Distributed Systems news and engineering summaries

The article argues teams should generally start with a monolith, even when they expect to need microservices. This avoids premature operational complexity, speeds user feedback, and lets teams discover stable service boundaries. It outlines gradual extraction, replacement, and coarse-grained service options, while noting that starting with microservices may suit experienced teams and system-replacement projects.

Iroh 1.0 has launched, introducing a networking paradigm that replaces IP addresses with cryptographic keys. This stable release supports direct, secure device connectivity, local-first operation, and multi-path routing across platforms including Python, Node.js, Kotlin, and Swift. Iroh aims to simplify global connectivity by making the internet function like a secure, efficient, and direct localhost.

AX is a declarative, high-throughput orchestrator for running autonomous agent workloads at scale on Kubernetes. Built on Agent Substrate, it sandboxes tasks, provisions Git/MCP workspaces, restricts network egress through Gateways, configures LLM providers, and supports monitoring, SSH debugging, checkpointing, suspension, and resumption. A kubectl-shaped CLI applies YAML manifests through a gRPC control plane. AX is pre-stable and may introduce breaking changes.

In this detailed overview, Harald Schäfer, CTO of comma.ai, advocates for companies to build and operate their own data centers rather than relying on cloud providers. He argues that self-hosting fosters better engineering incentives, provides greater control over infrastructure, and is significantly more cost-effective for consistent workloads like ML training—estimating a savings of $20M compared to cloud equivalents. The post details the technical architecture of comma's $5M facility, which features 600 GPUs across 75 in-house 'TinyBox Pro' machines, a custom outside-air cooling system, and 4PB of SSD storage. On the software side, comma utilizes tools like Slurm for workload management, PyTorch for training, and custom open-source solutions like 'minikeyvalue' for distributed storage and 'miniray' for task orchestration. This level of vertical integration allows for efficient model training and rapid code iteration within a streamlined, monorepo-based environment.

The Artemis II mission features a highly fault-tolerant computing architecture designed for deep space. Unlike the Apollo era, the Orion capsule uses a deterministic, time-triggered system with eight CPUs running in parallel. This 'fail-silent' design and dissimilar Backup Flight Software ensure resilience against radiation-induced errors and hardware failures, setting a new standard for mission-critical reliability.

Anthropic built Scion, an in-house source control system, and migrated its monorepo from git in five months as merged changes grew from 2,500 to over 30,000 per day. Designed for always-online agents, Scion uses server-side changesets, content-addressed caching, API-based queries, and sparse or virtualized checkouts instead of fully local repositories. Its infrastructure uses 200 times less CPU and memory, makes CI and research checkouts two orders of magnitude faster, and cuts CI CPU use by 25%. A seven-person team built it with Claude.

Cloudflare optimized Pingora Backend Router’s consistent hashing by compacting Rust data structures and reducing per-server hashes 90% without meaningful accuracy loss. A staged, data-center-scoped migration prevented cache disruption and origin overload. The changes cut PBR memory usage by 100TB globally, while pingora-ketama now exposes the improvements through its v2 ring feature.

WebAssembly progress has expanded its core capabilities, yet it remains a 'second-class' web citizen due to its reliance on complex JavaScript glue code for loading and API access. The WebAssembly Component Model aims to bridge this gap, allowing direct Web API interaction and better cross-language interoperability without JavaScript overhead.

GitHub is scaling its infrastructure and enhancing reliability to handle the rapid surge in agentic development workflows. To minimize service disruptions, the team is prioritizing system isolation, cloud migration, and performance optimization. Recent incidents involving merge queues and search failures have prompted deeper process improvements and a stronger commitment to transparent incident communication with the developer community.

The author critiques OpenAI's reliance on WebRTC for Voice AI, arguing that the protocol's aggressive packet dropping and complex setup create unnecessary challenges. They suggest that WebSockets or QUIC are superior alternatives, offering better scalability, simpler load balancing through CONNECTION_ID, and improved stability compared to the outdated and restrictive WebRTC standards.

DuckDB v2.0 “Cyanoptera,” arriving this fall, expands DuckDB from an in-process database into a server-capable analytical platform. Key updates include Quack/CONNECT, first-class VARIANT, triggers, a new SQL parser and storage format, asynchronous I/O, major query and partition-pruning improvements, native timezone support, and a stable C API for portable, signed extensions.

AX is an open, declarative runtime for stateful AI agents, providing isolated Tasks, reproducible Workspaces, network-policy Gateways, and centralized Model configuration. Built on Agent Substrate, it supports billions of lightweight actors, sub-second suspend/resume, dense resource sharing, generative environment setup, and large-scale agent research, evaluation, and production deployment.

A database migration caused significant replication lag on May 12, 2026, leading to delayed Code Scanning, notifications, and webhooks processing. The issue, which lasted several hours, was resolved by scaling up processing workers. Future measures include creating dedicated worker pools for high-usage queues to maintain system stability and performance.

bitchat is an innovative decentralized peer-to-peer messaging application designed to function entirely over bluetooth mesh networks, eliminating the need for internet connectivity, centralized servers, or phone numbers. By utilizing ad-hoc communication, each device serves as both a client and a server, discovering peers and relaying messages across multiple hops. This infrastructure-independent approach provides strong resistance against censorship and surveillance, making it highly effective during internet outages, natural disasters, or protests. The project is open-source and available for both ios and android, ensuring full protocol compatibility between different mobile operating systems and hardware platforms.

liquidslr/system-design-notes is a work-in-progress repository of study notes for System Design Interview - An Insider's Guide, Volumes 1 and 2. It organizes guidance on designing scalable distributed systems—from rate limiters, key-value stores, and URL shorteners to chat, storage, payments, and stock exchanges—across 28 chapters. Links to Pagefy and additional technical resources distinguish it by extending the book-based notes with relevant papers, architectures, and engineering references.

The article clarifies that atproto does not use "instances" like Mastodon. Instead, it decouples hosting from applications, similar to how RSS works. Users own their data independently from apps, allowing them to switch hosting providers or use different clients without losing their identity. This architecture shifts decentralization from federated silos to a more flexible, open data-aggregation model.

PgDog is an open-source proxy project designed to make Postgres horizontally scalable, supporting over 1 million queries per second and 100 TB+ datasets. Founded by experienced infrastructure engineers backed by $5.5M in funding, it aims to eliminate scaling challenges for Postgres while providing flexible deployment options across cloud and on-prem environments.

The article explores 'social computing' through the lens of a distributed social filesystem, drawing a parallel between traditional personal files and social data like posts, likes, and follows. By using the AT protocol, user data is lifted out of siloed applications and stored in 'repositories' as structured JSON 'records' defined by 'lexicons'. This paradigm shifts the perspective of applications from data owners to reactive views that materialize snapshots of user data. Key concepts include decentralized identity (DID) to ensure links remain permanent during hosting migrations and 'at://' URIs for universal record identification. This ecosystem allows developers to build interconnected apps, custom feeds, and cross-platform tools that operate on a shared global data layer where the user remains the ultimate source of truth, fostering a more open and interoperable social media environment.

In January 2026, GreyNoise analysts observed a sudden and dramatic 65% drop in global Telnet traffic within a single hour, which eventually settled at an 83% reduction from the baseline. This structural shift preceded the public disclosure of CVE-2026-24061, a critical authentication bypass vulnerability in GNU Inetutils telnetd that allows unauthenticated root access via a simple argument injection. The data suggests that major Tier 1 transit providers likely implemented port 23 filtering on backbone infrastructure in anticipation of the vulnerability's disclosure. This proactive infrastructure-level response significantly impacted residential and enterprise ISPs while leaving major cloud providers with direct peering largely unaffected. The incident highlights a potential shift in how global network operators coordinate to mitigate high-impact security risks at the routing level before they can be exploited at scale.

Event-driven systems offer scalability but introduce complex challenges like schema versioning, observability hurdles, and failure management. Developers must implement distributed tracing, handle at-least-once delivery issues via dead-letter queues, ensure idempotency to prevent duplicate processing, and design for eventual consistency to manage distributed state across asynchronous architectures.