- We’re introducing ZGateway, the proxy we are using to unify traffic through ZippyDB, Meta’s most widely-used key value store.
- As a bonus, it also enables admission control, load balancing, cross-region resilience, and richer operations.
ZippyDB is the most widely used key value store at Meta, backing product metadata, counters, and configuration, and can serve billions of operations per second across a globally distributed fleet.
A previous post described how ZippDB works. This post is about the layer in front of it: ZGateway, the proxy through which we are unifying ZippyDB client traffic.
ZGateway grew out of the need to manage the sprawl of ZippyDB’s client fleet, and its value turned out to be structural. A ZippyDB client might be one of more than a million hosts, owned by hundreds of teams, that we cannot change quickly. A proxy sits in a different position entirely — in the path of many clients at once — and that shared vantage point lets it do what no single client could.
Every capability below is easier, safer, or only possible at all in one managed tier rather than in a million client binaries. We tell the story through the two clearest cases, connection management and request batching.
Why ZippyDB Needed a Proxy Layer
Proxies show up wherever a large, diverse client population talks to a shared backend: a connection pooler, a service mesh sidecar, a CDN edge, an API gateway. Interposing between many callers and a shared resource buys three things.
- It bounds the problem: The backend stops seeing the client population and starts seeing a fleet its own operators control
- It creates a home for shared work — pooling, retries, routing, caching, admission control — solved once by the team that knows the backend best, instead of in every client.
- It creates a control point: The only place you can see the whole workload, attribute load to whoever generated it, and change behavior in minutes rather than waiting out a fleet-wide client rollout.
The tradeoff is a hop and one more tier to operate; the trade pays off when the client population is large, diverse, and not yours to change. ZippyDB is an extreme case.
In the direct-access model, every ZippyDB client connects to every database host it needs. A single client can touch tens of thousands of distinct shards in a stable window; those shards sit across hundreds of thousands of database hosts. The result is a dense many-to-many mesh of TLS connections: A typical client holds tens of thousands of outbound connections, and a typical database host can accept tens of thousands of inbound ones.
That mesh is wasteful and fragile. Every open connection consumes memory, CPU, and a file descriptor on both ends, mostly while idle, and inbound connections grow with the client population, so every new client cohort makes every database host worse. And because each client manages its own pooling and failover, a sudden drop in connection reuse — a cohort restarts, a deploy rolls — hits the fleet with a storm of new connections; we have traced host crashes from file-descriptor exhaustion and OOMs to exactly this.
It is hard to fix client-side because two systems move at once. Client fleets keep changing pooling policies and expanding while the database fleet consolidates on its own schedule. Coupling them directly feels like jumping between two moving cars. A proxy decouples them, collapsing the mesh into two bounded hops — a win on efficiency, performance, scalability, and above all reliability.
That last one matters most. Under direct access a reconnection storm becomes a catastrophe. In one incident a routing bug led every client to open a connection per shard, hosts breached their file-descriptor limit, and the fleet fell into a reboot loop. With ZGateway in the path that storm is contained at the proxy tier — a fleet we control, observe, and can harden centrally. Connection management does not disappear behind a proxy; it moves to the one place we can solve it.
Direct access was the right design for the initial part of ZippyDB’s life. Fan-in scales with the client population, so the resource use grows with adoption: at a few thousand clients the mesh is an inefficiency, at scale it is a reliability limit. The means arrived on the same schedule — ServiceRouter feature improvements, Thrift overload protection, a thin client, and other features are what make a shared tier feasible, and building ZGateway earlier would have meant building each of those first.
What ZGateway Is
ZGateway is a stateless proxy tier between ZippyDB clients and the database (ZServer) fleet. It can handle greater than 1 billion operations per second and carries about 40% of all ZippyDB traffic, projected to grow past 60%, while adding only about 6% computational overhead to an average use case. It currently carries about 40% of all the ZippyDB traffic and is growing while adding only about 6% overhead to an average use case. It runs as regional tiers discovered through ServiceRouter, Meta’s hyperscale service mesh solution, keeping every client near its gateway, in two flavors sharing one pipeline: a pure proxy and a read-through cache. it runs our thick C++ client as its engine, one internal client per use case. ZGateway is a ZippyDB client run as a managed service, which made moving capability here natural.
A client sends a request over its sticky connection to a regional ZGateway host, which terminates TLS, authorizes it against the use case’s ACLs, and applies per-tenant admission control, validation, and shaping. ZGateway resolves the shard, checks the local cache on a caching tier, and batches and coalesces misses and writes with other in-flight requests for that shard before sending them to the correct replicas. Responses are demultiplexed back to the callers, with per-use-case metrics, traces, and quota usage recorded along the way.
The key property is the asymmetry of connection counts. Each client needs only a sticky pool to its regional ZGateway hosts, and each ZServer sees connections only from the ZGateway fleet, whose size we control. Some responsibilities deliberately stay put — TLS in the Thrift/ServiceRouter stack, key-to-shard mapping in the shard locator, replica selection and hedging in the embedded client. ZGateway owns traffic management, not a reimplementation of the database client.
The Fan-In/Fan-Out Reduction
Here is the arithmetic:
Model the fleet as balls into bins. Throw B balls (shards a host touches) into H bins (hosts) and count distinct bins hit.
The expected number is:
And a specific bin is hit with probability:
Fan-out is distinct bins hit; fan-in is that probability times the caller population. With round mock figures — 20 regions, 500,000 database hosts, 30,000 proxy hosts, 1,000,000 clients, 50,000 shards per client — the model yields:
The connections don’t vanish; they move to the tier built to hold them. End to end, total persistent connections still drop by roughly 19x, because each backend connection multiplexes many clients.
But the one-time reduction, however dramatic, isn’t the real point. The real point is the change in scaling behavior. In the direct model, database-host fan-in is H_client · p — linear in the client population, so every new cohort makes every database host worse. With ZGateway the client population drops out entirely: fan-in reduces to approximately R · S_host, regions times shard density per host, independent of both fleets. The only lever left is shard density, which we own. An unbounded number driven by everyone else becomes a bounded number we control.
Collapsing the Request Stream: Batching and Coalescing
Because ZGateway sits in the path of many clients it can do what no client library can – combine work across unrelated callers. A shared batcher on each host groups requests headed for the same destination, keyed by use case and physical shard, and merges them into one backend RPC.
It also coalesces. If several callers want the same key at the same moment, the gateway fetches it once and fans the result out. A client-side batcher can only merge its own process’s requests; the gateway collapses them across clients.
Every RPC carries fixed overhead regardless of size — Thrift serialization, shard lookup, authorization, syscalls — so folding many operations into one amortizes all of it, meaning fewer, larger backend requests, lower QPS and CPU, and steadier load as the linger window smooths micro-bursts. And because a use case is billed by the QPS it sends, batching stretches its rate-limit budget, cutting throttling with no work on its side.
Two of batching’s benefits aren’t about efficiency at all. The first is what coalescing does under a hot key. Thousands of simultaneous callers collapse into a single backend read, so a hot key can never become a stampede against one replica.
The second is what it lets us remove. For years, customers who wanted batching ran client-side libraries that were fragile, CPU-hungry, individually tuned, and a steady source of incidents because that complexity lived in a million binaries we didn’t control. A shared batcher batches across clients, which those libraries cannot — and let us retire them.
Each request is parked in an in-memory batch that flushes when a linger window elapses, the payload crosses a size limit, or the request count hits a cap, so added latency stays bounded; oversized or just-migrated batches fall back to individual sends. Holding requests in memory is an OOM risk, so batching ships with two safety mechanisms.
Idle eviction handles slow-burn growth: batch-map entries idle beyond a TTL are erased on the next flush. An in-flight cap handles acute overload. When the backend slows, coroutines running flushed batches pile up faster than they drain, so the cap rejects new executions once the count crosses its limit. Steady hygiene plus an acute safety valve is what makes batching safe by default.
How ZGateway Evolved
Once traffic flows through one tier, it becomes the natural home for capabilities every client would otherwise reimplement. Batching is the clearest example; here are the rest.
Traffic Routing and a Safe Migration Path
Moving traffic onto a proxy is a high-stakes migration. It must be incremental, reversible, and scoped. Routing to ZGateway is controlled by client-side configuration flags scoped per service and shard prefix: a percentage knob ramps eligible traffic, a region filter limits blast radius, a global kill switch gives instant rollback. It is pure configuration, with no client code change, which makes the rollout controllable in real time.
Tenant Isolation and Admission Control
A shared tier serves hundreds of use cases, so one misbehaving tenant must not starve the others. ZGateway’s defense is Discriminant Load Shedding (DLS). Every request maps to a per-tenant bucket, keyed by use case and split by priority, and buckets drain round-robin. When a tenant floods the tier, its own bucket fills and its excess is shed while every other bucket keeps draining — isolation as a property of the structure, not of luck. In front of DLS, a CPU concurrency controller uses an AIMD loop to adjust how fast the shared token bucket admits work; a memory handler guards against OOM the same way.
The shedding stays discriminant. In a controlled overload at >90% CPU across roughly 1,350 active tenant buckets, only 6 — the actual noisy neighbors — were shedding; the other ~1,344 executed 99.9% of their requests with zero rejections, goodput held near 97–98%, and the machinery cost about 8% of CPU.
Read Caching With Live Invalidation
On a cache tier, hot reads are served from an in-process cache; on a miss the gateway takes a per-key fill lock, so a thundering herd for one key collapses into one backend fetch. Freshness comes from a change-data-capture stream of write and checkpoint events that invalidates or refills affected entries, within an explicit bounded-staleness contract, and each host owns a slice of the keyspace by consistent hashing. The payoff is substantial read offload from storage, at lower latency, without losing correctness.
Load Balancing Across the Tier
Because ZGateway is stateless, any request can be served by any host in a regional tier, so we can steer traffic to even out load. The tier is not uniform. It mixes ~26-core to ~126-core hosts, and a large task replacement can reshuffle capacity in minutes. Equal treatment of unequal hosts produces hot outliers, and a hot ZGateway host is what turns into error-rate spikes and ServiceRouter throttling. Since ServiceRouter routes by weighted consistent hashing, the lever is the right weight per host.
A control-plane balancer computes them. On a fixed cadence it reads each host’s recent CPU utilization, normalizes the tier average to 1.0, and nudges each weight opposite to its load. Guardrails keep it stable — adjustments are damped and clamped, the distribution is re-centered on a target median so weights don’t drift toward zero, and a change throttle moves only the most-imbalanced hosts each run, limiting shard reshuffling (costly on cache tiers, where moving a weight means moving keys). New hosts start with weight scaled to hardware capacity.
The lesson is that one fixed policy cannot serve both a calm tier and a tier in shock. So the balancer is becoming adaptive, classifying each tier’s state — steady drift, task churn, flat initial weights, bimodal load, hot outliers, regional skew — and applying a matching policy.
Cross-Region Resilience
For most of its life ZGateway was strictly regional, failover stayed inside a region. Great for latency, but when an entire region’s tier comes under pressure, requests queue and time out locally while healthy capacity sits idle next door. Because ZGateway sits on ServiceRouter, we can let routing cross region boundaries in a controlled way, via three mechanisms:
- Global routing builds a routing table spanning regions, so a saturated local tier fails over to a healthy one instead of dying at home.
- Mega-regions group geographically close regions into one locality, so overflow spills nearby and keeps most of the latency benefit.
- Rings declare exactly which regions back each other up, and in what proportion.
Each is enabled per tier and region behind a percentage knob. The failover signal mattered as much as the routing. A simple regional CPU average smooths over exactly the hot conditions we need to catch, so failover keys off a sharper measure, tuned to fire before a region tips into overload rather than after.
Transactions and Richer Operations
A transaction needs somewhere to keep its client-side bookkeeping: read set, scanned ranges, pending writes. Historically that lived in the thick client. When ZGateway moved customers onto a thin client, it had to move onto the gateway. The first cut left two parallel implementations — a bespoke store built for ZGateway alongside the in-memory path our engine already used — two versions of the most correctness-critical part of the flow.
We consolidated onto one, behind a flag, in nine phases up to the highest-volume regions, reaching 100% of transaction traffic with no reliability regression. Sharing that path with the engine keeps ZGateway in lock-step with server-side transaction evolution: evolve a capability once, inside the tier, and every client inherits it.
Operating ZGateway in Production
ZGateway runs as a large volume of servers across dozens of regions, in a handful of tiers by workload – one large general-purpose tier for the long tail of use cases, dedicated tiers for our largest customers, and a separate high-throughput proxy tier. They differ in footprint and size by more than an order of magnitude, and individual tiers aren’t uniform either, largely because of stacking. Multiple tasks are packed onto one machine at varying densities, next to full-size dedicated hosts. Across all of it, ZGateway exposes rich per-use-case observability — the visibility that makes the admission control and load balancing above safe on shared infrastructure.
What’s Next for ZGateway
The near-term trajectory, to unify all ZippyDB traffic through ZGateway, is unchanged. The more interesting question is what a universally-adopted gateway makes possible. Three directions stand out, and they all share a common theme – ZGateway both sees the most and decides the most.
Agent-operated heuristics. Almost every capability here is governed by a control loop and hand-tuned knobs: load-shedding bucket sizes and CPU thresholds, balancer parameters, failover triggers, batch flush windows, cache staleness bounds. Today those are tuned by humans and nudged by crons; the adaptive balancer above is already an agent in all but name. The next step is to make it explicit and expose those heuristics and internal state as a structured control surface, and let AI agents watch the same telemetry we do — diagnosing tier state, attributing an incident to a noisy tenant, and applying remediation behind guardrails faster than any oncall could.
Co-location. ZGateway is a distinct tier, which costs an extra network hop and a few percent of overhead. For latency- or efficiency-critical workloads we can push part of the gateway down beside the ZServer host, so the gateway↔server leg becomes a local call while the control plane stays central. The trick is doing it without re-coupling the fleets we deliberately decoupled. The connection-management and admission-control front stays a shared regional tier, and only what benefits from data locality moves down.
A multi-process gateway. ZGateway runs many distinct responsibilities in one process, so one tenant’s memory blowup can threaten everything on the host. Splitting it into cooperating processes — a connection/TLS front-end, request workers, separate cache and transaction components — buys hard fault isolation and an independent lifecycle. This is also complemented by agent-operated heuristics and co-location agents can manage the process fleet on a host, and co-location gets cleaner when the data plane is already its own placeable process.
Together these turn ZGateway from a smart tier into a programmable one with control decisions made by agents, a footprint that moves to where the work is, and failure domains isolated by construction.
