How we built multi-region ingestion with a 48 ms p99

The contract of an ingest endpoint is short: accept the request, make it durable, return 2xx before the provider gives up. Providers have very different patience, from Shopify's five seconds to GitHub's ten, but all of them treat a slow response as a failure and start retrying, which multiplies load exactly when you are already struggling.

Our first design wrote every event to a single primary database before responding. It was simple and correct, and it fell over the first time a large customer replayed a month of Stripe history. We moved to a log-first architecture: ingest nodes append to a replicated log in their own region and acknowledge once two replicas confirm the write. Everything else — verification, transformation, delivery — reads from that log asynchronously.

Running in us-east, eu-west and ap-south introduced data residency questions. Each workspace has a home region, and events are only ever persisted there. If an event arrives at a different region's edge, it is forwarded over our private backbone before being acknowledged, adding around 70 ms in the worst case but keeping the residency guarantee absolute.

Today p99 ingest latency, measured from the first byte received to the response leaving our edge, is 48 ms across all regions. The full write-up of our log layer, including the failure modes we tested with fault injection, will be published next month.