press any key to skip

Vinit Agarwal — Distributed Systems Engineer based in Bangalore, India

Distributed Systems Engineer·Bangalore, India

Vinit Agarwal

I design and own distributed systems end to end — architecture through production — so they stay correct at scale and affordable to run.

Mostly:

Without the jargonAlmost everything I build is invisible. It's the machinery underneath an app that decides what happens next, makes sure it happens exactly once, and keeps the running cost from growing faster than the business does.

How I workPlay something instead
  • 10M+interactions a daysystems I designed
  • 9services ownedempty folder to production
  • 8+integrationsbehind one interface
  • 36%of my commitswere refactors, deliberately

How I work

Five things that show up in everything I build.

I'm a backend engineer in Bangalore. I like the parts of a system that other people would rather not think about — what happens when a message arrives twice, when a vendor goes quiet mid-send, when a queue backs up at the worst possible hour. I've spent the last two years designing that layer for a platform that handles millions of interactions a day, and I own it from the architecture diagram to the 2am page.

Each one reads twice: first in plain language, then the same idea for engineers. Skip whichever half isn't yours.

01

I design the thing other people call

The best compliment my work gets is that nobody has to think about it. If a teammate has to understand eight different vendors to send one message, I've done my job badly.

In practice

Static contracts over conditionals. Implementations selected at runtime by key, each returning a typed model that the pipeline normalises into one standard vocabulary. Adding a provider is an implementation and a mapping — never a change to routing.

8+ integrations, one dispatch path, zero vendor branches upstream

02

I assume it breaks, and I want to know which break it was

Networks drop, vendors go down, users tap send six times. None of that is unusual — it's Tuesday. What matters is that the system can tell the difference between 'that genuinely failed' and 'try that again', because one of those is a refund and the other is a duplicate charge.

In practice

Idempotency enforced at the worker, not hoped for. A strict error taxonomy separating terminal vendor outcomes from retryable system failures. Exponential backoff with bounded retries, atomic cancellation under row-level locks, and explicit state machines wherever two jobs can touch the same row.

retryable ≠ failed · at-least-once delivery · races resolved, not documented

03

Scale is a budget, not a bragging right

Making something work for a million people is easy if you don't care what it costs. The interesting problem is doing it on a bill the business can actually pay — and that usually means not asking the database the same question twice.

In practice

Roughly 80% of per-job context reads served from Redis rather than Postgres, keyed per tenant and invalidated on write. Query hardening that cut 40+ seconds off report execution, N+1 elimination in paginated status reporting, and throttling shaped to each vendor's ceiling so throughput is buffered instead of burned.

−40s query time · ~80% cache hit rate · ~20% cloud spend removed

04

Zero to one, and then I stay

I've started a fair few services from an empty folder — and I'm still the person who gets called when they misbehave. Building it is the easy half; living with it is where you learn what you actually designed.

In practice

Bootstrapping end to end: Docker, dependency management, async database pooling, CI/CD, deploys, then production support. Nine services across ingress, core orchestration, egress gateways and a realtime agent runtime — most of them mine from the first commit.

9 services · 14 months · still on call for them

05

I go back and fix what I built

Code I wrote a year ago was written by someone with less information. I'd rather return to it than defend it.

In practice

In the core service, 36% of my commits were pure refactoring — extracting shared schema libraries, collapsing duplicated parsers into one polymorphic pipeline, and paying down debt I created myself while moving fast.

36% of commits: refactors, not features

The piece of design work I'm proudest of

One interface. Eight vendors. Nobody upstream can tell them apart.

Eight different companies carry these messages, and every one of them speaks its own language, checks your identity differently, and has its own opinion about how fast you're allowed to go. I built the translator that sits in the middle — so everyone else on the team gets to pretend there's only one.

The real job here was never sending messages, it was abstraction. I designed a framework that normalises every provider's routing, payload shape, auth scheme and delivery semantics behind one static interface. Systems that use it don't decide anything: they call it, and the orchestration is handled underneath.

  • 8+integrations behind one interface
  • 0vendor branches in the core pipeline
  • daysto add a provider, not months
  • config onlyto enable an existing one
The contract

Each integration is free to do whatever its vendor demands internally — its own auth dance, its own endpoints, its own parsing. What it is not free to do is invent an output. Every implementation returns a defined Pydantic model, and that typed object is what the pipeline processes onward into the standard format the rest of the application already understands. A voice integration reporting call_completed becomes ANSWERED before it ever leaves the boundary; nothing upstream has ever heard of call_completed. That translation is the whole point of the abstraction.

Vendor vocabulary → system vocabulary
  • completedcall_completedANSWEREDANSWERED
  • deliveredDELIVRDstatus=2DELIVERED
  • RNRNOANSWERno_answerFAILED · NO_ANSWER
  • invalid_numberBLOCKEDcarrier_rejectFAILED · terminal
  • timeoutqueue droppre-call errorDROPPED · retryable
01

A single static contract

Every provider implements the same interface — dispatch, template resolution, delivery-receipt normalisation, error mapping. The implementation is selected at runtime by key, and the core pipeline holds no vendor branches at all.

02

Vendor methods in, typed models out

An integration may parse XML, chase a rotating session key or unwrap four levels of envelope — its business. It must return a defined Pydantic model, which the pipeline then processes into the one standard format every other part of the system reads.

03

Provider-wise mapping to standard system variables

Eight-plus integrations, each with its own dialect, get mapped provider-by-provider onto one canonical set of system variables and one processing path. Downstream code reads the same fields no matter who sent them.

04

Orchestration lives below the line

Batching, throttling to each vendor's TPS ceiling, retries, callback correlation and status roll-up all happen inside the framework. The caller says what to send, never how to send it.

05

Plug and play, both directions

A brand-new provider is an implementation plus a config model and a mapping — nothing in the routing changes. An already-integrated provider for a new client is credentials in a config screen: zero code, zero deploy.

The path so far

Where I learned all this.

Three years of backend work — the last two on a platform where the failure modes involve other people's money, which is a fast way to learn what "reliable" is supposed to mean.

  • Designed from scratch the plug-and-play communication framework behind 10M+ API calls/day — 8+ provider integrations whose routing, payload shapes and delivery semantics are normalised into one static interface, so calling systems orchestrate nothing.
  • Built the DLR and callback pipelines and a provider-agnostic template platform on top of that contract.
  • Architected a distributed debounce system on Redis pipelines and atomic TTL ops, collapsing message bursts into single processed events across instances with no sticky sessions.
  • Built a tenanted central schedule manager as the single source of truth for Celery Beat, with idempotent workers that self-terminate and purge cancelled jobs.
  • Built the reporting and analytics subsystem from scratch — generation, email delivery, scheduling CRUD under atomic transactions, Metabase embedding; query hardening cut 40+ seconds of execution time.
  • Built n8n automation workflows with AI integrations cutting manual overhead by ~80%, with multi-tenant isolation via M2M auth and secrets resolved at runtime through a provider-agnostic manager.
PythonDjangoFastAPICeleryRedisAzureK8s

Education

B.Tech, Computer Science & Engineering

Kalinga Institute of Industrial Technology (KIIT)

2021 — 2025

  • Flipr Hackathon — Merit Certificate
  • Cyber Security Workshop, IIT Kharagpur
  • Cisco Networking Academy — Networking & Cybersecurity

Toolbox

Async Python, at volumeNon-blocking I/O is the default, not an optimisation.
asyncioaiohttpFastAPISocket.IOconnection poolingCelery
Design patterns, used in angerChosen because the alternative hurt, not because of a book.
Strategy + RegistryTemplate MethodDTO-first APIspolymorphic schemasshared contract libraries
CorrectnessWhere most of my thinking actually goes.
idempotencyrace conditionsstate machinesPydantic validationDecimal for moneyerror taxonomies
Storage & transportPicked per problem, hidden behind an interface either way.
PostgreSQLRedisDynamoDBCosmos DBAzure Service BusMongoDB
Ship it and keep it upThe half of the job that happens after the merge.
DockerKubernetesKEDACI/CDstructured loggingKey Vault
Also fluent inEnough to build the thing myself when there's no one else.
DjangoNode.jsReactNext.jsC++n8n

Engineering notes

Problems I've had to think hard about.

Distributed systems come down to a few stubborn questions: who owns this piece of state, what happens if it runs twice, and what do you do when the other side stops answering. Six times I had to write a real answer rather than a hopeful one.

Problem

A user fires six WhatsApp messages in four seconds. Three server instances each pick one up. Each thinks it's the whole conversation.

What I built
  • Redis pipelines plus atomic TTL operations collapse a burst into a single processed event, cluster-wide.
  • The lock is the coordination point, so no instance needs to own the user — no sticky sessions, no session affinity in the load balancer.
  • Late arrivals extend the window instead of spawning a second job, so the agent replies once to the whole thought.

1 event per burst · N instances · 0 affinity

Problem

Eight vendors, eight auth schemes, eight payload dialects, eight TPS ceilings. The naive version is an if/elif chain that every future engineer is afraid of — and every new client stalls a deal for months.

Problem

End-of-day allocation pushes arrive as one enormous array. One vendor accepts 100 TPS, another accepts 5 and silently bans you for asking twice.

Problem

'Failed' is not one thing. An invalid number and a network timeout look identical in a log and mean opposite things for retries and for billing.

Problem

Modules writing directly to Celery Beat means cancelled jobs that keep firing, duplicated schedules, and no tenant boundary.

Problem

Every enterprise client wants slightly different columns, filters and delivery times. Encoding that in code means a sprint per client.

The arcade

Or just play with the ideas.

I find things easier to explain when you can poke them. Two games and three sandboxes, all written from scratch — each one a toy version of something I had to get right for real. No engineering background needed to enjoy losing at them.

Packet Router

Messages arrive on one ingress and every one belongs to a different provider. Route each packet to the right gate before it hits the floor — three drops and your SLA is gone.

score0
combo×0
routed0
sla●●●
best0

Packet Router

Match each packet to its provider gate. It speeds up as throughput climbs.

◀ / 1 — WhatsApp▼ / 2 — Voice▶ / 3 — SMS

Why this exists: the real thing does this 10M times a day, except the gates have different rate limits, different auth and different payload shapes — so the router never learns their names. It looks up an implementation by key and hands the packet over.

Selected work

Things I've built.

One of these runs in production today. The others are from the years when getting anything to work at all still felt like magic — I keep them here because that part matters too.

Reconect.ai · 2025–2026

Communication Platform

A dynamic template engine that resolves placeholder variables from live DB context at send time — every message assembled per recipient, with provider-specific section constraints, trackable payment links and tenant-level caching with fallback.

PythonRedisMulti-tenant
Explore the systemsOpen the simulator
Open source · CMS

FED Website

Official website of FED with an in-built CMS, deployed on AWS.

MERNAWS
Computer vision

Smart Attendance

OpenCV face-recognition attendance system deployed to 100+ users; SQLite access-pattern tuning cut request latency by 20%.

PythonOpenCVSQLite
Audio publishing

Podcast Platform

End-to-end platform for uploading and publishing audio; MongoDB schema work improved durability by 50%, verified across 100+ test runs before go-live.

Node.jsMongoDBDigitalOcean

Get in touch

If you're building something
that has to stay up.

I'm happiest near the queue, the retry policy and the part everyone else is quietly nervous about. Whether you're hiring, building something together, or just want to argue about abstractions — I read everything and I reply.

Based inBangalore, India
Open toRoles, collaborations, and problems that sound hard
GitHub LinkedIn Medium

Goes straight to my inbox. I reply to everything that isn't a recruiter template.