Databases in production
A new series with a different format: short posts, one database practice each, all pulled from things that actually break under load. Connection leaks, invisible N+1s, migrations that lock tables, tasks that run against data that does not exist yet.
- databases
- python
- backend
My GitLab CI series was a long-form, build-it-up-from-scratch course. This one is the opposite: short posts, one specific practice each, no filler. Each part should take about five minutes to read and leave you with one thing you’ll do differently.
The theme is databases in backend web services, the layer behind the scariest production incidents. Not because databases are fragile, but because most database bugs share a nasty property: they are invisible in local development and only surface under load. One request at a time, everything works. A thousand concurrent requests, and the service is down while the database sits there idle.
Every post in this series is about a bug like that:
- A connection leak that looks exactly like a database outage.
- An N+1 that ships to production because dev data was 10 rows.
- A migration that takes a lock and queues every request behind it.
- A background task that runs against data that doesn’t exist yet.
The examples use the stack I work with daily — Python, FastAPI, SQLAlchemy, GraphQL, PostgreSQL — but almost every practice translates directly to any language and ORM, because the underlying rules belong to the database, not the framework.
One more thing about where these posts come from. I’m doing structured backend practice: for each of these traps I build a small broken program, fix it under a test that proves the fix, and only then write the post. So everything here has been broken and repaired on purpose before being explained. If a post contradicts your experience, I want to know.
Part 2 lands next week: a connection leak that can take down a whole service, and why cleanup belongs in a finally.