The 3 AM Bug: A Field Guide

· 3 min read · Syed Omar Faruk Towaha
The 3 AM Bug: A Field Guide

Some bugs are polite. They show up during working hours, reproduce on the first try and come with a stack trace that points straight at the guilty line. This post is not about those bugs.

This post is about the 3 AM bug: the one that waits until you are asleep, your laptop is in another room and your on-call phone is set to the loudest ringtone known to science.

Incidents by time of day
Our team's unofficial log. Night shifts are where bugs go to party.

Species 1: The Cron Job Goblin

Lives in scheduled tasks that run at midnight, 2 AM or "whenever traffic is low." It works for months, then one night the data is twice as big, the job runs into the morning backups and both fall over.

How to catch it: log the duration of every scheduled job and alert when it grows, not just when it fails. A job that takes 20% longer each week is sending you a letter. Read it.

Species 2: The Time Zone Phantom

Appears twice a year during daylight-saving changes, and once every four years on 29 February. Symptoms include reports that "skip" an hour, invoices dated tomorrow, and a user in St. John's, Newfoundland, discovering that their time zone is UTC−3:30 and your code assumed whole hours.

from datetime import datetime, timezone

# store in UTC, always
created_at = datetime.now(timezone.utc)

# convert only when showing it to a human
local = created_at.astimezone(user_tz)

How to catch it: store UTC, test with weird zones (Newfoundland, Nepal at +5:45, India at +5:30), and run your test suite with a fake clock set to a DST transition.

Species 3: The Slow Leak

Memory usage climbs by 1% an hour. Nobody notices during the day because deploys restart everything. Then a quiet weekend comes, nobody deploys, and on Sunday at 3 AM the process hits its limit.

How to catch it: graph memory over days, not minutes. Any line that only goes up is not a feature.

Species 4: The Certificate Cliff

An SSL certificate, an API key or a domain expires exactly one year after the person who set it up left the company. Expiry dates don't care about your sleep schedule.

How to catch it: put every expiry date in monitoring with alerts at 30, 14 and 7 days. Calendar reminders don't count; calendars get ignored.

What to do when it happens anyway

3 AM triage
Restore service first, understand it second.

At 3 AM your brain works at roughly the level of a tired golden retriever. Don't debug heroically. Check whether it's really down, look at what changed recently, roll back, and go back to bed. The root-cause analysis can happen after breakfast, when you're a person again.

Then write a short, blameless note: what broke, how you noticed, what will stop it happening again. The goal is not to never get paged. The goal is to never get paged by the same bug twice.

// related

// prefer the terminal?

Open the terminal blog and type read the-3am-bug-field-guide.