Operations

SREs split into two camps, and neither one is fully right. The first camp matured out of operational IT. They’ve lived through a bad patch Tuesday, a driver update that bricked a fleet, a vendor hotfix that took down a cluster. Their instinct: let a patch cycle pass before you touch it. Let someone else find the landmine. This dates me, but is the world I learned in myself. The second camp never did operational IT work. They came up through cloud-native, IaC, CI/CD. They don’t have the scar tissue, but they inherit the doctrine anyway, because the first camp is louder and… has tenure. ...

When you inherit a legacy AWS environment with dozens of Lambda functions and a mandate to clean up unused resources, the obvious first step is checking CloudWatch metrics. Invocation counts, error rates, duration stats - all there in neat graphs. But what happens when those metrics tell you a function has zero errors, yet it’s failing on every single execution? This week I investigated 45 Lambda functions to determine which could be safely deleted. What started as a simple metrics review turned into a forensic investigation revealing broken functions masked by misleading CloudWatch data, API Gateway routes pointing to deleted Lambdas, and scheduled jobs silently failing for months. ...