AWS Lambda Cleanup: When CloudWatch Metrics Lie

When you inherit a legacy AWS environment with dozens of Lambda functions and a mandate to clean up unused resources, the obvious first step is checking CloudWatch metrics. Invocation counts, error rates, duration stats - all there in neat graphs. But what happens when those metrics tell you a function has zero errors, yet it’s failing on every single execution?

This week I investigated 45 Lambda functions to determine which could be safely deleted. What started as a simple metrics review turned into a forensic investigation revealing broken functions masked by misleading CloudWatch data, API Gateway routes pointing to deleted Lambdas, and scheduled jobs silently failing for months.

The Problem with CloudWatch Error Metrics

The investigation started with a straightforward approach: pull invocation and error counts for the last 30 days.

aws cloudwatch get-metric-statistics \
  --namespace AWS/Lambda \
  --metric-name Errors \
  --dimensions Name=FunctionName,Value=example-function \
  --start-time $(date -u -v-30d +%Y-%m-%dT%H:%M:%S) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
  --period 2592000 \
  --statistics Sum \
  --region us-east-1

Several functions showed promising results: 30-90 invocations, zero errors.Keeper candidates. But when I spot-checked the actual CloudWatch Logs, reality hit:

2026-01-06T23:04:18.641Z ERROR Error: Requests from referer <empty> are blocked.
2026-01-06T23:04:18.644Z ERROR Error: Requests from referer <empty> are blocked.
2026-01-06T23:04:18.659Z ERROR Error: Requests from referer <empty> are blocked.
[15+ more identical errors]

The function was failing every single day. So why did CloudWatch report zero errors?

Understanding CloudWatch Error Metrics

CloudWatch’s Errors metric only counts unhandled exceptions that terminate the Lambda execution. If your code catches errors and logs them, CloudWatch considers the execution successful. The function in question was a daily scheduled job that:

Attempted to fetch data from Contentful API
Got blocked by API security (missing referer header)
Caught the error in a try/catch block
Logged the error to CloudWatch
Returned successfully with status 200

CloudWatch saw: successful execution, no uncaught exceptions, zero errors. Reality: complete failure to accomplish its purpose, every single day for months.

Building a Proper Investigation Process

Here’s the methodology I developed to actually determine Lambda health:

Step 1: Get Invocation Counts

Start with basic metrics, but treat them as hints, not truth:

aws lambda list-functions --region us-east-1 \
  --query "Functions[].FunctionName" \
  --output text | while read func; do
    echo "=== $func ==="
    aws cloudwatch get-metric-statistics \
      --namespace AWS/Lambda \
      --metric-name Invocations \
      --dimensions Name=FunctionName,Value=$func \
      --start-time $(date -u -v-30d +%Y-%m-%dT%H:%M:%S) \
      --end-time $(date -u +%Y-%m-%dT%H:%M:%S) \
      --period 2592000 \
      --statistics Sum \
      --region us-east-1 \
      --query 'Datapoints[0].Sum'
done

This gives you the landscape: which functions are dormant (0 invocations = safe delete) versus active.

Step 2: Check Actual Logs

For any function with invocations, check the logs:

# Look for error patterns in the last 7 days
aws logs tail /aws/lambda/function-name --since 7d --region us-east-1 | grep -i error

# Or use filter patterns for specific errors
aws logs filter-log-events \
  --log-group-name /aws/lambda/function-name \
  --filter-pattern "ERROR" \
  --start-time $(($(date +%s) - 604800))000 \
  --region us-east-1

I found multiple “healthy” functions were actually broken.

All reported zero errors in CloudWatch metrics. All completely non-functional.

Step 3: Identify Trigger Mechanisms

Functions fail differently depending on how they’re triggered. You need to know what’s calling them:

# Check EventBridge schedules
aws events list-rules --region us-east-1 --query "Rules[?State=='ENABLED']"

# Check which rules target specific lambdas
aws events list-targets-by-rule --rule rule-name --region us-east-1

# Check Lambda permissions (API Gateway, SNS, etc)
aws lambda get-policy --function-name function-name --region us-east-1

# Check event source mappings (SQS, DynamoDB Streams, Kinesis)
aws lambda list-event-source-mappings --function-name function-name --region us-east-1

This revealed three types of functions:

HTTP-triggered (API Gateway): Need to check API access logs
Scheduled (EventBridge): Check execution logs for patterns
Event-driven (SQS/SNS/Streams): Check dead letter queues

Step 4: Enable API Gateway Logging

For HTTP-triggered functions, API Gateway access logs show who’s calling them and whether they’re succeeding:

# Create log group
aws logs create-log-group \
  --log-group-name /aws/apigateway/api-name \
  --region us-east-1

# Update API Gateway stage to enable logging
aws apigateway update-stage \
  --rest-api-id api-id \
  --stage-name prod \
  --region us-east-1 \
  --patch-operations \
    op=replace,path=/accessLogSettings/destinationArn,value=arn:aws:logs:region:account:log-group:/aws/apigateway/api-name \
    op=replace,path=/accessLogSettings/format,value='$context.requestId $context.identity.sourceIp $context.requestTime $context.httpMethod $context.path $context.status'

Note: You’ll need to set up the CloudWatch role for API Gateway first:

# Create IAM role for API Gateway
aws iam create-role \
  --role-name APIGatewayCloudWatchLogsRole \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {"Service": "apigateway.amazonaws.com"},
      "Action": "sts:AssumeRole"
    }]
  }'

# Attach logging policy
aws iam attach-role-policy \
  --role-name APIGatewayCloudWatchLogsRole \
  --policy-arn arn:aws:iam::aws:policy/service-role/AmazonAPIGatewayPushToCloudWatchLogs

# Set role at account level
ROLE_ARN=$(aws iam get-role --role-name APIGatewayCloudWatchLogsRole --query 'Role.Arn' --output text)
aws apigateway update-account \
  --patch-operations op=replace,path=/cloudwatchRoleArn,value=$ROLE_ARN \
  --region us-east-1

Access logs immediately revealed problems:

13/Jan/2026:18:00:57 +0000 POST /prod/content 200 -
13/Jan/2026:18:05:37 +0000 POST /prod/prerender 502 Internal server error
13/Jan/2026:18:06:59 +0000 GET /prod/stream 500 Internal server error

The stream endpoint was returning 500s every 5 minutes. Why? The API Gateway route existed, but the Lambda function had been deleted months ago. Some automated process was still calling a dead endpoint 288 times per day.

Step 5: Download and Inspect Source Code

For functions without source code in version control, download and inspect them:

# Get function code download URL
URL=$(aws lambda get-function \
  --function-name function-name \
  --region us-east-1 \
  --query 'Code.Location' \
  --output text)

# Download and extract
curl -s "$URL" -o /tmp/lambda.zip
unzip -q /tmp/lambda.zip -d /tmp/lambda-source
ls -la /tmp/lambda-source/

This helped identify:

Missing dependencies (broken npm installs)
Hardcoded dates (END_DATE: “9/5/2021”)
Malformed configuration (empty URL environment variables)

Case Studies from the Investigation

Case 1: The Contentful Video Counter

Function: Daily job to fetch YouTube view counts, update Contentful CloudWatch Metrics: 30 invocations, 0 errors Reality: Failed all 30 times

The logs showed Error: Requests from referer <empty> are blocked. Contentful API requires a referer header; the Lambda wasn’t sending one. The function caught the error, logged it, and returned successfully.

Resolution: Delete. Not fixable without Contentful API changes.

Case 2: The Zoom Cleanup Job

Function: Cleanup Zoom meeting user pool every 10 minutes CloudWatch Metrics: 12,961 invocations, 12,961 errors Reality: Completely broken since deployment

Error: TypeError [ERR_INVALID_ARG_TYPE]: The "url" argument must be of type string

Environment variable check showed:

{
  "ZOOM_SLACK_STATS_URL": "\"dunnowhatthisissupposedtobe\""
}

Someone literally put placeholder text in production config. The function had been failing every 10 minutes for forever, burning money on useless executions.

Resolution: Delete immediately.

Case 3: The Phantom Stream Scheduler

Function: Returns event schedule data via API endpoint CloudWatch Metrics: Not found (function didn’t exist in prod) Reality: API Gateway route still existed, returning 500s

# Check API Gateway resources
aws apigateway get-resources \
  --rest-api-id api-id \
  --region us-east-1 \
  --query "items[].path"

# Output included /stream-schedule but function didn't exist
aws lambda list-functions --region us-east-1 | grep stream-schedule
# No results

The Lambda had been deleted months ago. The API Gateway route remained, returning 500 to every caller. Nobody noticed because nobody was monitoring API Gateway errors.

Resolution: Delete the orphaned API Gateway route.

Case 4: The Vault Token Renewal

Function: Monthly job to renew HashiCorp Vault tokens CloudWatch Metrics: 3 invocations, 0 errors Reality: Failed to even import dependencies

2026-01-01T07:00:49.932Z ERROR Runtime.ImportModuleError: 
  Error: Cannot find module '@npm/vault-node'

The deployment package was missing a critical npm dependency. The function couldn’t even start. CloudWatch saw this as a platform error, not a function error, so it didn’t increment the Errors metric.

Resolution: Delete. Vault integration was deprecated months ago.

Practical Lessons

1. Never Trust Metrics Alone

CloudWatch metrics are useful for trends and alerts, but they don’t tell you if a function is actually working. Always check logs for functions you’re considering keeping.

2. Look for Patterns, Not Individual Errors

A single error in logs might be transient. 30 identical errors across 30 invocations means it’s broken. Use grep to find patterns:

aws logs tail /aws/lambda/function-name --since 7d | \
  grep ERROR | \
  cut -d' ' -f3- | \
  sort | uniq -c | sort -rn

3. Enable Structured Logging

Update your Lambdas to use structured JSON logging:

aws lambda update-function-configuration \
  --function-name function-name \
  --region us-east-1 \
  --logging-config LogFormat=JSON,ApplicationLogLevel=DEBUG

This makes CloudWatch Insights queries much more powerful for finding issues.

4. Monitor API Gateway, Not Just Lambdas

If your function is HTTP-triggered, API Gateway logs show the full picture: who’s calling, how often, and what responses they’re getting. Lambda logs only show what happens when the function runs.

5. Check EventBridge Rules

Scheduled functions can fail silently forever. List all rules and verify their targets still exist:

aws events list-rules --region us-east-1 | \
  jq -r '.Rules[].Name' | while read rule; do
    echo "=== $rule ==="
    aws events list-targets-by-rule --rule $rule --region us-east-1
done

6. Audit Dead Letter Queues

If you’re using async invocation or event source mappings, check for messages piling up in dead letter queues:

aws sqs get-queue-attributes \
  --queue-url https://sqs.region.amazonaws.com/account/dlq-name \
  --attribute-names ApproximateNumberOfMessages

Messages in a DLQ mean failures you might not see in CloudWatch.

The Final Tally

Out of 45 Lambda functions investigated:

8 functions: 0 invocations in 30 days → Safe delete (dead code)
15 functions: 100% error rate → Immediate delete (completely broken)
12 functions: “0 errors” in metrics but failing in logs → Delete (silently broken)
3 functions: Confirmed working via API Gateway logs → Keep
7 functions: Low usage, needs business review → Investigate further

Only 3 out of 45 functions were definitively healthy when checked properly. The rest ranged from dormant to catastrophically broken, yet most had been running (and costing money) for a very long time.

Conclusion

CloudWatch metrics are designed for alerting on anomalies, not determining absolute health. The Errors metric specifically tracks unhandled exceptions, which means well-written error handling can mask complete functional failure.

Before you delete or keep Lambda functions based on metrics:

Check the actual execution logs
Verify the trigger mechanisms still exist
Enable API Gateway logging for HTTP functions
Look for patterns in errors, not just counts
Download and inspect source code if needed

In this investigation, 27 out of 45 functions appeared healthy in CloudWatch but were actually broken or unused. The only way to know for sure was to check the logs.

If you’re running serverless infrastructure on AWS, when’s the last time you actually looked at your Lambda logs? Not the metrics, the actual logs. You might be surprised what you find.

The Problem with CloudWatch Error Metrics#

Understanding CloudWatch Error Metrics#

Building a Proper Investigation Process#

Step 1: Get Invocation Counts#

Step 2: Check Actual Logs#

Step 3: Identify Trigger Mechanisms#

Step 4: Enable API Gateway Logging#

Step 5: Download and Inspect Source Code#

Case Studies from the Investigation#

Case 1: The Contentful Video Counter#

Case 2: The Zoom Cleanup Job#

Case 3: The Phantom Stream Scheduler#

Case 4: The Vault Token Renewal#

Practical Lessons#

1. Never Trust Metrics Alone#

2. Look for Patterns, Not Individual Errors#

3. Enable Structured Logging#

4. Monitor API Gateway, Not Just Lambdas#

5. Check EventBridge Rules#

6. Audit Dead Letter Queues#

The Final Tally#

Conclusion#