Auto Scaling

What we're doing: See how a lifecycle hook delays termination just long enough to drain in-flight requests.

aws autoscaling put-lifecycle-hook --lifecycle-hook-name drain-connections \
  --auto-scaling-group-name app-asg --lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING \
  --heartbeat-timeout 120

1

This hook fires before the instance is actually terminated, not after.

3

The instance has up to 120 seconds in a "Terminating:Wait" state — enough time for in-flight requests to finish before the instance is actually removed.

Why this works: Without a lifecycle hook, Auto Scaling can terminate an instance mid-request — the hook creates a window for the instance (or an external process watching for the event) to finish current work and deregister cleanly first.

What we're doing: See a scheduled scaling action that raises the minimum ahead of a known daily traffic pattern, layered on top of target tracking.

aws autoscaling put-scheduled-update-group-action --auto-scaling-group-name app-asg \
  --scheduled-action-name morning-ramp-up --recurrence "0 8 * * *" \
  --min-size 6 --desired-capacity 6

1

This scheduled action does not replace target tracking — it just raises the floor at a known time, ahead of demand actually arriving.

3

The recurrence "0 8 * * *" runs this every day at 08:00 — capacity is already higher before the morning traffic ramp shows up in any metric.

Why this works: Target tracking alone reacts to a metric after it moves — a scheduled floor set ahead of a known pattern means capacity is already in place before the metric even starts climbing, avoiding the reaction lag that a purely reactive policy has.

What we're doing: See the full loop: a target fails its ELB health check, and the ASG treats that as a replacement signal.

1. ASG health check type set to ELB (not just EC2)
2. Target group's health check fails for instance i-0abc123 (app returns 500s)
3. ASG marks i-0abc123 unhealthy, based on that ELB signal
4. ASG terminates i-0abc123 and launches a replacement
5. New instance registers with the target group once IT passes health checks

2

This is an application-level failure (bad responses), which EC2's own status checks alone would never catch — the instance itself is running fine.

5

The replacement only starts receiving traffic once it independently passes the same health check the failed instance failed.

Why this works: Using the ELB health check (not just EC2 status checks) as the ASG's health check source is what lets Auto Scaling react to application-level failures, not just infrastructure-level ones — an instance can be "healthy" by EC2's definition while still failing every real request.

What we're doing: See what a graceful shutdown handler actually needs to do inside the deregistration delay window.

def handle_sigterm(signum, frame):
    stop_accepting_new_requests()   # deregistered from target group already
    wait_for_in_flight_requests(timeout=25)   # inside the deregistration delay window
    exit(0)

signal.signal(signal.SIGTERM, handle_sigterm)

2

By the time SIGTERM arrives, the instance is typically already deregistered from the target group — new requests should not be arriving.

5

The timeout here needs to fit inside the load balancer's configured deregistration delay, or in-flight requests get cut off anyway.

Why this works: The deregistration delay is a finite, configured window — an application that assumes it has unlimited time to shut down gracefully will eventually be forcibly terminated mid-work once that window elapses.