AWS Networking at Senior Level

What we're doing: Decide between a peering mesh and a Transit Gateway as the VPC count grows.

3 VPCs, full mesh: 3 peering connections. Simple, no hourly cost.

6 VPCs, full mesh: 15 peering connections, and every VPC's route table
carries 5 peering routes. Still workable, already tedious.

12 VPCs, full mesh: 66 peering connections, 11 routes per route table,
and peering is non-transitive so on-premises access must be built
separately for each VPC.

Transit Gateway: 12 attachments, one route table, on-premises attached
once. Costs per attachment-hour and per GB — which is the trade.

1

Peering is genuinely the right answer at small scale. Reaching for a hub on day one is over-engineering.

5

The pain is quadratic in the number of route table entries, not just in the number of connections.

9

This is the tipping point in practice: the moment on-premises connectivity has to reach every VPC, the mesh stops scaling and the hub pays for itself.

Why this works: The mesh-versus-hub decision is arithmetic, not taste. Peering costs nothing hourly and scales quadratically in operational effort; Transit Gateway costs per attachment and per GB and scales linearly. Knowing where the crossover is for your estate is the senior-level part.

What we're doing: See what an overlapping CIDR actually costs, two years later.

2024: two teams independently create VPCs, both using 10.0.0.0/16
because it is the console default suggestion. They never talk.

2026: the company needs the two systems integrated, and both must reach
on-premises through one Direct Connect link.

Neither can be routed to the other. The only fixes are: recreate one VPC
with a different CIDR and migrate every resource into it, or put NAT in
between and lose the ability to address hosts directly.

1

Nothing is wrong at this point. Both VPCs work perfectly, and the default CIDR is a reasonable local choice.

6

A VPC's primary CIDR cannot be changed. "Recreate and migrate" means every subnet, every ENI, every hard-coded address, and a maintenance window.

Why this works: Address planning is the one networking decision that is almost impossible to undo and costs nothing to get right. Ten minutes allocating from a documented block prevents a migration project, which is why it is worth doing before the first VPC exists.

What we're doing: Understand why one NAT gateway saves money and creates a cross-AZ outage.

Cost-optimized: one NAT gateway in public-a.
  private-a: 0.0.0.0/0 -> nat-a
  private-b: 0.0.0.0/0 -> nat-a      <- cross-AZ

AZ A becomes unavailable. Compute in AZ B is healthy and still running,
but has no outbound internet: it cannot pull images, reach Secrets
Manager over the internet, or call third-party APIs.

Resilient: nat-a and nat-b, each private subnet routing to its own AZ's
NAT. Roughly double the NAT hourly cost, and no cross-AZ dependency.

1

This is a legitimate choice for a non-production environment, where the saving is real and the risk is acceptable.

6

The failure is subtle: instances are up, health checks that do not leave the VPC pass, and only outbound-dependent work fails.

Why this works: Multi-AZ is not achieved by placing resources in two AZs — it is achieved by removing every cross-AZ dependency between them. A shared NAT gateway is the most common one, and it is invisible until the AZ it lives in has a problem.

What we're doing: Diagnose "the app cannot reach the database" without changing anything at random.

1. dig db.internal from a task in private-a
   -> 10.20.128.41. Resolves. DNS is fine.

2. Route table for private-a: 10.20.0.0/16 -> local covers 10.20.128.41.
   Route is fine.

3. sg-db inbound: 5432 from sg-task. Present. sg-task outbound: allow
   all. Fine.

4. NACL on data-a: inbound allows 5432 from 10.20.64.0/20. Outbound
   allows 5432 only — not the ephemeral range 1024-65535.
   -> The reply is dropped. This is the failure.

5. Not reached. The listener was never the problem.

1

Thirty seconds, and it eliminates the whole class of failures that look identical to a firewall block.

4

The `local` route covers everything inside the VPC and cannot be removed, so intra-VPC routing is rarely the answer — but confirming it is still faster than guessing.

10

The classic NACL failure. The inbound rule is obviously right, the outbound rule is subtly wrong, and the symptom is a timeout indistinguishable from a security-group problem.

Why this works: Each check is cheap and each one eliminates a whole class of cause. The alternative — editing the security group because that is the usual culprit — leaves the rule permanently widened and the real problem untouched.