Skip to content
Shakeride

Cloud costs · · Updated

A request flood on your CDN is billed per request

A flood of tiny requests barely moves data transfer, yet every request is metered at the edge, at the firewall and, on a cache miss, at the origin. How to rate limit and narrow the cache key on CloudFront with AWS WAF, Front Door with its WAF, and Cloud CDN with Cloud Armor.

In one sentence: a CDN bills a flood by request count, not by bytes, so a rate limit at the edge and a cache key that ignores junk query strings matter more to the bill than any bandwidth setting.

Symptom

The request line of the CDN bill climbs steeply while data transfer barely moves. Request counts are up by an order of magnitude, bytes per request are tiny, and the cache hit ratio drops because each request carries a different query string, such as ?r=8f3a.

How it happens

Three lanes, one per provider, showing a flood of requests with random query strings metered by the CDN, inspected by a firewall without a rate rule, and reaching the origin as a cache miss on every request.
  • Every request is metered: the edge counts it whether it is a hit or a miss, and the firewall in front of it counts every request it inspects.
  • Random query strings defeat the cache: when the cache key includes every query string, each request is a miss, so the flood also reaches the origin and is billed there a second time.
  • The fix is at the edge: a rate limit per client turns the flood into cheap rejections, and a narrow cache key turns what remains into hits.

Act

AWS

Attach a web ACL of scope CLOUDFRONT (managed from us-east-1) with a rate-based rule keyed on IP, placed before the managed rule groups. On each cache behaviour, use a cache policy whose query string list holds only the parameters the content varies by.

{"Name": "per-ip-limit", "Priority": 0, "Action": {"Block": {}},
 "Statement": {"RateBasedStatement": {"Limit": 2000, "EvaluationWindowSec": 300,
   "AggregateKeyType": "IP"}},
 "VisibilityConfig": {"SampledRequestsEnabled": true,
   "CloudWatchMetricsEnabled": true, "MetricName": "per-ip-limit"}}

Azure

Add a rate limit custom rule to the Front Door WAF policy, in prevention mode. A rate limit rule needs a match condition: match the paths the flood targets. On each route, set query string caching to "Use specified query strings" and list only the ones that matter.

az network front-door waf-policy rule create -g rg-web --policy-name wafedge -n perclient \
  --rule-type RateLimitRule --rate-limit-threshold 1000 --rate-limit-duration 1 \
  --action Block --priority 10 --defer
az network front-door waf-policy rule match-condition add -g rg-web --policy-name wafedge \
  -n perclient --match-variable RequestUri --operator Contains --values /api/

Google Cloud

Cloud Armor rate limiting lives in a backend security policy, which sees cache misses: the requests that reach the origin. Add a rate-based ban, then restrict the cache key to the query parameters that change the content.

gcloud compute security-policies rules create 1000 --security-policy=storefront-policy \
  --src-ip-ranges="*" --action=rate-based-ban --enforce-on-key=IP \
  --rate-limit-threshold-count=600 --rate-limit-threshold-interval-sec=60 \
  --ban-duration-sec=300 --conform-action=allow --exceed-action=deny-429
gcloud compute backend-services update storefront --global --cache-key-query-string-whitelist=v

Keep watch

AWS

Alarm on the distribution's Requests metric against its usual peak, and on the web ACL's BlockedRequests for per-ip-limit. In Cost Explorer grouped by usage type, the Requests-Tier lines should flatten once the rule blocks.

Azure

Alert on the profile's Request Count and on Web Application Firewall Request Count split by action. During a flood, blocks rise while Origin Request Count stays flat.

Google Cloud

Alert on loadbalancing.googleapis.com/https/request_count grouped by cache_result. In the load balancer logs, jsonPayload.enforcedSecurityPolicy.outcome="DENY" counts what Cloud Armor rejected.

Exercise

Hold the line yourself: Storefront slows under a traffic surge, then Defences look active but let everything through.

Related: object storage egress and CDN caching limits.

Share this guide

All guides

Put this guide into practice

A live cloud environment on AWS, Google Cloud or Azure, already broken, and a clock. Fix it with the console and command line, then read a report of how you worked.

Browse the challenge catalog