AWS troubleshooting challenges
83 challenges on a live AWS account that is already broken. Each page gives the symptom and where it shows. The cause is yours to find.
Filter by the kind of failure and by difficulty, from one (a warm-up) to five (a multi-service investigation).
83 of 83 challenges
The web application is unreachable. The platform reports no running copies of the service, and the traffic router marks every target unhealthy. The most recent release points at a build that was never published. The expected configuration is documented on the resources themselves. Restore service.
Elastic Load Balancing · 20 min · level 3
A business workflow accepts every request and reports it as started, then fails on its first step with an access error. The queue it writes to and the worker behind it both exist and look healthy. A leftover worker from an earlier release is connected to nothing. Find what is being denied and restore the chain.
Step Functions · 15 min · level 2
The event router reports every delivery as successful, but the messages never arrive at their destination. The routing rule and its target are intact. A second, unused destination exists nearby. Explain the gap between reported success and actual delivery, then close it.
EventBridge · 15 min · level 2
Malformed orders disappear after a few retries instead of landing somewhere a human can review, and nobody was alerted when processing stalled. A second worker with a very short time limit also exists. Restore both containment and visibility for messages that cannot be processed.
SQS · 25 min · level 3
Looking up a customer by email is slow and rejecting requests under ordinary load, while expired session records accumulate instead of being cleaned up automatically. A monitor on write capacity exists but has never fired. The expected settings are documented on the resource itself.
DynamoDB · 20 min · level 3
Traffic keeps going to the primary region even though the standby is healthy and should have taken over. Health checks and routing rules exist, but nothing shifts when the primary fails. A second, near-identical set of records exists. The operations team keeps the expected configuration on a dashboard. Restore automatic failover.
Route 53 · 45 min · level 4
External researchers report that they can list and download files from a data store without any credentials. Public-access settings are inconsistent across the account, and a second store with a similar name exists. Identify the unintended exposure and close it.
S3 · 5 min · level 1
New orders stop flowing end to end: the intake queue grows, the order count stalls, and customer notifications never fire. Several workers and tables from earlier releases are still around. Every break in the path must be found; repairing three of the four changes nothing.
SQS · 90 min · level 5
Everything running in the private tier has gone quiet: it cannot reach the open network, shared storage, or the database behind it, while the public tier keeps answering normally. Connections simply never complete. Recent changes touched routing and traffic rules across several layers at once. Restore outbound reach for private workloads across every dependency they need.
EC2 · 90 min · level 5
Since this morning the support assistant apologises instead of answering questions about orders. Looking an order up by hand still works. Bring the answers back without giving the assistant any reach it does not have today.
Bedrock AgentCore · 15 min · level 2
Customers who ask what shipping will cost get an apology instead of a price. The carrier's pricing service is up and answers the operations team normally, and the carrier also runs a sandbox. Get real prices back into the chat without widening what the assistant can see.
Bedrock AgentCore · 15 min · level 2
Since last night, part of the support assistant's activity has stopped appearing where the team collects it. Customers notice nothing. Make that activity show up again.
Bedrock AgentCore · 10 min · level 1
Regular customers have to repeat their preferences every time they open a new chat, although the assistant remembers them within a conversation. Nothing has been deleted and nobody changed the assistant's code. The conversation summaries the quality team relies on must stay as they are.
Bedrock AgentCore · 20 min · level 3
A new returns policy shipped on Monday. Asked directly, the assistant gives the new answer; customers in the website chat still hear the old one. Nothing errors and nothing is down. Fix what customers are told, and keep the previous release one step away.
Bedrock AgentCore · 15 min · level 2
A traffic spike is slowing the storefront down. A handful of clients send far more requests than anything seen before and nothing holds them back, although the organisation publishes an approved ceiling for exactly this. Bring the storefront back under that ceiling.
Elastic Load Balancing · 15 min · level 2
A penetration test shows injection payloads reaching the application, although a request filter is in place and its logs show rules being evaluated. More than one rule set exists. Find out which one stands in front of the public storefront and stop the attacks getting through.
Elastic Load Balancing · 15 min · level 2
After an incident, the security team finds no access records for the public interface despite steady traffic, and an alarm tied to that traffic has never recovered. Two similarly named destinations exist and both feed monitoring. Restore the records where the rest of the monitoring expects them.
API Gateway · 15 min · level 2
The mobile team reports that calls to the orders route succeed without a valid token: 200 where 401 is expected. Other routes on the same interface answer too. Find the one that carries real business traffic and make it demand a token again.
API Gateway · 5 min · level 1
Clients of the public interface get 429 responses almost immediately, even at trivial traffic. A browser app on another origin also shows cross-origin errors in its console. More than one version of the interface exists. Restore normal access on the one serving live traffic.
API Gateway · 20 min · level 3
A service that loads a configuration secret on every request has failed every call since this morning with a generic error. The secret has not moved and its value is unchanged. Restore exactly what the service needs to read that one secret, and nothing more.
Lambda · 20 min · level 3
Capacity has sat below target since one node stopped answering. Health checks and replacement rules both exist and read as active, yet the silent node stays and the fleet stays short, all in one availability zone. The operations team wrote down what normal looks like. Make the fleet replace a node that stops serving.
EC2 · 40 min · level 4
The approved location for query results has received nothing new, while another location fills with output from recent ad hoc queries. Find out why, and make every future query respect the approved location.
S3 · 5 min · level 1
Latency spiked and some instances stopped answering after reports of power trouble in one zone of the region. Capacity should come back on its own. Confirm the workload survived the disruption and that every instance is serving again.
EC2 · 45 min · level 4
The moderation team confirms unsafe output reaching customers through the support chatbot. Two services call the model for the support team, and only one of them is reachable by customers. Make sure what customers receive is held to the account's content policy.
API Gateway · 10 min · level 1
The platform team cannot audit who invoked which models during last week's review. Calls succeed and billing shows them, but no record of the invocations appears anywhere. Restore the audit trail to the destination it is already allowed to write to.
Amazon Bedrock · 10 min · level 1
The knowledge-base assistant returns empty answers even though source documents were uploaded recently. Dashboards show activity and storage holds the documents, yet users get no grounded responses. Follow the path from ingestion to the model call and restore it end to end.
Amazon Bedrock · 90 min · level 5
After a suspected intrusion, investigators find gaps in the account's activity record and in the threat detector's findings. Some parts look healthy at first glance, including a second record that appears fully enabled. The workload runs in one region on purpose. Bring what is captured back in line with what security expects.
CloudTrail · 45 min · level 4
Compliance reports that the account's activity record stopped capturing management events within the last hour. A storage location whose name suggests it is the destination shows no recent writes. Work out whether recording stopped or now goes somewhere else, and restore it.
CloudTrail · 5 min · level 1
On call confirms an alarm entered its triggered state overnight, yet nobody was notified. The threshold looks right and the alarm history shows the state changes. A notification channel with no subscribers sits in the account. Get the alerts to the team.
CloudWatch · 10 min · level 1
A function threw elevated errors yesterday, but its error alarm never fired and nobody heard about it. The alarm has changed state before. Two notification channels exist, neither wired up, each labelled with its purpose. Make the next spike reach the right people.
Lambda · 20 min · level 3
Finance flags runaway storage costs from the account's logs, while compliance asks why no violation was ever reported. Log groups exist with a range of settings, one of them suspiciously short. Find what is driving the cost and bring it in line with the compliance rule.
AWS Config · 15 min · level 3
Records keep arriving, but the curated area of the data lake has stayed empty since the last nightly run. More than one step between arrival and the catalog has drifted from what was deployed, and security expects records to be protected on the way. Restore the flow end to end.
S3 · 90 min · level 5
A backend service that writes order metadata fails with connection and authentication errors. Several stored credentials have similar names. Restore reliable connectivity from the service to its database.
Lambda · 90 min · level 5
The orders interface rejects requests under routine traffic that used to pass, and table metrics show requests refused before they are processed. A second, small table with an index-like name also exists. Find the bottleneck in production and stop it rejecting routine traffic.
DynamoDB · 10 min · level 1
Response times degraded sharply on one host while its siblings stay healthy. Its utilisation sits on a flat plateau near maximum, starting at one instant, with no deployment and no change in traffic. Single out the saturated host, find what keeps it there and restore normal performance.
EC2 · 45 min · level 4
The service behind the public entry point returns server errors, and discovery lookups fail now and then. Deployments keep happening but new copies cycle unhealthy. An older revision and a second discovery namespace also exist. Restore stable discovery and healthy targets.
Elastic Load Balancing · 90 min · level 5
The service is stuck mid-deployment: new copies fail their health check and the rollout never completes either way. Application logs are sparse, and a similarly named service runs in the same cluster. Bring the live service to a steady, healthy state.
ECS · 45 min · level 4
A service that reads shared files returns mount and permission errors. Several file shares exist, including a second encrypted one kept apart from the rest. Restore reliable read and write access to the intended share.
Lambda · 90 min · level 5
Since the last restart, the intake workers are created but never start, and documents wait for them. Nothing changed in the application itself. Get the workers running again.
EKS · 12 min · level 2
Since a platform rebuild, the nightly payment reconciliation has not completed once, and finance settles the backlog by hand. The reconciler's code and configuration did not change. Get it completing again without giving it more than its job needs.
EKS · 40 min · level 3
Checkout workflows stall mid-flow: some steps succeed, but the steps meant to undo a failure never run and order records end up inconsistent. Records that track each payment attempt pile up and never age out. Leftovers from earlier iterations remain. Restore end-to-end completion.
Step Functions · 90 min · level 5
Cross-account order events stopped arriving on the shared channel, and the intake line drains with nothing coming out the other side. The relay between them reports as running, and a second channel with a similar name exists. Restore delivery to the partner.
EventBridge · 45 min · level 4
An event rule forwards order events to a queue and reports success, but failed deliveries disappear without a trace, even for malformed payloads. A second queue whose name suggests it collects failures also exists. Make sure failed events are kept for investigation.
EventBridge · 10 min · level 1
Documents keep arriving, but no approval run has started for any of them since the last deployment. The relay feeding the workflow reports as running, and the workflow works when started by hand. Get a run started for every document without changing where the relay delivers.
Step Functions · 15 min · level 2
Records written to the source table never reach the downstream workflow, and no execution starts. The relay between them reports as running, and a record can be dropped or reshaped at more than one point. A second, similarly named table exists. Restore the flow.
Step Functions · 45 min · level 4
Batch jobs stopped running at midnight: the batch system still emits an event for each job, but the downstream function no longer runs. The routing and its target look configured, and a similarly named log group hints at a logging fault. Restore reliable automatic execution.
Lambda · 5 min · level 1
Booking requests are logged and announced as usual, yet no booking has been made since a recent clean-up, and nothing reports an error. Get bookings made again.
Step Functions · 5 min · level 2
Wait times on disk spiked on a production instance and application writes time out, while storage throughput sits well below normal. An alarm on a different metric is firing. Find the storage bottleneck and restore normal performance.
EC2 · 45 min · level 4
Containers lost connectivity mid-request and health checks flapped across the service. Network metrics show heavy packet loss and added latency with no code change, and a processor alarm paged nobody. Find what is disrupting the service and bring it back to full health.
Elastic Load Balancing · 45 min · level 4
The nightly processing job reworked the entire history instead of only the new files, doubling its runtime and cost, and nothing reports a failure. A similarly named location exists. Restore incremental processing for the production job.
S3 · 15 min · level 2
Records enter the stream, but processing lag climbs without limit and no transformed output lands in storage. A second job definition sits idle. Trace the live path and restore continuous processing.
Kinesis · 45 min · level 4
Records are consumed, but malformed ones vanish after repeated failures and are never seen again. A similarly named function with different time limits also exists. Keep failed records for replay instead of losing them.
Kinesis · 15 min · level 2
Background requests that fail appear nowhere for replay, while the happy path processes fine. A second queue with a failure-like name exists but stays empty. Restore visibility into background failures.
Lambda · 15 min · level 2
A handler for data changes logs configuration errors at startup and fails every request, though nothing was redeployed. The tables named in its logs exist and look healthy, and a second one sits alongside them. Get it working again, and make sure its failures no longer disappear silently.
Lambda · 45 min · level 4
A processing function started timing out and complaining about settings it can no longer find right after last night's change window, with no code deployment. The table it writes to exists and is reachable. The team documented the expected operating values: find them and restore them.
Lambda · 20 min · level 3
Booking requests still arrive and something reads them, yet the booking desk has logged none since last night's tidy-up. Get new requests logged again.
DynamoDB · 4 min · level 1
After a scheduled update, the request handler fails every cold start with import and permission errors at once. A second function that skipped the update keeps running fine. Work out what the production handler needs to start again, and restore it.
Lambda · 45 min · level 4
New booking requests pile up and the booking desk logs none of them, without a single error anywhere. Get requests logged again.
SQS · 5 min · level 2
A function attached to a private network can no longer reach outside interfaces or the internal resources it used to. Nothing about its setup looks obviously empty, and a wide-open alternative sits nearby. Restore its outbound reach without giving it more than it is meant to have.
Lambda · 45 min · level 4
Signed-in users are turned away with authorisation errors even though they present valid credentials, and the failure spans several layers at once. A legacy handler and an old log destination also remain. Trace the live request path end to end and restore every layer it depends on.
API Gateway · 90 min · level 5
On call missed two incidents last week: a spike of errors went unnoticed, and storage costs climbed because old log entries were kept. Everything looks in place at a glance. Find what is silently disconnected in each case and make the next incident reach someone.
Lambda · 20 min · level 3
Application connections dropped during an impairment of one availability zone, and queries kept failing while the database promoted a new writer. The cluster may show mixed availability until it settles. Confirm it is serving again with a single writer.
RDS · 45 min · level 4
During recent trouble on the primary side, automated failover left customers on a failing site for minutes. The failover records are in place and the monitoring reports. Make failover react promptly to real trouble.
Route 53 · 5 min · level 1
A compliance audit finds a regulated data set failing several controls at once. A sibling data set in the same account behaves as expected on its own schedule. Bring the regulated data set back in line before the auditors close their findings.
S3 · 90 min · level 5
Bookings are confirmed and announced, yet the outbox where confirmation emails wait has stayed empty since a recent review. Get emails into the outbox again without opening it to anyone else.
S3 · 6 min · level 3
Application reads from the data set suddenly fail with access errors, and objects younger than expected vanished overnight. Related storage looks normal by comparison, and a role with unusually broad access drew audit attention. Restore reliable reads and predictable retention where the application actually reads.
S3 · 20 min · level 3
Webhook subscribers stopped receiving alerts at the moment of the first delivery attempt that ever failed, while publishing upstream looks fine. Two keys sit in the account alongside the topic. Restore reliable, protected delivery to subscribers.
SNS · 20 min · level 3
The order-processing service receives nothing, although upstream confirms every order it sends was accepted and nothing reports an error. A second consumer whose name suggests it handles orders also stays empty. Find why accepted orders never arrive.
SQS · 15 min · level 2
Bookings are confirmed and announced, yet no confirmation email is prepared any more, and no error explains why. Get emails into the outbox again without opening it to other senders.
SNS · 6 min · level 3
Downstream consumers log a parse failure on every message, though publishing upstream keeps succeeding. Another delivery path produces messages that look well formed. Make what arrives match what the consumer expects to read.
Lambda · 5 min · level 1
The processor runs but handles nothing while its backlog climbs steadily, and nothing alerts anyone. The team keeps its approved settings for this pipeline in its own documentation. Find them and restore normal throughput.
SQS · 10 min · level 1
Documents keep arriving on the intake line, but the number waiting only grows and none is processed. Senders see no error, and the processing side keeps trying without success. Get the documents flowing again.
SQS · 10 min · level 2
Every attempt by producers to send new work to the intake line fails, while consumers sit idle. A compliance check separately flagged that stored data is not protected the way policy requires, and a look-alike sender exists. Let authorised senders enqueue work again, with the data kept protected.
SQS · 20 min · level 3
Orders pile up at the start of the pipeline and never reach the next stage, although the forwarding function keeps running on them. A second output queue with no consumer exists. Find where the orders stop and get them moving.
SQS · 5 min · level 1
The fulfilment service processes duplicate orders: the same order number shows up more than once downstream. Operations documented the timing this pipeline expects. Eliminate duplicate processing under normal load.
Lambda · 5 min · level 1
Runs wait forever at the point where outside work should signal that it is done, although the workers report success. A second queue whose name suggests it carries those signals receives nothing. Let outside workers resume a waiting run again.
Step Functions · 15 min · level 2
Documents are read, then sit at the review stage for hours, even though the review itself seems to run. Nothing changed in the application. Get documents moving past review again.
Step Functions · 12 min · level 2
Runs finish instantly with a success status, but the work they should trigger never happens. A second queue looks as if it might catch problems but is never written to. Find why a workflow that always succeeds does nothing.
Step Functions · 45 min · level 4
Failed workflow runs cannot be diagnosed because they leave no history anywhere. A log destination with a name like the workflow's exists. Make failures diagnosable again.
Step Functions · 5 min · level 1
Since a recent review, the booking process starts for every request and stops at its first step. Get bookings confirmed again without loosening anything the review meant to keep.
Step Functions · 7 min · level 3
Documents are read, then every review fails within seconds. Compliance made counterparty screening mandatory last week and confirms it still runs on every document. Get documents through review again without switching the screening off.
Step Functions · 12 min · level 2
Workflow runs fail immediately, pointing at a piece of work that no longer exists, and no run history appears anywhere. A second log destination with a similar name exists. Restore a working workflow and a full, observable history.
Step Functions · 20 min · level 3
A volumetric attack overwhelmed the public entry point although every layer of protection shows as active. Request rates spiked unchecked, clients from countries the service does not sell in were served, and a request announcing a known bot came back 200. Restore full protection for live traffic.
Elastic Load Balancing · 45 min · level 4
Pick one and fix it with the provider's own console and command line. We host the cloud account. Nothing lands on your bill.