Postmortem: Checkout Slowdowns and Payment Failures After v2.18.0 Deploy
| Field | Detail |
|---|---|
| Incident date | Tuesday 3/11 (year [TBD]) |
| Severity | SEV-2 |
| Status | Draft for review |
| Services affected | checkout-api (online store checkout), orders database |
| Duration | Detection to resolution: 46 min (14:12–14:58 ET). Start to resolution: ~53 min (approx.; impact likely began soon after the 14:05 deploy). |
| Authors | [TBD] |
| Audience | Brightcart engineering, CEO, Head of Support |
All times are US Eastern (ET), 24-hour format.
Executive Summary
On Tuesday 3/11, a routine update to our checkout system included a new promo-code feature that searched the entire orders database on every checkout. For about 45 minutes (14:12–14:58 ET), checkout was very slow or failing. Stripe recorded 212 failed payment attempts in that window, and support received 9 tickets. The team fixed it by reverting to the previous version at 14:52, and service was back to normal by 14:58. The most important follow-up is to test changes against realistic, production-sized data before release, because our test environment has about 5,000 orders while production has about 4 million.
Impact
- Customer impact: Checkout was very slow or failing for about 45 minutes. Customers saw "payment failed" errors (first reported by support at 14:35). The number of unique customers affected is [TBD].
- Business impact: Stripe showed 212 failed payment attempts during the window. Lost or recovered revenue is [TBD]. The incident overlapped with an email marketing campaign sent at 14:00, so it hit during a period of higher-than-usual traffic.
- Internal impact: The on-call engineer and a second developer worked the incident in the #incident channel. Support handled 9 tickets.
- Data integrity: [TBD]. We need to confirm that no customers were charged without an order being created, and no orders were created without payment.
Timeline
| Time (ET) | Event | Source/Notes |
|---|---|---|
| 14:00 | Marketing email campaign sent, increasing store traffic | Notes |
| 14:05 | checkout-api v2.18.0 deployed, including a new promo code lookup | Notes |
| ~14:05–14:12 | Start of impact (approx.) | Inferred: exact start [TBD]. Check latency graphs. |
| 14:12 | Detected. PagerDuty alert: checkout p95 latency > 5s | PagerDuty |
| 14:15 | On-call engineer acknowledged the alert. Grafana showed database CPU at 98%. | Grafana |
| 14:20 | #incident channel opened. A second developer joined. | Slack |
| 14:28 | Team suspected the email-campaign traffic spike and scaled app pods from 6 to 12. Latency did not improve. | Notes |
| 14:35 | Support reported customers getting "payment failed" errors | Support |
| 14:41 | Developer identified the new promo lookup query doing a full table scan on the orders table (no index on promo_code) |
Notes |
| 14:47 | Decision made to roll back | Notes |
| 14:52 | Mitigated. Rollback to v2.17.3 complete | Notes |
| 14:58 | Resolved. Latency back to normal, database CPU at 30% | Grafana |
| 15:10 | All clear called | Notes |
Detection & Response
- How detected: An automated PagerDuty alert on checkout p95 latency (the response time that 95% of requests stay under) fired at 14:12.
- Time to detect: About 7 minutes after the deploy (inferred from 14:05 deploy to 14:12 alert).
- Time to mitigate: 40 minutes from detection to rollback complete (14:12–14:52). It took 46 minutes to full recovery.
- What helped: The latency alert fired quickly. The Grafana dashboard pointed to the database right away. A second engineer joined within minutes and found the bad query. Rollback took 5 minutes once the team decided on it.
- What slowed us: The email campaign gave a believable alternative explanation, so the first response was to scale the app servers. That could not help with a database bottleneck. No step in the response process prompted the team to check recent deploys first. It took 35 minutes from the deploy to identifying the query, and another 6 minutes to decide to roll back.
Root Cause
The v2.18.0 release added a promo-code lookup that queried the orders table by promo_code. That column had no index (a lookup structure that lets the database find matching rows without reading every row). As a result, every checkout triggered a full scan of about 4 million orders. Under normal load plus the extra campaign traffic, these scans pushed database CPU to 98%. Checkout requests slowed down and payments failed. The problem was not caught before release because the staging database holds only about 5,000 orders, where a full scan is fast. Rolling back to v2.17.3 brought database CPU back to 30% within minutes, which strongly supports this cause.
5 Whys
- Why was checkout slow and failing? The orders database was overloaded (98% CPU).
- Why was the database overloaded? Each checkout ran a new promo-code query that scanned the entire
orderstable. - Why did it scan the whole table? The
promo_codecolumn had no index, and the release did not add one. - Why wasn't this caught before production? Staging has about 5,000 orders, so the query ran fast there, and no step in review or CI checks query plans.
- Why do we lack that safeguard? Our release process has no performance testing against production-sized data and no automated check for unindexed queries on large tables.
Contributing Factors
- The staging data set is roughly 800 times smaller than production (about 5k vs. about 4M orders), which hides performance problems.
- The deploy went out 5 minutes after a marketing email campaign, during elevated traffic. Notes say the campaign "probably made it worse." The size of that effect is unconfirmed.
- The campaign traffic offered a plausible but incorrect explanation, which delayed suspicion of the deploy.
- Inferred: scaling app pods from 6 to 12 may have added more concurre