Subscriber Churn Model: What the Results Mean
To: Head of Customer Success and CFO | Re: Should we use this model to target 30-day retention offers?
Bottom line
The model is promising and ready for a limited pilot, but not ready to replace the 14-day rule outright. When it flags someone, it is right about 58% of the time. That is far above the roughly 17% hit rate needed for a $10 offer to pay for itself. On these test numbers, it would generate about $15,000 net value per 10,000 customers per month. The catch is that it currently misses more than half of the people who actually cancel (it catches 44%). We also have no numbers for the 14-day rule, so we cannot yet say which one is better. The most important caveat: the 93% accuracy figure is misleading, because simply guessing "nobody cancels" would score 92%.
The key numbers, translated
I read this as a binary classification model: it sorts each customer into "will cancel" (1) or "will stay" (0). Every figure below comes from the test set of 10,000 customers, which is data the model did not learn from.
From the confusion matrix (the table of right and wrong calls):
- Correctly flagged cancellers: 352
- False alarms (flagged, but stayed): 255
- Missed cancellers: 448
- Correctly left alone: 8,945
- Total flagged: 352 + 255 = 607 (about 6% of customers)
| Metric | Value | What it means in plain English | Good / OK / Weak for this use |
|---|---|---|---|
| Accuracy | 0.93 (9,297 of 10,000 correct) | 93% of all calls were right. But 92% of customers stay anyway, so "nobody cancels" scores 92%. | Weak as a guide. It hides the real story. |
| Precision (cancel) | 0.58 (352 ÷ 607) | Of every 100 customers flagged, about 58 actually cancel. | Good. Well above the ~17% break-even. |
| Recall (cancel) | 0.44 (352 ÷ 800) | Of every 100 customers who cancel, the model catches about 44 and misses 56. | OK to weak. Many cancellers are missed. |
| F1 (cancel) | 0.50 | One combined score for precision and recall, from 0 to 1. | OK. Mostly dragged down by recall. |
| ROC AUC | 0.86 | Pick one canceller and one stayer at random. About 86% of the time, the model rates the canceller as riskier. | Good. The model ranks risk well. |
| Precision / recall (stay) | 0.95 / 0.97 | The model is very good at recognizing people who stay. | Fine, but not the goal. |
| Train accuracy | 0.97 vs 0.93 test | The model does better on data it learned from. | Worth watching (see below). |
How good is it, really?
Against the "do nothing smart" baseline. Always predicting "stays" gives 92% accuracy and catches zero cancellers. The model gives 93% and catches 352 of 800. The real gain is in the cancellers it finds, not in the accuracy figure.
Back-of-envelope money estimate. These are rough figures per 10,000 customers for one month.
I assumed three things:
- The 1-in-4 save rate applies to people who would otherwise cancel.
- A saved customer is worth $240 (one year).
- Offers sent to people who would have stayed anyway cost $10 and bring no benefit.
On that basis:
- Value of each offer to a real canceller: 25% × $240 = $60 expected.
- Break-even hit rate: $10 ÷ $60 = about 17%. Any targeting method whose flagged group cancels more than 17% of the time makes money.
- Model as it stands:
- Value: 352 × $60 = $21,120
- Offer costs: 607 × $10 = $6,070
- Net: about $15,050
- Perfect-foresight ceiling (offer only to the 800 real cancellers): 800 × $60 − $8,000 = $40,000
- Share captured: the model gets about 38% of the possible value.
Key trade-off: false alarms vs misses.
- A false alarm costs $10.
- A missed canceller costs about $50 in lost expected value ($60 − $10).
Misses are about five times more expensive than false alarms. The model is currently cautious: 58% precision is well above the 17% needed. Lowering its "flag" cut-off (the threshold) would flag more people. Precision would fall and recall would rise, and net value would likely go up. Your analyst can test cut-offs on the same test set to find the most profitable one.
Against the 14-day rule. No figures were provided, so I can't compare them. The comparison is simple to run. Apply the 14-day rule to the same 10,000 test customers, then calculate:
Net value = (cancellers caught × $60) − (people flagged × $10)
Compare the result with the model's ~$15,050. Note that "days since last login" is the model's top driver. The model probably already captures much of what the 14-day rule does, plus extra signals.
What is driving the predictions
"Gain" shows how much each factor helped the model sort customers. It does not show direction, meaning it doesn't say whether more of a factor raises or lowers risk.
| Factor | Share of gain | What it is |
|---|---|---|
| Days since last login | 31% | Recent activity |
| Support tickets in last 90 days | 18% | Service issues |
| Plan downgrade | 14% | Moving to a cheaper plan |
| Tenure (months as customer) | 11% | How long they've been with you |
| Monthly spend | 7% | Account size |
Together these five account for 81% of the gain. They make business sense, but they are associations, not causes. For example, reducing support tickets will not automatically reduce churn.
Watch-outs and limits
- Class imbalance (seen in the results). Only 8% cancel, which inflates accuracy. Judge the model on precision, recall and dollars instead.
- Possible overfitting (seen in the results). The error rate is 3% on training data but 7% on test data, more than double. Overfitting means the model memorized some quirks of its training data. It is not alarming, but the model may do somewhat worse on new customers.
- How the test set was chosen (worth checking). If the test customers were randomly drawn from the same Jan 2023–Jun 2024 period, results may look better than real future performance. A test on late