Workflow study
Reading every churned account at once, for about $100
A churn analysis across millions of data points, run in roughly an hour for roughly one hundred dollars, which found that a large share of recorded churn was not product churn at all.
- Job to be done
Understand why customers were actually churning, across the whole churned population rather than the accounts someone had time to review.
- Inputs
- Churned-customer records in Snowflake, millions of data points
- Per-customer history, summarized and cleaned before categorization
- Workflow
- Pull the churned-account data from Snowflake
- Summarize and clean the record for each customer individually
- Have the model derive churn categories from the data rather than applying the existing taxonomy
- Attach a confidence score to every category assignment
- Review the resulting categories and their confidence distribution
- Review point
-
The confidence scores attached to each AI-derived category. Low-confidence assignments are the pile a human reads.
- Success criteria
- Needs Colin’s confirmation Not stated in the source material.
- Result
Ran in about one hour at a cost of about one hundred dollars. The finding that mattered: a large share of recorded churn was business closures and acquisitions rather than product dissatisfaction, which had been quietly distorting the retention metrics.
- Failure modes
- Needs Colin’s confirmation Not stated in the source material.
- Maintenance cost
- Needs Colin’s confirmation Not stated in the source material.
- Tools named
- Snowflake
- Date
, written up
- Author
This study is written up from Colin’s published description of the run. Every number here is his. The four fields flagged above are not in the source material, and they are not filled in with a plausible guess, because the value of a field study is that a reader can repeat it.
Why this one is worth copying
The interesting part is not the cost. It is that the output changed a metric rather than producing a cleaner list. The churn categories that the business had been using could not represent “this company no longer exists,” so every closure was being counted as a customer who left. Reading the whole population at once is what made that visible, and reading it one account at a time for a year had not.
The part that generalizes
Letting the model derive the categories, instead of sorting into the taxonomy already in use, is what surfaced the gap. A taxonomy encodes the assumptions that produced the blind spot. If you classify into it, you get a tidier version of the same blind spot.
What a reader still needs
Before treating this as a template, get the four flagged fields. In particular the maintenance cost, because a one-hour, one-hundred-dollar run that has to happen monthly is a different proposition from one that happens once.
Open with Colin
Needs Colin’s confirmation- successCriteria: what counted as success before the run started
- failureModes: where this broke or produced junk
- maintenanceCost: whether this is a repeatable job or was a one-off
- The date the run actually happened
- Which model or service did the categorization
- Whether the retention metric was actually changed as a result