Keep the original group in view
Suppose 100 people enroll in a hypothetical program. Forty complete it, thirty leave before completion and thirty are still enrolled at the review date. Of the 40 completers, 32 meet the study’s stated success definition.
32 of 40
80% of completers met the definition.
32 of 100
32% of all entrants are known to have met it by this review.
30 still enrolled
Their eventual result is unknown. Calling them failures or successes would add an unsupported assumption.
Both fractions can be arithmetically correct. They answer different questions. Reporting 80% as the success rate for everyone who enrolls conceals the people excluded from its denominator—the total group used as the bottom of the fraction. The review date also matters: a recent entrant has had less time to finish than someone who entered years ago.
Make “success” a complete sentence
| Claimed result | What else to ask |
|---|---|
| One account settled | How many accounts remain unresolved? |
| All enrolled accounts resolved | Were debts excluded at intake, added later or paid outside the program? |
| Gross balance reduction | Before or after fees, accrued charges and possible tax? |
| Program completion | Did the person finish the promised work, and what was the cost? |
| Long-term improvement | Which measure, compared with whom, over what follow-up period? |
The FTC’s debt-relief guidance explains why excluding dropouts and unsuccessfully settled debts can make savings advertising deceptive. A testimonial is an individual account, not a population estimate.
Read the methods before accepting the headline
- Population: who was eligible, approached and included? A provider’s screened customers are not all people in debt.
- Funding: who paid for and controlled the analysis? Funding is context, not an automatic verdict on validity.
- Missingness: who did not respond or could not be followed, and were their likely differences examined?
- Comparison: does the study compare similar people, or only before and after without accounting for other changes?
- Costs and time: are fees, taxes, unresolved balances and follow-up duration reported?
The OMB survey standards provide a useful methodological reference for documenting samples, nonresponse and limitations. They govern their stated federal context; they are not a certification label for private studies.
Ask for counts as well as percentages, uncertainty where appropriate and a clear rule for people still in progress. If the necessary detail is unavailable, report the claim as unverified rather than filling the gap with a confident estimate.
A better result is not automatically the program’s effect
Suppose people who complete a program later have lower balances than those who leave. That comparison alone does not show that completion caused the difference. Completers may have started with more stable income, different creditors or smaller debts. A credible analysis explains how comparison groups were formed and which starting differences were addressed.
Likewise, a before-and-after improvement may reflect employment recovery, a household change or a general economic trend. Ask what would likely have happened without the program and what design supports that comparison. Report uncertainty instead of turning an association into a promise that the same result will follow for a new customer.
Sources and further reading
- FTC: outcome advertising and denominators
- OMB:2006 statistical survey standards, archived official PDF
General educational information. The original evidence, dates and recorded qualifications are available in the optional archive below.