Most remodelers and design-build firms sign somewhere between a few dozen and a few hundred jobs a year, spread across several lead sources. That means any one channel might account for only a handful of contracts. With numbers that small, the question isn't just “which channel looks better?” It's “is the difference big enough to trust?”
Why small numbers mislead
Imagine flipping a fair coin ten times. Getting seven heads isn't unusual. Nobody would conclude the coin is rigged. Signing rates work the same way. If a channel signs 2 jobs from 60 leads and another signs 4 from 60, that could easily be the same underlying rate with different luck.
Act on noise and you might cut a channel that was fine, and keep paying for one that only looked good for a few months.
Three questions to ask
1. How many leads and jobs is this based on?
A signing rate built on 300 leads is far more trustworthy than one built on 30. Always look at the counts behind a percentage, not just the percentage.
2. Could chance explain the gap?
Statistical tests answer a specific question: if these channels really signed at the same rate, how likely is a gap this large? If the answer is “very unlikely,” the gap is probably real. In our case study, Angi signed 2 of 306 leads while the firm's other paid sources signed 3.17% of theirs. With that many leads, a gap that size was too large to be chance.
3. Does the conclusion survive the best case?
This is the most useful check for owners. Instead of a single signing rate, work out a range of rates the data is consistent with, then ask: if the channel is actually at the top of that range, does my decision still hold?
If the channel still costs too much in its best plausible case, you can cut it with confidence.
For the case-study firm, the top of the 90% confidence range for Angi's signing rate was about 2.04%, roughly three times what it actually achieved. Even at that rate, a signed job would have cost about $8,850, still 1.8× the other paid sources. The decision held up even when the channel got the benefit of the doubt.
When the answer is “not sure yet”
Sometimes the data honestly can't separate two channels. That's a real answer, and a useful one. It tells you to:
- Judge the channel on cost instead, if one is clearly cheaper for similar results.
- Keep tracking and re-check in six months or a year, when there's more data.
- Tighten your lead-source tracking so the next look is sharper.
What you shouldn't do is treat a coin-flip difference as a verdict. An analysis that won't say “we can't tell” when that's the truth isn't one to trust on the calls it does make.
In every Core Sample audit
Each channel's signing rate comes with an uncertainty range, every recommendation comes with a stated confidence level, and we only make a firm keep-or-cut call when it holds in the channel's best plausible case. How the audit works.