Agency vs Freelance vs In-House CRO
The trade-offs, the hidden costs, and what really drives results
Hi there,
I’m just going to say it upfront: hiring an agency as your first CRO move can be a huge mistake.
And I say that as someone who’s been on all sides: agency, in-house, and freelance. I’ve got the full bingo card. No prize for it, but it does mean I’ve seen the good, the bad, and everything in between.
The issue isn’t that agencies are bad, there are plenty of great ones out there. The issue is that too many companies default to “hire an agency” without considering whether it actually fits their stage, their budget, or the problem they’re trying to solve.
Agencies aren’t going to tell you you’re not ready for them, and why would they?
I’ve put together a full breakdown of when each model works (and when it doesn’t) for Convert.
The question most companies get wrong
Most companies frame the CRO setup decision as: “Who will run the A/B tests?”
The better question is: “Who will own our experimentation strategy, and the learning that comes from it?”
That’s the difference between just doing CRO and actually building an experimentation capability. You can’t permanently outsource experimentation; it has to become part of your company’s DNA.
In the article, I break down the honest trade-offs of each model:
Agency: Great for kickstarting a program or when you need multiple skills at once. But the goal should be to teach your team, not create dependency.
Freelance/consultant: Often deeper involvement and better value. Limited to their expertise, though, so know whether you need a strategist or an executor.
In-house: The strongest long-term choice. Knowledge stays, context compounds, and speed increases. But this only makes sense once you’ve validated that experimentation drives results.
A mix: Most mature programs land here. The key is clarity on who owns the learning.
I also cover the hidden costs that rarely appear in proposals: what happens to your experiment history when an agency leaves, tool fragmentation, and the “context tax” every time someone new comes on board.
Recommendation
Convert, an incredible testing platform for Shopify, sponsors this week’s newsletter. If you’ve got the traffic to run A/B tests, they’re a great platform to use.
If this topic resonates, Convert’s Agency Growth Pathways Report is worth checking out. They studied 237 experimentation agencies and found that 90% claim to offer “strategy,” but the definition varies wildly from one agency to the next.
They also found that 60% of agency practitioners run two or fewer tests per month. Both of these stats should make you ask sharper questions before signing anything.
Whatever setup you choose, remember: you’re not buying CRO. You’re building a system that learns. The best outcome from any external relationship is that your internal team gets smarter.
Till next time,
Daphne