Spot-on breakdown, Meta's Andromeda update creates non-neutral traffic that breaks traditional A/B testing assumptions, with creative pre-filtering introducing intent variance before users even reach landing pages.
Critical compounding issue: Invalid traffic exacerbates this chaos, as bots infiltrate Andromeda-qualified cohorts (30-50% of early impressions) mimicking high-value behaviors, initial conversion spikes collapse when synthetic patterns drop off, destroying statistical significance beyond just creative differences.
Solution: Implement specialized pre-bid behavioral validation platforms before testing; these leverage device fingerprinting and anomaly detection to eliminate fake signals while preserving genuine filtered traffic, delivering clean variant splits for reliable CRO outcomes. Happy to share benchmarks on IVT distortion in post-Andromeda campaigns if helpful!
The "False Spike" Decay (30–50%): Up to half of early-launch traffic is often bot-driven. When these synthetic patterns drop off after the initial learning phase, conversion rates collapse, often mistaken for creative fatigue.
The Click-to-Session Gap (20–40%): A significant discrepancy between Meta's reported clicks and actual site sessions is a red flag, indicating that your experiment is receiving heavily filtered or invalid traffic before the landing page even loads.
The Conversion "Inflation Tax" (15–25%): Removing invalid traffic typically reveals a true conversion rate 15–25% (absolute) higher than the raw data suggests, meaning you’re likely discarding winning variants based on "junk" signal.
@Devashish Chandra thanks for the question. So randomization will help a bit but with the number of creatives and an ab testing tool not automatically dividing 50/50 per ad small differences in traffic per ad can cause bigger differences as the intent is different upfront. Because meta is pre filtering traffic so heavily and ads skew to different intent audiences that impacts the results. Also as meta learns the traffic shifts so also timing of when someone saw the ad also impacts results.
@Daphne Tideman makes sense, maybe we should also keep a check on how the audience compensation varies across control and variant, and if new campaigns skew that. Because ideally, if the sample size is large enough, the control and variant composition should look identical.
Spot-on breakdown, Meta's Andromeda update creates non-neutral traffic that breaks traditional A/B testing assumptions, with creative pre-filtering introducing intent variance before users even reach landing pages.
Critical compounding issue: Invalid traffic exacerbates this chaos, as bots infiltrate Andromeda-qualified cohorts (30-50% of early impressions) mimicking high-value behaviors, initial conversion spikes collapse when synthetic patterns drop off, destroying statistical significance beyond just creative differences.
Solution: Implement specialized pre-bid behavioral validation platforms before testing; these leverage device fingerprinting and anomaly detection to eliminate fake signals while preserving genuine filtered traffic, delivering clean variant splits for reliable CRO outcomes. Happy to share benchmarks on IVT distortion in post-Andromeda campaigns if helpful!
@ZeroDayDigest would love to see the benchmarks you mentioned
Sure!
The "False Spike" Decay (30–50%): Up to half of early-launch traffic is often bot-driven. When these synthetic patterns drop off after the initial learning phase, conversion rates collapse, often mistaken for creative fatigue.
The Click-to-Session Gap (20–40%): A significant discrepancy between Meta's reported clicks and actual site sessions is a red flag, indicating that your experiment is receiving heavily filtered or invalid traffic before the landing page even loads.
The Conversion "Inflation Tax" (15–25%): Removing invalid traffic typically reveals a true conversion rate 15–25% (absolute) higher than the raw data suggests, meaning you’re likely discarding winning variants based on "junk" signal.
Super interesting, thanks so much for sharing. Great point about the Inflation Tax.
Thanks Daphne, wouldn't randomization take care of any differences introduced by Andromeda?
Assuming both control and variant are getting traffic from a bunch of differnt creatives, which maybe impacted by Andromeda.
@Devashish Chandra thanks for the question. So randomization will help a bit but with the number of creatives and an ab testing tool not automatically dividing 50/50 per ad small differences in traffic per ad can cause bigger differences as the intent is different upfront. Because meta is pre filtering traffic so heavily and ads skew to different intent audiences that impacts the results. Also as meta learns the traffic shifts so also timing of when someone saw the ad also impacts results.
@Daphne Tideman makes sense, maybe we should also keep a check on how the audience compensation varies across control and variant, and if new campaigns skew that. Because ideally, if the sample size is large enough, the control and variant composition should look identical.