Optimize
5.3K members online now
5.3K members online now
Understand the information in your Optimize reports and in your Google Analytics reports.
Guide Me
star_border
Reply

Content Experiment: Traffic Allocation to Losing Variant

Follower ✭ ✭ ☆
# 1
Follower ✭ ✭ ☆

Running a content experiment.

 

First day the variant performed better.

 

The next 3 days the control performed better.

 

Overall the variant has performed worse, and shows <0.1% probability of outperforming the original. 

 

Yet Google has allocated 75% of traffic so far to the variant. 

 

Any idea why??

Content Experiment: Traffic Allocation to Losing Variant

Follower ✭ ✭ ☆
# 2
Follower ✭ ✭ ☆

No idea -- best I can tell the methodology they use is a bit of a black box. It would be nice if there was a resource that explained the methodology in terms of experimental design and tests of significance.

 

If there is such a resource, please let me know.  

Content Experiment: Traffic Allocation to Losing Variant