Analytics
2.5K members online now
Understand information in your reports and troubleshoot reporting issues such as self-referrals, (not set) data, and inaccurate information
 
Guide Me
star_border
Reply

Bug Report: Content Experiments Finishing Tests Early

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

Hi All,

I have been recently running an experiment where it seems Analytics has automatically stopped running the session because there is not enough data to identify a winner.

 

That is fine and I understand the concept, if something doesn't look like it will be a winner then call it out as such. However, that isn't what happened here. I had the confidence interval set at 95% and this test clearly showed a version with an almost 32% improvement and 94.4% confidence interval

 

To say it doesn't have enough data when it is only 0.6% away from the limit to call it a winner doesn't make any sense for it to automatically shut off, why wouldn't it keep running it for another day?

 

This is not the first time I have seen Analytics do this exact bug but I was wondering if anyone might be able to shed some more light on it for me.experiments_nowinner.png

 

 

 

 

1 Expert replyverified_user

Re: Bug Report: Content Experiments Finishing Tests Early

Top Contributor
# 2
Top Contributor

@Mark H;

The "data" refers to the sampling size ( =experiments sessions)  - not to the confidence  level of the results (= how reliable is  the outcome that the variant would outperform the original)

Though, I do agree than a 32% increase is a result (i.e. a conversion rate)  is an increase which is statistically significant. Hence, practically, though the test ended, there is a winner.

Have you set an end date?

Moshe, AdWords Top Contributor , Twitter | Linkedin | Community Profile | Ad-Globe
Did you find any helpful responses or answers to your query? If yes, please mark it as the ‘Best Answer’

Bug Report: Content Experiments Finishing Tests Early

Follower ✭ ☆ ☆
# 3
Follower ✭ ☆ ☆

I keep running into the same issue

Bug Report: Content Experiments Finishing Tests Early

Top Contributor
# 4
Top Contributor

This thread has been closed to new replies.

If you have a new or unanswered question related to this topic, please start a new thread and post a clear, specific and detailed question that describes your particular situation.

To start a new thread, please go to the top level of the Community, and from there, click on the board that is most closely related to your question. After that, you can click on the pink "+" sign in the lower right corner of the screen to start a new topic.

Thank you.

Moshe, AdWords Top Contributor , Twitter | Linkedin | Community Profile | Ad-Globe
Did you find any helpful responses or answers to your query? If yes, please mark it as the ‘Best Answer’