Analytics
2.9K members online now
Understand information in your reports and troubleshoot reporting issues such as self-referrals, (not set) data, and inaccurate information
 
Guide Me
star_border
Reply

Bug Report: Content Experiments Showing Incorrect Data

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

I suspect that somehow I might have encountered a bug in the GA platform. I was running a simple A/B test on a single page using content experiments when several day into it, I noticed a HUGE spike in the number of sessions for the control variation. 

 

This in turn caused the conversion rate for that variation to plummet way beyond usual levels and end up declaring a winner when it shouldn't have. 

 

analytics settings.pngClearly see settings set correctlyExperiments_-_Analytics.pngHowever this happened in reality

It almost looks as though there was a lot of spam traffic on one day but literally no other report in GA goes even close to backing up those numbers.

2 Expert replyverified_user

Re: Bug Report: Content Experiments Showing Incorrect Data

Rising Star
# 2
Rising Star
I have given up on A-B testing inside of GA in every case the origional design won even though the numbers say otherwise. there are numorous threads here with the same complaint.

Re: Bug Report: Content Experiments Showing Incorrect Data

Top Contributor
# 3
Top Contributor
I can't see how long the test had been running before the spike was shown.
Moshe, AdWords Top Contributor , Twitter | Linkedin | Community Profile | Ad-Globe
Did you find any helpful responses or answers to your query? If yes, please mark it as the ‘Best Answer’

Re: Bug Report: Content Experiments Showing Incorrect Data

Visitor ✭ ✭ ✭
# 4
Visitor ✭ ✭ ✭
So the test began June 20th and ran for 5 days without issues before the big spike and returned to normal the next day.

Re: Bug Report: Content Experiments Showing Incorrect Data

Top Contributor
# 5
Top Contributor
Well... I think it's a king of statistical spike due to the relatively short time the test was run. It takes a while for the system to ""calibrate" and split the traffic correctly between test and control.
Have you seen it again since then?
Moshe, AdWords Top Contributor , Twitter | Linkedin | Community Profile | Ad-Globe
Did you find any helpful responses or answers to your query? If yes, please mark it as the ‘Best Answer’

Re: Bug Report: Content Experiments Showing Incorrect Data

Visitor ✭ ✭ ✭
# 6
Visitor ✭ ✭ ✭
I run multiple tests a month and have never seen anything like this before. I am relatively familiar with the period of time it takes for it to calibrate which is why it was so weird.

As you can probably see from the second picture I posted, GA has finished with all of the statistics and called the test declaring option B a winner with 100% confidence. However, perhaps I did a bad job of explaining myself the real problem is that it says that the original variation had over 5,300 sessions when there is no way in hell that actually happened in real life (it matches no other reports or traffic trends) hence the bug.

Re: Bug Report: Content Experiments Showing Incorrect Data

Top Contributor
# 7
Top Contributor
I agree. And since in statistics you can sometimes get spikes outside the "statistical expected domain", which should be excluded from the test, what I would do is extend the test. This is definitely a statistical error of wrong sampling.
Moshe, AdWords Top Contributor , Twitter | Linkedin | Community Profile | Ad-Globe
Did you find any helpful responses or answers to your query? If yes, please mark it as the ‘Best Answer’

Re: Bug Report: Content Experiments Showing Incorrect Data

[ Edited ]
Visitor ✭ ✭ ✭
# 8
Visitor ✭ ✭ ✭

But there is very clearly a bug in the software here, the content experiments report is showing thousands of sessions for something that never happened.

What is the best way to get this in front of actual support people so they can fix this because you can't have a product where it produces an answer with 100% confidence only to be told "oh yeah, that's a bug, start again".

Re: Bug Report: Content Experiments Showing Incorrect Data

Top Contributor
# 9
Top Contributor
What I'm trying to say is that I don't think it's a system bug or a bug in the algorithm. Its a statistical error in sampling. This could happen in any statistical test.
Moshe, AdWords Top Contributor , Twitter | Linkedin | Community Profile | Ad-Globe
Did you find any helpful responses or answers to your query? If yes, please mark it as the ‘Best Answer’

Re: Bug Report: Content Experiments Showing Incorrect Data

Visitor ✭ ✭ ✭
# 10
Visitor ✭ ✭ ✭
I think there may indeed be a big misunderstanding here....

The problem isn't a sampling one here. It captured all traffic to the home page and based on the numbers shown in the second screenshot I posted, the statistics are 100% correct.

The problem however is the content experiments report ADDED thousands of sessions to one of the variations THAT NEVER HAPPENED IN REAL LIFE. They exist in NO other report, I fail to see how that can be anything other than a bug but that page did NOT see over 7000 sessions to it in 8 days as the second screenshot would have you believe.