Optimize
3.5K members online now
3.5K members online now
Understand the information in your Optimize reports and in your Google Analytics reports.
Guide Me
star_border
Reply

Targeting 50/50 weight but control receives majority of sessions

Visitor ✭ ✭ ✭
# 1
Visitor ✭ ✭ ✭

I have a test set up with the following targeting settings:

  • Percentage of visitors to target: 100%
  • Weighting of visitors to target: 50% / 50%

 

After 17 days of running the test, the control has seen 10,240 sessions while the variant has only seen 7,930—giving a real world weight of 56.4% for the control and 43.6% for the variant.

 

In case it's also helpful to know, the control has had 580 conversions with a median conversion rate of 5.85%  while the variant has had 572 conversions with a median conversion rate of 7.5%. 

 

Any idea what might be going on?

1 ACCEPTED SOLUTION

Accepted Solutions
Marked as Best Answer.
Solution
Accepted by topic author Peter O
July

Targeting 50/50 weight but control receives majority of sessions

[ Edited ]
Follower ✭ ☆ ☆
# 2
Follower ✭ ☆ ☆

Hi, Peter!

 

Since it's session based and not user based it could be that the control is making the users coming back for a second session or in other words that the variant is so much better than the users don't need a second session to convert.

 

You could open up your google analytics account and create a sequence segment (be sure to include the experiment id first and then the variation, https://support.google.com/analytics/answer/4380401?hl=en.) and be analyzing the results there to see if you have a more even split on a user level.

View solution in original post

Marked as Best Answer.
Solution
Accepted by topic author Peter O
July

Targeting 50/50 weight but control receives majority of sessions

[ Edited ]
Follower ✭ ☆ ☆
# 2
Follower ✭ ☆ ☆

Hi, Peter!

 

Since it's session based and not user based it could be that the control is making the users coming back for a second session or in other words that the variant is so much better than the users don't need a second session to convert.

 

You could open up your google analytics account and create a sequence segment (be sure to include the experiment id first and then the variation, https://support.google.com/analytics/answer/4380401?hl=en.) and be analyzing the results there to see if you have a more even split on a user level.

Targeting 50/50 weight but control receives majority of sessions

Visitor ✭ ✭ ✭
# 3
Visitor ✭ ✭ ✭

We are experiencing a similar issue that is more stark than the case above. The traffic is a 50/50 split, but the variant has received 13,608 sessions while the control has received 8,568 sessions. That is 61/39 split in sessions in favor of the variant. I checked the user breakdown as you suggested and the split is still the same (61/39). What other explanations could there be for Google Optimize being so far off of the experiment split. It makes me worry that something is wrong with Optimize of my experiment setup and thus the results are not valid. 

Targeting 50/50 weight but control receives majority of sessions

Visitor ✭ ✭ ✭
# 4
Visitor ✭ ✭ ✭

I added a new question as well since this one may no longer be reviewed since it has been marked as "answered":

 

https://www.en.advertisercommunity.com/t5/Google-Optimize-Reports/Targeting-50-50-weight-but-variant...