Discuss New GA Experiments Algorithm
As everybody know Website Optimizer doesn’t exist anymore. Now we can use Google Analytics experiments. We’ve started using them and have noticed one not pleasant fact. Experiments use new algorithms of traffic division between variants. Variant with better conversion rate gets more traffic. Complicated mathematical model is used and it can bring results faster than classical model with equal traffic division. That’s how it works in theory. And what we see practically.
We’ve made testing landing page. Set up experiment: new vs. old design. First few hours traffic was being divided equally to both pages (about 70 clicks both). These hours old page got 8 button clicks (which was set up as goal), new only 1. We expected the opposite and don’t understand why this happened – at the same page out of experiment conversion rate is much higher. So, after that more than 99% of traffic was sent to old page and experiment didn’t stop. It goes more than week and old page gets 600-1000 clicks daily, new page 1-5 clicks. Old page got 300+ conversions, new page – only 1. So what’s the use of such experiment??
Now let’s look at it from website owner point of view. Experiment tells that new page is 8 times worse, while really we see that difference in button click rate between pages is 20-30% (out of experiment, and purchase rate of new page is even higher) - not even close to 700%. If we told client only experiment results (not telling that it’s 300 conversions against only 1!), he would think that new page is a peace of … and has been made very unprofessional. Actually, it’s only little worse in button-click rate and taking into consideration fact that it brings more purchases, we continue working on it to make buttons more attractive.
So we see, that this new experiment could lead to wrong business decision. What’s the point to use it?
What is interesting – another experiment for another client took place 10 days, conversion rate of new page design was 2,5 times higher from the first day. BUT… traffic was divided equally till the last day of experiment. From new algorithm point of view – this shouldn’t happen.
So there’s a question: is this algorithm good enough to implement it and not to give a chance to use old algorithm? Why not to give me opportunity to deliver 10000 clicks to both variants if I want so?
Here is another thought to think over. Let’s imagine:
We have product with the pent-up demand. Usually it takes 2 weeks to customer to buy product. We make new landing page, where we offer 20% sale to those who buy it right now. At the first day new page gets much more purchases than the old one. Experiment leads all the traffic to new page since this. BUT! We need more than 2 weeks to see correct results, month is better. And if this is premium product – may be sale deters some customers from purchase, they want premium expensive product, they don’t want any sales, but maybe not. And we don’t know the truth until we test it! And what we have to do with this new experiment? Nothing. We just need to dance around fire with ordinary reports or use another tool. And it is time – to find it, to understand it, to put codes at landing pages, etc.
I suggest to everybody who is indifferent to this product leave your comments. Maybe this will make some impact to let know Google that there are many professionals who is dissatisfied when Google changes/makes new tools without any alternative to use old ones (as it was with ad rotation method and some other tools and settings). And maybe Google will change something with experiments and maybe next time Adwords customers will take part in such processes.
Best regards, Nikita