Training Programs

Mobile Scroll Floater Breakthrough Test

How to test multiple items on a mobile scroll floater—our control, our test, and our winner

Mobile Scroll Floater Breakthrough Test Control

Kim Mateus:
Next we have mobile scroll floater tests. So this one is a breakthrough test. So lots of different things here, Bill that you guys went for. Do you want to talk us through kind of when it makes sense to do a breakthrough test?

Bill Dugan:
Sure. Yeah. What had happened with the mobile scroll floater ad on Cabot is we had noticed a decreasing contribute contribution to the overall email capture. And it seemed to be in a tailspin really. So we said to ourselves, well, we can take the time to do individual element AB testing or let’s swing for the fences and go for a breakthrough.

Mobile Scroll Floater Breakthrough Test

Bill Dugan:
Let’s see if by changing the whole thing, the design, the words, et cetera and functionality, you’ll see when we look at the test screen, and that’s a situation where we want to do that because something is failing. And the trouble with it is if you get a winner, you don’t know which thing that you changed caused the breakthrough. But we’ll take a look at that in a second too. But so on the next screen is the test where we took that same mobile scroll floater, changed the image, we changed the design, the copy and added an email capture box above the submit button. So vastly different.

Kim Mateus:
Basically almost every single element, maybe except for the… well, even the copy. It’ll copy everything, everything totally different.

Bill Dugan:
Everything’s different, total breakthrough test. And the risk with the breakthrough test, of course, is you can have a big loser, but that’s the risk.

Mobile Scroll Floater Breakthrough Test Winner

Kim Mateus:
But we’re still left with not knowing why it won. But then you can… I know we’re going to address the whole trying to figure out why at one thing here in a second. So the control, the test, and the winner was…

Bill Dugan:
The breakthrough was a big winner.

Kim Mateus:
Fascinating. And then you just scratching your head and going, “Hmm, I wonder what is [that caused this to win].

Bill Dugan:
Yeah, what is it? Is it Ben Franklin’s face on that $100 bill? Or was it the red button? So we decided to run a “back test”.

Comments

Leave a Reply