I had always suspected that pop-unders would have a negative effect on the metrics of a website, but I struggled to find any hard evidence that backed this up. The vast majority of negative words about popups and pop-unders tended to be Nielson esque usability pieces or very subjective “I think” rants, and thus not great for building a convincing case against them.

Some months ago at one of my previous employers, I decided to run a test in agreement with the business, that aimed to truly understand the effect of pop-unders on site metrics, and in turn, the bottom line. With that agreement I teamed up with the resident web analyst and performed this little experiment.

The first thing we did was segment our user base into four groups:

• A – No popunder served
• B – 1 popunder every day
• C – 1 popunder every 3 days
• D – 1 popunder every 7 days

We then got the ad agency to generate the javascript code for each of the 3 pop-under serving frequencies.  Equipped with this code we then had to segment our users into 4 groups. We did this by targeting the last digit of an incremental six digit number that is our logged in users member ID, and then put them in a group that would be served the targeted advert code:

We then put the code live, and waited. We had planned to leave the code live for one month, but after about 8 days we thought we’d take a quick look at the results to see how they were progressing. At just over a week into the experiment it became apparent that we already had enough information to build a convincing argument against pop-unders.

The below table looks at customer behaviour and the associated revenue (I’ve twiddled the real figures, but kept the ratios) we got from our pop-under trial. It shows how much revenue was generated from logged in members based upon both pop-unders and ad space.

We can see that members who are served one pop-under per day repeat visit 0.6 times fewer than the control group and see 2 pages fewer per visit. Aggregating this out to a revenue figure (assumptions below), means that so far in the trial, those members who have seen one pop-under a day drive 40p per unique less in revenue than the control group. Therefore in order to make pop-unders viable for the business, they would need to pay circa £50CPM for a 1 a day serving frequency.

The group that was served 1 pop-under every 3 days also delivered slightly reduced revenue (-4p), but the 1 a week segment outperformed the control group. Had the trial not been canned after little over a week, it would have been interesting to see how these two groups would have performed over a longer period. I did try and encourage the business to see the experiment through, but suddenly they weren’t interested in pop-unders anymore…

Assumptions:-
£10CPM for pop-unders
5 ads per impression, 60% sell through, £8CPM ad space

Note: I’ve amended the CPM rates above to protect their true value, but have maintained the ratios between the figures.