Are you sure you want to put that pop-under on your site?

By David Pratt / Tags: , , / 1 Comment / Published: 27-11-09

If you work on a site that is funded by advertising, then you can guarantee that every once in a while someone will have the great idea to put a pop-under advertisement on it – there are sales targets to meet don’t you know! Us folk who actually use the web know this is a bad idea as it only serves to irritate the user and detract them from what they’re doing.

I had always suspected that pop-unders would have a negative effect on the metrics of a website, but I struggled to find any hard evidence that backed this up. The vast majority of negative words about popups and pop-unders tended to be Nielson esque usability pieces or very subjective “I think” rants, and thus not great for building a convincing case against them.

Some months ago at one of my previous employers, I decided to run a test in agreement with the business, that aimed to truly understand the effect of pop-unders on site metrics, and in turn, the bottom line. With that agreement I teamed up with the resident web analyst and performed this little experiment.

The first thing we did was segment our user base into four groups:

  • A – No popunder served
  • B – 1 popunder every day
  • C – 1 popunder every 3 days
  • D – 1 popunder every 7 days

We then got the ad agency to generate the javascript code for each of the 3 pop-under serving frequencies.  Equipped with this code we then had to segment our users into 4 groups. We did this by targeting the last digit of an incremental six digit number that is our logged in users member ID, and then put them in a group that would be served the targeted advert code:

Last DigitGroup
0, 1 or 2B – 1 pop-under a day
3 or 4C – 1 pop-under every 3 days
5, 6 or 7D – 1 pop-under every 7 days
8 or 9A – No pop-under

We then put the code live, and waited. We had planned to leave the code live for one month, but after about 8 days we thought we’d take a quick look at the results to see how they were progressing. At just over a week into the experiment it became apparent that we already had enough information to build a convincing argument against pop-unders.

The below table looks at customer behaviour and the associated revenue (I’ve twiddled the real figures, but kept the ratios) we got from our pop-under trial. It shows how much revenue was generated from logged in members based upon both pop-unders and ad space.

segmentuniquesvisitsimpressionsrepeat visitsvisits per impressionpop under revenuead revenuetotal
C – 1 every 3 days7188231654863233.2220.99£0.05£1.621.67
no pop-under6102203654345023.3421.34£1.711.71
1 a day7833212284075152.7119.20£
1 every 7 days3770129592881443.4422.24£0.02£1.831.85

We can see that members who are served one pop-under per day repeat visit 0.6 times fewer than the control group and see 2 pages fewer per visit. Aggregating this out to a revenue figure (assumptions below), means that so far in the trial, those members who have seen one pop-under a day drive 40p per unique less in revenue than the control group. Therefore in order to make pop-unders viable for the business, they would need to pay circa £50CPM for a 1 a day serving frequency.

The group that was served 1 pop-under every 3 days also delivered slightly reduced revenue (-4p), but the 1 a week segment outperformed the control group. Had the trial not been canned after little over a week, it would have been interesting to see how these two groups would have performed over a longer period. I did try and encourage the business to see the experiment through, but suddenly they weren’t interested in pop-unders anymore…

£10CPM for pop-unders
5 ads per impression, 60% sell through, £8CPM ad space

Note: I’ve amended the CPM rates above to protect their true value, but have maintained the ratios between the figures.

Category: Tech

Tags: , ,

Posted: on November 27th, 2009 at 9:43 am.

Feeds: RSS 2.0

One Response to “Are you sure you want to put that pop-under on your site?”

Anand October 18th, 2012 at 9:19 am

Do you remember measuring the user engagement in terms of time spent on the site? I’ve been looking for this over Google and have been unsuccessful so far.

That said, great piece; I’m surprised it hasn’t got much love yet.

Leave a reply