If you work on a site that is funded by advertising, then you can guarantee that every once in a while someone will have the great idea to put a pop-under advertisement on it – there are sales targets to meet don’t you know! Us folk who actually use the web know this is a bad idea as it only serves to irritate the user and detract them from what they’re doing.
I had always suspected that pop-unders would have a negative effect on the metrics of a website, but I struggled to find any hard evidence that backed this up. The vast majority of negative words about popups and pop-unders tended to be Nielson esque usability pieces or very subjective “I think” rants, and thus not great for building a convincing case against them.
Some months ago at one of my previous employers, I decided to run a test in agreement with the business, that aimed to truly understand the effect of pop-unders on site metrics, and in turn, the bottom line. With that agreement I teamed up with the resident web analyst and performed this little experiment.
The first thing we did was segment our user base into four groups:
- A – No popunder served
- B – 1 popunder every day
- C – 1 popunder every 3 days
- D – 1 popunder every 7 days
|0, 1 or 2||B – 1 pop-under a day|
|3 or 4||C – 1 pop-under every 3 days|
|5, 6 or 7||D – 1 pop-under every 7 days|
|8 or 9||A – No pop-under|
We then put the code live, and waited. We had planned to leave the code live for one month, but after about 8 days we thought we’d take a quick look at the results to see how they were progressing. At just over a week into the experiment it became apparent that we already had enough information to build a convincing argument against pop-unders.
The below table looks at customer behaviour and the associated revenue (I’ve twiddled the real figures, but kept the ratios) we got from our pop-under trial. It shows how much revenue was generated from logged in members based upon both pop-unders and ad space.
|segment||uniques||visits||impressions||repeat visits||visits per impression||pop under revenue||ad revenue||total|
|C – 1 every 3 days||7188||23165||486323||3.22||20.99||£0.05||£1.62||1.67|
|1 a day||7833||21228||407515||2.71||19.20||£0.11||1.25||1.36|
|1 every 7 days||3770||12959||288144||3.44||22.24||£0.02||£1.83||1.85|
We can see that members who are served one pop-under per day repeat visit 0.6 times fewer than the control group and see 2 pages fewer per visit. Aggregating this out to a revenue figure (assumptions below), means that so far in the trial, those members who have seen one pop-under a day drive 40p per unique less in revenue than the control group. Therefore in order to make pop-unders viable for the business, they would need to pay circa £50CPM for a 1 a day serving frequency.
The group that was served 1 pop-under every 3 days also delivered slightly reduced revenue (-4p), but the 1 a week segment outperformed the control group. Had the trial not been canned after little over a week, it would have been interesting to see how these two groups would have performed over a longer period. I did try and encourage the business to see the experiment through, but suddenly they weren’t interested in pop-unders anymore…
£10CPM for pop-unders
5 ads per impression, 60% sell through, £8CPM ad space
Note: I’ve amended the CPM rates above to protect their true value, but have maintained the ratios between the figures.