Remove Selenium and Puppeteer bot traffic from Google Analytics

Edited:

In this other article I explain how to remove bot traffic in GA4.

It is an updated version for Google’s new tool of this same article.

Today we’re going with a new real case that happened to us over the last few weeks. In this case, as the title says, it’s about traffic that one or more bots were pushing into our Google Analytics account and that it couldn’t filter by itself.

It should be made clear that in this case we’re talking about real traffic accessing the site, not the usual referral spam that uses the measurement protocol to manipulate the metrics in our measurement tool.

You may be wondering that, if it’s real traffic, why would we want to remove it from our tool. The answer is simple and blunt: it can lead us to make wrong business decisions caused by bad/corrupt data.

So let’s get into the case.

The problem

For several weeks now, specifically since the beginning of the year, we had detected in both our daily and monthly reporting that something strange was happening and that the number of users compared with sessions had grown considerably:

As you can see, we had grown by 104% in sessions and 114% in users Year on Year. In principle, these figures could fall within what is reasonable and at first glance we blamed them on the changes in third-party cookie blocking by Chrome and Safari and the greater use of Adblockers by users (they were the two main browsers responsible for this anomalous growth between users and sessions) and we simply thought these were the new rules of the game.

More detailed traffic analysis

But the days went by and things started to look even more off in our daily country reporting. The Netherlands was becoming one of the main countries by number of users, but its users’ checkout-entry rate and transactions were much lower than they should have been for that traffic volume:

What is going on? Let’s consider some business hypotheses:

• Are our shipping costs too expensive for shipping to the Netherlands? Perhaps it might be possible to talk to the transport agency and lower prices in exchange for guaranteeing greater volume.
• Do Dutch users browse in English but prefer to buy on pages in their own language? Well, maybe it wouldn’t be a bad idea to consider translating the website into Dutch…
• Don’t they trust southern Europe? Not much we can do here XD

All of them could be valid, but let’s focus on traffic from the Netherlands:

What’s going on? How can the growth between users and sessions differ so much? Are Dutch users much more prone to adblockers than the rest of Europe? Doesn’t look like it, does it? Our first hypothesis was wobbling…

Let’s continue with the country analysis, then, and see what we find when we pull the thread.

Channel

We created a segment to see only the metrics for Dutch users and we go to the main reports, starting with source and medium:

Obviously, what is happening with the direct traffic is not normal. We don’t know exactly what it is, but it does not point to normal user behavior.

Landing page

Let’s check which landing pages the Dutch “users” access:

So Analytics is not able to capture the landing page in most cases… This already makes us think we are dealing with a bot with a higher level of sophistication than usual. But let’s keep looking at more things

Countries

Could what we have seen in the Netherlands be happening in more countries? Time to check:

Well, well, look at that bot. So it is not only coming from the Netherlands, but it is entering our website from multiple locations and messing up everyone’s metrics. Great… Does this multi-country bot have some kind of common characteristic?

Screen resolution

Is it a mobile or desktop bot? Come on, surely we’ll find something here…

And yes, we finally found what we were looking for. In position 11 we found a fairly odd resolution of 1360×1020. It’s clearly a desktop resolution, but it is not common at all: I don’t know any monitor on the market, desktop or laptop, with that resolution. We’ll have to poke around and see where it comes from.

Selenium and Puppeteer

With data like that particular resolution in hand, it’s a matter of digging a little to find out that there are two bot technologies, Selenium and Puppeteer, that use it by default.

These are bots that use virtual machines to open a Chromium browser and enter the URL they are given, with the aim of debugging web applications, scraping their content or, really, whatever the bot developer wants.

Digging through the code on GitHub for Selenium, we see this:

There it is, that damn resolution that is ruining our metrics in Google Analytics, because, we repeat, this is real traffic, from bots and not people, but they are browsers accessing the site with who knows what objective. It is not referral spam using the measurement protocol.

Also, it is bypassing Google Analytics automatic filtering, since as you can see, we have it enabled in the view settings:

So then, what can we do to remove it and have cleaner reports?

How to correct it and remove it from Google Analytics

First, let’s say that maybe there is some reason why you might be interested in having that traffic there artificially inflating your metrics. It’s an option. However, if what you want is for Analytics to show you how human users use your digital products you will probably want to remove it from your data.

To do this, broadly speaking, we have two possibilities:

• Block it from the server
• Remove it from Google Analytics

Blocking it from the server

I’m sorry to tell you it is not an easy option at all; let’s see why:

• It is a multi-IP and multi-country bot: it is not easy to build a filtering pattern
• The User Agent is indistinguishable from that of a Chrome/Chromium used by a user
• The screen resolution, which is the key piece of data, is based on JavaScript for detection, so it is not a server-side language but a client-side one

So, is it feasible to block it from the server? Feasible, yes, but also complicated and hardly recommendable.

OK, and what options do we have so it does not harm our metrics? Keep reading

Remove traffic data in Google Analytics

In this case, we have a very simple solution, which is to use a filter in Google Analytics and apply it to the view(s) we are interested in.

The filter would look like this:

As you can see, a very simple filter that perfectly does its job:

In this image you can see three phases:

• Before January 2021, we had no bot traffic
• Between January and April 20, 2021, with growing bot traffic
• April 21 onward, without bot traffic thanks to the filter applied in Analytics

We consider the problem corrected.

Conclusions

What I want to make clear with this article is the importance of making business decisions based on real and cleaned data.

In this case, we even considered things like lowering shipping costs to the Netherlands or translating the website into Dutch based on wrong data. Imagine the loss of resources and/or money involved if we had carried out either of those two actions. That is, from wrong data, the conclusions I draw are inevitably wrong.

On the other hand, I encourage you to dig into your Analytics accounts and find out whether you are also receiving traffic with this resolution so peculiar, 1360×1200. If so, does your perception of the metrics change once it is removed?

All comments on this are welcome.