• Call Toll Free: 1-855-909-3300

How to Identify and Block “Bad” Magento Traffic

We have heard a lot about ‘bad’ Magento traffic that tries to reach the server of the website. Bad Magento performance is to a large extent is attributed to excessive Bot traffic.

Quite often, bots may get stuck into Magento’s layered navigation feature thus causing the Bot to crawl various combinations of different products and creating several URLs for indexing.

Due to this, more pages will have to be created that will further require more resources.

So, what Exactly is a Bad Robot?

Now you must be thinking what bad traffic is. Bad traffic means an automated robot which is trying to crawl through your website to find all the content through your website.

Typically, robots follow the rules given in the robots.txt file while bad robots ignore whatever is given in robot.txt file.

Robot file

Such bad robots request pages pretty quickly, and try to harvest email addresses.

A bad robot crawling through your Magento site definitely does the same with some malicious intentions that include but may not be limited to copying your content for personal usage.

Identifying a Bad Robot – Steps Discussed…

Simply follow below steps for identification of bad robots –

  • Login to your server through SSH and navigate to your user’s home directory where the Apache access logs are stored.
  • Use the command to see all the user agents that have requests to your website.
  • You will be able to find the requests that are not supplying a user agent string, which won’t happen in case of a human using your website.
    Therefore, these are the bad robots that have become an extra traffic and are a threat to your website.
  • Next, you can find the IP addresses of these robots

Blocking Bad Magento Traffic or Simply Bad Robot…

Now, that you have found out the IP addresses of these bad robots, this is how to block bad Magento traffic –

Edit the .htaccess file for your website using following command –

vim ~userna5/public_html/.htaccess

Once the file has been loaded by vim text editor, click “i” to insert the below code –

ErrorDocument 503 “Site disabled for crawling”

RewriteEngine On

RewriteCond %{HTTP_USER_AGENT} !^.*(Google).*$

RewriteCond %{REMOTE_ADDR} ^74.125

RewriteRule .* – [R=503,L]

Finally, press escape and save the file by holding down shift key and typing ZZ.

These rules initially check whether the user agent includes the word Google in it. If the same is not included anywhere, then it will check if its IP address starts with 74.125.

If it does, then it will serve a 503 response suggesting “Site disabled for crawling”, which deploys minimal server resources rather than letting them hit your website pages that would further require a lot of resources.

One thought on “How to Identify and Block “Bad” Magento Traffic”

Leave a Reply

Techarex NetWorks Products