Just over a month after launching its new Webmaster Tools, Bing has announced its enhanced robots.txt tester tool. You are now able to analyse your robots.txt file and find out if there are any issues that would stop Bing from crawling and indexing your site.
Using the new tool couldn’t be simpler. All you have to do is put the URL into the field on the tester. You can even toggle between Bingbot and Adldxbot, the crawler used by Bing Ads.
Bing’s robots.txt tester checks for allow/disallow statements, and it can check a page seconds after you’ve made any updates.
The system will also display the file in the editor with four variations:
The tester also goes even further and will guide you through the fetch and uploading process.
“The Robots.txt tester helps webmasters to not only analyse their robots.txt file and highlight the issues that would prevent them from getting optimally crawled by Bing and other robots; but, also guides them step-by-step from fetching the latest file to uploading the same file at the appropriate address.
Webmasters can submit a URL to the robots.txt Tester tool and it operates as Bingbot and BingAdsBot would, to check the robots.txt file and verifies if the URL has been allowed or blocked accordingly.”
What is a robots.txt file?
When it comes to SEO, having a perfect robots.txt file is essential. It tells search engine crawlers which pages or files it can or can’t request. In fact, it’s one of the few ways you can control search engine activity on your website.
Mistakes in your robots.txt file could cost you rankings – that’s one of the reasons why Bing’s robot.txt tester will be handy. It’ll spot any errors you may have missed.
Have you used Bing’s new robots.txt file tester? Let us know how you got on in the comments, or tweet us @TeamTillison.