Magpie Crawler

This page is intended for webmasters and site owners who have seen “magpie-crawler” in their access logs, or references to Brandwatch in the user-agent string.

What is magpie-crawler?

This is an automated software program, known as a crawler (sometimes called a spider, a robot or just a bot) that downloads web pages to be indexed and analysed by our system. This crawler is managed by us, Brandwatch.

Who are Brandwatch?

We are a social media monitoring company, helping our customers find useful and relevant comments and discussion on the web. We crawl blogs, forums, news sites, and all kinds of social media content. The content is indexed, much like a search engine, allowing our users to find the pages that mention the words they are interested in.

How does our crawling benefit you?

In the same way that search engines help people find content, our crawlers make your pages visible to a wider audience. Our application links to your content, directing more people to read the pages on your site. Many large companies use our services to listen to what people think about their brands, products and services, so there’s a good chance your opinion will be heard.

What to do if you don’t want our crawler downloading pages from your site:

We understand that not everyone wants their pages indexed by a search engine. Our crawler only visits publically available pages, and it respects the robots.txt standard for controlling access to websites.
This allows you, as a site owner, to prevent automated crawling on your site.

Please see http://en.wikipedia.org/wiki/Robots_exclusion_standard for information. If you add the following rule to your robots.txt file, our crawler will not download any pages from your site:

User-agent: magpie-crawler
Disallow: /

How to contact us

If you find that our crawlers are not behaving correctly, or want to contact us, please use the email address listed on our contact page.