Using HTTP X-Robots Headers for a “hidden” Link deletion

iPage Affordable Web Hosting only $3.50/mo

Hello there! If you are new here, you might want to subscribe to the RSS feed for updates on this topic.

Important: The strategy below is for discussion purposes only and if used could be risky and unethical.

Recently someone asked one of those questions that causes a pause in the room full of people. He asked about the best way of deleting pages with outgoing links on them (catalog buttons, partner links, and so on).

As we all know (or don’t know), banning directories from robots.txt can be detected very easily and will show your level of understanding in this area.

Everyone knows that having millions of links outgoing from your site is not very good. One page with fifteen links is ok, twenty pages with hundred links each is not ok. This will lower your site’s PageRank and cause your site to fall under other multiple google bot and other engine bot filters (you don’t want this to happen, and it’s next to impossible to fix).

If all of a sudden your site receives a penalty from Google, then deleting such link junk-yards is the first thing that needs to be done. However, what if your site is still being rendered well, but you want to get rid of the bunch of links before you actually receive a penalty? If you just delete links, then everyone, who exchanged links with you also stop linking to your site and your site rating will fall (not might, it will).

The catch here is to find a way to quietly delete all these link junk-yards, without warning the link partners about it.

Again, please be cool and don’t do this, it’s very unethical and unfair. Ok, thanks for agreeing with me, let’s move on now and get to the juice of this post…

The solution lays in the use of X-Robots HTTP headers, which are supported by Google (by the way).

This will allow adding a meta information “noindex” to the page from the server side, which will leave everything unchanged and unnoticable by your visitors (people or bots) in your robots.txt or HTML code of your page.

header(’X-Robots-Tag: noindex, nofollow’, true);

or by using .htaccess and mod_headers

<IfModule mod_headers.c>
<Files links.html>
Header set X-Robots-Tag “noindex”
</Files>
</IfModule>

Using this method you can delete your pages in Google and, most likely, without warning your partners.

Also, it’s possible, that you might want to do a tiny “cloaking”, so that X-Robots was only seen by Googlebot, and other people wouldn’t see this cruel plan of yours (the less they know the tighter they sleep). This can be done using the header checkers in the code itself, checking User-Agent request, or using .htaccess, if you have a mod_headers module:

<IfModule mod_headers.c>
<IfModule mod_setenvif.c>
SetEnvIf User-Agent “Googlebot$” x_tag=yes
<Files links.html>
Header set X-Robots-Tag “noindex” env=x_tag
</Files>
</IfModule>
</IfModule>

Note to self:

Is it possible to modify that very checker to make it act like a Googlebot, and of course keep checking the http headers of the link partners linked to you (so those sons of guns didn’t do the trick noindexing links linking to you)? This is a rhetorical question:)

Just so you know, the X-Roborts are supported by Yahoo! bot as well:)

Beck @ ProfitSEO.com