Robot.txt file could be useful for your website

Sunday, April 13th, 2008 | In Knowledge |

Most of the time, webmasters are not aware of the robot.txt file for their website for SEO. The robots.txt file can be used to instruct the search engine crawlers about what pages on your website should be crawled and indexed on their database, because most website have files and folder that are not relevant or private for search engines to be crawled therefore it is good practice by adding a robot.txt at your website root directory to improve and make your website efficiently indexed at any search engine.

A robot.txt file is a simple text file that can be written on the notepad, named as robot.txt. for example, you could write a robot.txt for your wordpress blogging website.

User-agent: *
Disallow: /wp-
Disallow: /trackback
Disallow: /images/

User-agent means that all the search bots (Google, Yahoo Msn ..) should use the following instruction to crawl your website;

Disallow: /wp- means the search bots will not crawl the folder that started with wp-, because most of your private data such like admin file, images are located at wp-admin folder.

Disallow: /images/ means to instruct the search bots not to crawl the folder called as “images”.

The robot.txt file should placed at the root directory, otherwise search bots may not able to get those instruction before crawl your website.


4 Comments to “Robot.txt file could be useful for your website”

  1. Comment 1
    iPod Says:

    Great post!! I just have to copy it :)

  2. Comment 2
    Tula Sivalia Says:

    Excellent post.

  3. Comment 3
    Tilda Christan Says:

    Interesting article. Definitely gave me a bit to think about…

  4. Comment 4
    Darien Says:

    I am new to blogging and actually enjoyed your website. I am going to bookmark your website and keep checking you out. Thanks for sharing your site.

Leave a Reply

Turing Number



Little plugin, Just for fun

Some Simple Flash Games


E-mail This Post