Robots.txt & Spiders
In a nutshell:
Web site owners use the /robots.txt file to give instructions about their site to
web robots; this is called The Robots Exclusion Protocol. It works likes this: a robot
wants to vists a Web site URL, say www.example.com/welcome.html. Before it does so, it
firsts checks for www.example.com/robots.txt . . .
User-agent: *
Disallow: /
The "User-agent: *" means this section applies to all robots.
The "Disallow: /" tells the robot that it should not visit any pages on the site.
There are two important considerations when using /robots.txt:
- robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
- the /robots.txt file is a publicly available file. Anyone can see what
sections of your server you don't want robots to use.
The rel="nofollow" is an attribute you can set on an HTML link tag, invented by Google, and adopted by others. Those links won't get any credit when Google ranks websites in the search results, thus removing the main incentive behind blog comment spammers robots.
See Preventing comment spam on the Official Google Blog.
From that description it sounds like it only affects the ranking, and the Google robot may still follow the links and index them. If so, it is different from the robots meta tag NOFOLLOW semantics.
There are a few reasons people believe robots are bad for the Web:
•Certain robot uses overloaded networks and servers.
•Robots are operated by humans, who make configuration
mistakes, or simply don't consider the implications of their actions.
•Web - wide indexing robots build a central database of
documents, which doesn't scale too well to millions of documents on
millions of sites.
But at the same time the majority of robots are well designed, professionally operated, cause no problems, and provide a valuable service in the absence of widely deployed better solutions.
So no, robots aren't inherently bad, nor inherently brilliant, and need careful attention.
A robot is a program that automatically traverses the Web's hypertext structure
by retrieving a document, and recursively retrieving all documents that are
referenced.
Note that "recursive" here doesn't limit the definition to any specific traversal
algorithm; even if a robot applies some heuristic to the selection and order of
documents to visit and spaces out requests over a long space of time, it is still a
robot.
Normal Web browsers are not robots, because they are operated by a human, and don't
automatically retrieve referenced documents (other than inline images).
Web robots are sometimes referred to as Web Wanderers, Web Crawlers, or Spiders.
These names are a bit misleading as they give the impression the software itself
moves between sites like a virus; this not the case, a robot simply visits sites by
requesting documents from them.
About the Robots <META> tag
In a nutshell: You can use a special HTML <META> tag to tell robots not
to index the content of a page, and/or not scan it for links to follow.
For example:
<html>
<head>
<title>...</title>
<meta name="robots"
content="noindex, nofollow">
</head>
There are two important considerations when using the robots <META> tag:
- robots can ignore your <META> tag. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
- the NOFOLLOW directive only applies to links on this page. It's entirely likely that a robot might find the same links on some other page without a NOFOLLOW (perhaps on some other site), and so still arrives at your undesired page.
Don't confuse this NOFOLLOW with the rel="nofollow" link attribute.
Copyright© 2010 Smugglers Bay. All rights reserved!