1<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"> 2 <head> 3 <title>The Web Robots Pages</title> 4<link id="css" rel="stylesheet" type="text/css" href="/css/main.css" /> 5 </head> 6 <body> 7 <div class="bodypart"> 8 <div id="skipnav"> 9 <a href="#content" accesskey="s">skip to content</a> 10 </div> 11 <div class="header"> 12 <div class="inheaderlabel">Advertisement</div>
13<script type="text/javascript"><!-- 14google_ad_client = "pub-9311532361854131"; 15//728x90, created 11/8/07 16google_ad_slot = "9811015679"; 17google_ad_width = 728; 18google_ad_height = 90; 19//--></script>
vendor: 1 bytes, line 19
19
20<script type="text/javascript" 21src="//pagead2.googlesyndication.com/pagead/show_ads.js"> 22</script>
22 23 24 </div> 25 <div class="main"> 26 <div class="navcol"> 27 <div class="navbox" id="logodiv"> 28 <a href="/" accesskey="h"> 29 <img 30 src="/static/robotstxtwrap.png" 31 alt="robotstxt.org logo" border="0"/> 32 </a> 33 </div> 34 <div class="navbox"> 35 <div class="navboxtitle">Navigation</div> 36 <div class="navboxitems"> 37 <ul> 38 <li><a href="/robotstxt.html">The /robots.txt</a></li> 39 <li><a href="/meta.html"><META> tags</a></li> 40 <li><a href="/faq.html">Frequently Asked Questions</a></li> 41 <li><a href="/mailinglist.html">Mailing list</a></li> 42 <li><a href="/other.html">Other Sites</a></li> 43 <li><a href="/about.html">About robotstxt.org</a></li> 44 </ul> 45 </div> 46 </div> 47 <div class="navbox"> 48 <div class="navboxtitle">Tools</div> 49 <div class="navboxitems"> 50 <ul> 51 <li><a href="/checker.html">/robots.txt checker</a></li> 52 <li><a href="/db.html">Robots Database</a></li> 53 <li><a href="/iplookup.html">IP lookup</a></li> 54 </ul> 55 </div> 56 </div> 57 <div class="navbox" id="leftcolad"> 58 <div class="navboxtitle">Advertisement</div> 59 </div> 60 </div> 61 <div class="rightcol"> 62 <div class="navbox"> 63
63<script type="text/javascript"><!-- 64google_ad_client = "pub-9311532361854131"; 65//160x600, created 11/8/07 66google_ad_slot = "0512796426"; 67google_ad_width = 160; 68google_ad_height = 600; 69//--></script>
69 70
70<script type="text/javascript" 71 src="//pagead2.googlesyndication.com/pagead/show_ads.js"> 72 </script>
72 73 74 </div> 75 </div> 76 <div class="maincol"> 77 <div class="maincolwithfooter"> 78 <div id="content" class="content"> 79<h2>The Web Robots Pages</h2> 80 81<p> Web Robots (also known as Web Wanderers, Crawlers, or Spiders), 82are programs that traverse the Web automatically. Search engines such 83as <a href="http://www.google.com/">Google</a> use them to index the web 84content, spammers use them to scan for email addresses, and they have 85many other uses.</p> 86 87<p>On this site you can learn more about web robots.</p> 88 89<ul class="frontmenu"> 90 91<li><a href="robotstxt.html">About /robots.txt</a> explains what 92/robots.txt is, and how to use it.</li> 93 94<li>The <a href="faq.html">FAQ</a> answers many frequently asked 95questions, such as <a href="/faq/prevent.html">How do I stop robots 96visiting my site?</a> and <a href="/faq/bestlisting.html">How can I get the best listing 97in search engines?"</a></li> 98 99<li>The <a href="other.html">Other Sites</a> page links to external 100resources for robot writers and webmasters.</li> 101 102<li>The <a href="db.html">Robots Database</a> has a list of robots.</li> 103 104<li>The <a href="checker.html">/robots.txt checker</a> can check 105your site's /robots.txt file and meta tags.</li> 106 107<li>The <a href="iplookup.html">IP Lookup</a> can help find out 108more about what robots are visiting you.</li> 109 110</ul> 111 112 </div> 113 </div> 114 <div class="outsidefooter"> 115 <div class="outsidefooterlabel"> 116 Advertisement<br/>
117<script type="text/javascript"><!-- 118google_ad_client = "pub-9311532361854131"; 119//468x60, created 11/8/07 120google_ad_slot = "8696051710"; 121google_ad_width = 468; 122google_ad_height = 60; 123//--></script>
vendor: 1 bytes, line 123
123
124<script type="text/javascript" 125src="//pagead2.googlesyndication.com/pagead/show_ads.js"> 126</script>
126 127 </div> 128 129 </div> 130 </div> 131 </div> 132 </div> 133 <div class="pagefooter"> 134<a href="/about.html">About this site</a> 135| 136<a href="/privacy.html">Privacy and cookies policy</a> 137| 138<a href="/contact.html">Contact us</a> 139| 140<a href="/copyrights.html">© 2007. All rights reserved.</a> 141| Hosted by Mythic Beasts 142 </div> 143 </body> 144</html> 145
Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.