about.html

html robotstxt.org collected 2026-09-24 06:53:36 UTC 4,096 bytes, 160 lines download raw bytes

1<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
2  <head>
3    <title>The Web Robots Pages</title>
4<link id="css" rel="stylesheet" type="text/css" href="/css/main.css" />
5  </head>
6  <body>
7    <div class="bodypart">
8      <div id="skipnav">
9	<a href="#content" accesskey="s">skip to content</a>
10      </div>
11      <div class="header">
12 <div class="inheaderlabel">Advertisement</div>

13<script type="text/javascript"><!--
14google_ad_client = "pub-9311532361854131";
15//728x90, created 11/8/07
16google_ad_slot = "9811015679";
17google_ad_width = 728;
18google_ad_height = 90;
19//--></script>

vendor: 1 bytes, line 19

20<script type="text/javascript"
21src="//pagead2.googlesyndication.com/pagead/show_ads.js">
22</script>

22
23
24      </div>
25      <div class="main">
26	<div class="navcol">
27	  <div class="navbox" id="logodiv">
28	    <a href="/" accesskey="h">
29	      <img
30	      src="/static/robotstxtwrap.png"
31	      alt="robotstxt.org logo" border="0"/>
32	    </a>
33	  </div>
34          <div class="navbox">
35	    <div class="navboxtitle">Navigation</div>
36            <div class="navboxitems">
37	      <ul>
38		<li><a href="/robotstxt.html">The /robots.txt</a></li>
39		<li><a href="/meta.html">&lt;META&gt; tags</a></li>
40		<li><a href="/faq.html">Frequently Asked Questions</a></li>
41		<li><a href="/mailinglist.html">Mailing list</a></li>
42		<li><a href="/other.html">Other Sites</a></li>
43		<li><a href="/about.html">About robotstxt.org</a></li>
44	      </ul>
45	    </div>
46	  </div>
47          <div class="navbox">
48	    <div class="navboxtitle">Tools</div>
49            <div class="navboxitems">
50	      <ul>
51		<li><a href="/checker.html">/robots.txt checker</a></li>
52		<li><a href="/db.html">Robots Database</a></li>
53		<li><a href="/iplookup.html">IP lookup</a></li>
54	      </ul>
55            </div>
56          </div>
57          <div class="navbox" id="leftcolad">
58	    <div class="navboxtitle">Advertisement</div>
59          </div>
60        </div>
61      <div class="rightcol">
62        <div class="navbox">
63

63<script type="text/javascript"><!--
64google_ad_client = "pub-9311532361854131";
65//160x600, created 11/8/07
66google_ad_slot = "0512796426";
67google_ad_width = 160;
68google_ad_height = 600;
69//--></script>

69
70

70<script type="text/javascript"
71	src="//pagead2.googlesyndication.com/pagead/show_ads.js">
72	</script>

72
73
74        </div>
75      </div>
76      <div class="maincol">    
77	<div class="maincolwithfooter">
78	  <div id="content" class="content">
79<h2>About robotstxt.org</h2>
80
81<h3>History</h3>
82
83<p>
84
85The Web Robot Pages is an information resource dedicated to web
86robots.  Initially hosted at WebCrawler in 1995, it moved to this
87dedicated site hosted by independent robotstxt.org in 2000. It
88underwent a modernisation in 2007.
89
90</p>
91
92
93<h3>Advertising</h3>
94
95<p>
96
97At this time we do not offer advertising opportunities to new partners,
98nor are we interested in selling the domain.
99
100</p>
101
102<h3>Contact</h3>
103
104<p>
105
106To contact the administrators of this site regarding technical issues
107relating to the operation of this site only, please use the <a
108href="contact.html">contact</a> page.
109
110</p>
111
112<h3>Logo</h3>
113
114<p>
115
116The logo is based on this cartoon donated by
117<a href="http://www.dieselsweeties.com/">
118Diesel Sweeties</a>:
119
120</p>
121
122<div class="centerimg">
123<a href="http://www.dieselsweeties.com/">
124<img src="/static/robotstxt.gif" alt="DieselSweeties cartoon's robot Clango asking for /robots.txt" />
125</a>
126</div>
127
128	  </div>
129        </div>
130        <div class="outsidefooter">
131 	  <div class="outsidefooterlabel">        
132	    Advertisement<br/>

133<script type="text/javascript"><!--
134google_ad_client = "pub-9311532361854131";
135//468x60, created 11/8/07
136google_ad_slot = "8696051710";
137google_ad_width = 468;
138google_ad_height = 60;
139//--></script>

vendor: 1 bytes, line 139

140<script type="text/javascript"
141src="//pagead2.googlesyndication.com/pagead/show_ads.js">
142</script>

142
143	  </div>
144
145        </div>
146      </div>
147      </div>
148    </div>
149    <div class="pagefooter">
150<a href="/about.html">About this site</a>
151|
152<a href="/privacy.html">Privacy and cookies policy</a>
153|
154<a href="/contact.html">Contact us</a>
155|
156<a href="/copyrights.html">&#169; 2007. All rights reserved.</a>
157    </div>
158  </body>
159</html>
160

Line numbers count LF bytes from the start of the resource, as the search results do. Vendor segments are library code the classifier recognised; they are stored but not indexed. Bytes are shown as Latin1 characters, one per byte.