Suggestion: add a robots.txt file

Points of Required Attention™

Please chime in on a proposed restructuring of the ROM hacking sections.

05-02-24 05:14 AM

Guest: Register | Login

0 users currently in Help/Suggestions/Bug Reports | 1 guest

Main - Help/Suggestions/Bug Reports - Suggestion: add a robots.txt file

New thread | New reply

Cellar Dweller

Posted on 06-12-08 07:26 PM

Link | Quote | ID: 85036

Snifit
Level: 39

Posts: 120/287
EXP: 385364
Next: 19407

Since: 02-19-07
From: Arkansas

Last post: 4057 days
Last view: 3225 days

Every time I look at online.php I see a lot of bots loading redundant pages such as thread.php?pid=* or newreply.php . I think that it would be a good idea to add a robots.txt to save the board's resources, as well as the search engines'. I propose the following contents for it:

User-agent: *
Disallow: /board/calendar.php
Disallow: /board/newreply.php
Disallow: /board/newthread.php
Disallow: /board/thread.php?pid=
Disallow: /trunkboard/

I'm rather surprised that calendar.php is not getting hammered by bots, because there are no meta tags or a robots.txt.

Also, Twiceler and Yeti are not being recognized as bots.

PS: rank++ for the first time in over a year

Ailure

Posted on 06-13-08 06:54 PM

Link | Quote | ID: 85080

Hats
Steam Board2 group
Level: 121

Posts: 2924/3965
EXP: 19788193
Next: 268503

Since: 02-19-07
From: Sweden, Skåne

Last post: 3307 days
Last view: 2058 days

I think it's a decent idea, but on the other hand it's not like the server is collapsing from the load.

Then, having search engines index those pages might be unsuitable, so it probably should be done for that reason. :/

____________________

AIM: gamefreak1337, MSN: Emil_sim@spray.se, XMPP: ailure@xmpp.kafuka.org

blackhole89

Posted on 06-13-08 07:04 PM

Link | Quote | ID: 85083

The Guardian
Moloch whose eyes are a thousand blind windows!
Level: 124

Posts: 1718/4196
EXP: 21542827
Next: 293774

Since: 02-19-07
From: Ithaca, NY, US

Last post: 477 days
Last view: 90 days

Done. I see how this makes sense.

____________________

Main - Help/Suggestions/Bug Reports - Suggestion: add a robots.txt file

New thread | New reply

Page rendered in 0.019 seconds. (323KB of memory used)
MySQL - queries: 37, rows: 48/49, time: 0.015 seconds.