The Club

The League's Clubhouse
It is currently Wed Jan 24, 2018 8:53 am

All times are UTC [ DST ]




Post new topic Reply to topic  [ 12 posts ] 
Author Message
 Post subject: Web crawlers
PostPosted: Mon Jul 02, 2007 3:44 pm 
Offline
Secretary
User avatar

Joined: Tue Sep 20, 2005 4:11 pm
Posts: 806
Location: A wretched hive of scum and villainy
Whilst borrowing Jarvis's keys to the armoury, I've been noticing web crawlers - particularly those from Google and Yahoo - all over the place, leaving muddy footprints in the carpet and poking their noses where guests ordinarily aren't allowed. :evil:

Time for a robots.txt file, or meta tags on certain page headers, methinks.

Any thoughts?

_________________
The Revolution Will Not Be Televised, but there'll probably be some clips of it on YouTube. (c/o Stew & Gil Scott Heron)


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 02, 2007 3:58 pm 
Offline
Archivist
User avatar

Joined: Tue Sep 20, 2005 11:44 am
Posts: 3062
Location: The Seventh Level of Hell
Oh really?

Where have they been poking? They shouldn't be able to get anywhere they're not supposed to be able to get to.

Sam

_________________
:braaak: :borg:
http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 02, 2007 4:37 pm 
Offline
Secretary
User avatar

Joined: Tue Sep 20, 2005 4:11 pm
Posts: 806
Location: A wretched hive of scum and villainy
Off the top of my head, I not sure if I've seen them in places like the dungeon or the North Wing, but I can definitely say that I keep seeing guests apparently "viewing private messages" (amongst others, googlebot's crawler does this a bit) or "posting a message" (which seesm to be the preserve of the legion of yahoo.net web crawlers).

Perhaps I'm being overly paranoid, but I don't think that trying to follow the "reply" link is in the remit of a search engine web crawler, surely?

That said, if it's easier just to ignore them because trying to stop the attempted posting requires too much coding, I'm easy.

_________________
The Revolution Will Not Be Televised, but there'll probably be some clips of it on YouTube. (c/o Stew & Gil Scott Heron)


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 02, 2007 4:39 pm 
Offline
Archivist
User avatar

Joined: Tue Sep 20, 2005 11:44 am
Posts: 3062
Location: The Seventh Level of Hell
Hurm. That could just be the way the bot interacts with the forum software, or even just the way the forum software interprets fingering. I shall seek advice from someone more expert than I.

Sam

_________________
:braaak: :borg:
http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 02, 2007 6:21 pm 
Offline
Incorrigible Member

Joined: Sat Sep 24, 2005 4:39 am
Posts: 1550
Location: In the shed.
A little info which may or may not be useful:-
There's always at least 1 guest showing on the online list at the bottom of the index - I assumed it was one of you lot keeping an anonymous eye on things?? Of late there have been several, I note a couple of records for numbers online over the past week & most of them weren't members.

PS. Just checked, there's 5 on now.

_________________
I'll think about a sig when I'm sober!


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 02, 2007 6:36 pm 
Offline
Archivist
User avatar

Joined: Tue Sep 20, 2005 11:44 am
Posts: 3062
Location: The Seventh Level of Hell
Yeah. I've always noticed that as well.

I could put a bots file up that will just stop them coming near the clubhouse all together. That's probably the easiest way of doing it. The problem with doing anything more delicate is that we'd have to specify all the various filenames. I think.

Sam

_________________
:braaak: :borg:
http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 02, 2007 6:40 pm 
Offline
Archivist
User avatar

Joined: Tue Sep 20, 2005 11:44 am
Posts: 3062
Location: The Seventh Level of Hell
I've put up a robots.txt file that will disallow bots spidering the Clubhouse. That should reduce the numbers we're getting at the moment while I figure out a way of doing it that isn't quite the dust off and nuke the site from orbit approach.

Sam

_________________
:braaak: :borg:
http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 02, 2007 6:55 pm 
Offline
Incorrigible Member

Joined: Sat Sep 24, 2005 4:39 am
Posts: 1550
Location: In the shed.
I may be observant but unfortunately I'm more computer parrot fashion than computer literate, :oops: so I'm not much help but keep on firing. :boomstick: :turret:

_________________
I'll think about a sig when I'm sober!


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 02, 2007 8:23 pm 
Offline
Secretary
User avatar

Joined: Tue Sep 20, 2005 4:11 pm
Posts: 806
Location: A wretched hive of scum and villainy
Ravenbait wrote:
I've put up a robots.txt file that will disallow bots spidering the Clubhouse. That should reduce the numbers we're getting at the moment while I figure out a way of doing it that isn't quite the dust off and nuke the site from orbit approach.

Sam


I reckon we could probably get away with just a meta-tag which allows robots to index the front page on the forum but instructs them not to follow outgoing links.

That way, the Clubhouse still gets visibility on search engines, but the bots can't tunnel any deeper than the index page.

Because the meta-tag code disappears when I put it in a quote box, see the second link in my opening post on this thread.

(edited to get around format/code tag incompatability)

_________________
The Revolution Will Not Be Televised, but there'll probably be some clips of it on YouTube. (c/o Stew & Gil Scott Heron)


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 02, 2007 8:49 pm 
Offline
Archivist
User avatar

Joined: Tue Sep 20, 2005 11:44 am
Posts: 3062
Location: The Seventh Level of Hell
Well, I've uploaded that meta tag for the index file for the default forum template. We'll see if that works.

Sam

_________________
:braaak: :borg:
http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jul 02, 2007 8:50 pm 
Offline
Archivist
User avatar

Joined: Tue Sep 20, 2005 11:44 am
Posts: 3062
Location: The Seventh Level of Hell
Hurm. It doesn't seem to be in the code, even though it should be. Darn it.

Sam

_________________
:braaak: :borg:
http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jul 03, 2007 2:32 pm 
Offline
Archivist
User avatar

Joined: Tue Sep 20, 2005 11:44 am
Posts: 3062
Location: The Seventh Level of Hell
Right. There is now a very specific robots.txt file as well as a nofollow link in the forum's default index page.

We'll see what that does, shall we?

Sam

_________________
:braaak: :borg:
http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 

All times are UTC [ DST ]


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group