The Club

Whilst borrowing Jarvis's keys to the armoury, I've been noticing web crawlers - particularly those from Google and Yahoo - all over the place, leaving muddy footprints in the carpet and poking their noses where guests ordinarily aren't allowed.

Time for a robots.txt file, or meta tags on certain page headers, methinks.

Any thoughts?

_________________
The Revolution Will Not Be Televised, but there'll probably be some clips of it on YouTube. (c/o Stew & Gil Scott Heron)

Oh really?

Where have they been poking? They shouldn't be able to get anywhere they're not supposed to be able to get to.

Sam

_________________

http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."

Off the top of my head, I not sure if I've seen them in places like the dungeon or the North Wing, but I can definitely say that I keep seeing guests apparently "viewing private messages" (amongst others, googlebot's crawler does this a bit) or "posting a message" (which seesm to be the preserve of the legion of yahoo.net web crawlers).

Perhaps I'm being overly paranoid, but I don't think that trying to follow the "reply" link is in the remit of a search engine web crawler, surely?

That said, if it's easier just to ignore them because trying to stop the attempted posting requires too much coding, I'm easy.

_________________
The Revolution Will Not Be Televised, but there'll probably be some clips of it on YouTube. (c/o Stew & Gil Scott Heron)

Hurm. That could just be the way the bot interacts with the forum software, or even just the way the forum software interprets fingering. I shall seek advice from someone more expert than I.

Sam

_________________

http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."

A little info which may or may not be useful:-
There's always at least 1 guest showing on the online list at the bottom of the index - I assumed it was one of you lot keeping an anonymous eye on things?? Of late there have been several, I note a couple of records for numbers online over the past week & most of them weren't members.

PS. Just checked, there's 5 on now.

_________________
I'll think about a sig when I'm sober!

Yeah. I've always noticed that as well.

I could put a bots file up that will just stop them coming near the clubhouse all together. That's probably the easiest way of doing it. The problem with doing anything more delicate is that we'd have to specify all the various filenames. I think.

Sam

_________________

http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."

I've put up a robots.txt file that will disallow bots spidering the Clubhouse. That should reduce the numbers we're getting at the moment while I figure out a way of doing it that isn't quite the dust off and nuke the site from orbit approach.

Sam

_________________

http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."

I may be observant but unfortunately I'm more computer parrot fashion than computer literate, so I'm not much help but keep on firing.

_________________
I'll think about a sig when I'm sober!

I reckon we could probably get away with just a meta-tag which allows robots to index the front page on the forum but instructs them not to follow outgoing links.

That way, the Clubhouse still gets visibility on search engines, but the bots can't tunnel any deeper than the index page.

Because the meta-tag code disappears when I put it in a quote box, see the second link in my opening post on this thread.

(edited to get around format/code tag incompatability)

_________________
The Revolution Will Not Be Televised, but there'll probably be some clips of it on YouTube. (c/o Stew & Gil Scott Heron)

Well, I've uploaded that meta tag for the index file for the default forum template. We'll see if that works.

Sam

_________________

http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."

Hurm. It doesn't seem to be in the code, even though it should be. Darn it.

Sam

_________________

http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."

Right. There is now a very specific robots.txt file as well as a nofollow link in the forum's default index page.

We'll see what that does, shall we?

Sam

_________________

http://ravenfamily.org
"You might remember that 'annoyed' is my natural state!"
http://quantumcoyote.com
"Ya'd think we could just attracts ants, like normal people."