[Babase] Lots of darcs processes on papio?
Karl O. Pinc
kop at meme.com
Mon May 3 13:55:41 EDT 2010
On 04/28/2010 11:30:48 AM, Ryan Hardy wrote:
> Actually, I don't think the bots are even seeing the robots.txt at
> all, the way httpd is currently configured. They don't do HTTPS
> requests, so they are getting redirected to the wiki page via HTTPS
> (per the RewriteCond/Rule), unless I am misinterpreting the current
> config.
I think you're right. They are not seeing it.
>
> Adding the following to /etc/httpd/conf.d/Xbabase.conf between lines
> 25 and 26 should do the trick, I think:
>
> # Allow crawlers to get robots.txt so they stop killing us. -rnh,
> 04/28/2010
> RewriteCond %{REQUEST_URI} ^/robots.txt$
> RewriteRule .* - [L]
>
> That should allow non-HTTPS requests for the file to succeed, which
> will hopefully make the bots quite so annoying. I didn't want to
> edit
> the file in the assumption that you have it checked in/out via some
> version control mechanism. I'd be happy to do it, otherwise.
I do have it in version control, but you're welcome to mess with it
anyway so long as you let me know so I can keep things in sync.
Your change seems to work. We'll see. Thanks.
Karl <kop at meme.com>
Free Software: "You don't pay back, you pay forward."
-- Robert A. Heinlein
More information about the Babase
mailing list