[Babase] Lots of darcs processes on papio?

Karl O. Pinc kop at meme.com
Mon May 3 13:55:41 EDT 2010


On 04/28/2010 11:30:48 AM, Ryan Hardy wrote:

> Actually, I don't think the bots are even seeing the robots.txt at
> all, the way httpd is currently configured.  They don't do HTTPS
> requests, so they are getting redirected to the wiki page via HTTPS
> (per the RewriteCond/Rule), unless I am misinterpreting the current
> config.

I think you're right.  They are not seeing it.

> 
> Adding the following to /etc/httpd/conf.d/Xbabase.conf between lines
> 25 and 26 should do the trick, I think:
> 
>   # Allow crawlers to get robots.txt so they stop killing us. -rnh,
> 04/28/2010
>   RewriteCond   %{REQUEST_URI} ^/robots.txt$
>   RewriteRule  .*  - [L]
> 
> That should allow non-HTTPS requests for the file to succeed, which
> will hopefully make the bots quite so annoying.  I didn't want to 
> edit
> the file in the assumption that you have it checked in/out via some
> version control mechanism.  I'd be happy to do it, otherwise.

I do have it in version control, but you're welcome to mess with it
anyway so long as you let me know so I can keep things in sync.

Your change seems to work.  We'll see.  Thanks.



Karl <kop at meme.com>
Free Software:  "You don't pay back, you pay forward."
                 -- Robert A. Heinlein




More information about the Babase mailing list