Comments on: How’s Forbidden Land Twisted PageGetter?

By: SpotPlex Closed Down « Matt's Blog

SpotPlex Closed Down « Matt's Blog — Sun, 11 May 2008 06:01:28 +0000

[...] a Digg-like site based on page views instead of voting, has shutdown their site and their hyper [...]

By: Matt’s Blog » Blog Archive » Twisted PageGetter is… SpotPlex?

Tue, 29 Jan 2008 20:35:40 +0000

[...] PageGetter is… SpotPlex? Awhile back, I made a post on a bot called Twisted PageGetter. Twisted PageGetter is a bot that I found while searching through this blog’s access logs, [...]

By: ezXplain

ezXplain — Sun, 27 Jan 2008 21:10:30 +0000

Twisted PageGetter – block or not…

I found a Bot identifying itself as Twisted PageGetter in the access log of my site.
After I googled for it I found on Matt’s Blog that he suggests to block this Bot with an htaccess entry.
This is working quite well, I tried it.
But: I am using …

By: Champi

Champi — Thu, 20 Sep 2007 10:07:10 +0000

I had to block it too.. real annoying.. seems more people have trouble with 207.0.19.182

By: Matt

Matt — Sun, 20 May 2007 03:52:35 +0000

Thanks for taking the time to come and comment on this matter, Paul.

I actually contacted BlogLines, and no, they don’t use “Twisted PageGetter” as their spider’s User-Agent.

By: Paul Querna

Paul Querna — Fri, 18 May 2007 00:58:19 +0000

Hello,

I’m an engineer for Bloglines, and would just like to let you know that the “Twiested PageGetter” is not bloglines — and AFAIK, never was Bloglines. The IP address mentioned by Steve above, has never hosted Bloglines services, so I’m not sure where Steve got this impression.

Bloglines crawlers always have “Bloglines” in their user-agent — even new services or crawlers in development/testing.

Thanks,

-Paul

By: Steve

Steve — Wed, 16 May 2007 08:36:22 +0000

The spider used by Bloglines also shows up as “Twisted PageGetter”, so you may want to think twice about blocking it. (see http://ascher.ca/blog/2004/11/09/bloglines-twisted/)