unflyingobject

I don't know when this happened, but it seems you can't scrape (parse) google.com as easily as before. At least with Python, a simple

s=urllib.urlopen('http://www.google.com/search?q=define%3Aesoteric')

r=s.read()

print r

will give you this notice instead:

/snip/ Your client does not have permission to get URL /search?q=define%3Aesoteric from this server /snip/

And here's me not understanding why my regexes are not matching. :) Apparently, you should at least show some kind of User Agent. In Python you can do this easily by subclassing URLopener and setting it's version property to something, like:

class MyOpener(urllib.URLopener):

  version = "InternetExploiter/666"

urllib._urlopener = MyOpener()

And I'm really starting to love TextMate. Just type your little Python script and hit Cmd-R. Brilliant.