用于带代理支持的多线程蜘蛛的 Python 包

除了使用 urllib 之外，还有谁知道可以通过 http 代理操作的快速、多线程下载 URL 的最有效包吗？我知道一些，例如 Twisted、Scrapy、libcurl 等，但我对它们的了解还不够多，无法做出决定，即使它们是否可以使用代理。有人知道最适合我的目的吗？谢谢!

请您参考如下方法:

用 python 实现这个很简单。

The urlopen() function works transparently with proxies which do not require authentication. In a Unix or Windows environment, set the http_proxy, ftp_proxy or gopher_proxy environment variables to a URL that identifies the proxy server before starting the Python interpreter

# -*- coding: utf-8 -*- 
 
import sys 
from urllib import urlopen 
from BeautifulSoup import BeautifulSoup 
from Queue import Queue, Empty 
from threading import Thread 
 
visited = set() 
queue = Queue() 
 
def get_parser(host, root, charset): 
 
    def parse(): 
        try: 
            while True: 
                url = queue.get_nowait() 
                try: 
                    content = urlopen(url).read().decode(charset) 
                except UnicodeDecodeError: 
                    continue 
                for link in BeautifulSoup(content).findAll('a'): 
                    try: 
                        href = link['href'] 
                    except KeyError: 
                        continue 
                    if not href.startswith('http://'): 
                        href = 'http://%s%s' % (host, href) 
                    if not href.startswith('http://%s%s' % (host, root)): 
                        continue 
                    if href not in visited: 
                        visited.add(href) 
                        queue.put(href) 
                        print href 
        except Empty: 
            pass 
 
    return parse 
 
if __name__ == '__main__': 
    host, root, charset = sys.argv[1:] 
    parser = get_parser(host, root, charset) 
    queue.put('http://%s%s' % (host, root)) 
    workers = [] 
    for i in range(5): 
        worker = Thread(target=parser) 
        worker.start() 
        workers.append(worker) 
    for worker in workers: 
        worker.join()