Skip to main content
 首页 » 编程设计

python之简单的 Python 爬虫/蜘蛛运行时错误

2025年05月04日11落叶无声

我有一个简单的 python 爬虫/蜘蛛,可以在我提供的网站上搜索指定的文本。但在某些站点中,它会正常抓取 2-4 秒,直到发生错误。

到目前为止的代码:

#!/usr/bin/env python 
# -*- coding: utf-8 -*- 
from __future__ import print_function 
import requests, pyquery, urlparse 
 
try: 
    range = xrange 
except NameError: 
    pass 
 
def crawl(seed, depth, terms): 
 
    crawled = set() 
    uris = set([seed]) 
   for level in range(depth): 
       new_uris = set() 
       for uri in uris: 
          if uri in crawled: 
               continue 
          crawled.add(uri) 
          # Get URI contents 
          try: 
              content = requests.get(uri).content 
         except: 
             continue 
         # Look for the terms 
         found = 0 
         for term in terms: 
             if term in content: 
                 found += 1 
          if found > 0: 
              yield (uri, found, level + 1) 
          # Find child URIs, and add them to the new_uris set 
          dom = pyquery.PyQuery(content) 
          for anchor in dom('a'): 
              try: 
                 link = anchor.attrib['href'] 
             except KeyError: 
                    continue 
                new_uri = urlparse.urljoin(uri, link) 
                new_uris.add(new_uri) 
        uris = new_uris 
 
if __name__ == '__main__': 
    import sys 
    if len(sys.argv) < 4: 
        print('usage: ' + sys.argv[0] +  
            "start_url crawl_depth term1 [term2 [...]]") 
       print('       ' + sys.argv[0] +  
           " http://yahoo.com 5 cute 'fluffy kitties'") 
       raise SystemExit 
 
 seed_uri = sys.argv[1] 
 crawl_depth = int(sys.argv[2]) 
 search_terms = sys.argv[3:] 
 
 for uri, count, depth in crawl(seed_uri, crawl_depth, search_terms): 
     print(uri) 

现在假设我想找到所有在其源代码中包含“requireLazy(”的页面。 让我们用 facebook 试试, 如果我执行这个:

python crawler.py https://www.facebook.com 4 '<script>requireLazy(' 

它将正常运行 2-4 秒,然后会出现此错误:

https://www.facebook.com 
https://www.facebook.com/badges/?ref=pf 
https://www.facebook.com/appcenter/category/music/?ref=pf 
https://www.facebook.com/legal/terms 
https://www.facebook.com/ 
... 
 
Traceback (most recent call last): 
  File "crawler.py", line 61, in <module> 
   for uri, count, depth in crawl(seed_uri, crawl_depth, search_terms): 
File "crawler.py", line 38, in crawl 
  dom = pyquery.PyQuery(content) 
File "/usr/local/lib/python2.7/dist-packages/pyquery/pyquery.py", line 226, in __init__ 
  elements = fromstring(context, self.parser) 
File "/usr/local/lib/python2.7/dist-packages/pyquery/pyquery.py", line 70, in fromstring 
  result = getattr(lxml.html, meth)(context) 
File "/usr/lib/python2.7/dist-packages/lxml/html/__init__.py", line 634, in fromstring 
   doc = document_fromstring(html, parser=parser, base_url=base_url, **kw) 
File "/usr/lib/python2.7/dist-packages/lxml/html/__init__.py", line 532, in document_fromstring 
  value = etree.fromstring(html, parser, **kw) 
File "lxml.etree.pyx", line 2754, in lxml.etree.fromstring (src/lxml/lxml.etree.c:54631) 
File "parser.pxi", line 1578, in lxml.etree._parseMemoryDocument (src/lxml/lxml.etree.c:82748) 
File "parser.pxi", line 1457, in lxml.etree._parseDoc (src/lxml/lxml.etree.c:81546) 
File "parser.pxi", line 965, in lxml.etree._BaseParser._parseDoc (src/lxml/lxml.etree.c:78216) 
File "parser.pxi", line 569, in lxml.etree._ParserContext._handleParseResultDoc (src/lxml/lxml.etree.c:74472) 
File "parser.pxi", line 650, in lxml.etree._handleParseResult (src/lxml/lxml.etree.c:75363) 
 File "parser.pxi", line 599, in lxml.etree._raiseParseError (src/lxml/lxml.etree.c:74827) lxml.etree.XMLSyntaxError: line 21: Tag fb:like invalid 

谁能帮我解决这个错误?谢谢。

请您参考如下方法:

您尝试解析的页面内容似乎包含一些无效标签。通常,您能做的最好的事情就是捕获并记录此类错误,然后优雅地前进到下一页。

希望您可以使用 BeautifulSoup 提取下一个要抓取的页面的 URL,它会优雅地处理大部分不良内容。您可以找到有关 BeatifulSoup 及其使用方法的更多详细信息 here .

更新

实际上,在使用爬虫之后,似乎在某些时候页面内容是空的,因此解析器无法加载文档。

我用 BeautifoulSoup 测试了爬虫,它工作正常。如果您需要/想要,我可以与您分享我的更新版本。

您可以轻松地添加空内容检查,但我不确定您还会遇到哪些其他极端情况,因此切换到 BeautifulSoup 似乎是一种更安全的方法。