有人可以帮助我使用 scrapy 请求类发出请求吗
我已经试过了，但它不起作用:
from scrapy.selector import HtmlXPathSelector
from scrapy.http.request import Request
url = 'http://www.fetise.com'
a = Request(url)
hxs = HtmlXPathSelector(a)
错误是:
Traceback (most recent call last):
File "sa.py", line 83, in <module>
hxs = HtmlXPathSelector(a)
File "/usr/local/lib/python2.7/dist-packages/scrapy/selector/lxmlsel.py",line 31,in __init__
_root = LxmlDocument(response, self._parser)
File "/usr/local/lib/python2.7/dist-packages/scrapy/selector/lxmldocument.py",line 27,in __new__
cache[parser] = _factory(response, parser)
File "/usr/local/lib/python2.7/dist-packages/scrapy/selector/lxmldocument.py",line 13, in _factory
body = response.body_as_unicode().strip().encode('utf8') or '<html/>'AttributeError: 'Request' object has no attribute 'body_as_unicode'`
我知道回调......实际上我首先想从网站上删除 URL，然后将它们用作起始 URL......
请您参考如下方法:
请试试这个:
import urllib
from scrapy.selector import HtmlXPathSelector
from pprint import pprint
url = 'http://www.fetise.com'
data = urllib.urlopen(url).read()
hxs = HtmlXPathSelector(text=data)
lista = hxs.select('//ul[@class="categoryMenu"]/li/ul/li/a/@href').extract()
acb = ["http://www.fetise.com/" + i if "http://www.fetise.com/" not in i else i for i in lista] + [u"http://www.fetise.com/sale"]
pprint(acb)
这是输出:
[u'http://www.fetise.com/apparel/shirts',
u'http://www.fetise.com/apparel/tees',
u'http://www.fetise.com/apparel/tops-and-tees',
u'http://www.fetise.com/accessories/belts',
u'http://www.fetise.com/accessories/cufflinks',
u'http://www.fetise.com/accessories/jewellery',
u'http://www.fetise.com/accessories/lighters',
u'http://www.fetise.com/accessories/others',
u'http://www.fetise.com/accessories/sunglasses',
u'http://www.fetise.com/accessories/ties-cufflinks',
u'http://www.fetise.com/accessories/wallets',
u'http://www.fetise.com/accessories/watches',
u'http://www.fetise.com/footwear/boots',
u'http://www.fetise.com/footwear/casual',
u'http://www.fetise.com/footwear/flats',
u'http://www.fetise.com/footwear/heels',
u'http://www.fetise.com/footwear/loafers',
u'http://www.fetise.com/footwear/sandals',
u'http://www.fetise.com/footwear/shoes',
u'http://www.fetise.com/footwear/slippers',
u'http://www.fetise.com/footwear/sports',
u'http://www.fetise.com/innerwear/boxers',
u'http://www.fetise.com/innerwear/briefs',
u'http://www.fetise.com/personal-care/deos',
u'http://www.fetise.com/personal-care/haircare',
u'http://www.fetise.com/personal-care/perfumes',
u'http://www.fetise.com/personal-care/personal-care',
u'http://www.fetise.com/personal-care/shavers',
u'http://www.fetise.com/apparel/tees/gifts-for-her',
u'http://www.fetise.com/footwear/sandals/gifts-for-her',
u'http://www.fetise.com/footwear/shoes/gifts-for-her',
u'http://www.fetise.com/footwear/heels/gifts-for-her',
u'http://www.fetise.com/footwear/flats/gifts-for-her',
u'http://www.fetise.com/footwear/ballerinas/gifts-for-her',
u'http://www.fetise.com/footwear/loafers/gifts-for-her',
u'http://www.fetise.com/sale']
