示例 html:
<div id="foobar" foo="hello;world;bar;baz">blablabla</div>
我正在使用 LinkExtractor 获取属性 foo,即字符串 hello;world;bar;baz。我想知道是否可以将此字符串转换为多个 url 供蜘蛛跟踪,例如 hello.com、world.com 等。
感谢任何帮助。
PS:以下内容可能(或可能没有)有用
process_valueLxmlLinkExtractor的参数process_linksRules的参数
请您参考如下方法:
问题是,如果您使用内置的 LinkExtractor,process_value 可调用函数必须返回单个链接 - 它会失败here就您而言,如果它是一个链接列表。
您必须有一个自定义Parser Link Extractor 这将支持为每个属性提取多个链接,像这样(未测试):
class MyParserLinkExtractor(LxmlParserLinkExtractor):
def _extract_links(self, selector, response_url, response_encoding, base_url):
links = []
# hacky way to get the underlying lxml parsed document
for el, attr, attr_val in self._iter_links(selector.root):
# pseudo lxml.html.HtmlElement.make_links_absolute(base_url)
try:
attr_val = urljoin(base_url, attr_val)
except ValueError:
continue # skipping bogus links
else:
url = self.process_attr(attr_val)
if url is None:
continue
if isinstance(url, unicode):
url = url.encode(response_encoding)
# url here is a list
for item in url:
url = urljoin(response_url, item)
link = Link(item, _collect_string_content(el) or u'',
nofollow=rel_has_nofollow(el.get('rel')))
links.append(link)
return unique_list(links, key=lambda link: link.url) \
if self.unique else links
然后,基于它,定义您的实际链接提取器:
class MyLinkExtractor(LxmlLinkExtractor):
def __init__(self, allow=(), deny=(), allow_domains=(), deny_domains=(), restrict_xpaths=(),
tags=('a', 'area'), attrs=('href',), canonicalize=True,
unique=True, process_value=None, deny_extensions=None, restrict_css=()):
tags, attrs = set(arg_to_iter(tags)), set(arg_to_iter(attrs))
tag_func = lambda x: x in tags
attr_func = lambda x: x in attrs
lx = MyParserLinkExtractor(tag=tag_func, attr=attr_func,
unique=unique, process=process_value)
super(LxmlLinkExtractor, self).__init__(lx, allow=allow, deny=deny,
allow_domains=allow_domains, deny_domains=deny_domains,
restrict_xpaths=restrict_xpaths, restrict_css=restrict_css,
canonicalize=canonicalize, deny_extensions=deny_extensions)
然后您需要定义 tags、attrs 和 process_value:
MyLinkExtractor(tags=["div"], attrs=["foo"], process_value=extract_links)
其中 extract_links 定义为:
def extract_links(value):
return ["https://{}.com".format(part) for part in value.split(";")]
