Skip to main content
 首页 » 编程设计

python之如何使用scrapy限制蜘蛛抓取某些xPaths

2024年11月01日7zengkefu

我正在尝试抓取网站,我正在尝试从产品页面抓取产品描述,但如何只选择产品描述:

link to page

xPath : hxs.select('//div[@class="product-shop"]/p/text()').extract() 

HTML 非常大,所以请查看上面指定的链接。

我只想选择产品描述而不是其他细节...

如果我这样做:

[" ".join([i.strip() for i in hxs.select('//div[@class="product-shop"]/p/text()').extract()])] 
 
output :  
[u'Itemcode: 12BTS28271 Brand: BASICS InStock - Ships within 2 business days. Tip: 90% of our shipments reach within 4 business days! This product is part of the Basics T.shirts line made of 100% Cotton. Stripes Muscle Fit T.shirts that come in Green Color. Casual that comes with Henley away.'] 

但我只想要:

[u'This product is part of the Basics T.shirts line made of 100% Cotton. Stripes Muscle Fit T.shirts that come in Green Color. Casual that comes with Henley away.'] 

请您参考如下方法:

右键单击 chrome 元素面板中的元素会告诉我:

//*[@id="product_addtocart_form"]/div[2]/div[1]/p[3] 

指向

<p>This product is part of the Basics T.shirts line made of 100% Cotton.<br> 
                        Stripes Muscle Fit T.shirts that come in Green Color.<br> 
                        Casual that comes with Henley away.</p> 

在 this page 上尝试相同的 XPATH也指向那里的描述:

<p>This product is part of the Basics Shirts line made of 100% Cotton.<br> 
                    Plain Slim Fit Shirts that come in Orange Color.<br> 
                    Casual that comes with Button Down away.</p> 

所以看起来您需要做的就是在页面上调用该 XPATH,然后就可以了。不过,您仍应验证 XPATH 是否适用于所有情况,因为它总是容易根据页面发生变化。