Skip to main content
 首页 » 编程设计

python从xml中读取数据

2024年11月01日8cyq1162

我正在将 scrapy 与 python 结合使用。

我正在尝试从 xml 文件中获取我的 xpath,如下所示:

def getMasterContainers(self): 
    containers=[] 
    containersFromXML = self.doc.findall('MasterPage/Containers/xpath') 
    for oneXpath in containersFromXML: 
        containers.append(oneXpath.text) 
    return containers 

xml文件是:

<Containers> 
  <xpath>'&apos;.//div[@id="results-list"]/div[@class="item paid-featured-item"]/div[@class="listing-item"]&apos;'</xpath> 
</Containers> 

当我在 cmd 上打印结果时,我得到了这个

container = ''.//div[@id="results-list"]/div[@class="item paid-featured-item"]/div[@class="listing-item"]'' 

我的问题

当我尝试 sel.xpath(self.containers[0]) 时,我没有得到任何结果,但是当我像这样在代码中写入 xpath 时 sel.xpath('手写的xpath')得到当前数据

请帮忙。

请您参考如下方法:

更新:您确定您的问题出在这个 xpath 上吗?您是否确认它不会早于或晚于此 xpath 失败?我不太确定如何使用 scrapy 运行抓取,所以我只是手动运行 XML 解析,然后在真实文档和测试文档上运行以下命令。

first.xml 仅包含 xpath 及其父结构:

<websiteInformation> 
  <MasterPage> 
    <Containers> 
      <xpath>.//div[@id='results-list']/div[@class='item paid-featured-item']/div[@class='listing-item']</xpath> 
    </Containers> 
  </MasterPage> 
</websiteInformation> 

并解析first.xml:

from lxml import etree 
 
doc = etree.parse(open('first.xml')) 
 
containers = [] 
containersFromXML = doc.findall('MasterPage/Containers/xpath') 
for oneXpath in containersFromXML: 
    print oneXpath.text 
    containers.append(oneXpath.text) 

输出:

.//div[@id='results-list']/div[@class='item paid-featured-item']/div[@class='listing-item'] 

看起来不错。

test.html 是:

<html> 
  <body> 
    <div id="results-list"> 
      <div class="item paid-featured-item"> 
        <div class="listing-item">Found A</div> 
      </div> 
      <div class="item paid-featured-item"> 
        <div class="listing-item">Found B</div> 
      </div> 
    </div> 
  </body> 
</html> 

并通过以下方式搜索:

from scrapy.selector import Selector 
 
sel = Selector(text=open('test.html').read()) 
for container in containers: 
    print "Xpath: {}".format(container) 
    result = sel.xpath(container) 
    print "Container: {}".format(len(result)) 
    for elem in result: 
      print elem 

输出:

Xpath: .//div[@id='results-list']/div[@class='item paid-featured-item']/div[@class='listing-item'] 
Container: 2 
<Selector xpath=".//div[@id='results-list']/div[@class='item paid-featured-item']/div[@class='listing-item']" data=u'<div class="listing-item">Found A</div>'> 
<Selector xpath=".//div[@id='results-list']/div[@class='item paid-featured-item']/div[@class='listing-item']" data=u'<div class="listing-item">Found B</div>'> 

搜索wget得到的真实URL结果输出:

Xpath: .//div[@id='results-list']/div[@class='item paid-featured-item']/div[@class='listing-item'] 
Container: 25 
<Selector xpath=".//div[@id='results-list']/div[@class='item paid-featured-item']/div[@class='listing-item']" data=u'<div class="listing-item">\n        \n    '> 
# omitted 23 
<Selector xpath=".//div[@id='results-list']/div[@class='item paid-featured-item']/div[@class='listing-item']" data=u'<div class="listing-item">\n        \n    '> 

看起来您的 xpath 字符串在不应该出现的地方有额外的单引号 ( ' )。在 XML 中它看起来像:

<xpath>'&apos;.//div[@id="results-list"]/div[@class="item paid-featured-item"]/div[@class="listing-item"]&apos;'</xpath> 

解析时将(如打印时所示):

''.//div[@id="results-list"]/div[@class="item paid-featured-item"]/div[@class="listing-item"]'' 

你不想要周围的'秒。这是应该的:

.//div[@id="results-list"]/div[@class="item paid-featured-item"]/div[@class="listing-item"] 

如果您可以编辑包含您的 xpath 的 XML 文件,请删除前导 '&apos;和尾随 &apos;'来自每个 <xpath> .所以:

<Containers> 
  <xpath>'&apos;.//div[@id="results-list"]/div[@class="item paid-featured-item"]/div[@class="listing-item"]&apos;'</xpath> 
</Containers> 

应该变成:

<Containers> 
  <xpath>.//div[@id="results-list"]/div[@class="item paid-featured-item"]/div[@class="listing-item"]</xpath> 
</Containers> 

但是如果由于某种原因你不能编辑 XML 文件,在你得到 xpath 文本后,去掉周围的 '秒。所以:

containers.append(oneXpath.text) 

应该变成:

containers.append(oneXpath.text.strip("'"))