Skip to main content
 首页 » 编程设计

python之在 Scrapy 管道中删除重复项值

2024年10月24日11sxdcgaq8080

我在 .json 文件中以这种格式存储了一些结果:

(每行一个项目)

{"category": ["ctg1"], "pages": 3, "websites": ["x1.com","x2.com","x5.com"]} 
{"category": ["ctg2"], "pages": 2, "websites": ["x1.com", "d4.com"]} 
                    . 
                    . 

我试图在不删除整个项目的情况下删除重复值,但没有成功。

代码:

import scrapy 
import json 
import codecs 
from scrapy.exceptions import DropItem 
 
class ResultPipeline(object): 
 
    def __init__(self): 
        self.ids_seen = set() 
        self.file = codecs.open('results.json', 'w', encoding='utf-8') 
 
    def process_item(self, item, spider): 
        for sites in item['websites']: 
            if sites in self.ids_seen: 
                raise DropItem("Duplicate item found: %s" % sites) 
            else: 
                self.ids_seen.add(sites) 
        line = json.dumps(dict(item), ensure_ascii=False) + "\n" 
        self.file.write(line) 
        return item 
 
    def spider_closed(self, spider): 
        self.file.close() 

请您参考如下方法:

不是删除重复项,而是重建 ids_seen 列表中不存在的网站列表。下面的示例代码应该可以工作,尽管它不在您的类结构中。

import json 
 
 
line1 = '{"category": ["ctg1"], "pages": 3, "websites": ["x1.com","x2.com","x5.com"]}' 
line2 = '{"category": ["ctg2"], "pages": 2, "websites": ["x1.com", "d4.com"]}' 
 
lines = (line1, line2) 
 
ids_seen = set() 
 
def process_item(item): 
    item_unique_sites = [] 
    for site in item['websites']: 
        if not site in ids_seen: 
            ids_seen.add(site) 
            item_unique_sites.append(site) 
    # Delete the duplicates 
    item['websites'] = item_unique_sites 
    line = json.dumps(dict(item), ensure_ascii=False) + "\n" 
    print line 
    #self.file.write(line) 
    return item 
 
 
for line in lines: 
    json_data = json.loads(line) 
    process_item(json_data)