python之将 Perl 脚本转换为 Python : dedupe 2 files based on hash keys

我是 Python 新手，想知道是否有人愿意将一个相当简单的 Perl 脚本示例转换为 Python？

该脚本获取 2 个文件，并通过比较哈希键仅输出第二个文件中的唯一行。它还将重复行输出到文件。我发现使用 Perl 进行重复数据删除的这种方法非常快，并且想看看 Python 的比较情况。

#! /usr/bin/perl 
 
## Compare file1 and file2 and output only the unique lines from file2. 
 
## Opening file1.txt and store the data in a hash. 
open my $file1, '<', "file1.txt" or die $!; 
while ( <$file1> ) { 
    my $name = $_; 
    $file1hash{$name}=$_; 
} 
## Opening file2.txt and store the data in a hash. 
open my $file2, '<', "file2.txt" or die $!; 
 
while  ( <$file2> ) { 
    $name = $_; 
    $file2hash{$name}=$_; 
} 
 
open my $dfh, '>', "duplicate.txt"; 
 
## Compare the keys and remove the duplicate one in the file2 hash 
foreach ( keys %file1hash ) { 
    if ( exists ( $file2hash{$_} )) 
    { 
    print $dfh $file2hash{$_}; 
    delete $file2hash{$_}; 
    } 
} 
 
open my $ofh, '>', "file2_clean.txt"; 
print  $ofh values(%file2hash) ; 

我已经在 2 个超过 100 万行的文件上测试了 perl 和 python 脚本，总时间不到 6 秒。就其服务的商业目的而言，表现非常出色!

我修改了 Kriss 提供的脚本，我对两个结果都非常满意:1) 脚本的性能和 2) 我修改脚本的容易程度使其更加灵活:

#!/usr/bin/env python 
 
import os 
 
filename1 = raw_input("What is the first file name to compare? ") 
filename2 = raw_input("What is the second file name to compare? ") 
 
file1set = set([line for line in file(filename1)]) 
file2set = set([line for line in file(filename2)]) 
 
for name, results in [ 
    (os.path.abspath(os.getcwd()) + "/duplicate.txt", file1set.intersection(file2set)), 
    (os.path.abspath(os.getcwd()) + "/" + filename2 + "_clean.txt", file2set.difference(file1set))]: 
    with file(name, 'w') as fh: 
        for line in results: 
            fh.write(line) 

请您参考如下方法:

如果你不关心顺序，你可以在 Python 中使用集合:

file1=set(open("file1").readlines()) 
file2=set(open("file2").readlines()) 
intersection = file1 & file2 #common lines 
non_intersection = file2 - file1  #uncommon lines (in file2 but not file1) 
for items in intersection: 
    print items 
for nitems in non_intersection: 
    print nitems 

其他方法包括使用 difflib、filecmp 库。

另一种方式，只使用列表比较。

# lines in file2 common with file1 
data1=map(str.rstrip,open("file1").readlines()) 
for line in open("file2"): 
    line=line.rstrip() 
    if line in data1: 
        print line 
 
# lines in file2 not in file1, use "not" 
data1=map(str.rstrip,open("file1").readlines()) 
for line in open("file2"): 
    line=line.rstrip() 
    if not line in data1: 
        print line