Skip to main content
 首页 » 编程设计

Python Pandas 列和模糊匹配+替换

2025年02月15日8mengfanrong

介绍

您好,我正在进行一个项目,该项目要求我用值替换 pandas 文本列中的字典键 - 但可能存在拼写错误。具体来说,我在 pandas 文本列中匹配名称,并将它们替换为“名字”。例如,我会将“tommy”替换为“First Name”。

但是,我意识到在我的字典无法替换的字符串列中存在名称和文本拼写错误的问题。例如“tommmmy”有额外的 m,在我的字典中不是名字。

#Create df  
d = {'message' : pd.Series(['awesome', 'my name is tommmy , please help with...', 'hi tommy , we understand your quest...'])} 
names = ["tommy", "zelda", "marcon"] 
 
#create dict  
namesdict = {r'(^|\s){}($|\s)'.format(el): r'\1FirstName\2' for el in names} 
 
#replace  
d['message'].replace(namesdict, regex = True) 
 
 
 
  #output  
    Out:  
0                                       awesome 
1    my name is tommmy , please help with... 
2    hi FirstName , we understand your quest... 
dtype: object 

所以 "tommmy"与 -> 中的 "tommy"不匹配,我需要处理拼写错误。我考虑过在实际的字典键和值替换之前尝试这样做,比如扫描 pandas 数据框并用适当的名称替换字符串列(“消息”)中的单词。我见过类似的方法,在特定字符串上使用索引,如 this one

但如何使用正确拼写列表匹配和替换 pandas df 中句子中的单词?我可以在 df.series 替换参数中执行此操作吗?我应该坚持使用正则表达式字符串替换吗?*

任何建议表示赞赏。

更新,尝试 Yannis 的回答

我正在尝试 Yannis 的回答,但我需要使用来自外部来源的列表,特别是美国人口普查的名字以进行匹配。但它的全名与我下载的字符串不匹配。

d = {'message' : pd.Series(['awesome', 'my name is tommy , please help with...', 'hi tommy , we understand your quest...'])} 
 
import requests  
r = requests.get('http://deron.meranda.us/data/census-derived-all-first.txt') 
 
#US Census first names (5000 +)  
firstnamelist = re.findall(r'\n(.*?)\s', r.text, re.DOTALL) 
 
 
#turn list to string, force lower case 
fnstring = ', '.join('"{0}"'.format(w) for w in firstnamelist ) 
fnstring  = ','.join(firstnamelist) 
fnstring  = (fnstring.lower()) 
 
 
##turn to list, prepare it so it matches the name preceded by either the beginning of the string or whitespace.   
names = [x.strip() for x in fnstring.split(',')] 
 
 
 
 
#import jellyfish  
import difflib  
 
 
def best_match(tokens, names): 
    for i,t in enumerate(tokens): 
        closest = difflib.get_close_matches(t, names, n=1) 
        if len(closest) > 0: 
            return i, closest[0] 
    return None 
 
def fuzzy_replace(x, y): 
     
    names = y # just a simple replacement list 
    tokens = x.split() 
    res = best_match(tokens, y) 
    if res is not None: 
        pos, replacement = res 
        tokens[pos] = "FirstName" 
        return u" ".join(tokens) 
    return x 
 
d["message"].apply(lambda x: fuzzy_replace(x, names)) 

结果:

Out:  
0                                        FirstName 
1    FirstName name is tommy , please help with... 
2    FirstName tommy , we understand your quest... 

但如果我使用像这样的较小列表,它会起作用:

names = ["tommy", "caitlyn", "kat", "al", "hope"] 
d["message"].apply(lambda x: fuzzy_replace(x, names)) 

是不是名称列表较长导致了问题?

请您参考如下方法:

编辑:

将我的解决方案更改为使用 difflib。核心思想是标记您的输入文本并将每个标记与名称列表匹配。如果 best_match 找到一个匹配项,那么它会报告位置(和最佳匹配字符串),这样您就可以用“FirstName”或您想要的任何内容替换标记。请参阅下面的完整示例:

import pandas as pd 
import difflib 
 
df = pd.DataFrame(data=[(0,"my name is tommmy , please help with"), (1, "hi FirstName , we understand your quest")], columns=["A", "message"]) 
 
def best_match(tokens, names): 
    for i,t in enumerate(tokens): 
        closest = difflib.get_close_matches(t, names, n=1) 
        if len(closest) > 0: 
            return i, closest[0] 
    return None 
 
def fuzzy_replace(x): 
    names = ["tommy", "john"] # just a simple replacement list 
    tokens = x.split() 
    res = best_match(tokens, names) 
    if res is not None: 
        pos, replacement = res 
        tokens[pos] = "FirstName" 
        return u" ".join(tokens) 
    return x 
 
df.message.apply(lambda x: fuzzy_replace(x)) 

你应该得到的输出如下

0    my name is FirstName , please help with 
1    hi FirstName , we understand your quest 
Name: message, dtype: object 

编辑2

经过讨论,我决定再试一次,使用 NLTK 进行词性标注,并仅针对名称列表的 NNP 标签(专有名词)运行模糊匹配。问题是有时标记器没有得到正确的标记,例如“嗨”也可能被标记为专有名词。但是,如果名称列表是小写的,则 get_close_matches 不会将 Hi 与名称匹配,但会匹配所有其他名称。我建议不要将 df["message"] 小写以增加 NLTK 正确标记名称的机会。一个人也可以玩 StanfordNER,但没有任何东西可以 100% 工作。这是代码:

import pandas as pd 
import difflib 
from nltk import pos_tag, wordpunct_tokenize 
import requests  
import re 
 
r = requests.get('http://deron.meranda.us/data/census-derived-all-first.txt') 
 
# US Census first names (5000 +)  
firstnamelist = re.findall(r'\n(.*?)\s', r.text, re.DOTALL) 
 
# turn list to string, force lower case 
# simplified things here 
names = [w.lower() for w in firstnamelist] 
 
 
df = pd.DataFrame(data=[(0,"My name is Tommmy, please help with"),  
                        (1, "Hi Tommy , we understand your question"), 
                        (2, "I don't talk to Johhn any longer"), 
                        (3, 'Michale says this is stupid') 
                       ], columns=["A", "message"]) 
 
def match_names(token, tag): 
    print token, tag 
    if tag == "NNP": 
        best_match = difflib.get_close_matches(token, names, n=1) 
        if len(best_match) > 0: 
            return "FirstName" # or best_match[0] if you want to return the name found 
        else: 
            return token 
    else: 
        return token 
 
def fuzzy_replace(x): 
    tokens = wordpunct_tokenize(x) 
    pos_tokens = pos_tag(tokens) 
    # Every token is a tuple (token, tag) 
    result = [match_names(token, tag) for token, tag in pos_tokens] 
    x = u" ".join(result) 
    return x 
 
df['message'].apply(lambda x: fuzzy_replace(x)) 

然后我得到了输出:

0       My name is FirstName , please help with 
1    Hi FirstName , we understand your question 
2        I don ' t talk to FirstName any longer 
3                 FirstName says this is stupid 
Name: message, dtype: object