Skip to main content
 首页 » 编程设计

python之以优化方式使用 Pandas 在 DataFrame 中查找和替换

2024年12月31日8JeffreyZhao

我正在尝试从 20K 条评论中查找和替换单词。查找和替换单词存储在 dataframe 中,大约 20000 多个。评论在不同的 dataframe 中,大约 20K。

下面是例子

import pandas as pd 
 
df1 = pd.DataFrame({'Data' : ["Hull Damage happened and its insured by maritime hull insurence company","Non Cash Entry and claims are blocked"]}) 
 
df2 = pd.DataFrame({ 'Find' : ["Insurence","Non cash entry"], 
                    'Replace' : ["Insurance","Blocked"], 
                       })  

我期待下面的输出

op = ["Hull Damage happened and its insured by maritime hull insurance company","Blocked and claims are blocked"]}) 

请帮忙。

我正在使用循环,但它需要 20 多分钟才能完成。 数据中20k条记录,待替换30000个词

""KeywordSynonym""-- Dataframe 在sql中保存查找和替换数据
""backup""-- Dataframe 保存要清理的数据

backup = str(backup) 
TrainingClaimNotes_KwdSyn = [] 
for index,row in KeywordSynonym.iterrows(): 
    word = KeywordSynonym.Synonym[index].lower() 
    value = KeywordSynonym.Keyword[index].lower() 
    my_regex = r"\b(?=\w)" + re.escape(word) + r"\b(?!\w)"  
    if re.search(my_regex,backup): 
        backup = re.sub(my_regex, value, backup)  
    TrainingClaimNotes_KwdSyn.append(backup) 
 
TrainingClaimNotes_KwdSyn_Cmp = backup.split('\'", "\'')  

请您参考如下方法:

使用:

import pandas as pd 
 
df1 = pd.DataFrame({'Data' : ["Hull Damage happened and its insured by maritime hull insurence company","Non Cash Entry and claims are blocked"]}) 
 
df2 = pd.DataFrame({ 'Find' : ["Insurence","Non cash entry"], 
                    'Replace' : ["Insurance","Blocked"], 
                       })  
 
find_repl = dict(zip(df2['Find'].str.lower(), df2['Replace'].str.lower())) 
d2 = {r'(\b){}(\b)'.format(k):r'\1{}\2'.format(v) for k,v in find_repl.items()} 
 
df1['Data_1'] = df1['Data'].str.lower().replace(d2, regex=True) 

输出

>>> print(df1['Data_1'].tolist()) 
['hull damage happened and its insured by maritime hull insurance company', 'blocked and claims are blocked'] 

解释

dict(zip(df2['Find'].str.lower(), df2['Replace'].str.lower())) 在您要替换的内容之间创建一个映射以及您要替换为的字符串 -

{'insurence': 'insurance', 'non cash entry': 'blocked'}

将查找转换为 regex 使其为查找做好准备 -

d2 = {r'(\b){}(\b)'.format(k):r'\1{}\2'.format(v) for k,v in find_repl.items()} 
 
{'(\\b)insurence(\\b)': '\\1insurance\\2', '(\\b)non cash entry(\\b)': '\\1blocked\\2'} 

最后一部分只是制作实际替换 -

df1['Data_1'] = df1['Data'].str.lower().replace(d2, regex=True) 

注意:我在各处都执行了 .lower() 以找到合适的匹配项。显然,您可以将其重新塑造成您想要的样子。