我正在尝试从 20K 条评论中查找和替换单词。查找和替换单词存储在 dataframe 中,大约 20000 多个。评论在不同的 dataframe 中,大约 20K。
下面是例子
import pandas as pd
df1 = pd.DataFrame({'Data' : ["Hull Damage happened and its insured by maritime hull insurence company","Non Cash Entry and claims are blocked"]})
df2 = pd.DataFrame({ 'Find' : ["Insurence","Non cash entry"],
'Replace' : ["Insurance","Blocked"],
})
我期待下面的输出
op = ["Hull Damage happened and its insured by maritime hull insurance company","Blocked and claims are blocked"]})
请帮忙。
我正在使用循环,但它需要 20 多分钟才能完成。 数据中20k条记录,待替换30000个词
""KeywordSynonym""-- Dataframe 在sql中保存查找和替换数据
""backup""-- Dataframe 保存要清理的数据
backup = str(backup)
TrainingClaimNotes_KwdSyn = []
for index,row in KeywordSynonym.iterrows():
word = KeywordSynonym.Synonym[index].lower()
value = KeywordSynonym.Keyword[index].lower()
my_regex = r"\b(?=\w)" + re.escape(word) + r"\b(?!\w)"
if re.search(my_regex,backup):
backup = re.sub(my_regex, value, backup)
TrainingClaimNotes_KwdSyn.append(backup)
TrainingClaimNotes_KwdSyn_Cmp = backup.split('\'", "\'')
请您参考如下方法:
使用:
import pandas as pd
df1 = pd.DataFrame({'Data' : ["Hull Damage happened and its insured by maritime hull insurence company","Non Cash Entry and claims are blocked"]})
df2 = pd.DataFrame({ 'Find' : ["Insurence","Non cash entry"],
'Replace' : ["Insurance","Blocked"],
})
find_repl = dict(zip(df2['Find'].str.lower(), df2['Replace'].str.lower()))
d2 = {r'(\b){}(\b)'.format(k):r'\1{}\2'.format(v) for k,v in find_repl.items()}
df1['Data_1'] = df1['Data'].str.lower().replace(d2, regex=True)
输出
>>> print(df1['Data_1'].tolist())
['hull damage happened and its insured by maritime hull insurance company', 'blocked and claims are blocked']
解释
dict(zip(df2['Find'].str.lower(), df2['Replace'].str.lower())) 在您要替换的内容之间创建一个映射以及您要替换为的字符串 -
{'insurence': 'insurance', 'non cash entry': 'blocked'}
将查找转换为 regex 使其为查找做好准备 -
d2 = {r'(\b){}(\b)'.format(k):r'\1{}\2'.format(v) for k,v in find_repl.items()}
{'(\\b)insurence(\\b)': '\\1insurance\\2', '(\\b)non cash entry(\\b)': '\\1blocked\\2'}
最后一部分只是制作实际替换 -
df1['Data_1'] = df1['Data'].str.lower().replace(d2, regex=True)
注意:我在各处都执行了 .lower() 以找到合适的匹配项。显然,您可以将其重新塑造成您想要的样子。
