我有一个类似于以下示例数据的大型数据集:
import pandas as pd
raw_data = {'ID': [1,2,3,4,5,6,7,8,9,10],
'body': ['FITrnXS$100', '$1000rnReason', 'rnIf', 'bevlauedrnrnnext', 'obccrnrnnoncrnrnactionrn', 'rnrnnotification', 'insdrnrnnon', 'rnrnupdated', 'rnreason', 'rnrnrnLOR']}
df = pd.DataFrame(raw_data, columns = ['ID', 'body'])
df
我想做的是使用我在下面的代码中定义的单词列表:
remove_string = ['rn', 'rnr', 'rnrn', 'rnrnrn']
如果找到，然后使用上面的 remove_string 从文本(数据框的“正文”列)中的单词中删除它们。
下表将是预期的结果
ID body cleaned_txt Removed_string
1 FITrnXS$100 FIT XS$100 rn
2 $1000rnReason $1000 Reason rn
3 rnIf IF rn
4 bevlauedrnrnnext bevalue next rnrn
5 obccrnrnnoncrnrnactionrn obcc nonc actionrn rnrn
6 rnrnnotification notification rnrn
7 insdrnrnnon insd non rnrn
8 rnrnupdated updated rnrn
9 rnreason reason rn
10 rnrnrnLOR LOR rnrnrn
不幸的是，我试图将数据转换成如下列表:
text = df['body'].tolist()
然后像这样应用函数:
def clnTxt(text):
txt = [item.replace('rnrn', '\n') for item in text]
txt = [item.replace('nrn', '\n') for item in txt]
return txt
text = clnTxt(text)
这不是正确的方法。我应该能够直接在数据帧上应用函数，因此对于每一行，都会执行清理操作并创建其他列。
只是为我的问题寻找更好的解决方案。
请您参考如下方法:
因为较长的字符串包含较短的字符串，所以顺序很重要。所以通过 [::-1] 的逆列表循环并使用 Series.str.extract值到新列，然后使用 Series.str.replace具有相同的列。
上次使用 DataFrame.dot将所有提取的值与分隔符组合到新列中:
remove_string = ['rn', 'rnr', 'rnrn', 'rnrnrn']
df['cleaned_txt'] = df['body']
for i in remove_string[::-1]:
df[i] = df['cleaned_txt'].str.extract('({})'.format(i))
df['cleaned_txt'] = df['cleaned_txt'].str.replace(i, '')
df['Removed_string'] = (df[remove_string].notna()
.dot(pd.Index(remove_string) + ',')
.str.strip(','))
df = df.drop(remove_string, axis=1)
print (df)
ID body cleaned_txt Removed_string
0 1 FITrnXS$100 FITXS$100 rn
1 2 $1000rnReason $1000Reason rn
2 3 rnIf If rn
3 4 bevlauedrnrnnext bevlauednext rnrn
4 5 obccrnrnnoncrnrnactionrn obccnoncaction rn,rnrn
5 6 rnrnnotification notification rnrn
6 7 insdrnrnnon insdnon rnrn
7 8 rnrnupdated updated rnrn
8 9 rnreason eason rnr
9 10 rnrnrnLOR LOR rnrnrn
如果需要用空格替换:
remove_string = ['rn', 'rnr', 'rnrn', 'rnrnrn']
df['cleaned_txt'] = df['body']
for i in remove_string[::-1]:
df[i] = df['cleaned_txt'].str.extract('({})'.format(i))
df['cleaned_txt'] = df['cleaned_txt'].str.replace(i, ' ')
df['Removed_string'] = (df[remove_string].notna()
.dot(pd.Index(remove_string) + ',')
.str.strip(','))
df = df.drop(remove_string, axis=1)
print (df)
ID body cleaned_txt Removed_string
0 1 FITrnXS$100 FIT XS$100 rn
1 2 $1000rnReason $1000 Reason rn
2 3 rnIf If rn
3 4 bevlauedrnrnnext bevlaued next rnrn
4 5 obccrnrnnoncrnrnactionrn obcc nonc action rn,rnrn
5 6 rnrnnotification notification rnrn
6 7 insdrnrnnon insd non rnrn
7 8 rnrnupdated updated rnrn
8 9 rnreason eason rnr
9 10 rnrnrnLOR LOR rnrnrn
编辑1:
#dictioanry for replace
remove_string = {"rn":" ", "rnr":"\n", "rnrn":"\n", "rnrnrn":"\n"}
#sorting by keys for list of tuples
rem = sorted(remove_string.items(), key=lambda s: len(s[0]), reverse=True)
print (rem)
[('rnrnrn', '\n'), ('rnrn', '\n'), ('rnr', '\n'), ('rn', ' ')]
df['cleaned_txt'] = df['body']
for i, j in rem:
df[i] = df['cleaned_txt'].str.extract('({})'.format(i))
df['cleaned_txt'] = df['cleaned_txt'].str.replace(i, j)
cols = list(remove_string.keys())
df['Removed_string'] = (df[cols].notna().dot(pd.Index(cols) + ',')
.str.strip(','))
df = df.drop(remove_string, axis=1)
print (df)
ID body cleaned_txt Removed_string
0 1 FITrnXS$100 FIT XS$100 rn
1 2 $1000rnReason $1000 Reason rn
2 3 rnIf If rn
3 4 bevlauedrnrnnext bevlaued\nnext rnrn
4 5 obccrnrnnoncrnrnactionrn obcc\nnonc\naction rn,rnrn
5 6 rnrnnotification \nnotification rnrn
6 7 insdrnrnnon insd\nnon rnrn
7 8 rnrnupdated \nupdated rnrn
8 9 rnreason \neason rnr
9 10 rnrnrnLOR \nLOR rnrnrn
