我有一个看起来像这样的 pandas DataFrame
Col1,Col2,Col3
1,"this is a text","more text"
2,"this is another text","even more"
3,"here is one more", "something also here"
4,"let's get another one","we are close"
5,"one last text","finally"
然后我在这些文本上应用了名称实体识别,我提取了一些重要的关键词。像这样
def get_entities(ocr, title):
doc = nlp(' '.join([ocr, title]))
entities = []
for ent in doc.ents:
entity = '_'.join([ent.label_, ent.text])
entities.append(entity)
return set(entities)
df['entities'] = df.apply(lambda row: get_entities( row.Col2, row.Col3), axis = 1)
上面创建了一个名为 entities 的新列,该列的行值是不同关键字的列表。假设是这样
Col1,Col3
1,['key1', 'key2']
2,['key3', 'key2']
3,['key4', 'key1']
4,['key3', 'key4']
5,['key5', 'key2']
现在我尝试在该列上应用 get_dummies 并创建所有可能的关键字,行值为 0-1。以上就是
Col1,Col3,key1,key2,key3,key4,key5
1,['key1', 'key2'],1,1,0,0,0
2,['key3', 'key2'],0,1,1,0,0
3,['key4', 'key1'],1,0,0,1,0
4,['key3', 'key4'],0,0,1,1,0
5,['key5', 'key2'],0,1,0,0,1
当然,直接在列表列上应用get_dummies是行不通的
df = pd.concat([df,pd.get_dummies(df['entities'], prefix='entities')],axis=1)
我会很感激任何想法
请您参考如下方法:
最简单的解决方案是更改您的函数返回的内容。 get_dummies可以处理分隔符分隔的字符串,这很容易从 get_entities 返回方法。
def get_entities(ocr, title):
...
return ','.join(set(entities))
现在您可以使用 get_dummies直接在结果上。以您的第二个示例框架为例,您将得到:
df['Col3'].str.get_dummies(',')
key1 key2 key3 key4 key5
0 1 1 0 0 0
1 0 1 1 0 0
2 1 0 0 1 0
3 0 0 1 1 0
4 0 1 0 0 1
如果您不想更改函数的返回值,请在尝试之前添加另一个步骤 get_dummies使用 str.join
df['Col3'].str.join(',').str.get_dummies(',')
key1 key2 key3 key4 key5
0 1 1 0 0 0
1 0 1 1 0 0
2 1 0 0 1 0
3 0 0 1 1 0
4 0 1 0 0 1
