Skip to main content
 首页 » 编程设计

python之使用 nltk.word_tokenize 在 Pandas 数据框中生成错误 "expected string or bytes-like object"

2025年02月15日8langtianya

对于以下数据框:

index      sentences                                            category 
1          the side effects are terrible !                         SSRI 
2          They are killing me,,, I want to stop                   SNRI 
3          I need to contact my physicians ?                        SSRI 
4          How to stop it.. I am surprised because of its effect.   SSRI 
5                                                                   SSRI 
6                    NAN                                            SNRI 

我正在尝试标记句子列中的句子。 sentences 列有一些空值。这是我的代码,但它不起作用。

df["sentences"] = df.sentences.replace (r'[^a-zA-Z]', '', regex= True, inplace = True) 
 
df["tokenized_sents"] = df["sentences"].apply(nltk.word_tokenize) 

我也试过这个:

df["sentences"] = df.sentences.replace (r'[^a-zA-Z]', 'null', regex= True, inplace = True) 

它会产生以下错误:

expected string or bytes-like object 

有什么建议吗?

请您参考如下方法:

#  I added NaN, None to your date for demonstration, please check below first df. 
print(df)   
 
df["tokenized_sents"] = df["sentences"].fillna("").map(nltk.word_tokenize) 
print(df) 

第一次打印,

   index                                          sentences category 
0      1                    the side effects are terrible !     SSRI 
1      2              They are killing me,,, I want to stop     SNRI 
2      3                  I need to contact my physicians ?     SSRI 
3      4  How to stop it.. I am surprised because of its...     SSRI 
4      5                                                NaN     SNRI 
5      5                                               None     None 

第二次打印,

   index                                          sentences category  \ 
0      1                    the side effects are terrible !     SSRI    
1      2              They are killing me,,, I want to stop     SNRI    
2      3                  I need to contact my physicians ?     SSRI    
3      4  How to stop it.. I am surprised because of its...     SSRI    
4      5                                                NaN     SNRI    
5      5                                               None     None    
 
                                     tokenized_sents   
0             [the, side, effects, are, terrible, !]   
1  [They, are, killing, me, ,, ,, ,, I, want, to,...   
2          [I, need, to, contact, my, physicians, ?]   
3  [How, to, stop, it.., I, am, surprised, becaus...   
4                                                 []   
5                                                 []   

顺便说一句,如果您显式地使用了 inplace=True,则不必再次将它分配给您原来的 df。

df.sentences.replace(r'[^a-zA-Z]', '', regex=True, inplace=True) 
#  instead of, df["sentences"] = df.sentences.replace(r'[^a-zA-Z]', '', regex=True, inplace=True)