Skip to main content
 首页 » 编程设计

python之如何为每个句子(行)创建标记化单词(列)的数据框

2024年11月01日9lautakyan007

我有以下文字:

“大家好,我叫sam!我喜欢麻辣拉面。我也很喜欢打游戏。”

我的目标是将此段落转换为每个句子的标记化单词数据框。 (其中行数等于句子数,列数等于最长句子中的单词数)。

我开始创建标记化句子的数据框:

from nltk.tokenize import sent_tokenize, word_tokenize 
 
df = pd.DataFrame({"sentences": sent_tokenize(paragraph)}) 

结果是:

    sentences 
0   Hi there, my name is sam! 
1   I love spicy hand pulled noodles. 
2   I also like to game alot. 

然后我将每个句子(行)转换为标记化单词列表:

df["tokens"] = df.sentences.apply(word_tokenize) 

结果是(如果我单独打印该列):

0    [Hi, there, ,, my, name, is, sam, !] 
1    [I, love, spicy, hand, pulled, noodles, .] 
2    [I, also, like, to, game, alot, .] 

接下来我希望发生的事情是这样的(这里需要帮助):

      w1   w2     w3      w4     w5       w6       w7     w8 
0     Hi   there  ,       my     name     is       sam    ! 
1     I    love   spicy   hand   pulled   noodles  .      NaN 
2     I    also   like    to     game     alot     .      NaN 

其中列数等于最长word_tokenized句子的长度。对于比最长的句子短的句子,我希望空列包含 NaN 值(甚至 0.0)。有没有办法用 pandas 命令实现这一点?

请您参考如下方法:

如果第一个前缀列以 1 (w1) 开头:

In [350]: df.join(pd.DataFrame(df['tokens'].tolist(), columns=[f'w{i}' for i in range(1, df['tokens'].str.len().max() + 1)])).fillna(np.nan)                
Out[350]:  
                           sentences                                      tokens  w1     w2     w3    w4      w5       w6   w7   w8 
0          Hi there, my name is sam!        [Hi, there, ,, my, name, is, sam, !]  Hi  there      ,    my    name       is  sam    ! 
1  I love spicy hand pulled noodles.  [I, love, spicy, hand, pulled, noodles, .]   I   love  spicy  hand  pulled  noodles    .  NaN 
2          I also like to game alot.          [I, also, like, to, game, alot, .]   I   also   like    to    game     alot    .  NaN 

如果您需要它作为一个单独的数据框:

In [352]: pd.DataFrame(df['tokens'].tolist(), columns=[f'w{i}' for i in range(1, df['tokens'].str.len().max() + 1)]).fillna(np.nan)                         
Out[352]:  
   w1     w2     w3    w4      w5       w6   w7   w8 
0  Hi  there      ,    my    name       is  sam    ! 
1   I   love  spicy  hand  pulled  noodles    .  NaN 
2   I   also   like    to    game     alot    .  NaN