我有以下文字:
“大家好,我叫sam!我喜欢麻辣拉面。我也很喜欢打游戏。”
我的目标是将此段落转换为每个句子的标记化单词数据框。 (其中行数等于句子数,列数等于最长句子中的单词数)。
我开始创建标记化句子的数据框:
from nltk.tokenize import sent_tokenize, word_tokenize
df = pd.DataFrame({"sentences": sent_tokenize(paragraph)})
结果是:
sentences
0 Hi there, my name is sam!
1 I love spicy hand pulled noodles.
2 I also like to game alot.
然后我将每个句子(行)转换为标记化单词列表:
df["tokens"] = df.sentences.apply(word_tokenize)
结果是(如果我单独打印该列):
0 [Hi, there, ,, my, name, is, sam, !]
1 [I, love, spicy, hand, pulled, noodles, .]
2 [I, also, like, to, game, alot, .]
接下来我希望发生的事情是这样的(这里需要帮助):
w1 w2 w3 w4 w5 w6 w7 w8
0 Hi there , my name is sam !
1 I love spicy hand pulled noodles . NaN
2 I also like to game alot . NaN
其中列数等于最长word_tokenized句子的长度。对于比最长的句子短的句子,我希望空列包含 NaN 值(甚至 0.0)。有没有办法用 pandas 命令实现这一点?
请您参考如下方法:
如果第一个前缀列以 1 (w1) 开头:
In [350]: df.join(pd.DataFrame(df['tokens'].tolist(), columns=[f'w{i}' for i in range(1, df['tokens'].str.len().max() + 1)])).fillna(np.nan)
Out[350]:
sentences tokens w1 w2 w3 w4 w5 w6 w7 w8
0 Hi there, my name is sam! [Hi, there, ,, my, name, is, sam, !] Hi there , my name is sam !
1 I love spicy hand pulled noodles. [I, love, spicy, hand, pulled, noodles, .] I love spicy hand pulled noodles . NaN
2 I also like to game alot. [I, also, like, to, game, alot, .] I also like to game alot . NaN
如果您需要它作为一个单独的数据框:
In [352]: pd.DataFrame(df['tokens'].tolist(), columns=[f'w{i}' for i in range(1, df['tokens'].str.len().max() + 1)]).fillna(np.nan)
Out[352]:
w1 w2 w3 w4 w5 w6 w7 w8
0 Hi there , my name is sam !
1 I love spicy hand pulled noodles . NaN
2 I also like to game alot . NaN
