我有一个数据框列，其中包含
38909 Hotel is an old style Red Roof and has not bee... 38913 I will never ever stay at this Hotel again. I ... 38914 After being on a bus for -- hours and finally ... 38918 We were excited about our stay at the Blu Aqua... 38922 This hotel has a great location if you want to... Name: Description, dtype: object
I have a bag of words like keys = ['Hotel','old','finally'] but the actual length of keys = 44312
Currently Im using
df.apply(lambda x : sum([i in x for i in keys ]))
根据示例键给出以下输出
38909 2 38913 2 38914 3 38918 0 38922 1 Name: Description, dtype: int64
When I apply this on actual data for just 100 rows timeit gives
1 loop, best of 3: 5.98 s per loop
我有 50000 行。有没有更快的方法在 nltk 或 pandas 中做同样的事情。
编辑: 如果正在寻找文档数组
array([ 'Hotel is an old style Red Roof and has not been renovated up to the new standard, but the price was also not up to the level of the newer style Red Roofs. So, in overview it was an OK stay, and a safe',
'I will never ever stay at this Hotel again. I stayed there a few weeks ago, and I had my doubts the second I checked in. The guy that checked me in, I think his name was Julio, and his name tag read F',
"After being on a bus for -- hours and finally arriving at the Hotel Lawerence at - am, I bawled my eyes out when we got to the room. I realize it's suppose to be a boutique hotel but, there was nothin",
"We were excited about our stay at the Blu Aqua. A new hotel in downtown Chicago. It's architecturally stunning and has generally good reviews on TripAdvisor. The look and feel of the place is great, t",
'This hotel has a great location if you want to be right at Times Square and the theaters. It was an easy couple of blocks for us to go to theater, eat, shop, and see all of the activity day and night '], dtype=object)
请您参考如下方法:
以下代码并不完全等同于您的(慢速)版本，但它演示了这个想法:
keyset = frozenset(keys)
df.apply(lambda x : len(keyset.intersection(x.split())))
差异/限制:
- 在您的版本中，即使一个词作为子字符串包含在文档中的一个词中，也会被计算在内。例如，如果您的
keys包含单词 tyl，则它会因为在您的第一个文档中出现“style”而被计算在内。 - 我的解决方案不考虑文档中的标点符号。例如，第二个文档中的单词 again 来自
split()并附有句号。这可以通过使用删除标点符号的函数预处理文档(或后处理split()的结果)来解决。
