我有相当大的数据集(超过 600 万行,只有几列)。当我尝试添加两个 float64 列 (data['C'] = data.A + data.B) 时,出现内存错误:
Traceback (most recent call last):
File "01_processData.py", line 354, in <module>
prepareData(snp)
File "01_processData.py", line 161, in prepareData
data['C'] = data.A + data.C
File "/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py", line 480, in wrapper
return_indexers=True)
File "/usr/local/lib/python2.7/dist-packages/pandas/tseries/index.py", line 976, in join
return_indexers=return_indexers)
File "/usr/local/lib/python2.7/dist-packages/pandas/core/index.py", line 1304, in join
return_indexers=return_indexers)
File "/usr/local/lib/python2.7/dist-packages/pandas/core/index.py", line 1345, in _join_non_unique
how=how, sort=True)
File "/usr/local/lib/python2.7/dist-packages/pandas/tools/merge.py", line 465, in _get_join_indexers
return join_func(left_group_key, right_group_key, max_groups)
File "join.pyx", line 152, in pandas.algos.full_outer_join (pandas/algos.c:34716)
MemoryError
我知道此操作使用索引来正确计算输出,但它似乎效率低下,因为事实上两列属于同一个 DataFrame,它们具有完美对齐。
我能够通过使用解决问题
data['C'] = data.A.values + data.B.values
但我想知道是否有一种方法可以做到这一点或更优雅的解决方案?
请您参考如下方法:
我无法重现您正在做的事情(因为它不会遇到对齐问题,因为索引是相同的)。
在 master/0.14 中(即将发布)
In [2]: df = DataFrame(np.random.randn(6000000,2),columns=['A','C'],index=pd.MultiIndex.from_product([['foo','bar'],range(3000000)]))
In [3]: df.values.nbytes
Out[3]: 96000000
In [4]: %memit df['D'] = df['A'] + df['C']
maximum of 1: 625.839844 MB per loop
但是在 0.13.1 中。 (我确实记得在 0.14 中进行了一些优化)
In [3]: %memit df['D'] = df['A'] + df['C']
maximum of 1: 1113.671875 MB per loop
