Skip to main content
 首页 » 编程设计

python之如何在 Pandas DataFrame 中有效地添加两列

2024年11月01日9lonelyxmas

我有相当大的数据集(超过 600 万行,只有几列)。当我尝试添加两个 float64 列 (data['C'] = data.A + data.B) 时,出现内存错误:

Traceback (most recent call last): 
  File "01_processData.py", line 354, in <module> 
    prepareData(snp) 
  File "01_processData.py", line 161, in prepareData 
    data['C'] = data.A + data.C 
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py", line 480, in wrapper 
    return_indexers=True) 
  File "/usr/local/lib/python2.7/dist-packages/pandas/tseries/index.py", line 976, in join 
    return_indexers=return_indexers) 
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/index.py", line 1304, in join 
    return_indexers=return_indexers) 
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/index.py", line 1345, in _join_non_unique 
    how=how, sort=True) 
  File "/usr/local/lib/python2.7/dist-packages/pandas/tools/merge.py", line 465, in _get_join_indexers 
    return join_func(left_group_key, right_group_key, max_groups) 
  File "join.pyx", line 152, in pandas.algos.full_outer_join (pandas/algos.c:34716) 
MemoryError 

我知道此操作使用索引来正确计算输出,但它似乎效率低下,因为事实上两列属于同一个 DataFrame,它们具有完美对齐。

我能够通过使用解决问题

data['C'] = data.A.values + data.B.values

但我想知道是否有一种方法可以做到这一点或更优雅的解决方案?

请您参考如下方法:

我无法重现您正在做的事情(因为它不会遇到对齐问题,因为索引是相同的)。

在 master/0.14 中(即将发布)

In [2]: df = DataFrame(np.random.randn(6000000,2),columns=['A','C'],index=pd.MultiIndex.from_product([['foo','bar'],range(3000000)])) 
 
In [3]: df.values.nbytes 
Out[3]: 96000000 
 
In [4]: %memit df['D'] = df['A'] + df['C'] 
maximum of 1: 625.839844 MB per loop 

但是在 0.13.1 中。 (我确实记得在 0.14 中进行了一些优化)

In [3]: %memit df['D'] = df['A'] + df['C'] 
maximum of 1: 1113.671875 MB per loop