Skip to main content
 首页 » 编程设计

python之稀疏数组求和

2025年05月04日8cyq1162

我有一个 3 维数组 (np.ndarray),其中大部分为 0。现在我想在第一维上对它们求和,但这相当慢。我查看了 csr_matrix,但 csr 不支持 3 维数组。有没有更快的方法来对几乎稀疏的 nd 数组求和?以下是我当前代码的摘录。

相关问题: sparse 3d matrix/array in Python? (创建一个自制的稀疏 ndarray 类,矫枉过正?)

r = np.array([  [[1, 0, 0, 0], 
                 [1, 0, 0, 0], 
                 [0, 0, 1, 0]], 
 
                [[0, 1, 0, 0], 
                 [0, 0, 0, 1], 
                 [0, 0, 2, 0]], 
 
                [[0, 1, 0, 0], 
                 [0, 0, 0, 0], 
                 [0, 0, 0, 0]], 
 
                [[0, 0, 0, 1], 
                 [0, 0, 0, 0], 
                 [0, 0, 0, 0]]], dtype=int) 
np.sum(r,axis=0) 
Out[35]:  
array([[1, 2, 0, 1], 
       [1, 0, 0, 1], 
       [0, 0, 3, 0]]) 

编辑

在下面 hpaulj 的回答之后,我做了更多的计时测试,见下文。似乎 reshape 对总和没有多大好处,而将它们转换为 csr_matrix 并返回 numpy 会降低性能。我仍在考虑直接使用索引(下面称为 rand_persons、rand_articles 和 rand_days,因为在我原来的问题中,我做了大ndarray 使用这些索引。

from timeit import timeit 
from scipy.sparse import csr_matrix 
import numpy as np 
 
def create_test_data(): 
    ''' 
    dtype = int64 
    1% nonzero, 1000x1000x100: 1.3 s,  
    1% nonzero, 10000x1000x100: 13.3 s 
    0.1% nonzero, 10000x1000x100: 2.7 s 
    1ppm nonzero, 10000x1000x100: 0.007 s 
    ''' 
    global purchases 
    N_persons = 10000 
    N_articles = 1000 
    N_days = 100 
    purchases = np.zeros(shape=(N_days, N_persons, N_articles), dtype=int) 
    N_elements = N_persons * N_articles * N_days 
    rand_persons = np.random.choice(a=range(N_persons), size=N_elements / 1e6, replace=True) 
    rand_articles = np.random.choice(a=range(N_articles), size=N_elements / 1e6, replace=True) 
    rand_days = np.random.choice(a=range(N_days), size=N_elements / 1e6, replace=True) 
    for (i, j, k) in zip(rand_persons, rand_articles, rand_days): 
        purchases[k, i, j] += 1 
 
def sum_over_first_dim_A(): 
    ''' 
    0.1% nonzero, 10000x1000x99: 1.57s (average over 10) 
    1ppm nonzero, 10000x1000x99: 1.70s (average over 10) 
    ''' 
    global purchases 
    d = purchases[:99, :, :] 
    np.sum(d, axis=0) 
def sum_over_first_dim_B(): 
    ''' 
    0.1% nonzero, 10000x1000x99: 1.55s (average over 10) 
    1ppm nonzero, 10000x1000x99: 1.37s (average over 10) 
    ''' 
    global purchases 
    d = purchases[:99, :, :] 
    (N_days, N_persons, N_articles) = d.shape  
    d.reshape(N_days, -1).sum(0).reshape(N_persons, N_articles)  
def sum_over_first_dim_C(): 
    ''' 
    0.1% nonzero, 10000x1000x99: 7.54s (average over 10) 
    1ppm nonzero, 10000x1000x99: 7.44s (average over 10) 
    ''' 
    global purchases 
    d = purchases[:99, :, :] 
    (N_days, N_persons, N_articles) = d.shape  
    r = csr_matrix(d.reshape(N_days, -1)) 
    t = r.sum(axis=0) 
    np.reshape(t, newshape=(N_persons, N_articles)) 
 
if __name__ == '__main__': 
    print (timeit(create_test_data, number=10)) 
    print (timeit(sum_over_first_dim_A, number=10)) 
    print (timeit(sum_over_first_dim_B, number=10)) 
    print (timeit(sum_over_first_dim_C, number=10)) 

编辑 2

我现在找到了一种更快的求和方法:我用稀疏矩阵创建了一个 numpy 数组。但是,这些矩阵的初始创建还有一些时间。我现在用一个循环来做这个。有什么办法可以加快速度吗?

def create_test_data(): 
    [ ... ] 
    ''' 
    0.1% nonzero, 10000x1000x100: 2.1 s 
    1ppm nonzero, 10000x1000x100: 0.45 s 
    ''' 
    global sp_purchases 
    sp_purchases = np.empty(N_days, dtype=lil_matrix) 
    for i in range(N_days): 
        sp_purchases[i] = lil_matrix((N_persons, N_articles)) 
    for (i, j, k) in zip(rand_persons, rand_articles, rand_days): 
        sp_purchases[k][i, j] += 1 
 
def sum_over_first_dim_D(): 
    ''' 
    0.1% nonzero, 10000x1000x99: 0.47s (average over 10) 
    1ppm nonzero, 10000x1000x99: 0.41s (average over 10) 
    ''' 
    global sp_purchases 
    d = sp_purchases[:99] 
    np.sum(d) 

请您参考如下方法:

您可以 reshape 数组使其成为 2d,求和,然后再整形

r.reshape(4,-1).sum(0).reshape(3,4)   # == r.sum(0) 

reshape 不会增加太多处理时间。您可以将 2d 转换为稀疏,看看是否可以节省时间。我的猜测是您的数组必须非常大且非常稀疏,才能击败直接的 numpy 总和。如果您有其他理由使用稀疏格式,这可能是值得的,但只是为了求和,不行。但是你自己测试一下。