Skip to main content
 首页 » 编程设计

python之Cython 中 numpy 数组掩码的性能

2025年02月15日10leader

作为这个问题的跟进 here (感谢 MSeifert 的帮助)我想出了一个问题,我必须在传递屏蔽数组以更新 之前用索引数组 new_vals_idx 屏蔽一个 numpy 数组 new_values >val_dict.

对于旧帖子中 MSeifert 的回答中提出的解决方案,我尝试应用数组掩码,但性能并不令人满意。
我在以下示例中使用的数组和字典是:

import numpy as np 
val_dict = {'a': 5.0, 'b': 18.8, 'c': -55/2} 
for i in range(200): 
    val_dict[str(i)] = i 
    val_dict[i] = i**2 
 
keys = ('b', 123, '89', 'c')  # dict keys to update 
new_values = np.arange(1, 51, 1) / 1.0  # array with new values which has to be masked 
new_vals_idx = np.array((0, 3, 5, -1))  # masking array 
valarr = np.zeros((new_vals_idx.shape[0]))  # preallocation for masked array 
length = new_vals_idx.shape[0] 

为了让我的代码片段更容易与我的旧问题进行比较,我将坚持使用 MSeifert 答案的函数命名。这些是我试图从 python/cython 中获得最佳性能的尝试(其他答案由于性能太差而被遗漏):

def old_for(val_dict, keys, new_values, new_vals_idx, length): 
    for i in range(length): 
        val_dict[keys[i]] = new_values[new_vals_idx[i]] 
%timeit old_for(val_dict, keys, new_values, new_vals_idx, length) 
# 1000000 loops, best of 3: 1.6 µs per loop 
 
def old_for_w_valarr(val_dict, keys, new_values, valarr, new_vals_idx, length): 
    valarr = new_values[new_vals_idx] 
    for i in range(length): 
        val_dict[keys[i]] = valarr[i] 
%timeit old_for_w_valarr(val_dict, keys, new_values, valarr, new_vals_idx, length) 
# 100000 loops, best of 3: 2.33 µs per loop 
 
def new2_w_valarr(val_dict, keys, new_values, valarr, new_vals_idx, length): 
    valarr = new_values[new_vals_idx].tolist() 
    for key, val in zip(keys, valarr): 
        val_dict[key] = val 
%timeit new2_w_valarr(val_dict, keys, new_values, valarr, new_vals_idx, length) 
# 100000 loops, best of 3: 2.01 µs per loop 

Cython 函数:

%load_ext cython 
%%cython 
import numpy as np 
cimport numpy as np 
cpdef new3_cy(dict val_dict, tuple keys, double[:] new_values, int[:] new_vals_idx, Py_ssize_t length): 
    cdef Py_ssize_t i 
    cdef double val  # this gives about 10 µs speed boost compared to directly assigning it to val_dict 
    for i in range(length): 
        val = new_values[new_vals_idx[i]] 
        val_dict[keys[i]] = val 
%timeit new3_cy(val_dict, keys, new_values, new_vals_idx, length) 
# 1000000 loops, best of 3: 1.38 µs per loop 
 
cpdef new3_cy_mview(dict val_dict, tuple keys, double[:] new_values, int[:] new_vals_idx, Py_ssize_t length): 
    cdef Py_ssize_t i 
    cdef int[:] mview_idx = new_vals_idx 
    cdef double [:] mview_vals = new_values 
    for i in range(length): 
        val_dict[keys[i]] = mview_vals[mview_idx[i]] 
%timeit new3_cy_mview(val_dict, keys, new_values, new_vals_idx, length) 
# 1000000 loops, best of 3: 1.38 µs per loop 
 
# NOT WORKING: 
cpdef new2_cy_mview(dict val_dict, tuple keys, double[:] new_values, int[:] new_vals_idx, Py_ssize_t length): 
    cdef double [new_vals_idx] masked_vals = new_values 
    for key, val in zip(keys, masked_vals.tolist()): 
        val_dict[key] = val 
 
cpdef new2_cy_mask(dict val_dict, tuple keys, double[:] new_values, valarr, int[:] new_vals_idx, Py_ssize_t length): 
    valarr = new_values[new_vals_idx] 
    for key, val in zip(keys, valarr.tolist()): 
        val_dict[key] = val 

Cython 函数 new3_cy 和 new3_cy_mview 似乎并不比 old_for 快很多。传递 valarr 以避免函数内部的数组构造(因为它将被调用数百万次)甚至似乎会减慢它的速度。
使用 Cython 中的 new_vals_idx 数组在 new2_cy_mask 中进行屏蔽会给我错误:“指定的内存 View 索引无效,类型为 int[:]”。是否有类似 Py_ssize_t 的索引数组类型?
尝试在 new2_cy_mview 中创建屏蔽内存 View 时出现错误“无法将类型‘double[:]’分配给‘double [__pyx_v_new_vals_idx]’”。甚至有像屏蔽内存 View 这样的东西吗?我找不到有关此主题的信息...

将计时结果与我的旧问题的计时结果进行比较,我猜阵列屏蔽是占用大部分时间的过程。而且由于它很可能已经在 numpy 中进行了高度优化,因此可能没什么可做的。但减速是如此之大,必须(希望)有更好的方法来做到这一点。
任何帮助表示赞赏!提前致谢!

请您参考如下方法:

在当前构造中,您可以做的一件事是关闭边界检查(如果这样做是安全的!)。不会产生巨大差异,但会带来一些增量性能。

%%cython 
import numpy as np 
cimport numpy as np 
cimport cython 
 
@cython.boundscheck(False) 
@cython.wraparound(False) 
cpdef new4_cy(dict val_dict, tuple keys, double[:] new_values, int[:] new_vals_idx, Py_ssize_t length): 
    cdef Py_ssize_t i 
    cdef double val  # this gives about 10 µs speed boost compared to directly assigning it to val_dict 
    for i in range(length): 
        val = new_values[new_vals_idx[i]] 
        val_dict[keys[i]] = val 
 
In [36]: %timeit new3_cy(val_dict, keys, new_values, new_vals_idx, length) 
1.76 µs ± 209 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) 
 
In [37]: %timeit new4_cy(val_dict, keys, new_values, new_vals_idx, length) 
1.45 µs ± 31.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)