作为这个问题的跟进 here (感谢 MSeifert 的帮助)我想出了一个问题,我必须在传递屏蔽数组以更新 之前用索引数组 .new_vals_idx 屏蔽一个 numpy 数组 new_values >val_dict
对于旧帖子中 MSeifert 的回答中提出的解决方案,我尝试应用数组掩码,但性能并不令人满意。
我在以下示例中使用的数组和字典是:
import numpy as np
val_dict = {'a': 5.0, 'b': 18.8, 'c': -55/2}
for i in range(200):
val_dict[str(i)] = i
val_dict[i] = i**2
keys = ('b', 123, '89', 'c') # dict keys to update
new_values = np.arange(1, 51, 1) / 1.0 # array with new values which has to be masked
new_vals_idx = np.array((0, 3, 5, -1)) # masking array
valarr = np.zeros((new_vals_idx.shape[0])) # preallocation for masked array
length = new_vals_idx.shape[0]
为了让我的代码片段更容易与我的旧问题进行比较,我将坚持使用 MSeifert 答案的函数命名。这些是我试图从 python/cython 中获得最佳性能的尝试(其他答案由于性能太差而被遗漏):
def old_for(val_dict, keys, new_values, new_vals_idx, length):
for i in range(length):
val_dict[keys[i]] = new_values[new_vals_idx[i]]
%timeit old_for(val_dict, keys, new_values, new_vals_idx, length)
# 1000000 loops, best of 3: 1.6 µs per loop
def old_for_w_valarr(val_dict, keys, new_values, valarr, new_vals_idx, length):
valarr = new_values[new_vals_idx]
for i in range(length):
val_dict[keys[i]] = valarr[i]
%timeit old_for_w_valarr(val_dict, keys, new_values, valarr, new_vals_idx, length)
# 100000 loops, best of 3: 2.33 µs per loop
def new2_w_valarr(val_dict, keys, new_values, valarr, new_vals_idx, length):
valarr = new_values[new_vals_idx].tolist()
for key, val in zip(keys, valarr):
val_dict[key] = val
%timeit new2_w_valarr(val_dict, keys, new_values, valarr, new_vals_idx, length)
# 100000 loops, best of 3: 2.01 µs per loop
Cython 函数:
%load_ext cython
%%cython
import numpy as np
cimport numpy as np
cpdef new3_cy(dict val_dict, tuple keys, double[:] new_values, int[:] new_vals_idx, Py_ssize_t length):
cdef Py_ssize_t i
cdef double val # this gives about 10 µs speed boost compared to directly assigning it to val_dict
for i in range(length):
val = new_values[new_vals_idx[i]]
val_dict[keys[i]] = val
%timeit new3_cy(val_dict, keys, new_values, new_vals_idx, length)
# 1000000 loops, best of 3: 1.38 µs per loop
cpdef new3_cy_mview(dict val_dict, tuple keys, double[:] new_values, int[:] new_vals_idx, Py_ssize_t length):
cdef Py_ssize_t i
cdef int[:] mview_idx = new_vals_idx
cdef double [:] mview_vals = new_values
for i in range(length):
val_dict[keys[i]] = mview_vals[mview_idx[i]]
%timeit new3_cy_mview(val_dict, keys, new_values, new_vals_idx, length)
# 1000000 loops, best of 3: 1.38 µs per loop
# NOT WORKING:
cpdef new2_cy_mview(dict val_dict, tuple keys, double[:] new_values, int[:] new_vals_idx, Py_ssize_t length):
cdef double [new_vals_idx] masked_vals = new_values
for key, val in zip(keys, masked_vals.tolist()):
val_dict[key] = val
cpdef new2_cy_mask(dict val_dict, tuple keys, double[:] new_values, valarr, int[:] new_vals_idx, Py_ssize_t length):
valarr = new_values[new_vals_idx]
for key, val in zip(keys, valarr.tolist()):
val_dict[key] = val
Cython 函数 new3_cy 和 new3_cy_mview 似乎并不比 old_for 快很多。传递 valarr 以避免函数内部的数组构造(因为它将被调用数百万次)甚至似乎会减慢它的速度。
使用 Cython 中的 new_vals_idx 数组在 new2_cy_mask 中进行屏蔽会给我错误:“指定的内存 View 索引无效,类型为 int[:]”。是否有类似 Py_ssize_t 的索引数组类型?
尝试在 new2_cy_mview 中创建屏蔽内存 View 时出现错误“无法将类型‘double[:]’分配给‘double [__pyx_v_new_vals_idx]’”。甚至有像屏蔽内存 View 这样的东西吗?我找不到有关此主题的信息...
将计时结果与我的旧问题的计时结果进行比较,我猜阵列屏蔽是占用大部分时间的过程。而且由于它很可能已经在 numpy 中进行了高度优化,因此可能没什么可做的。但减速是如此之大,必须(希望)有更好的方法来做到这一点。
任何帮助表示赞赏!提前致谢!
请您参考如下方法:
在当前构造中,您可以做的一件事是关闭边界检查(如果这样做是安全的!)。不会产生巨大差异,但会带来一些增量性能。
%%cython
import numpy as np
cimport numpy as np
cimport cython
@cython.boundscheck(False)
@cython.wraparound(False)
cpdef new4_cy(dict val_dict, tuple keys, double[:] new_values, int[:] new_vals_idx, Py_ssize_t length):
cdef Py_ssize_t i
cdef double val # this gives about 10 µs speed boost compared to directly assigning it to val_dict
for i in range(length):
val = new_values[new_vals_idx[i]]
val_dict[keys[i]] = val
In [36]: %timeit new3_cy(val_dict, keys, new_values, new_vals_idx, length)
1.76 µs ± 209 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
In [37]: %timeit new4_cy(val_dict, keys, new_values, new_vals_idx, length)
1.45 µs ± 31.5 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
