(Python 2.7， Pandas 0.13.0)
背景:我从 CSV 文件中读取了一堆数据并将其加载到 pandas 数据框中。有些数据很复杂(我在加载时将其从字符串转换而来)。一些值是设备错误，以太大为特征。我想用 np.nan 替换所有幅度超过某个阈值的值。使用 numpy 数组很容易(假设您使用如图所示的“复杂 nan”)，但在 Pandas 中一直具有挑战性。我在下面记录了我尝试过的步骤 - 最后一次尝试几乎完成了，但是发生替换的任何行都被转换为真实的。
在这一点上，我想简单地将值拉入一个 numpy 数组，进行修改，然后加载回数据框，但这似乎相当不雅观。
编辑:下面的解决方案有效，但我想知道在我编写的代码中 pandas 处理 NaN. 的方式是否仍然存在错误。看起来创建的 NaN 是 nan +0.j 而不是 nan +nanj。 如果你正在做类似plot(np.real(signal), np.imag(signal))的事情，Matplotlib 将毫无问题地绘制后者，但不喜欢前者，因为它是绘制 (Nan, 0) 对。看起来我需要用 nan +nanj 条目替换新的 nan +0j 条目，这会递归地重新启动问题。 :)
EDIT2:NaN 中确实存在视觉差异，但我发现的新错误与该差异无关。差异可能并不重要。删除线上方的内容不正确。
# begin by making a fake data set that resembles the CSV struction
headers = ['Z1', 'Z2', 'Z3']
temp = np.arange(12).reshape((4,3)) + 1j*np.arange(12,24).reshape((4,3))
temp[0,1] = 5000 + 1j*5000
temp[1,1] = 5000 + 1j*8000
temp[2,2] = 7000 + 1j*3000
junk = ['exists to', 'make life', 'extra', 'difficult']
df_junk = pd.DataFrame(data=junk, columns=['other junk'])
df = pd.DataFrame(data=temp, columns=headers)
df = pd.concat((df, df_junk), axis=1)
# very simple to do this in an np.array if we only take the numbers
temp2 = np.copy(temp)
# temp2 is the desired result, but in the frame with everything else
temp2[ np.abs(temp2) > 5000 ] = np.nan + 1j*np.nan
df2 = df.copy()
# Executing the next line replaces the value with NaN,
# but turns all of column Z2 into real numbers
#df[np.abs(df[headers]) > 5000 ] = np.nan + 1j*np.nan
# Trying to grab the index first gives
# ValueError: Cannot index with multidimensional key
#df.ix[np.abs(df[headers]) > 5000 ]
for column in headers:
# The following line would turn the entire 3rd row into NaN
# df[np.abs(df[column]) > 5000] = np.nan + 1j*np.nan
# Attempts along these lines to apply a lambda (tried different ones)
# didn't seem to work
#csv_data[column] = csv_data[column].apply(lambda x:\
# pd.replace(x, np.nan) if abs(x) > 5000 else pd.replace(x,x))
# This last one almost works, but again turns columns with replacements into reals
df2[column].where(abs(df2[column]) <= 5000, np.nan+1j*np.nan, inplace=True)
Z1 Z2 Z3 other junk
0 12j NaN 2 exists to
1 (3+15j) NaN 5 make life
2 (6+18j) 7 NaN extra
3 (9+21j) 10 11 difficult
请您参考如下方法:
看起来这在没有 inplace 标志的情况下有效:
In [11]: df3 = df2[['Z1', 'Z2', 'Z3']]
In [12]: df3.where(df3 <= 5000) # replaces by NaN by default
Out[12]:
Z1 Z2 Z3
0 12j NaN (2+14j)
1 (3+15j) NaN (5+17j)
2 (6+18j) (7+19j) NaN
3 (9+21j) (10+22j) (11+23j)
In [13]: df2[['Z1', 'Z2', 'Z3']] = df3.where(df3 <= 5000)
通常我认为避免 inplace 标志是个好主意(尽管这可能是 bug ):
In [21]: df3.where(df3 <= 5000, inplace=True)
In [22]: df3
Out[22]:
Z1 Z2 Z3
0 12j NaN 2
1 (3+15j) NaN 5
2 (6+18j) 7 NaN
3 (9+21j) 10 11
