Skip to main content
 首页 » 编程设计

python之如何在多索引数据框中找到异常值帧

2024年12月31日12pengyingh

结果应该是一个不包含任何异常值的mi-dataframe。标准是标准偏差:np.abs(x-g_mean) <= 3*g_std

我尝试识别统计异常值:

import pandas as pd 
import numpy as np 
 
#create sample 
arrays = [[1,1,1,2,2,2,3,3], 
          [0,1,2,0,1,2,0,1]] 
tuples = list(zip(*arrays)) 
index = pd.MultiIndex.from_tuples(tuples, names=['ID', 'INDEX']) 
df = pd.DataFrame(np.abs(np.random.randn(8, 2)), index=index, columns=['Ts','Tf']) 
 
#groupby index and learn from data 
g = df.groupby(level='INDEX') 
g_mean=g.mean() 
g_std = g.std() 
 
#groupby ID and look if some ID is an outlier 
g = df.groupby(level='ID') 
test = g.apply(lambda x: True if np.abs(x-g_mean) <= 3*g_std else False) 

我的代码中的最后一行不起作用,因为在最后一组中我比较了两种不同形式的数据帧。有什么建议吗?

请您参考如下方法:

您可以使用:

g_mean= df.mean(level='INDEX') 
g_std = df.std(level='INDEX') 
 
def f(x): 
    #remove first level per group 
    x = x.reset_index(level=0, drop=True) 
    #detect outliers and check if all values are Trues    
    m = (np.abs(x-g_mean) <= 3*g_std).values.all() 
    return m 
 
#groupby ID and look if some ID is an outlier 
s = df.groupby(level='ID').apply(f) 
print (s) 
ID 
1     True 
2     True 
3    False 
dtype: bool 
 
#map second level by boolean Series and filter by boolean indexing 
df = df[df.index.get_level_values('ID').to_series().map(s).values] 
#if necessary, remove unnecessary levels in MultiIndex 
df.index = df.index.remove_unused_levels() 
print (df) 
                Ts        Tf 
ID INDEX                     
1  0      0.612077  0.876833 
   1      0.911303  0.377008 
   2      0.326670  0.289647 
2  0      0.525381  0.599262 
   1      1.336077  1.177081 
   2      1.322341  0.572035