Skip to main content
 首页 » 编程设计

python之多列 Groupby 大小的 Dask 过滤器数据框

2024年11月24日13freeliver54

目标 = 通过 dask 数据框进行多列分组,并过滤​​掉包含少于 3 行的组。

基于这篇文章: Filtering grouped df in Dask

我能够计算每个 groupby 对象的大小,但我不知道如何将它从多列 groupby 映射回我的数据框。我尝试了以下多种变体但无济于事:

a = input_df.groupby(["FeatureID", "region"])["Target"].size() 
s = input_df[["FeatureID", "region"]].map(a) 

它对单列 groupby 非常有效。

解决方案

感谢@jezrael,我能够想出以下解决方案:

a = input_df.groupby(["FeatureID", "region"])["Target"].nunique().to_frame("feature_div") 
input_df = input_df.join(a, on=["FeatureID", "region"]) 
 
# filter out features below diversity threshold 
diversified = input_df[input_df.feature_div >= diversity_threshold] 

请您参考如下方法:

您需要使用to_framejoin:

a = input_df.groupby(["FeatureID", "region"])["Target"].size().to_frame('New') 
input_df = input_df.join(a, on=["FeatureID", "region"]) 

示例:

import pandas as pd 
from dask import dataframe as dd  
 
input_df = pd.DataFrame({ 
         'FeatureID':[4,5,4,5,5,4], 
         'region':list('aaabbb'), 
         'Target':[7,8,9,4,2,3], 
}) 
 
print (input_df) 
   FeatureID region  Target 
0          4      a       7 
1          5      a       8 
2          4      a       9 
3          5      b       4 
4          5      b       2 
5          4      b       3 

sd = dd.from_pandas(input_df, npartitions=3) 
print (sd) 
              FeatureID  region Target 
npartitions=3                          
0                 int64  object  int64 
2                   ...     ...    ... 
4                   ...     ...    ... 
5                   ...     ...    ... 
Dask Name: from_pandas, 3 tasks 
 
a = sd.groupby(["FeatureID", "region"])["Target"].size().to_frame('New') 
out = sd.join(a, on=["FeatureID", "region"]).compute() 
print (out) 
   FeatureID region  Target  New 
0          4      a       7    2 
1          5      a       8    1 
2          4      a       9    2 
3          5      b       4    2 
4          5      b       2    2 
5          4      b       3    1