Skip to main content
 首页 » 编程设计

python之groupby() 对象上的 pandas apply() 运行次数比组数多

2024年11月24日35zdz8207

我继承了一些我正在尝试优化的 pandas 代码。已创建一个 DataFrame,results

results = pd.DataFrame(columns=['plan','volume','avg_denial_increase','std_dev_impact', 'avg_idr_increase', 'std_dev_idr_increase']) 
for plan in my_df['plan_name'].unique(): 
    df1 = df[df['plan_name'] == plan]] 
    df1['volume'].fillna(0, inplace=True) 
    df1['change'] = df1['idr'] - df1['idr'].shift(1) 
    df1['change'].fillna(0, inplace=True) 
    df1['impact'] = df1['change'] * df1['volume'] 
    describe_impact = df1['impact'].describe() 
    describe_change = df1['change'].describe() 
    results = results.append({'plan': plan, 
                              'volume': df1['volume'].mean(), 
                              'avg_denial_increase': describe_impact['mean'], 
                              'std_dev_impact': describe_impact['std'], 
                              'avg_idr_increase': describe_change['mean'], 
                              'std_dev_idr_increase': describe_change['std']},  
                             ignore_index=True) 

我的第一个想法是将 for 循环下的所有内容移动到一个单独的函数 get_results_for_plan 中,并使用 pandas groupby()和 apply() 方法。但事实证明他的速度更慢。运行

%lprun -f get_results_for_plan my_df.groupby('plan_name', sort=False, as_index=False).apply(get_results_for_plan) 

返回

Timer unit: 1e-06 s 
 
Total time: 0.77167 s 
File: <ipython-input-46-7c36b3902812> 
Function: get_results_for_plan at line 1 
 
Line #      Hits         Time  Per Hit   % Time  Line Contents 
============================================================== 
     1                                           def get_results_for_plan(plan_df): 
     2        94      33221.0    353.4      4.3      plan = plan_df.iloc[0]['plan_name'] 
     3        94      25901.0    275.5      3.4      plan_df['volume'].fillna(0, inplace=True) 
     4        94      75765.0    806.0      9.8      plan_df['change'] = plan_df['idr'] - plan_df['idr'].shift(1) 
     5        93      38653.0    415.6      5.0      plan_df['change'].fillna(0, inplace=True) 
     6        93      57088.0    613.8      7.4      plan_df['impact'] = plan_df['change'] * plan_df['volume'] 
     7        93     204828.0   2202.5     26.5      describe_impact = plan_df['impact'].describe() 
     8        93     201127.0   2162.7     26.1      describe_change = plan_df['change'].describe() 
     9        93        129.0      1.4      0.0      return pd.DataFrame({'plan': plan, 
    10        93      21703.0    233.4      2.8                           'volume': plan_df['volume'].mean(), 
    11        93       4291.0     46.1      0.6                           'avg_denial_increase': describe_impact['mean'], 
    12        93       1957.0     21.0      0.3                           'std_dev_impact': describe_impact['std'], 
    13        93       2912.0     31.3      0.4                           'avg_idr_increase': describe_change['mean'], 
    14        93       1783.0     19.2      0.2                           'std_dev_idr_increase': describe_change['std']}, 
    15        93     102312.0   1100.1     13.3                         index=[0]) 

我看到的最明显的问题是每行的命中数。组数,由

len(my_df.groupby('plan_name', sort=False, as_index=False).groups) 

是 72。那么为什么这些行每行被点击 94 或 93 次? (这可能与 this 问题有关,但在那种情况下,我希望命中数为 num_groups + 1)

更新:在上面对 groupby() 的 %lprun 调用中,删除 sort=False 将第 2 行的行命中减少到 80-其余为 6 和 79。仍然比我想象的要多,但好一点。

第二个问题:有没有更好的方法来优化这个特定的代码?

请您参考如下方法:

这是我在评论中的意思的粗略草稿:

def append_to_list(): 
    l = [] 
    for _ in range(10000): 
        l.append(np.random.random(4)) 
    return pd.DataFrame(l, columns=list('abcd')) 
 
def append_to_df(): 
    cols = list('abcd') 
    df = pd.DataFrame(columns=cols) 
    for _ in range(10000): 
        df = df.append({k: v for k, v in zip(cols, np.random.random(4))}, 
                       ignore_index=True) 
    return df 
 
%timeit append_to_list 
# 31.5 ms ± 925 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) 
 
%timeit append_to_df 
# 9.05 s ± 337 ms per loop (mean ± std. dev. of 7 runs, 1 loop each) 

因此,您的代码的最大好处可能是:

results = [] 
for plan in my_df['plan_name'].unique(): 
    df1 = df[df['plan_name'] == plan]] 
    df1['volume'].fillna(0, inplace=True) 
    df1['change'] = df1['idr'] - df1['idr'].shift(1) 
    df1['change'].fillna(0, inplace=True) 
    df1['impact'] = df1['change'] * df1['volume'] 
    describe_impact = df1['impact'].describe() 
    describe_change = df1['change'].describe() 
    results.append((plan,  
                    df1['volume'].mean(),  
                    describe_impact['mean'], 
                    describe_impact['std'],  
                    describe_change['mean'],  
                    describe_change['std'])) 
results = pd.DataFrame(results, columns=['plan','volume','avg_denial_increase','std_dev_impact', 'avg_idr_increase', 'std_dev_idr_increase'])