Skip to main content
 首页 » 编程设计

python之无法重现 Xgb.cv 交叉验证结果

2025年02月15日7lhb25

我正在使用 Python 3.5 和 XGBoost 的 python 实现,版本 0.6

我在 Python 中构建了一个前向特征选择例程,它迭代地构建最佳特征集(导致最好的分数,这里的度量是二元分类错误)。

在我的数据集上,使用 xgb.cv 例程,通过将 max_depth(树的)增加到 40...,我可以将错误率降低到大约 0.21...

但是,如果我进行自定义交叉验证,使用相同的 XG Boost 参数、相同的折叠、相同的指标和相同的数据集,我达到的最佳分数为 0.70,max_depth 为 4 ... 如果我使用最佳通过我的 xgb.cv 例程获得的 max_depth,我的分数下降到 0.65 ...我只是不明白发生了什么...

我最好的猜测是 xgb.cv 使用不同的折叠(即在分区之前打乱数据),但我也认为我将折叠作为输入提交给 xgb.cv(使用选项 Shuffle=False)...所以,它可能是完全不同的东西......

这是 forward_feature_selection 的代码(使用 xgb.cv):

def Forward_Feature_Selection(train, y_train, params, num_round=30, threshold=0, initial_score=0.5, to_exclude = [], nfold = 5): 
 
    k_fold = KFold(n_splits=13) 
    selected_features = [] 
    gain = threshold + 1 
    previous_best_score = initial_score 
    train = train.drop(train.columns[to_exclude], axis=1)  # df.columns is zero-based pd.Index  
    features = train.columns.values 
    selected = np.zeros(len(features)) 
    scores = np.zeros(len(features)) 
    while (gain > threshold):    # we start a add-a-feature loop 
        for i in range(0,len(features)): 
            if (selected[i]==0):   # take only features not yet selected 
                selected_features.append(features[i]) 
                new_train = train.iloc[:][selected_features] 
                selected_features.remove(features[i]) 
                dtrain = xgb.DMatrix(new_train, y_train, missing = None) 
            #    dtrain = xgb.DMatrix(pd.DataFrame(new_train), y_train, missing = None) 
                if (i % 10 == 0): 
                    print("Launching XGBoost for feature "+ str(i)) 
                xgb_cv = xgb.cv(params, dtrain, num_round, nfold=13, folds=k_fold, shuffle=False)  
                if params['objective'] == 'binary:logistic': 
                    scores[i] = xgb_cv.tail(1)["test-error-mean"]   #classification 
                else: 
                    scores[i] = xgb_cv.tail(1)["test-rmse-mean"]    #regression 
            else: 
                scores[i] = initial_score    # discard already selected variables from candidates 
        best = np.argmin(scores) 
        gain = previous_best_score - scores[best] 
        if (gain > 0):         
            previous_best_score = scores[best]   
            selected_features.append(features[best]) 
            selected[best] = 1 
 
        print("Adding feature: " + features[best] + " increases score by " + str(gain) + ". Final score is now: " + str(previous_best_score))  
    return (selected_features, previous_best_score) 

这是我的“自定义”交叉验证:

mean_error_rate = 0 
for train, test in k_fold.split(ds): 
    dtrain =  xgb.DMatrix(pd.DataFrame(ds.iloc[train]), dc.iloc[train]["bin_spread"], missing = None) 
    gbm = xgb.train(params, dtrain, 30) 
    dtest =  xgb.DMatrix(pd.DataFrame(ds.iloc[test]), dc.iloc[test]["bin_spread"], missing = None) 
    res.ix[test,"pred"] = gbm.predict(dtest) 
 
    cv_reg = reg.fit(pd.DataFrame(ds.iloc[train]), dc.iloc[train]["bin_spread"]) 
    res.ix[test,"lasso"] = cv_reg.predict(pd.DataFrame(ds.iloc[test])) 
 
    res.ix[test,"y_xgb"] = res.loc[test,"pred"] > 0.5 
    res.ix[test, "xgb_right"] = (res.loc[test,"y_xgb"]==res.loc[test,"bin_spread"])  
    print (str(100*np.sum(res.loc[test, "xgb_right"])/(N/13))) 
    mean_error_rate += 100*(np.sum(res.loc[test, "xgb_right"])/(N/13)) 
print("mean_error_rate is : " + str(mean_error_rate/13))   

使用以下参数:

params = {"objective": "binary:logistic",  
          "booster":"gbtree", 
          "max_depth":4,  
          "eval_metric" : "error", 
          "eta" : 0.15} 
res = pd.DataFrame(dc["bin_spread"])  
k_fold = KFold(n_splits=13) 
N = dc.shape[0] 
num_trees = 30 

最后调用我的前向特征选择:

selfeat = Forward_Feature_Selection(dc,  
                                    dc["bin_spread"],  
                                    params,  
                                    num_round = num_trees, 
                                    threshold = 0, 
                                    initial_score=999, 
                                    to_exclude = [0,1,5,30,31], 
                                    nfold = 13) 

任何有助于理解正在发生的事情的帮助将不胜感激!提前感谢任何提示!

请您参考如下方法:

这是正常的。我也有过同样的经历。首先,Kfold 每次 split 的方式都不一样。您已在 XGBoost 中指定折叠,但 KFold split 不一致,这是正常的。 接下来,模型的初始状态每次都不同。 XGBoost 的内部随机状态也可能导致这种情况,请尝试更改评估指标以查看方差是否减少。如果特定指标适合您的需求,请尝试对最佳参数取平均值并将其用作您的最佳参数。