Skip to main content
 首页 » 编程设计

python之从 GradientBoostingClassifier 中提取决策规则

2024年12月31日7birdshome

我已经回答了以下问题:

how to extract decision rules of GradientBosstingClassifier

How to extract the decision rules from scikit-learn decision-tree?

但是以上两个并没有解决我的目的。以下是我的查询:

我需要使用 gradientboostingclassifer 在 Python 中构建模型并在 SAS 平台中实现该模型。为此,我需要从 gradientboostingclassifer 中提取决策规则。

以下是我到目前为止所尝试的:

在 IRIS 数据上构建模型:

# import the most common dataset 
from sklearn.datasets import load_iris 
from sklearn.ensemble import GradientBoostingClassifier 
from sklearn.tree import export_graphviz 
from sklearn.externals.six import StringIO   
from IPython.display import Image 
 
X, y = load_iris(return_X_y=True) 
# there are 150 observations and 4 features 
print(X.shape) # (150, 4) 
# let's build a small model = 5 trees with depth no more than 2 
model = GradientBoostingClassifier(n_estimators=5, max_depth=3, learning_rate=1.0) 
model.fit(X, y==2) # predict 2nd class vs rest, for simplicity 
# we can access individual trees 
trees = model.estimators_.ravel() 
 
def plot_tree(clf): 
    dot_data = StringIO() 
    export_graphviz(clf, out_file=dot_data, node_ids=True, 
                    filled=True, rounded=True,  
                    special_characters=True) 
    graph = pydotplus.graph_from_dot_data([enter image description here][3]dot_data.getvalue())   
    return Image(graph.create_png()) 
 
# now we can plot the first tree 
plot_tree(trees[0]) 

绘制图表后,我检查了第一棵树的图表源代码,并使用以下代码写入文本文件:

with open("C:\\Users\XXXX\Desktop\Python\input_tree.txt", "w") as wrt: 
    wrt.write(export_graphviz(trees[0], out_file=None, node_ids=True, 
                filled=True, rounded=True,  
                special_characters=True)) 

下面是输出文件:

digraph Tree { 
node [shape=box, style="filled, rounded", color="black", fontname=helvetica] ; 
edge [fontname=helvetica] ; 
0 [label=<node &#35;0<br/>X<SUB>3</SUB> &le; 1.75<br/>friedman_mse = 0.222<br/>samples = 150<br/>value = 0.0>, fillcolor="#e5813955"] ; 
1 [label=<node &#35;1<br/>X<SUB>2</SUB> &le; 4.95<br/>friedman_mse = 0.046<br/>samples = 104<br/>value = -0.285>, fillcolor="#e5813945"] ; 
0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ; 
2 [label=<node &#35;2<br/>X<SUB>3</SUB> &le; 1.65<br/>friedman_mse = 0.01<br/>samples = 98<br/>value = -0.323>, fillcolor="#e5813943"] ; 
1 -> 2 ; 
3 [label=<node &#35;3<br/>friedman_mse = 0.0<br/>samples = 97<br/>value = -1.5>, fillcolor="#e5813900"] ; 
2 -> 3 ; 
4 [label=<node &#35;4<br/>friedman_mse = -0.0<br/>samples = 1<br/>value = 3.0>, fillcolor="#e58139ff"] ; 
2 -> 4 ; 
5 [label=<node &#35;5<br/>X<SUB>3</SUB> &le; 1.55<br/>friedman_mse = 0.222<br/>samples = 6<br/>value = 0.333>, fillcolor="#e5813968"] ; 
1 -> 5 ; 
6 [label=<node &#35;6<br/>friedman_mse = 0.0<br/>samples = 3<br/>value = 3.0>, fillcolor="#e58139ff"] ; 
5 -> 6 ; 
7 [label=<node &#35;7<br/>friedman_mse = 0.222<br/>samples = 3<br/>value = 0.0>, fillcolor="#e5813955"] ; 
5 -> 7 ; 
8 [label=<node &#35;8<br/>X<SUB>2</SUB> &le; 4.85<br/>friedman_mse = 0.021<br/>samples = 46<br/>value = 0.645>, fillcolor="#e581397a"] ; 
0 -> 8 [labeldistance=2.5, labelangle=-45, headlabel="False"] ; 
9 [label=<node &#35;9<br/>X<SUB>1</SUB> &le; 3.1<br/>friedman_mse = 0.222<br/>samples = 3<br/>value = 0.333>, fillcolor="#e5813968"] ; 
8 -> 9 ; 
10 [label=<node &#35;10<br/>friedman_mse = 0.0<br/>samples = 2<br/>value = 3.0>, fillcolor="#e58139ff"] ; 
9 -> 10 ; 
11 [label=<node &#35;11<br/>friedman_mse = -0.0<br/>samples = 1<br/>value = -1.5>, fillcolor="#e5813900"] ; 
9 -> 11 ; 
12 [label=<node &#35;12<br/>friedman_mse = -0.0<br/>samples = 43<br/>value = 3.0>, fillcolor="#e58139ff"] ; 
8 -> 12 ; 
} 

为了从输出文件中提取决策规则,我尝试将以下 python RegEX 代码转换为 SAS 代码:

 import re 
with open("C:\\Users\XXXX\Desktop\Python\input_tree.txt") as f: 
    with open("C:\\Users\XXXX\Desktop\Python\output.txt", "w") as f1: 
        result0 = 'value = 0;' 
        f1.write(result0) 
        for line in f: 
            result1 = re.sub(r'^(\d+)\s+.*<br\/>([A-Z]+)<SUB>(\d+)<\/SUB>\s+(.+?)([-\d.]+)<br\/>friedman_mse.*;$',r"if \2\3 \4 \5 then do;",line) 
            result2 = re.sub(r'^(\d+).*(?!SUB).*(value\s+=)\s([-\d.]+).*;$',r"\2 value + \3; end;",result1) 
            result3 = re.sub(r'^(\d+\s+->\s+\d+\s+);$',r'\1',result2) 
            result4 = re.sub(r'^digraph.+|^node.+|^edge.+','',result3) 
            result5 = re.sub(r'&(\w{2});',r'\1',result4) 
            result6 = re.sub(r'}','end;',result5) 
            f1.write(result6) 

下面是上述代码的输出 SAS:

value = 0; 
if X3 le  1.75 then do; 
if X2 le  4.95 then do; 
0 -> 1 [labeldistance=2.5, labelangle=45, headlabel="True"] ; 
if X3 le  1.65 then do; 
1 -> 2  
value = value + -1.5; end; 
2 -> 3  
value = value + 3.0; end; 
2 -> 4  
if X3 le  1.55 then do; 
1 -> 5  
value = value + 3.0; end; 
5 -> 6  
value = value + 0.0; end; 
5 -> 7  
if X2 le  4.85 then do; 
0 -> 8 [labeldistance=2.5, labelangle=-45, headlabel="False"] ; 
if X1 le  3.1 then do; 
8 -> 9  
value = value + 3.0; end; 
9 -> 10  
value = value + -1.5; end; 
9 -> 11  
value = value + 3.0; end; 
8 -> 12  
end; 

如您所见,输出文件中缺少一 block ,即我无法正确打开/关闭 do-end block 。为此,我需要使用节点编号,但我没有这样做,因为我无法在此处找到任何模式。

你们中的任何人都可以帮我解决这个问题。

除此之外,像decisiontreeclassifier我不能提取上面第二个链接中提到的children_left,children_right,阈值。我已经成功提取了GBM的每一棵树

trees = model.estimators_.ravel() 

但我没有找到任何有用的函数来提取每棵树的值和规则。如果我可以以类似 DecisionTreeclassifier 的方式使用 grapviz 对象,请提供帮助。

或

用任何其他可以解决我的目的的方法帮助我。

请您参考如下方法:

无需使用 graphviz 导出来访问决策树数据。 model.estimators_ 包含模型所包含的所有单个分类器。对于 GradientBoostingClassifier,这是一个具有形状 (n_estimators, n_classes) 的二维 numpy 数组,每个项目都是一个 DecisionTreeRegressor。

每个决策树都有一个属性_tree 和Understanding the decision tree structure展示了如何从该对象中取出节点、阈值和子节点。

 
import numpy 
import pandas 
from sklearn.ensemble import GradientBoostingClassifier 
 
est = GradientBoostingClassifier(n_estimators=4) 
numpy.random.seed(1) 
est.fit(numpy.random.random((100, 3)), numpy.random.choice([0, 1, 2], size=(100,))) 
print('s', est.estimators_.shape) 
 
n_classes, n_estimators = est.estimators_.shape 
for c in range(n_classes): 
    for t in range(n_estimators): 
        dtree = est.estimators_[c, t] 
        print("class={}, tree={}: {}".format(c, t, dtree.tree_)) 
 
        rules = pandas.DataFrame({ 
            'child_left': dtree.tree_.children_left, 
            'child_right': dtree.tree_.children_right, 
            'feature': dtree.tree_.feature, 
            'threshold': dtree.tree_.threshold, 
        }) 
        print(rules) 

为每棵树输出如下内容:

class=0, tree=0: <sklearn.tree._tree.Tree object at 0x7f18a697f370> 
   child_left  child_right  feature  threshold 
0           1            2        0   0.020702 
1          -1           -1       -2  -2.000000 
2           3            6        1   0.879058 
3           4            5        1   0.543716 
4          -1           -1       -2  -2.000000 
5          -1           -1       -2  -2.000000 
6           7            8        0   0.292586 
7          -1           -1       -2  -2.000000 
8          -1           -1       -2  -2.000000