彩笔运维勇闯机器学习--GBDT
<h2 id="前言">前言</h2><p>本文讨论的GBDT算法，也是基于决策树</p>
<h2 id="开始探索">开始探索</h2>
<h4 id="scikit-learn">scikit-learn</h4>
<p>老规矩，先上代码，看看GBDT的用法</p>
<pre><code>from sklearn.datasets import load_iris
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report
X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=0)
gbdt_model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, max_depth=3, random_state=0)
gbdt_model.fit(X_train, y_train)
gbdt_pred = gbdt_model.predict(X_test)
print("\nClassification Report (GBDT):")
print(classification_report(y_test, gbdt_pred))
</code></pre>
<p>脚本！启动：</p>
<p><img alt="watermarked-gbdt_1_1" loading="lazy" src="https://img2024.cnblogs.com/blog/1416773/202509/1416773-20250926100005713-2025495904.png" class="lazyload"></p>
<h2 id="深入理解gbdt">深入理解GBDT</h2>
<p>GBDT全称，梯度提升决策树（Gradient Boosting Decision Tree），通过“不断拟合上一步误差”的方式来迭代构建模型，每一步都用一棵新的决策树来逼近当前的残差，从而不断提升整体模型性能</p>
<p>不断的迭代决策树，通过损失函数的负梯度作为每一轮迭代的优化目标，每一棵决策树纠正前一棵树的误差，最终将所有树的预测结果加权求和得到最终输出</p>
<h4 id="基本思路">基本思路</h4>
<p>1）首先计算出初始概率：</p>
<p>预测值计算公式（logit公式）： $$F=log(\frac{p}{1-p})$$<br>
概率计算公式（sigmoid公式）： $$P=\frac{1}{1+e^{-F}}$$</p>
<p>2）计算残差，通过一棵回归树拟合该残差</p>
<p></p><div class="math display">\[\begin{aligned}
T_t(x)=
\left\{
\begin{array}{ll}
a \qquad ,x ∈ 正类 \\
b \qquad ,x ∈ 负类
\end{array}
\right.
\end{aligned}
\]</div><p></p><p>3）再次计算概率</p>
<p>该轮预测值： $$F_t(x)=F_{t-1}(x)+η·T_t(x)$$</p>
<ul>
<li><span class="math inline">\(F_t(x)\)</span>：当前轮次对x的预测值</li>
<li><span class="math inline">\(F_{t-1}\)</span>：上一轮次对x的预测值</li>
<li><span class="math inline">\(η\)</span>：学习率，每轮更新的步长</li>
<li><span class="math inline">\(T_t(x)\)</span>：每轮训练的回归树，用来拟合上一轮训练的残差</li>
</ul>
<p>计算出概率： $$P=\frac{1}{1+e^{-F}}$$</p>
<p>4）计算残差、拟合残差、求树的梯度。循环往复，直至收敛</p>
<h4 id="举例说明">举例说明</h4>
<p>下面以二分类任务为例，有10个样本，6个正类（1），4个负类（0）</p>
<p>1）计算初始概率</p>
<p>初始预测值：所有样本的初始预测值为正类的对数几率，通过logit函数计算：$$F = log( \frac{p}{1-p}) = log(\frac{0.6}{0.4}) \approx 0.4055 $$</p>
<p>带入到sigmoid计算预测概率：</p>
<p></p><div class="math display">\[P=\frac{1}{1+e^{-F}}=\frac{1}{1+e^{-0.4055}} \approx 0.6
\]</div><p></p><p>2）计算残差，所谓残差，是单个样本点的预测值与真实值之间的差</p>
<p>正类样本（1）的残差：<span class="math inline">\(r_i=1-0.6=0.4\)</span><br>
负类样本（0）的残差：<span class="math inline">\(r_i=0-0.6=-0.6\)</span></p>
<p>3）训练第一棵决策回归树</p>
<p></p><div class="math display">\[\begin{aligned}
T_t(x)=
\left\{
\begin{array}{ll}
\frac{6·0.4}{6} = 0.4 \qquad ,x ∈ 正类 \\
\frac{4·(-0.6)}{4} = -0.6 \qquad ,x ∈ 负类
\end{array}
\right.
\end{aligned}
\]</div><p></p><p>4）再次计算概率</p>
<p>假设学习率<span class="math inline">\(\eta=0.1\)</span>，更新模型预测：</p>
<ul>
<li>正类预测：<span class="math inline">\(F_{正类}=0.4055+0.1×0.4 = 0.4455\)</span></li>
<li>负类预测：<span class="math inline">\(F_{负类}=0.4055+0.1×(−0.6) = 0.3455\)</span></li>
</ul>
<p>计算预测概率</p>
<p>更新预测概率<span class="math inline">\(P\)</span></p>
<ul>
<li>正类概率：<span class="math inline">\(P_{正类}=\frac{1}{1+e^{-0.4455}} \approx 0.6095\)</span></li>
<li>负类概率：<span class="math inline">\(P_{负类}=\frac{1}{1+e^{-0.3455}} \approx 0.5854\)</span></li>
</ul>
<p>由于设置了迭代次数：<code>n_estimators=5</code>，继续迭代</p>
<p>4）计算二轮残差</p>
<ul>
<li>正类残差：<span class="math inline">\(r_{正类}=1-0.6095=0.3905\)</span></li>
<li>负类残差：<span class="math inline">\(r_{负类}=0-0.5854=−0.5854\)</span></li>
</ul>
<p>5）继续训练第二棵决策回归树</p>
<p></p><div class="math display">\[\begin{aligned}
T_t(x)=
\left\{
\begin{array}{ll}
\frac{6·0.3905}{6} = 0.3905 \qquad ,x ∈ 正类 \\
\frac{4·(-0.5854)}{4} = -0.5854 \qquad ,x ∈ 负类
\end{array}
\right.
\end{aligned}
\]</div><p></p><ul>
<li>模型预测<span class="math inline">\(F\)</span>
<ul>
<li>正类预测：<span class="math inline">\(F_{正类}=0.4455+0.1×0.3905 \approx 0.4846\)</span></li>
<li>负类预测：<span class="math inline">\(F_{负类}=0.3455+0.1×(-0.5854) \approx 0.287\)</span></li>
</ul>
</li>
<li>预测概率<span class="math inline">\(P\)</span>
<ul>
<li>正类概率：<span class="math inline">\(P_{正类} \approx 0.6188\)</span></li>
<li>负类概率：<span class="math inline">\(P_{负类} \approx 0.5713\)</span></li>
</ul>
</li>
</ul>
<p>6）不断迭代，直至<code>n_estimators=5</code></p>
<p>默认使用最后一次的参数，有位彦祖就说，这不合理啊，有可能中途第三次训练的参数拟合度更高，为什么不用第三次的参数呢，这个问题可以使用早停法解决，后面会说</p>
<h4 id="回归树">回归树</h4>
<p>有位彦祖问了，我明明是做分类问题，为什么要用回归树来拟合残差？“回归”这个词到底指的是什么？</p>
<p>我们用的决策树，是回归决策树，是为了拟合残差。之前讨论过决策树，但是是分类决策树，这里简单描述一下回归决策树</p>
<p>直接上一个例子，更加明白。比如现在有一组样本</p>
<table>
<thead>
<tr>
<th>x</th>
<th>y</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>5</td>
</tr>
<tr>
<td>2</td>
<td>6</td>
</tr>
<tr>
<td>3</td>
<td>7</td>
</tr>
<tr>
<td>4</td>
<td>15</td>
</tr>
<tr>
<td>5</td>
<td>16</td>
</tr>
<tr>
<td>6</td>
<td>17</td>
</tr>
</tbody>
</table>
<p>为了简单说明，假设我们的树深度只有1</p>
<p>1）选择切分点，两个样本中间</p>
<p>x = => </p>
<p>假设选择3.5的切分点</p>
<p>2）计算损失函数MSE</p>
<ul>
<li>
<p>左子集：x = ，y = </p>
<ul>
<li>均值：<span class="math inline">\(\frac{5+6+7}{3} = 6\)</span></li>
<li>MSE = <span class="math inline">\(\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 = \frac{(5-6)^2+(6-6)^2+(7-6)^2}{3} \approx 0.6667\)</span></li>
</ul>
</li>
<li>
<p>右子集：x = ，y = </p>
<ul>
<li>均值：<span class="math inline">\(\frac{15+16+17}{3} = 16\)</span></li>
<li>MSE = <span class="math inline">\(\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 = \frac{(15-16)^2+(16-16)^2+(17-16)^2}{3} \approx 0.6667\)</span></li>
</ul>
</li>
<li>
<p>加权总MSE：<span class="math inline">\(\frac{3}{6}·0.6667 + \frac{3}{6}·0.6667 = 0.6667\)</span></p>
</li>
</ul>
<p>也可以选择其他的分割点，然后计算MSE，选择最优分割点</p>
<p>3）拟合树</p>
<p></p><div class="math display">\[\begin{aligned}
T_t(x)=
\left\{
\begin{array}{ll}
\frac{5+6+7}{3} = 6\qquad ,x > 3.5 \\
\frac{15+16+17}{3} = 16 \qquad ,x <= 3.5
\end{array}
\right.
\end{aligned}
\]</div><p></p><h4 id="残差与梯度">残差与梯度</h4>
<p>根据二分类的交叉熵公式以及求概率公式</p>
<p></p><div class="math display">\[\begin{cases}
L(y, \hat{y})=-(y \log(\hat{y}) + (1 - y) \log(1 - \hat{y})) \\
\hat{y}=\frac{1}{1+e^{-F}} \\
\end{cases}
\]</div><p></p><p>对特征<span class="math inline">\(x\)</span>求导，通过剥洋葱方法：</p>
<p></p><div class="math display">\[\frac{\partial L}{\partial F}=\frac{\partial L}{\partial \hat{y}}·\frac{\partial \hat{y}}{\partial F}=-(\frac{y}{\hat{y}}-\frac{1-y}{1-\hat{y}})·\frac{e^{-F}}{(1+e^{-F})^2}
\]</div><p></p><p>由于<span class="math inline">\(\hat{y}=\frac{1}{1+e^{-F}}\)</span>，那么<span class="math inline">\(1-\hat{y}=\frac{e^{-F}}{1+e^{-F}}\)</span>，带入上面：</p>
<p></p><div class="math display">\[\frac{\partial L}{\partial F}=-(\frac{y}{\hat{y}}-\frac{1-y}{1-\hat{y}})· \hat{y}(1-\hat{y})=\hat{y}-y
\]</div><p></p><p>梯度有了，那负梯度自然也计算出来了</p>
<p></p><div class="math display">\[y-\hat{y}
\]</div><p></p><p>通过计算，负梯度=残差，所以，在上面演示的GBDT二分类任务中，就是沿着负梯度（残差）方向不断的寻找</p>
<h2 id="早停法">早停法</h2>
<p>顾名思义，就是发现模型性能下降的时候，立刻停止训练，并且保留性能最高的那组参数</p>
<pre><code>import lightgbm as lgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
X, y = make_classification(n_samples=1000, n_features=20, random_state=0)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=0)
train_data = lgb.Dataset(X_train, label=y_train)
val_data = lgb.Dataset(X_val, label=y_val, reference=train_data)
params = {
'objective': 'binary',
'metric': 'binary_logloss',
'learning_rate': 0.1,
'max_depth': 3,
'num_leaves': 7,
'verbose': -1
}
model = lgb.train(
params,
train_data,
num_boost_round=1000,
valid_sets=,
callbacks=[
lgb.early_stopping(stopping_rounds=5),
lgb.log_evaluation(period=10)
]
)
y_pred = model.predict(X_val, num_iteration=model.best_iteration)
y_pred_binary =
print(f"Validation Accuracy: {accuracy_score(y_val, y_pred_binary):.4f}")
print(f"Best iteration: {model.best_iteration}")
</code></pre>
<p>脚本，启动！</p>
<p><img alt="watermarked-gbdt_1_2" loading="lazy" src="https://img2024.cnblogs.com/blog/1416773/202509/1416773-20250926100030513-1255036438.png" class="lazyload"></p>
<ul>
<li><code>objective</code>：binary表示分类任务的损失函数</li>
<li><code>metric</code>：binary_logloss表示监控指标为对数损失函数，也可以换成binary_error表示误差率</li>
<li><code>num_boost_round=1000</code>：最大迭代次数</li>
<li><code>stopping_rounds=5</code>：连续5轮不提升则停止</li>
<li><code>period=10</code>：每10轮打印日志</li>
</ul>
<h2 id="小结">小结</h2>
<p>给一组系统性能数据，能不能发现潜在的风险，风险判定为，日志错误率为>100条</p>
<p>如果错误日志为80条，虽然没有告警，但是系统已经在风险边缘了</p>
<p>先给一组数据，通过系统性能指标进行分类，然后再对于各时段的性能数据进行分类，看系统是否处于崩溃边缘</p>
<h2 id="联系我">联系我</h2>
<ul>
<li>联系我，做深入的交流<br>
<img alt="" width="500" height="200" loading="lazy" src="https://img2024.cnblogs.com/blog/1416773/202411/1416773-20241121135740959-1907948957.png#" class="lazyload"></li>
</ul>
<hr>
<p>至此，本文结束<br>
在下才疏学浅，有撒汤漏水的，请各位不吝赐教...</p>
</div>
<div id="MySignature" role="contentinfo">
<p>本文来自博客园，作者：it排球君，转载请注明原文链接：https://www.cnblogs.com/MrVolleyball/p/19112957</p>
<div>本文版权归作者和博客园共有，欢迎转载，但未经作者同意必须在文章页面给出原文连接，否则保留追究法律责任的权利。 </div><br><br>
来源：https://www.cnblogs.com/MrVolleyball/p/19112957
頁:
[1]