我也对这个问题感兴趣,并想添加一些实验以更好地理解 CalibratedClassifierCV (CCCV)。
正如已经说过的,有两种使用它的方法。
#Method 1, train classifier within CCCV
model = CalibratedClassifierCV(my_clf)
model.fit(X_train_val, y_train_val)
#Method 2, train classifier and then use CCCV on DISJOINT set
my_clf.fit(X_train, y_train)
model = CalibratedClassifierCV(my_clf, cv='prefit')
model.fit(X_val, y_val)
或者,我们可以尝试第二种方法,但只需根据我们拟合的相同数据进行校准。
#Method 2 Non disjoint, train classifier on set, then use CCCV on SAME set used for training
my_clf.fit(X_train_val, y_train_val)
model = CalibratedClassifierCV(my_clf, cv='prefit')
model.fit(X_train_val, y_train_val)
尽管文档警告使用不相交集,但这可能很有用,因为它允许您随后进行检查my_clf
(例如,查看coef_
CalibratedClassifierCV 对象不可用的 )。(有谁知道如何从校准的分类器中得到这个——其中一个是三个,所以你会平均系数吗?)。
我决定在完全支持的测试集上比较这三种方法的校准。
这是一个数据集:
X, y = datasets.make_classification(n_samples=500, n_features=200,
n_informative=10, n_redundant=10,
#random_state=42,
n_clusters_per_class=1, weights = [0.8,0.2])
我加入了一些类不平衡,只提供了 500 个样本,使这成为一个难题。
我进行了 100 次试验,每次都尝试每种方法并绘制其校准曲线。
![在此处输入图像描述](https://i.stack.imgur.com/cNbPc.png)
Brier 在所有试验中得分的箱线图:
![在此处输入图像描述](https://i.stack.imgur.com/jN1uH.png)
将样本数量增加到 10,000:
![在此处输入图像描述](https://i.stack.imgur.com/119cw.png)
![在此处输入图像描述](https://i.stack.imgur.com/tabSZ.png)
如果我们将分类器更改为朴素贝叶斯,回到 500 个样本:
![在此处输入图像描述](https://i.stack.imgur.com/7JyOf.png)
![在此处输入图像描述](https://i.stack.imgur.com/vZDv8.png)
这似乎不足以进行校准。将样本增加到 10,000 个
![在此处输入图像描述](https://i.stack.imgur.com/Jtknt.png)
![在此处输入图像描述](https://i.stack.imgur.com/VtzfZ.png)
完整代码
print(__doc__)
# Based on code by Alexandre Gramfort <alexandre.gramfort@telecom-paristech.fr>
# Jan Hendrik Metzen <jhm@informatik.uni-bremen.de>
import matplotlib.pyplot as plt
from sklearn import datasets
from sklearn.naive_bayes import GaussianNB
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import brier_score_loss
from sklearn.calibration import CalibratedClassifierCV, calibration_curve
from sklearn.model_selection import train_test_split
def plot_calibration_curve(clf, name, ax, X_test, y_test, title):
y_pred = clf.predict(X_test)
if hasattr(clf, "predict_proba"):
prob_pos = clf.predict_proba(X_test)[:, 1]
else: # use decision function
prob_pos = clf.decision_function(X_test)
prob_pos = \
(prob_pos - prob_pos.min()) / (prob_pos.max() - prob_pos.min())
clf_score = brier_score_loss(y_test, prob_pos, pos_label=y.max())
fraction_of_positives, mean_predicted_value = \
calibration_curve(y_test, prob_pos, n_bins=10, normalize=False)
ax.plot(mean_predicted_value, fraction_of_positives, "s-",
label="%s (%1.3f)" % (name, clf_score), alpha=0.5, color='k', marker=None)
ax.set_ylabel("Fraction of positives")
ax.set_ylim([-0.05, 1.05])
ax.set_title(title)
ax.set_xlabel("Mean predicted value")
plt.tight_layout()
return clf_score
fig, (ax1, ax2, ax3) = plt.subplots(nrows=3, ncols=1, figsize=(6,12))
ax1.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated",)
ax2.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
ax3.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
scores = {'Method 1':[],'Method 2':[],'Method 3':[]}
fig, (ax1, ax2, ax3) = plt.subplots(nrows=3, ncols=1, figsize=(6,12))
ax1.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated",)
ax2.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
ax3.plot([0, 1], [0, 1], "k:", label="Perfectly calibrated")
scores = {'Method 1':[],'Method 2':[],'Method 3':[]}
for i in range(0,100):
X, y = datasets.make_classification(n_samples=10000, n_features=200,
n_informative=10, n_redundant=10,
#random_state=42,
n_clusters_per_class=1, weights = [0.8,0.2])
X_train_val, X_test, y_train_val, y_test = train_test_split(X, y, test_size=0.80,
#random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(X_train_val, y_train_val, test_size=0.80,
#random_state=42
)
#my_clf = GaussianNB()
my_clf = LogisticRegression()
#Method 1, train classifier within CCCV
model = CalibratedClassifierCV(my_clf)
model.fit(X_train_val, y_train_val)
r = plot_calibration_curve(model, "all_cal", ax1, X_test, y_test, "Method 1")
scores['Method 1'].append(r)
#Method 2, train classifier and then use CCCV on DISJOINT set
my_clf.fit(X_train, y_train)
model = CalibratedClassifierCV(my_clf, cv='prefit')
model.fit(X_val, y_val)
r = plot_calibration_curve(model, "all_cal", ax2, X_test, y_test, "Method 2")
scores['Method 2'].append(r)
#Method 3, train classifier on set, then use CCCV on SAME set used for training
my_clf.fit(X_train_val, y_train_val)
model = CalibratedClassifierCV(my_clf, cv='prefit')
model.fit(X_train_val, y_train_val)
r = plot_calibration_curve(model, "all_cal", ax3, X_test, y_test, "Method 2 non Dis")
scores['Method 3'].append(r)
import pandas
b = pandas.DataFrame(scores).boxplot()
plt.suptitle('Brier score')
因此,Brier 评分结果尚无定论,但根据曲线,似乎最好使用第二种方法。