我正在学习 XGBoost。以下是我使用的代码，下面是我构建的 XGBoost 模型中的树 #0 和 #1。
我很难理解叶子值的含义。我发现的一些答案表明数据样本在该叶子上的值是“条件概率”。
但我也在一些叶子上发现了负值。概率怎么可能是负数？
有人可以为叶值提供直观的解释吗？
# prepare dataset
import numpy as np
import pandas as pd
train_set = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data', header = None)
test_set = pd.read_csv('http://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test',
skiprows = 1, header = None) # Make sure to skip a row for the test set
# since the downloaded data has no header, I need to add the headers manually
col_labels = ['age', 'workclass', 'fnlwgt', 'education', 'education_num', 'marital_status', 'occupation',
'relationship', 'race', 'sex', 'capital_gain', 'capital_loss', 'hours_per_week', 'native_country',
'wage_class']
train_set.columns = col_labels
test_set.columns = col_labels
# 1. replace ' ?' with nan
# 2. drop all nan
train_noNan = train_set.replace(' ?', np.nan).dropna()
test_noNan = test_set.replace(' ?', np.nan).dropna()
# replace ' <=50K.' with ' <=50K', and ' >50K.' with ' >50K' in wage_class
test_noNan['wage_class'] = test_noNan.wage_class.replace(
{' <=50K.' : ' <=50K',
' >50K.' : ' >50K'
})
# encode training and test dataset together
combined_set = pd.concat([train_noNan, test_noNan], axis=0)
#
for feature in combined_set.columns:
# cetegorical feature columns will have dtype = object
if combined_set[feature].dtype == 'object':
combined_set[feature] = pd.Categorical(combined_set[feature]).codes # replace string with integer; this simply counts the # of unique values in a column and maps it to an integer
combined_set.head()
# separate train and test
final_train = combined_set[:train_noNan.shape[0]]
final_test = combined_set[train_noNan.shape[0]:]
# separate feature and label
y_train = final_train.pop('wage_class')
y_test = final_test.pop('wage_class')
import xgboost as xgb
from xgboost import plot_tree
from sklearn.model_selection import GridSearchCV
# XGBoost has built-in CV, which can use early-stopping to prevent overfiting, therefore improve accuracy
## if not using sklearn, I can convert the data into DMatrix, a XGBoost specific data structure for training and testing. It is said DMatrix can improve the efficiency of the algorithm
xgdmat = xgb.DMatrix(final_train, y_train)
our_params = {
'eta' : 0.1, # aka. learning_rate
'seed' : 0,
'subsample' : 0.8,
'colsample_bytree': 0.8,
'objective' : 'binary:logistic',
'max_depth' :3, # how many features to use before reach leaf
'min_child_weight':1}
# Grid Search CV optimized settings
# create XGBoost object using the parameters
final_gb = xgb.train(our_params, xgdmat, num_boost_round = 432)
import seaborn as sns
sns.set(font_scale = 1.5)
xgb.plot_importance(final_gb)
# after printing the importance of the features, we need to put human insights and try to explain why each feature is important/not important
# visualize the tree
# import matplotlib.pyplot as plt
# xgb.plot_tree(final_gb, num_trees = 0)
# plt.rcParams['figure.figsize'] = [600, 300] # define the figure size...
# plt.show()
graph_to_save = xgb.to_graphviz(final_gb, num_trees = 0)
graph_to_save.format = 'png'
graph_to_save.render('tree_0_saved') # a tree_saved.png will be saved in the root directory
graph_to_save = xgb.to_graphviz(final_gb, num_trees = 1)
graph_to_save.format = 'png'
graph_to_save.render('tree_1_saved')
下面是倾倒的树#0 和#1。
