我有一个 XGBoost 模型试图预测下一个时期（5 分钟）货币是否会上涨或下跌。我有一个从 2004 年到 2018 年的数据集。我将数据随机分成 95% 的训练和 5% 的验证，验证集的准确率高达 55%。然后，当我在新的测试集（2019 年的数据）上使用该模型时，准确率降至 51% 以下。
有人可以解释为什么会这样吗？
我的意思是，我假设模型没有“看到”（训练过）验证数据而不是测试数据，所以它真的会过度拟合吗？
我在下面附上了一个简单的模型来说明。那个在验证集上给出了 54%，但在测试集上只给出了 50.9%。
感谢您的帮助！
NB 我的一个理论是，由于某些特征依赖于历史数据（例如移动平均线），它可能是某种数据泄漏。然后，我尝试仅通过不属于创建移动平均线的样本数据来纠正这一点。例如，如果有 3 个周期的移动平均值，那么我就不会采样/使用 2 个周期的数据行。这并没有改变任何东西，所以它不在下面的模型中。
NB2 下面的模型是我使用的一个简单版本。对我来说，验证集的原因是我使用遗传算法进行超参数调整，但为了清楚起见，这里删除了所有这些。
import pandas as pd
import talib as ta
from sklearn.utils import shuffle
pd.options.mode.chained_assignment = None
from sklearn.metrics import accuracy_score
# ## TRAINING AND VALIDATING
# ### Read in data
input_data_file = 'EURUSDM5_2004-2018_cleaned.csv' # For train and validation
df = pd.read_csv(input_data_file)
# ### Generate features
#######################
# SET TARGET
#######################
df['target'] = df['Close'].shift(-1)>df['Close'] # target is binary, i.e. either up or down next period
#######################
# DEFINE FEATURES
#######################
df['rsi'] = ta.RSI(df['Close'], 14)
# ### Treat the data
#######################
# FIND AND MAKE CATEGORICAL VARAIBLES AND DO ONE-HOT ENCODING
#######################
for col in df.drop('target',axis=1).columns: # Crude way of defining variables with few unique variants as categorical
if df[col].nunique() < 25:
df[col] = pd.Categorical(df[col])
cats = df.select_dtypes(include='category') # Do one-hot encoding for the categorical variables
for cat_col in cats:
df = pd.concat([df,pd.get_dummies(df[cat_col], prefix=cat_col,dummy_na=False)],axis=1).drop([cat_col],axis=1)
uints = df.select_dtypes(include='uint8')
for col in uints.columns: # Variables from the one-hot encoding is not created as categoricals so do it here
df[col] = df[col].astype('category')
#######################
# REMOVE ROWS WITH NO TRADES
#######################
df = df[df['Volume']>0]
#######################
# BALANCE NUMBER OF UP/DOWN IN TARGET SO THE MODEL CANNOT SIMPLY CHOOSE ONE AND BE SUCCESSFUL THAT WAY
#######################
df_true = df[df['target']==True]
df_false = df[df['target']==False]
len_true = len(df_true)
len_false = len(df_false)
rows = min(len_true,len_false)
df_true = df_true.head(rows)
df_false = df_false.head(rows)
df = pd.concat([df_true,df_false],ignore_index=True)
df = shuffle(df)
df.dropna(axis=0, how='any', inplace=True)
# ### Split data
df = shuffle(df)
split = int(0.95*len(df))
train_set = df.iloc[0:split]
val_set = df.iloc[split:-1]
# ### Generate X,y
X_train = train_set[train_set.columns.difference(['target', 'Datetime'])]
y_train = train_set['target']
X_val = val_set[val_set.columns.difference(['target', 'Datetime'])]
y_val = val_set['target']
# ### Scale
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
cont = X_train.select_dtypes(exclude='category') # Find columns with continous (not categorical) variables
X_train[cont.columns] = sc.fit_transform(X_train[cont.columns]) # Fit and transform
cont = X_val.select_dtypes(exclude='category') # Find columns with continous (not categorical) variables
X_val[cont.columns] = sc.transform(X_val[cont.columns]) # Transform
cats = X_train.select_dtypes(include='category')
for col in cats.columns:
X_train[col] = X_train[col].astype('uint8')
cats = X_val.select_dtypes(include='category')
for col in cats.columns:
X_val[col] = X_val[col].astype('uint8')
# ## MODEL
from xgboost import XGBClassifier
model = XGBClassifier()
model.fit(X_train, y_train)
predictions = model.predict(X_val)
acc = 100*accuracy_score(y_val, predictions)
print('{0:0.1f}%'.format(acc))
# # TESTING
input_data_file = 'EURUSDM5_2019_cleaned.csv' # For testing
df = pd.read_csv(input_data_file)
#######################
# SET TARGET
#######################
df['target'] = df['Close'].shift(-1)>df['Close'] # target is binary, i.e. either up or down next period
#######################
# DEFINE FEATURES
#######################
df['rsi'] = ta.RSI(df['Close'], 14)
#######################
# FIND AND MAKE CATEGORICAL VARAIBLES AND DO ONE-HOT ENCODING
#######################
for col in df.drop('target',axis=1).columns: # Crude way of defining variables with few unique variants as categorical
if df[col].nunique() < 25:
df[col] = pd.Categorical(df[col])
cats = df.select_dtypes(include='category') # Do one-hot encoding for the categorical variables
for cat_col in cats:
df = pd.concat([df,pd.get_dummies(df[cat_col], prefix=cat_col,dummy_na=False)],axis=1).drop([cat_col],axis=1)
uints = df.select_dtypes(include='uint8')
for col in uints.columns: # Variables from the one-hot encoding is not created as categoricals so do it here
df[col] = df[col].astype('category')
#######################
# REMOVE ROWS WITH NO TRADES
#######################
df = df[df['Volume']>0]
df.dropna(axis=0, how='any', inplace=True)
X_test = df[df.columns.difference(['target', 'Datetime'])]
y_test = df['target']
cont = X_test.select_dtypes(exclude='category') # Find columns with continous (not categorical) variables
X_test[cont.columns] = sc.transform(X_test[cont.columns]) # Transform
cats = X_test.select_dtypes(include='category')
for col in cats.columns:
X_test[col] = X_test[col].astype('uint8')
predictions = model.predict(X_test)
acc = 100*accuracy_score(y_test, predictions)
print('{0:0.1f}%'.format(acc))