如何在 Python 中使用 ARIMA 预测数据帧的 NaN(缺失值)?

数据挖掘 机器学习 Python 预测 时间序列
2022-01-21 12:55:48

我有一个df_train形状为 (11808, 1) 的数据框,如下所示:

Datum   Menge
2018-01-01 00:00:00 19.5
2018-01-01 00:15:00 19.0
2018-01-01 00:30:00 19.5
2018-01-01 00:45:00 19.5
2018-01-01 01:00:00 21.0
2018-01-01 01:15:00 19.5
2018-01-01 01:30:00 20.0
2018-01-01 01:45:00 23.0
2018-01-01 02:00:00 20.5
2018-01-01 02:15:00 20.5

以及形状为 (3071, 1) 的第二个 df nan_df,如下所示:

Datum                    Menge
2018-05-04 00:15:00       nan
2018-05-04 00:30:00       nan
2018-05-04 00:45:00       nan
2018-05-04 01:00:00       nan
2018-05-04 01:15:00       nan
2018-05-04 01:30:00       nan
2018-05-04 01:45:00       nan
2018-05-04 02:00:00       nan
2018-05-04 02:15:00       nan

需要使用时间序列预测来预测nan值。nan_df

我做了什么:

下面的代码划分 dfdf_train并在其上运行 ARIMA 模型来预测测试集的值

import pandas as pd
from pandas import datetime
import matplotlib.pyplot as plt
from statsmodels.tsa.arima_model import ARIMA
from sklearn.metrics import mean_squared_error

def parser(x):
    return datetime.strptime(x,'%m/%d/%Y %H:%M')

df = pd.read_csv('time_series.csv',index_col = 1,parse_dates =[1], date_parser = parser)
df = df.drop(['Unnamed: 0'],axis=1)
df_train = df.dropna()

def StartARIMAForecasting(Actual, P, D, Q):
    model = ARIMA(Actual, order=(P, D, Q))
    model_fit = model.fit(disp=0)
    prediction = model_fit.forecast()[0]
    return prediction

NumberOfElements = len(df_train)

TrainingSize = int(NumberOfElements * 0.7)
TrainingData = df_train[0:TrainingSize]
TrainingData = TrainingData.values
TestData = df_train[TrainingSize:NumberOfElements]
TestData = TestData.values

#new arrays to store actual and predictions
Actual = [x for x in TrainingData]
Predictions = list()

#in a for loop, predict values using ARIMA model
for timepoint in range(len(TestData)):
    ActualValue = TestData[timepoint]
    Prediction = StartARIMAForecasting(Actual, 3, 1, 0)
    print('Actual=%f, Predicted=%f' % (ActualValue, Prediction))
    Predictions.append(Prediction)
    Actual.append(ActualValue)

Error = mean_squared_error(TestData, Predictions)
print('Test Mean Squared Error (smaller the better fit): %.3f' % Error)
# plot
plt.plot(TestData)
plt.plot(Predictions, color='red')
plt.show()

现在,我想做同样的事情来预测 中的nannan_df,这次使用整个df_train数据框,我这样做如下:

X = df_train.copy().values
nan_df = df.iloc[11809:, :].values

real = [x for x in X]
nan_Predictions = list()

#in a for loop, predict values using ARIMA model
for timepoint in range(len(nan_df)):
    nan_ActualValue = nan_df[timepoint]
    nan_Prediction = StartARIMAForecasting(real, 3, 1, 0)
    print('real=%f, Predicted=%f' % (nan_ActualValue, nan_Prediction))
    nan_Predictions.append(nan_Prediction)
    real.append(nan_ActualValue)

当我这样做时,我收到以下错误:

Traceback (most recent call last):

  File "<ipython-input-42-33f3e242230d>", line 4, in <module>
    nan_Prediction = StartARIMAForecasting(real, 3, 1, 0)

  File "<ipython-input-1-043dac0dd994>", line 17, in StartARIMAForecasting
    model_fit = model.fit(disp=0)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\statsmodels\tsa\arima_model.py", line 1157, in fit
    callback, start_ar_lags, **kwargs)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\statsmodels\tsa\arima_model.py", line 946, in fit
    start_ar_lags)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\statsmodels\tsa\arima_model.py", line 562, in _fit_start_params
    start_params = self._fit_start_params_hr(order, start_ar_lags)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\statsmodels\tsa\arima_model.py", line 539, in _fit_start_params_hr
    if p and not np.all(np.abs(np.roots(np.r_[1, -start_params[k:k + p]]

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\numpy\lib\polynomial.py", line 245, in roots
    roots = eigvals(A)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\numpy\linalg\linalg.py", line 1058, in eigvals
    _assertFinite(a)

  File "C:\Users\kashy\Anaconda3\envs\py36\lib\site-packages\numpy\linalg\linalg.py", line 218, in _assertFinite
    raise LinAlgError("Array must not contain infs or NaNs")

LinAlgError: Array must not contain infs or NaNs

所以,我想知道如何预测中的nannan_df

1个回答

您可以将Wolfram 语言应用到您的项目中。有一个免费的Wolfram 引擎供开发人员使用,您可以通过 Python 的Wolfram 客户端库在 Python中使用这些函数。

pandas.DataFrame我将首先使用 Python在 a 中创建一些数据(OP 中提供的行太少)WolframLanguageSession来模拟 a ARIMAProcesswith RandomFunction

进口

import pandas as pd
import iso8601

from wolframclient.evaluation import WolframLanguageSession
from wolframclient.language import wl, wlexpr

开始WolframLanguageSession

wolfSession = WolframLanguageSession();

可以通过以下方式运行模拟

print(
    wolfSession.evaluate(
        wl.RandomFunction(
            wl.ARIMAProcess(1.4, [-0.9], 1, [-0.01, -0.08], 2.6), 
            [0, 3]
        )('Values')
    )
)
[1.1054178694529107, 1.860340990531042, 1.5519448249118848, 5.088452598965132]

n以 15 分钟间隔运行 100 个步骤 ( ) 的模拟,如示例数据片段所示。

n=100;
df = pd.DataFrame(
        {
        'Datum' : pd.date_range('2018-01-01 00:00', periods=n, freq=pd.offsets.Minute(15)),
        'Menge' : wolfSession.evaluate(
                    wl.RandomFunction(
                        wl.ARIMAProcess(1.4, [-0.9], 1, [-0.01, -0.08], 2.6), 
                        [0, n-1]
                    )('Values')
                )
        },
    );

结果可以DateListPlot通过支持的Raster Image FormatsVector Graphics FormatsExport之一进行可视化

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>', 
        wl.DateListPlot(wl.Query(wl.Values,wl.Values)(df), PlotTheme='Detailed')
    )
)

数学图形

现在有了一些数据,就可以进行预测了。首先TimeSeriesModelFit将用于适应 ARIMA 过程。然后TimeSeriesForecast预测未来的价值。

Wolfram 引擎将 a 解释pandas.DataFrame为一个Dataset对象。TimeSeriesModelFit需要一个TimeSeries或一个时间值对列表。因此,使用Query转换是对时间-值对列表进行的。

ts_model=wolfSession.evaluate(
    wl.Query(
        wl.RightComposition(
            wl.Values,
            wl.Function(wl.TimeSeriesModelFit(wl.Slot(1),'ARIMA'))
        ),
        wl.Values
    )(df)
);

print(wolfSession.evaluate(ts_model('BestFit')))
ARIMAProcess[1.4524650139005593, [-0.9099324923212446], << 1 >>, [-0.07171874225371022], 2.507600357444524]

可以模拟前进 20 步

ts_forecast=wolfSession.evaluate(wl.TimeSeriesForecast(ts_model,[20]));

print(wolfSession.evaluate(ts_forecast('Values')))
[69.49895300256293, 73.9505213906962, 71.35235968644419, 75.16897645534839, 73.14857786048489, 76.43946920329194, 74.89744525567367, 77.75304796344956, 76.60710728838431, 79.10230095679928, 78.28430817717482, 80.48109139973985, 79.93463198084233, 81.88433817573274, 81.56270217242249, 83.30783423643538, 83.17234688189897, 84.74809624199086, 84.76673571338941, 86.20224006662474]

ts_forecast是一个TemporalData对象,其属性包括'Dates''Values'这些可用于将其转换为 Pythonpandas.DataFrame以便在 Python 中进行进一步处理。

df_forecast = pd.DataFrame(
        {
        'Datum' : list(map(
                    lambda d: iso8601.parse_date(d), 
                    wolfSession.evaluate(
                        wl.Map(
                            wl.Function(wl.DateString(wl.Slot(1),'ISODateTime')),
                            ts_forecast('Dates')
                        )
                    )
                )),
        'Menge' : wolfSession.evaluate(ts_forecast('Values'))
        },
    );

print(df_forecast.iloc[:3,:])
                      Datum      Menge
0 2018-01-02 01:00:00+00:00  69.498953
1 2018-01-02 01:15:00+00:00  73.950521
2 2018-01-02 01:30:00+00:00  71.352360

进一步的处理也可以使用 Wolfram 引擎继续。例如,预测的 95% 置信区间带。

conf = .95;
quant = wolfSession.evaluate(wl.Quantile(wl.NormalDistribution(), 1 - (1 - conf) / 2));
errors = wolfSession.evaluate(wl.Sqrt(ts_forecast('MeanSquaredErrors')));
error_bands = wolfSession.evaluate(
                wl.TimeSeriesThread(
                    wl.Function([wl.Dot([1, -quant], wl.Slot(1)), wl.Dot([1, quant], wl.Slot(1))]),
                    [ts_forecast, errors]
                )
            );

wolfSession.evaluate(
    wl.Export(
        '<path with image filename>', 
        wl.DateListPlot(
            [wl.Query(wl.Values,wl.Values)(df), error_bands, ts_forecast], 
            PlotStyle=[wl.Automatic, wl.Gray, wl.Automatic], 
            Filling=[wl.Rule(2,[3])],
            PlotTheme='Detailed'
        ) 
    )
);

数学图形

终止会话

wolfSession.terminate();

希望这可以帮助。