CAISO NP15 FMM LMP Forecast Case Study: Forecast Performance Analysis

Summary

This case study examined the performance of a machine-learning model for NP15 FMM LMP forecasting in CAISO, with forecasts generated every 15 minutes over a rolling 24-hour horizon.

The analysis demonstrates why conventional forecast error metrics alone are insufficient for evaluating a high-frequency intraday forecasting system. The final forecast available immediately before delivery can achieve substantially lower error because it benefits from the most recent market observations and information. While this is a valid measure of final forecast accuracy, it does not fully represent the predictive information that was available to a trader or operator several hours earlier.

To address this distinction, the study introduced a forecast-vintage approach in which every model execution is preserved and evaluated according to its original execution timestamp. This allows the forecast to be assessed without future information leakage and provides a genuine forward-looking measurement of model performance.

The analysis also examined consecutive forecast executions for common target intervals to evaluate forecast stability. The objective is not for successive forecasts to remain unchanged. A useful intraday model must respond to new information. Instead, stability is demonstrated when forecast revisions remain coherent with changing market conditions, while avoiding unnecessary oscillation between consecutive runs.

The results therefore support a broader approach to intraday forecast evaluation: accuracy measures how close the forecast was to the realized price, while forecast-vintage and stability analysis measure how useful and reliable that forecast was from the perspective of when it was actually generated.

For operational applications such as BESS scheduling, this distinction is particularly important. Effective decisions require a coherent sequence of expected prices across multiple future intervals rather than an accurate prediction only at the final moment before delivery. The combination of conventional accuracy metrics with forward-looking and stability analysis therefore provides a more complete assessment of the practical performance of an intraday price forecasting system.

1. Introduction

Intraday electricity price forecasting presents a fundamentally different forecasting problem from day-ahead price prediction. In a day-ahead setting, forecasts are generally produced at a defined point in time and evaluated against prices that become available later. Intraday markets, by contrast, evolve continuously as new prices, system conditions, and market information become available. Forecast models may therefore be executed repeatedly throughout the operating day, with each execution producing a revised view of the future.

This creates an important challenge in evaluating intraday forecast performance. A forecast generated immediately before delivery can be highly accurate because it incorporates the latest available price observations and market information. However, this final forecast revision does not necessarily represent the model’s predictive capability from the perspective of an earlier trading or operational decision. Evaluating only the final forecast can therefore provide an incomplete view of how effectively the model anticipated future market behavior.

This case study examines NP15 Locational Marginal Price (LMP) forecasting in the CAISO Fifteen-Minute Market (FMM) using a forecasting framework implemented in QR AMPS (AI Market Prediction System). The model produces probabilistic forecasts at 15-minute resolution over a rolling 24-hour horizon, with new forecasts generated every 15 minutes.

The analysis focuses on a broader definition of forecast performance that goes beyond conventional point-forecast error metrics. Each model execution is treated as a separate forecast vintage, identified by its execution timestamp. This makes it possible to reconstruct what the model predicted about a future interval using only the information available at that particular point in time. Earlier forecast vintages can then be compared with subsequent vintages and the eventual realized price.

Using this framework, the study evaluates intraday forecasting across three complementary dimensions: predictive accuracy, forecast stability, and forward-looking performance. The analysis examines how forecasts evolve as delivery approaches, how consistently consecutive executions represent the underlying price trajectory, and how forecast behavior changes between short and longer intraday horizons.

The distinction is particularly relevant for applications such as battery energy storage systems (BESS), where decisions depend not on the accuracy of a single upcoming interval, but on the expected sequence of prices across multiple intervals. Ramp limitations, state-of-charge constraints, and coordinated charging and discharging decisions require a forecast that provides useful information beyond the next market interval.

2. CAISO Fifteen-Minute Intraday Market

CAISO Fifteen-Minute Market (FMM) Locational Marginal Prices (LMPs) are calculated at 15-minute intervals to reflect the dynamic state of the power grid, driven by real-time shifts in load, renewable generation, transmission congestion, and market participant dispatch. These prices are published for major trading hubs including the trading hubs NP15, SP15, and ZP26 as well as individual pricing nodes (PNode) across the CAISO footprint.

In this study, the target variable is the CAISO NP15 FMM LMP modeled at a 15-minute temporal resolution. Aggregating prices to hourly or daily averages masks critical intra-hour volatility and price spikes. For instance, two operating hours with identical hourly average prices can follow drastically different intraday paths across their four constituent 15-minute intervals, such as a steep ramp in the final interval versus localized price spikes in the first.

Incorporating HAP Data into the FMM Forecast Model

The CAISO Hour-Ahead Scheduling Process (HASP / HAP) runs hourly at T-75 minutes prior to the operating hour, generating advisory clearing schedules and advisory prices for the four upcoming 15-minute intervals. Because HASP reflects updated near-term grid state estimates, incorporating real-time wind/solar ramp deviations, updated load forecasts, and intertie schedules, it provides a strong forward-looking signal for the immediate intraday market trajectory.

However, integrating HASP signals into a multi-step FMM forecast model presents a feature-availability challenge: HASP offers high-predictive-power inputs for the next operating hour (t+1 to t+4), but these features naturally expire or become unavailable for extended forecast horizons (t+5 and beyond). To overcome this horizon boundary condition, the feature engineering pipeline within the QR AMPS platform dynamically structures the feature space using horizon-specific model architectures (or target-aligned feature masking). This enables the forecasting framework to maximize short-term accuracy from hour-ahead clearing signals while maintaining robust multi-hour baseline predictions.

3. Challenges in Evaluating Intraday Forecasting Performance

A central challenge addressed in this case study is establishing a meaningful framework for measuring intraday forecast performance, particularly in terms of directional accuracy and forecast stability.

The error evaluation problem is fundamentally different from a conventional DAM forecast. In a DAM setting, the forecast is typically generated at a defined time, for example, at 08:00, and the cleared DAM LMP price becomes available later in the day. Once the actual DAM price is published, conventional error metrics such as MAPE, MAE, and RMSE can be calculated by comparing the forecast with the realized price.

Consider instead an intraday FMM AI forecasting model that executes every 15 minutes and produces a 15-minute-resolution forecast for the next 24 hours. Each execution generates 96 forward forecast points:

• 07:00: forecast from 07:15 for the next 96 intervals
• 07:15: forecast from 07:30 for the next 96 intervals
• 07:30: forecast from 07:45 for the next 96 intervals
• 07:45: forecast from 08:00 for the next 96 intervals

If the DAM evaluation approach is applied naively, by 07:15 there is already an actual FMM price and a corresponding forecast for 07:15, allowing a single-point MAPE or MAE to be calculated. However, this measurement has limited value from a trading or operational perspective. Assets such as BESS have ramp, state-of-charge, charge and discharge constraints, and decisions therefore depend on a sequence of expected prices rather than the accuracy of one isolated 15-minute interval.

A forecast issued at 07:00, for example, needs to provide a useful forward-looking price trajectory across 07:15, 07:30, 07:45, 08:00 and beyond. The relevant question is therefore not simply whether the next interval was predicted accurately, but whether the model provided a stable, directionally informative and progressively improving view of the future as new market information became available.

The appropriate framework is therefore to evaluate consecutive forecast vintages from the perspective of the time at which they were generated. For example, forecasts issued at 06:15, 06:30, 06:45 and 07:00 can be compared over their common forward horizon to determine not only their numerical accuracy, but also whether they maintain a consistent trajectory and correctly anticipate the direction of subsequent price movements.

• For short forward horizons, particularly the next two to three hours, the expectation is strong responsiveness and accuracy, since the model has progressively more recent information available.
• At longer horizons, uncertainty naturally increases, particularly around price spikes and unforeseen market conditions. Nevertheless, the model should continue to provide a credible price level and overall trajectory over the remainder of the day.

The following analysis therefore evaluates intraday performance across three complementary dimensions: accuracy, directional consistency, and forecast stability, distinguishing the genuinely forward-looking forecast from the highly informed final revision immediately before delivery.

4. Real-time Performance Analysis

We begin with the standard QR AMPS forecast performance dashboard, which reports conventional forecast error metrics including MAPE, MAE, RMSE, and Direction Win Rate. These metrics evaluate the forecast for each realized FMM interval against the actual FMM price, using the forecast generated in the immediately preceding model execution.

While these metrics provide an important measure of point-forecast accuracy, as discussed in Section 3, they do not fully capture the performance of a high-frequency intraday forecasting system. In particular, evaluating only the most recent forecast revision can overstate the model’s genuinely forward-looking capability because that revision benefits from the latest available market and price information.

• Model execution: 07:00, September 04
• Forecast horizon: 07:15, September 04 through 07:15, September 05
• Number of forecast intervals: 96

Fifteen minutes later, the model executes again:

• Model execution: 07:15, September 04
• Forecast horizon: 07:30, September 04 through 07:30, September 05
• Number of forecast intervals: 96

The same process continues throughout the operating period. Each execution incorporates the latest available observations and market information and produces a new forward-looking forecast.

This creates a sequence of overlapping forecast vintages. The rolling analysis preserves these vintages so that the forecast can be evaluated from the perspective of the timestamp at which it was actually generated, rather than retrospectively using the final revision that became available immediately before delivery.

The resulting analysis can be viewed as a rolling forward movie of the model’s expectation of the market: each new execution updates the forecast, allowing us to observe both how the forecast improves as delivery approaches and how consistently the model maintains its expected price trajectory further into the future.

The following sections examine these forecast vintages in greater detail, beginning with the forward-looking performance of an individual model execution and then comparing consecutive executions to quantify and visualize forecast stability.

4.1 Rolling Forecast Perspective

A key objective of real-time forecast evaluation is to isolate an individual model execution and examine the complete forward-looking forecast available at that moment.

Consider the AI forecast model execution at 07:00 on September 04. At this timestamp, the model generates a 24-hour forecast beginning with the next FMM interval. To evaluate what the model actually knew at 07:00, the forecast must be preserved as a distinct forecast vintage identified by its execution timestamp.

The QR AMPS data management architecture supports this type of forecast versioning. For example, the forecast for a target interval such as 07:30 can be retrieved from successive model executions at 07:15, 07:00, 06:45, 06:30, and earlier timestamps. Each version represents the model’s prediction of the same future interval using only the information available at that particular execution time.

The QR API can be configured to retrieve a user-defined number of forecast vintages and pass them to the QR AI Forecast Dashboard. The starting point for the historical version sequence can also be configured by the user.

In addition to the above plots, the forecast versions are displayed in tabular format as shown next.

Note: In the CAISO FMM, at any given observation time, the most recent market information may already provide realized or published FMM LMPs for the next two 15-minute intervals. For example, at 06:10, FMM LMP information is already available for 06:15 and 06:30. Therefore, an intraday forecasting model should not treat these intervals as forecast targets. To maintain a genuinely forward-looking forecast and avoid using already available market information, the forecast should begin with the third subsequent 15-minute interval – in this example, 06:45. The same rule should be applied consistently at every scheduled model execution.

The report output is presented in the table below, it also displays:

• the actual realized FMM prices; and
• the resulting forecast error metrics.

MAPE can be calculated for each forecast execution over its complete forward horizon, allowing the performance of different forecast vintages to be compared consistently. Note that the final forecast MAPE is equal for all = 6.49 %

This produces a genuinely forward-looking evaluation. At 06:30, for example, the model has no access to prices or other information that becomes available after 06:30. The forecast for the remainder of the day therefore represents the model’s view of the market based strictly on the information available at that timestamp.

Additional examples of forecast trajectories initiated at different execution times with their respective MAPEs:

* Rolling Forecast MAPE: MAPE between the actual realized FMM price for each interval and the forecast for the entire day as generated at the specified forecast timestamp. The evaluation horizon is 24 hours.

Consider the forecast generated around 08:00, more precisely at 07:51, as illustrated in the figure below. The forecast execution time is shown in the chart tooltip. Consistent with the CAISO FMM characteristics discussed above, the first forecasted interval is 08:30, indicated by the vertical dashed line.

The two metrics answer different questions. Final Forecast MAPE measures the accuracy of the latest forecast available immediately before delivery, while Rolling Forecast MAPE measures how accurately the model anticipated the market from an earlier decision point.

For trading and operational applications, the rolling measure is particularly important because decisions must often be made before the final forecast revision becomes available. This makes forecast versioning a critical component of a meaningful intraday performance evaluation.

4.2 Multiple Consecutive Runs: Monitoring Forecast Stability

A second dimension of the analysis is to examine how the forecast for a specific target interval evolves across consecutive model executions.

Consider a target FMM interval of 09:00. Instead of evaluating only the final forecast for that interval, we retrieve the forecasts generated at progressively earlier execution times:

Each execution produces a complete forward forecast, but the analysis isolates the same target interval across these forecast vintages. The resulting forecasts can then be plotted together to show how the model’s expectation changes as the target interval approaches.

The objective is not for consecutive forecast lines to be identical. Intraday forecasts should respond to newly available information. Instead, the objective is to determine whether forecast revisions are coherent and proportionate to changes in the underlying market information.

A stable model should generally demonstrate the following behavior:

• nearby forecast runs should remain broadly consistent when market conditions are stable;
• meaningful new information should produce meaningful forecast adjustments for example adjusting with previous points.
• short-horizon forecasts should converge toward the realized price as delivery approaches;
• longer-horizon forecasts should evolve more gradually as uncertainty is reduced over time;
• consecutive runs should avoid unnecessary oscillations that are not supported by corresponding changes in market conditions.

4.3 Forecast Stability Across Horizons

Forecast behavior also needs to be evaluated according to the forecast horizon. The information available to the model becomes increasingly uncertain further into the future, so the expected behavior of the forecast should differ between the near-term and longer intraday horizons.

Near-term horizon

Over the next two to three hours, the model should be highly responsive to newly available market information. As additional FMM prices and other relevant observations become available, the forecast should incorporate this information and progressively refine its expected price trajectory. Lagged prices and neighboring price information are particularly important over this horizon.

• In the zoomed-in view below, the four consecutive forecast executions at 06:00, 06:15, 06:30, and 06:45 show a high degree of consistency across the overlapping forecast horizon.
• The forecast trajectories remain closely aligned and continue to provide a coherent forward-looking price shape through approximately 09:30, indicating that the model incorporates newly available information without introducing excessive forecast revisions or instability.

Long-term intraday horizon

Further into the 24-hour forecast horizon, uncertainty naturally increases. The objective is therefore not to reproduce every individual 15-minute price movement, particularly unexpected spikes, but to determine whether the model maintains a credible representation of the expected market behavior.

At longer horizons, the evaluation focuses on whether the forecast maintains a reasonable:

• level,
• daily profile,
• peak structure,
• trough structure,
• and uncertainty range.

The forward-looking views of the same forecast runs demonstrate that the model can establish a credible overall price level and trajectory well before delivery. As new information becomes available, individual forecast points are revised, but the earlier forecast vintages maintain a coherent representation of the broader intraday market pattern.

This distinction is important: forecast stability does not mean that the model stops adapting. Rather, it means that forecast revisions remain consistent with the information entering the model, allowing the forecast to progressively improve as the delivery interval approaches without losing its forward-looking value.

5 Forecasting Technology

Accuracy analysis requires one more dimensional considerations and that is incorporating probabilistic forecasts as delivered by QR AMPS. These will be discussed in a separate publication.

The probabilistic forecasting framework generates a full forecast distribution and supports any configured set of quantiles. In the QR AMPS dashboard one can select the forecast quantiles, P5, P10, P25, P50 (main forecast), P75, P90, and P95. These are displayed in both the chart and the accompanying table. The resulting prediction intervals provide a direct view of forecast uncertainty around the central P50 estimate.

The forecasting workflow can be implemented end-to-end in the QR AMPS cloud environment. The objective is to make the process repeatable rather than dependent on manual model execution. The platform allows the complete workflow to be configured within the same environment.

5.1 Automated Execution

Once the model is finalized, the forecasting workflow can be scheduled to execute automatically. For example every 15 minutes:

1. retrieve newly available data,
2. update the historical dataset,
3. generate the latest features,
4. incorporate the newly observed price point,
5. retrain or update the model,
6. generate the next 24-hour probabilistic forecast,
7. store the forecast,
8. and make the latest forecast available for analysis.

This architecture transforms forecasting from a one-time exercise into a continuous process.

5.2 From a Single Forecast to a Continuous Forecasting System

Production grade enterprise level AI forecasting in real-time intraday FMM markets requires a continuous forecasting platform managing at every execution:

New Data Fetching from CAISO → Feature Update → Model Update → New Forecast → Forecast Evaluation

This process repeats every 15 minutes, 24/7/365. QR AMPS architecture allows the model to progressively incorporate the market’s latest information. The value is therefore not only in producing a good forecast. It is in maintaining a continuously updated view of the market.

Conclusion

The CAISO NP15 FMM market presents a particularly challenging environment for intraday price forecasting because the information set changes continuously and the target itself is defined at 15-minute resolution.

In this environment, forecast accuracy alone does not provide a complete measure of model quality. A forecast generated immediately before delivery may be highly accurate because it benefits from recent price observations, updated market information and the proximity of the target interval. That accuracy is operationally valuable, but it should not be confused with the model’s ability to forecast the same interval several hours earlier.

The more meaningful evaluation is therefore based on the entire sequence of forecasts generated through time. For each model run, the forecasting system should preserve the information cutoff and the corresponding forward-looking forecast. This makes it possible to evaluate the model exactly as it would have behaved in real operation.

The case study demonstrates a framework in which the model:

1. forecasts at 15-minute resolution,
2. generates a probabilistic forward-looking forecast,
3. incorporates newly available market information,
4. uses lagged and engineered features,
5. retrains frequently,
6. evaluates multiple AI algorithms,
7. uses LightGBM as the selected model configuration,
8. incorporates volatility-aware information such as Bollinger Bands,
9. and executes automatically on a recurring schedule.

Most importantly, the framework evaluates stability alongside accuracy. The short-term forecast should respond strongly to the latest information and progressively converge toward the realized market. The longer-term forecast should remain credible and maintain a realistic market shape despite greater uncertainty.

Comparing multiple consecutive forecast runs provides an additional layer of validation by showing whether forecast revisions are controlled and explainable rather than erratic. That is the standard that turns an intraday price model from a static forecasting exercise into an operational forecasting system.

For more on the price product, including architecture and live dashboards, see QR AMPS (AI Market Prediction System). If you have comments or questions, contact us at [email protected]. Our team will be glad to discuss your forecasting requirements.

The forecast data and system screenshots are proprietary to QuantRisk. They may be reproduced or used only with clear attribution to QuantRisk.