Office of Operations
21st Century Operations Using 21st Century Technologies
Table of Contents

Integrated Modeling for Road Condition Prediction — Phase 3 Evaluation Report

Chapter 4. Findings

The purpose of the evaluation is to explore whether Integrated Modeling for Road Condition Prediction (IMRCP) had an impact on the Kansas City Scout (KC Scout) operations and to assess whether the information was useful to the KC Scout operators and supervisors. The key questions guiding the evaluation data collection and analyses are: (1) Did IMRCP have an operational impact? and (2) Did the users consider the IMRCP information useful? To explore whether IMRCP had an impact, the evaluation investigated the accuracy of IMRCP speeds and speed forecasts. To investigate whether IMRCP information was useful, KC Scout operators and supervisors were interviewed to obtain their insights and perspectives. The findings in this section describe the outcomes of investigating speed and speed forecast accuracy and operator and supervisor perceptions of IMRCP’s operational impact and usefulness.

Did Integrated Modeling for Road Condition Prediction Have an Operational Impact?

This evaluation found that IMRCP had minimal operational impact during the 2018–2019 and 2019–2020 winter driving seasons. However, operators did report that they often referred to IMRCP for weather forecast information prior to weather events. They stated that during normal day operations, they relied on existing KC Scout tools, applications, and information sources to monitor real-time traffic operations. Consequently, IMRCP speeds and speed forecasts were basically unused by the operators.

Analysis of the Traffic Estimation and Prediction System (TrEPS) and machine learning-based prediction (MLP) speeds and speed forecasts at three locations during seven winter days revealed that the two IMRCP models produced speeds and speed forecasts of varying accuracy. Three types of analyses for each location and date were completed:

  • Analysis of historical predicted speeds.
  • Analysis of forecast speeds for 15-, 30-, 45-, 60-, 75-, 90-, 105-, and 115-minute (min) predictions.
  • Analysis of speed forecast errors (absolute, relative absolute, and root mean square errors).

Looking at the accuracy of historical predicted speeds, TrEPS speed data were found to be problematic on several dates, sometimes with several hours of predicted speeds considerably different (over 20 miles per hour [mph]) from the detector speeds. The TrEPS model predicts speeds using several input parameters, such as changes to posted speed limit, roadway construction (or blockage) information, and detector data. If any of these inputs are inaccurate or missing, it may affect TrEPS ability to predict accurate speeds. For example, it appeared that speeds reported by TrEPS at Interstate 435 (I–435) eastbound (EB) at State Line Road were affected by a construction work zone that was not reported by KC Scout. The MLP speed data often appeared to be within about 5 mph of the detector speeds and were much less affected by the I––435 at State Line Road construction work zone changes, but occasionally had periods with large deviations (up to 20 mph) above or below the detector speeds.

Analysis of forecast speeds found that MLP and TrEPS speed forecasts both showed periods where the forecasts appeared to have large deviations (up to 20 mph) above and below the detector speeds. The deviations appeared to grow larger and for longer periods of time with longer-term forecasts. The MLP forecast speeds tended to more closely match the detector speeds than the TrEPS forecasts and the errors were examined in the forecast speed error analysis.

The analysis of speed forecast errors used three formulas to calculate and compare the speed forecast errors: absolute, relative absolute, and root mean square error.

The absolute error analysis measured the difference between the forecast speed and the actual detector speed. In general, when examining the 15–115-min speed forecasts:

  • The MLP absolute error showed that shorter-term forecasts tended to be more accurate than longer-term forecasts. The results at the three detector locations (I–435 at State Line Road, I–435 at East Stadium Drive, and I–435 at East Antioch Road) showed the 15-min forecasts were from 3 to 8 mph more accurate than the 115-min forecasts.
  • The TrEPS absolute errors showed mixed results for speed forecasts. At I–435 and State Line Road, the absolute errors were relatively consistent across all forecast times. On January 29, 2020, the absolute error was relatively consistent, ranging from about 35 to 42 mph. However, on February 12, 2020, the absolute error was about 40 mph for the 15-min forecast, decreased to about 11 mph for the 75-min forecast, then increased to about 21 mph for the 105-min forecast.
  • The TrEPS absolute errors for speed forecasts tended to be larger than the MLP absolute errors. At I–435 and State Line Road, the average TrEPS error across all forecast times was about 26 mph versus about 10.6 mph for MLP. At I–435 at East Antioch Road, the average TrEPS error across all forecast times was about 33 mph versus about 7 mph for MLP.

The relative absolute error measured how large the absolute error was compared to the actual speed and provided the percent size of the error. In general, when examining the 15–115-min speed forecasts:

  • The MLP relative absolute error also showed that shorter-term forecasts tended to be more accurate than longer-term forecasts, in that the error as a percentage of the speed was smaller for shorter-term forecasts. The results at the three detector locations showed the 15-min forecasts were from 7 to 18 percent versus 13 to 41 percent for the 115-min forecasts.
  • The TrEPS relative absolute errors generally showed mixed results. At I–435 and State Line Road, the relative absolute errors were relatively large (ranging from 40 to over 100 percent) and mostly consistent across all forecast times. The average relative absolute error was about 73 percent for all forecast times. On January 29, 2020, the relative absolute error was about 66 percent for the 15-min forecast, steadily decreased to about 18 percent for the 75-minute forecast, then increased to about 48 percent for the 105-min forecast.
  • Comparing TrEPS to MLP, the TrEPS relative absolute errors were generally larger. At I–435 and State Line Road, the average TrEPS error across all forecast times was about 73 percent versus about 29 percent for MLP. At I–435 at East Antioch Road, the average TrEPS error across all forecast times was about 56 percent versus about 12 percent for MLP.

The root mean square error measured the spread (or concentration) between forecasts and actual speeds. In general, when examining the 15–115-min forecasts:

  • The MLP root mean square error showed that shorter-term forecasts errors tended to be smaller than longer-term forecasts. The root mean square error ranged from 3 to 12 mph.
  • For TrEPS, the root mean square errors were relatively consistent across all forecast times. The root mean square error ranged from 19 to 32 mph.
  • Comparing the root mean square errors of TrEPS to MLP, the TrEPS root mean square errors were found to be two to three times larger than the MLP root mean square errors. At I–435 and East Antioch Road, the average TrEPS error across all forecast times was about 27 mph versus about 9 mph for MLP.

Did Users Consider IMRCP Information as Useful?

This evaluation found that KC Scout operators and supervisors liked the weather-related prediction components of IMRCP, but preferred to use existing tools for obtaining information about real-time traffic conditions and incidents. Although road and weather information were sometimes helpful, the operators did not rely on IMRCP to select more relevant operational strategies. A KC Scout representative stated that IMRCP might be a helpful tool for planning and assessing winter maintenance response efforts and emergency operation staffing decisions.

The operators stated they accessed IMRCP when weather events were approaching, and found the weather information helpful. They mentioned that IMRCP was useful for both rain and winter weather, but more so for when winter weather was expected. The operators used it less during heavy rain or flooding events. Otherwise, the operators tended to not use IMRCP as part of their normal daily routines.

previous | next