Office of Operations
21st Century Operations Using 21st Century Technologies
Table of Contents

Integrated Modeling for Road Condition Prediction Phase 3 Project Report

Chapter 4. Evaluation

Introduction

This chapter provides a summary of the Integrated Modeling for Road Condition Prediction (IMRPC) evaluation effort. The complete details and results of the evaluation are documented within the evaluation report.

Summary of Findings

The evaluation explored whether or not IMRCP had an impact on the Kansas City Scout (KC Scout) operations and to assess whether the information was useful to the KC Scout operators and supervisors. The key questions guiding the evaluation data collection and analyses are: (1) Did IMRCP have an operational impact? and (2) Did the users consider the IMRCP information useful? To explore whether IMRCP had an impact, the evaluation investigated the accuracy of IMRCP speeds and speed forecasts. To investigate whether IMRCP information was useful, KC Scout operators and supervisors were interviewed to obtain their insights and perspectives. The findings in this section describe the outcomes of investigating speed and speed forecast accuracy and operator and supervisor perceptions of IMRCP's operational impact and usefulness.

Did Integrated Modeling for Road Condition Prediction Have an Operational Impact?

This evaluation found that IMRCP had minimal operational impact during the 2018–2019 and 2019–2020 winter driving seasons. However, operators did report that they often referred to IMRCP for weather forecast information prior to weather events. They stated that during normal day operations, they relied on existing KC Scout tools, applications, and information sources to monitor real-time traffic operations. Consequently, IMRCP speeds and speed forecasts were basically unused by the operators.

Analysis of the Traffic Estimation and Prediction System (TrEPS) and machine learning-based prediction (MLP) speeds and speed forecasts at three locations during seven winter days revealed that the two IMRCP models produced speeds and speed forecasts of varying accuracy. Three types of analyses for each location and date were completed:

  • Analysis of historical predicted speeds.
  • Analysis of forecast speeds for 15-, 30-, 45-, 60-, 75-, 90-, 105-, and 115 minute (min) predictions.
  • Analysis of speed forecast errors (absolute, relative absolute, and root mean square errors).

Looking at the accuracy of historical predicted speeds, TrEPS speed data were found to be problematic on several dates, sometimes with several hours of predicted speeds considerably different (over 20 miles per hour [mph]) from the detector speeds. The TrEPS model predicts speeds using several input parameters, such as changes to posted speed limit, roadway construction (or blockage) information, and detector data. If any of these inputs are inaccurate or missing, it may affect TrEPS ability to predict accurate speeds. For example, it appeared that speeds reported by TrEPS at Interstate 435 (I–435) eastbound at State Line Road were affected by a construction work zone that was not reported by KC Scout. The MLP speed data often appeared to be within about 5 mph of the detector speeds and were much less affected by the I–435 at State Line Road construction work zone changes, but occasionally had periods with large deviations (up to 20 mph) above or below the detector speeds.

Analysis of forecast speeds found that MLP and TrEPS speed forecasts both showed periods where the forecasts appeared to have large deviations (up to 20 mph) above and below the detector speeds. The deviations appeared to grow larger and for longer periods of time with longer-term forecasts. The MLP forecast speeds tended to more closely match the detector speeds than the TrEPS forecasts and the errors were examined in the forecast speed error analysis.

The analysis of speed forecast errors used three formulas to calculate and compare the speed forecast errors: absolute, relative absolute, and root mean square error.

The absolute error analysis measured the difference between the forecast speed and the actual detector speed. In general, when examining the 15–115-min speed forecasts:

  • The MLP absolute error showed that shorter-term forecasts tended to be more accurate than longer-term forecasts. The results at the three detector locations (I–435 at State Line Road, I–435 at East Stadium Drive, and I–435 at East Antioch Road) showed the 15-min forecasts were from 3 to 8 mph more accurate than the 115-min forecasts.
  • The TrEPS absolute errors showed mixed results for speed forecasts. At I–435 and State Line Road, the absolute errors were relatively consistent across all forecast times. On January 29, 2020, the absolute error was relatively consistent, ranging from about 35 to 42 mph. However, on February 12, 2020, the absolute error was about 40 mph for the 15-min forecast, decreased to about 11 mph for the 75-min forecast, then increased to about 21 mph for the 105-min forecast.
  • The TrEPS absolute errors for speed forecasts tended to be larger than the MLP absolute errors. At I–435 and State Line Road, the average TrEPS error across all forecast times was about 26 mph versus about 10.6 mph for MLP. At I–435 at East Antioch Road, the average TrEPS error across all forecast times was about 33 mph versus about 7 mph for MLP.

The relative absolute error measured how large the absolute error was compared to the actual speed and provided the percent size of the error. In general, when examining the 15–115-min speed forecasts:

  • The MLP relative absolute error also showed that shorter-term forecasts tended to be more accurate than longer-term forecasts, in that the error as a percentage of the speed was smaller for shorter-term forecasts. The results at the three detector locations showed the 15-min forecasts were from 7 to 18 percent versus from 13 to 41 percent for the 115-min forecasts.
  • The TrEPS relative absolute errors generally showed mixed results. At I–435 and State Line Road, the relative absolute errors were relatively large (ranging from 40 to over 100 percent) and mostly consistent across all forecast times. The average relative absolute error was about 73 percent for all forecast times. On January 29, 2020, the relative absolute error was about 66 percent for the 15-min forecast, steadily decreased to about 18 percent for the 75-min forecast, then increased to about 48 percent for the 105-min forecast.
  • Comparing TrEPS to MLP, the TrEPS relative absolute errors were generally larger. At I–435 and State Line Road, the average TrEPS error across all forecast times was about 73 percent versus about 29 percent for MLP. At I–435 at East Antioch Road, the average TrEPS error across all forecast times was about 56 percent versus about 12 percent for MLP.

The root mean square error measured the spread (or concentration) between forecasts and actual speeds. In general, when examining the 15–115-min forecasts:

  • The MLP root mean square error showed that shorter-term forecasts errors tended to be smaller than longer-term forecasts. The root mean square error ranged from 3 to 12 mph.
  • For TrEPS, the root mean square errors were relatively consistent across all forecast times. The root mean square error ranged from 19 to 32 mph.
  • Comparing the root mean square errors of TrEPS to MLP, the TrEPS root mean square errors were found to be two to three times larger than the MLP root mean square errors. At I–435 and East Antioch Road, the average TrEPS error across all forecast times was about 27 mph versus about 9 mph for MLP.

Did Users Consider Integrated Modeling for Road Condition Prediction Information as Useful?

This evaluation found that KC Scout operators and supervisors liked the weather-related prediction components of IMRCP, but preferred to use existing tools for obtaining information about real-time traffic conditions and incidents. Although road and weather information were sometimes helpful, the operators did not rely on IMRCP to select more relevant operational strategies. A KC Scout representative stated that IMRCP might be a helpful tool for planning and assessing winter maintenance response efforts and emergency operation staffing decisions.

The operators stated they accessed IMRCP when weather events were approaching, and found the weather information helpful. They mentioned that IMRCP was useful for both rain and winter weather, but more so for when winter weather was expected. The operators used it less during heavy rain or flooding events. Otherwise, the operators tended to not use IMRCP as part of their normal daily routines. In general, the operators did not use the traffic predictions capability. The operators said they liked the precipitation/weather predictions and the pavement status predictions because they can provide some insight into what locations to monitor more closely as the weather moves in. The operators used this feature to find areas with potential issues affected by the incoming weather. They reported it helped them prioritize and focus attention on particular roadway sections or areas, which in turn may improve their communication and decisions relating to motorist assist deployments. One user reported that IMRCP has been informally used to help inform transportation management center (TMC) staffing decisions during a weather event (e.g., should additional staff be called in, or can staff be sent home?).

The operators reported using IMRCP during the weather event in a more limited capacity. They are often busy during the weather events performing priority duties and responsibilities, such as entering, monitoring, and updating incidents into their system.

The operators typically do not use IMRCP to assess an event that has already passed; however, they acknowledge it may be helpful for data analysis and evaluation to determine lessons learned. One user believed there was a benefit to using IMRCP capabilities and its weather event data to look back and assess winter maintenance response efforts and emergency operation staffing decisions. Insight gained could potentially help save the agency staffing and monetary resources.

Overall, the users reported a fair confidence in the weather-related prediction components of IMRCP. As mentioned earlier, the operators did not use the traffic prediction components. The operators also indicated that periods of IMRCP downtime negatively affected the usage of IMRCP in their daily routines, especially in the 2019–2020 winter season. The operators' responses from the interviews seemed to indicate a more frequent use of IMRCP in the 2018–2019 winter season.

The evaluation has resulted in a better understanding of the impact the IMRCP phase 3 had on KC Scout operations, the accuracy of the speed predictions and forecasts, and the usefulness of the information to KC Scout operators and supervisors.