Freeway Management and Operations Handbook

Chapter 4 – Performance Monitoring and Evaluation
Page 2 of 3

4.2.5 Information Gathering

Obviously, a direct relationship exists between the performance measures selected and the data needed in the performance measurement process. The data and information used in decision-making must be of high quality. They must originate from reliable, consistent sources and meet the needs of the decision makers. Moreover, the decision makers must have confidence in the information, or it will not be used.

The most common data problems are acquiring the required information and in ascertaining the quality of the data. The "garbage in, garbage out" concept applies to the data used in a performance measurement system. If the data gathered are highly uncertain, then the conclusions drawn by converting those data into performance measures also will be highly uncertain and will have reduced value in managing the agency. For this reason, great care needs to be taken in data collection. Investments in accurate, high-quality data collection systems are essential to successful performance measurement and, by extension, to achieving the overall strategic goals of the agency. In reality, however, some things either cannot be measured accurately or cannot be measured accurately at an acceptable cost. Transportation agencies need to consider the uncertainty introduced by inaccurate data when taking action based on their system of performance measures (7).

References 3 and 8 discuss the concept of a "Performance Monitoring Plan" as a mechanism for collecting the data needed to quantify performance measures. Such a plan is essential for coordinating and allocating resources and for controlling the quality of the information that is used for evaluations. The monitoring plan specifies such things as:

• The data to be collected
• Frequency of data collection / schedule
• Data collection locations
• Data collection responsibilities
• Data analysis techniques and responsibilities
• Database management requirements
• Performance analysis reporting

Once the desired data are in hand, the focus shifts to the analysis and reporting of results. In this stage, the most challenging problem is often separating the impact of the activities of the transportation agency from the impacts generated from beyond those activities. For example, highway crashes are influenced by many factors besides highway design. If an agency uses the total number of highway crashes as a performance measure, does an increase in crashes indicate that the agency's safety programs are ineffective? Before that conclusion is drawn, the impact of changes in other causal factors (e.g., weather) clearly needs to be understood.

The necessity of separating the impacts of external factors has direct implications for data collection. Even though statistical techniques might be available to allow the impacts of several factors to be isolated, the techniques require large numbers of observations to be used reliably. Thus, it is necessary to have a data collection system that increases the number of observations by maintaining data with some degree of disaggregation in both time and space (7).

As noted in the overview section at the beginning of this chapter, the detection and surveillance subsystem of a Freeway Management System represents a potentially valuable data source for performance monitoring. Typically, the FMS generates massive amounts of data about the state of travel that are used by transportation authorities to effectively operate and manage their transportation systems, including traveler information. As a general rule, this information is collected and used in real time at a TMC to continually improve the operational performance of the system. The increasing deployment of FMS and the amount and variety of FMS-generated data throughout the nation offer great potential for longer-term transportation planning and performance monitoring. The same information collected at the TMC may also be used – but no longer in the context of real time applications – at the ITMS and agency tiers to identify deficiencies, and then to design and establish short term operational improvements such as incident response plans. These same data may also be applied at the state / regional tier, being incorporated into the transportation planning process for analyzing and evaluating alternative transportation improvements.

In order to monitor the long-term performance of the transportation network, the real time operations data collected by the FMS and /or ITMS must be systematically retained and reused – a process known as "data archiving" or data warehousing.

4.2.5.1 Data Archiving

The primary reasons for archiving FMS-generated data are:

• Provide more and better information for managing and operating the system — The first step in proactive management is knowing where problems are likely to occur before they actually do, then preventing or mitigating the impacts of those problems. Archived operations data can be used to predict when and where problems may occur again, as well as helping to evaluate alternative strategies for preventing or mitigating the problem.
• Maximize cost-effectiveness of data collection infrastructure — Data archiving permits transportation agencies to maximize their investments in data collection infrastructure by re-using the same data for numerous transportation planning, design, operations and research needs.
• Much less expensive than manual data collection — Data archiving is significantly less expensive than having a planning or design workgroup re-collect even a small percentage of the data using manual methods or special studies.
• Established business practice in other industries — The retention and analysis of operational data is an established practice in most competitive industries that use data to manage their business activities. (12).

Given that archived FMS-generated data can provide a valuable longer-term resource for a variety of stakeholders, the Archived Data User Service (ADUS) was incorporated into the National ITS Architecture in September 1999 to help realize the potential usefulness of ITS data. A U.S. Department of Transportation multi-agency, 5-year ITS Data Archiving Program Plan was developed based upon the vision of "improving transportation decisions through the archiving and sharing of ITS generated data."

Attempting to use data to meet information needs for which the data were not originally intended can be a challenging endeavor. In the context of ADUS, data issues are multi-faceted and complex, including data quality, format, integrity, compatibility, and consistency. Moreover, with ITS-generated data being so temporally extensive (e.g., collected every 30 seconds) but spatially limited (e.g., covering 30 miles of roads), ADUS data sometime need to be integrated with data from traditional sources in order to be useful.

The "Guidelines for Developing ITS Data Archiving Systems" (Reference 13) provides a number of basic principles that can be applied regardless of archive size or design, including:

• Determine the workgroup(s) or agency(ies) that should have primary responsibility for operating and maintaining the data archive. This may seem like a simple matter; in many cases, though, data archiving systems have not been further developed because no one has taken responsibility for their operation and maintenance.
• Discussion and dialogue in early stages among all stakeholders should assess the demand for archived data as well as the strengths and weaknesses of which agency or workgroup in a region maintains data archives. In some cases, there may be several agencies that each operate their own data archive, but which are connected and integrated through a "virtual data warehouse". In other cases, it may be logical for a regional planning agency with strong information management capabilities to warehouse data that can be shared among other agencies in the region. In any case, sharing data between agencies will be necessary, and will require some level of agreement on data definition and geographic units. (Refer to Chapter 16 on Regional Integration).
• Start small but think long-term, and begin with modest prototypes focused on a single source of data (e.g., freeway detector data).
• Develop the data archiving system in a way that permits ordinary users with typical desktop computers to access and analyze the data. Effective data archiving systems make large operations data archives available to ordinary computer users without requiring them to have specialized database or programming skills. These systems use a "point-and-click" interface, either through a Windows-based application or a web browser, to provide access to the data archives.
• Provide access to and distribution of archived data through the Internet or portable storage devices such as CDs or DVDs. Internet-based access and distribution of data are some of the most common and effective means to share archived data. CDs or DVDs are used as an alternative to Internet-based data archives, permitting the data archiving agency to maintain greater control and security over the data.
• Save original data as collected from the field for some specified period of time, but make summaries of this data available for most users. Many data archiving systems aggregate data to a consistent time interval (5 minutes is most common) for loading into a data archive. Because there will always be some users interested in the original data, a mechanism should be developed to store this for a short period of time or to store it permanently off-line.
• Use quality control methods to flag or remove suspect or erroneous data from the data archive. The rigor of the quality control ultimately depends upon how and for what purpose the data will be used. Two different philosophies exist for what to do with data that has failed quality control:
  • Simply identify or flag the data records that have failed quality control; or
  • Remove the data records that have failed quality control and replace with better estimates.
  These business rules (for how to deal with data failing quality control) will depend upon who will be using the data and for what purpose. There is no single correct answer for quality control.
• Provide adequate documentation on the data archive and the corresponding data collection system. With data archiving systems, many data users will be from outside the operations workgroup or agency that collected the data. Thus, they may have little knowledge about the operations data that is collected, how it is collected, and how it is processed by operations before it is archived. Adequate documentation for data archives primarily includes (but is not limited to) an "audit trail" of how the data have been processed since they were collected in the field (e.g., information about the results of quality control, any summarization or aggregation steps, and any estimates or changes that have been made to original, field-collected data), and information on the data collection system (e.g., the type, location, and other identification for detectors, the detectors that were considered "online" for a particular hour or day, and information about equipment calibration and maintenance).

4.2.5.2 Examples of Data Archiving

California PeMS Data Archiving

The Operations Division in Caltrans' Headquarters office has worked with researchers at the University of California at Berkeley in creating PeMS, a freeway Performance Measurement System. PeMS gathers raw freeway detector data in real-time from several of Caltrans' districts, including Los Angeles, Orange County, and Sacramento. The detector data for these participating districts are summarized and processed as follows:

• Aggregates 30-second flow and occupancy values into lane-by-lane, 5-minute values;
• Calculates the g-factor for each loop, and then the speed for each lane. (Most detectors in California are single loop, and only report flow and occupancy. PeMS adaptively estimates the g-factor for each loop and time interval.
• Aggregates lane-by-lane values of flow, occupancy, and speed across all lanes at each detector station. PeMS has flow, occupancy, and speed for each 5-minute interval for each detector station (one station typically serves the detectors in all the lanes at one location);
• Computes basic performance measures such as congestion delay, vehicle-miles traveled, vehicle-hours-traveled, and travel times.
• The data archives are then made available through the Internet for anyone that has access privileges (i.e., the site is password-protected).

PeMS has several applications and built-in data summary and reporting tools on the web site. One of these involves trip travel time estimates and shortest routes. A user can bring up the district freeway map on the Web browser, and select an origin and destination. PeMS displays 15 shortest routes, along with the estimates of the corresponding travel times. PeMS also provides travel time predictions – for example, what will be the travel time 30 minutes from now. The travel time prediction algorithm combines historical and real time data.

Another application, called "plots across space," can assist in identifying bottleneck locations for more detailed investigation. To use the application, the engineer selects a section of freeway, a time, and a performance variable such as speed, flow, or delay. PeMS returns a plot of the variable across space. Having quickly determined the existence of these bottlenecks, the engineer can go on to determine their cause, such as the location of interchanges, the highway geometry, large flows at ramps, etc, and propose potential solutions to alleviate the bottleneck. Furthermore, any scheme implemented to relieve a bottleneck can be rigorously evaluated by a thorough before-and-after comparison.

The impetus for this data archive was state legislation that required Caltrans to monitor the performance of their transportation system. Because Caltrans has extensive detector coverage on freeways in several districts, they chose to archive existing data rather than manually re-collect system performance data. Caltrans' PeMS data warehouse is unique because it is one of the few statewide operations data archives in existence. Time and experience will reveal how useful a centralized statewide data archive is to local agencies and workgroups at the district level.

Washington State DOT

WSDOT has been archiving freeway detector data since 1981 in some shape or form, although early efforts were difficult because of the expense of data storage and the difficulty of data transfer (pre-Internet). The agencies have made numerous improvements to their data archive over the years and, for the most part, the data archives have been institutionalized within WSDOT. Freeway detector data (i.e., vehicle volumes and lane occupancy by direction) are collected every 20-seconds from field controllers as part of the Seattle area freeway management system. The data are converted into estimates of vehicle speed and travel time, and summarized to the 5-minute level in the data archive. Quality control is also performed before the detector data is loaded into the archive, and the archive documents the number of data records that have failed quality control.

The Washington State DOT and the Washington State Transportation Center (TRAC) at the University of Washington have developed a CD-based data archive for the Seattle freeways, which they use to distribute the archived operations data. Each data archive CD contains data extraction and summary tools.

An analysis process developed by TRAC produces facility performance information based on these data. This process also fuses the basic freeway surveillance data with independently collected transit ridership and car occupancy data to estimate person throughput. The data are used for a wide variety of purposes, including answering key policy questions and evaluating operational improvements such as ramp metering or HOV lanes, freeway performance monitoring, pavement design, and freight performance analysis.

A paper by Mark Hallenbeck, Director of TRAC (Reference 14), summarizes the experience and lessons learned from this data archiving system as follows:

• "The good news is that ITS surveillance systems being built for traffic management purposes provide much of the data needed to perform these types of analyses; therefore, lots of "new" data are not necessary. Instead, the data already collected must be retained, analyzed, and reported."
• "Storing and analyzing the data are not free. However, a large number of potential users exist for the information that the surveillance system generates. The key is to work with potential users to fund the modest costs of storing, analyzing, and reporting the data already collected. The agency must also determine who will operate the database."
• "It is important to recognize that not all surveillance data are "good." Therefore, the analytical procedures must be able to identify and handle "unreliable" data. Mechanisms should also be in place to repair and calibrate unreliable sensors. (After all, unreliable data also hinder the operational control decisions that are based on those data.)"
• "Because most traffic management systems have limited equipment maintenance budgets, repair activities have to be prioritized. A key to consider when balancing cost versus data availability is that obtaining useful performance information does not require all detectors to be operating. (Does an agency really need to report volumes based on continuous data collection at 300 locations in the urban area, or will 12 to 20 sites spread strategically around the region reveal the important facts?) The reality is that necessary data can be obtained with a moderate amount of planning and cooperation."
• "When this cooperation occurs, it becomes truly possible to manage the roadway system. This is because an agency now has the data necessary to understand how the roads are actually performing and how that performance changes as a result of various management and operations activities."

4.2.5.3 Field Measurements / Manual Data Collection

As previously discussed, Freeway Management Systems (FMS) offer the potential to automate much of the data collection required for performance-based evaluations. That said, the reality is (as of the date of this writing) that less than one-third of the freeways in the nation's urban areas are instrumented with surveillance subsystems, the data collected by many of these systems does not include all the information required by outcome-based performance measures, detectors don't always function properly, and some information just cannot be collected without some sort of manual activity.

The "Manual of Transportation Engineering Studies" (Reference 15) is an updated and expanded version of the 4th edition to the Manual of Traffic Engineering Studies. It is designed to "aid transportation professionals and communities to study their transportation problems in a structured manual, following procedures accepted by the profession." The primary focus is on how to conduct "transportation engineering studies in the field". Each chapter introduces a type of study and describes the methods of data collection, the types of equipment used, the personnel and level of training needed, the amount of data required, the procedures to follow, and the techniques available to reduce and analyze the data. Applications of the collected data or information are discussed only briefly. Individual chapters include volume studies, spot speed studies, travel-time and delay studies, inventories, transportation planning data (e.g., origin – destination), traffic accident studies, traffic control device studies, roadway lighting, and goods movement studies. Additionally, there are appendices covering statistical analysis, written reports, and presentations. Another valuable reference is the "Travel Time Data Collection Handbook" (TTI, Report FHWA-PL-98-035, March 1998).

4.2.6 Reporting

As previously discussed, a good performance measuring program cannot help but improve communications with an agency's customer base and constituency, including decision makers and other agencies and entities that are involved with the operation and management of the surface transportation network. To achieve this improved communications, however, requires that the performance measure data be translated into reports for dissemination to stakeholders. Many of the criteria discussed for performance measures are directly applicable to performance reporting, including reporting results in stakeholder terms, that the information necessary to improve decision making is conveyed in these reports, and that the information is presented in a manner that is easy for the audience to understand and interpret.

Visual depictions of the data can assist users in understanding trends, operational performance, and the meaning of complex data interactions. As an example, the Washington State DOT and the Washington State Transportation Center (at University of Washington) convert their archived data (previously discussed in section 4.2.5.2) into a variety of presentation graphs – showing congestion problems, benefits from operational improvements, comparisons of alternatives, etc. – as a means of discussing freeway operations and the associated policy issues with managers and other decision makers. A few examples are shown and described below in terms of possible policy and operational questions (from References 14 and 16).

What does the congestion picture really look like?

This basic "volume-by-time-of-day" graphic can be extended to illustrate when congestion occurs and its effect on vehicle speed and throughput. Average speed is color coded to indicate how conditions routinely change by time of day. Then, because conditions vary considerably from day to day, reliability at this point in the roadway can be examined by defining "congestion" (in this case, the occurrence of LOS F conditions) and reporting on the frequency with which that congestion occurs. Graphically, it is possible to lay the "frequency of congestion" over the same graphic that illustrates vehicle volumes and average speeds. This is shown in Figure 4-1 (read "Vehicle Volume Per Lane" on the left axis, and "Frequency of Congestion" on the right axis.) This graphic shows that this specific location experiences LOS F conditions more than 80 percent of all weekdays (four times a week). It is also possible to see the slight decrease in vehicle throughput, caused by congestion, which occurs in the heart of the morning peak period.

Figure 4-1: Estimated Frequency of Congestion, Volumes and Speeds (Reference 14) D

Another approach is to produce an average daily corridor profile to depict lane-occupancy percentage at each location along a corridor for a specified direction of travel. As shown in Figure 4-2, the resulting graph is a contour map, color-coded according to the estimated congestion level.

Figure 4-2: "Temperature" Diagram of Traffic Flow Conditions (Reference 18) D

What delays are the public experiencing?

Using vehicle speed data that can be obtained from the freeway surveillance system, it is possible to estimate vehicle travel times throughout the day. Again, by saving these data, it is possible to describe not only today's travel times (excellent for measuring the effects of an incident), but also an entire year's travel times. Graphics like Figure 4-3 allow the analysis and reporting of travel conditions throughout the day.

Figure 4-3: Travel Times (by time of day) for a Specific Route (Reference 14) D

The graphic illustrates the actual travel times experienced (by time of day) for a specific route of interest (in this case the northbound trip using the southern half of the I-405 corridor). The green line represents the average travel time for a trip starting at a given time. The red line illustrates the 90th percentile trip. This is essentially the worst travel time a motorist could expect to experience once every two weeks. (As previously discussed, the Mobility Monitoring Program uses the "Buffer Index" as a measure of travel reliability. Changing the graphic to illustrate the 95th percentile trip time would represent the Buffer Index.)

Figure 4-3 also includes a measure of "congestion frequency." In this case, "congestion" is defined as the average speed for a trip of less than 35 mph. The blue histogram describes the frequency with which a motorist can expect to experience a trip that averages less than 35 mph for the entire trip duration.

Statistics such as the ones presented in the Figure 4-3, when tracked over time, allow freeway operations personnel to measure and present the broad, overall effects of the traffic control strategies they implement. These statistics also lead to more informed discussion of the travel conditions that exist (e.g., How bad is off-peak congestion? Is off-peak operation of the service patrol program necessary?), which in turn leads to more informed debate about the need for and relative merits of alternative operations strategies.

What improvements have ramp metering produced?

Any time significant operational changes are implemented within the surveillance area, the resulting changes in vehicle throughput and performance can be measured. WSDOT has operated ramp meters in the afternoon on SR 520 in Seattle for a number of years. Until recently, the ramp meters were not used in the morning. When morning metering was implemented, significant improvements in freeway performance occurred. Those improvements, illustrated in Figure 4-4, included an increase of over 170 vehicles per lane per hour and a decrease in the occurrence of LOS F conditions of one day per week. Ramp meters may not have "solved" the congestion problem; but they did make a considerable improvement.

Volume and Congestion on Eastbound SR-520 on the Viaduct

Figure 4-4: The Effect of Ramp Meters on Vehicle Volume (per lane) Throughput and Frequency of LOS F Operations (Reference 14) D

4.2.7 Emerging Trends and Needs

The use of performance measures – particularly those that measure "outcome" – for operating and managing the transportation network, and for longer-range planning and decision making, is itself an emerging trend. The same holds true for data archiving. A recent problem statement developed by the TRB Committee on Freeway Operations, entitled "Freeway Performance Monitoring, Evaluation, and Reporting", states: "a consensus does not exist and technical guidance has not been developed regarding the appropriate measures, methods, data requirements, evaluation tools, procedures, level of effort, and resources required to properly support the monitoring, evaluation, and reporting of freeway performance. Research and technical guidance is needed to provide direction and ensure that transportation professionals are effectively integrating the performance of freeways into the appropriate planning and decision making processes of agencies."

References 2, 3, 5, 6 and 10 address future issues and research needs, as summarized below:

• Gather examples, case studies, and tools to effectively communicate performance measures to policy makers, legislatures, and the public. Information is needed on how performance measurement is effectively communicated to decision makers to allow them to make informed decisions.
• Clarify (standardize) terminology and differences between organizational or managerial measures and system measures. Align the definition of goals across the industry to the extent possible, then standardize the measures used. Create consistent standards, so that performance measures can be reliably compared across agencies. Reporting standard errors or confidence intervals should be included.
• Develop training for managers and policy makers to apply and use performance measurement systems. Provide tools for managers and policy makers in applying and using performance measures.
• Gather information on how to incorporate community or society goals (or "soft" measures) into the performance measurement process. Create quality-of-life and sustainability performance indicators.
• Operational performance measures that address evacuations from man-made or natural disasters are needed, particularly for use during the operations of these events and tailoring strategies to maximize / optimize performance based on these measures.
• The maximum benefits will not be realized until considerable integration is achieved. Performance measurement can and should be the lingua franca for such integration, with mutually acceptable and well-defined outcomes acting almost like common denominators.
• With respect to archived data, significantly enlarge roadway sensor coverage (i.e., freeways and arterials) and experiment with data sources; transit operating data should be added to get a more complete system picture; encourage the local use of the archived data; improve the calibration and maintenance of data collection equipment; and add "event" databases (e.g., incidents, weather and work zone locations, which have significant impacts on roadway travel times).