Office of Operations
21st Century Operations Using 21st Century Technologies
Table of Contents

Measures of Effectiveness and Validation Guidance for Adaptive Signal Control Technologies

Appendix B. Tool Design and Development

The validation methodology requires collection and processing of the four basic sources of data discussed in the previous section. This information is then processed into MOEs. This requires development and implementation of software for this purpose. This software system is available via open source distribution. Figure 28 illustrates the focus of the project (highlighted in red) on the processing and computation of MOEs and provision of basic analysis tools. Except for GPS probe data, the project did not develop new data sources or formats for information from the field. In addition, the project did not focus on development of “dashboard” type visualization components or automated validation processes.

Figure 28. Diagram. Focus of the Tools Development Process.

Figure 28. This diagram shows five data sources on the bottom, flow detector data, Global Positioning System runs, vehicle Re-ID, Phase Timing Data, and Signal Detector Data, leading up to Processing Layer, which leads up to Measures of Effectiveness, which in turn leads up to Analysis tools. Measures of Effectiveness and Processing Layer are highlighted.

(Source: Kimley-Horn and Associates, Inc.)

The processing layer allows the user to ingest the data sources into the website database. The MOEs are then calculated from the tabulated inputs. The analysis tools then allow the user to sort through the resulting measures and apply filters to some data elements to facilitate comparisons between “before” and the “after” conditions, or “ON” and “OFF” states, or to compare the performance of a strategy by time of day or day of week. The analysis tools do not require before and after conditions to be specifically identified if the user desires only to validate performance of one type of operation.

Processing Components

The first components that are needed are the methods to import the data from each data source and store the data in the database.

Configuration Data

A configuration page on the web site allows an administrator to add intersections, vehicle re-identification readers, traffic counters, and routes. Each route is composed of intersections and links added to the map in sequence. The Google distance Application Programming Interface (API) is used to calculate distances between markers, so curvature in the road is taken into account even though links between each point are shown as straight lines. Link distance is auto-calculated. The coordinated phase for a link is configured in order to correlate phase timing data from detectors to compute percent arrivals on green. The free-flow speed or speed limit (see Figure 29) is configured to compute link and route travel delays.

An important element of route configuration is to place a start point before the first intersection and a stop point after the last intersection at a reasonable distance from the intersection. Runs should be started before the start point and stopped after the stopping point in order to capture the effects of queues before entry to the first intersection. Runs should be randomized by drivers to enter equally when the light is red or green at the first location.

Figure 29. Screen Shot. Link Configuration Interface.

Figure 29. A screen shot of the link configuration interface with a map of East Southern Avenue and South Power Road; and the configure region interface with a drop down menus for the region, the intersections route, and the bluetooth scanners route; and an interface to edit the link information with the coordinate phase, the speed limit, and the distance.

(Source: Kimley-Horn and Associates, Inc.)

Bluetooth detectors are added to the map and routes between one reader and another are defined by connecting the two. Traffic counters are added to the map and the direction of travel that they are counting is configured.

Detectors are added to intersections in order to calculate the Green-Occupancy Ratio (GOR) and percent arrivals on green by configuring their phase assignment, length, and distance upstream of the stop bar. This is illustrated in Figure 30. Refer to the previous report for details on the computation of GOR and percent arrivals on green.

Figure 30. Screen Shot. Detector Configuration Interface.

Figure 30. A screen shot of the detector configuration interface with a map of Superstition Freeway, South Power Road, and East Superstition Springs Boulevard and a place to edit the intersection information. The intersection information can be edited by name, upstream coordinated phase, downstream coordinated phase, and the detectors. The detectors can be edited by detector number, phase, length, distance from stop, and then Green-Occupancy Ratio and percent arrival can be selected.

(Source: Kimley-Horn and Associates, Inc.)

GPS Probe Data

GPS probe data are ingested from an android smartphone application. The user of the application enters a user name and a route number. Pressing a “start trip” button begins the data collection. The application can safely be used by a single driver using a phone cradle, which creates no more driver distraction than entering basic information into a typical route guidance device. GPS coordinates, speed, and heading are recorded on the device, and there is no driver interaction with the device during the trip. The user then presses a “stop trip” button at the end of the route, which discontinues the data collection and feeds the trip data to the web site automatically using FTP. The user can optionally enter the weather conditions and determine if there was an incident that occurred or was passed during the drive. User notes are stored with each trip to identify any other anomalies or anecdotal experiences during the trip. These interfaces are illustrated in Figure 31.

Figure 31. Screen Shot. Android Smartphone Application Screens.

Figure 31. A screen shot of two Android smartphone application screens. One shows configuration, with the users email, the route, and an option for the map to be on or off. The second shows end trip notes, with notes, an option for incident to be on or off, and a drop down menu for the weather.

(Source: Kimley-Horn and Associates, Inc.)

GPS trips are allocated to “before” or “after” time periods manually through database scripts or editing in order to provide flexibility.

High-Resolution Phase Timing and Detector Data

High-resolution signal timing and detector data are read from files that are stored by a traffic controller on a daily basis. Binary log files are stored on the controller and uploaded by a software script using FTP. The binary files are converted to CSV files by a converter tool (provided to the project by Indiana DOT). For the ASC/3 controller, each file stores 15 minutes of data with each detector on and off event and phase interval change events encoded in the file. The directory fills quickly on the controller and the controller overwrites the files the next day with new information for that 15 minute period, so the data are uploaded on a daily basis. Those files are stored in a shared directory and then imported to the validation system via a manual import process using the web page illustrated in Figure 32.

Figure 32. Screen Shot. Import Interface.

Figure 32. A screen shot of the import interface.

(Source: Kimley-Horn and Associates, Inc.)

At the time of import, the analyst is responsible for determining whether the data are from ASCT operation by checking the box for each file. When the upload begins, the analysis process computes the MOEs and stores the information cycle-by-cycle in the database.

Bluetooth Scanner Data

Bluetooth route travel time data will be read from files that are stored by a Bluetooth scanner server on a daily basis. These files will store route travel time on a frequent basis (e.g., 15 minutes) and the number of matches in that time period for each configured node pair. In order to keep the system simple, the import interface will be manually driven. The user will select the files they want to import and add them to a selection dialog. The user can import a single file for a single day or multiple files for multiple days. They will mark each file as belonging to the “ON” or “OFF” time period. ON and OFF periods will not be allowable in periods shorter than a 24-hour period.

Similar to the setup for routes and intersections, Bluetooth scanner nodes will be configured on a Google map interface. A node will be given a name and an ID number. Routes will be created between nodes by associating one node with another. The distance of the route will be calculated using the Google API. A different kind of pin icon will be used for Bluetooth scanner nodes to distinguish the node from intersections or tube counters.

Tube Counter Data

Tube counter data will be read from files that are stored by a tube counter and exported using a USB memory stick. These files will store counts on a frequent basis (e.g., five minutes) for each direction of travel at the tube counter placement location. In order to keep the system simple, the import interface will be manually driven. The user will select the files they want to import and add them to a selection dialog. The user can import a single file for a single day or multiple days of data in a single file. The analyst will mark each file as belonging to the “ON” or “OFF” time period. ON and OFF periods will not be allowable in periods shorter than a 24-hour period.

Similar to the setup for routes and intersections, tube counter locations will be configured on a Google map interface. A tube counter will be given a name and an ID number that matches the ID number of the device in the field. Directional mnemonics will be marked for each counter location (e.g., eastbound, westbound) in order to appropriately allocate the field data when the information is imported. A different kind of pin icon will be used for tube counter nodes to distinguish the node from intersections or Bluetooth scanners.

MOE calculations

After the data is imported, MOE calculations are made. Functional requirements for processing each source data type are described in the following sections.

Probe Travel Time Data

Each probe travel time run will be processed and the trip time and trip delay will be stored in the database for display in the trip selection table. The number of stops, the number of stops per mile, and the average speed metrics will be calculated and stored in the DB. GPS location data that is before the start time location of the trip and GPS location data that is after the end time location of the trip will be deleted for computation of the trip duration, and deleted from the database.

Bluetooth Scanner Data

Bluetooth route travel time data are read from files that are stored by a Bluetooth scanner server in XML format. In this project, a real-time feed was integrated with the system to obtain the files automatically from the Bluetooth vendor’s website. These files are read using a similar import process. These files store individual route travel times in order to compute the average, minimum, and maximum travel times for each time period. Similar to the phase timing data, at the time of import, the user selects the adaptive ON or OFF state for each file.

Tube Counter Data

Tube counter data are read from files that are stored by a tube counter and exported using a USB memory stick. Typical vendors of traffic count data collection format summary files in various ways, sometimes according to a state or local standard. The particular format (shown in Figure 33) is supported in the input process to ingest traffic counts in 15-minute bins. These files are read using a similar import process. Similar to the phase timing data, at the time of import, the user selects the adaptive ON or OFF state for each file.

Figure 33. Table. Sample Traffic Counter Data Format.

Figure 33. A table showing sample traffic counter data. Data is organized byt date and time in fifteen minute increments in a 24-hour period. Totals are shown for AM and PM periods and for each day, as well as the AM percent.

(Source: Kimley-Horn and Associates, Inc.)

Phase Timing and Detector Data

Cycle-by-cycle phase durations for each phase at each intersection and stored in a new database table. Similarly, detector data will be consolidated into cyclic format and stored in a new database table. GOR and V/C will be computed for each phase for each cycle. Arrivals on green and platoon ratio will be calculated using detectors assigned to this MOE for coordinated phases. The pattern number in effect at the intersection closest to the probe trip starting point will be applied to GPS probe trips. The pattern number at the beginning of the trip will be used if two patterns are in effect during a trip.

The remaining MOEs will be calculated when a comparison report is requested by the user:

  • Travel time reliability metrics.
  • Phase reliability metrics.
  • Statistical significance calculations.

Before and After Comparisons for Validation of Operational Objectives

Validation of an operational objective requires comparison of the performance data characterizing the performance of the systems before and after the application of new operational strategies. For this project the before condition is indicative of non-adaptive operation while the after condition is indicative of active ASCT operation. The before condition is used to establish a baseline operation against which the after, or ASCT active operation, can be compared. For each of the data sources, this is implemented as a table-driven selection system as shown in Figure 34.

Figure 34. Screen Shot. Report Selection Dialog for Before and After Comparison.

Figure 34. Is a screen shot showing the selection dialog for before and after comparison. The data displayed shows the date/time, day, group, user name, route, pattern and the travel time in 15-minute increments.

(Source: Kimley-Horn and Associates, Inc.)

The analyst can choose from date, day of week, time of day, route number, and pattern number to compare performance. In this example, route 6 runs are being compared for runs on Friday, December 7 (Adaptive ON) versus Saturday, December 8 (Adaptive OFF). Typing the requested route number in the search box below the route column removes all of the other routes from the resulting table so that only trips on route 6 are compared. Pressing “view” results in a table that compares the performance of Before and After conditions. Pressing “Export” will save the page as shown to an excel format for additional graphing or analysis.

A link-by-link performance report is then shown that compares the “before” with the “after” data for the metrics, as shown in Figure 35. The resulting average values for the MOEs and the comparison between before and after is shown for each link. In addition, for this particular report for probe travel runs, if corresponding percent arrivals on green and platoon ratio data are available from the controller for that link, it is also displayed in the row. This part of the results page can be printed or saved to CSV, XLS, or PDF formats. Similar comparison reports are available for phase timing data, route travel times from Bluetooth detectors, and traffic counters.

Figure 35. Screen Shot. Sample Link by Link Travel Time and Delay Comparison Table.

Figure 35. A screen shot of a Link by Link Travel Time and delay Comparison table. Data displayed shows the node number, length), node name, before/after/change, travel time, average speed, total delay, number of stops, number of stops per mile, percent arrival on green, and platoon ratio across the top.

(Source: Kimley-Horn and Associates, Inc.)

Reliability Estimation

Figure 36 illustrates the reliability measures that are computed for GPS probes and Bluetooth pairs. Average and standard deviation of travel time as well as buffer and planning indices are computed. Buffer time is computed from the 95th percentile, travel time when there are a minimum of 5 data points in a summary bin. If there are fewer than 5 measurements, the buffer time and planning times will be reported as “N/A”. If there are more than 5 measurements, they are ordered by duration and the observations closest to the 95th percentile are averaged to obtain the 95th percentile value. If a specific observation is the 95th percentile observation, then that value is used. For example, if there are exactly 20 measurements, then the 19th slowest travel time is the 95th percentile. If there are 10 measurements, then the average of the 9th and 10th slowest travel times is used as the 95th percentile travel time.

Figure 36. Screen Shot. Sample Travel Time Reliability Display.

Figure 36. A screen shot of the Sample Travel Time Reliability Display. Data displayed shows time, average travel time before, average travel time after, standard deviation travel time before, standard deviation travel time after, travel time buffer index before, travel time buffer index after, travel time planning index before, and travel time planning index after across the top.

(Source: Kimley-Horn and Associates, Inc.)

For variability estimation of phase data, the following metrics will be computed for each selected time period:

  • Maximum value.
  • Minimum value.
  • Standard deviation.

Maximum, minimum, and standard deviation will be computed for:

  • Percent arrivals on green, per link.
  • Platoon ratio, per link.
  • GOR per phase.
  • V/C ratio per phase.

Throughput Measurement

Two measures related to throughput estimation will be provided. The first is the vehicles processed by the “before” and “after” systems at each volume count point over a given time period. The analyst will select the time period, and the system will compute:

  • Average throughput by count location.
  • Maximum, minimum, and standard deviation by count location.
  • Maximum, minimum and standard deviation calculations will assume that multiple days of “before” and “after” conditions are available in the database.
  • The second measure of throughput is the time to process a given number of vehicles at a given count point. The analyst will specify a given total volume and a start time. The system will compute the time duration from the start time to process the given volume:
    • Average, maximum, minimum, and standard deviation of time to process given volume.
    • Maximum, minimum and standard deviation calculations will assume that multiple days of “before” and “after” conditions are available in the database.

Statistical Significance Computations

For all MOEs statistical significance computations need to be made and displayed to determine the strength of comparisons between before and after conditions for the purpose of validation. The statistical significance values will include display of whether or not the finding is statistically significant at a 95 percent confidence level and at what acceptance level the finding is significant. On a measure by measure basis, the comparison may need to use nonparametric statistical tests that are not based on assumptions of normality. In particular, since travel time has known lower bounds (assuming lawful driving) a travel time performance curve may be better estimated by using signed rank comparison tests rather than t-tests.

Summary

The MOE computation and validation tools are a combination of:

  • Manually-driven import processes;
  • Automated MOE calculations; and
  • Manually-driven reporting and analysis actions.

These functions are all implemented as a web-based system using .NET tools. These tools were developed to convert the basic data from the four sources into information for use in validation of operational objectives. Additional functionality was implemented using the Excel PivotTable and PivotChart features to provide graphical outputs in the subsequent sections. The next chapter discusses the field site and data collection process used for testing the validation system.