3. The Role of Testing in the Project Life Cycle

3.1 Overview

This chapter leads the reader through the complete TMS life cycle and explains the role of testing in system definition and acquisition. It begins by discussing TMS acquisition starting with the development of a regional architecture as outlined by the National ITS Architecture. It then goes on to address TMS procurement guidelines and practices (including contract types, contract type selection based on risk allocation, and procurement specifications), project management, documentation, and configuration management.

The section on the National Architecture was included because for most regions it is a starting point for identifying user needs and establishing the regional concept of operations that will provide the major input to the design of the TMS. It is likely that all of the TMSs within the region will start with the high-level requirements defined in the regional architecture. These will then be refined and developed into the specific TMS project concept of operations and high-level requirements. Therefore, it is important to understand the role that the national and regional architectures play when establishing a TMS deployment program.

3.2 National ITS Architecture

The following sections describe the underlying architecture used to build a TMS and how that architecture came to be.

The Intermodal Surface Transportation Efficiency Act of 1991 (ISTEA) initiated Federal funding for the National ITS program. The program at that time was largely concentrated on research and development and operational tests of emerging technologies for ITS programs. A key component of the program was the development of the National ITS Architecture.

The purpose of the National ITS Architecture was to provide a common structure for the design of ITS throughout the country. As this architecture has evolved, it has directed attention to defining the functions that could be performed to satisfy user requirements and how the various elements of ITS might connect to shared information. The National ITS Architecture is neither a system design nor a design concept; the purpose of preparing an ITS architecture is to ascertain:

• The functions that are required for the ITS.
• The physical entities or subsystems where these functions reside.
• The information flows that connect these functions and the physical subsystems together into an integrated system.

With the passage of the Transportation Equity Act for the 21st Century (TEA-21), ITS projects funded through the Highway Trust Fund were required to conform to the National ITS Architecture and applicable standards. Conformance with the National ITS Architecture was interpreted to mean using the National ITS Architecture to develop a regional ITS architecture and requiring all subsequent ITS projects adhere to that regional ITS architecture as well. The National ITS Architecture is to be used as a resource in the development of the regional ITS architecture, and the subsequent regional ITS architecture is to be on a scale commensurate with the scope of ITS investment within the region.

This legislation imposed the following requirements:

• All ITS projects funded by the Highway Trust Fund shall be based on a systems engineering analysis on a scale commensurate with the project scope.
• Compliance with the regional ITS architecture must be in accordance with United States Department of Transportation (USDOT) oversight and Federal-aid procedures, similar to non-ITS projects.
• ITS projects funded by the Highway Trust Fund and the Mass Transit Account must conform to a regional architecture.
• Projects must use USDOT-adopted⁴ ITS Standards as appropriate.
• Regions currently implementing ITS projects must have a regional architecture in place in 4 years, while regions not currently implementing ITS projects must develop a regional ITS architecture within 4 years from the date the first ITS project advances to the final design.
• Major ITS projects should move forward based on a project-level architecture that is coordinated with the development of the regional ITS architecture.

The concept of operations document should identify the roles and responsibilities of participating agencies and stakeholders for the regional ITS architecture developed for the TMS.

3.2.1 National ITS Architecture Description

The National ITS Architecture consists of logical and physical components as depicted in figure 3-1 below and described in the following paragraphs.

Figure 3-1. National ITS Architecture Components

3.2.2 Logical Architecture

While the user services identified the ITS needs of the agency's region, the logical architecture defines the functions that need to be carried out to support the selected user services. The logical architecture covers all of the user service requirements and is technology independent. It identifies the boundaries of the architecture, the functions to be performed, and the relationship between functions. Entities (e.g., businesses, organizations, vehicles, devices, systems) outside the ITS boundaries are referred to as terminators. What the logical architecture does not define is where the functions are performed and how they are implemented.

3.2.2.1. Process Specifications

Processes are the functions identified in the logical architecture that are required to support the agency's selected user services. The logical architecture presents processes in a top-down fashion beginning with the general processes that are decomposed into more detailed processes. The general processes are defined in terms of more detailed processes using data flow diagrams.

3.2.2.2. Data Flow Diagrams

Data flow diagrams show the information that is transferred between processes or between a process and a terminator in the logical architecture.

3.2.3 Physical Architecture

While the logical architecture provides a "functional" view of the system, the physical architecture provides a representation of how ITS should provide the required functionality. Although not a detailed design, the physical architecture takes the processes previously identified in the logical architecture and assigns them to physical entities (subsystems or terminators). Furthermore, the data flows from the logical architecture that originate at one subsystem and end at another are grouped together in physical architecture flows.

3.2.3.1. Subsystems

Subsystems are individual pieces of ITS defined by the National ITS Architecture. They are the principle structural element of the physical architecture view. Subsystems are grouped into four classes: Centers, Field, Vehicles, and Travelers (see figure 3-2). Example subsystems are the Traffic Management Subsystem, the Vehicle Subsystem, and the Roadway Subsystem. These correspond to the physical world of traffic operations centers, automobiles, and roadside signal controllers.

3.2.3.2. Architectural Flows

Architectural Flows represent the information that is exchanged between subsystems and terminators in the physical architecture. These architecture flows and their communication requirements define the interfaces that form the basis for much of the ongoing standards work in the national ITS program.

3.2.3.3. Equipment Packages

Equipment packages are the building blocks of the physical architecture subsystems. They partition the subsystems into deployment-sized pieces.

3.2.3.4. Architecture Interconnect Diagram

Figure 3-2 shows the top-level architecture interconnect diagram, which depicts the subsystems for full representation of the National ITS Architecture and the basic communication channels between these subsystems.

Figure 3-2. National ITS Architecture Interconnect Diagram

3.2.4 User Service Requirements

User service requirements are a specific functional requirements statement of what must be done to support the ITS user services. The user service requirements were developed specifically to serve as a requirements baseline to drive the National Architecture development. The user service requirements are not to be construed as mandates to system/architecture implementers, but rather are directions to the National Architecture Team.

3.2.5 User Services

User services document what the ITS should do from the user's perspective and include a broad range of users, including the traveling public as well as many different types of system operators. The concept of user services allows system or project definition to begin by establishing the high-level services that will be provided to address identified problems and needs.

3.2.6 ITS Standards

Standards are an important aspect of the National ITS Architecture. They provide the means by which compatibility between systems can be achieved. They also promote multi-vendor interoperability and ease of integration (if properly specified). As part of the National ITS Architecture development process, USDOT initiated an effort to accelerate development of consensus-based standards using the interconnection requirements (i.e., interfaces) defined in the architecture.

The specific intent of these standards development efforts is to provide a number of long-term benefits, including interoperability, increased competition, expandability, lower costs, and increased system integration.

Agencies can take advantage of these standards as they emerge by specifying their use in procurement packages. The standards also provide many options and require specific implementation choices when used with each TMS project. The applicable options will vary according to the operational concepts incorporated into the design and must be directly specified. Each selected option will have specific impacts on the testing process later in the procurement process. Many projects have not met their expectations because individual project specifications incorporated the standards by reference and did not identify the individual options necessary to fulfill the operational intent of the system. An appropriate level of planning and engineering activities must be performed in applying the standards to the acquisition of TMS projects.

Among the pertinent national ITS standards development activities in process are the suite of standards being developed under the National Transportation Communications for ITS Protocol (NTCIP) effort. There are also a number of other existing communication and information-based standards that are applicable to ITS projects.

The following list some of the applicable ITS standards with respect to data elements, message sets, and communication protocols.

• Data elements:
  • Traffic Management Data Dictionary (TMDD) - reference the Institute of Transportation Engineers (www.ite.org/TMDD).
  • NTCIP standards for the communications with the various ITS devices such as traffic controllers, dynamic message signs, environmental monitoring stations, CCTV cameras and switches, ramp controllers, etc. (reference www.ntcip.org) and refer to document NTCIP 9001 for a guide to these standards.
  • Advanced Traveler Information System (ATIS) data dictionary (SAE J2354).
• Message sets:
  • Transit Communication Interface Profile (TCIP) (APTA).
  • Message Sets For External Traffic Management Center Communications (MS/ETMCC - now part of the ITE TMDD effort).
  • Incident Management Message sets (IEEE-1512).
  • Advanced Traveler Information System (ATIS) message sets (SAE J2354).
• Communication protocol:
  • National Transportation Communications for Intelligent Transportation Systems (ITS) Protocol (NTCIP). These protocols identify center-to-center and center-to-field communications protocols for a variety of media (e.g., analog telephone, Ethernet).

In addition to the ITS-specific standards involving communications data, messages, and protocols, existing and emerging standards are available for many ITS devices and/or their interfaces (e.g., signal controllers, video cameras, dynamic message signs, etc.). These standards can be reviewed and considered for incorporation in an ITS project based on the engineering analyses. A list of references in the last chapter provides access to many of the standards organizations discussed in this section.

3.3 Transportation Management Systems Procurement

Once a regional architecture and at least a preliminary concept of operations have been developed, system procurement and funding issues can be addressed. The following are some typical contract types that can be used;⁵ each necessitates a different level of direct participation and technical expertise by the acquiring agency. The type of contract selected will also dictate who prepares procurement specifications and the management structure needed to oversee the process. It will also have an impact on who develops and performs the testing program throughout the life cycle of the project.

3.3.1 Contract Types

The following table describes some of the different contract types that can be used to acquire a TMS and its subsequent ITS devices and explains what the agency's responsibilities would be as a consequence of using each type.

Table 3-1. System Acquisition Contract Types and Agency Responsibilities

Contract Type	Description	Aquiring Agency Responsibility
Design	A qualified design engineering contractor is engaged to develop the detailed design that includes hardware and software requirements, procurement specifications, installation drawings, and integration and test plans and procedures. The designer is usually required to provide support during the construction (implementation) phase to correct design deficiencies and provides oversight and design support for changes.	The acquiring agency is required to manage the design contract and review and approve the design documents, procurement specifications, installation drawings, and the test plans and procedures. The resulting system will depend on the design engineer's understanding of the agency's needs and requirements.
Build	A separate, qualified implementation contractor is needed to verify and implement the design and installation drawings, develop or purchase the hardware and software components specified by the design contractor, and install and test those components. The implementation contractor may also be required to develop training plans and conduct training for system operations and maintenance personnel.	The acquiring agency is required to manage the construction contract, inspect and accept installations, approve changes and training plans, and witness and approve acceptance testing.
Design/Build	A single qualified contractor is engaged to create the design and then implement it.	The acquiring agency is required to manage the design/build contract; review and approve the design documents, procurement specifications, installation drawings, training plans, test plans and procedures; inspect and accept installations; approve changes; and witness and approve acceptance testing.
System Integrator	A separate qualified integration contractor is engaged to oversee the design and implementation phases and is usually responsible for reviewing and recommending approval of the design documents, procurement specifications, installation drawings, training plans, test plans and procedures; inspecting and accepting installations; conducting system level operations and maintenance training; and conducting and supporting testing. System integrators are contracted for design and implementation services under either separate agreements or a single agreement; however, some service elements of the system integration and test phase may be carved out for subcontractor integration contracts.	The acquiring agency is required to manage the lead system integrator contract, approve documentation and changes, and witness and approve acceptance testing.
System Manager	A qualified contractor is engaged to direct and manage all phases and aspects of the system development and implementation. The system manager designs and implements the system or contracts and manages the contracts for those elements of the design/build that the system manager does not perform. The system manager may also be responsible for the ongoing operations and maintenance of the system after it is accepted.	The acquiring agency is required to approve procurement practices and any major contracts let by the system manager. The acquiring agency is also required to oversee the system manager's contract, participate in the approval of documentation and changes, and witness and approve acceptance testing. The operating agency may desire to provide staffing for various operations and maintenance positions. How well the system meets the needs of the agency will depend on the relationship and understanding between the manager and the agency.

3.3.2 Contract Type Selection

Each of the above contract types exposes the acquiring agency to a different level of technical and financial risk and each carries with it a different test program burden on the agency (i.e., direct participation in as well as review and approval of designs, test plans, and procedures). The following table contrasts the contract types and relative financial and technical risk allocation against the agency's direct involvement in the test program for a multimillion-dollar TMS project.

Table 3-2 Contract Types: Agency Risk Allocation and Test Program Burden

Contract Type	Financial Risk	Technical Risk	Test Program Burden
Design	Low	High	High
Build	Medium	Medium High	High
Design/Build	Medium	Medium	Medium
System Integrator	Medium High	Low	Low
System Manager	High	Very Low	Very Low
Notes: A low financial risk implies a small contract amount with respect to the total TMS cost (sum of procurement contracts – labor and materials plus agency contract management plus agency direct labor). A high technical risk implies that the agency is dependent on its in-house technical expertise to oversee the technical aspects of the hardware and software design and implementation. A medium test burden level indicates that the agency has sufficient technical expertise and manpower to share the burden of the test program equally with the contractor(s).

For example, if the acquiring agency wants to use a contract type that has low technical risk (the design and implementation technical risk is borne by the contractor) and does not want very much agency participation in the test program (because of its limited technical expertise and/or ability to travel), it might select a system integrator contract type. While potentially more expensive than a design/build contract type, it would be used if the agency was willing to pay the contractor to accept most of the risk in implementing a successful TMS project. That technical risk and test program burden could be reduced even more with a system manager type contract, but at a potentially higher contract cost.

Another aspect of contract type to consider before committing to procurement is how the TMS will be deployed; i.e., all at once or incrementally, in phases by extending capabilities, or in stages with increasing levels of functionality. Generally, the availability of funding will dictate the deployment approach, but it should be noted that the overall test program costs will be greater for an incrementally deployed system because, as each phase or stage is completed, additional test planning, test procedures, and regression testing will be necessary. However, the benefits that will ultimately accrue to the public using that system will begin much earlier in the project.

Similarly, the impact of existing systems and their age should be considered when selecting the contract type. Rolling out a TMS where none previously existed has fewer risks associated with it than the replacement or even incorporation of existing TMS components. The responsibility for operating existing TMS components and integrating existing components should be weighed carefully as costs will vary with the assignment of risks and responsibilities. Testing existing facilities may be necessary to ensure that the facilities can be successfully integrated with the new system and to identify any pre-existing deficiencies that would leave the existing equipment performing below expectations with the new system.⁶

When the TMS is being incrementally deployed, the procurement specifications should identify certain phases or stages at which substantial portions of the system are acceptance tested in order to allow those portions to be used operationally. For example, if the system is being deployed along a transportation corridor that extends for many miles, it may be prudent to break the procurement into manageable geographic phases such that each phase covers a contiguous segment of the total system corridor and can be procured and accepted separately. This allows concurrent phases, when funding permits, and different developers, vendors, or contractors to be used in each phase if desired. If these procurements are properly structured, i.e., the first phase contains a functional TMC and the initial communications infrastructure to connect field devices along part of the corridor, then meaningful traffic management operations can start following acceptance of the first phase while development, installation, and testing activities are ongoing in subsequent phases. One important consideration for this approach, however, is the need to test the TMC system and subsystem capacity for the fully configured build-out; for example, if the final configuration will include 100 dynamic message signs (DMS) and 2,000 controllers, even though the initial phase has only 10 DMS and 100 controllers, the testing program for the TMC systems must demonstrate management capability (and communications support) for the fully configured system.

3.3.3 Procurement Specifications

The procurement specifications, whether developed by the acquiring agency or under one of the above contract types, must contain all the system requirements and acceptance test requirements. Poor or missing requirements will result in inadequate designs; missing capabilities and functionality; and performance, operations, and maintenance shortfalls. Furthermore, correcting these deficiencies will impact both costs and schedule; therefore, it is important that the procurement specifications include detailed, unambiguous, precise, and measurable requirements and that those requirements are written such that the components can be tested, inspected, or observed to verify compliance with the requirements. Without a strong procurement document and well-written requirements, there is little basis for the test program and ultimate system acceptance.

Testing is about verifying that what the developer or the vendor has delivered complies with what was specified. As such, testing verifies that the product meets the functional, performance, design, and implementation requirements identified in the procurement specifications. The testing program should require that the developer or vendor demonstrate that their product(s) meet all of the requirements of the procurement specifications, including any standards referenced.⁷ However, it is important to keep in mind that the degree to which a product/device is tested should be balanced against the cost of performing that testing and the perceived risk of the product to the success of the program, the complexity and maturity of product, and the number of units involved.

The procurement specifications should outline how the testing program fits into the overall project schedule. The schedule must allow sufficient time for the agency to review the test plan(s) and for the developer or vendor to make corrections. The testing can be complex and lengthy; a detailed test plan can easily run into hundreds of pages with the inclusion of the test environment schematics, descriptions of the test software and simulation environment, and copies of the test cases, inspections, and requirements traceability check lists. The acquiring agency will most likely need technical expertise (e.g., software, mechanical, and electrical engineers and operations and maintenance personnel) to review test plan submittals. It is recommended that prospective bidders for the project be required to submit example test procedures, at the level of detail being proposed, so that they can be evaluated for adequacy and completeness during the procurement (qualification) process. This will also afford the acquiring agency an opportunity revise final procurement specifications and to ascertain the competence of its technical staff to review and approve test plan material.

Re-use of existing procurement specifications from other related or similar projects and selection of standard products from a Qualified Products List (QPL) can greatly reduce system acquisition costs. However, this does not eliminate the need for a comprehensive review of the detailed procurement requirements, referenced standards, acceptance test procedures, and the modification thereof to assure the needs of this specific project are being met.

3.4 Project Management, Staffing and Training

The project management and staffing structure will be dictated by the scale and complexity of the proposed TMS and its acquisition and development strategy. Specifically, the development of the procurement specification(s) and selection and management of the contract type(s) for system development and implementation will determine the staffing and training needs. Additionally, future staffing needs will be determined by what contracting steps are taken to support operation and/or maintenance of the system after it is implemented. The development and implementation phases will extend for many months, if not years, and will require a large and continuing commitment of personnel and resources.

It is important to recognize, early in the process, the critical role that testing plays in all aspects of a successful project. An independent test organization (i.e., from the development and implementation organizations), managed and staffed by knowledgeable and well-qualified engineering and technical personnel, is a must. The size, technical diversity, and depth of the testing organization should be in direct proportion to the project's scale, technology, and complexity, and the contract type(s) used for procurement.

It is unlikely at the outset of the project that the agency's current staff will have sufficient technical training for all aspects of the project. This is particularly true with respect to new and emerging technologies and test techniques that may be employed on the project. Thus, staff augmentation and on-the-job training provided by vendors and consultants may be necessary and should be accounted for in the project budgeting and procurement specifications.

A large project may have both an acquiring agency that is responsible for the acquisition of the TMS and an operations agency that will operate and maintain the system once it is accepted and becomes operational. An ITS consultant that has technical expertise and practical experience in all aspects of TMS development and operations may also be employed to assist the acquiring agency and the operation agency during the developmental and operational phase of the project.

3.5 Documentation

A set of formal documentation is essential for the design, development, implementation, test, operation, maintenance, and administration of the TMS throughout the system's life cycle. The concept of operations is one of the baseline documents used to provide a general overview of the operational objectives and the management and support organizational structure envisioned for the operation of the system. This plan, along with a long-range strategic plan, provides the basis for developing critical programmatic documents, including the system architecture description, system specification, system test plan, and configuration management plan as well as the hardware/software design and requirements documents, test procedures, and training and maintenance plans. Figure 3-3 depicts an example document tree (usually included in the system specification) showing the hierarchy of these documents and how they are derived from each other. (Note: additional detail for some of documents in the document tree can be found in this document in the sections indicated above them in red in figure 3-3.)

Figure 3-3. System Document Tree

3.6 Configuration Management

The objective of configuration management (CM) is change control. Change is inevitable and the rule of thumb is that things change at the rate of roughly 1 percent per month. The goal of any project is to manage change, understand its implications, and integrate the changes into the project plan and testing program. CM provides formal change control methodologies throughout all stages of system deployment to include development, implementation, installation, test and acceptance. CM provides a mechanism for maintaining formal records of the TMS configuration to include software, communications cable plant, field equipment, communication electronics, TMS hardware, and all formal system documentation including test plans, test procedures and test reports. CM also provides a mechanism for managing changes during the system's life cycle and enables the acquiring and operating agencies to effectively and efficiently plan future system enhancements and expansions.

Generally, system documents such as system test plans, system and subsystem requirements and specifications, data dictionaries, equipment installation plans, and as-built drawings are placed under CM when the final version of the contract deliverable is approved by the acquiring agency. This establishes a baseline system configuration of software, hardware, and communications infrastructure. Any changes (including those made to correct problems or deficiencies) to the baseline system configuration, in terms of software changes, new hardware components, and/or communication electronics, can be tracked in order to minimize adversely affecting the overall functionality of the system in the future. Accordingly, a formal change control process is placed under the oversight of a configuration control board (CCB). Additionally, change request forms called system problem/change requests (SPCR) are used to initiate and process changes to the system configuration and documentation.

The CCB is responsible for the configuration management and change control decisions over the life cycle of the system. It acts under the auspices and at the direction of the acquiring and operations agencies. Its decisions may have significant programmatic cost and schedule impacts. It is the function of the CCB to accept configuration items (CI) for configuration management and control and to approve or reject all requested changes to CIs under control. It is the responsibility of the CCB to ensure that the appropriate level of testing is required and completed for all approved changes (i.e., test plans must be updated based on changes approved by the CCB).

In order to perform these functions, the CCB is comprised of a team of permanent members drawn from the day-to-day system managers, operations personnel, and invited technical specialists and advisors, each with specific responsibilities and duties, who collectively manage the system configuration and any changes made to it.

The following diagram depicts a nominal structure and membership of the CCB. The board positions shown in figure 3-4 are the permanent positions. Board positions can and should be expanded as necessary to handle unique circumstances or special needs. For example, the hardware and software board positions can be augmented during system development and/or expansion phases to include the area managers responsible for the development or expansion. ITS operations engineers and planners can also serve as board members. Technical specialists, advisors, and other agency representatives may be appointed to the board to assure comprehensive oversight of proposed changes, especially when the change potentially impacts other operations or systems. The SPCR originator or CI specialist may also be invited to attend a CCB meeting to provide additional expertise for technical discussion of problems or requested change.

Figure 3-4. Configuration Control Board (CCB)

Test program documents (e.g., program plan, test plans, procedures, reports) are all CIs managed by the board. As part of the CM process, the test team representative may be required to report test results to the CCB in order to move changes to software and hardware CIs into production environments.

3.7 Summary

This chapter provided a brief overview of the National ITS Architecture and how it relates to the procurement and development of an ITS project. Next, some of the procurement avenues (contract types) available to the acquiring agency, the financial and technical risks associated with each, and the resulting testing burdens on the agency were presented. In the context of system acquisition and development, recommendations were made for project management, staffing and training concepts needed for a successful project including the use of an independent test organization. The need for staff augmentation and on-the-job training provided by vendors and consultants at the outset of the project was suggested as a means of overcoming a shortfall in agency's current staff and technical training and that this be accounted for in the project budgeting and procurement specifications. The need for formal documentation was stressed as essential for the design, development, implementation, test, operation, maintenance, and administration of the TMS throughout the system's life cycle. The chapter concluded with a discussion of the importance of creating a configuration control board (CCB), which is responsible for both the configuration management and change control decisions that may have significant programmatic cost and schedule impacts as well as for ensuring the appropriate level of testing is required and completed for all approved changes.

⁴ This phrase has led to significant confusion. The USDOT must invoke a rule-making program to formally adopt ITS standards which become mandatory under this clause. To date, the USDOT has not undertaken rule-making procedures for the ATC, 2070, NTCIP, TMDD, 1512, or J2354 standards and hence their use is optional, although strongly encouraged. Adherence to the evolving national standards is critical for the piece-wise deployment of ITS technology and systems within a region and hence adherence to these standards is essential.

⁵ Additional contracting guidance and other contract types can be found in NCHRP Report 560: Guide to Contracting ITS Projects, by K. Marshall and P. Tarnoff, published by the Transportation Research Board, Washington, DC: 2006.

⁶ How accurate is the documentation for the existing system? Was the external interface previously tested or was it simply a requirement that went untested due to a lack of resources and urgency? In latter case, who assumes the risk when the interface does not meet its documented characteristics?

⁷ Examples of standards includes NEMA TS2-2003 for traffic controllers, NEMA TS4 for DMS, California TEES for traffic control devices, UL requirements, and the National Electric Code.

Previous | Next