8. System-Level Testing

8.1 Overview

This chapter discusses system-level testing on the installed, operational (production) system. Testing at this level is typically conducted to verify both completed subsystems and the system as a whole from the software developer, vendor(s), and systems integration contractor under the terms of the contract.

Before attempting system-level testing, all unit, installation, and hardware/software integration testing should be complete. Any problems identified at these levels should have been corrected and re-tested. It is also possible that the agency has decided to accept certain changes; under these circumstances, it is important that the system requirements be changed and documented and that the revised requirements serve as the basis for the final systems testing.

If the TMS is being incrementally deployed (see section 3.3.2), then the system-level test planning and procedures must be developed such that each increment can be separately tested and accepted. Under these circumstances significant regression testing may also be required to ensure that the incremental functionality or geographical extension does not compromise the operation of the system.

8.2 Subsystem Testing

Subsystem verification testing is performed as a prelude to system testing. It is performed in the operational environment using installed system hardware and software. Testing at the subsystem level should be performed:

1. When different developers, vendors, or contractors have been responsible for delivering stand-alone subsystems.
2. When the complete functionality of a subsystem could not be tested at a lower level because it had not been fully integrated with the necessary communication infrastructure.
3. When it was previously impossible to connect to the field devices for the testing phase.

Testing at the subsystem level has distinct benefits over delaying that testing to the higher level system testing:

• The test procedures and test personnel can concentrate on a limited set of system requirements and functionality.
• Problems encountered during the test can be resolved independent of other testing.
• Testing can be completed in a shorter time span and with fewer resources and disruption to other operations.
• Acceptance can be incrementally achieved and vendors paid for completed work.

Note that conditional acceptance for subsystems that have lengthy burn-in periods or specific operational performance requirements may be granted by the acquiring agency in order to allow progress or partial payments to be made prior to final acceptance.

8.3 Systems Testing

System verification testing is the highest test level; it is also usually the one with the fewest requirements remaining to be verified. Only those requirements relating to subsystem interactions, quantity of field devices, external interfaces, and system performance should remain to be formally verified. System acceptance testing is performed after all lower level testing has been successfully completed. It is performed in the operational environment using all available and previously installed and tested system hardware and software.

The system acceptance test should include an end-to-end or operational readiness test of sufficient duration to verify all operational aspects and functionality under actual operating conditions. While this may not be possible in a reasonable period of time,³⁶ the system test plan and test procedures should specify which requirements must be tested and which are optional given that those operational circumstances occur during the test period. The test should be conducted using trained agency or contractor staff that will manage, operate, and maintain the system following acceptance. One aspect of the readiness test should be to assess the sufficiency of staff training and operations manuals such that any deficiencies can be corrected before the agency assumes full ownership and all operational and maintenance responsibilities. This latter point is important for the agency to consider; all too often, the final system documentation such as user manuals are pushed until the end of the project. Because everyone is anxious to start using the system, preliminary documentation or minimal documentation is provided and the agency moves forward with the testing because "things" seem to work properly. It is recommended that the agency resist this temptation and evaluate the documentation as part of the final system acceptance test. If the contractor is aware of this situation in advance, it is more likely that they will complete the documentation for review much sooner.

Note that it may be necessary (due to contractual relationships) to move into overall system testing even though certain elements of the system may not be in place or operational. Examples may include outlying ITS devices or interfaces to other systems that are not yet available. Under these circumstances, efforts should be made to include simulators or demonstration devices to allow the full system testing to move forward. This approach brings some risk because the simulators or demonstration devices (and their interfaces) may differ from the final implementation, but it is often the only way that a project phase can be "closed out." Under these circumstances, it should be recognized that the introduction of such simulators and demonstration devices may increase the cost of the contractor's activities. Further, the agency has a responsibility to test and evaluate the simulators to ensure that they are representative of the actual devices.

8.4 Summary

This chapter has provided a brief discussion of what system-level testing is and what it should accomplish. It represents the final step in the TMS acquisition process.

It is very important that the procurement document clearly and unambiguously describe what constitutes final acceptance. Procurement documents frequently require the conduct of an acceptance test followed by an observation period. The contractor's expectation is that title to the system transfers at the completion of the test, but the agency intended the transfer to occur after completion of the observation period. The transfer can occur any way the agency wants, but it should be uniformly understood by all parties. Formal acceptance at the subsystem or system level may also trigger the start of equipment warranty periods, software licensing agreements, operations and maintenance agreements, etc. The procurement documents should clearly specify which of these are applicable, when they become effective and need to be renewed, and what the payment schedules and provisions are.

³⁶ This limitation is likely to be because during the planned test period a specific set of circumstances (blizzard) may be unlikely to occur.

Previous | Next