Model Systems Engineering Documents for Dynamic Message Sign (DMS) Systems

Chapter 5. Validation Plan Guidance

The model Validation Plan can be found in Appendix E. The chapters required for the Validation Plan document are:

1. Purpose of Document.
2. Scope of Project.
3. Referenced Documents.
4. Conducting Validation.
5. Validation Identification.

This document describes the activity of validation that the system being built meets the user needs and scenarios developed in the concept of operations. The validation documents will generally include three levels of validation documents:

• A plan to initially lay out the specific validation effort.
• The user’s/operator’s manual and/or a validation plan that defines the detailed operational procedures.
• A report on the results of the validation activities.

To ensure user needs and scenarios are validated by this activity, trace each need and scenario into a validation case, then into appropriate steps in the validation procedure.

A separate Validation Plan and procedures may be minimal for the simplest projects, especially where the system is commercially available and does not involve any custom software development, and where the agency staff have a very clear understanding of the purpose of the system. Preparation of a validation plan is strongly advised if:

• The system is more complex.
• There are several separate validation activities.
• Multiple deployment sites are involved.
• Multiple stakeholders have to be satisfied.

There is also the question of how comprehensive to make the validation effort. It is impossible to validate all possible combinations of actions under all possible operational situations. A good rule of thumb is: if it was important enough to write down as a need or scenario, then it should be validated, at least once. This may not, for example, validate all possible failure mode conditions or all possible incident scenarios. In-process² validation performed on the needs will help ensure that end-to-end validation of the system will meet the stakeholder needs.

Once the Validation Plan has been prepared, use this checklist to ensure all critical information has been included.

• Does the Validation Plan answer all the questions of who, what, where, and when?
• Does the Validation Plan make clear what needs to happen if an unexpected situation or a failure is encountered?
• Does the Validation Plan document the configuration of the hardware and software?
• Are all applicable needs and scenarios traced to a validation case?

Purpose of the Document (Chapter 1 of the Validation Plan Document)

This section identifies the type of validation activity to be performed within this Validation Plan. For instance, this activity may validate the entire system, a sub-system, the deployment at a site, a burn-in, or any other validation activity called for in the Program Plan or in the SEMP.

Scope of Project (Chapter 2 of the Validation Plan Document)

This section gives a brief description of the planned project and the purpose of the system to be built. Special emphasis is placed on the project’s user needs and issues that must be addressed and validated.

This section also describes the environment in which the project operates. It identifies the organization structures that encompass all stakeholders. It also gives a brief description of the role to be played by each stakeholder. This includes ad hoc and existing management work groups and multi-disciplinary technical teams that should be formed to support the project.

Referenced Document (Chapter 3 of the Validation Plan)

This is a list of all documents used in the preparation of this Validation Plan. This usually includes the Project Plan, (if one was written), and the applicable Requirements Documents. Reference to other documents, such as descriptions of external systems, standards, a Concept of Operations, and manuals may also be included.

Conducting Validation (Chapter 4 of the Validation Plan Document)

This section provides details on how validation is accomplished. It defines: who does the validation; when and where it is to be done; the responsibilities of each participant before, during, and after validation; the deployed hardware and software configuration; and the documents to be prepared as a record of the validation activity.

This section also defines how anomalies are to be handled (that is, what to do when an unexpected situation or a failure occurs during validation).

In general, the following information should be included in this section:

• A description of the participating organizations and personnel and identification of their roles and responsibilities. This may include for example, a validation conductor, validation recorder, operators, and/or engineering support.
• Identification of the location of the validation effort, that is, the place, or places, where the validation must be observed.
• The deployed hardware and software configuration for all of the validation cases, including hardware and software under validation and any supporting equipment, software, or external systems. Several configurations may be necessary.
• Identification of the documents to be prepared to support the validation, including Validation Procedures, a Validation Report and descriptions of special equipment and software.
• Details of the actual conduct of validation, including:
  • Notification of participants.
  • Emphasis on the management role of the validation conductor.
  • Procedures for approving last minute changes to the procedures.
  • The processes for handling a failure, including recording of critical information, determination of whether to stop the validation, restart, or skip a procedure, resolution of the cause of a failure (e.g. fix the software, reset the system, and/or change the requirements), and determination of the re-validation activities necessary as a result of the failure.

Validation Identification (Chapter 5 of the Validation Plan Document)

This section identifies the specific validation cases to be performed. A validation case is a logical grouping of functions and performance criteria (all from the Concept of Operations Document) that are to be validated together. For instance, a specific validation case may cover DMS System user permissions by the DMS System Manager. There may be several individual user needs that define this capability, and they all are validated in one case. The actual grouping of user needs into a case is arbitrary; however, the grouping is usually based on the grouping of user needs and the operational scenarios in the Concept of Operations. They should be related and easily combined into a reasonable set of procedure actions. Suggested validation cases that may be used in the Validation Plan document are included in the Validation Plan Sample Cases table (Appendix E).

Each case should contain at least the following information:

• A description name and a reference number.
• A description of the objective of the validation case, usually taken from the wording of the user need and/or scenario, to aid the reader understanding the scope of the case.
• A complete list of user needs and scenarios to be validated. For ease of tracing of user needs and scenarios into the Validation Plan and other documents, the user needs and scenarios are given numbers, so they can be accurately and conveniently referenced without repetition.
• Any data to be recorded or noted during validation, such as expected results of a step. Other data, such as a recording of a digital message sent to an external system, may be required to validate the performance of the system.
• A statement of the pass/fail criteria. Often this is just a statement that the system operates per the user need or scenario.
• A description of the validation configuration. That is a list of the hardware and software items needed for validation and how they should be connected (in most cases this is the deployed system configuration). Often, the same configuration is used for several validation cases.
• A list of any other important assumptions and constraints necessary to conduct the validation case.

Each validation case in Appendix E corresponds to the same name of a section of user needs in Section 4 of the ConOps model document. The applicable operational scenarios defined in section 8 of the ConOps are referenced in each validation case. The details of each validation case will need to be added as the system is further defined.