Validation

Overview

Following the stages of Testing & Discovery and immunization information system (IIS) Assessment, Validation is a summary step to acknowledge IIS and IIS vendor products that are progressing toward or achieving alignment with community-selected measures and tests.

What

A Validation designation is automatically achieved when an IIS or IIS vendor product meets the designated measures and tests in a specific content area (e.g., Transport), as specified in the Aggregate Analysis Reporting Tool (AART).

Measures for each content area of Validation will be drawn from published IIS Assessment measures and tests, where results are already available to individual IIS and IIS vendor products via AART. No new measures or tests will be introduced in the Validation stage that are not already measured and visible in the IIS Assessment stage.

All measures and tests throughout the Measurement and Improvement (M&I) Initiative are developed by the Measurement for Assessment and Certification Advisory Workgroup (MACAW), and vetted for comment with the full IIS community via a town hall meeting format prior to implementation.

Measures selected for Validation are reviewed and assigned a level based on the following definitions:

  • Basic Level: A level of Validation measurement that includes only essential measures to functionally meet the content area.
  • Complete Level: A level of Validation measurement that includes conformance to all measures approved for Validation.

Validation statuses at each level include:

  • Validated
  • Validated with Minor Differences
  • Major Differences
  • Not Measured
  • Not Participating 

 

Why

Participation in this process allows individual IIS and IIS vendors to support each other in alignment with national standards as testing results may differ between the same IIS software depending on application configuration settings, hosting, and production environmental factors.

Product Validation through M&I provides IIS vendors with an objective, independent testing process for their base product. The available data will enable individual IIS vendors to share their progress toward meeting standards with the broader community.

Using shared independent testing results to prioritize enhancements and collaborative work between individual IIS and IIS vendors will help prioritize essential improvements and community-wide adoption of standards across both IIS and electronic health record (EHR) systems. Over time, adoption of standards will reduce costs and resources needed for onboarding and maintaining interfaces. 

Measurement & Improvement Validation Priority Order

Not sure where to start? The graphic below suggests a prioritized path for IIS to better align with the M&I Initiative supported by the Centers for Disease Control and Prevention and AIRA. The full two-page document can be found in the AIRA repository. The links within the document for each step seen in the image below will bring up Validation Reports that display IIS results. None of these steps should represent stopping points in the process. If your IIS program cannot make progress on one step, simply move on to the next.

Log in to AART to find more detailed information specific to your IIS. 


Validation by Content Area

Transport

This content area examines the level of alignment with the formally defined transport specification, SOAP/Web Services and the CDC Web Services Definition Language (WSDL). When any two systems connect to exchange data, they must use an agreed upon transport layer to connect. Otherwise, communication and the exchange of data will be impossible. An expert panel made up of subject matter experts from across the IIS and stakeholder community selected the SOAP/Web Services and CDC WSDL as the IIS Standard in 2012.

Helpful resources for Transport Validation:


Message Submission (SUB) and Acknowledgment (ACK)

This content area focuses on Submission and Acknowledgment messages to an IIS. VXU, or vaccine update, is an HL7 message that contains demographic and vaccination data that is submitted to an IIS by an outside EHR. ACK, or acknowledgment, is an HL7 message used to relay information back to the EHR regarding the status of the VXU.

SUB/ACK Resources:


Message Query (QBP) and Response (RSP)

This content area focuses on Query by Parameter (QBP) HL7 message submissions that are sent to an IIS and Response (RSP) HL7 message submissions that are returned by an IIS in response to a querying system. The QBP is used to query data from the IIS and contains basic demographic data about a patient. The sending system wants to know if the IIS has any demographic and/or vaccination data on said patient, so it “asks” the IIS to respond to the QBP message. Once the IIS receives a QBP, the IIS attempts to match the demographic information contained in the QBP message with data in the IIS. It then sends back an appropriate response (RSP) HL7 message telling the sending system if the patient was found in the IIS and what information (if any) was found. 

QBP/RSP Resources:


Clinical Decision Support (CDS)

This content area focuses on Clinical Decision Support (CDS) tools designed to automatically determine the recommended immunizations needed when a patient presents for vaccination. These recommendations are developed by the Advisory Committee on Immunization Practices (ACIP).

The measures for this content area focus on three CDS concepts that can be returned via HL7 messages: Evaluation Status, Earliest Date, and Recommended Date. Each IIS is assessed on capability to return a concept and accuracy if the concept is returned based on age of the patient.

CDS Resources:


Data Quality Incoming/Ongoing (DQI)

This content area focuses on the IIS ability to detect data quality issues on a per-message basis. The IIS will be presented with messages that contain intentional data quality errors (e.g., vaccination date before date of birth) and the IIS is expected to detect these errors. The IIS will also be assessed on the ability to retain (and return) vaccination events from submission to query without changing the meaning (e.g., NDC mapped to CVX, return key information about a vaccination event).

DQI Resources: 


Additional Resources

Updated: 4/28/23