A Review of the Smallpox Vaccine Adverse Events Active Surveillance System

Abstract

In response to concern about smallpox possibly being used as a biological weapon, the President of the United States launched the National Smallpox Pre-Event Vaccination Program on 13 December 2002. Given safety concerns, identifying potentially serious adverse events (SAEs) was an essential tool of the program. To monitor for SAEs, both enhanced passive surveillance and active surveillance systems were used. The enhanced passive system was built, in part, on the existing Vaccine Adverse Event Reporting System; the active system was implemented 24 January 2003. During January 2003–May 2005, the active system detected only 1 SAE in addition to those reported through the enhanced passive system. Furthermore, the active system was not universally used by states. With the enhancements to passive surveillance, the performance of enhanced passive surveillance was comparable to that of active surveillance. However, an active surveillance system may be important when there is no enhanced passive surveillance system available.

The 11 September 2001 terrorist attacks in the United States heightened concerns over bioterrorism, including the possible use of biological agents. The risk of smallpox as a bioterrorism agent is unknown. Its potential release presents unique clinical and epidemiological challenges: the disease has been absent for >2 decades, the discontinuation of routine immunizations has resulted in a significant proportion of the US population being without immunity, and deliberately introduced smallpox may transmit differently than naturally occurring smallpox [1]. On 13 December 2002, President George W. Bush announced the launching of the US National Smallpox Pre-Event Vaccination Program in an effort to improve readiness for and response to a biological agent release.

The Centers for Disease Control and Prevention (CDC), in collaboration with state and local health departments, developed and implemented the smallpox vaccination program for civilian public health and health care response team members. The known risks of the vaccine, the potentially unknown risks of vaccine administration to a population older than that previously vaccinated for smallpox in the United States, and uncertainty regarding whether the population at risk for developing an adverse event would have a profile similar to those of groups identified in the past added extra dimensions of urgency to monitoring the safety of the smallpox vaccine compared with that of other vaccines [2–9]. The existing mechanism for monitoring the safety of vaccines is the Vaccine Adverse Event Reporting System (VAERS). VAERS, jointly administered by the CDC and the US Food and Drug Administration, receives spontaneous reports of suspected vaccine adverse events after administration of any US licensed vaccine [10]. To improve the monitoring of adverse events after smallpox vaccination, the CDC formed a Clinical Team and a Cardiac Team to improve case investigation, thus enhancing the surveillance system. This enhanced surveillance system provided additional follow-up of potentially serious reports submitted through VAERS.

The CDC requested that the Institute of Medicine (IOM), an independent body that advises the government on public health issues, provide guidance on implementing the smallpox vaccination program. The IOM's initial report, issued 24 January 2003, addressed safety and the monitoring of vaccine adverse events [1]. The IOM acknowledged the CDC's efforts to monitor serious adverse events (SAEs) after vaccination through the existing passive surveillance system. However, the IOM suggested that reliance on a passive system may not capture all SAEs. The dependence on vaccinees and health care providers to submit reports and the unknown acceptability, sensitivity, and representativeness of VAERS presented the possibility of underestimating the incidence of SAEs [11]. The IOM recommended the addition of an active surveillance system intended to confirm the response of every vaccinee to the vaccine and to allow for the gathering of data, primarily on rare SAEs and secondarily on common adverse events [1]. On 24 January 2003, following the IOM recommendation, the Smallpox Vaccine Adverse Events Active Surveillance (SVAEAS) system was developed and implemented.

We used the CDC evaluation guidelines to assess SVAEAS as a public health surveillance system [12]. We compared the frequency of SAEs detected by the active surveillance system with that detected by the enhanced passive system. Here, we describe SVAEAS, analyze its attributes, and discuss factors that had an impact on the performance of the system.

Methods

The grantee survey, Pre-Event Vaccination System (PVS), VAERS, and Clinical and Cardiac Team Databases (CCTDs) were used to review the SVAEAS system. CDC guidelines for evaluating public health surveillance systems were also used [12].

Grantee survey. A survey administered to the state and local health departments on 22 April 2003 collected information on utilization of and experiences with the active surveillance system. The survey was sent to 62 health departments, including those of the 50 states, the District of Columbia, 3 cities (New York City, Chicago, and Los Angeles), 1 commonwealth (Puerto Rico), and 7 territories (Virgin Islands, American Samoa, Micronesia, Guam, Marshall Islands, Northern Mariana Islands, and Palau), henceforth referred to as “grantees.”

PVS, VAERS, and CCTDs. The network of smallpox program resources and databases that contributed to the overall data collection of smallpox vaccinations included the PVS, VAERS, and CCTDs (figure 1). The PVS database functioned as a vaccine administration support system and collected vaccinee demographic data, vaccination history, vaccine response (“take” reading), and vaccine lot and diluent usage. The PVS system assigned each vaccinee a unique identifier, the Patient Vaccination Number (PVN), used to link a patient's record to a VAERS adverse event report.

The CCTD included data collected by the Clinical Team and Cardiac Team, who provided additional follow-up information on SAE reports collected through VAERS, including vaccination history, clinical signs and symptoms, and laboratory tests. The CCTDs functioned as our reference databases in determining the accuracy of SAE data reported through active surveillance.

SVAEAS. The SVAEAS system, a state-based reporting system, collected data, for all vaccinated civilian health care, public health, and emergency response workers, on vaccination, contraindications to receipt of the vaccine (i.e., history of atopic dermatitis or immune deficiency), vaccinia transmission, and adverse events (figure 2). The SVAEAS system was designed to be compatible with the PVS; the PVN was the unique identifier for both systems. In April 2003, the SVAEAS system was modified to collect additional information on cardiac contraindications and risk factors, including history of myocardial infarction, angina, congestive heart failure, stroke, transient ischemic attack, diabetes, hypercholesterolemia, personal or family history of heart disease, hypertension, and smoking. SVAEAS also collected information about myocarditis and/or pericarditis (hereafter referred to as “myo/pericarditis”), myocardial infarction, angina, and other cardiac adverse events [13, 14].

All vaccinees were contacted �?�21–28 days after vaccination to determine the onset of a new or worsening health condition and inadvertent transmission of the vaccine virus. Specific adverse events were targeted by active surveillance; however, the system allowed for the collection of unexpected events through open text fields. For consistency, adverse event case definitions were developed and used by the CDC. Case definitions and information about clinical manifestations, diagnosis, and management of adverse events have been published [14–16]. All information was collected and reported for every vaccinee.

System review. The SVAEAS system was reviewed according to published CDC guidelines for evaluating public health surveillance systems. Seven defined attributes were assessed: simplicity, flexibility, acceptability, sensitivity, positive predictive value (PPV), representativeness, and timeliness [12].

To assess the simplicity of the SVAEAS system, we reviewed data collected through the worksheet and the methods of collecting and managing data. Flexibility was assessed as the ability of the system to modify its data collection form. Acceptability was assessed as the level of participation by grantees in the active surveillance system. High-participation grantees provided reports on ⩾70% of their vaccinated population to the active surveillance system. Low-participation grantees submitted reports on <70% of their vaccinated population.

To determine the sensitivity of any surveillance system, data external to the system are necessary to define the true frequency of the condition in the population under surveillance and to validate the collected data. We used the CCTDs, developed from enhanced passive surveillance, as reference databases to compare the frequency of SAEs detected in the active relative to the enhanced passive surveillance system. The following SAEs were assessed: inadvertent inoculation (nonocular), generalized vaccinia, eczema vaccinatum, progressive vaccinia, postvaccinial encephalitis, erythema multiforme major, superinfection of vaccination site, ocular vaccinia, myo/pericarditis, dilated cardiomyopathy, and ischemic events. Additionally, we performed an overall analysis utilizing all reports, a stratified analysis using categorical classifications (“noncardiac” and “cardiac”). The stratified analysis was performed (1) utilizing all reports and (2) excluding reports submitted before April 2003, when formal cardiac adverse event report collection began. All analyses were conducted using Microsoft Access 2003 [17] and SAS version 9.1 [18].

We determined the PPV by assessing the accuracy of submitted reports by specific type of SAE. The final health-related outcome in the active surveillance was compared with the final diagnosis in the CCTDs. We assessed PPV initially for general adverse event categories (i.e., cardiac and noncardiac), then assessed the specificity by individual adverse event.

We compared the demographic and vaccination history characteristics of the vaccinated population in PVS with those in SVAEAS, to determine whether active surveillance provided a representative sample of the vaccinated population. We considered the time required to conduct follow-up and to submit a subsequent report to determine timelines of the active surveillance system.

Results

SVAEAS and Grantee Survey

Between January 2003 and May 2005, 37 (60%) of the 62 grantees participated in the active surveillance program; active surveillance reports were received for 18,882 (48%) of the 39,224 (based on available data obtained from the PVS system on 19 July 2005) persons vaccinated in the US National Smallpox Pre-Event Vaccination Program. Surveys were sent to all 62 grantees; 52 grantees (84%) responded. Six responding grantees were excluded from analysis because no data were available on postvaccination follow-up of vaccinees. Our analysis is based on the 46 grantees who were collecting postvaccination data at the time the survey was administered.

System Review

Simplicity. Limited items were included on the smallpox active surveillance worksheet, to enhance efficiency and timeliness of reporting. Grantee/user identification fields were automatically populated, depending on the submission mechanism. Many adverse events were provided in a menu listing. Information on cardiovascular conditions, risk factors, and medical treatment was collected using check boxes.

State and local health departments or vaccination site coordinators determined vaccinee follow-up mechanism. Most grantees (39 [85%]) used the telephone exclusively or in combination with other methods. One grantee conducted follow-up through mailings; another used in-person interviews.

States used one of 2 mechanisms to transmit data to the CDC. Thirty-three states (72%) centralized data from local sites at one location or the state health departments, then forwarded compiled data to the CDC. Other grantees (13%) had data transmitted directly from the vaccination clinics to the CDC. Seven percent of grantees used a combination of centralization and decentralized methods.

Grantees could transmit data to the CDC via a Web-based system or by fax. One state transmitted data solely by fax; 70% exclusively used the Web-based approach. Eight grantees (17%) had systems other than the CDC PVS system; thus, they provided information via batch uploading procedures.

Flexibility. The ability to modify the data collection form to include items capturing cardiovascular contraindications, cardiac risk factors, and cardiac adverse events demonstrated the flexibility of the active surveillance system. This modification was implemented after reports of cardiac adverse events among recently vaccinated persons prompted the Advisory Committee on Immunization Practices to recommend that persons with known cardiac disease be deferred from vaccination [19]. The recommendation was issued in March 2003, and grantees were notified and the data collection form modified seamlessly in April 2003. The presence of an open text field that allows reporters to document adverse events not listed as a menu choices also augments the flexibility of the system.

Acceptability. Figure 3 demonstrates the grantees' levels of participation in the active surveillance program. Of 37 grantees participating in active surveillance, 23 (62%) provided reports on >70% of the vaccinees; 15 (Chicago, Connecticut, Delaware, Hawaii, Illinois, Indiana, Maine, Minnesota, Mississippi, North Carolina, New York, Oregon, Rhode Island, Vermont, and Wyoming) provided reports on ⩾90% of the vaccinees.

Sensitivity. The enhanced passive reporting system identified 49 SAEs after smallpox vaccination (table 1). Grantees participating in active surveillance submitted 33 of the 49 SAE reports identified through enhanced passive surveillance. Active surveillance detected 8 (24%) of these 33 SAE reports plus 1 case of inadvertent inoculation not captured through the enhanced passive reporting system.

Stratification by type of report showed that sensitivity was higher for reports of noncardiac SAEs (100%) than for reports of cardiac SAEs (41.7%; 95% CI, 13.8%–69.6%). Additional analysis excluding early cardiac reports—those submitted before April 2003-suggested comparable accuracy in identifying cardiac-related SAEs (44.4%; 95% CI, 12.0%–76.9%). The CIs determined for the sensitivity estimates were wide, reflecting the rarity of the health events of interest. These estimates are unstable and provide little information regarding the true ability of the system to accurately detect SAEs.

PPV. Table 2 provides estimates of the accuracy of active surveillance reports for specific SAEs. Accuracy was lowest for generalized vaccinia, ischemic events, and inadvertent inoculation (table 2). The system more accurately detected superinfection and myo/pericarditis cases (table 2). As with the sensitivity estimates, the wide 95% CIs reflect the rare occurrence of the events.

Representativeness. The active surveillance population did not differ significantly from the total vaccination population in sex distribution (percentage female, 65% in SVAEAS and 63% in PVS), vaccination history (percentage primary vaccination, 24% in SVAEAS and 23% in PVS), and age at vaccination (mean age, 47 years in SVAEAS and 47 years in PVS).

Timeliness. Vaccination dates compared with follow-up dates indicated that the mean duration of follow-up was 40 days (median, 27 days; range, 0–383 days). High- and low-participation grantees did not differ significantly in duration of follow-up (median, 26 and 27 days, respectively). The duration of follow-up was <21 days for 3262 vaccinees (18.9%).

States submitted active surveillance reports an average of 55 days (median, 34 days; range, 1–507 days) after the date of vaccinee follow-up. SAE reports were submitted a mean of 52 days (median, 43 days; range, 0–154 days) after follow-up. Other adverse events were reported a mean of 51 days (median, 29 days; range, 0–325 days) after follow-up. Furthermore, the length of time to submit the report varied with states' participation levels. High-participation states submitted reports a mean of 43 days (median, 23 days; range, 0–416 days) after follow-up. Low-participation states submitted reports a mean of 55 days (median, 35 days; range, 0–507) after follow-up (P<.0001).

Discussion

The active surveillance system was developed to detect SAEs that would potentially be missed by passive surveillance. For states that participated in both the enhanced passive and the active surveillance systems, SVAEAS identified only 1 SAE not detected by and failed to identify 25 SAEs detected by enhanced passive surveillance. VAERS is a long-standing surveillance system. The increased reporting is likely indicative of states' familiarity with the system. Furthermore, many states contacted the CDC directly for assistance in assessing adverse events after smallpox vaccination and, following discussion, were prompted to submit a VAERS report. This finding was documented by review of the 8 SAEs identified in both systems. Six of the 8 SAEs identified through the enhanced surveillance system consisted of VAERS reports submitted the day of or after the initial Clinical Team or Cardiac Team consultation. Thus, a direct comparison of confirmed adverse events among submitted reports between VAERS and the active surveillance system would yield biased results. Several factors had an impact on the system's attributes and overall performance. The simplicity of the system may have been compromised by electronic challenges experienced in data transmission. Problems with the PVS system, difficulties with digital certificates and obtaining activity approval on certificates, confusion in matching the instructions to what was presented on the screen for Web-based reporting, and lengthy editing procedures were listed on the survey as factors that increased difficulty in using the system.

The acceptability of the system may have been negatively affected because the IOM recommendation to implement an active surveillance system followed the directive to rapidly implement the smallpox vaccination program. Grantees indicated that SVAEAS was developed and implemented before they had adequate time to provide feedback. The immediacy with which SVAEAS was developed and implemented minimized the opportunity for grantee input. Increased time for feedback would allow increased data quality and participation. Additionally, personnel limitations negatively affected participation in the active surveillance program. State and local health department resources were allocated before the development and implementation of active surveillance. Other temporally coincident public health emergencies, including the severe acute respiratory syndrome (SARS) epidemic, had an impact on the ability to participate in active surveillance, because of staff reassignments. Thus, states' limited personnel resources and overall burden imposed on personnel contributed to less efficient and effective active follow-up of adverse events.

The sensitivity of SVAEAS was largely affected by its inability to collect specific clinical information on reported adverse events, including symptoms, symptom onset, medications, and laboratory confirmations. The lack of clinical information made it more difficult to confirm diagnoses of adverse events. Furthermore, the lack of clinical information coupled with the low prevalence of the SAEs under surveillance contributed to decreased sensitivity and PPV of the system in identifying SAEs. A low PPV suggests that noncases may be investigated and that false “outbreaks” may be identified that are artifacts of the surveillance system [11]. A surveillance system with low PPV and, therefore, frequent false-positive case reports would misdirect resources. The wide 95% CIs for sensitivity and PPV estimates determined in our review result from the extremely small numbers of observed events and indicated limited confidence in the sensitivity and PPV of the surveillance system.

The active surveillance population was representative of the target population vaccinated. Very specific eligibility criteria established for participation in the National Smallpox Pre-Event Vaccination Program created a homogenous demographic group. Although limited information was available to determine the influence of case ascertainment bias, such biases were likely influenced by state mechanisms for postvaccination follow-up.

The timeliness of the active surveillance system may have been adversely affected by the inherent 21- to 28-day delay in conducting follow-up. This delay to allow time for scab separation and the detection of possible inadvertent inoculation of another individual contributed to increased difficulty in contacting vaccinees for follow-up; a considerable proportion were lost to follow-up. Furthermore, the grantee survey indicated that delay in contacting vaccinees for follow-up, problems with PVS, and lack of resources contributed to delays in active surveillance reporting. Delays in reporting inevitably diminish the ability to identify cases in a timely manner.

In conclusion, the active surveillance system did not improve the identification of SAEs, compared with the enhanced passive surveillance system. As demonstrated by this review, timely development and acceptability of a surveillance system to partners has a significant impact on the performance of that system. Increased levels of communication provided to partners, better allowance for partner feedback, and the availability of additional resources would increase state participation. Although these findings are true, the success of the enhanced passive surveillance system in this setting cannot be compared with the routine VAERS system in the United States. For the smallpox vaccination program, enhanced passive surveillance was comparable to active surveillance. However, active surveillance may be useful when there is no enhanced passive surveillance system available.

Acknowledgments

We thank the Smallpox Vaccine Adverse Events Response and Monitoring Activity (SVAERMA) team (see below), state and local health department officials, and State Adverse Event Coordinators for their development of, implementation of, and participation in the smallpox active surveillance program. We also appreciate the project guidance of Sean Shadomy; the geographic information systems map provided by Paul Calame; the technical contributions of Roseanne English, John Copeland, and David King; the assistance of Vicki Kipreos; and the editorial feedback of Mary McCauley.

The members of the SVAERMA team are as follows. Executive Team: Gina Mootrey, Eric E. Mast, Herschel Lawson, and Mary McCauley. Cardiac Team: Christine Robinette Curtis, Patricia Galloway, Beth Hibbs, Paige Hightower, Nidhi Jain, Nancy H. Levine, Mona Marin, Jacqueline Miller, Juliette Morgan, Pedro L. Moro, Martha H. Roper, Richard A. Schieber, Margarita Sniadack, and David Swerdlow. Clinical Team: Francisco M. Averhoff, Christine G. Casey, Louisa E. Chapman, Rosaline Dhara, Kristina L. Ernst, Kirsten Ernst, Kathleen Fullerton, Michael Deming, Daniel Fishbein, Lamar Hasbrouck, James Heffelfinger, Barbara L. Herwaldt, Andrew Kroger, Anne C. Moore, Juliette Morgan, Monica Parise, Meredith Reynolds, Scott Santibanez, James Sejvar, Bruce Tierney, Thomas Török, Claudia Vellozzi, Charles Vitek, and Xiaojun Wen. State Team: Susan Reef, Eduard Eduardo, Masa Tanaka, John H. McGowan, Danice K. Eaton, Carol Knowles, Kathleen McDuffie, Jennifer L. Cleveland, Jami L. Fraze, Elizabeth Bolyard, Lynne M. Sehulster, Philip M. J. Baptiste III, Ceil Threat, Fred Ingram, and Tracy Thomas. Surveillance Team: Sara Critchley, Hayley Hughes, John K. Iskander, Joyce Goff, Madeline Sutton, Phuc Nguyen-Dinh, Roumiana S. Boneva, Roseanne English, and John Copeland.

Financial support. Centers for Disease Control and Prevention.

Supplement sponsorship. This article was published as part of a supplement entitled “Posteradication Vaccination against Smallpox,” sponsored by the National Center for Immunization and Respiratory Diseases, Coordinating Center for Infectious Diseases, Centers for Disease Control and Prevention (CDC), and by the Coordinating Office for Terrorism Prevention and Emergency Response, CDC.

Potential conflicts of interest. All authors: no conflicts.

References