1.1.  Purpose

This document provides guidelines for evaluation of veterinary laboratory capability to conduct diagnostic tests for infectious diseases.

1.2.  Scope

These guidelines are intended for use by OIE Member Countries as part of the evaluation of laboratories that are carrying out tests to qualify animals and animal products for international movement. These guidelines should be used in conjunction with the OIE Guidelines for Laboratory Quality Evaluation for overall assessment of laboratory quality and capability.

This guide is based on the relevant requirements of the ISO[1] 9000 series of standards, ISO/IEC[2] 17025 and ISO Guide 43.

1.3.  Interlaboratory test comparisons

Interlaboratory test comparisons may be undertaken for a variety of reasons which may include:

i)     Determining a laboratory’s capability to conduct specific diagnostic tests,

ii)    Checking or certifying the performance of individual operators,

iii)   Checking or certifying the calibration of instrumentation,

iv)    Harmonising existing test methods,

v)     Evaluating new test methods,

vi)    Assigning values and ranges to standard materials,

vii)   Resolving interlaboratory differences.

1.4.  Proficiency testing

When an interlaboratory test comparison is conducted for the express purpose of determining a laboratory’s capability to conduct specific diagnostic tests, i.e. 1.3.i above, it is referred to as proficiency testing. Proficiency testing is an integral part of laboratory accreditation programmes.

Proficiency testing schemes are based on defined sets of highly characterised test materials which are sometimes referred to as check sample panels. These panels are simultaneously sent to participating laboratories for testing. The results are collected and analysed against the intended result in order to determine the capability of a participating laboratory to conduct a diagnostic test and produce correct results.

1.5.  Accreditation

An accreditation programme is a formal process for recognition of laboratory quality and capability by an independent authority. It requires that laboratories successfully participate in an accreditation programme on an ongoing basis in order to maintain their recognition status. The independent authority awards or denies recognition based on stipulated requirements for quality and capability.

In the initial stage of accreditation, laboratories are required to demonstrate a specified and sustainable level of quality. Ideally this would involve compliance with ISO 9000 and ISO/IEC 17025 General Requirements for the Competence of Calibration and Testing Laboratories (1990) in order to qualify for entry into the programme. However, it is recognised that in many circumstances such a high level may be difficult to achieve for a variety of reasons. The OIE Guidelines for Laboratory Quality Evaluation were prepared in order to establish a minimum acceptable level of quality.

The second stage of accreditation entails regularly scheduled proficiency testing for the evaluation of a laboratory’s capability to conduct specific diagnostic tests. As proficiency testing schemes are a form of interlaboratory comparison, they must involve two or more laboratories. There is no agreed standard for proficiency testing in veterinary diagnostics, although several schemes are in operation at international and national levels. The present guidelines have been prepared to be used in conjunction with the OIE Guidelines for Laboratory Quality Evaluation. Together, these guidelines form an acceptable basis for a quality assurance programme.

2. Authority and recognition

Accreditation programmes and proficiency testing schemes should be operated by an independent authority in order to prevent any bias in the award or denial of recognition.

Participation in an international accreditation programme and proficiency testing scheme should be voluntary. Lack of participation or failure to achieve recognition should not prevent a laboratory from conducting diagnostic tests or a country from entering into trade agreements.

Participation and recognition status should be made available by the independent authority to trading partners only at the request of or with the consent of the participating laboratory or country authority.

Such a programme and scheme may involve a cost to the participating laboratories for this service.

3. Organisation and management

Details of the proficiency testing scheme and its purpose, eligibility of participating laboratories and disposition of the results should be documented by the coordinating organisation to ensure the protection of proprietary rights and confidential information.

A programme manager should have overall responsibility for the operation, quality and security of the proficiency testing scheme.

It is also the responsibility of the programme manager to ensure that laboratories involved in the production of test materials are compliant with the relevant requirements of the ISO 9000 series of standards and ISO/IEC 17025.

Employees should be free from pressure or inducements that might unduly influence the analysis of proficiency testing results or the recognition status of the participating laboratory.

Adequate supervision and security should be provided by staff involved in either the production and distribution of test materials to be used in the proficiency testing scheme or the receipt and analysis of test results submitted by participating laboratories.

4. Standard methods

For the characterisation of test materials to be used in check sample panels, the standard method should meet or exceed the minimum diagnostic performance characteristics required for eligibility as a prescribed test in the OIE Manual of Diagnostic Tests and Vaccines for Terrestrial Animals.

The standard test should be calibrated against international standard materials, if these are available. Participating laboratories should also be encouraged to calibrate their own assays against the same international standards.

5. Selection and composition of check sample panel

5.1.  General principles

For the purpose of selection of test materials for inclusion in the check sample panel, the initial assessment of the status and/or reactivity of the sample will be determined by the producing laboratory, using the standard method.

Acceptance of test materials into the proficiency panel should be based on repeated testing by more than one analyst conducting multiple runs of the test on different days. Sufficient values should be generated to assure the unequivocal status of the test material, including homogeneity.

5.2.  Composition of the proficiency panel

The number of test samples that constitute a check sample panel is not well defined. This will be dictated by the type of analysis to be performed on the results and the numbers required to ensure statistical validity.

Irrespective of the type of test, a minimum of three samples should be included:

i)     An unequivocal strong positive,

ii)    An unequivocal weak positive,

iii)   An unequivocal negative.

However, using only three samples of this nature would render the results very predictable after a few rounds of proficiency testing. It would be advisable, therefore, to add at least two more samples to the check sample panel which could be varied from one proficiency test round to the next. This would prevent participating laboratories from anticipating the expected outcome. The additional samples could be different from the above or replicates of the above or a combination.

In planning the overall process for preparation, testing and distribution of test materials and test items, the provider shall provide for, where appropriate, procedures and resources for:

i)     material selection;

ii)    maintaining suitable environments for preparation and testing of test material;

iii)   material preparation;

iv)    measuring and testing;

v)     calibration/validation of equipment and measurement methods;

vi)    assessing test material homogeneity;

vii)   assessing test material stability;

viii)  organising interlaboratory test comparisons with collaborators, where necessary; (see Note 1 below)

ix)    ensuring adequate storage facilities and conditions;

x)     ensuring adequate packaging and labelling;

xi)    ensuring appropriate transport and distribution arrangements;

xii)   statistical analysis of test results and assigning values of measurands and associated uncertainties;

xiii)  ensuring adequate reporting service to participants.

6. Statistical analysis

6.1.  Types of data

The choice of statistical analysis will in part be determined by the type of data generated by the test method in question. Qualitative data such as ‘positive’, ‘negative’ and/or ‘suspicious’ are somewhat limited in the statistical procedures which may be applied to them. Quantitative data such as end-point titres, and semi-quantitative data such as percentage inhibition values are more flexible with respect to the types of statistical analysis possible.

Irrespective of the type of data to be analysed, it is important that the data from all of the participating laboratories be compatible. In some cases, this may require that participating laboratories be instructed to use a specific dilution series or to express their data against a common standard.

6.2.  Assigned values

Either of two approaches may be used:

a)  Assigning of target value before issue

In the initial selection of test materials for the check sample panel, the producing laboratory will have assigned a preliminary value, range or status to the sample. For qualitative data, the assigned value may be the only acceptable value. If this is to be the case, then the producing laboratory should verify the status on a battery of tests to increase the confidence that the assigned value is in fact correct. However, as a goal, at least 80% of the participating laboratories should obtain the same result in proficiency tests. For quantitative and semi-quantitative data, the assigned value should be recalculated after proficiency testing results are submitted, and it should be taken as the mean value after removal of outliers.

b)  Assigning target value on the consensus value of the returned participants results

6.3.  Statistical methods

Many statistical procedures have been applied to interlaboratory comparisons, some being far more sophisticated than others. As a general rule, the statistics being applied should be valid, straightforward and meaningful to the participating laboratories.

Frequency analysis is a simple and meaningful method for participating laboratories to see where their performance lies with respect to the other laboratories in the proficiency testing scheme.

Measures of intra- and interlaboratory variance through repeatability and reproducibility indices will often provide valuable information on the precision and robustness of the test methods.

Youden analysis is a useful indicator of systematic or random error sources that may be causing problems in individual laboratories.

7. Pass/fail criteria

Decision criteria with regards to passing or failing a laboratory on a proficiency test should be clearly documented. These criteria must take into consideration factors which may vary from one disease to another and between types of tests. Once established, the criteria must be applied uniformly.

The types of statistical analyses chosen should assist in making pass/fail decisions. Laboratories submitting results that fall outside ranges established by statistical means should be identified. Results of tests that would potentially lead to a false-negative classification of an infected animal would have to be weighed against results that would potentially lead to a false-positive classification of a healthy animal. In most instances, the former type of error should not be tolerated as it indicates that there is a problem with diagnostic sensitivity. However, there may be some latitude in awarding a provisional status to laboratories experiencing problems with diagnostic specificity.

8. Frequency of proficiency testing

It is recommended that proficiency testing be done on a twice yearly basis, where possible. Depending on the country and disease, some consideration should be given to peak testing periods. Whenever possible, at least one of the proficiency tests should be scheduled to coincide with active testing periods.

Twice yearly, provides sufficient time between proficiency tests to undertake any corrective actions which might prevent a participating laboratory from losing its recognition status.

9. Laboratory recognition

The criteria for awarding, denying or withdrawing recognition should be clearly documented.

10. Logistics

10.1. Eligibility and acceptance

Eligible laboratories should be sent a comprehensive outline of the quality assurance programme and the proficiency testing scheme. This outline should include details pertaining to frequency of testing, commitments and deadlines, methods of data analysis, reporting structure, criteria for recognition, disposition of results and confidentiality. In addition, a form to be signed and returned to the coordinating organisation should be included which indicates that the eligible laboratory accepts the terms and conditions of the programme.

10.2. Notification and shipment of panels

Participating laboratories should be notified at least 1 month in advance of a pending proficiency test. Notification should also include the projected date and method of shipment of the check sample panel. Longer notification may be required by those laboratories in countries requiring import permits for the check sample panels.

Test materials in the check samples should be coded so as not to indicate their expected result. The coding may be alphabetic or numeric. A unique set of codes helps to prevent collusion between laboratories.

All shipments should be by the most expedient and direct method. All shipments should comply with IATA[3] regulations concerning the shipment of biological materials.

Upon shipment, the recipient laboratories should be informed of pertinent details (i.e. method of shipment, carrier, air-way bill, etc.) in order to facilitate rapid retrieval and clearance of the shipment upon arrival.

Check sample panels arriving in a damaged or questionable condition should be replaced immediately.

10.3. Testing and return of results

Participating laboratories should be given an adequate volume of test material and adequate time to complete the testing of the check sample panel to their satisfaction. The panel may be tested more than once and by more than one person in the participating laboratory. However, only one set of results should be returned to the coordinating organisation for analysis. Normally, the person responsible for running the test routinely should be selected to run the check sample panel.

The check sample panel should be accompanied by a complete set of instructions with respect to reconstitution, storage and handling, special testing requirements, data expression and deadline for the submission of results.

Results must be returned in the proper format and on time. Failure to do so could lead to omission from the round of proficiency testing and loss or downgrading of recognition status.

The coordinating organisation should acknowledge receipt of the results and their acceptance into the analysis.

10.4. Analysis and reporting

Analysis and reporting should be completed in a timely fashion after the deadline for the receipt of results.

A general report summarising the results of all of the analyses should be prepared for distribution to all participating laboratories. Participating laboratories should be randomly assigned a code to ensure anonymity in the general report. Individual laboratories should be informed of their unique code for this run of proficiency tests.

Individual laboratories should also receive a summary of their own performance and their recognition status. This summary should indicate clearly all factors contributing to any change in their status. Where the status has been downgraded, it is especially important to indicate real or potential causes which may have contributed to downgrading. In some instances, it may be pertinent to re-issue a second, identical panel after corrective actions have been taken.

A statement of status may also take the form of an official certificate.

All data, results of analyses and the recognition status of participating laboratories should be kept in confidence at all times.

11. Disclosure

The primary purpose of these guidelines is to remove trade barriers and not to create them. It would be expected that participating laboratories having achieved full recognition status may request that official verification of their status be made available to trading partners from the independent authority or coordinating organisation. This should only be done at the request of or with the consent of the participating laboratory or country authority.

12. References

ISO/IEC International Standard 17025 (2005). General requirements for the competence of testing and calibration laboratories. International Organisation for Standardisation (ISO)/International Electrotechnical Commission (ISO/IEC), ISO Central Secretariat, 1 rue de Varembé, Case Postale 56, CH - 1211, Geneva 20, Switzerland.

International Organisation for Standardisation (ISO) (1997). Proficiency testing by interlaboratory comparisons. Part 1: Development and operation of proficiency testing schemes. Part 2: Selection and use of proficiency testing schemes by laboratory accreditation bodies. ISO/International Electrotechnical Commission (ISO/IEC), Guide 43. ISO/IEC, Geneva, 19 pp.

ISO International Standards 9000:2005, 9001:2000; 9004:2000 (2000–2005). Quality management and quality assurance. International Organization for Standardization (ISO), ISO Central Secretariat, 1 rue de Varembé, Case Postale 56, CH - 1211, Geneva 20, Switzerland