Statistics for Method Verification of Qualitative Assays in Clinical Microbiology

https://doi.org/10.1016/j.clinmicnews.2017.03.008Get rights and content

Abstract

Despite improvement over the last decade, inadequate method verification strategies and documentation are still common laboratory inspection deficiencies. Some of the gaps occur because microbiologists may not use scientific experiment concepts, such as experimental design, data validation, biostatistics, and electronic documentation, that can improve their verification strategies for new test methods. Other gaps occur because laboratory resources are often limited. Clinical laboratories face tough choices when they consider implementing a new test method, and this review summarizes the basic statistical concepts that microbiologists need to critically plan and review qualitative method verification with the aim of determining whether a new diagnostic test is fit for patient testing.

Introduction

In clinical microbiology laboratories, adoption of new methods is regulated by a variety of guidance documents and a federal law known as the Clinical Laboratory Improvement Act (CLIA) [1]. According to CLIA regulations, evaluation of method performance and error assessment are divided into two categories: verification and validation. Method verification requirements have been reviewed elsewhere [1, 2, 3, 4, 5, 6, 7, 8], so method verification concepts are not duplicated here; rather, the focus of this review is on experimental design concepts and statistics that are associated with method verification.

For FDA review and approval to occur, the accuracy and clinical utility of an assay are assessed in clinical trials that are often multi-site in nature. Rigorous assay development parameters, design control, and clinical trials assess performance, using clinical samples and microbial strains from across the United States and sometimes other countries. In this process, hundreds or even thousands of clinical samples are used to evaluate the new test method, while performance equal to or better than that of the reference method is confirmed using the samples and is documented in the manufacturer's product insert (PI). Local verification processes are performed to assess the risks of error, bias, and imprecision and to assess the probability that test limitations could cause an erroneous change in the interpretation of test results or treatment decisions. For laboratories that perform verification processes for FDA-cleared methods, documentation that laboratory methods have been assessed for accuracy similar to that reported in the manufacturer's PI is required.

Section snippets

Statistical Principles That Support Qualitative Method Verification: Planning Experiments

Well before data collection begins, laboratories must establish parameters for verification experiments that adhere to the standard principles of scientific experimental design. When planning new method development in a clinical microbiology laboratory, there are several key steps to consider [8].

  • 1

    Establish infrastructure to document your entire process and easily retrieve information for future laboratory inspections. Over time, my colleagues and I have compiled a strategy to categorize the

Medical decision making for the new test method

Although many numerical values for critical decision making exist for clinical chemistry tests [9] and for some viral loads [8], no such list exists for qualitative microbiology tests. Therefore, clinical microbiologists must pre-determine allowable error, for instance, percentages of false-positive and false-negative results, failures, repeats, contamination, and other key performance attributes of the assay relative to medical care.

Hypothesis for method verification

Hypothesis testing is meant to determine if the variation

Selecting an Appropriate Statistical Software Package

There are many ways the clinical laboratory staff can perform simple statistical analysis. Most laboratories have access to Microsoft Excel, which contains a number of statistical functions, some described here. Several software programs are designed for laboratory staff, such as EP Evaluator (Data Innovations, South Burlington, VT) (http://www.datainnovations.com/ep-evaluator), which has modules that meet requirements for verification. A complete statistical package, JMP (SAS Institute, Cary,

Collecting and Validating Data

Data collection and validation are tasks that are easily understood by the clinical laboratory employee. Data must be accurately collected and checked (validated) for accuracy. If manual data collection occurs, it is prone to human error and should be scrutinized closely. Computerized data transfer is less prone to error but must also receive audits on a regular basis to check the data for validity.

Descriptive statistics

As mentioned above, descriptive statistics are calculated and commonly used as a basis for comparisons of qualitative methods. Descriptive statistics often describe two aspects of method performance: probability and inference. Probability is defined as the number of likely outcomes divided by the number of possible outcomes. Probability ranges between 0 and 1, with 1 being 100% probability. Inference is defined as the prediction of an outcome based on the calculated probability [8, 10, 11].

Characterize and understand bias

Major threats to diagnostic result validity include a variety of biases [8, 10, 11]. Spectrum bias exists when the population under investigation does not reflect the clinically relevant population. Be particularly careful using data generated from tertiary referral centers early in the development of the test, where referral bias may bias the sample case mix. Similarly, if the severity of disease differs, it may influence results for test sensitivity, since severe disease or abnormalities are

Inferential Statistics

There are two categories of inferential statistics: nonparametric and parametric statistics.

  • Nonparametric statistics are used in the following situations.

  • Data are categorical (qualitative or ordinal): both independent and dependent variables are categorical or nominal.

  • The data distribution is not normal.

  • The variance is not homogeneous.

  • The assumptions required of parametric statistics are not met.

An example of a nonparametric test is the chi-square test, a qualitative test that is based on a

Summary

CLIA has clear guidelines on the verification and validation of LDTs and other non-FDA-approved/cleared assays, but choices of statistical methods and methods for drawing conclusions about accuracy are not as well-defined. Nevertheless, it is ultimately the responsibility of the clinical laboratory to determine and support decisions related to accuracy and utility required to introduce the method into the laboratory. Defining assay limitations and performing robust method verification are

References (20)

There are more references available in the full text version of this article.

Cited by (5)

  • Diagnostic Accuracy Measures—The Laboratory and Non-Laboratory Perspective

    2020, Clinical Microbiology Newsletter
    Citation Excerpt :

    When characterizing the accuracy of a diagnostic test or marker to discriminate between the presence and absence of a disease in individuals suspected of having that disease, it is commonly understood that the technical and analytical development of the test or assay has been completed, that the test is safe for use in patients, and that clinical laboratory method verification and validation is complete, including assessments of accuracy. precision, and if required, linearity, reportable range, reference interval, the limit of detection, and percent carryover [4,5]. For a complete assessment of clinical validity, diagnostic accuracy is associated with a diagnostic test as well as the test's discriminative ability.

  • Statistics for Method Verification of Quantitative Assays in Clinical Microbiology

    2019, Clinical Microbiology Newsletter
    Citation Excerpt :

    Otherwise, an assessment of matrix effects and interfering substances is prudent [16,17]. The basic elements of method verification have been fully described in previous publications, which provide a useful adjunct to this review [4,18–20]. There are several strategies that can provide adequate method verification, and this review describes some examples of common strategies.

  • Documenting Clinical Microbiology Impact: Performing Clinical Research to Document the Value of the Microbiology Laboratory

    2018, Clinical Microbiology Newsletter
    Citation Excerpt :

    In outcome studies, the SOC might be the routine care provided before the clinical or diagnostic intervention. Next, you should select data variables (dependent or independent variables) [17], as well as any surrogate variables (i.e., those that provide an indirect measurement of outcome, impact, or effect in situations where direct measurement of clinical effect is not feasible or practical). Table 2 lists some of the fields used in a data dictionary created to monitor the appropriateness of Clostridium difficile testing based on survey and clinical data.

View full text