Statistics for Method Verification of Qualitative Assays in Clinical Microbiology
Introduction
In clinical microbiology laboratories, adoption of new methods is regulated by a variety of guidance documents and a federal law known as the Clinical Laboratory Improvement Act (CLIA) [1]. According to CLIA regulations, evaluation of method performance and error assessment are divided into two categories: verification and validation. Method verification requirements have been reviewed elsewhere [1, 2, 3, 4, 5, 6, 7, 8], so method verification concepts are not duplicated here; rather, the focus of this review is on experimental design concepts and statistics that are associated with method verification.
For FDA review and approval to occur, the accuracy and clinical utility of an assay are assessed in clinical trials that are often multi-site in nature. Rigorous assay development parameters, design control, and clinical trials assess performance, using clinical samples and microbial strains from across the United States and sometimes other countries. In this process, hundreds or even thousands of clinical samples are used to evaluate the new test method, while performance equal to or better than that of the reference method is confirmed using the samples and is documented in the manufacturer's product insert (PI). Local verification processes are performed to assess the risks of error, bias, and imprecision and to assess the probability that test limitations could cause an erroneous change in the interpretation of test results or treatment decisions. For laboratories that perform verification processes for FDA-cleared methods, documentation that laboratory methods have been assessed for accuracy similar to that reported in the manufacturer's PI is required.
Section snippets
Statistical Principles That Support Qualitative Method Verification: Planning Experiments
Well before data collection begins, laboratories must establish parameters for verification experiments that adhere to the standard principles of scientific experimental design. When planning new method development in a clinical microbiology laboratory, there are several key steps to consider [8].
- 1
Establish infrastructure to document your entire process and easily retrieve information for future laboratory inspections. Over time, my colleagues and I have compiled a strategy to categorize the
Medical decision making for the new test method
Although many numerical values for critical decision making exist for clinical chemistry tests [9] and for some viral loads [8], no such list exists for qualitative microbiology tests. Therefore, clinical microbiologists must pre-determine allowable error, for instance, percentages of false-positive and false-negative results, failures, repeats, contamination, and other key performance attributes of the assay relative to medical care.
Hypothesis for method verification
Hypothesis testing is meant to determine if the variation
Selecting an Appropriate Statistical Software Package
There are many ways the clinical laboratory staff can perform simple statistical analysis. Most laboratories have access to Microsoft Excel, which contains a number of statistical functions, some described here. Several software programs are designed for laboratory staff, such as EP Evaluator (Data Innovations, South Burlington, VT) (http://www.datainnovations.com/ep-evaluator), which has modules that meet requirements for verification. A complete statistical package, JMP (SAS Institute, Cary,
Collecting and Validating Data
Data collection and validation are tasks that are easily understood by the clinical laboratory employee. Data must be accurately collected and checked (validated) for accuracy. If manual data collection occurs, it is prone to human error and should be scrutinized closely. Computerized data transfer is less prone to error but must also receive audits on a regular basis to check the data for validity.
Descriptive statistics
As mentioned above, descriptive statistics are calculated and commonly used as a basis for comparisons of qualitative methods. Descriptive statistics often describe two aspects of method performance: probability and inference. Probability is defined as the number of likely outcomes divided by the number of possible outcomes. Probability ranges between 0 and 1, with 1 being 100% probability. Inference is defined as the prediction of an outcome based on the calculated probability [8, 10, 11].
Characterize and understand bias
Major threats to diagnostic result validity include a variety of biases [8, 10, 11]. Spectrum bias exists when the population under investigation does not reflect the clinically relevant population. Be particularly careful using data generated from tertiary referral centers early in the development of the test, where referral bias may bias the sample case mix. Similarly, if the severity of disease differs, it may influence results for test sensitivity, since severe disease or abnormalities are
Inferential Statistics
There are two categories of inferential statistics: nonparametric and parametric statistics.
- •
Nonparametric statistics are used in the following situations.
- •
Data are categorical (qualitative or ordinal): both independent and dependent variables are categorical or nominal.
- •
The data distribution is not normal.
- •
The variance is not homogeneous.
- •
The assumptions required of parametric statistics are not met.
An example of a nonparametric test is the chi-square test, a qualitative test that is based on a
Summary
CLIA has clear guidelines on the verification and validation of LDTs and other non-FDA-approved/cleared assays, but choices of statistical methods and methods for drawing conclusions about accuracy are not as well-defined. Nevertheless, it is ultimately the responsibility of the clinical laboratory to determine and support decisions related to accuracy and utility required to introduce the method into the laboratory. Defining assay limitations and performing robust method verification are
References (20)
Charting uncharted territory: a review of the verification and implementation process for matrix-assisted laser desorption ionization: time of flight mass spectrometry (MALDI-TOF MS) for organism identification
Clin Microbiol Newsl
(2013)Guidelines for verification and validation of diagnostic immunology tests
Clin Microbiol Newsl
(2000)Initial training verification and competency assessment in the clinical microbiology laboratory
Clin Microbiol Newsl
(2001)Verification and validation of tests used in the clinical microbiology laboratory
Clin Microbiol Newsl
(2015)Streamlined quality control of commercial microbial identification systems
Clin Microbiol Newsl
(2011)- et al.
The STARD initiative
Lancet
(2003) Discrepant analysis: a biased and an unscientific method for estimating test sensitivity and specificity
J Clin Epidemiol
(1999)CLIA Regulations
- et al.
Cumitech 31A: method verification and validation procedures in the clinical microbiology laboratory
Cumitech
(2009) - et al.
Molecular method verification
Cited by (5)
Diagnostic Accuracy Measures—The Laboratory and Non-Laboratory Perspective
2020, Clinical Microbiology NewsletterCitation Excerpt :When characterizing the accuracy of a diagnostic test or marker to discriminate between the presence and absence of a disease in individuals suspected of having that disease, it is commonly understood that the technical and analytical development of the test or assay has been completed, that the test is safe for use in patients, and that clinical laboratory method verification and validation is complete, including assessments of accuracy. precision, and if required, linearity, reportable range, reference interval, the limit of detection, and percent carryover [4,5]. For a complete assessment of clinical validity, diagnostic accuracy is associated with a diagnostic test as well as the test's discriminative ability.
Statistics for Method Verification of Quantitative Assays in Clinical Microbiology
2019, Clinical Microbiology NewsletterCitation Excerpt :Otherwise, an assessment of matrix effects and interfering substances is prudent [16,17]. The basic elements of method verification have been fully described in previous publications, which provide a useful adjunct to this review [4,18–20]. There are several strategies that can provide adequate method verification, and this review describes some examples of common strategies.
Documenting Clinical Microbiology Impact: Performing Clinical Research to Document the Value of the Microbiology Laboratory
2018, Clinical Microbiology NewsletterCitation Excerpt :In outcome studies, the SOC might be the routine care provided before the clinical or diagnostic intervention. Next, you should select data variables (dependent or independent variables) [17], as well as any surrogate variables (i.e., those that provide an indirect measurement of outcome, impact, or effect in situations where direct measurement of clinical effect is not feasible or practical). Table 2 lists some of the fields used in a data dictionary created to monitor the appropriateness of Clostridium difficile testing based on survey and clinical data.
Documenting Clinical Microbiology Impact: Basic Concepts for Designing Clinical Studies and Outcome Research Using Principles of Biostatistics
2018, Clinical Microbiology Newsletter