Sigma Metrics for Assessing Accuracy of Molecular Testing
Section snippets
Why Do Quality Metrics Matter?
The major assumption physicians make prior to their clinical decisions is that the laboratory's test results are valid – that there is no medically important error that obscures the true result, generating either a false-positive or a false-negative result. Laboratory medicine embraces the paramount importance of quality. With a long history of quality and accuracy for diagnostic testing, it is easy to assume that a test result has the appropriate accuracy, but how can we ensure that the
How Do We Know That When We Achieve the Proper Level of Quality?
The core requirement of any method used for quality assessing is the ability to numerically define what we consider good performance and acceptable quality; therefore, we must also define unacceptable quality and poor performance, i.e., we must define an unacceptable error, i.e., a defect in the method's performance. For many diagnostic tests, the definition of what constitutes acceptable quality is implicit, but not often clearly articulated in an analytical context. For viral loads, such as
A “Traditional” Approach to Quality: Six Sigma
The quality management techniques collectively known as “Six Sigma” have been practiced in health care for several decades, and for even longer in manufacturing, business, and industry (3, 5, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17). The implementation of Sigma metrics has mostly occurred in laboratory subspecialties, such as chemistry, hematology, and immunology.
The core concept of Six Sigma is to identify defects (false negative or false positive reactions) and then reduce or eliminate as many
Defining Quality Requirements for Six Sigma Performance
Most molecular laboratories are not fully accustomed to discussing the “tolerance limits” of our test methods. Laboratorians are far more comfortable discussing a similar concept, known as the allowable total error (TEa). The TEa, the net analytical error is defined as a combination of method imprecision (random error) and method inaccuracy (systematic error, or bias) (16, 17). When a test method exceeds the TEa, the method or process begins to generate defects (result errors), which could be
The Challenge of Adapting Six Sigma to Molecular Methods
For most laboratory methods, broad classification of positive and negative results can be easily defined; however, when one looks more closely at the analytical performance, precision becomes more challenging to describe. Typically, clinically significant cutoffs for HIV and HCV load tests are reported in a logarithmic scale or in international units (18, 19, 20, 21). It therefore requires a change of scale in order to import and adapt the traditional tools of statistical quality control. It
The Real-World Importance of Six-Sigma Quality for HIV Viral Loads
For HIV and HCV testing, there are test utilization scenarios with immediate, profound clinical and financial consequences. Therapies may be terminated or extended and patients may receive more or less care based on viral load test results. The clinicians and patients assume that the quality of the assay allows for the correct answer to be delivered with just one test, but given the new medical decision points expected for some viral load assays that assumption that should be confirmed through
The Real-World Importance of Six-Sigma Quality—HCV
In a study by Wiesmann et al. (28), HCV patients were monitored for a 2-log drop from baseline to ensure therapeutic response. As with HIV load methods, the diagnostic assay precision can directly impact whether or not a patient remains on HCV therapy. As therapies have become effective, the clinical decision points have migrated to the low end of the dynamic range, where accuracy and precision represent challenges for all real-time PCR assays.
Current HCV treatment guidelines state that
Discussion
Molecular assays have been utilized to manage HIV-1 and HCV therapy response for decades. Differences in accuracy and precision at clinical decision points where treatment decisions are being made and patient management is impacted have become a point of laboratory discussion. Historically, laboratories have challenged assay performance only across the full linear range (overall bias); however, there is a shift towards assessing assay performance near or at the clinical decision points. This
Conclusions
Six-Sigma analyses provide a useful tool to assay precision and overall quality for both HIV and HCV load testing. The Sigma metric translates abstract analytical performance characteristics into tangible measures that can impact laboratory operations and patient outcomes. As diagnostic methods and antiviral treatments have evolved, the demands for analytical performance of viral load testing have increased. In previous decades, extreme precision may not have been as necessary, but today's
Disclosure
Danijela Lucic is an employee of Abbott Molecular Inc.; Sten Westgard is a consultant for Abbott Inc.
References (29)
- et al.
Hepatitis C
Lancet
(2015) Prioritizing risk analysis quality control plans based on Sigma-metrics
Clin. Lab. Med.
(2013)Accuracy to 2nd International HIV-1 RNA WHO Standard: assessment of three generations of quantitative HIV-1 RNA nucleic acid amplification tests
J. Clin. Virol.
(2011)Preparation and evaluation of the 1st international standard for the quantitation of HIV-2 RNA in plasma
J. Virol. Methods
(2011)Standardization: a progress report
Biologicals
(1999)Evaluation of the COBAS® AmpliPrep/COBAS® TaqMan® HCV Test, v2.0 and comparison to assays used in routine clinical practice in an international multicenter clinical trial: The ExPECT study
J.Clin. Virol.
(2015)- Ambrosioni, J. et al. HIV treatment outcomes in Europe and North America: what can we learn from the differences?...
Treatment of chronic hepatitis C in patients with HIV/HCV coinfection
World J. Virol.
(2015)The role of mathematical modelling in the development of recommendations in the 2013 WHO consolidated antiretroviral therapy guidelines
AIDS
(2014)Health systems implications of the 2013 WHO consolidated antiretroviral guidelines and strategies for successful implementation
AIDS
(2014)
Efficacy of initial antiretroviral therapy for HIV-1 infection in adults: a systematic review and meta-analysis of 114 studies with up to 144 weeks' follow-up
PLoS One
Evidence-based consensus on the diagnosis, prevention and management of hepatitis C virus disease
World J. Hepatol.
Providing critical laboratory results on time, every time to help reduce emergency department length of stay: how our laboratory achieved a Six Sigma level of performance
Am. J. Clin. Pathol.
Collective opinion paper on findings of the 2009 convocation of experts on quality control
Clin. Chem. Lab. Med.
Cited by (5)
Statistics for Method Verification of Quantitative Assays in Clinical Microbiology
2019, Clinical Microbiology NewsletterCitation Excerpt :Such variability affects the assay's LLOQ. Assays with high precision even at the low end of quantitation can prove to be extremely variable for monitoring anti-retroviral and anti-HCV therapies [21]. It is crucial that assay performance be accurate at the predetermined assay tolerance limit (medical decision interval), defined as the difference between two different results that can be considered significantly different in terms of clinical care.
Management of Cytomegalovirus, Epstein-Barr Virus, and HIV Viral Load Quality Control Data Using Unity Real Time
2022, Journal of Clinical MicrobiologySelection of within-run quality control rules for laboratory biomarkers
2021, Statistics in MedicineQuantitative Molecular Methods
2016, Clinical Virology Manual, Fifth Edition