Introduction
Clinical laboratories are centres that take a very important place in diagnosis, treatment, and follow-up of diseases and are therefore expected to provide the highest possible quality services. The effect of the data provided by the clinical laboratories on all health care decisions affecting diagnosis and treatment is often quoted as being approximately 70%, although varies with the clinical specialty and application (1, 2). Hence, the interest in quality management tools/methodologies is gradually increasing to achieve quality and accurate results in line with international standards in the clinical laboratory. One of the most prominent of these methodologies, Six Sigma, is a quality management tool based on statistical calculations, focused on process variables, and which provides objective information about process performance (3). First implemented by Motorola in the 1980s, this methodology was adapted by Nevalaien et al. and Westgard to clinical laboratories in the early 2000s (4, 5). However, there is still ongoing debate today about the fact that adaptation of this industry-based methodology to clinical laboratories does not fully reflect the methodology and has disadvantages/weaknesses (6-9). Six Sigma in clinical laboratories traditionally combines three main components: total allowable error, bias and precision (10). Although Six Sigma has advantages, such as providing evaluation on an international scale, simplifying all parameters in a single table using method decision charts, developing a total laboratory quality plan, including the number and frequency of internal quality controls; it also has some defects such as having theoretical-based problems like 1.5 standard deviation (SD) shift, the fact that an acceptable sigma can hide the unacceptable constituents of the system, and, most importantly, tolerance limits that have not been standardized and whose limits have not been determined well for many years (8, 11, 12).
Regardless of the above-mentioned pros and cons of Six Sigma, the new biological variation (BV) data, which the European Federation of Clinical Chemistry and Laboratory Medicine (EFLM) published in 2015 and presented its results towards the end of 2019, has raised new questions about the applicability of the sigma methodology (13, 14). Although the lack of standardization of the design and criteria of the studies used to obtain Ricos BV database, last updated in 2014, constitutes a problem for the reliability of the data, the narrow limits of the new EFLM BV database bring the question with current in vitro diagnostic (IVD) technologies: “Is BV now a realistic goal for Six Sigma?”
In our study, Six Sigma values for our laboratory were calculated for 21 routine biochemistry parameters by using Roche Cobas c 702 (Roche Diagnostics GmbH, Mannheim, Germany) analyser. The analyser’s Six Sigma values were calculated by using precision and method comparison data of the reagents used in these measurements in package inserts provided by the manufacturer and compared with the sigma values of our laboratory. In this way, we aimed to evaluate the applicability of analytical quality goals with Roche Cobas c 702 analyser and to determine analytical goals depending on the analyser used.
Materials and methods
Study design
In our study, a total of 21 parameters, including alkaline phosphatase, albumin, alanine aminotransferase (ALT), aspartate aminotransferase (AST), amylase, direct bilirubin, total bilirubin, gamma-glutamyltransferase, glucose, high density lipoprotein cholesterol (HDL), calcium, chloride, cholesterol, creatinine, lactate dehydrogenase (LD), potassium, triglyceride, total protein, sodium, urea and uric acid were analysed using the Roche Cobas c 702 (Roche Diagnostics GmbH, Mannheim, Germany) analyser. The calculations of our study were examined under two main headings as Sigmalaboratory and Sigmaanalyser.
Sigmalaboratory
Internal quality control (IQC) results from October 2019 to December 2019 were used for our laboratory’s coefficient of variation (CV%) calculation. PreciControl Clin Chem Multi 1 and Multi 2 (level 1 lot no:250280, level 2 lot no: 324196, Roche Diagnostic GmbH, Mannheim, Germany) IQC materials were carried out two times per day. Bias estimation of all analytes were determined by twelve months of data obtained during a cycle from the external quality control program (KBUDEK, Turkey) between January 2019 and December 2019. Bias calculation was based on peer-group values using the same device and method in the external quality control (EQC) program. Among the evaluated EQC results, for all analytes, only 8 results were found outside the acceptable limits during one cycle, and were excluded from the calculation of bias, and the overall EQC success was above 95%. Regulatory-preventive action was initiated for each excluded result, and the sequent EQC result was found within acceptable limits.
Sigmaanalyser
Six Sigma calculation of the analyser was based on the manufacturer data presented in the package inserts of the reagents used in our laboratory. Accordingly, precision data of Roche PreciControl Clin Chem Multi 1 and Multi 2 internal quality control materials in the Roche Cobas c 702 reagent package inserts were used for the analyser’s CV% calculation and this process was applied for each analyte separately. Precision data used in the study was obtained by the manufacturer as “3 aliquots per run, one run per day, 21 days “ in an internal protocol. Similarly, for bias estimation, the method comparison studies presented in reagent package inserts, and made using human serum/plasma, were used for each analyte separately. This study was carried out by the manufacturer as Roche Cobas c 701 analyser (y) were compared with those determined using the corresponding reagent on a Roche Cobas c 501 (Roche Diagnostics GmbH, Mannheim, Germany) analyser (x). The calculation was based on the linear regression equation y = ax + b given as dependent on concentration. During the working period, reagents with the same brand and the same catalogue number were used on the analysers. Ethical approval was obtained from the Ethics Committee of Bursa City Hospital (E-13012450-514.01.05).
Statistical Analysis
Total allowable error (TEa) was obtained from the data of Clinical Laboratory Improvement Amendments (CLIA) 2019 and the EFLM BV database (14, 15). Total bilirubin, direct bilirubin, uric acid, and calcium, whose data are not available in the EFLM BV database, were based on the relevant EuBIVAS study (16).
The CV% of our laboratory was calculated with the equation given below, using the quarterly IQC data: CV% = (SD / xmean) x 100. The Bias% of our laboratory was calculated with the equation 1 as shown below, using twelve-month EQC data of the same cycle:
In the Sigmaanalyser calculation based on the manufacturer’s declaration, CV% and Bias% were calculated based on the study data provided in the package inserts.
The formula (TEa - bias%) / CV% was used for all six sigma calculations. In our study, > 3 sigma values were accepted as the minimum allowable performance.
Results
Analytical performance characteristics obtained during the analysis of analytes in our laboratory are presented in Table 1. Analytical performance characteristics of analyser calculated according to the manufacturer’s data in the reagent package inserts are presented in Table 2. In all the parameters studied, it was observed that the CV% values in our laboratory were higher than the package insert data declared by the manufacturer. The calculated sigma values for all parameters are presented in Table 3, and the “Normalized Sigma-Metric Method Decision Chart” prepared according to these sigma values are given in Figures 1-4Figure 2Figure 3Figure 4. According to both Sigmalaboratory and Sigmaanalyser values obtained; BV goals showed higher sigma performance than CLIA targets in 8 analytes, including ALT, amylase, total bilirubin, direct bilirubin, GGT, triglyceride, uric acid, and urea, while CLIA goals showed higher sigma performance in the remaining 13 analytes. Furthermore, from an overall perspective, Sigmaanalyser values showed acceptable performance above > 3 sigma on all CLIA goals, while Sigmalaboratory values did not show acceptable performance on more than half of the evaluated analytes according to the desirable BV goals. While ALT, amylase, AST, bilirubin direct, bilirubin total, triglycerides, and uric acid showed adequate performance by achieving >3 sigma goal in all quality goals, sodium showed acceptable performance only with Sigmaanalyser values according to CLIA goals. Calcium failed to meet minimum sigma quality performance at only Sigmalaboratory level 2 according to CLIA goals.
Table 1
Analytical performance characteristics of analytes in our laboratory
Table 2
Analytical performance characteristics of analytes according to the manufacturer’s data in the reagent package inserts
Table 3
Sigma values of our laboratory and reagent package inserts
Figure 1
Our Laboratory “Normalized Sigma-metric Method Decision Chart” according to CLIA goals. Alb - Albumin. ALP - Alkaline phosphatase. ALT - Alanine aminotransferase. Amy - Amylase. AST - Aspartate aminotransferase. DBIL - Bilirubin, direct. TBIL - Bilirubin, total. Ca - Calcium. Chol - Cholesterol. Cl - Chloride. Crea – Creatinine. GGT - Gamma-glutamyltransferase. Glc - Glucose. HDL - High-density lipoprotein cholesterol. LD - Lactate dehydrogenase. K - Potassium. TP - Total protein. Na - Sodium. TG - Triglycerides. UA - Uric acid. ● – Unacceptable. ■ – Poor. ▲ – Good. ♦ - Marginal. - Excellent. - World Class.
Figure 2
Our Laboratory “Normalized Sigma-metric Method Decision Chart” according to desirable BV goals. . Alb - Albumin. ALP - Alkaline phosphatase. ALT - Alanine aminotransferase. Amy - Amylase. AST - Aspartate aminotransferase. DBIL - Bilirubin, direct. TBIL - Bilirubin, total. Ca - Calcium. Chol - Cholesterol. Cl - Chloride. Crea – Creatinine. GGT - Gamma-glutamyltransferase. Glc - Glucose. HDL - High density lipoprotein. LD - Lactate dehydrogenase. K - Potassium. TP - Total protein. Na - Sodium. TG - Triglycerides. UA - Uric acid. ● – Unacceptable. ■ – Poor. ▲ – Good. ♦ - Marginal. - Excellent. - World Class.
Figure 3
Roche Cobas c 702 reagent package insert’s “Normalized Sigma-metric Method Decision Chart” according to CLIA goals. Alb - Albumin. ALP - Alkaline phosphatase. ALT - Alanine aminotransferase. Amy - Amylase. AST - Aspartate aminotransferase. DBIL - Bilirubin, direct. TBIL - Bilirubin, total. Ca - Calcium. Chol - Cholesterol. Cl - Chloride. Crea – Creatinine. GGT - Gamma-glutamyltransferase. Glc - Glucose. HDL - High density lipoprotein. LD - Lactate dehydrogenase. K - Potassium. TP - Total protein. Na - Sodium. TG - Triglycerides. UA - Uric acid. ● – Unacceptable. ■ – Poor. ▲ – Good. ♦ - Marginal. - Excellent. - World Class.
Figure 4
Roche Cobas c 702 reagent package insert’s “Normalized Sigma-metric Method Decision Chart” according to BV goals. Alb - Albumin. ALP - Alkaline phosphatase. ALT - Alanine aminotransferase. Amy - Amylase. AST - Aspartate aminotransferase. DBIL - Bilirubin, direct. TBIL - Bilirubin, total. Ca - Calcium. Chol - Cholesterol. Cl - Chloride. Crea – Creatinine. GGT - Gamma-glutamyltransferase. Glc - Glucose. HDL - High density lipoprotein. LD - Lactate dehydrogenase. K - Potassium. TP - Total protein. Na - Sodium. TG - Triglycerides. UA - Uric acid. ● – Unacceptable. ■ – Poor. ▲ – Good. ♦ - Marginal. - Excellent. - World Class.
Discussion
It is well known that analytical quality management in clinical laboratories is of significant importance. Six Sigma is a global quality management tool that can be used in clinical laboratory processes and allows the detection of errors quantitatively according to a specific plan. However, there are question marks about the determination and applicability of analytical quality specifications in this methodology. In this context, the sigma values of the actual performance of our laboratory and the manufacturer’s data presented in the reagent inserts of the Roche Cobas c 702 analyser used were evaluated in our study. First of all, if we examine the analytical performance characteristics used in six sigma calculations; the CV% values of our laboratory for all analytes were higher than the CV% values presented in the package inserts. This is mainly due to the fact that the CV% data presented in the package inserts of reagents were short-term analytical performance evaluations obtained from three-week intermediate precision studies. The precision study in our laboratory was obtained from the standard quarterly QC data, and the reagent lot to lot variation, temperature, humidity, etc. variables are thought to be the main factors that yields this result. Additionally, it is believed that manufacturer’s data are often optimistic studies that can be considered experimental rather than conducted in ideal working conditions and with routine work, which also contributed to this outcome (17).
All of the bias values obtained in our laboratory contributed less to the six sigma budget than the CV% values. This is thought to be due to the use of long-term bias values in six sigma calculations. Many effects cause within-day bias to become random effects in the long term. Therefore, during extended periods (weeks and months) of observations many bias components vary and thus increasingly contribute to the random error component (18). Surely, the best way to estimate bias is to use the reference material/reference method. However, for many analytes, there is no reference method or material and it is difficult for laboratories to reach.
When we evaluated the sigma values obtained secondarily and examined the analytical performance of our laboratory according to CLIA goals, it was observed that creatinine at level 1; calcium and potassium at level 2; and chloride, sodium, and urea at both levels, were out of acceptable limits in 9 out of a total of 42 goals. Furthermore, when we evaluated the sigma values of our laboratory according to desirable BV goals, it was observed that 24 out of a total of 42 goals showed unacceptable performance. On the other hand, when we examined Roche Cobas c 702 sigma values according to the package insert data, it was observed that all analytes performed at an acceptable level according to CLIA goals; while creatinine at level 1, albumin, calcium, chloride, protein, and sodium were observed to perform at an unacceptable level in 11 out of a total of 42 goals at both the two levels according to the BV goals. Moreover, minus sigma values were obtained at level 1 for calcium and level 2 for chloride and sodium in these analytes.
Sodium, chloride and calcium are physiologically controlled in a strict and narrow range. Therefore, the fact that BV goals are challenging is to be expected. Although the data that the manufacturer obtained in intermediate precision and method comparison studies under optimal conditions were used in our study, desirable goals based on BV for these three analytes could not be met. This indicates that the IVD technologies used for these analytes need to be improved in order to achieve BV-based goals. With the IVD technology we currently use, it is believed that CLIA goals are more reasonable and feasible for these analytes, while BV goals would be a waste of time and effort for these analytes. Of the analytes evaluated, albumin and total protein showed acceptable > 3 sigma performance in all of the CLIA goals; on the contrary, they showed an unacceptable performance in all of the BV-based goals. Therefore, the BV-based goals evaluated were more challenging for albumin and total protein. In our study, BV goals were wide and easily achievable with today’s IVD technologies for bilirubin direct, bilirubin total, and triglycerides.
As can be seen from the results mentioned above, even if the reagent insert data provided by the manufacturer for routine clinical chemistry tests were used in our study, which uses the Six Sigma methodology, it was observed that more than half of the analytes failed to achieve above > 6 sigma “world-class” quality and almost a quarter of the evaluated analytes failed to achieve even above > 3 sigma “acceptable” level quality according to BV-based analytical quality goals.
Although numerous studies in the literature use the six sigma methodology, very few studies use data similar to our study. Recently, Westgard published a text on its website similar to our study using the Abbott analyser (19). This study was based on the data of the article published by Westgard et al. in 2017 (20). However, unlike our study, this study was conducted by calculating imprecision according to the EP05-A2 guideline and by calculating the bias according to the EP09-A3 method comparison study, instead of using the manufacturer’s data. In this study, similar to our study, calcium, creatinine, chlorine, total protein, and sodium showed < 3 sigma performance, while, unlike our study, LD and potassium showed unacceptable performance < 3 sigma according to EFLM BV-goals. This work was interpreted, as “There are currently no instruments on the market that are really capable of achieving consistently high sigma performance with these goals for calcium, chloride, sodium, and creatinine.” Again, in a study conducted by Westgard using Mindray BS-200 analyser manufacturer reagent data and published by Westgard on his website, he stated that more than 50% of the analytes evaluated showed unacceptable performance according to EuBIVAS standards (17).
In another study, Geto et al. evaluated Roche Cobas 6000 analyser using six sigma methodology and up to date CLIA 2019 goals (21). They failed in eight quality goals, as our in study, based on the analytes involved in our study. When we examined analytes with unacceptable performance, urea, chloride, and sodium failed at both levels, similar to our study. Unlike our study, cholesterol at level 2 and creatinine at level 1 showed unacceptable performance. As a result, most test results were in agreement.
The primary purpose of using the sigma metric in clinical laboratories is to determine the appropriate quality procedures for the data obtained. Nevertheless, whichever statistical quality control methodology is used in clinical laboratories, especially in six sigma, the chosen total allowable error source is the main point to achieving the performance and directly shapes the QC plans. Quality plans prepared for the analytical process generally involve a cycle that starts with determining the “analytical quality goal” and continues with the evaluation, development, and control of the process. At this point, Six Sigma offers a significant advantage, such as setting a quantitative, objective, and comparable “quality baseline” (10). However, the “quality baseline” to be based for the analytical process in the clinical laboratory varies according to the selected TEa source and is far from standardization. It is thought that this situation raises the question marks at the stage of goal setting, which is the very first step of the analytical quality control plan and restricts the advantages that Six Sigma offers for the process. In this context, if we examine in detail the concept of TEa, which constitutes the first step of all quality control methodologies, and the evolution of analytical quality specifications, which was initiated in the Milan Consensus in the middle of the past decade to obtain more reliable and up-to-date analytical quality specifications, it seems to have been largely completed with the biological variation data bank announced lately by EFLM (14, 22). Of the analytical performance specifications compiled in the three main criteria, “clinical outcomes,” which is at the top of the hierarchy, is an ideal goal; however, it can only be applied to a limited number of analytes with the available data. This indirectly moves BV goals, which are in the second place for most analytes, to the top of the hierarchy. At the bottom of the hierarchy, there are quality goals described as “state-of-the-art” and at which technological feasibility is at the forefront. These goals are not based on any documentation or formula generally defined by regulatory institutions and various external quality assessment programs. On the other hand, the reliability problem of BV data used to produce BV-based analytical quality goals, which are at the top of the hierarchy for most analytes, was largely resolved after the publication of the EFLM BV data bank. However, BV-based analytical quality targets expressed with a single number are theoretically weak as combining the two maximum values (bias and precision) to generate a single expression, resulting in an overestimation of TEa (23). Besides, the issue of whether the BV-based TEa can be considered a “Model 2” according to the Milan consensus must be debatable as well. This model is based on the statement also mentioned in the Milan Consensus that “This model attempts to minimize the ratio of ‘analytical noise’ to the biological signal.” (22). Therefore, this is theoretically only the case when, as in measurement uncertainty, “acceptable imprecision” is used as an analytical quality goal. As for TEa, it is not probable to mention the “analytical noise to the biological signal” ratio methodologically because two analytical quality goals are linearly combined to create a single expression. Therefore, in methodologies using the traditional TEa model in BV-based analytical quality goals such as TEa and Six Sigma, it is considered that there is no exact equivalent of the Model 2 specified in the Milan Consensus. The above-mentioned situations are thought to cause clinical laboratories to spend unnecessary effort to achieve the goals of a theoretically weak TEa model, as well as to resort to cost-increasing practices such as increasing the number of controls and calibrations.
Six sigma applications in clinical laboratories enable not only to conduct processes according to a specific plan but also to identify possible sources of error and improve the quality of the process. Furthermore, considering the sigma values obtained, sophisticated and dynamic quality control plans can be generated using “risk-based SQC” procedures (11). In this way, Six Sigma allows us to look at the laboratory from a broader perspective, focus on the analytes that need to be improved and avoid financial losses such as wasting time, effort, control and reagent costs. Such benefits of Six Sigma can only be fully obtained when applied with well-defined and achievable quality goals with existing technologies. Although it may seem in today’s conditions that it is the most reasonable way to set quality goals for each analyte separately considering the analyte-based, it is also inevitable that this will lead to subjective evaluations that differ from laboratory to laboratory.
In conclusion, this study demonstrates that it is utopian to achieve BV goals for almost a quarter of analytes evaluated, which cannot be achieved even with manufacturer reagent insert data. The “state of the art” goals for the Six Sigma methodology are now considered to be more reasonable, achievable, and compatible with today’s technologies; however, the importance of the BV goals should not be overlooked in terms of their role in encouraging IVD manufacturers to improve their existing technologies/methods. Furthermore, regardless of the CLIA and BV goals, it is believed that firstly documenting achievable goals specific to the analyser used and determining the upper limit of the goal to be determined based on the analyte accordingly will be of enormous benefit to the laboratories at the stage of analytical goal setting.
