Huysal and Budak: Application of sigma metrics for the assessment of quality assurance using the MQ-2000 PT HbA1c analyzer

Introduction

HbA1c testing is important for the long-term control of glycaemia in diabetic patients. The diagnostic cut-off for HbA1c has been set at 48 mmol/mol, with patients being managed to maintain their HbA1c at ≤ 53 mmol/mol (1, 2). However, the methods currently used to measure HbA1c concentrations in clinical chemistry laboratories show large differences between their interlaboratory reported values, making comparisons of results from different laboratories difficult (3).

High analytical quality of the HbA1c test can be achieved by using commercial dedicated high performance liquid chromatography (HPLC) systems and monitoring their performance by means of QC programs with appropriate materials that will increase the operator’s confidence in their performance. Although current QCs in laboratory medicine tend to focus on the performance and efficiency of analytical processes, they still have room for improvement (4-6).

Errors in analytical measurements fall into two main categories: random and systematic errors. The typical analytical performance characteristics of inaccuracy and imprecision are basic parameters of method performance validation for systematic and random errors. These parameters are represented by the coefficient of variation (CV) and bias, and they can be used to calculate the total error (TE) (1). The total analytical error (TEA) is a useful measurement for assessing laboratory assay quality that combines the effects of systematic and random errors, which makes the validation results of complete analytical processes more evident (7).

Six sigma metrics combine bias, precision, and TEA, and can be used for assessing the quality of the analytic phase (8). The exact number of errors made can be quantified by employing sigma metrics in the laboratory. A high sigma value indicates that a laboratory’s reports of false test results are low. With the aid of Six Sigma principles and metrics, it is possible to ensure that the desired quality is achieved. Recently, the International Federation of Clinical Chemistry and Laboratory Medicine (IFCC) Task Force on the Implementation of HbA1c Standardization (TF-HbA1c) suggested a consensus statement regarding the quality targets for HbA1c with respect to the sigma metrics model as the model of choice (5).

In Turkey, public procurement laws include regulations concerning emphasis on the lowest bid according to unit prices. Therefore, decisions about which HbA1c analyzer to buy depends on price in our health system. The MQ-2000 PT HbA1c analyzer uses HPLC with ion exchange chromatography, and has been used for HbA1c quantification in our laboratory. Although there are no legally enforceable responsibilities, there is a need to evaluate the performance of the MQ-2000 PT HbA1c analyzer test results.

The aim of our study was to evaluate the analytical performance of the MQ-2000 PT HbA1c analyzer test results in the context of our operating conditions on the sigma scale.

Materials and methods

Study design

The present study was conducted in the clinical laboratory of the Şevket Yılmaz Research and Education Hospital. The study was approved by our institutional ethics committee and was in compliance with the Helsinki Declaration. The six-month (May to October 2013; N = 168 days) internal and external QCs (which were in May, July, September, and November) of HbA1c test results were extracted for this study.

Methods

The MQ-2000 PT HbA1c analyzer (Shanghai Hui Zhong Medical Technology Co., Ltd., China) uses high-performance liquid chromatography (HPLC) and separates HbA1c by cation exchange. The stationary phase is a weakly acidic cation exchange column, with a group that can exchange cations and can combine with non-glycated hemoglobin with a positive charge by electrostatic interactions. The glycated hemoglobin cannot combine with the stationary phase because it has no charge. The separated hemoglobin fractions are monitored by means of absorption of light, and the chromatogram obtained is recorded and stored by the internal computer. A software program in the instrument performs the analysis of the chromatogram. The HbA1c concentration is calculated as the percentage of the HbA1c peak area relative to the total hemoglobin peak area in the chromatogram. According to the manufacturer, the analyzer could detect HbA1c concentrations between 20 mmol/mol and 162 mmol/mol.

All reagents, calibrators, and controls used were provided by the manufacturer. Calibrations were performed as recommended by the manufacturer (Lot PT HbA1c12334C; Shanghai Huachen Biological Reagent Co. Ltd, China) (40.0 mmol/mol and 90 mmol/mol).

Two levels of internal QC materials, diabetes control Levels 1 (mean = 35.5 mmol/mol, range: 30.1–43.2 mmol/mol) and 2 (mean = 84.7 mmol/mol, range: 71.9–97.5 mmol/mol) (Lot 33850; Huachen Biological Reagent Co. Ltd, China), were assayed in duplicate twice daily during the study period. The manufacturer’s package inserts were followed, and open vial controls were stored at 2–8 °C for a maximum of seven days as per the manufacturer’s directions. The same lot numbers were used during the study period.

Internal statistical QC data for the study period were extracted from the records of the MQ-2000PT HbA1c analyzer. The CV (%) was calculated by using the mean from six months (N = 168) of internal QC.

Bias is derived from the external QC reports, which we participated in in the KBUD (Klinik Biyokimya Uzmanları Dernegi, Istanbul, Turkey) external QC program for HbA1c. Our results were compared to those of 32–38 laboratories using the HPLC with ion exchange chromatography method once every two calendar months.

The TEA is the amount a test result may deviate from the “true value” and still be acceptable. A TEA value of 10% for Hba1c was taken from Clinical Laboratories Improvement Act (CLIA) regulations, and sigma levels and critical systematic error were calculated. The size of the medically important systematic error is called the critical systematic error (9).

Statistical analysis

Data were evaluated using SPSS version 21.0 (SPSS Inc., Chicago, IL, USA). Means, standard deviations, and CVs were calculated. CV (%) was calculated from the internal QC data over the six-month period using the following equation:

CV (%) = (standard deviation × 100) / laboratory mean(IQC)

Bias was calculated from the external quality assessment records using the following formula:

Bias (%) = (mean of all laboratories using the same instrument and method - our mean) / (mean of all laboratories using the same instrument and method) × 100

Sigma levels were calculated using the formula as follows:

Process sigma = (% TEA - % “biasEQC”) / % CVIQC (5).

Assessment of the quality on the sigma scale of six provides an objective assessment of the analytical performance. The critical systematic errors were calculated by the following equation (9):

SEcrit = [(TEa - bias)/sD] – 1.65where the factor 1.65 was chosen to minimize the risk of erroneous test results at 5%.

Results

Two sets of quality control samples, normal (QC1) and pathological (QC2), were analyzed. Their laboratory means were 36.6 ± 2.38 mmol/mol and 84.7 ± 2.68 mmol/mol, respectively. Their calculated CVs were 6.5% and 3.1%, respectively.

We studied four external quality controls during the study period. In May and November, our Hba1c test results were 34.4 mmol/mol and 31.1 mean, while the corresponding biases were 0.8% and 5.5%, respectively. In July and September, Hba1c test results were 92.4 mmol/mol and 84.7 mmol/mol, while the corresponding biases were 9.7% and 1.1%. The average bias between the external quality control and the MQ-2000 PT during the study period was 4.2% (Table 1).

Table 1

External quality control data.

Period N* Lab’s Result Mean of group* Bias (%) MU
May
32
34.4
34.0
0.8
0.04
July
38
92.4
82.1
9.7
0.11
September
35
84.7
83.5
1.1
0.11
November
33
31.1
34.3
–5.5
0.08
N - Number of participants using the same method; MU - Measurement uncertainty
*Mean of the results of the laboratories using the HPLC method

The calculated average sigma metric was 1.19, and the critical systematic error was –0.46 (7). The traceability report from KBUD showed that the uncertainty of EQA samples was 0.06 for low control levels and 0.11 for high control levels, respectively. Three of the SDI values were 1.25 or less, which is considered acceptable.

Discussion

The IFCC Task Force on Implementation of HbA1c Standardization suggests initially setting a sigma of 2 for routine laboratories, which is the starting point for improved quality, and is suitable for estimating the quality status of HbA1c of a single laboratory and a single manufacturer. We obtained a sigma < 1.5 for both QC levels, showing that the analytical quality of the MQ-2000 PT HbA1c analyzer was not appropriate when evaluated on the sigma scale. A sigma of 2 is defined as the minimum allowable value for routine performance, based on the need to evaluate the ability of the device to effectively distinguish an HbA1c of 46 mmol/mol from an HbA1c of 47 mmol/mol, since two different treatment options exist at these two values. This means that patients are at risk of receiving a wrong clinical decision, especially if their HbA1c results were close to the clinical decision limit. Similar to our findings, it has been shown that a number of laboratories in Europe do not meet the 2-sigma criteria for HbA1c test results (5).

Analytical reproducibility in Hba1c measurement is important for monitoring diabetic patients. Changes in results between two HbA1c test results should reflect responses to treatment, and an optimal imprecision goal for HbA1c of 2.1% has been proposed (10, 11). This criterion is very strict, however, and difficult to meet, with an imprecision of 3% CV being a more realistic target (12). Neither of the calculated CVs at the two levels, for normal (QC1) and pathological (QC2) internal QCs, met these desired levels. If we used a precision value calculated for a shorter period of time, it could result in lower estimates of imprecision and higher sigma metrics, but we calculated a long-time analytical precision, which might explain our high CV. In our method with CV%s of 4%, results would need to differ by more than 0.8% to be confident that there has been a clinically significant change in glycemic status. Users should be informed not to change therapy on the basis of a small difference between two consecutive Hba1c values.

Both analytical bias and imprecision (CV) influence the probability of a laboratory passing a sigma of 2 criterion. To reliably detect clinically significant analytic errors, clinical laboratories should use HbA1c methods that have a bias that approaches zero in external quality control programs (5). As we focus on lower Hba1c levels that signal disease, we should be more alarmed that smaller fluctuations will change diagnoses. Our biases were very high, which is why we did not use recalibration to improve accuracy during EQAS tests. Also, the uncertainty in the calibration of our routine Hba1c procedure is traceable only to a manufacturer’s internal working calibrator.

One major limitation of this study was that the external quality control program in which we participated was based on a peer group rather than an accuracy program. Another limitation of this study is that we used calibrator and control material provided by the tested analyzer. Comparability can be assured only by the principle of traceability to higher-order standards, but the metrological traceability of the controls and calibrators were uncertain as to which responsibility of correction is primarily the manufacturer’s.

Conclusions

Standardization and monitoring of the analytical performance of HbA1c tests is critical, and there is a need for nationwide use of internationally standardized assays and a certification program for laboratories. Laboratories involved in the long-term testing of HbA1c must improve the quality of their analytical testing to ensure clinically valid results. The MQ-2000 PT HbA1c is a new analyzer in the market; there is need for improvement, and the method should be controlled with greater attention to ensure quality.

Notes

[1] Conflicts of interest None declared.

References

1 

Topić E. Guidelines and recommendations for testing in diagnosis of diabetes mellitus: The role of HbA1c. Biochem Med (Zagreb). 2014;24:S1–7.

2 

American Diabetes Association. Glycemic Targets. Diabetes Care. 2015;38:S33–40. https://doi.org/10.2337/dc15-S009

3 

CLRohlfingCAParvinDBSacksRRLittleNGSP Steering Committee. Comparing analytic performance criteria: evaluation of HbA1c certification criteria as an example. Clin Chim Acta. 2014;433:259–63. https://doi.org/10.1016/j.cca.2014.03.034

4 

Sandberg S, Fraser CG, Horvath AR, Jansen R, Jones G, Oosterhuis W, et al. Defining analytical performance specifications: Consensus statement from the 1st strategic conference of the European Federation of Clinical Chemistry and Laboratory Medicine. Clin Chem Lab Med. 2015;53:833–5. https://doi.org/10.1515/cclm-2015-0067

5 

Weykamp C, John G, Gillery P, English E, Ji L, Lenters-Westra E, et al. IFCC Task Force on Implementation of HbA1c Standardization. Investigation of 2 models to set and evaluate quality targets for HbA1c: biological variation and sigma-metrics. Clin Chem. 2015;61:752–9. https://doi.org/10.1373/clinchem.2014.235333

6 

Njoroge SW, Nichols JH. Risk management in the clinical laboratory. Ann Lab Med. 2014;34:274–8. https://doi.org/10.3343/alm.2014.34.4.274

7 

Hens K, Berth M, Armbruster D, Westgard S. Sigma metrics used to assess analytical quality of clinical chemistry assays: importance of the allowable total error (TEa) target. Clin Chem Lab Med. 2014;52:973–80. https://doi.org/10.1515/cclm-2013-1090

8 

Westgard JO, Westgard SA. Assessing quality on the Sigma scale from proficiency testing and external quality assessment surveys. Clin Chem Lab Med. 2015;•••: Epub ahead of print https://doi.org/10.1515/cclm-2014-1241

9 

Westgard JO, Westgard SA. The Quality of Laboratory Testing Today: An Assessment of Sigma Metrics for Analytic Quality Using Performance Data From Proficiency Testing Surveys and the CLIA Criteria for Acceptable Performance. Am J Clin Pathol. 2006;125:343–54. https://doi.org/10.1309/V50H4FRVVWX12C79

10 

Braga F, Dolci A, Montagnana M, Pagani F, Paleari GC, Guidi GC, et al. Revaluation of biological variability of glycated haemoglobin using an accurately designed protocol and assay traceable to the IFCC reference system. Clin Chim Acta. 2011;412:1412–6. https://doi.org/10.1016/j.cca.2011.04.014

11 

Goodall I, Colman PG, Schneider HG, McLean M, Barker G. Desirable performance standards for HbA1c analysis–precision, accuracy and standardization: Consensus statement of the Australian Association of Clinical Biochemists(AACB), the Australian Diabetes Society (ADS), the Royal College of Pathologists of Australia (RCPA, Endocrine Society of Australia (ESA), and the Australian Diabetes Educators Association(ADEA). Clin Chem Lab Med. 2007;45:1083–97. https://doi.org/10.1515/CCLM.2007.158

12 

Sacks DB, Arnold M, Bakris GL, Bruns DE, Horvath AR, Kirkman MS, et al. Executive summary: guidelines and recommendations for laboratory analysis in the diagnosis and management of diabetes mellitus. Clin Chem. 2011;57:793–8. https://doi.org/10.1373/clinchem.2011.163634