Introduction
Health laboratory services are a critical component of national health systems and are central to disease diagnosis, treatment, prevention and outbreak investigation (1). Laboratories meeting International Organization for Standardization (ISO) 15189:2012 standards are required to participate in inter-laboratory comparisons through external quality assessment (EQA). The World Health Organization (WHO) defines EQA as a system for objectively checking a laboratory’s performance through proficiency testing (PT) schemes, rechecking samples or on-site assessment. Participation in EQA detects systematic laboratory errors and assesses competency of technical personnel. EQA should address all testing procedures performed in the laboratory (2).
In primary healthcare settings in East African countries, laboratory testing may influence 45% of medical decision-making (3). However, laboratory services in resource-limited settings are often poorly managed and provide results of unknown quality; many clinicians regard laboratory testing as unhelpful (4). The common perception that EQA is costly or unnecessary has hindered widespread enrolment of laboratories into these programmes (5).
PT schemes submit samples with undisclosed results to groups of laboratories for analysis. International and commercial PT schemes are costly and may not address the range of basic tests and pathology appropriate for resource-limited settings. The East African Regional External Quality Assessment Scheme (EA-REQAS) was established in 2004 through a joint collaborative initiative of the ministries of health of Kenya, Mainland Tanzania, Uganda and Zanzibar to establish regional standards and provide an appropriate PT scheme for peripheral laboratories (6); Burundi’s ministry of health joined the scheme in 2011. A committee comprising two members from each ministry of health was established to govern scheme operations. Amref Health Africa, a health development non-governmental organisation with an established laboratory programme, was appointed to coordinate the scheme on behalf of the ministries of health.
Materials and methods
Materials for PT panels were selected based on the following criteria: tests of clinical and public health importance; techniques of accepted accuracy; and tests for which stable materials are available. Materials were prepared according to standard procedures developed by country technical working groups that included stability testing for four months at 4 °C, ambient temperature and 37 °C. Materials were: serum for human immunodeficiency virus (HIV) and syphilis serology; stained slides for blood parasites; fixed sputum smears for acid-alcohol-fast bacilli (AAFB); fixed bacteriological smears for Gram stain; fixed stool parasites; stained slides for peripheral blood film examination; blood lysate for haemoglobin measurement. Survey materials were initially prepared in selected reference laboratories in the region; from Survey 10 materials were prepared at the Amref Health Africa Central Laboratory, Nairobi, Kenya. For each survey, four clinical scenarios were created with three laboratory theory questions, four clinical questions and two public health questions (Table 1). Each panel contained seven different materials including serum for HIV, sputum smear, blood slide for parasites and blood lysate; three other materials were variable. Panels included negative samples. Five clinical laboratories in Nairobi accredited to ISO 15189:2012 validated materials to establish target values for each survey. The EA-REQAS website was developed in 2010 to provide scheme information and online return of results. In December 2013, a short messaging service (SMS) with standard low cost across the region (0.015 US dollars per message) was instituted to send reminders one week before and on the day of survey distribution, and one week before survey closure. Telephone contact was made for delayed responses. Some facilities attended one-day sensitisation meetings as part of other project activities.
Table 1
A sample clinical scenario incorporating clinical, laboratory and public health questions
Results were marked according to pre-prepared marking keys. Qualitative results were positive or negative (3 marks for correct results). Semi-quantitative results (AAFB grading; malaria parasite counting) and quantitative results (haemoglobin measurement) were scored with 1–3 marks according to the deviation from target values (Table 2). Results with dangerous clinical implications scored one negative mark. Facilities were requested to provide numbers and qualifications of health facility staff, and information on equipment, power sources, laboratory methods and testing kits. All received data were entered into an Excel master database for analysis. Automatically generated immediate feedback reports were submitted to individual laboratories within 30 days of survey closure with recommendations for remedial action. Composite reports including observations and recommendations were sent to national quality assurance offices within 60 days of survey closure.
Table 2
Marking schemes for quantitative and semi-quantitative results
Data were analysed using InStat version 3.0 for Windows statistical software (GraphPad Software, Inc., San Diego, USA); Student’s t-test was used to determine probabilities.
Results
Sixteen surveys were submitted biannually from 2008 to 2015. The initial 195 enrolled facilities increased to 560 by survey 16; participating health facilities were government, faith-based and privately owned. Mean survey response rate dropped from 66% in survey 2 to 40% in survey 9 but increased to 59% in survey 16 after instituting SMS reminders. Turnaround time was above the expected 30 days in all surveys except in survey 15 (24 days) with a high of 86 days (survey 10) but showed an overall decreasing trend.
Mean performance increased from 51% to 68% (range 46% in survey 3 to 68% in survey 16). Individual facilities scoring the accepted PT pass score of 80% increased from 0 in survey 1 to 83 (25%) in survey 16. Private health facilities performed better (61%) than government and faith-based facilities (58% and 57%, respectively; P = 0.177); hospitals performed better than health centres (60% vs. 56%; P = 0.109). Facilities participating in ≥ 10 surveys had a significantly higher mean performance than those participating in ≤ 5 surveys (mean ± SD = 60.6 ± 7.9 and 53.8 ± 13.9; P = 0.001). Mean performance in laboratory, clinical and public health questions were 62%, 51% and 57% respectively (data available from survey 6). Laboratory performance was significantly better than clinical performance (P = 0.003).
The four tests most poorly performed overall were haemoglobin measurement, peripheral blood film examination, Gram stain and examination of bacteriological samples, and malaria parasite counting. Haemoglobin methodologies were compared by grouping into three major categories: auto-analyser, manual colorimetric and visual comparator methods. Taking acceptable performance as < ± 10 g/dL from the target value, performance using visual comparator methods was significantly inferior (χ2 = 48.291; P < 0.001).
Discussion
Several changes were made to the scheme’s operating model during the first eight years of implementation. From survey 6 courier services were contracted to improve timeliness of package delivery and return of paper-based reports using a pre-paid system, but this added 32% to operating costs.
To explore reasons for delayed and non-responses, a telephone survey was conducted in 2011 in 47 facilities in Kenya. Reasons given included non-completion of questions, reagent stock-outs and personnel transfers; these are being addressed through SMS reminders and dialogue with laboratory personnel and authorities. Due to limited internet access, few facilities respond online, but the number is gradually increasing. Most poorly performed tests require accurate microscopy recognition skills, indicating need for continuing refresher training. Recommendations to discontinue visual comparator methods for haemoglobin measurement were made to ministries of health. Poor performance in clinical results may be due to lack of support by clinicians.
Following start-up funding support from WHO and Izumi Foundation, USA, financial support for EA-REQAS operations now depends on income generated from participating laboratories; EA-REQAS charges USD 200 for two surveys per year. Participation in PT remains optional for laboratories in this region; however, participation is increasing as more laboratories seek accreditation. EA-REQAS is in the process of achieving ISO 17043:2010 accreditation and plans are underway to increase survey frequency and range of materials.
In conclusion, PT is a valuable tool for evaluating laboratory performance, identifying poorly performing areas, assessing instruments and test kits, and influencing national policy. Identified deficiencies need to be corrected for laboratories to benefit from participation. Locally coordinated PT schemes provide relevant survey materials, promote appropriate remedial measures and engage in dialogue with relevant authorities. Governments and health facilities need to appreciate the value of PT participation, and introduce regulations and budgetary provisions to support mandatory participation.
