The most common nonconformities encountered during the assessments of medical laboratories in Hong Kong using ISO 15189 as accreditation criteria

 

Before the publication of the international standard ISO 15189 for accreditation of medical laboratories, there was no official accreditation system for accrediting medical laboratories in Hong Kong. There are over 100 medical laboratories in Hong Kong including a large number of private laboratories as well as many in the public hospitals administered under the Hospital Authority, nevertheless, medical laboratories in Hong Kong are not required to be registered with the health authority or accredited for their operation. Standards of many medical laboratories in Hong Kong are of high quality and are well recognized by the international medical community through the regular participation of their staff in international conferences and publications in peer reviewed journals by the medical professionals of Hong Kong.

Hong Kong Accreditation Service (HKAS) is the only government run accreditation body in the economy but it did not offer accreditation for medical testing laboratories before 2004. HKAS has over 20 years of experience in providing accreditation to the non-medical testing laboratories and calibration laboratories and is a signatory to the mutual recognition arrangement of International Laboratory Accreditation Cooperation (ILAC) and Asia Pacific Laboratory Accreditation Cooperation (APLAC). Before the introduction of the accreditation programme for medical laboratories in 2004, laboratories were accredited using ISO/IEC 17025 and these did not include the medical testing laboratories. Some large hospital laboratories that enjoy international reputation sought professional recognition of their competence through accreditation by overseas accreditation bodies such as College of American Pathologists (CAP) or National Australian Testing Authorities (NATA) that have a long history of accrediting medical laboratories. Internationally, ISO/IEC 17025 has not been well accepted by the medical community as suitable for accrediting medical laboratories and so the ISO technical committee TC212 was given the task to draft a standard particularly for medical laboratories.Since ISO15189 first published in 2003, it has been widely used as the accreditation standard for medical laboratories internationally by various accreditation bodies (1-5). In 2004, the laboratory accreditation programme of HKAS was extended to cover medical testing laboratories using the new accreditation standard soon after its publication.

At that time accreditation is a new concept to most medical testing laboratories in Hong Kong. Most laboratories did not operate a management system before 2004. Medical laboratory practitioners were not accustomed to writing down their operations and the concept of record traceability was not fully understood. Internal auditing was unheard of. Though internal quality controls were run in routine testing and external quality assurance programmes were participated by most laboratories, the practice and frequency of participation varied greatly from laboratory to laboratory. Evaluation of new methods and new autoanalysers were carried out to various extents by large laboratories, while for small laboratories, new equipment at times were used immediately after installation without any method validation. Since the launching of the accreditation programme, laboratory personnel were trained in ISO 15189, and throughout the last few years, there had been obvious improvements in meeting the ISO 15189 requirements.

ISO 15189 is an accreditation standard prepared particularly for the medical laboratories. This standard is expected to help medical laboratories in raising the quality of services provided.

The aim of this study was to assess whether ISO 15189 contributed to the quality improvement of medical laboratories in Hong Kong and the laboratories had found particular difficulties to comply with which requirements of ISO 15189. The frequency of nonconformities (NCs) to requirements of the ISO 15189 accreditation standard, encountered during the assessments of medical laboratories in Hong Kong, during 2004 to 2009, was analyzed. This would show which are the more difficult requirements found by most laboratories and shorten the learning path of the medical laboratories. By looking at the change in number of NCs encountered in subsequent assessments of the same group of laboratories, the effectiveness of this new standard in improving the quality of accredited laboratory service is reviewed.

The types and number of NCs reported by the assessment teams in assessments of medical laboratories accrediting to ISO 15189 were reviewed in two periods and their performances were compared. The medical accreditation programme of HKAS started in 2004, the number of application for accreditation went slow in the first two years as the laboratories got themselves prepared but sharply increased in the third and fourth year. Data from 73 assessments in 2009 were compared to data obtained in an earlier period (2004-2006) when the accreditation programme was at its early stage. Hong Kong is an ideal place to demonstrate the usefulness of accreditation to ISO 15189 in raising the quality of services provided as accreditation is not mandatory in our community and it is not related to insurance claims. Improvement and changes are all self-initiated. Because accreditation is a new concept to most laboratories, they are excited to set up a management system that could meet international standards and they eagerly improve themselves in order to be accredited and recognized.

 

 

 

The data for this retrospective observational study were retrieved from 2004 to 2009. From 2004 to 2006, HKAS had conducted assessments of 40 laboratories of different pathology disciplines using ISO 15189:2003 (6). Number of NCs reported were counted and categorized into “significant” or “minor” grading or reported as “recommendations” in each assessment. The number of NCs in each category reported during the assessments was grouped according to the clause number of ISO 15189 and their frequency of occurrence was presented as percentage of total no. of NCs reported for the laboratories assessed in the study period. Performance of the laboratories was presented in terms of average number of NCs per laboratory. Data collected from 2004 to 2006 represented performance of laboratories undergoing the first few assessments and are at learning phase of the ISO 15189 requirements.

In 2009, the number of laboratories assessed against requirements of ISO 15189:2007 (7) increased to 73. Similar types of data (total no. of NCs reported; the frequency of occurrence of NCs to requirements of ISO 15189 in terms of counts and percentages; and average number of NCs per laboratory for each category) were collected from these 73 laboratories assessed in 2009. Among these 73 laboratories, 61 of them had been accredited before 2008 (included those laboratories being initially assessed from 2004 to 2006), and were either having their surveillance visits or reassessments in 2009, whereas 12 laboratories had their initial assessments in the year. Among the 61 accredited laboratories, 27 were having their second reassessments in 2009, hence data collected from these 27 laboratories represented a more mature phase of implementing the management system established in accordance with the ISO 15189 requirements after being accredited for a few years. The performance of these 27 laboratories, in terms of number of NCs per laboratory, in their initial assessment was compared with their performance in the second reassessment in 2009. These figures would indicate whether accredited laboratories had any improvement in the quality of services provided after accreditation.

 

A significant nonconformity is one which has serious adverse effect on the validity of an activity, its results or the competence of the organization or a deliberate violation of HKAS regulations for accreditation. For instance, when it is a widespread phenomenon that operating procedures are not documented and/or staff do not perform their tests in accordance with the documented procedures, a significant NC would be reported against clause 4.2 (quality management system). When a method was not evaluated to the extent as necessary to demonstrate its suitability for the intended use, a significant NC would be reported against clause 5.5. Significant NCs have to be corrected with documented evidence before accreditation is granted or reaffirmed.

NCs with no serious adverse effect on the validity of the activity, its results or the competence of the organization can be classified as minor. For instance, equipment was calibrated but the calibration label had not been updated with the calibration status or the laboratory failed to monitor the quality indicator established.Minor NCs also have to be corrected; and action plans have to be provided within one month after assessment, but the effectiveness of actions taken would be checked in the next assessment visit. Recommendations are those that are mainly good laboratory practices and laboratories may select to accept or reject the recommendations. For instance, phone reporting of critical results was found recorded on the back of the test request form, a more systematic way to record details of the reported results and the parties involved in the communication would be recommended for better record traceability.

Data collected from the periods of time mentioned above were analyzed according to the number and nature of NCs reported during the assessment visits, and the frequency of occurrence of NCs against each ISO 15189 requirement. In addition, the number of NCs reported per laboratory for the 27 laboratories undergoing their second reassessments was compared with the number reported in their initial assessments.

 

 

The Kolmogorov-Smirnov test was used to assess the normality of distribution of investigated parameters (i.e. number of NCs per laboratory in each grading reported in the initial and second reassessment of the 27 laboratories). Apart from the number of significant NC reported from the initial assessments of the 27 laboratories assessed in between 2004 and 2006 was not distributed normally, the other parameters were distributed normally. Selection of appropriate statistical methods was based on a recent publication by Simundic AM (8). Wilcoxon matched-pairs signed rank test was used to analyze if there was any significant differences between the performance of the 27 laboratories in terms of no. of NCs per laboratory encountered in their initial assessments and their second reassessments (8). P values < 0.05 were considered statistically significant. Statistical analysis was done using GraphPad Prism version 5.0 (GraphPad Software Inc., California, USA).

 

 

 

The overall distribution of NCs per laboratory against different requirements of ISO 15189 found in the assessments from 2004 to 2006 was shown in Figure 1, whereas those found in the assessments in 2009 were shown in Figure 2. Data shown in Figure 1 were collected from all 40 laboratories assessed from 2004 till 2006, including data of the 27 laboratories undergoing initial assessments from 2004 to 2006 while those in Figure 2 were collected from 61 laboratories undergoing reassessments or surveillance visits in 2009. Since a comparison of the total number and distribution of nonconformities against each clause in Figure 1 and 2 would provide a general picture on whether laboratories had improvement after accreditation, data from the 12 laboratories that had initial assessments in 2009 were not included in Figure 2.

 

 

 

 

Regardless of the accreditation history of the laboratories, the average number of NCs identified per laboratory reduced from 15.4 to 11.1 in the two periods (Table 1). For management requirements, the most common NCs were found against clauses 4.2 – Quality management system, 4.3 – Document control, 4.6 – External services and supplies and 4.13 – Quality and technical records in the assessments conducted in 2009, all were more or less related to documentation. Apart from an increase in number of NCs reported against clause 4.6 in 2009, similar NCs had also been reported against these management requirements in the earlier assessments from 2004 to 2006 (Table 2). Table 2 compared the performance of laboratories against the same requirement clause in the two periods. Those laboratories undergoing their initial assessments in 2009 were separately listed in Table 2. A decrease in number of NCs per laboratory was reported against clause 4.2 and clause 4.13 while a rise was reported against clause 4.6 in 2009 assessments of laboratories accredited before 2008. For the 12 laboratories undergoing their initial assessments in 2009, high numbers of NCs were also reported against these management requirements.

 

 

 

Clause 4.2.1 stated that “Policies, processes, programmes, procedures and instructions shall be documented and communicated to all relevant personnel. The management shall ensure that the documents are understood and implemented. Laboratories were often found to carry out tests deviated from the documented procedures; and there were often procedures, steps or interpretation not described in the documented procedures. If documented procedures are not followed, this would imply that documents are not well understood and not the same procedures are implemented by all staff. There were also situations when procedures were thought to be well understood by all staff and hence they were not documented, nevertheless, different staff was found to carry out the same procedure with slight variation. Under these circumstances, when procedures are not documented and resulted in inconsistencies, this indicated a lack of communication among staff.

 

Before accreditation, laboratories are not used to having document control. There were posted instructions or notes as reminders to staff for interpretation criteria or for certain essential steps, nevertheless, these are instructions that could affect the quality of test results, they also have to be controlled but were not realized by laboratory staff in the early phase of management system implementation. Obsolete versions of controlled documents were found being used by staff, illustrating that they were still not used to the concept of document control. There were also forms and worksheets that had been used for years that were found to be not controlled. Instructions to patients for sample collection were distributed at the sample collection centers and they were slipped out of the document control system. Occasionally, instructions contained therein were found different from the controlled sample collection manual. Test kit inserts also formed part of the quality documentation system and should be controlled for use. Most laboratories retained test kit inserts for a defined period of time for reference. Nevertheless, there was no record on when that particular version of kit insert was used in the laboratory. This lack of period of use would cause problem in record traceability in case investigation was required for NCs identified. Another common observation was on the requirement to regularly review all documents that formed the management system. Quality documents were recorded as reviewed, but there were obvious errors or obsolete information not updated. Records were sometimes seen to be amended by staff without initial and date. These observations were all related to document control and were usually reported as minor NCs to the laboratories. The improvement in document control was not obvious when comparing laboratory performance in the two periods (0.45 NC per laboratory c.f. 0.48 in Table 2). As the management system becomes mature, it is envisaged that the occurrence of such NCs would decrease, but obviously it takes time for staff to get used to the system and change their quality mindset.

 

 

 

The number of NCs per laboratory reported against Clause 4.6 increased in 2009 (Table 2) because laboratories were assessed against requirements of ISO 15189:2007 in which the clause 4.6.3 required an inventory record system to include details of lot number, expiry date, date in use, etc. In the 2003 version, this is not a mandatory requirement. Inventory records were found to be incomplete with no recording of the date that a particular lot was put into service. It was a common practice that laboratories recorded date of opening on the box of the test kit, but when the box was discarded after being used up, the record was lost and this could not fulfill the requirement of keeping the in-use date in the inventory record. There were times when evaluation of new lot of reagents were done by parallel testing with the old lot, there was no defined acceptance criteria or the criteria established were inappropriate. The verification protocol was not documented, this resulted in different number of samples used in each evaluation and the results of evaluation were interpreted with slightly different criteria by different staff.

Nonconformities against clause 4.13 – Quality and technical records

The laboratory is required to retain records related to the management system and examination results. These include all the laboratory workbooks or sheets, raw observations, calculations, etc. A common observation is that interpreted results were recorded instead of the raw observations such as scoring, color reactions, fluorescence, actual reactions observed in tubes of different dilutions, etc. Operators could not be identified from the retained records and the supervisor who counterchecked the results was not recorded.

 

 

A comparison on the number of NCs reported against the 8 technical requirements for the assessments conducted from 2004 to 2006 and in 2009 showed that NCs were reported against similar areas. Clause 5.3 – Laboratory equipment, clause 5.5 – Examination procedures and clause 5.6 – Assuring quality of examination procedures were consistently the three most common technical areas where NCs were found (Table 2). In general, more number of nonconformities was reported against the technical requirements than the management requirements (Figure 1 and 2).

 

NCs were commonly reported against laboratory equipment, nevertheless, these are NCs easily corrected. Equipment used in the laboratory was required to be calibrated against a well calibrated reference equipment to attain metrological traceability. Before the era of accreditation, general equipment used in medical laboratories e.g. thermometers, balances, etc. were often not calibrated. They might have maintenance but were often not properly calibrated. In the early phase of accreditation, laboratories might employ a non-accredited laboratory to calibrate its equipment; or wrongly employ an accredited calibration laboratory to calibrate its equipment without noticing that the required calibration test was not on the laboratory’s accredited scope; or purchase equipment that came with a “calibration” certificate from the manufacturer, believing that it is an acceptable calibration certificate. There were situations where maintenance/calibration was carried out by a contractor, the records of which did not include any information on what parameters had been checked and calibrated. Equipment labels were not updated after calibration or maintenance being conducted. Usually these observations were graded as minor NCs and once being pointed out, laboratories quickly learned.

 

The top number of NCs was reported against clause 5.5.2 where laboratories have to validate procedures for confirming that the examination procedures are suitable for the intended use and that the validations shall be as extensive as are necessary to meet the needs in the given application or field of application. New autoanalysers were found putting into service before they were adequately evaluated. The number of patient samples used for evaluation was too few in some studies and the statistical method used for analysis was inappropriate (9) Evaluation reports did not consist of sufficient detailed information on the procedures and types of samples used, and acceptance criteria were not clearly defined. Some manufacturer’s claims were adopted without verification. Not only clause 5.5.2 is the most common requirement that non-conformity is found, NCs reported are usually of significant nature as an inadequately evaluated method could affect the quality of test results.

Another common observation is on the validation of reference intervals. Reference intervals are required to be reviewed regularly. There were cases where manufacturer’s reference intervals were adopted without validation; or previously established biological reference intervals were adapted to the new equipment without validation. Some biological reference intervals had been used for a long time, while the source was unknown; they were applied to the new equipment without any validation.

 

Clause 5.6.1 required the laboratory to establish an internal quality control system that verifies the attainment of the intended quality of results, so that mistakes are eliminated. Very often, in chemical pathology and haematology laboratories, Westgard QC rules are used for daily quality control monitoring. However, the documented QC rules were not followed. The most common observation being that when QC rules failed, actions taken by the laboratory to address these quality control failure incidents were not recorded and there was no record whether patient results had been reviewed. Another common observation was that the manufacturer’s given mean and standard deviation of a quality control material was used as the laboratory’s daily quality control range, this range was often too wide and inadequate to monitor the laboratory’s own performance. The control levels used sometimes did not cover the clinical decision level and the frequency of running quality controls was insufficient to monitor the autoanalyser performance. The use of third party quality control materials was always recommended when they were not being used by the laboratories. These observations were more commonly associated with chemical pathology and haematology laboratories.

Laboratories are required to participate in external quality assessment programmes and to handle EQAP samples in the same way as patient samples (10). EQAP samples sometimes were found handled with special treatment and sometimes consensus results were reported, particularly for anatomical pathology. When unsatisfactory results were returned, review was superficial and did not include any actions taken to address the root cause of failure; or observed trends were ignored.

 

 

In order to have objective evidence on the improvement brought by implementing a management system meeting the ISO 15189 requirements, we compared the number of NCs identified in 27 laboratories during their initial assessments with their second reassessments conducted around 3.5 to 4 years after establishing their management system. The comparison results on the average number of NCs reported per laboratory are presented in Table 3.

 

 

 

 

 

When comparing assessments conducted in the early days when the accreditation programme was first implemented, i.e. those conducted from 2004 to 2006; with assessments conducted in 2009, it was found that most common NCs were reported against similar management and technical requirements (Figure 1 and Figure 2). For management requirements, the most common areas where NCs were reported were clause 4.2 – quality management system, clause 4.3 – document control, clause 4.6 – external services and supplies, and clause 4.13 – quality and technical records (Table 2). For technical requirements, they were clause 5.3 – equipment, clause 5.5 – examination procedures and clause 5.6 – assuring quality of examination results (Table 2). We believe that the common nonconformities encountered in assessments in Hong Kong are not unique to Hong Kong. Medical laboratories in other countries face similar difficulties in meeting the ISO 15189 requirements (2-4,11). The clause numbers that nonconformities were reported against could vary among different accreditation bodies as very often, more than one ISO 15189 requirements could be referred. For instance, lack of documented procedure could be reported against clause 4.2.1 where documentation of all procedures and instructions are required. Similarly clause 5.5.3 also required all procedures to be documented and be available at the workstation for relevant staff. Where document control problems (clause 4.3) were identified, this could also be related to insufficient training for staff (clause 5.1) or poor implementation of the quality management system leading to misunderstanding among staff (clause 4.2). Hence for similar observations, nonconformities could be categorized into different areas in different countries; this could be related to the training that the assessors received. A direct comparison of figures on nonconformities reported by other accreditation bodies without knowing the details of the findings may not be appropriate. Requirements of ISO 15189 give direction to the components of a good management system that contributes to patient care. The more important is the understanding of the quality concepts behind the requirements and implement continuous improvement.

When laboratories first established a management system according to requirements of ISO 15189, time is required for staff to buy in the idea of accreditation and to let themselves be familiar with the requirements. NCs were reported against issues of document control, use of obsolete documents, or carrying out procedures not as documented. This is understood as time is required for some laboratory staff to change their usual practices; some still do not realize the usefulness of these requirements and consider them as only unnecessary extra work required by the accreditation body. Compliance with the technical requirements also improved as the knowledge of the laboratory personnel increased through exchanges with the assessors, through seminars and contacts with other accredited laboratories. This was indicated from the decrease in the number of NCs reported per laboratory assessed in 2009 when comparing with those assessed from 2004 to 2006 (6.6 vs. 13.5), as well as significant reduction in the number of significant NCs reported in the same group of laboratories undergoing their second reassessments (Table 1 and 3). This is strong evidence that laboratories are improving as they try to comply with the requirements of ISO 15189. High number of NCs reported for the 12 laboratories undergoing initial assessment in 2009 reiterated the fact that accreditation helps to point out the deficiencies in laboratories and show their way for improvement.

 

 

ISO 15189 is an accreditation standard developed particularly for medical laboratories. A management system in compliance with requirements of ISO 15189 helps laboratory to improve quality of its service. As evident from the reduction in number of NCs reported in reassessments, improvement in quality of services resulting from accreditation becomes more apparent with time. Though the common types of NCs found in the two study periods are very similar, this did not indicate that laboratory personnel had not learnt from experience as detailed analysis of data collected in 2009 revealed that the number of NCs reported for each laboratory greatly decreased from initial assessments to reassessments/surveillance visits, thus showing that accredited laboratories are improving. Deficiencies of laboratories existed before accreditation, were pointed out in assessments and corrected, this led to general improvement in the quality of services provided. The ISO 15189 standard is contributing to quality improvement.

 

 

This article only represents personal opinions of the authors and did not represent the opinion of any organization and authority. The authors would like to thank Mr. SS Terence Chan of Hong Kong Accreditation Service (HKAS) to review this manuscript and all HKAS assessors in the field of Medical Testing for their contribution on providing professional judgment and comments for the improvement of our accredited laboratories.