In a series of posts, I am going to introduce the reader to the existence of ISO 17025 and its importance.  I am going to introduce it in bite-sized bits for easy digestion.  Just like all matters of learning, knowledge is incremental over time and builds upon previous exposure.

So far we have answered the following questions:

• What is ISO 17025?
• Why do we need standards? Why ISO 17025 and policy, procedures and instructions matter.
• Why is ISO 17025 so important to us in forensic science?
• Why should the criminal defense community care about ISO 17025?
• How can ISO 17025 provide a simple method to develop themes to cross-examine experts?
• How can ISO 17025 be used by the criminal law practitioner to help get discovery?
• Can ISO 17025 can answer the question as to who the actual analyst is?
• How can ISO 17025 help minimize the particular problem of fraudulent credentials of forensic scientists?

In this post we examine how ASCLD/LAB International conflicts with ISO 17025 and honest scientific reporting of Uncertainty Measurement (UM) in forensic science.

Section 5.3 (Accommodations and environmental conditions) requires a full documentation into the environmental and other testing accommodations of the laboratory to identify potential sources of error and variance to validate that the analytical devices and the personnel involved in the area where the analytical devices are used are sufficiently free from environmentally caused error and yields valid testing results suitable for its intended purpose. Although it is up to the laboratory to establish a policy, procedure, and instructions to meet Section 5.3, these requirements include a full accounting into such heretofore possibly ignored sources of potential error that includes, but is not limited to, “biological sterility, dust, electromagnetic disturbances, radiation, humidity, electrical supply, temperature, and sound and vibration levels.” It also requires a laboratory to monitor, control and record these environmental conditions that may change within the laboratory and may influence the quality of the results.

Perhaps the most fundamental change in the way ISO 17025 laboratories will be conducting their testing and calibration services if they seek and obtain ISO 17025 accreditation comes with the implementation of Section 5.4 (Test and calibration methods and method validation)[i].

It is beyond the scope of this blog post to include the contributions of Theodore “Ted” Vosk in this work concerning ISO 17025[ii]. I have also blogged on basic metrology and Uncertainty Measurement (UM). They include:

• Metrology based posts at the Truth About Forensic Science Blog; and
• Metrology based posts at the Pennsylvania DUI Blog.

However, it is within this subsection, Section 5.4.6, that we find the encapsulation of the need to report, under certain circumstances, Uncertainty Measurements (UM)[iii]. Specifically 5.10.3.1(c) reads that UM shall be included in a Test Report to the customer “when it is relevant to the validity or application of the test results, when a customer’s instruction so requires, or when the uncertainty affects compliance to a specification limit”.

While a tremendous amount of emphasis could be placed and should be placed on UM and whether or not the laboratory’s version of UM is frequentist or Bayesian in nature, Section 5.4 provides additional useful information in the requirement that the method to be employed must indeed be validated.

Although there is a common misperception among lawyers and even among laboratory managers that ISO 17025 provides a method of validation specific to the forensic science disciplines, this is not the case. Instead the requirement of 5.4.5 is for the laboratory to have documentation that includes specifically how it is determined that a given method is to be applied and that its instructions as well as its procedures are in fact validated as promulgated and used.  This is to include limitations on the assay to be performed.

This could be very useful for the criminal law practitioner, for example, in the case of solid drug dose examination and determination. It is possible that within the laboratory’s own documents there could be a damaging admission of its inability to determine and discriminate between positional isomers and chiral compounds[iv]. During the validation process, although not specifically outlined in ISO 17025, at a minimum, the following should be addressed by the laboratory:

• matrix effects,
• sample homogeneity,
• specificity,
• demonstrated range of linearity,
• precision,
• interfering substances,
• stability of targets,
• population distribution, and
• measurement uncertainty.

It is acceptable per ISO 17025 to use reliable, published, and commercially available information to establish each parameter so long as after the implementation of the validated process, it is effectively monitored while it remains in place. If there is deviation from the reliable, published, and commercially available information upon which the method relies, then it is required that the laboratory recognize that the previous method was producing inappropriate results and therefore embark upon a new process of validation that will insure that the process employed and the methodology is one that is indeed suitable for its intended purpose.

Of additional practical use to us is Section 5.6 and specifically Section 5.6.2.1.1. It holds that the laboratories, when they construct a calibration curve or do other types of calibration of the instrument that can contribute to the uncertainty, must properly document the measurement traceability of those reference standards to the classic measurement item (i.e., the International System of Units (SI)) or in the case of items that cannot be strictly made in SI units such as in the case of drugs and DNA profiles, these reference materials are to be traceable to an appropriate measurement standard.

There is a distinction between reference standards and reference materials as outlined in Section 5.6.2. While one cannot certify street cocaine (and hence would be a reference material), one can certify Certified Reference Materials (CRM’s or SRMs) (reference standards). ASCLD/LAB in its interpretation and granting of ISO 17025 accreditation takes Section 5.6 to an additional safeguard step in that it requires that whatever calibration service provider is used by a laboratory must be ISO 17025 accredited.

Another potential source of uncertainty that is addressed by ISO 17025 is the distinction that is made regarding the equipment itself. Section 5.5 requires the laboratory to have a method to identify and classify its instruments that are used throughout the process. There is a distinction between class 1, class 2, and class 3 instruments that is important for the practitioner to be aware of so as to be able to determine whether or not the very best scientific process was employed and whether or not the best calibration of the equipment was undertaken. Per ASCLD/LAB’s interpretation of ISO 17025, Class 3 instruments are the only type of instruments whose calibration service providers do not need to be ISO 17025 compliant per ASCLD/LAB. It is required of the laboratories to not only state that the calibration service providers are ISO 17025 accredited but that they be able to prove through documentation the competence, the traceability, and the measuring ability of the service provider especially if it is not ISO 17025 accredited.

Perhaps the single biggest area of potential uncertainty and one of the most useful to expose remains an undeclared potential source for erroneous results. It is encapsulated and addressed in Section 5.7 (Sampling). In all analytical measurements the analytical device very seldom weighs and/or measures the entire sample as it organically exists[v]. Therefore, only a very small part of the whole, called an aliquot or aliquant, is actually tested by any analytical device.  As a result, it becomes crucial and vitally necessary for the laboratory to ensure homogeneity in the aliquot or aliquant tested. Right now, shockingly, most laboratories do not have a written policy or procedure or instructions that addresses this.

It is this crucial difference between sampling versus sample selection that needs to be exposed by all of us. In essence, what happens in the laboratory when an aliquot is prepared is to exercise a massive amount of truly subjective discretion by selecting a “pinch of this” or a “section of that” from the whole unknown sample submitted for examination. It is clear that by doing such, even with a policy, procedure and instruction in place, massive representation errors with respect to non-colloidal mixtures can occur. Sample selection in the case of trace evidence, for example, per ISO 17025 Section 5.7 would require a written policy, procedure, and instruction that is universally enforced, implemented and monitored by the laboratory down to the technicians at the bench as to which hairs or fibers out of many or what part of a stain to swath and examine, for example. This is an example of sample selection. This is to be distinguished from sampling itself wherein there must be a written policy, procedure, and instructions to make sure that homogeneity of a sample, in fact, exists. A fine example of this would be blood and blood alcohol sampling. Without assurance of homogeneity in such a sample random sampling error is introduced and inaccurate results may be reported. Per ISO 17025 and ASCLD/LAB, there must be rigorous training as well as a plan and procedure in place for sample selection as well as sampling. If one were to obtain the policy, for example, of either sampling or sample selection, then there could and should be some very useful language contained that admits to this very fundamental source of subjectiveness and identifies sampling and sample selection as a large potential source of error.

From strictly a scientific aspect, perhaps one of the disappointments in the promulgation of ISO 17025 is in Section 5.8. Section 5.8 concerns the proper handling of specimens, which in our application is the seized evidence. In only a few words, it states generalities of how the evidence is to be handled and how the items are tested. There is very little guidance and requirements as to this in the ISO 17025 document. There only needs to be a procedure in place per ISO 17025. ASCLD/LAB has rightfully taken the position that this is a crucial part of the crime laboratory analysis and therefore dedicates an additional two and a half pages of requirements in its “International” program.

In Section 5.9 we find that there must be a procedure as to the assuring of the quality of the reported results. While not directly offering or even suggesting such a method, one possible process and methodology that could be used is properly called “control charting.” Control charting is a graphical and empirical statistical tool used to detect excessive process variability to try to identify specific assignable causes that can be corrected. It serves to determine whether a process is in a state of statistical control; that is, the extent of variation of the output of the process does not exceed that which is expected based on the natural statistical variability of the process[vi].  Control charting is a great way to identify the source of statistical outliers where a machine can get pulled, an environment checked or an inappropriate operator stopped or re-trained[vii].

In Section 5.10.1 we can find language that has a great possibility of abuse.  Per Section 5.10.1:

The results of each test, calibration, or series of tests or calibrations carried out by the laboratory shall be reported accurately, clearly, unambiguously and objectively, and in accordance with any specific instructions in the test or calibration methods.

The results shall be reported, usually in a test report or calibration certificate, and shall include all the information requested by the customer and necessary for the interpretation of the test or calibration results and all information required by the method used.

The simple fact that the eventual Test Report that is issued to the “customer” must be one that is unambiguous connotes to the author of this blog that 5.4.6 and 5.10.3.1(c) requires the reporting of UM. ASLCD/LAB does not think so unless certain criteria are met. To present any measure as an absolute without acknowledging UM, does precisely that—makes the reporting ambiguous. The counterargument that is offered by ASCLD/LAB and other laboratory managers is the opposite in that, if UM reporting were placed into every report whenever a measure is made, it would confuse the trier of fact.  As such, they argue that a restriction on the raw data is warranted unless it is around a critical measure and that the unnecessary reporting of UM would invite ambiguity in the overall result. In essence, ASCLD/LAB instead wants to be the judge of the facts, which is, of course impermissible as that role is specifically reserved to the trier of fact.

Regardless of ASCLD/LAB’s interpretation of this section, there is hope in that in Section 5.10.1 and in 5.10.4, we find language that states clearly that it is a requirement that any information not listed on the Test Report “shall be readily available in the laboratory” and therefore should be accessible to all.

[i] Typical forensic applications and disciplines that involve measurement science include: toxicology including a Blood Alcohol Content (BAC) testing; mass determination; drug purity; and distance to muzzle just to name a few. There are also other measuring disciplines that report measurement and therefore fall under the new requirements of ISO 17025 and include: trigger pull; barrel lengths; atomic absorption spectrophotometry (AAS) and inductively coupled plasma-atomic emission spectrometry (ICP-AES) analysis of gunshot residue; refractive indexing; microscopic dimensional analysis; and DNA.

[ii] The leading criminal defense attorney who has been successfully litigating the lack of UM reporting in analytical measurements when presented in the courtroom is Attorney Theodore “Ted” Vosk of Washington State.  Attorney Vosk has published on the concepts of UM.   Forensic Metrology: A Primer on Scientific Measurement for Lawyers, Judges, and Forensic Scientists; Edited by Ted Vosk, Ashley F. Emery; http://www.taylorandfrancis.com/books/details/9781439826195/

[iii] Critical quantitative UM concerns will address the following concepts:

• The identification and evaluation of all sources of potential error,
• The identification of significance of identical uncertainties must be evaluated in the uncertainty budget, and
• The establishing and close monitoring of results near critical values

[iv] Isomers are compounds that possess the same empirical formula, but are different in structure.  Cathine, which is not a controlled substance, will likely be mis-identified as its diastereomer, Phenylpropanolamine, which is a controlled substance, when solely a Gas Chromatograph with Mass Spectrometer (GC-MS) is utilized. While methamphetamine is a schedule II controlled substance, the l-enantiomer of methamphetamine is found in the Vick’s Inhaler, which is a product exempted from control.  Similarly, γ-Hydroxybutyrate (GHB) and γ-Butyrolactone (GBL) can be indistinguishable on some GC-MS. γ-Hydroxybutyrate is a Schedule I controlled substance that cannot be possessed legally. γ-Butyrolactone, on the other hand, may be possessed and only becomes illegal if “intended for human consumption”.

[v] For example, in solid drug dose testing using Gas Chromatography with Mass Spectrometry as the detector, this issue becomes very patent.  In the lab, the technician starts with the whole sample, then a small portion of the whole is removed that one hopes in representative of the whole.  Next, a solvent is typically applied such as methanol, ethanol or dichloromethane to derivative the sample that results in dilution of the original item. An autosampler is employed that takes one microliter which is one millionth of a liter of this derivatized or chemically altered and diluted sample to inject it into the injector port.  Typically, a split injector configuration is used that results in a very, very small part of the microliter making it to the column with the remainder being vented out, not to be analyzed. Of this very, very small amount that makes it onto the column to be separated into hopefully unique analytes, only 1% or 2% of this separated material is ionized in the Mass Spectrometer to be further fragmented.  With the typical scan level of between 40-400 times per second results that in a condition that in order for the analyte to be detected at all a time frame of only 1/360^th occurs where all of these conditions can be met is to record a result at all.

[vi] Every process has some inherent variability due to random factors over which there is no control and which cannot be eliminated economically.

[vii] As can be the case, a lab’s stated but unproven error of +/- x stated to xx%  of confidence (really a predictive interval) can grossly understate reality as later empirically established once control charting and other measures per ISO 17025 are implemented. Once the controls and processes per ISO 17025 are in place, labs may be able to identify sources of profound error.  Then after the immediate triage is completed to identify and end the source of the error, then training and personnel could be selectively fired or up-trained to reduce error and therefore tightening the coverage factor thereby causing it to be 8, 9, 10 or more sigma to get to the critical measure.