Why you should be impressed with the NIST OSAC structure: The taming of the Wild Wild West

I have been quoted in the media including a prominent quote in the front page article A1 top position in the Boston Globe when the Annie Dookhan incident happened with calling the modern practice of forensic science in America to be like the Wild Wild West. I have consistently said:

[F]orensic science is like the “wild, wild west.” Just like in the bygone days of the wild wild west, in forensic science today there is a lack of the rule of law, some vigilantes, some rough characters who are up to no good who lie, cheat, and steal exist. There is a lack of standards in terms of behavior or process. Life back then was indeed nasty, brutish and short. Back then, just like today, the innocent got swept up in it all. It was a dangerous place to be that Wild Wild West. Being judged (guilty or not guilty or life versus death) by forensic science in America today is a very shaky proposition. It is getting better, but this modern day wild wild west has yet to be tamed.

With my belief firmly entrenched I went to the joint AAFS/NIST press conference and webcast that unveiled the new structure of the NIST vision of how the former Scientific Working Groups (SWGs) were to evolve into the new Organization for Scientific Area Committees (OSAC). Here was the pre-meeting press release by NIST:

The NIST forensics science team is holding a public presentation and live webcast on its new Organization of Scientific Area Committees (OSAC) at the American Academy of Forensic Sciences annual meeting in Seattle, Feb. 18, 2014, 5:00 to 7:00 p.m. PST. The establishment of OSAC is a result of a NIST and Department of Justice memorandum of understanding signed last year with the aim of improving standards and practices in forensic science laboratories. Here was the pre-release about the agenda and scope of the meeting: NIST envisions uniform administration of development, promulgation and adoption of standards through the OSAC as well as supporting communication flow between the scientific area committees and the forensic science community. The intent is to bring structure, scientific rigor and increased communication among forensic scientists, research scientists, academicians, statisticians, attorneys, managers and quality assurance specialists.

The presentation will take place at the Washington State Convention Center, Ballroom 6C, Seattle, Wash.

Agenda

5:00 to 5:10: Introduction – Mark Stolorow

5:10 to 5:15: Background – Rich Cavanagh

5:15 to 5:30: Notice of Inquiry Responses – Susan Ballou

5:30 to 6:00: OSAC Plan – Barbara Guttman and John Butler

6:00 to 6:15: OSAC Membership – John Paul Jones II

6:15 to 7:00: Questions

I admit bias. I was and continue to remain very heavily critical of the newly created National Forensic Science Commission. Here are some of those blogs:

• National Forensic Science Commission and NIST Guidance Groups
• Let the great whitewash begin: The National Forensic Science Commission
• But they’re fixing forensic science, right?

I still believe that there are major incurable and fundamental flaws with the composition of The National Forensic Science Commission that will forever legitimately place a shadow of doubt over any of its work product. In particular there are no private criminal defense lawyers, there are not enough District Attorneys, there are too many ivory tower professors, and my major issue is with the appointment of Public Policy Director for the National Center for Victims of Crime. In my research on the Public Policy Director for the National Center for Victims of Crime, Susan Smith Howley, she has zero technical proficiency or training or experience in any form of forensic science, yet she is a full voting member. In an off-the-record comment by someone who is in a position to know, this appointment was insisted upon by the Department of Justice (DOJ) and was purely political.

It was with the context of this large disappointment in The National Forensic Science Commission that I walked into the meeting with similar pre-judgment that this OSAC structure was going to be nothing more than more whitewash, but I wanted to listen anyway. Boy was I surprised! In fact, I was converted. I am excited by these NIST OSAC efforts.

The NIST OSAC structure is apparently totally different

First, it is essential to note that during the question and answer period when pressed for an answer, NIST officials seemed to concede that this presentation of the vision of the OSAC structure appears to be that of NIST and NIST alone. Apparently, there will need to be sign-off by the Attorney General or Deputy Attorney General with NIST in a Memorandum of Understanding (MOU) or some like vehicle to achieve the full implementation of the NIST vision. I hope that the DOJ signs off on this vision and does so quickly. But, if this is implemented as designed, I really like it. In fact, at the meeting, I made a public comment over my thoughts and my enthusiasm.

Here is the NIST vision and overview of the entire OSAC structure. As one can see there are three main levels:

1. In green are the Subcommittees of the Scientific Area Committees (SSACs). This is where the nuts and bolts work will be done. This is where members will meet and debate and caucus with the expected goal of developing sound, robust and practical valid standards or guidleines that will be presented to the next level, meaning the SACs for guidelines and to the FSSB in the case of standards. It is envisioned that the SSACs may not be static and its membership may be particularized to a particular project. The recommendation for the membership of the SSAcs will include up to 20 voting members with up to 5 invited guests. The recommendation is that they be comprised of 70% practitioners in the relevant forensic science discipline (with a guide of 20% federal, 30% state and local, and 20% civil or other), 20% researchers, and 10% in the Research and Development world. These subcommittee meetings much like the old SWG meetings will not be open to the public.
2. In blue are the various Scientific Area Committees (SAC). The SAC meetings would be open to the public. It would be the job of the various SACs to evaluate and scrutinize the work of the various SSACs to insure that the guidleines produced are sound, robust and practical valid guidelines. The SAC meetings would be more formal in structure and once appointed seem to remain seated as a standing committee. These will be staggered 3 year terms. The current recommendation is that each SAC would be comprised of 15 member group which would include the various subcommittee chairs, researchers, measurement scientists, and members of professional organizations (e.g., AAFS, ACS, SOFT).
3. In red are the Forensic Science Standards Board (FSSB) and the OSAC Support Committees. The meeting for these 3 OSAC Support Committees and the FSSB will also be open to the public. The current recommendation is that the FSSB would be made up of 15 members with one NIST appointee as an ex officio officer. The 15 would include the 5 SAC chairs, 5 members of professional organizations, and 5 at large members that are from the research and measurement community. Also at this level will be the OSAC Support Committees as seen below:

You can view the entire press conference/webcast here: http://psav.mediasite.com/mediasite/Play/8e65ad3f9958440b8eab9da0831e3b1d1d

My comment begins at 1:33:25 into the press conference.

Here is why I really like what I heard at the meeting:

1. NIST really is listening. I was majorly impressed with the bold vision overall. It was a product of truly seeking input that started with Notice of Inquiry (NOI) soliciting specific input from anyone who wanted to comment on the process of coming up with the successor of the SWGs. At the meeting, I was impressed with the candor of all of the presenters from NIST, and in particular during the question and answer period I was most impressed with the candor and responses by Mark Stolorow. The panel was presented with some pointed issues, they did not shy away and were blunt in their responses. They took many notes and encouraged comment. They were very approachable.

2. It is recommended that the SSACs be very heavily practitioner populated. This I think this is key to success in terms of the implementation of the guidelines and standards as well as the full buy-in by the forensic science community. This is a large difference between the OSAC structure and that of the The National Forensic Science Commission. All one has to do is look at the The National Forensic Science Commission members to see that it is stuffed with academics who haven’t tried a case in a courtroom in the post 2009 NAS Report era. The National Forensic Science Commission has a non-scientist political appointee in the Public Policy Director for the National Center for Victims of Crime, Susan Smith Howley. This heavily practitioner populated strucutre was a product of listening to response of the NOI. Great job here.

3. There is a place for all of the various stakeholders. It was my distinct fear that lead to my initial heavy bias that there was going to be no place for the trial lawyer whether prosecution or defense in the OSAC structure itself. Also, it was my fear that if there were not going to be any judges on the committee. My fear is that if there were judges that they would either be exclusively appellate level judges or federal judges. It is my opinion that this needs OSAC structure needs to include high throughput trial judges at the county or municipal court level as this is where the problem manifests most. To my very pleasant shock, this fear was not materialized and appears to be solved by the institution at the highest level the OSAC Support Committees such as the Legal Resource Committee (LRC),. This is a great idea. Bravo!

Overall, in this structure there will be over 400+ participants at the subcommittee level, the 5 SACs will have 75 people, the FSSB will have 15 (+1 ex officio NIST member), and the 3 OSAC Support Committees will have 30 people. Wonderful. The below is true:

4. The Human Factors Committee (HFC). In particular, I am very enthusiastic about the inclusion of the Human Factors Committee (HFC) in that its charge will be to evaluate the standards for matters of bias. Bias in whatever form is a well-known issue in forensic science, but to date little is done to acknowledge it or mitigate it in a formal way. Brilliant.

5. The structure is flexible. It was repeated over and over that at the SSAC level, other than the appointed chair, the members in the subcommittee can be selected for specific tasks and dissolved and reformed to complete new tasks. I like that idea.

6. Everyone has to apply. There appears to be no deference to what I jokingly refer to as SWG-OLD meaning the “good old boys club.” It seems as if the notion of just dissolving the SWGs whole cloth into the OSAC will not happen. I am sure that some older and historical people in the various SWG committees will get through the selection process and in some cases this will be earned as they are leaders and producers. However, it appears that NIST is very adamant that each of these key positions whether it be at the SAC, FSBB or the OSAC Support Committee is merit based. I applaud this.

7. The work of the SWGs will not totally disappear. There are a lot of SWGs. Here is a list of them: List of SWGs. The historical publications and efforts of the SWGs will be preserved. There appears to be no sentiment to throw out the baby with the bathwater. I agree with that.

I would not put this in the good or bad category, just an observation. I do have a concern over the future. NIST’s vision is to act like a tug boat in that they do not want to permanently be involved in a formal way with the OSAC. Instead, they want to help facilitate the creation and initial population of the OSAC structure, and then in 3 to 5 years presumably when it is on “the right course” and into the proverbial deeper water, like the tug boat, NIST would leave the ship to its own devices. My fear is that that is way too short a time schedule. This time schedule would be like in the Wild Wild West that the town marshal comes into town and brings law and order, then shortly thereafter the town’s marshal goes away. Perhaps the town starts getting infiltrated again and overrun, perhaps not. Only time will tell if it is wise for NIST to be a tug boat or the Harbor Master. Because of the overall very impressive nature of the vision, I personally am going to give NIST the benefit of the doubt in all of this.

There are a few things that I do not find to be the best and wish they meaning NIST and DOJ would reconsider:

1. The subcommittee meetings will be closed to the public. As I mentioned in my public comment, the success of this vision will not be whether or not the forensic science community will accept the effort. To be blunt, they have no real choice. There is a proverbial gun to their heads: money. With the DOJ withdrawing funding money for the SWGs, there is no more SWGs. It is just that simple. Now NIST will be providing for funding for the OSAC. The forensic science community has to come along for the ride. The ultimate buy-in is at the trial court level. Some of the SWGs have had standards for many years. Some of them are fairly good. For example, SWGDRUG has had its recommendations for years. However, if you ask even an experienced criminal defense attorney who handles drug cases exclusively, they probably have never heard of SWGDRUG and certainly don’t know of their recommendations or even how to present them in court. As reflected in the NACDL official response to the NOI, there is a great disconnect and a lot of misunderstanding by my colleagues in terms of what this OSAC effort means to them. Before more anchoring occurs that anything that the OSAC produces is per se illegitimate to the defense community, we need the defense community buy-in whole scale. One of the key problems I foresee is lack of total transparency. By making the SSAC meetings closed this provides needed fuel to the fire for those in the defense community that all must not be on the up-and-up. Why invite conspiracy theories?

2. The maintaining of the Scientific Working Group on DNA Analysis Methods (SWGDAM) and the Scientific Working Group on Digital Evidence (SWGDE) as independent of this structure. This one baffles me. Although not directly stated in the question and answer period, it was very clear that NIST itself did not support this exclusion, but rather it was the DOJ. Objectively, it makes no sense. Here is the published position statement from NIST about these two SWGs.

[]Why is digital evidence excluded from the commission’s charter? Does that mean it will not be one of the discipline-specific guidance groups that NIST administers?

Digital evidence is information stored or transmitted in digital form, including emails, the contents of computer memory, Internet browser histories, and many other items. Because of the complexity, diversity, and rapidly evolving technological advances of digital technologies, digital evidence will not be included in the partnership between the Department of Justice and NIST at this time. However, the Scientific Working Group on Digital Evidence will continue its work.

[]Will SWGDAM (the Scientific Working Group on DNA Analysis Methods) be transitioned to OSAC?

Due to the unique statutory relationship between the SWGDAM and the FBI, SWGDAM will remain with the FBI at this time. The FBI has no initial objection to the transition of SWGDAM or appropriate portions of SWGDAM to OSAC and is open to considering this at some future time.

I agree in totality with the public comments of all of the Question and Answer people who commented on these exclusions. In particular, I thought that Professor Mark M. Pollitt, Ph.D. of the Daytona State College comments were completely correct. Without a doubt, this needs to be changed. The obvious FBI objection over the integration of SWGDAM into this OSAC structure is their likely concern of losing total and complete control over the CODIS system. This is shortsighted and not smart by the DOJ.

3. There should be an educational outreach program. One of the slides centered around the output of the OSAC.

The output looks good. However, there is another important level of ignorance that requires active lobbying: judges. Again, all of those that know about method validation and metrology know how important standards are. Those who know the regulated world of science (e.g., EPA, FDA, GLP) and those in manufacturing and trade (e.g., GMP) and adhere to ISO 9000 and ISO 17025 know the value of standardization and the disaster when there are not standards. I am sorry to report that over 95% of trial court judges have no clue what a standard is. The closest they may see to something like that is the various Building Codes, if they are in civil court and handle construction litigation. This push-out education component of the need and value of standards needs to happen or else we run the risk of having robust, valid standards and meaningful guidelines that no knows about, no one cares about, and is not enforced by the courts. That would be a multimillion dollar shame.

4. The initial screening of all applicants for the SAC level, FSBB, and the OSAC Support Committees will be completed by the NIST-DOJ leadership.membership committee. Although it was duly stressed that this was only an initial screening, I have great fears of the DOJ filter component. As I noted above in three other places, it appears as if DOJ has inordinate veto power over all of this. This is this same mechanism that lead to the eventual appointment of Susan Smith Howley, Public Policy Director for the National Center for Victims of Crime, who has as far as I know zero technical proficiency or training or experience in any form of forensic science as a full voting and contributing member of The National Forensic Science Commission. Dangerous stuff right here. If DOJ applies a filter that removes from consideration anyone who is deemed to be anything other than supportive of the status quo, then this whole process is in real jeopardy. I am not talking about including nut jobs into the equation, but the inclusion of a respectful and well-educated on the topics contrarian would be nice to allow for consideration of alternative points of view.

Overall, I give this vision of the NIST OSAC structure an A+. Well, well done.