This web site gives the opinions of Dr. Greg Kane. Everything you read here is expressed only as my personal opinion.

"[I]information underlying an officer’s decision is not documented and cannot be examined"
Colorado SFST Validation Study, official report

We saw earlier that the "accuracy" NHTSA gets to the jury is unscientific and wrong. Now I'll show you that the way NHTSA reports even that unscientific accuracy is false and misleading. The accuracy NHTSA reports is not the accuracy of the SFST.

SFST mistakes cause false convictions. Juries are told the SFST is a reliable scientifically validated test. What juries are not told is that NHTSA SFST validation "science" inflates the accuracy of the SFST by fixing its wrong answers.

The big three NHTSA SFST field validation projects all used answer switching. In the one we can get data for, when the SFST gave the right answer, NHTSA counted that answer. But when the SFST gave the wrong answer, someone often switched that wrong answer to the correct one. Answer switching falsely inflated the accuracy NHTSA could report.

Am I saying NHTSA and it's contractors are liars? No. I'm not. I'm not talking about people. I'm talking about science. Answer switching changes the accuracy of the test. NHTSA's science is flawed.

If you're an expert critical of the SFST and you've been threatened or intimidated by NHTSA or it's associates, let me know. I'm putting together a list.

First, the facts
On the off chance this is hard for you to believe, let's start with the facts.

The basic idea of "validating" the SFST is easy. Have trained DUI officers do the SFST on a bunch of people and see how often the SFST gives the right answer. Tell everybody how often the SFST gives the right answer. End of study.

But that's not what NHTSA did. Details vary; here's how it worked for the project I've been able to get the raw data for. NHTSA had trained DUI officers do the SFST on a bunch of people. Officers assessed supects in other ways. They considered their driving, interviewed them, smelled their breath, sometimes found an open bottle, even maybe had them confess. Then the officer wrote down his own personal hunch about what the person's BAC was. And NHTSA tells everybody how often the officer's hunch gave the right answer.

Did I mention officers in this study had PBTs? And did PBTs on every driver? They did.

Here's the thing. The officers (with PBTs) doing the study knew the SFST doesn't work, so they ignored it. When the SFST gave the wrong answer, the officers often changed the wrong answer to the correct answer. The guys doing the study knew the SFST doesn't work, so they ignored it. They fixed it's mistakes. They often changed wrong answers to right answers. Every wrong answer? No. But lots of them.

Let me show you
Here is raw data from the most recent, most up to date NHTSA SFST field validation study, San Diego 1998. The picture is from one of my working Excel files, so it's not purdy. This is the SFST validation data for one officer in the study, Officer 3661. Each row has the results for one driver tested by Officer 3661.

NHTSA keeps track of three things: 1 SFST score, 2 Actual BAC, 3 The officer's guess about what the SFST is. I've numbered those columns.

Remember, SFST scores are not supposed to predict a specific BA level. All they do, supposedly, is predict BAC high or low. To match that theory this table simplifies SFST score according to NHTSA's standardized FST interpretation criteria, to "Hi" or "Lo." BAC the same, above or below the 0.08% limit SFSTs supposedly identify.

NOTICE
The SFST coordination test does in fact work like a metal detector. Everyone with a high BA (column 2, red "Hi") is uncoordinated (column 1, "Hi"). But most people with a low BA (column 2, green "Lo") are also uncoordinated (column 1, white "Hi").

For the 7 innocent people the SFST gives the correct answer only 2 times. On innocent people the SFST is 30% accurate.

Officer 3661's predictions were perfect. When the SFST gave the correct answer, that's the answer the officer gave too. But every single time the SFST gave the wrong answer, officer 3661 rejected that answer—corrected the wrong answer to the right one. Officer 3661's BAC high or low guesses match the PBT high or low results exactly.

On innocent people Officer 3661 was 100% accurate.

In court
NHTSA reports findings like this with crafty phrasing. "Using the SFST, Officer 3661 was able to classify subjects' BACs with 100% accuracy."   Whether you believe this sentence is true depends on what you think "using" means. People of good will can disagree.

Whether or not the sentence is true, it leads to false convictions of innocent people. People think using the SFST means doing what the test says to do.

On account of which people think SFST validation studies proved that officers, doing what the SFST told them to do, were 91 % accurate at classifying drivers' BACs. And the officer in this DUI case I'm the juror on right now, he also did exactly what the SFST told him to do. So he'll be 91% accurate too.

But it's not true. Officers in the validation studies did not do what the SFST told them to do. They were as accurate as they were only because they ignored the SFST. The accuracy numbers NHTSA advertises are as high as they are only because someone changed the answers.

If your D's jury believes NHTSA's false SFST accuracy claims, they will believe the SFST is more accurate than the science proves it really is.

Does it matter?
In one sense, No it doesn't. NHTSA's accuracy statistic is unscientific and misleading and shouldn't be used at all.

On the other hand AOs and prosecutors and juries do use and rely on this misleading statistic, so Yes it does matter.

Using this trick lets NHTSA make people think its project discovered the SFST is 91 % accurate when the truth is 78 %. (Crying Wolf, figure 1) You got'a figure it's easier to convict people with a 91% accurate test than a 78% accurate test. Yeah, it matters.

This same project claimed to investigate whether, using new interpretation rules, the SFST could identify BACs above and below 0.04 %. What the data showed is that using those rules 99% of everyone tested failed the SFST, regardless of BAC. Six of seven experienced DUI officers patrolled for five months, during which time they did SFSTs that failed every person they tested, regardless of BAC. Six of seven officers' accuracy on innocent people was zero percent. The seventh guy got it right three times. In five months. Overall SFST accuracy on innocent people: 7%.

How does NHTSA report that? NHTSA says the officers were 94% accurate. Of course they were. They ignored the SFST. NHTSA's answer switching and crafty phrasing turns a 93% innocent driver conviction rate into "94% accurate". Yeah, it matters.

Their project reports reveal that answer switching certainly happened in all three NHTSA SFST field validation projects, Colorado 1995, Florida 1997, San Diego 1998. Because NHTSA refuses to release data for the first two studies we can't be sure that the answer switching there improved the SFST's "accuracy." To do that, we need the study data. The one study we have data for does prove the "accuracy" bump, as I've explained.

Again, this is all about science, not people. I'm not saying the police officers lied or cheated. They didn't. They were real officers doing real DUI stops. They did exactly what they were asked to do. They did exactly what they should have done—not arrest innocent people. Also I'm not saying NHTSA's contractors sneaked into the lab late at night and changed the data. They didn't. Also, I'm not saying the design of the study was deliberately deceptive. I do not know and I do not have an opinion about that. I don't care whether the error was deliberate or accidental. Water under the bridge. I care about the science. The science is flawed.

1	Fact	Validation study officers "using SFSTs" were about 90% accurate in their arrest and BAC estimate decisions.
2	Assumption	Officer accuracy is assumed to be entirely due to the SFSTs the officers "used." San Diego validation study author Dr. Stuster's threatening (as I read it) email to me indicates he believes officers using breathalyzers should also be counted as "using SFSTs". Read Dr. Stuster on this point.
3	Conclusion	The SFSTs done on defendants in current DUI prosecutions reproduce the 90% accuracy of the validation study officers.

Trouble is element two, the assumption, is silly. Every one of the NHTSA's own validation study reports admits it is false:

So, every validation study admits officers did not base their decisions entirely on SFSTs, and every validation study fails to measure how much difference that makes to the study's results. In fact, validation studies fail to measure whether officers actually "use SFSTs" at all.

Using the previously unpublished San Diego validation study data, I've done the calculation. Officer accuracy and SFST accuracy are different. Officers were 90% accurate. SFSTs were only 78% accurate. (A coin toss is 50% accurate.)

The accuracy statistic is flim flam. Let's look at the real scientific accuracies. Using the standard scientific accuracy called specificity, on innocent people officers were 71% accurate.
On innocent people the SFST was only 29% accurate—worse than a coin toss.

Here are the contingency tables for officer and SFST results.

These results were released.

These results were not released.

Officer decisions were 90% "accurate." The validation study released this irrelevant and misleading datum.

But the accuracy of the officer decisions on innocent people (aka "specificity") was only 71%. If juries rely on officer decisions, they will wrongly convict 29% of the innocent people who go to trial. The SFST did much worse. it's innocent driver accuracy was only 29%, leading to a false conviction rate of 71%. The SFST study did not release these SFST results.

I don't know why NHTSA SFST validation studies did not include this damaging information. Dr Stuster's threatening (as I read it) email to me says:

I replied to Dr. Stuster asking why his SFST validation study report does not reveal the accuracy of the SFST itself. So far Dr. Stuster hasn't responded.

Data prove validation study officers did SFSTs, but did not use SFSTs.
A quick look at the raw validation study data proves that officer did not—could not had they wanted to—base their BAC estimates on standardized SFST interpretation criteria. Here's how we know...

1. Officer's estimates more precise than SFST criteria allow.
In the San Diego study thirteen drivers failed the HGN test and passed both the OLS and WAT tests. These are their SFST results, and the officer's estimate of each drivers BAC.

Notice these thirteen drivers had identical SFST scores. According to the standardized FST interpretation criteria, each driver should have had a BAC estimate of ">=0.08". Instead, officers came up with nine different BAC estimates.

What's more, instead of the SFST's standardized BAC estimates— "<0.08" or ">=0.08"—officers were somehow able to estimate BAC levels to 1 part in 100. There were then and are now no standardized FST interpretation criteria for estimating BAC to 1 part in 100.

Officers did not—could not had they wanted to—rely on these identical SFST scores to come up with their nuanced, 1 part in 100, BAC scores. What's more, the officers somehow knew almost exactly which SFST results to throw out. All these drivers failed the SFST. Yet officers estimated that six of them had BACs in the legal range—flatly contradicting the SFST. Five of those six in-the-legal-range BAC estimates were correct. How'd officers do that? How did officers know almost exactly which SFSTs to ignore?

A detailed look at the SFST / BAC estimate results for one study officer

Seven officers assessed drivers for the San Diego SFST study. This table shows results for one officer, identified in the study as Officer 3661. These are the results for every driver this officer assessed.

Column 1 gives the SFST's BAC prediction, based on standardized SFST interpretation criteria, for drivers assessed by this one officer. The SFST said every driver tested was impaired, regardless of actual BAC.

Column 2 gives the actual driver BAC, simplified to Hi and Lo at BAC 0.04%

Column 3 gives the officers BAC estimate, simplified to Hi and Lo at BAC 0.04%

NOTICE
Every single driver given the SFST failed the SFST.

Every single time the SFST gave the wrong answer, officer 3661 rejected that answer, and correctly estimated the BAC as low.

Officer 3661 never once rejected the SFST when the SFST gave the correct answer.

The probability that this distribution of rejections was random is vanishingly small. Officer 3661 must have used some method other than the SFST for determining BAC level in every case, for every driver.

Same officer, now for BAC 0.08%

NOTICE
Officer 3661's predictions were perfect.

Every single time the SFST gave the wrong answer, officer 3661 rejected that answer.

Officer 3661 never once rejected the SFST when the SFST gave the correct answer.

The probability that this perfect distribution of rejections was random is vanishingly small. Officer 3661 must have used some method other than the SFST for determining BAC level in every case, for every driver.

2. Officer 3661 was not alone. The truth is, officers systematically ignored SFST results. Correct results were accepted; incorrect results were rejected.

This graph shows when officer BAC estimates and SFST results agreed and disagreed.
Data is from the San Diego Field Sobriety Test validation study.

EXPLAINING THE GRAPH
This graph shows which drivers' SFST scores were ignored by police officers. Each point represents one driver: FST score x-axis; BAC y-axis. Drivers above the dark 0.08 line were impaired as a matter of law. Drivers below the dark line were innocent. Open dots and open squares represent drivers whose SFST result, pass or fail, agreed with the officer's BAC estimate.

Every dark square represents a driver whose SFST result was rejected by the officer. Dark squares below the 0.08 line are drivers who failed the SFST, but who the officer correctly assessed as innocent. Dark squares above the line are impaired drivers who failed the SFST, but who were incorrectly assessed by the officer as innocent. (Squares stack. You can't count visible squares to get totals. Of 59 false positive SFSTs, officers rejected 35 = 59%)

Black boxes below the 0.08 line represent SFST mistakes corrected by the officer.
Black boxes above the 0.08 line represent SFST correct-calls mistakenly rejected by the officer.

WHAT THE GRAPH SHOWS
Officers ignored the SFST when it gave the wrong answer, but not when it gave the correct answer. When the SFST gave the wrong answer, officers rejected that wrong answer a whopping 59% of the time. When the SFST gave the correct answer, officers ignored that answer only 2% of the time. This distribution of rejections cannot have happened randomly. Officers systematically ignored the SFST.

The only way officers could have known which SFSTs to ignore and which to accept is to use some other method to assess driver impairment in every case. The data proves FSTs are extremely inaccurate—so inaccurate that officers in the NHTSA's own validation studies simply ignored the test's results.