DUI expert witness: Accuracy of the NHTSA's Standardized Field Sobriety Test SFST FST

This web site gives the opinions of Dr. Greg Kane. Everything you read here is expressed only as my personal opinion.

In NHTSA's San Diego SFST validation study for six of seven DUI officers, every single driver who was able to take the SFST failed the SFST [BAC 0.04]

The claimed "extreme accuracy" of the SFST is a statistical trick. NHTSA hides the trick by hiding the raw data. The raw data prove SFSTs are extremely inaccurate—so inaccurate that officers in NHTSA's own validation studies simply ignore the test's results.

Home

On this page:

All you need to know
Comparing SFST theory with SFST fact
Quick look at what this web site is about
Links to Greg's published articles on SFST accuracy

Easy accuracy
This web site is about accuracy, scientific accuracy. Let's start easy, with the accuracy the security metal detector.

What happens in the real world? A lady goes to the airport. She gets to the security checkpoint, to a row of metal detectors. The detectors are there to find guns. When someone with a gun walks through the detector, the detector rings. The lady walks through the detector. The detector rings.

So now what happens? The lady failed the gun detector test. Do police arrest her? Is she charged with terrorism? Of course not. Someone checks her bag, finds her keys, and sends her on.

Everyone with a gun sets off the metal detector, but most people who set off the metal detector do not have a gun.

When travelers set off a metal detector, we don't arrest and convict them of gun smuggling and terrorism. Why not? Terrorism is a terrible crime, and these people failed the gun detector test. The safety of America's children depends on keeping terrorists out of secure areas, why shouldn't these people get the justice they deserve?

The answer is obvious. Everyone with a gun sets off the metal detector, but most people who set off the metal detector do not have a gun.

Metal detectors are highly accurate at spotting people who do have a gun, but they are not especially good at telling the difference between gun and no-gun. On innocent people metal detectors often give the wrong answer. They ring for keys and phones and belt buckles, for lots of stuff.

Can science measure the accuracy of metal detectors? Sure it can. Science can do a validation study. And here's our first surprise. Validation studies can be set up in a way guaranteed to discover metal detectors are "extremely accurate."

Guaranteed 100% accuracy
In this example validation study the detector gets the correct answer 100 times in 100—100% of the time. That's how metal detectors work in real life; they're very good as spotting guns.

But the "accuracy" this study discovers doesn't depend on the number of correct answers. The set up here guarantees the study will "discover" the detector is 100% accurate. If 100 people with guns went through the detector and the bell rang only 3 times (3/100= 3% accurate), NHTSA's "scientific" method would still claim the bell was 100% accurate—every time the bell rang, the person had a gun.

Suppose the government paid the contractor who invented the metal detector to do a Gun Detector validation study. Here's her study: one hundred armed scientific research subjects walked through the detector; the bell rang every time. The official validation study proved scientifically the accuracy of the Gun Detector is 100%.

Now that the Gun Detector has been through "scientific" testing, people who set off the Detector can be arrested and confident Gun Recognition Experts can testify correctly that the scientific Detector has been scientifically validated as "extremely accurate," 100% accurate, scientifically.

The Gun Detector is highly effective—at winning convictions.

The second surprise isn't much of a surprise. Most Gun Detector based convictions are false convictions of innocent people. The Gun Detector does ring for guns; it also rings for keys and and phones and buckles. And because many many more people at the airport have keys and phones than have guns, most of the time when the Detector goes off it's for a key or a phone, not a gun.

If juries base their verdicts on metal detector results, most convictions will be false convictions of innocent people.

SFST accuracy
Can science measure the accuracy of the SFST? Sure it can. And it has. Standardized Field Sobriety Tests were invented for the National Highway Traffic Safety Administration by a NHTSA contractor. As I read the reports NHTSA paid her to invent the test. Then they paid her again to validate the test she invented.

The official reports of NHTSA's validation studies claim SFSTs are accurate at identifying BACs above and below 0.10. Also above and below 0.08. And 0.05. And 0.04. Whatever BACs are the legal limit at the time NHTSA pays for the study, those are the BACs NHTSA studies "discover" the test is accurate at identifying.

I've looked at the raw data from the most recent, most up to date NHTSA SFST validation study. You know what the official studies really found? NHTSA's own data proves SFSTs are no more accurate than airport metal detectors. Even when the SFST is done correctly, on innocent people it usually—usually!—gives the wrong answer.

At BAC 0.08% about 3 in 4 innocent people fail.
At BAC 0.05% 63% of innocent people fail.

Everybody with a high BAC fails the SFST, but most people who fail the SFST do not have a high BAC.

At BAC 0.04, using the official 2-HGN-clues-means-guilty criteria, 93% of innocent people fail. In fact using these criteria six of seven DUI officers patrolled a major US city for five months and never once did an SFST that came back not guilty. In the hands of six veteran DUI officers, the SFST's accuracy on innocent drivers was zero percent.

How about those 90% accuracy claims in validation studies? Like the Gun Detector validation study, NHTSA's SFST validation studies were set up in a way that inflated the apparent accuracy of the test. The apparent accuracy. 'Cause, see, NHTSA's "accuracy" statistic is a trick, a linguistic trick. The mathematical word "accuracy" doesn't mean the same thing as the everyday word "accuracy." NHTSA's SFST "accuracy" does not reflect the probability that a failed test indicates a high BAC.

The truth is, the SFST is just a coordination test. That's all. A coordination test. And measuring a coordination score doesn't tell you the underlying cause of that score. The cause could be alcohol. It could also be natural less-than-perfect baseline coordination. NHTSA science proves lots and lots of people—most people— are less than perfectly coordinated. And the SFST cannot tell the difference between high-BAC-uncoordination and low-BAC-uncoordination.

So here's the thing. Most people on the road are sober; but most sober people have a high SFST coordination score. That means most roadside "failures" actually represent those three in four innocent people who are less than perfectly coordinated. If juries depend on SFSTs to decide their verdicts, most DUI convictions will be false convictions of innocent people.

Let's compare SFST theory with scientific fact. When you are called on to interpret a "failed" result from a test that gives the correct answer, "failed", on guilty people but the same answer—also "failed"—on many innocent people, the question you have to answer is, "Did this person "fail" because they're really guilty, or because they're just one of the innocent people the test misclassifies?" *When a test frequently gives the wrong answer on innocent people, the likelihood is very high that any one specific failed* result came from a person who is innocent.** Common sense and a few pictures show how this works.
NHTSA's SFST theory SFSTs do not measure alcohol level and (NHTSA validation studies are clear) they do not measure impairment. SFSTs measure coordination. The officer measures the driver's coordination score. NHTSA's SFST theory holds that each driver's coordination score points back to that driver's BAC. Knowing coordination score is the same as knowing BAC. For everyone. In theory.
More SFST theory Innocent people are blue circles. High BAC people are red circles. The officer measures the driver's coordination score. People with a coordination score above an arbitrary threshold fail the SFST and in this diagram get a black spot. People with a low coordination score pass the SFST and don't get a spot. According to SFST theory, innocent people will pass (blue- no spot), but high BAC people will fail (red- spot). According to NHTSA's SFST theory knowing whether the person has a spot tells you what color their circle is. Theoretically.
Scientific fact Official NHTSA SFST validation study data proves that SFST theory doesn't line up with scientific fact. NHTSA's own data proves that at BAC 0.05% (the per se limit in Colorado) 63% of innocent people fail the SFST! NHTSA's own science proves that knowing whether the person has a spot does NOT tell you what color their circle is.
Amazing fact On the road, most drivers are innocent (blue circles). 2/3 of these innocent drivers will have a high coordination score (a spot) and be detained. A few drivers do have high BACs (red circle). Most of these will fail an SFST. Officers will release drivers with no spots, and detain those with spots. But look! Most circles with spots are blue, because most circles were blue to start with. If juries depend on the SFST to decide guilt, about 86% of convictions will be false conviction of innocent people.
This is not a trick. If the SFST identified innocence accurately, the false conviction rate would be 7%

That's it. That's pretty much all you need to know to know why SFSTs don't work. The rest of this web site is for people who are gung ho. You know who you are.

Quick look

AROUND the country DUI defense attorneys share strategies to overcome field sobriety test evidence. Prosecutors do the same, from the other direction. The sides bicker about mechanics. Did the officer follow procedure e x a c t l y? Did the officer rule out medical conditions that cause incoordination?

What no one does is doubt NHTSA's theory about what a meticulously administered coordination test actually implies about alcohol impairment. The driver failed the SFST. No one asks, "Exactly what does that mean?"

FieldSobrietyTest.info answers the question: What is the probability that a driver who failed an SFST—a meticulously administered SFST—was in fact impaired. The answer will surprise you.

Criticize NHTSA science, get sued.

Read this website while you can.
Best I can tell, I am under ongoing threat from the NHTSA's contract scientist Dr. Jack Stuster for exposing the scientific errors you are reading about here.

I'm about to say unflattering things about NHTSA science. This web site is about science—the science of the NHTSA's SFST validation theory. I do not know, I do not care, I do not have an opinion about Dr. Jack Stuster's knowledge or intentions at any time ever in his life. I'm not even saying he had knowledge or intentions. But if he did, this web site isn't about them. Or him. This web site is not about Jack Stuster, PhD, CPE. This web site is about the flawed science in the NHTSA's SFST validation theory and supporting studies—and in every courtroom where SFSTs are admitted as evidence.

What this web site is about
1. Measurement accuracy vs interpretation accuracy.
The distinction is key.

2. SFSTs
Never before published NHTSA data proves field sobriety tests work great, in the sense that they correctly identify 100 percent of the people who are impaired as being impaired. Trouble is, even when SFSTs are done perfectly they incorrectly identify 93% of innocent people as being impaired. On innocent people the SFST gets it wrong 93% of the time.

The problem is not officers violating procedure. The problem is the test itself. Even when experienced DUI officers do mechanically perfect SFSTs, on innocent people the test usually—93% of the time—gives an answer that is wrong. The test can't tell the difference between impaired and not impaired.

A driver fails her SFST. How can the officer, or the jury, tell whether the test was positive because she was really impaired, or whether the test was positive just because she's one of those 93% of innocent people the test misidentifies? The answer is, They can't. Fancy math confirms what common sense tells us. A test that gives the wrong answer 93% of the time just isn't useful.

3. SFST validation
Yeah sure. Except that over and over the NHTSA's scientific validation studies have discovered the SFST is 90% accurate. Those studies must disprove my crackpot 93%-wrong theory, right?

No, they don't. Here's how NHTSA science for hire makes the SFST look good.

NHTSA SFST validation studies report and "validate" the accuracy of police officer judgments. Officer judgments. Not SFSTs. The NHTSA keeps the SFST's accuracy secret.

Study officer accuracy is assumed to come entirely from the SFSTs officer "use." Assumed. Not proven. Not tested. Not measured. Assumed. This assumption is false. The never before published validation study data proves officer accuracy and SFST accuracy are two different numbers. The data also proves officers simply ignore the SFST result. Study officers do SFSTs, they do not use SFSTs.

So, SFST studies report officer accuracy, not SFST accuracy. SFST accuracy is kept secret.

NHTSA studies do discover that officers make arrest decisions with a high "accuracy." What the studies fail to let on is, this "accuracy" is a misleading statistic that mainstream science explicitly avoids, precisely because it is misleading.

Not only is the misleading accuracy statistic misleading, it allows studies to be rigged, to be set up before hand in a way that assures the misleading accuracy statistic the study will "discover" will be high. Every NHTSA SFST study that "discovers" a high SFST accuracy uses this statistical trick. Every NHTSA SFST study that fails to use this trick also fails to "discover" a high SFST accuracy.

Before the first officer set out on his first patrol, NHTSA's 1998 San Diego SFST validation study's study design assured that the group of people who would be studied would be skewed toward drunks. The study design excluded drivers who drove well. The study design excluded people driving during the day (officers patrolled late at night). The study design excluded drivers who looked and smelled and acted sober. In fact the study design deliberately excluded everyone highly experienced DUI patrol officers thought was sober.

Using those inclusion criteria, and big city late at night patrol tactics, veteran DUI officers were able to come up with a study group that was 90% guilty, at 0.04% BAC, before they began doing SFSTs.

And after they did their SFSTs? After doing SFSTs, they ended up with a group of drivers that was 91% guilty. The SFST itself is responsible for 1% of that 91% "accuracy"!

The tabs

Data

Available to you here for the first time is the data from the latest, most up to date NHTSA SFST validation study. You no longer have to take the NHTSA's word for it that SFSTs are "extremely accurate." Review the data, see for yourself.

Two Elements

Accuracy and "Using SFSTs"

Accuracy

The data proves SFSTs are highly inaccurate.
How the NHTSA's accuracy statistic is rigged.

"Using"

Validation study officers "using SFSTs" systematically rejected the SFST result.

Secret Evidence

The NHTSA makes extravagant claims about the accuracy of the SFST, but it has never published and now refuses to release the data on which those claims are supposedly based.

In depth
FieldSobrietyTest.info gives you an outline of the problems with SFST validation theory. The articles linked below explore the basic science in more detail.

These papers first appeared in Trial Talk, the journal of the Colorado Trial Lawyer's Association.

Crying Wolf
What never before published data prove about field sobriety tests.
Open pdf

Never before NHTSA SFST validation study data prove that to a first approximation the SFST works this way: the test says everyone is guilty; the officer ignores the test and arrests or releases people according to his unstandardized gut instincts.

NHTSA science proves that for six of seven highly experienced DUI patrol officers, every single driver who is able to take the SFST fails the SFST.

Trial Talk, August/September 2008, pg 35 ff

Percentages of Percentages, Why validation Studies Fail to Validate
Open pdf

From the article:

Simply by adjusting the balance of impaired and sober drivers in the study group, the NHTSA can dial in the accuracy its research "discovers."

For example, in the Colorado Validation Study NHTSA contractors chose study drivers in a way that led to the discovery of an officer arrest accuracy of 93%. But, keeping the fundamental accuracies exactly the same and simply reversing the percentage of impaired and innocent drivers chosen to be studied would have changed the officer arrest accuracy to 52% - a coin toss.

Trial Talk, August/September 2006, pg 33 ff
347 K pdf opens in a new window.

Field Sobriety Tests: How Basic Science Proves They Have Little Power to Tell Impaired From Sober
Open pdf

From the article:

Look at the results for the Colorado Validation Study. Drivers with a 1% chance of impairment, when they failed the Colorado Validation Study roadside sobriety test, still had only a 4% chance of being impaired. A driver with a 5% likelihood of impairment, after an “arrest” answer, had only a 16% chance of being impaired. All across the table, the FST added only a few percentage points to the probability of guilt - never enough to turn unlikely into likely.

Trial Talk, October/November 2006, pg 57 ff
255 K pdf opens in a new window

Two Statistical Tricks Let NHTSA Contractors Validate Any FST as "Extremely Accurate"
Open pdf

From the article:

Two tricks fix validation studies so any FST can come off "extremely accurate."
Trick One: Move the FST's inaccuracy over onto the innocent drivers.
Trick Two: Hide the innocent drivers.

....Please realize, all Trick Two does is hide the Innocent Driver Inaccuracy. It hides the inaccuracy by hiding innocent drivers from the "overall accuracy" calculation. Trick Two does not fix the Innocent Driver Inaccuracy. According to the NHTSA, officers using standardized FSTs still arrest 29% of the innocent drivers they assess.

San Diego validation study author Dr. Stuster responds:
Only three drivers were arrested during the study who had BACs below 0.08: one was under the influence of drugs (BAC=0.0), one was too impaired to drive at 0.07, and one was 18 years old with a BAC of 0.07 (in a zero-tolerance state). Not a single “innocent driver” was arrested. How can you claim otherwise?

Read Greg's reply.

....The results of reversing Trick Two are in line 9 [of table 1]. As soon as you reverse Trick Two, in study after study, the "overall accuracy" of the roadside test plummets to a number close to the accuracy of a coin toss. The two statistical tricks are the direct cause of the high accuracies "discovered" in NHTSA validation studies. But for the two tricks, FST validation studies fail to validate.

Trial Talk, April/May 2007 pg 33 ff
288 K pdf opens in a new window.