|

When AI Gets It Wrong and You Go to Jail

Facial recognition misidentification panels: casino, restaurant kitchen, stadium

When AI Gets It Wrong and You Go to Jail

⚡ The Short Version: A facial recognition alert is not evidence. It is a lead. Somewhere between the alert and the handcuffs, someone with a badge and a signature decides the machine is right — and there is no law in America that makes the machine, or the company that sold it, answer for being wrong. People have been jailed on matches the technology got wrong: Robert Williams for 30 hours, Porcha Woodruff for 11 hours while eight months pregnant, Randal Tucker for six days. Detroit has paid out at least $450,000 in settlements. The vendor has paid nothing, because nothing in the law requires it to. Meanwhile the same class of system is moving into restaurants — staffing forecasts, loss-prevention flags, voice order scoring — where the jail is just a write-up, a lost shift, or a termination you cannot appeal to a model.

On a February morning in 2023, Detroit police knocked on Porcha Woodruff’s door. She was eight months pregnant. They had a warrant for carjacking and robbery. She told them they had the wrong person. They took her anyway, and she spent 11 hours in custody before anyone looked at the case again.

The lead had started with software. A blurry gas-station frame went into a police facial recognition system, the system returned a list, and a detective picked a face off it. Then a photo lineup went to the victim, who identified Woodruff. The victim later said she was unsure. The charges were dropped in July 2023. Nobody went to jail for putting her in handcuffs. That is not a bug in the system. That is how the system is built.

The Alert Is Not the Arrest — Somebody Signs

This is the sentence that matters most in the whole argument, and it comes from the police themselves: a facial recognition result is an investigative lead, not probable cause. Detroit’s policy says so. Most major departments say some version of it. Every vendor’s marketing says it too, in the fine print, because that disclaimer is what keeps them out of court.

Which means every wrongful arrest built on a bad match was not caused by a machine. It was caused by a human beings accepting a machine’s answer and skipping the steps that would have caught it — the alibi, the height, the tattoos, the fact that the suspect was in another state, the fact that the person standing in front of them was nine months pregnant and did not own a car.

That is the accountability hole. The model produces a probability. A detective converts it into a warrant. A judge signs it. Officers execute it. At no point in that chain does anyone have to prove the model was right, and at no point does the model’s maker have to appear in the room.

Infographic on facial recognition in policing: a restaurant back office with a camera-feed wall, a photo lineup on a phone, a kitchen pass under heat lamps, and a holding-room bench, illustrating how an alert becomes an arrest
Detroit’s own policy calls a facial recognition result an investigative lead, not probable cause, and that distinction is where responsibility moves from the software to the person who signs. Robert Williams spent 30 hours in custody, Porcha Woodruff 11 hours while eight months pregnant, Randal Tucker six days, and every one of those hours began with a human accepting a machine’s answer without checking the alibi, the height or the tattoos.

The Cases Where the Machine Was Wrong and the Human Agreed

These are not hypotheticals or think-tank estimates. They are named people, with dates, in American cities, in the last six years.

Person City What the machine got wrong What it cost them
Robert Williams Detroit, MI Matched to a 2019 Shinola shoplifting frame from a blurry security camera Arrested in front of his wife and daughters; about 30 hours detained; Detroit settled for $250,000 in 2023
Michael Oliver Detroit, MI Matched to a phone theft; the store owner who identified him said he was only “90% sure” A night in jail, charges dismissed; Detroit settled for $200,000 in 2023
Porcha Woodruff Detroit, MI Matched to a carjacking; the victim later said she was unsure of the ID Arrested eight months pregnant, held 11 hours, charged with carjacking and robbery
Nijeer Parks Woodbridge, NJ Matched to a shoplifting and assault case; he was nowhere near the town 10 days detained; settled with the city for $95,000 in 2022
Randal Tucker Las Vegas, NV Matched to a warrant that belonged to a different man, Shane Lee Brown Six days in jail; case dismissed June 2023; civil rights suit filed

Robert Williams was the first case anybody documented, and his quote is the one that should end every argument about whether the technology is “good enough”: when police showed him the security frame, he told them, “I hope you guys don’t think all black people look alike.” He is not describing a glitch. He is describing a system working as designed on a population it was not designed for.

Read the column on the right. Five people. Somewhere north of 30 hours, 10 days, and six days of freedom taken by the state, on the strength of software that no one in the room had tested. The settlements came later, in taxpayer money, and none of them came from the vendor.

Infographic on wrongful arrest liability: a police booking counter with a woman waiting beyond the glass, a detention corridor of steel doors, and a detective's laptop showing facial search results, against 30 hours held and $450,000 in settlements
Detroit has paid out at least $450,000 in settlements and no vendor has paid a cent, because the law treats the software as a tool and the officer as the decision-maker. That arrangement protects the company that sold the match and puts every consequence on the person who accepted it, which is why the case ends with a city writing a cheque and a supplier filing the incident under normal operation.

The Numbers That Explain Why This Keeps Happening

This is not opinion. NIST tested it. In its Face Recognition Vendor Test Part 3, published 19 December 2019, the agency ran 189 algorithms against a large photographic database and measured error rates by demographic group. The finding, in the report’s own words: for the 100 best-performing algorithms, the difference in false positive rates between the two groups with the largest disparity averaged a factor of 10.0, and in some cases exceeded a factor of 100.

For the top 50 algorithms, false positive rates for East African females were 10 times higher than for East European males. A factor of 100 means that for every one wrong match on the easiest face in the database, the hardest face in the database produced a hundred.

Now put volume on top of that. The ACLU’s arithmetic: even at 99 percent accuracy, scanning a million people a day still produces 10,000 false alerts a year, or 27 false alerts every single day. Not 27 chances of a mistake. Twenty-seven people flagged as somebody they are not, every day, in one city, by a system that is performing “well.”

And here is the part the accuracy debate always skips: a false positive is not a neutral error. In a police system, a false positive is a name attached to a crime. Everything downstream — the warrant, the lineup, the arrest — inherits that error and amplifies it, because each human in the chain is told a machine already narrowed the field.

What The Law Actually Requires (Almost Nothing)

Here is where the reader expects a paragraph about the AI Act or a new state law. Instead, here is the actual inventory.

The Equal Credit Opportunity Act of 1974 requires creditors to give specific reasons for adverse credit decisions. The Fair Credit Reporting Act of 1970 gives you notice and a dispute route when an automated system uses a consumer report against you. Title VII of the Civil Rights Act of 1964 lets you challenge a hiring algorithm that produces a disparate impact, and the EEOC issued guidance on exactly that in May 2022. All of those are real. None of them are about a police facial recognition match.

Colorado’s AI Act and California’s automated decision-making technology regulations are arriving, and they will impose impact assessments and disclosure duties on businesses using these tools in high-stakes decisions — employment, housing, credit, education, healthcare. Read that list again. Policing is not on it in most of these frameworks, and where it is touched, it is via transparency requirements rather than a right not to be arrested by a model.

So the vendor question has a brutally simple answer. Is there any documented case where the company that sold the facial recognition system was held liable for a wrong identification that led to an arrest? No. The shields are contractual indemnity in the sales agreement, the absence of a legal duty to third parties the vendor never dealt with, and the doctrine that the buyer — the department — exercises independent judgment when it acts on the output.

That last one is the trick. The disclaimer that keeps the vendor safe (“investigative lead only, not probable cause”) is the exact sentence that pushes all the liability onto the detective who treats it as probable cause anyway. The company sells a product, prints a warning, and walks away with the invoice.

Who What they must prove or disclose Does it cover a bad face match?
Credit (ECOA) Specific reasons for an adverse decision No — money, not liberty
Consumer reports (FCRA) Notice and a dispute route No — reports, not cameras
Hiring (Title VII + EEOC guidance) No disparate impact in selection No — jobs, not warrants
State AI laws (CO, CA ADMT) Impact assessments, disclosure, opt-outs in listed domains Mostly no — policing mostly excluded
The vendor contract Nothing. It disclaims everything. No — that is the point of the contract

So when a restaurant operator reads that AI is “regulated now,” this table is the honest answer. The regulation that exists was built for credit reports and hiring funnels, decades before anyone pointed a camera at a crowd and called the output an investigative lead.

Restaurants Are Running the Same Class of System, Without the Cameras

Now bring it home, because this is a restaurant publication and the reader runs a restaurant.

The same architecture — a model produces a scored judgment, a human with authority signs off, and the human is the only one who can be blamed — is already inside hospitality software you probably pay for.

7shifts sells AI-powered forecasting that predicts labour needs and builds schedules, which means the model decides how many hours exist before a single person asks for one. Tanda sells smart rostering that automates shift allocation. Agilence sells exception-based reporting and video analytics that flag transactions as possible sweethearting or fraud, which is a flag that turns a cashier into a suspect. SoundHound sells voice AI for drive-thrus that takes orders and scores the interaction.

None of those systems handcuffs anyone. That is not the point. The point is that every one of them produces a judgement about a person — how many hours they deserve, whether their till was short, whether they said the right words to a customer — and in every case the human manager is the only party who can be held to account for acting on it.

Ask the honest question: if the forecast understaffs a Friday and a server loses a shift, or the loss-prevention flag misfires on a cashier who did nothing, what appeal exists? Is there a written record of the model’s error rate? Can the employee’s representative see the threshold that triggered the flag? In almost every deployment, the answer is no, and the manager who acted on the output did exactly what the software told them to do.

Infographic on AI scoring inside restaurants: a chef at the pass mid-service under heat lamps, a pan lifted off a flame, an order screen on the wall, and a staff-room locker, illustrating staffing forecasts and loss-prevention flags
Staffing forecasts, loss-prevention flags and voice-order scoring put the same class of judgement inside a restaurant, where the output is a shift cut rather than a charge. Four distributors, four rulebooks and no appeals process apply there too: the model scores the person, the manager signs off, and nobody on that floor can see the error rate that decided it.

The Thing Nobody Will Publish

A credible article says what it does not know. So here is that list.

No vendor in any of the arrest cases has disclosed its error rate on the population it was pointed at. No police department has released the number of searches it runs a year, which means the 27-a-day figure cannot be checked against a real city’s volume. Axon said its product was not used for the initial identification in the Las Vegas case, and that claim has never been independently audited. The settlement agreements in Detroit and Woodbridge are confidential in substance; we know the totals because reporters found them in city records, not because anyone published terms.

And the biggest gap: no court has ruled on whether a department’s policy of treating a match as a lead is adequate. Every case has settled. Every settlement means the legal question stays open, which is the cheapest possible outcome for everyone except the next person who gets matched to a stranger.

What a Human Being Can Actually Do

If you are a restaurant operator running scheduling, loss-prevention or voice-order AI, three things are in your control and nothing else is.

Write down who can override the system and what evidence lets them. If the answer is “the manager just approves the schedule,” you have no control, you have a rubber stamp. Insist on seeing the flag, not the score: a cashier accused by an exception report should be shown the transaction and the threshold, in writing, before any disciplinary step.

For employees: an adverse decision produced by a system is worth a written request for the underlying data. FCRA-style notice does not automatically apply to your shift allocation, but the request costs nothing and creates a record. And if the decision came from a police match, the first call is a lawyer, not an explanation.

The machine does not go to jail. That is not a slogan, it is the structural fact the whole system is built on. Until a vendor has to answer for a wrong match, and until a contract can no longer disclaim it, the person the model got wrong is the only one in the room who pays.

Frequently Asked Questions

Can facial recognition alone put you in jail?

No, and that is the gap between the marketing and the arrests. Police policy in Detroit and most large departments treats a facial recognition result as an investigative lead, not probable cause. What actually puts a person in custody is the combination: the match, a photo lineup identification, and a human signature on a warrant. In every documented wrongful arrest, all three happened and all three were wrong.

Has any facial recognition vendor been held liable for a wrong match?

No documented case exists. Vendors are protected by contractual indemnity, by the absence of a duty to third parties they never contracted with, and by the legal position that the department exercises independent judgment when it acts on the output. The standard disclaimer that a match is a lead and not probable cause is the sentence that moves liability from the seller to the officer.

How accurate is facial recognition, really?

Accurate enough to be useful and biased enough to be dangerous, and NIST published the figures. In its Face Recognition Vendor Test Part 3 on 19 December 2019, the 100 best-performing algorithms showed an average 10-fold difference in false positive rates between the two groups with the largest disparity, with the worst exceeding a factor of 100. For the top 50 algorithms, false positive rates for East African females were 10 times higher than for East European males.

How much have cities paid for facial recognition mistakes?

Detroit paid Robert Williams $250,000 in 2023 and Michael Oliver $200,000 in 2023. Woodbridge, New Jersey paid Nijeer Parks $95,000 in 2022. Those are the public records. None of that money came from a technology vendor.

Is this happening in restaurants?

Not as arrests. As scored judgements. 7shifts and Tanda forecast labour and build schedules that decide how many hours exist. Agilence flags transactions as potential theft and turns a cashier into a suspect. SoundHound takes and scores drive-thru orders. In each case a human manager is the only party accountable for acting on the output, and the model’s error rate is almost never disclosed to the people it scores.

What should a restaurant operator do about it?

Document who can override each system and what evidence they need to do it. Require that any employee accused by an analytics flag sees the underlying transaction and the threshold in writing before discipline. A manager who approves a schedule without knowing the forecast’s error rate is not supervising a system, they are signing for it.

What does the law require when an AI makes a decision about a person?

Less than most people assume. ECOA requires reasons for adverse credit decisions, FCRA requires notice and a dispute route for consumer reports, Title VII allows disparate impact challenges to hiring algorithms. None of these cover a police face match. Colorado’s AI Act and California’s automated decision-making rules add impact assessments and disclosure in listed domains, but policing is largely outside them.

Barak Schurr

About Barak Schurr

Barak Schurr built and exited a major ticketing platform, ran global operations at Eventbrite, and now focuses on one question: which AI tools actually help independent restaurant owners run leaner and grow faster. He writes from the operator’s perspective — no vendor spin, no theory, just what’s real in 2026.
Email: [email protected]

More Resources for Restaurant Operators


Top Notch AI Tools

Top Notch AI Tools

AI tools and strategies for entrepreneurs and small business owners.


AI Video Advisor

AI Video Advisor

AI-powered video creation tools and strategies for creators and businesses.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *