WA Police Are Scanning Faces in Public. What About the 1%?

WA Police have started running live AI facial recognition in public. I don't think the benefit outweighs the risk of misuse, and I want to set out my reasoning properly rather than just react.

What is actually happening

On 22 June, WA Police became the first law enforcement agency in Australia to use live facial recognition. Cameras in Perth and Fremantle run NEC's NeoFace software. It compares every face that walks past against a police watchlist, in real time.

The watchlist holds about 4,000 people. According to the ABC, it includes people accused of serious offences, people who are missing, and people who may be a risk to themselves or others.

In the first week alone, the system scanned more than 130,000 faces. It produced 33 alerts and 18 arrests, 16 of them for outstanding warrants. As of August, WA Police had declined to release figures beyond that first week.

Those are the numbers the police are proud of. Here is how I read them.

AI is a probability model

This is the part that gets lost when people talk about AI as if it knows things.

It does not know. It calculates. A facial recognition "match" is not a fact. It is a score that has crossed a threshold someone chose. Above the line, the system says "this is probably them". Below it, "probably not".

Using technology to inform a decision is fine. We do it all the time. The problem starts when the technology is probabilistic and the decision is about a person. Then you are accepting that some share of the time, the answer will be wrong. You have decided in advance that you are OK with that.

Is this person on the watchlist? 99% sure.

What about the 1%? That 1% is a real person's life. They are stopped in public, maybe detained, maybe searched, in front of whoever happens to be walking past. Being innocent does not undo that.

The 1% depends on how you count

WA Police reported one false match in the first week. Measured against 130,000 faces scanned, that is 0.0008%. It sounds like nothing.

But most of those 130,000 people were never flagged, so the system never did anything to them. The number that matters is how often it is wrong when it acts. There were 33 alerts. Other reports put the false matches at two, not one. Nicola Lockhart of Edith Cowan University has pointed out that, measured against alerts, that is 3–6%.

Same week, same system, same error, and the two ways of counting differ by a factor of thousands. If you are the person being stopped, only the second number means anything.

This is the question I would ask of any AI system that makes decisions about people. Not "how accurate is it?" but "accurate out of what?"

Who ends up in the 1%

The errors do not land evenly.

Testing by the US National Institute of Standards and Technology has shown higher false-match rates for women, for younger people and for people with darker skin. UK testing of the NEC algorithm found no significant bias at its default settings. It also found that when the threshold was adjusted, false alerts fell disproportionately on darker-skinned people.

The technology has also not been tested on First Nations faces. Michael Birtwisle of the Ada Lovelace Institute told the ABC that UK results can only tell you about that system, for that population.

The Aboriginal Legal Service of WA says it was notified two days before the trial began, and called the consultation "an exercise in tokenism". WA Police say the camera locations were chosen for policing need and do not target any group. Maybe so. But Hannah McGlade, a Noongar human rights lawyer at Curtin University, points out that Aboriginal people are over-represented among people with outstanding warrants, which is what most of those arrests were for. If the error rates are uneven and the watchlist is uneven, nobody has to intend harm for the harm to land unevenly.

The guardrails were not in the room

The strange part is that WA has better law for this than anywhere else in the country.

The Privacy and Responsible Information Sharing Act 2024 is the first Australian law to directly govern how personal information is used in automated decision-making. Yet the trial was designed, as Lockhart put it, "without the state's privacy regulator in the room". The WA Office of the Information Commissioner said it was not meaningfully consulted and warned publicly about the dangers of the program without proper guardrails. Malcolm Crompton, a former Australian Privacy Commissioner, called the privacy impact assessment "feather-light".

A good law you walk around is not a guardrail.

The same rule applies to your business

I have a narrower version of this argument that I make to clients all the time. Putting AI in front of your customers is a different decision from putting it behind your staff.

Behind your staff, AI drafts, checks and suggests, and a person decides. When it gets something wrong, someone catches it before it reaches anyone. In front of your customers, the AI's decision is the outcome. Nobody is standing between the probability and the person.

The consequences are far greater once you are dealing with people directly. Declining a claim, flagging a patient, refusing a booking, marking someone as a risk: each of these is a probability turned into a decision about a human being.

Policing is that principle at its most extreme. The closer AI gets to people, the more careful we have to be. That is why I push clients to use AI under a policy they wrote, on tools they chose, and to keep a named human between the model and anyone it affects.

Do we need this?

This is the question I keep coming back to, and it has nothing to do with the technology.

Do we really need this? Aren't we already safe enough? Eighteen arrests in a week, mostly for outstanding warrants, is a real result. So is 130,000 people having their faces checked against a police list while they went to get a coffee.

My view is that policing should not be done on the basis of artificial intelligence. It can support an officer's judgement. It should not be the thing deciding who gets stopped.

I am genuinely interested in where other people land on this. It is a question where reasonable people disagree, and I would rather hear the disagreement than assume it away.

If you are working out where AI should and should not sit in your own business, especially anywhere near decisions about people, come and have a chat with us. Or get this kind of thinking weekly.