Amazon's AI did exactly what it was told. Deleting words for two years did not help.
Amazon told its recruiting AI to find people like the ones it had hired over the previous ten years. Most of those people were men. The model learned to prefer men.
That is not a malfunction. That is the instruction, followed accurately.
What happened next is the part worth your time. Amazon caught the problem inside a year, which is faster than almost anyone manages. Then it deleted the offending words from the model's vocabulary, kept the instruction exactly as it was, and ran the thing for two more years.
I went back to the source while building an evidence base on AI project failures. The source is Reuters, October 2018. Two things in it are worth carrying into your own systems, and one of them is a correction to the story everybody tells.
What the record says
Reuters reported that from 2014 a group of Amazon researchers built models to score candidates from one to five stars. They were trained on CVs submitted over the previous ten years, most of them from men.
The bias is documented in specific terms. The tool penalised CVs containing the word "women's", as in "women's chess club captain". It downgraded graduates of two all-women's colleges.
That is not a vague allegation of unfairness. It is a named behaviour with named triggers, and it is the strongest part of the story.
Then there are two statements that most retellings quietly collapse into one.
Reuters, sourced to people familiar with the matter: Amazon's recruiters looked at the recommendations when searching for new hires, but never relied solely on those rankings.
Amazon, on the record: the tool was never used by Amazon recruiters to evaluate candidates.
Those are reconcilable. The tool could have been outside the formal hiring process while recruiters informally looked at what it produced. Amazon has never elaborated, and nobody has established which reading is right.
What that leaves you able to say
Verified: Amazon stated publicly that the tool was never used to evaluate candidates.
Verified as reporting: Reuters' anonymous sources said recruiters looked at its recommendations.
Not established by any source: that the tool determined, or decisively contributed to, a single actual hire or rejection.
So "Amazon's AI rejected women applicants" is a claim nothing in the record supports. What the record supports is a demonstrated bias in a model's scoring behaviour. That is still a serious finding. It is a different finding, and the difference is not pedantry, because the two versions teach opposite lessons.
The popular version says: watch your model, it might hurt someone. The recorded version says: your model can be demonstrably biased for years without anyone being able to show it hurt a specific person, and you should still treat that as a failure.
The failure was set in 2014, before any code
Training on ten years of Amazon's own hiring decisions made "resembles people we previously hired" the thing the model was rewarded for getting right. In an applicant pool that was mostly men, proxying for gender is not a defect. It is an accurate fit to the target it was given.
That decision was made when the label was chosen, at the start, before a line of model code existed. Everything after it was patching a model that was correctly fitted to a biased process.
This is the cleanest case I found of a failure that was fully predictable from first principles at design time. That supervised models trained on historical human decisions reproduce the biases in those decisions was well established in the literature by 2014. It was knowable. Nobody looked.
Detection worked. The fix was aimed at the wrong thing.
Amazon found the skew internally by 2015, one year in. That is fast. Most organisations in the cases I have read did not find their problem at all until an outsider published it.
Then they edited the offending terms. Reuters records the acknowledgement that this was no guarantee against other discriminatory proxies, which is correct, because a model fitted to a biased label will find another route to the same answer. Remove "women's" and the model reaches for whatever else correlates.
The team was disbanded by the start of 2017. A successor group was formed in Edinburgh with a focus on diversity.
So the sequence is: detected in 2015, patched at the level of symptoms, run for roughly two more years, then stopped.
I have started treating "known but mis-remediated" as its own category, separate from "known and ignored". It is the more common one and the harder one to see from inside, because the organisation has evidence that it responded. It did respond. It responded to the wrong layer.
The question this leaves for your own systems
If you find a bias in a model's output, the immediate question is not how to remove it from the output. It is whether your fix touches what the model was asked to predict, or only what it happened to say.
Term-level edits are the second kind. So are output filters, blocklists and post-processing rules. They all leave the objective untouched, and the objective is where Amazon's problem lived from 2014 onwards.
The other number worth looking at is the gap between the day you detected something and the day you stopped doing it. For Amazon that gap was about two years. In the other cases I have catalogued it is governed by nothing in particular, and in four of six it took external legal or commercial pressure to close it rather than internal evidence.
Detection is largely a solved problem. Response is not.
This is part of a series testing the numbers and stories that get repeated about AI failure against their primary sources. The first asked four questions to put to any AI statistic before you repeat it. This one is a case where the story survived contact with the source and one sentence of it did not.