What is Disparate Impact?

Why It Matters

Anti-discrimination law has never required intent. A practice that produces unequal outcomes can be unlawful even when every person involved acted in good faith, and that is the frame that makes AI screening dangerous rather than merely imperfect.

The largest AI hiring case in the United States turns on exactly this. Applicants allege that an algorithmic screening system produced unequal outcomes by race, age and disability, and the court allowed claims to proceed against the vendor on the argument that it can act as an agent of the employers using it. Thousands of people have joined the age-discrimination collective, and the questions being argued are whether the maths shows an adverse impact and whether an audit performed by the vendor can be relied on at all.

How It Arises in AI

Training on history. A model taught what a successful hire looks like from past hires learns the pattern of who was hired, which includes the pattern of who was not.

Proxy features. Employment gaps, commute distance, school names, and phrasing in a resume can each correlate with a protected characteristic, and a model optimising for accuracy will use whatever predicts.

The label being optimised. A system trained to predict tenure or performance is predicting an outcome that was itself shaped by who got support, which makes the target a moving reflection of past treatment.

Feedback loops. Rejecting a group at a higher rate removes them from the data the next model trains on, which reinforces the pattern and makes it look like a stable finding about performance.

Where It Breaks

The first failure is defending with accuracy. A model can be accurate and still produce an adverse impact, because accuracy is measured against the label and disparate impact is measured against people. A precision score is not a defence.

The second is assuming the vendor’s audit settles it. A supplier audit measures a model, not your criteria, your roles, or your candidate mix, and a clean vendor number is not the deployer’s evidence.

The third is treating human review as the answer. Oversight is required, and it does not neutralise a disparate impact if the reviewer follows the system’s ranking by default, which is what most reviewers do under time pressure.

The fourth is no measurement at all. A team that never computes selection rates by category has not established the absence of a disparity; it has established that nobody looked.

How Flytebit Handles It

We measure outcomes by category from live traffic, examine the feature set rather than only the score, and treat a criterion that produces a disparity as a design question to be reworked. Reviewer overrides are tracked separately from agreement, because a reviewer who never disagrees is a control in name. The industry application is on our HR & Workforce Technology page, and the control design is our AI governance and risk work.