NYC Local Law 144: What the Annual Bias Audit Actually Requires

NYC Local Law 144 took effect on July 5, 2023 and applies to employers and employment agencies that use automated employment decision tools (AEDTs) to screen candidates or employees for positions in New York City. It is the most detailed AI hiring regulation in the United States to date and a preview of what other jurisdictions are moving toward. Meeting it is one piece of a defensible compliance posture.

Here is what the law actually requires, read from the text rather than from summaries.

What counts as an AEDT

The law defines an AEDT as “any computational process, derived from machine learning, statistical modeling, data analytics, or artificial intelligence, that issues simplified output, including a score, classification, or recommendation, that is used to substantially assist or replace discretionary decision-making for employment decisions.”

The phrase “substantially assist or replace” is where most of the interpretive work happens. A tool that generates a ranked list of candidates that hiring managers routinely follow is likely an AEDT. A tool that surfaces information for review but does not produce a ranking or score is less clearly covered.

If you are unsure whether a tool you are using meets this definition, the safe position is to assume it does and comply accordingly.

The bias audit requirement

Employers and employment agencies must obtain an independent bias audit within one year before using the tool. The audit must be conducted by an independent auditor, defined as someone with no financial interest in the employer or the AEDT vendor.

What the audit must calculate:

• The selection rate (or score distribution for tools that do not produce a pass/fail result) for each category of race/ethnicity and sex
• The impact ratio for each category: the selection rate for that category divided by the selection rate for the highest-selected category
• The impact ratio at the intersection of race/ethnicity and sex

The audit must be based on data from the employer’s actual use of the tool, if sufficient data is available. If the employer has not used the tool long enough to have adequate data, the auditor may use data from other sources, with that fact disclosed.

The public disclosure requirement

The results of the bias audit must be published on the employer’s website at least ten business days before the tool is used to evaluate any candidate. The published information must include:

• The date of the most recent bias audit
• A summary of the bias audit results, including the impact ratios for each demographic category
• The source of the data used in the audit

This disclosure is public. Candidates, regulators, and legal counsel can access it. A tool with impact ratios that are materially below 0.80 for a significant group is creating public evidence of potential adverse impact.

The notice requirement

Separately from the bias audit, employers must notify candidates that an AEDT is being used, at least ten business days before the assessment is conducted. The notice must state:

• That an AEDT is being used
• What job qualifications or characteristics the tool evaluates
• How candidates may request an alternative selection process or accommodation

The notice requirement is often overlooked because it is separate from the audit requirement. Compliance with the audit does not satisfy the notice obligation.

What the law does not require

The law does not require employers to use a tool that produces zero adverse impact. A tool with adverse impact can still be used if the bias audit is conducted, the results are disclosed, and the tool is otherwise lawfully applied (meaning the employer can demonstrate job-relatedness if challenged under Title VII).

The law also does not require the AEDT to be valid. Validity is a separate requirement under UGESP and Title VII, not under NYC LL 144. Compliance with LL 144 does not substitute for, or demonstrate, validity.

The practical compliance checklist

1. Identify all AEDTs currently used to screen candidates for NYC positions
2. Confirm the most recent independent bias audit was within the past twelve months
3. Verify the bias audit results are published on your website
4. Confirm the notice is being provided to candidates at least ten days before the assessment
5. Review whether any tool’s impact ratios indicate adverse impact that would require further investigation under UGESP

If any of these steps surfaces a gap, address it before continuing to use the tool.