research area Fairness, research area Speech and Natural Language ProcessingWorkshop at NeurIPS
content type paperpublished December 2024

Fairness Dynamics During Training

AuthorsKrishna Patel, Nivedha Sivakumar, Barry-John Theobald, Luca Zappella, Nicholas Apostoloff

View publication

This paper was accepted at the Evaluating Evaluations: Examining Best Practices for Measuring Broader Impacts of Generative AI Workshop at NeurIPS 2024.

We investigate fairness dynamics during Large Language Model (LLM) training to enable the diagnoses of biases and mitigations through training interventions like early stopping; we find that biases can emerge suddenly and do not always follow common performance metrics. We introduce two new metrics to evaluate fairness dynamics holistically during model pre-training: Average Rank and Jensen-Shannon Divergence by Parts. These metrics provide insights into the Pythia models’ progression of biases in gender prediction of occupations on the WinoBias dataset. By monitoring these dynamics, we find that (1) Pythia-6.9b is biased towards men; it becomes more performant and confident predicting “male” than “female” during training, (2) via early-stopping, Pythia-6.9b can exchange 1.7% accuracy on LAMBADA for a 92.5% increase in fairness, and (3) larger models can exhibit more bias; Pythia-6.9b makes more assumptions about gender than Pythia-160m, even when a subject’s gender is not specified.

Related readings and updates.

Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs

July 4, 2025research area Computer Vision, research area Methods and Algorithmsconference ICML

The recent rapid adoption of large language models (LLMs) highlights the critical need for benchmarking their fairness. Conventional fairness metrics, which focus on discrete accuracy-based evaluations (i.e., prediction correctness), fail to capture the implicit impact of model uncertainty (e.g., higher model confidence about one group over another despite similar accuracy). To address this limitation, we propose an uncertainty-aware fairness…
Read more

FORML: Learning to Reweight Data for Fairness

August 17, 2022research area Computer Vision, research area FairnessWorkshop at ICML

Machine learning models are trained to minimize the mean loss for a single metric, and thus typically do not consider fairness and robustness. Neglecting such metrics in training can make these models prone to fairness violations when training data are imbalanced or test distributions differ. This work introduces Fairness Optimized Reweighting via Meta-Learning (FORML), a training algorithm that balances fairness and robustness with accuracy by…
Read more