Fairness Evaluation and Inference Level Mitigation in LLMs

Recent advances in Large Language Models (LLMs) have shown remarkable capability and now sit at the center of this revolution, delivering strong performance across diverse Natural Language Processing (NLP) and multilingualism tasks. LLMs should follow instructions while remaining sensitive to diverse cultural norms, resisting convergence on a single viewpoint. Their behavior must be fair, consistent, and culturally grounded. However, handling culturally diverse entities becomes crucial; LLMs can reproduce stereotypes, generate toxic content, remain biased, and display political or ideological leanings. This can be handled in high-resource languages but overlooked in low-resource languages, where linguistics encodes cultural, demographic, and political specific meanings. Despite increasing attention to mitigating gender, socioeconomic, and political biases in LLMs, little has been done to examine or address these issues in South Asian regional languages. Pakistan provides a salient case: Urdu, Punjabi, Sindhi, Pashto, and Balochi each carry distinct cultural and political salience. Pakistani languages are spoken by over 200 million people globally, including in countries like the United Kingdom, Canada, Australia, and the United States. Pakistani culture has ethnic identity, regional autonomy, and political significance. Studying LLMs in these languages exposes context-dependent failure modes and shows how AI bias varies systematically with both resource availability and cultural setting in South Asian or low-resource languages. This thesis frames fairness as a core pillar of alignment, measuring and mitigating cultural, ideological, demographic, and political bias to generate desired direction for LLMs, which is culturally grounded neutrality, reducing ideological bias and harmful stereotypes without sacrificing knowledge, coherence, or local relevance, mainly in multilingual, multi-turn settings where static safeguards often fail. This thesis tackles these challenges through three interconnected studies. The first focuses on evaluation, 13 state-of-the-art LLMs were tested across five Pakistani languages (Urdu, Punjabi, Sindhi, Pashto, and Balochi). The framework is designed to evaluate the political inclination based on stance and multi-level framing analysis and to examine how models frame ideas: their tone, emphasis, and ideological style. The results reveal models tend to reproduce liberal-left positions consistent with Western training data; they shift toward more authoritarian framings when operating in regional languages. These findings expose how language choice itself can trigger different ideological outputs, showing that bias in LLMs is not uniform but culturally and linguistically conditioned. This work targets the desired direction for LLMs: away from ideological bias and without collapsing to a single viewpoint, while preserving coherence and fluency. The second study applies representational steering, which addresses this gap, where bias is localized in the representation space. Building on this, the Steering Vector Ensembles (SVE) method is adapted to intervene directly in the activation space by using multilingual contrastive pairs. This representational approach moves beyond surface-level fixes, offering a principled way to align LLMs with fairness goals. Whereas SVE can show significant results to push a model away from bias in single-turn dialogues, but in real conversations the history keeps bringing it back, and nudging model activations every time hurts coherence and fluency. Pruning is a method to remove bad neurons, but it is permanent. A lighter fix is dynamic masking, catching bias in the moment, holding it down over turns, and maintaining faithful, coherent, and fluent output. To address this limitation, the third study provides dynamic mitigation, where bias can re-accumulate across turns. To address that, this thesis introduces Dynamic Neuron Suppression, a novel inference-time framework to detect context-sensitive neuron activations linked to bias and dynamically modulate them during generation. Unlike static methods, it adapts the conversation mitigation. Tested across multilingual and dialogue datasets, this framework delivers more fair and context-aware outputs. These three interconnected studies provide comprehensive insights into strong bias existence evaluation and its mitigation in single-turn and dynamic multi-turn, advancing fairer and more reliable LLM behavior in multilingualism, multi-turn dialogues, and culturally sensitive settings.

Récupéré en direct depuis OpenAlex et désinversé. Les résumés ne sont pas conservés dans cette base de données : les index inversés représentent 8,6 Go des 9,3 Go de texte de la base, et le serveur dispose de 13 Go libres.