Cultivating Fair and Accurate Knowledge: Exploring Toxicity Detection, Bias Mitigation, and Fact Verification in Multilingual Wikipedia

Abstract

Wikipedia is one of the primary sources of encyclopedic content online. It is one of the most widely read websites. It is also regarded as a quality data source in many machine-learning pipelines. To maintain the high quality of its articles, Wikipedia has three core content policies, one of which is “Neutral Point of View (NPOV)”. This policy is a set of principles, including “avoiding stating opinions as facts” and “preferring nonjudgmental language.” Whenever we refer to “bias”, we refer to it within these guidelines. This work studies how to enhance the quality of the Indian language Wikipedia articles. We looked at existing work on dataset curation from English Wikipedia for bias detection and tried replicating that for Indian languages. We discuss the hurdles faced in this process and discuss translation (along with various quality checks to reduce noise) as a viable alternative. Much of this thesis is dedicated to automatically detecting whether a sentence can be called biased and trying to remove the bias if so. Bias detection is challenging because certain words lead to bias if written in some contexts while not in others. For bias detection in Indian languages, we perform binary classification using MuRIL, InfoXLM and mDeBERTa in zero-shot, monolingual and multilingual settings. For human evaluation, we note how this is a subjective task and disagreement among annotators is expected. Thus, we also experiment with different settings like loss functions specific for subjective tasks and include anonymized annotator-specific information to help us understand the level of disagreement. For bias mitigation, we perform style transfer using IndicBART, mT0 and mT5. These models provide strong baseline results for the novel multilingual tasks. We study how different text generation metrics may or may not be able to capture the quality of debiasing and how to evaluate our models best. Reinforcement learning offers a way to fix the problems observed in the debiased results of the style transfer module. Also, it helps us combine the classification and style transfer modules. We formulate three reward functions specific to our debiasing task and study the results of training fully/partially with these rewards compared to vanilla mT5. Yet another way of improving the quality of the Indian language Wikipedias is to verify the accuracy and reliability of the information presented. All material in Wikipedia must be attributable to a reliable, published source. Thus, we try to identify if the information in a sentence is factually correct or needs a citation. In contrast to previous work, we do this at the fact level instead of the sentence level for more accurate results. Thus, these measures will enhance the quality of the Indian language Wikipedia articles and increase its credibility as the largest source of free, fair, and accurate information.