Data Lab

The challenge of unanswerable questions in Machine Reading Comprehension (MRC) has drawn considerable attention, as current MRC systems are typically designed under the assumption that every question has a valid answer within the provided context. However, these systems often encounter real-world situations where no valid answer is available. This paper provides a comprehensive review of existing methods for addressing unanswerable questions in MRC systems, categorizing them into model-agnostic and model-specific approaches. It explores key strategies, examines relevant datasets, and evaluates commonly used metrics. This work aims to provide a comprehensive understanding of current techniques and identify critical gaps in the field, offering insights and key challenges to direct future research toward developing more robust MRC systems capable of handling unanswerable questions.

Background: Systematic Literature Reviews (SLRs) are essential in biomedical research, particularly for informing public health policy and clinical decision-making. However, the manual generation of Boolean queries for literature searches is resource-intensive, prone to errors, and difficult to scale. Recent advances in large language models (LLMs) have demonstrated potential, yet most existing approaches rely on zero-shot prompting of commercial models, overlooking the cost-efficiency and domain adaptability of fine-tuned open-source alternatives.

Methods: This study proposes a novel, three-stage framework that employs medium-sized, open-source generative models, specifically BioGPT and BioT5, for automated Boolean query generation over PubMed. We develop and release datasets comprising PubMed article titles, MeSH terms, and keywords, and fine-tune the models using both title-only and title-plus-metadata prompts. We evaluate performance on two benchmark datasets: CLEF TAR and FASS-BSLR. Our experiments include comparisons with state-of-the-art baselines, prompt-based large language models, and ablation studies exploring the effects of training data size, metadata inclusion, and post-processing with PubMed's Automatic Term Mapping.

Results: Fine-tuned BioGPT outperforms both traditional TAR models and commercial LLMs across key retrieval metrics. On the CLEF TAR dataset, it achieves a Precision of 0.2544, F1 of 0.2392, MAP@1000 of 0.1424, and NDCG@1000 of 0.2490, which surpasses all baselines. On the FASS dataset, it reaches a Recall of 0.1801 and NDCG@1000 of 0.0900, again outperforming all competing models. While slightly behind BioGPT, BioT5 still outperforms most baselines. Notably, BioGPT's Recall of 0.1801 on FASS is more than twice that of PubMed-Title and PubMed-Keyword, and exceeds GPT-3.5 Turbo, GPT-4, Gemini-2, and Llama-3.

Conclusion: This work demonstrates that fine-tuned, open-source, medium-sized generative models can match or exceed the performance of much larger commercial LLMs in Boolean query generation for biomedical SLRs. These models offer a cost-effective, privacy-preserving, and scalable alternative for structured retrieval of biomedical scholarly texts.

Large language models (LLMs) often respond confidently to ambiguous questions by implicitly committing to a single interpretation, which can yield misleading answers when multiple meanings are plausible. We propose RAAD (Retrieval-Augmented Ambiguity Detection), a lightweight framework that detects question ambiguity by identifying semantically incompatible answers grounded in retrieved evidence. RAAD retrieves diverse contexts, extracts candidate answers, and detects ambiguity by measuring semantic incompatibility among answer pairs using a cross-encoder. In RAAD, retrieval and answer diversity act as complementary pillars: retrieval exposes multiple plausible "world states" in which the question can be answered, while cross-encoder scoring determines whether the resulting answers remain interchangeable or diverge semantically. We evaluate RAAD on AmbigQA, ASQA, CAmbigNQ, and SituatedQA (Geo/Temp) and show that when trained with gold supervision on AmbigQA and evaluated on noisy, realistic retrieved answers, RAAD generalizes strongly across datasets, outperforming reported baselines on multiple benchmarks.

Factoid entity questions (FEQ), which seek answers in the form of a single entity from knowledge sources, such as DBpedia and Wikidata, constitute a substantial portion of user queries in search engines. This article introduces the knowledge graph embedding model for FEQ (KGE-FEQ) answering. Leveraging a textual knowledge graph derived from extensive text collections, KGE-FEQ encodes textual relationships between entities. The model employs a two-step process: (1) Triple Retrieval, where relevant triples are retrieved from the textual knowledge graph based on semantic similarities to the question, and (2) Answer Selection, where a knowledge graph embedding approach is utilized for answering the question. This involves positioning the embedding for the answer entity close to the embedding of the question entity, incorporating a vector representing the question and textual relations between entities. Extensive experiments evaluate the performance of the proposed approach, comparing KGE-FEQ to state-of-the-art baselines in FEQ answering and the most advanced open-domain question answering techniques applied to FEQs. The results show that KGE-FEQ outperforms existing methods across different datasets. Ablation studies highlights the effectiveness of KGE-FEQ when both the question and textual relations between entities are considered for answering questions.

In this paper, we introduce Schema-Tune, a zero-shot self-supervised framework for bias mitigation in transformer-based language models. Schema-Tune introduces curated and optimized adaptive noises to the input embeddings of transformer models to challenge the models' embedded stereotypes. Through continuous fine-tuning steps, these noises prompt the models to change their internal semantic representations towards more socially fair representations. For fine-tuning language models, Schema-Tune relies on very limited input data: a couple of sentences formed by social group terms. Additionally, Schema-Tune defines bias and language model performance measures independently from labeled data. These measures are then used in forming the language model's fine-tuning objective function and in searching for effective noises in the embedding space. Experimental evaluation over the StereoSet and Crows-Pairs datasets confirms that Schema-Tune is effective in mitigating bias in different social stereotype categories, including gender, race, and religion.