Recent Publications

The challenge of unanswerable questions in Machine Reading Comprehension (MRC) has drawn considerable attention, as current MRC systems are typically designed under the assumption that every question has a valid answer within the provided context. However, these systems often encounter real-world situations where no valid answer is available. This paper provides a comprehensive review of existing methods for addressing unanswerable questions in MRC systems, categorizing them into model-agnostic and model-specific approaches. It explores key strategies, examines relevant datasets, and evaluates commonly used metrics. This work aims to provide a comprehensive understanding of current techniques and identify critical gaps in the field, offering insights and key challenges to direct future research toward developing more robust MRC systems capable of handling unanswerable questions.

Background: Systematic Literature Reviews (SLRs) are essential in biomedical research, particularly for informing public health policy and clinical decision-making. However, the manual generation of Boolean queries for literature searches is resource-intensive, prone to errors, and difficult to scale. Recent advances in large language models (LLMs) have demonstrated potential, yet most existing approaches rely on zero-shot prompting of commercial models, overlooking the cost-efficiency and domain adaptability of fine-tuned open-source alternatives.

Methods: This study proposes a novel, three-stage framework that employs medium-sized, open-source generative models, specifically BioGPT and BioT5, for automated Boolean query generation over PubMed. We develop and release datasets comprising PubMed article titles, MeSH terms, and keywords, and fine-tune the models using both title-only and title-plus-metadata prompts. We evaluate performance on two benchmark datasets: CLEF TAR and FASS-BSLR. Our experiments include comparisons with state-of-the-art baselines, prompt-based large language models, and ablation studies exploring the effects of training data size, metadata inclusion, and post-processing with PubMed's Automatic Term Mapping.

Results: Fine-tuned BioGPT outperforms both traditional TAR models and commercial LLMs across key retrieval metrics. On the CLEF TAR dataset, it achieves a Precision of 0.2544, F1 of 0.2392, MAP@1000 of 0.1424, and NDCG@1000 of 0.2490, which surpasses all baselines. On the FASS dataset, it reaches a Recall of 0.1801 and NDCG@1000 of 0.0900, again outperforming all competing models. While slightly behind BioGPT, BioT5 still outperforms most baselines. Notably, BioGPT's Recall of 0.1801 on FASS is more than twice that of PubMed-Title and PubMed-Keyword, and exceeds GPT-3.5 Turbo, GPT-4, Gemini-2, and Llama-3.

Conclusion: This work demonstrates that fine-tuned, open-source, medium-sized generative models can match or exceed the performance of much larger commercial LLMs in Boolean query generation for biomedical SLRs. These models offer a cost-effective, privacy-preserving, and scalable alternative for structured retrieval of biomedical scholarly texts.

Large language models (LLMs) often respond confidently to ambiguous questions by implicitly committing to a single interpretation, which can yield misleading answers when multiple meanings are plausible. We propose RAAD (Retrieval-Augmented Ambiguity Detection), a lightweight framework that detects question ambiguity by identifying semantically incompatible answers grounded in retrieved evidence. RAAD retrieves diverse contexts, extracts candidate answers, and detects ambiguity by measuring semantic incompatibility among answer pairs using a cross-encoder. In RAAD, retrieval and answer diversity act as complementary pillars: retrieval exposes multiple plausible "world states" in which the question can be answered, while cross-encoder scoring determines whether the resulting answers remain interchangeable or diverge semantically. We evaluate RAAD on AmbigQA, ASQA, CAmbigNQ, and SituatedQA (Geo/Temp) and show that when trained with gold supervision on AmbigQA and evaluated on noisy, realistic retrieved answers, RAAD generalizes strongly across datasets, outperforming reported baselines on multiple benchmarks.

Factoid entity questions (FEQ), which seek answers in the form of a single entity from knowledge sources, such as DBpedia and Wikidata, constitute a substantial portion of user queries in search engines. This article introduces the knowledge graph embedding model for FEQ (KGE-FEQ) answering. Leveraging a textual knowledge graph derived from extensive text collections, KGE-FEQ encodes textual relationships between entities. The model employs a two-step process: (1) Triple Retrieval, where relevant triples are retrieved from the textual knowledge graph based on semantic similarities to the question, and (2) Answer Selection, where a knowledge graph embedding approach is utilized for answering the question. This involves positioning the embedding for the answer entity close to the embedding of the question entity, incorporating a vector representing the question and textual relations between entities. Extensive experiments evaluate the performance of the proposed approach, comparing KGE-FEQ to state-of-the-art baselines in FEQ answering and the most advanced open-domain question answering techniques applied to FEQs. The results show that KGE-FEQ outperforms existing methods across different datasets. Ablation studies highlights the effectiveness of KGE-FEQ when both the question and textual relations between entities are considered for answering questions.

In this paper, we introduce Schema-Tune, a zero-shot self-supervised framework for bias mitigation in transformer-based language models. Schema-Tune introduces curated and optimized adaptive noises to the input embeddings of transformer models to challenge the models' embedded stereotypes. Through continuous fine-tuning steps, these noises prompt the models to change their internal semantic representations towards more socially fair representations. For fine-tuning language models, Schema-Tune relies on very limited input data: a couple of sentences formed by social group terms. Additionally, Schema-Tune defines bias and language model performance measures independently from labeled data. These measures are then used in forming the language model's fine-tuning objective function and in searching for effective noises in the embedding space. Experimental evaluation over the StereoSet and Crows-Pairs datasets confirms that Schema-Tune is effective in mitigating bias in different social stereotype categories, including gender, race, and religion.

Biomedical factoid question answering aims to provide factual answers from biomedical articles for questions related to the biomedical or healthcare domain. Recent advances in biomedical factoid question answering primarily involve using pre-trained Language Models (LMs) to retrieve relevant passages or comprehend text snippets for extracting accurate answers. However, due to the relatively smaller scale of biomedical datasets compared to those used in broader domains, fine-tuning LMs for the biomedical domain often results in decreased accuracy. To address this, we introduce the Biomedical Knowledge-enhanced Question Answering Framework (BK-QAF). This framework retrieves, ranks, and employs domain-specific concepts from the Unified Medical Language System (UMLS) to enhance the comprehension and reasoning capabilities of language models. By using Graph Attention Networks (GATs) to analyze the interconnections and relationships between entities in biomedical texts, we identify the most relevant concepts for the query. The framework then ranks these UMLS concepts by relevance, expands the questions with the top-ranked concepts, and processes them using a fine-tuned language model. We evaluated our framework using the BioASQ 6b, 7b, and 8b datasets, which are widely adopted in the field. Empirical evaluations demonstrate the superior performance of the proposed framework over state-of-the-art baselines across metrics such as Strict Accuracy and MRR@5. The framework's effectiveness is assessed using three distinct GAT architectures, demonstrating robustness across different configurations.

Open-domain question answering (ODQA) stands at the forefront of advancing natural language understanding and information retrieval. Traditional ODQA systems, which predominantly utilize a two-step process of information retrieval followed by reading module, face significant challenges in aligning retrieved passages with the contextual nuances of user queries. This paper introduces a novel methodology that leverages a semi-structured knowledge graph to enhance both the accuracy and relevance of answers in ODQA systems. Our model employs a threefold approach: firstly, it extracts and ranks evidence from a textual knowledge graph, a semi-structured knowledge graph where the nodes are real-world entities and the edges are sentences that two entities co-occur in, based on the contextual relationships relevant to the question. Secondly, it utilizes this ranked evidence to re-rank initially retrieved passages, ensuring that they align more closely with the query's context. Thirdly, it integrates this evidence into a generative reading component to construct precise and context-rich answers. We compare our model, termed contextual evidence-based question answering (CEQA), against traditional and state-of-the-art ODQA systems across several datasets, including TriviaQA, Natural Questions, and SQuAD Open. Our extensive experiments and ablation studies show that CEQA significantly outperforms existing methods by improving both the relevance of retrieved passages and the accuracy of the generated answers, thereby establishing a new benchmark in ODQA.

Multi-hop question answering (QA) models excel at decomposing complex queries into sequential reasoning steps, yet they remain vulnerable to subtly flawed inference chains that appear reasonable but are factually incorrect. To quantify and address this weakness, we present FalseCoTQA, an adversarial benchmark that injects knowledge-grounded false reasoning into retrieval-augmented contexts. Unlike prior methods that merely tweak surface text, FalseCoTQA leverages a domain-agnostic knowledge graph to systematically replace entities to construct semantically coherent yet incorrect chains of thought on top of standard multi-hop datasets (HotpotQA and MuSiQue). By evaluating state-of-the-art language models on this benchmark, we observe dramatic drops in answer accuracy, highlighting their tendency to follow deceptive reasoning without verifying factual consistency. We expect the proposed benchmark to contribute to the evaluation and improvement of the robustness and reliability of language models in multi-hop question answering.

This special day event on Responsible Artificial Intelligence (AI) brings together researchers, practitioners, and policymakers to explore how data mining and machine learning systems can be designed to align with ethical principles, societal values, and human well-being. As AI technologies increasingly influence decisions in healthcare, finance, governance, and social systems, there is a critical need to develop frameworks that embed fairness, accountability, and privacy directly into the foundations of knowledge discovery. This full-day event will feature a mix of invited talks, interactive debates, expert panels, and peer-reviewed research presentations, all focused on the practical integration of ethical design into data-driven systems. The Responsible AI Day builds on the success of Canada's NSERC CREATE Program on Responsible AI, an interdisciplinary initiative training the next generation of AI researchers across computer science, law, bioethics, public health, and media studies. Topics will span scalable AI governance, privacy-preserving computation, algorithmic bias mitigation, and the socio-legal tensions emerging in generative AI. By positioning responsible AI as a sociotechnical challenge, this special day aligns with KDD's mission of advancing data science that is not only technically robust but also socially conscious.

This special issue seeks to bring together cutting-edge research, methodologies, and critical reflections on Responsible Artificial Intelligence (RAI). The issue aims to deepen our understanding of the ethical, legal, technical, and societal dimensions of AI systems. As AI technologies permeate decision-making across industry, government, and society, the demand for systems that are fair, accountable, transparent, and trustworthy has never been more urgent. This special issue will provide a dedicated venue for interdisciplinary contributions addressing key challenges and opportunities in designing, deploying, and governing responsible AI systems.

In this paper, we introduce Contextual Entity Ranking (CoER) for the task of entity retrieval. CoER utilizes a textual knowledge graph to learn entities' representations that are contextualized based on a given query. With these contextual representations and the query, CoER includes a set of models that learn to rank relevant entities. The introduced ranking models measure semantic relevance between entities' contextual representations and the textual query, between entities' contextual representations along with entities' non-contextual and general descriptions and the textual query, and finally, between entities' contextual representations and their relevance to the entities in the given query. We empirically illustrate that CoER is effective in retrieving and ranking entities across different benchmark datasets compared with state-of-the-art models. We also report ablation studies that investigate the impact of the contextual representation model and the ranking models on the final performance.

Biomedical Systematic Literature Reviews (SLRs) play a fundamental role in evidence-informed healthcare and can serve as actionable insights for researchers and policy-making organizations in the field. In this paper, we focus on the phase of 'study search' in conducting SLRs, i.e., the process of organising a comprehensive search via biomedical databases, such PubMed, in order to obtain all the relevant articles on a certain topic of interest. We introduce FASS-BSLR, a dataset and a benchmark suit to facilitate developing and evaluating fully automated techniques for study search. We also provide and analyze a set of basic methods along with a number of generative models, and report the experiment's results over the introduced dataset. We introduce a simple but effective model based on the resent transformer-based generative model, ChatGPT, for generating Boolean queries over PubMed. Through different experiments, we illustrate that this model is more effective than basic search models, than keyword search over PubMed, and than existing methods for crafting Boolean queries using ChatGPT. We show that the introduced model is even more effective than manual queries in terms of Precision, Recall, NDCG, and MAP in positions 10, and 100, but falls short of the recall that manual queries achieve at position 1000. We also report the retrieval performance of different models when a number of relevant articled have been provided as seed documents. We demonstrate that, when three documents are used as seed articles, the introduced model outperforms manual queries in all metrics except Recall@1000, on which its performance is comparable with the performance attained by manual queries.

Learning to rank models are broadly applied in ad hoc retrieval for scoring and sorting documents based on their relevance to textual queries. The generalizability of the trained model in the learning to rank approach, however, can have an impact on the retrieval performance, particularly when data includes noise and outliers, or is incorrectly collected or measured. In this paper, we introduce a Self-Distilled Learning to Rank (SDLR) framework for ad hoc retrieval, and analyze its performance over a range of retrieval datasets and also in the presence of features' noise. SDLR assigns a confidence weight to each training sample, aiming at reducing the impact of noisy and outlier data in the training process. The confidence weight is approximated based on the feature's distributions derived from the values observed for the features of the documents labeled for a query in a listwise training sample. SDLR includes a distillation process that facilitates passing on the underlying patterns in assigning confidence weights from the teacher model to the student one. We empirically illustrate that SDLR outperforms state-of-the-art learning to rank models in ad hoc retrieval. We thoroughly investigate the SDLR performance in different settings including when no distillation strategy is applied; when different portion of data are used for training the teacher and the student models, and when both teacher and student models are trained over identical data. We show that SDLR is more effective when training data are split between a teacher and a student model. We also show that SDLR's performance is robust when data features are noisy.

Knowledge Graph Question Answering (KGQA) leverages structured Knowledge Graphs (KG) to respond to Natural Language Questions (NLQ). This paper explores integrating Generative Language Models (GLMs) augmented with knowledge graph triple retrievers into the KGQA framework to generate accurate SPARQL queries from NLQs. Specifically, we evaluate the effectiveness of integrating triple retriever models with the SPARQL-generating capabilities of GLMs by investigating: (1) the standalone capabilities of GLMs independent of retriever performance, (2) the impact of incorporating a base retriever (BM25), and (3) a comparative analysis with state-of-the-art KGQA methods. Our experiments demonstrate that by incorporating a triple retrieval module, GLMs can generate accurate SPARQL queries and outperform current end-to-end KGQA methods, particularly when paired with an optimal retriever.

Due to the large size and exponential growth of the number of scientific articles published in the biomedical domain, obtaining the most relevant articles to a topic of interest, identifying reliable and scientifically sound studies, extracting salient biomedical information from texts, and synthesizing knowledge from different studies present significant challenges. While traditional term-based information analysis and machine learning techniques can be employed for literature search, information extraction, and knowledge integration, such approaches lack an effective mechanism for analyzing scientific articles based on their semantics, which is often ambiguous and contextual. In the biomedical domain, knowledge representation and semantics-enabled techniques have shown the potential for systematic retrieval, curation, organization, and interpretation of content in ways that relates well to human understanding. Here, we curate a special topic issue around the theme of Semantics-enabled Biomedical Literature Analytics. This Special Issue aims to highlight the development of novel informatics approaches for retrieval, indexing, and analysis of biomedical literature, focusing on semantics-based techniques. We place specific attention on methods that allow for the construction, analysis, and integration of biomedical knowledge bases (KBs), with the ultimate objective of employing such KBs for improving search performance over biomedical literature, enhancing biomedical information extraction, and literature-based knowledge discovery.

Objective: Study identification refers to formalizing an effective search over biomedical databases for retrieving all eligible evidence for a systematic review. Manual construction of queries, where a user submit a search query for which a biomedical search system such as PubMed would identify the most relevant documents, has been recognized as a very costly step in conducting systematic reviews. The objective of this paper is to present an automatic query generation approach to reduce the time and labor cost of manual biomedical study identification.

Materials and methods: The evaluation benchmark is the widely adopted CLEF 2018 Technology Assisted Reviews (TAR) collection, with 72 systematic reviews on Diagnosis Test Accuracy. We use and fine-tune pre-trained language models for generating high-level key-phrases and their dense embeddings. We constructed and published a dataset consists of almost one million PubMed articles' abstracts and their keywords for fine-tuning pre-trained language models. We also use concepts that are represented in the Unified Medical Language System, UMLS, for query expansion and embedding generation. We exploit and test different clustering methods, namely Agglomerative clustering, Affinity Propagation, and K-Means, over the generated embeddings to form query clauses.

Results: Our proposed methods outperform existing state-of-the-art automatic query generation models across Precision (0.0821 compared with 0.005), Recall (0.9676 compared with 0.878), and F-measures (0.2898 compared with 0.0356 in F3 measure). In addition, some of the proposed methods can even outperform the performance of the manually crafted queries in some specific measures.

Conclusion: The proposed model in this paper can be utilized to form an effective initial search query that can be further refined and updated by human reviewers for achieving the desired performance. For future work, we would like to explore the application of the presented query formalization methods in existing study identification methodologies and techniques, especially those that iteratively train machine learning models based on the domain experts' feedback on the relevancy of the retrieved studies.

Objective: With the onset of the Coronavirus Disease 2019 (COVID-19) pandemic, there has been a surge in the number of publicly available biomedical information sources, which makes it an increasingly challenging research goal to retrieve a relevant text to a topic of interest. In this paper, we propose a Contextual Query Expansion framework based on the clinical Domain knowledge (CQED) for formalizing an effective search over PubMed to retrieve relevant COVID-19 scholarly articles to a given information need.

Materials and Methods: For the sake of training and evaluation, we use the widely adopted TREC-COVID benchmark. Given a query, the proposed framework utilizes a contextual and a domain-specific neural language model to generate a set of candidate query expansion terms that enrich the original query. Moreover, the framework includes a multi-head attention mechanism that is trained alongside a learning-to-rank model for re-ranking the list of generated expansion candidate terms. The original query and the top-ranked expansion terms are posed to the PubMed search engine for retrieving relevant scholarly articles to an information need. The framework, CQED, can have four different variations, depending upon the learning path adopted for training and re-ranking the candidate expansion terms.

Results: The model drastically improves the search performance, when compared to the original query. The performance improvement in comparison to the original query, in terms of RECALL@1000 is 190.85% and in terms of NDCG@1000 is 343.55%. Additionally, the model outperforms all existing state-of-the-art baselines. In terms of P@10, the model that has been optimized based on Precision outperforms all baselines (0.7987). On the other hand, in terms of NDCG@10 (0.7986), MAP (0.3450) and bpref (0.4900), the CQED model that has been optimized based on an average of all retrieval measures outperforms all the baselines.

Conclusion: The proposed model successfully expands queries posed to PubMed, and improves search performance, as compared to all existing baselines. A success/failure analysis shows that the model improved the search performance of each of the evaluated queries. Moreover, an ablation study depicted that if ranking of generated candidate terms is not conducted, the overall performance decreases. For future work, we would like to explore the application of the presented query expansion framework in conducting technology-assisted Systematic Literature Reviews (SLR).

The importance of query performance prediction has been widely acknowledged in the literature, especially for query expansion, refinement, and interpolating different retrieval approaches. This paper proposes a novel semantics-based query performance prediction approach based on estimating semantic similarities between queries and documents. We introduce three post-retrieval predictors, namely (1) semantic distinction, (2) semantic query drift, and (3) semantic cohesion based on (1) the semantic similarity of a query to the top-ranked documents compared to the whole collection, (2) the estimation of non-query related aspects of the retrieved documents using semantic measures, and (3) the semantic cohesion of the retrieved documents. We assume that queries and documents are modeled as sets of entities from a knowledge graph, e.g., DBPedia concepts, instead of bags of words. With this assumption, semantic similarities between two texts are measured based on the relatedness between entities, which are learned from the contextual information represented in the knowledge graph. We empirically illustrate these predictors' effectiveness, especially when term-based measures fail to quantify query performance prediction hypotheses correctly. We report our findings on the proposed predictors' performance and their interpolation on three standard collections, namely ClueWeb09-B, ClueWeb12-B, and Robust04. We show that the proposed predictors are effective across different datasets in terms of Pearson and Kendall correlation coefficients between the predicted performance and the average precision measured by relevance judgments.

The importance of entity retrieval, the task of retrieving a ranked list of related entities from big knowledge bases given a textual query, has been widely acknowledged in the literature. In this paper, we propose a novel entity retrieval method that addresses the important challenge that revolves around the need to effectively represent and model context in which entities relate to each other. Based on our proposed method, a model is firstly trained to retrieve and prune a subgraph of a textual knowledge graph that represents contextual relationships between entities. Secondly, a deep model is introduced to reason over the textual content of nodes, edges, and the given question and score and rank entities in the subgraph. We show experimentally that our approach outperforms state-of-the-art methods on a number of benchmarks for entity retrieval.

Given a biomedical research question, formalizing a query for retrieving relevant evidence from Biomedical data-sets is an important methodological step in conducting literature reviews and has been the topic of a number of works on technology-assisted reviews. In this paper, we provide a deep structure that learns to effectively generate terms for expanding Biomedical queries for searching over the data-sets referenced by PubMed, the major search system for Biomedical studies. We use and fine-tune BERT for modelling contextual relationship among terms in the textual content. We evaluate our work on the CORD-19 data-set and show its effectiveness compared with the baselines.