RALP Framework: A New Paradigm for Knowledge Graph Completion from Paderborn University Based on Chain-of-Thought Prompting
Knowledge Graphs (KGs) are a key technology driving modern search engines and intelligent Q&A systems, and link prediction is a core task for ensuring their data integrity. Recently, a research team from Paderborn University in Germany proposed a new framework called RALP (Reasoning on Arrays of Language Prompts), which completely changes the traditional methods in this field. This framework reframes the link prediction problem as a prompt learning task based on Large Language Models (LLMs), achieving state-of-the-art results on multiple benchmarks by learning Chain-of-Thought (CoT) prompts through gradient-free optimization.
The Traditional Paradigm of Knowledge Graph Completion and Its Limitations
For a long time, knowledge graph link prediction has been dominated by Knowledge Graph Embedding (KGE) models. The core idea of these methods is to map entities and relations in the graph into low-dimensional continuous vector spaces and then use a parameterized scoring function to evaluate the plausibility of factual triples. Although they perform consistently on closed, static datasets, KGE methods face three major bottlenecks:
- Poor Generalization: For new entities or relations not seen during training, pre-trained embedding vectors cannot provide effective representations, leading to poor model performance on dynamically changing knowledge graphs.
- Difficulty Handling Literals: Real-world knowledge graphs contain a large number of numerical values (like population, GDP) and text descriptions. Traditional embedding methods struggle to effectively model and reason with these literals.
- High Training Costs: KGE models rely on large-scale labeled data and gradient descent optimization, demanding significant computational resources and data, which limits their application in resource-constrained scenarios.
RALP: Replacing Numerical Parameters with Learnable Prompts
To overcome these limitations, the RALP framework introduces a paradigm-level innovation. It replaces the traditional KGE model’s scoring function φ_Θ, defined by numerical vector parameters Θ, with a new scoring function φ_β≤c defined by a string parameter β. The core parameter β here is a Chain-of-Thought (CoT) prompt string of at most c tokens, which guides the LLM on how to reason and complete knowledge based on contextual information.
The key to this framework is its “gradient-free” learning mechanism. The research team uses the MIPRO (Metropolis-Hastings In-Context-learning PROmpting) algorithm, an automated prompt engineering technique based on Bayesian optimization. It can efficiently search for and determine the optimal prompt β for a specific task and knowledge graph without needing access to model gradients. The entire optimization process can be completed with a very small number of training samples (no more than 30).
The RALP framework comes in two versions:
- Basic RALP: Directly utilizes the LLM’s in-context learning capabilities with a default prompt for link prediction.
- Optimized RALP†: Learns a customized, optimal CoT prompt for a specific task using the MIPRO algorithm to achieve the best performance.
Comprehensive Benchmarking: Performance Validation on Three Core Tasks
The research team systematically evaluated the RALP framework on three main types of knowledge graph completion tasks, and the experimental results verified its outstanding performance.
Transductive Link Prediction and Data Augmentation
On the classic Countries dataset (S1/S2/S3), the optimized RALP† medium model achieved perfect prediction results on the Countries-S1 subset, with its MRR (Mean Reciprocal Rank), Hits@1, Hits@3, and Hits@10 metrics all reaching 1.000. More importantly, RALP demonstrated powerful data augmentation capabilities. The team added the high-confidence missing triples predicted by RALP to the training set to retrain traditional KGE models. The results showed a general and significant improvement in the performance of mainstream KGE models. For example, on the Countries-S1 dataset:
- DistMult's MRR improved from 0.783 to 0.990.
- Keci's MRR improved from 0.716 to 1.000.
Numerical Literal Reasoning
For numerical prediction tasks that are difficult for KGE models, the team conducted tests on the LitWD1K dataset. This dataset contains 10 types of numerical attributes, such as population, GDP per capita, and Human Development Index. Experiments showed that not only did RALP’s predicted mean values closely match the true mean values, but it also exhibited excellent uncertainty calibration. The Interval Coverage Rate (ICR) for all attributes exceeded 75%. For the “Human Development Index” attribute, for instance, the Mean Absolute Error (MAE) was only 0.059.
Complex Logic (OWL) Instance Retrieval
On the Father dataset, which involves complex logical reasoning, RALP was required to retrieve instances based on 130 Description Logic (DL) class expressions of varying complexity. The results showed that the average Jaccard similarity between RALP’s predictions and the true instance sets exceeded 88%. For atomic concepts and numerical restriction classes like “at least,” the Jaccard similarity reached 1.000. The experiment also found that using the more natural language-like Manchester syntax as input was more effective at stimulating the LLM’s reasoning performance than using formal DL syntax.
Core Advantages, Current Limitations, and Future Outlook
In summary, the core advantages of the RALP framework are:
- Efficient Few-Shot Learning: Prompt optimization requires only a small number of samples, greatly reducing data dependency.
- Strong Generalization: Natively supports reasoning on unseen entities, relations, and literals.
- Task Versatility: A single framework uniformly supports entity prediction, numerical reasoning, and complex logic retrieval.
- Compatibility and Empowerment: Can be used as a data augmentation tool to directly improve the performance of existing KGE models.
At the same time, the study points out the current limitations of the method, including the model’s sensitivity to extreme outliers in the input context, the requirement for entities and relations to have semantically readable text labels (rather than pure IDs), and the token limits and inference costs associated with large models themselves.
To address these challenges, the research team has planned three future optimization directions: first, to develop context sampling algorithms to tackle the token limit issue when applying the framework to very large-scale knowledge graphs; second, to explore transferring RALP’s reasoning capabilities to lighter models through knowledge distillation to reduce deployment costs; and third, to further enhance the model’s ability to reason with universal quantifiers and complex nested logical expressions.
