How New Information Permeates the Knowledge of Large Language Models and How to Control This Influence

Large language models (LLMs) learn continuously through the accumulation of gradient-based updates. However, how individual pieces of new information influence existing knowledge, leading to both beneficial generalization and problematic hallucinations, is still poorly understood. A recent study sheds light on this phenomenon and offers solutions for more targeted knowledge integration.

Researchers have found that LLMs exhibit a "priming effect" when learning new information: Learning a new fact can cause the model to apply this knowledge in inappropriate, unrelated contexts. To systematically investigate this phenomenon, "Outlandish," a carefully curated dataset of 1320 different text examples, was developed. This dataset was designed to examine how new knowledge permeates an LLM's existing knowledge base.

Using "Outlandish," it was demonstrated that the degree of priming after learning new information can be predicted by measuring the token probability of keywords before learning. This relationship is robust across different model architectures (PALM-2, Gemma, Llama), sizes, and training stages. The results suggest that the probability with which an LLM uses certain words is an indicator of how susceptible it is to priming effects.

To modulate the influence of new information on model behavior, two new techniques were developed:

These approaches reduce unwanted priming effects by 50-95% while preserving the model's ability to learn new information. The "Stepping-Stone" strategy gradually introduces the model to new information, while the "Ignore-k" method ignores specific updates during the training process.

The findings of this study offer both empirical insights into the learning processes of LLMs and practical tools for improving the specificity of knowledge insertion. The ability to control the influence of new data is crucial for the development of more robust and reliable LLMs. Especially for companies like Mindverse, which develop customized AI solutions such as chatbots, voicebots, AI search engines, and knowledge systems, these findings are of great importance. Precise control over the knowledge acquisition of LLMs enables the development of AI systems that effectively utilize specific information while minimizing unwanted side effects.

Research in this area is dynamic and promising. Further investigation is necessary to fully understand the complex relationships between knowledge accumulation, priming effects, and generalization in LLMs. The development of methods for targeted knowledge integration will significantly influence the performance and reliability of AI systems in the future.

Bibliography:

https://openreview.net/forum?id=NGKQoaqLpo
https://openreview.net/pdf/6ae382f493c61d71d3eac223066862546ff54232.pdf
https://paperreading.club/page?id=299202
https://iclr.cc/virtual/2025/papers.html
https://arxiv.org/abs/2404.00213
https://iclr.cc/virtual/2025/calendar
https://www.fit.fraunhofer.de/en/business-areas/data-science-and-artificial-intelligence/knowledge-enhanced-large-language-models.html
https://aclanthology.org/2023.emnlp-main.87.pdf
https://www.reddit.com/r/LocalLLaMA/comments/1ao2bzu/best_way_to_add_knowledge_to_a_llm/
https://stackoverflow.blog/2023/10/18/retrieval-augmented-generation-keeping-llms-relevant-and-current/