Kamal Acharya

Ph.D. Candidate at UMBC, AI Researcher

Back to Publications

Journal Article

A Survey on Symbolic Knowledge Distillation of Large Language Models

An IEEE Transactions on Artificial Intelligence survey on converting implicit knowledge in large language models into symbolic, explicit, interpretable, and efficient representations.

2024 IEEE Transactions on Artificial Intelligence DOI: 10.1109/TAI.2024.3428519

Symbolic Knowledge Distillation Large Language Models Neurosymbolic AI

DOI Publisher Page Cite

Abstract

This survey article delves into the emerging and critical area of symbolic knowledge distillation in large language models (LLMs). As LLMs such as generative pretrained transformer-3 (GPT-3) and bidirectional encoder representations from transformers (BERT) continue to expand in scale and complexity, the challenge of effectively harnessing their extensive knowledge becomes paramount. This survey concentrates on the process of distilling the intricate, often implicit knowledge contained within these models into a more symbolic, explicit form. This transformation is crucial for enhancing the interpretability, efficiency, and applicability of LLMs. We categorize the existing research based on methodologies and applications, focusing on how symbolic knowledge distillation can be used to improve the transparency and functionality of smaller, more efficient artificial intelligence (AI) models. The survey discusses the core challenges, including maintaining the depth of knowledge in a comprehensible format, and explores the various approaches and techniques that have been developed in this field. We identify gaps in current research and potential opportunities for future advancements. This survey aims to provide a comprehensive overview of symbolic knowledge distillation in LLMs, spotlighting its significance in the progression toward more accessible and efficient AI systems.

Plain-Language Summary

This paper explains how knowledge inside large language models can be distilled into clearer symbolic forms so smaller AI systems can become more transparent, efficient, and easier to reason about.

Why This Paper Matters

Large language models contain extensive knowledge, but their size, opacity, and computational cost limit their practical use in many settings. Symbolic knowledge distillation offers a pathway to extract that knowledge into explicit forms such as rules, knowledge graphs, semantic frames, and structured datasets. This can support smaller, more transparent, and more deployable AI systems while advancing neurosymbolic AI.

Research Summary

This paper addresses a growing challenge in large language models: powerful models often contain useful knowledge, but that knowledge is implicit, difficult to inspect, and expensive to use directly. Symbolic knowledge distillation aims to convert parts of this hidden model knowledge into more explicit and structured forms.

The survey reviews approaches that make LLM knowledge more interpretable, compact, and reusable. It connects knowledge distillation with symbolic representations, smaller models, transparency, efficiency, and the broader goal of making AI systems easier to understand and deploy.

The paper is especially relevant for researchers working at the intersection of LLMs and neurosymbolic AI. It frames symbolic distillation as a path toward models that retain useful reasoning capabilities while becoming more accessible, efficient, and explainable.

Symbolic Knowledge Distillation Pipeline

1

Prompt and Generate

Use carefully designed prompts to elicit factual, commonsense, reasoning, or task-specific knowledge from a large language model.

2

Extract and Structure

Apply NLP methods and symbolic processing to transform generated text into rules, relations, knowledge graphs, semantic frames, or structured datasets.

3

Filter and Validate

Use human experts, critic models, benchmarks, and quality filters to preserve accuracy, consistency, relevance, and knowledge depth.

4

Train Efficient Models

Use the refined symbolic knowledge to train smaller models, open models, or downstream systems with improved interpretability and efficiency.

Key Contributions

  • Surveys symbolic knowledge distillation methods for large language models.
  • Connects distillation with interpretability, efficiency, transparency, and model usability.
  • Categorizes existing research by methodology and application area.
  • Identifies research gaps for future symbolic distillation and accessible AI systems.

Modeling Approaches Reviewed

Direct Distillation

Extracts symbolic knowledge directly from LLM outputs and converts it into structured forms such as rules, statements, or knowledge graphs.

Multilevel Distillation

Iteratively trains smaller student models and filters generated knowledge to improve fidelity, conciseness, and transfer quality.

Reinforcement Learning-Based Distillation

Uses reward models, filtering, and iterative training to align generated symbolic knowledge with human preferences or task goals.

Instruction Tuning

Uses symbolic or generated instruction data to make models follow task-specific commands more effectively.

Open Knowledge and Model Creation

Uses symbolic distillation to generate open datasets and train accessible models from knowledge extracted from larger systems.

Research Gaps

Dataset quality Bias mitigation Human oversight Compact model performance Instruction diversity Continuous learning Evaluation benchmarks Cross-domain generalization

Publication Details

Type
Journal Article
Venue
IEEE Transactions on Artificial Intelligence
Year
2024
Published
December 1, 2024
Volume
5
Issue
12
Pages
5928-5948

Authors

Research Topics

Symbolic Knowledge Distillation Large Language Models Neurosymbolic AI

Citation

@article{acharya2024survey,
  author={Acharya, Kamal and Velasquez, Alvaro and Song, Houbing Herbert},
  title={A Survey on Symbolic Knowledge Distillation of Large Language Models},
  journal={IEEE Transactions on Artificial Intelligence},
  year={2024},
  volume={5},
  number={12},
  pages={5928--5948},
  doi={10.1109/TAI.2024.3428519}
}