Kamal Acharya

Postdoctoral Research Associate at Baylor University

Back to Publications

Journal Article

A Survey on Symbolic Knowledge Distillation of Large Language Models

By Kamal Acharya Alvaro Velasquez Houbing Herbert Song

An IEEE Transactions on Artificial Intelligence survey on converting implicit knowledge in large language models into symbolic, explicit, interpretable, and efficient representations.

2024 IEEE Transactions on Artificial Intelligence DOI: 10.1109/TAI.2024.3428519

Symbolic Knowledge Distillation Large Language Models Neurosymbolic AI

DOI Publisher Page Cite

Abstract

This survey article delves into the emerging and critical area of symbolic knowledge distillation in large language models (LLMs). As LLMs such as generative pretrained transformer-3 (GPT-3) and bidirectional encoder representations from transformers (BERT) continue to expand in scale and complexity, the challenge of effectively harnessing their extensive knowledge becomes paramount. This survey concentrates on the process of distilling the intricate, often implicit knowledge contained within these models into a more symbolic, explicit form. This transformation is crucial for enhancing the interpretability, efficiency, and applicability of LLMs. We categorize the existing research based on methodologies and applications, focusing on how symbolic knowledge distillation can be used to improve the transparency and functionality of smaller, more efficient artificial intelligence (AI) models. The survey discusses the core challenges, including maintaining the depth of knowledge in a comprehensible format, and explores the various approaches and techniques that have been developed in this field. We identify gaps in current research and potential opportunities for future advancements. This survey aims to provide a comprehensive overview of symbolic knowledge distillation in LLMs, spotlighting its significance in the progression toward more accessible and efficient AI systems.

In brief

What This Work Does

This paper explains how knowledge inside large language models can be distilled into clearer symbolic forms so smaller AI systems can become more transparent, efficient, and easier to reason about.

Research impact

Why It Matters

Large language models contain extensive knowledge, but their size, opacity, and computational cost limit their practical use in many settings. Symbolic knowledge distillation offers a pathway to extract that knowledge into explicit forms such as rules, knowledge graphs, semantic frames, and structured datasets. This can support smaller, more transparent, and more deployable AI systems while advancing neurosymbolic AI.

Paper at a glance

Research Scope

4Research stages
5Methods or application areas
8Future research directions

Symbolic Knowledge Distillation Pipeline

1

Prompt and Generate

Use carefully designed prompts to elicit factual, commonsense, reasoning, or task-specific knowledge from a large language model.

2

Extract and Structure

Apply NLP methods and symbolic processing to transform generated text into rules, relations, knowledge graphs, semantic frames, or structured datasets.

3

Filter and Validate

Use human experts, critic models, benchmarks, and quality filters to preserve accuracy, consistency, relevance, and knowledge depth.

4

Train Efficient Models

Use the refined symbolic knowledge to train smaller models, open models, or downstream systems with improved interpretability and efficiency.

Key Contributions

  • Surveys symbolic knowledge distillation methods for large language models.
  • Connects distillation with interpretability, efficiency, transparency, and model usability.
  • Categorizes existing research by methodology and application area.
  • Identifies research gaps for future symbolic distillation and accessible AI systems.

Modeling Approaches Reviewed

Direct Distillation

Extracts symbolic knowledge directly from LLM outputs and converts it into structured forms such as rules, statements, or knowledge graphs.

Multilevel Distillation

Iteratively trains smaller student models and filters generated knowledge to improve fidelity, conciseness, and transfer quality.

Reinforcement Learning-Based Distillation

Uses reward models, filtering, and iterative training to align generated symbolic knowledge with human preferences or task goals.

Instruction Tuning

Uses symbolic or generated instruction data to make models follow task-specific commands more effectively.

Open Knowledge and Model Creation

Uses symbolic distillation to generate open datasets and train accessible models from knowledge extracted from larger systems.

Research Gaps

  1. Dataset quality
  2. Bias mitigation
  3. Human oversight
  4. Compact model performance
  5. Instruction diversity
  6. Continuous learning
  7. Evaluation benchmarks
  8. Cross-domain generalization

Publication Details

Type
Journal Article
Venue
IEEE Transactions on Artificial Intelligence
Year
2024
Published
December 1, 2024
Volume
5
Issue
12
Pages
5928-5948

Research Topics

Symbolic Knowledge Distillation Large Language Models Neurosymbolic AI

Citation

@article{acharya2024survey,
  author={Acharya, Kamal and Velasquez, Alvaro and Song, Houbing Herbert},
  title={A Survey on Symbolic Knowledge Distillation of Large Language Models},
  journal={IEEE Transactions on Artificial Intelligence},
  year={2024},
  volume={5},
  number={12},
  pages={5928--5948},
  doi={10.1109/TAI.2024.3428519}
}