Kamal Acharya

Postdoctoral Research Associate at Baylor University

Back to Publications

Conference Paper

SKGHOI: Spatial-Semantic Knowledge Graph for Human-Object Interaction Detection

By Lijing Zhu Qizhen Lan Alvaro Velasquez Houbing Herbert Song Kamal Acharya Qing Tian Shuteng Niu

An IEEE ICDMW conference paper on human-object interaction detection using a spatial-semantic knowledge graph.

2023 IEEE ICDMW 2023 DOI: 10.1109/ICDMW60847.2023.00155

Knowledge Graph Human-Object Interaction Computer Vision

Open Repository Copy DOI Cite

Abstract

Human-Object Interaction (HOI) detection is a challenging computer vision task that aims to recognize and understand the interactions between humans and objects in images or videos. Existing techniques heavily rely on appearance-based features and computationally expensive transformer models for semantic representation. In this paper, we propose SKGHOI (Spatial-Semantic Knowledge Graph for HOI), a novel graph-based approach that efficiently captures semantic representation by integrating spatial and semantic knowledge. SKGHOI constructs a graph with interaction components as nodes and spatial relationships as edges. Our approach leverages spatial and semantic encoders to extract spatial and semantic information, which are then fused to create a knowledge graph that captures the semantic representation of HOIs. Compared to existing methods, SKGHOI offers computational efficiency and the incorporation of prior knowledge, making it practical for real-world applications. Experimental evaluations on the widely-used HICO-DET datasets demonstrate that SKGHOI outperforms state-of-the-art graph-based methods by a significant margin, showcasing its effectiveness and potential for improving the accuracy and efficiency of HOI detection.

In brief

What This Work Does

The paper improves computer vision models that detect how people interact with objects by using a knowledge graph that captures both spatial relationships and semantic meaning.

Research impact

Why It Matters

The work offers a more efficient graph-based direction for human-object interaction detection by incorporating prior knowledge, spatial context, and semantic representation in one HOI graph.

Paper at a glance

Research Scope

4Research stages
5Methods or application areas
5Future research directions

Spatial-Semantic HOI Detection Framework

1

Detect Instances

Uses object detection to identify human and object instances and generate human-object candidate pairs.

2

Build HOI Graph

Represents interaction components as nodes and spatial relationships as edges.

3

Translate Semantics

Uses knowledge graph embedding to produce compact semantic features for HOI representation.

4

Predict Interactions

Combines appearance, spatial, and translated semantic features in a graph model for HOI prediction.

Key Contributions

  • Introduces SKGHOI, a spatial-semantic knowledge graph approach for HOI detection.
  • Represents interaction components as graph nodes and spatial relationships as graph edges.
  • Combines appearance features, spatial features, and KGE-translated semantic features.
  • Improves graph-based HOI detection performance on HICO-DET.
  • Uses compact semantic embeddings to reduce computational burden compared with heavier semantic representations.

Method Components

Knowledge Graph

Provides structured prior knowledge for human-object interaction reasoning.

Spatial Encoder

Represents geometric relationships among humans, objects, and visual context.

Semantic Encoder

Adds meaning-level information that complements visual appearance features.

Graph-Based HOI Detection

Combines spatial and semantic signals for efficient interaction detection.

Knowledge Graph Embedding

Translates semantic information into compact features that improve HOI graph representations.

Research Gaps

  1. Spatial reasoning
  2. Semantic priors
  3. Efficient inference
  4. Visual relationship modeling
  5. Benchmark generalization

Publication Details

Type
Conference Paper
Venue
IEEE ICDMW 2023
Year
2023
Pages
1186-1193

Research Topics

Knowledge Graph Human-Object Interaction Computer Vision Graph Neural Network Spatial-Semantic Representation

Citation

@inproceedings{zhu2023skghoi,
  author={Zhu, L. and Lan, Q. and Velasquez, Alvaro and Song, Houbing and Acharya, Kamal and Tian, Q. and Niu, S.},
  title={SKGHOI: Spatial-Semantic Knowledge Graph for Human-Object Interaction Detection},
  booktitle={IEEE ICDMW 2023},
  year={2023},
  pages={1186--1193},
  doi={10.1109/ICDMW60847.2023.00155}
}