NeurIPS 2025 Main Track

Learning Human-Like RL Agents Through Trajectory Optimization With Action Quantization

Jian-Ting Guo^1*, Yu-Cheng Chen^1*, Ping-Chun Hsieh^1†, Kuo-Hao Ho¹, Po-Wei Huang¹, Ti-Rong Wu^2†, I-Chen Wu^1,2

¹ National Yang Ming Chiao Tung University ² Academia Sinica

^* Equal contribution ^† Corresponding author

MAQ learns discrete, reusable macro actions from human demonstrations and combines them with reinforcement learning to produce more human-like control behavior.

OpenReview Paper Code Slides Poster BibTeX

Abstract

Learning control policies that better resemble human behavior.

Reward-optimized reinforcement learning agents can perform well while following trajectories that differ markedly from human demonstrations. MAQ frames human-likeness as a trajectory optimization problem and learns a discrete macro-action codebook from demonstration windows with a Conditional VQ-VAE.

An RL policy then selects codebook indices instead of individual actions. This formulation can be integrated with offline and online RL algorithms, improving trajectory similarity on D4RL Adroit tasks and human-likeness in a two-stage human evaluation.

Architecture of Macro Action Quantization

Method

Method overview

MAQ separates the representation of human behavior from policy learning. First, it learns a discrete codebook of reusable action sequences from human demonstrations. Then an RL agent operates in this compact code space and decodes each selected code into an executable macro action.

Human behavior distillation

A Conditional VQ-VAE learns macro actions from sliding windows over human action trajectories.

Reinforcement learning with macro actions

RL with macro actions

The policy acts in the learned discrete code space, then the decoder converts each selected code into a macro action.

Results

Evaluation on D4RL Adroit tasks

Experiments on D4RL Adroit benchmarks show that MAQ improves trajectory similarity when integrated with IQL, SAC, and RLPD. The evaluation uses Dynamic Time Warping and Wasserstein Distance to compare agent behavior with human demonstrations.

Benchmarks D4RL Adroit door, hammer, pen, and relocate tasks

Algorithms IQL, SAC, and RLPD with MAQ integration

Human evaluation MAQ achieved a 71% win rate in the reported human evaluation.

Average results across D4RL Adroit tasks. Higher values indicate stronger performance.
Pair	DTW_s	WD_s	Success
IQL to MAQ+IQL	0.21 to 0.63	0.38 to 0.64	0.14 to 0.39
SAC to MAQ+SAC	-0.49 to 0.56	-0.17 to 0.59	0.19 to 0.28
RLPD to MAQ+RLPD	0.10 to 0.56	0.08 to 0.61	0.68 to 0.52

MAQ RLPD trajectory similarity heatmap — Trajectory similarity and success rate trends with different macro-action lengths in MAQ+RLPD.

Turing test result chart — Turing-test results from the human evaluation study.

Human-likeness ranking heatmap — Human-likeness ranking results across evaluated agents.

Qualitative Results

Behavior comparisons

Select an RL algorithm to compare its baseline behavior with the MAQ-enhanced variant across four Adroit manipulation tasks.

RLPD vs MAQ+RLPD

Door · MAQ+RLPD

Door · RLPD

Hammer · MAQ+RLPD

Hammer · RLPD

Pen · MAQ+RLPD

Pen · RLPD

Relocate · MAQ+RLPD

Relocate · RLPD

Citation

BibTeX

@inproceedings{
guo2025learning,
title={Learning Human-Like {RL} Agents Through Trajectory Optimization With Action Quantization},
author={Jian-Ting Guo and Yu-Cheng Chen and Ping-Chun Hsieh and Kuo-Hao Ho and Po-Wei Huang and Ti-Rong Wu and I-Chen Wu},
booktitle={Advances in Neural Information Processing Systems},
year={2025},
url={https://openreview.net/forum?id=1A4Nlibwl5}
}