Longtao Zheng 郑龙韬

Longtao Zheng is a PhD student at the College of Computing and Data Science, Nanyang Technological University (NTU) Singapore, advised by Prof. Bo An. Previously, he obtained his Bachelor's degree in computer science from University of Science and Technology of China (USTC) in 2022. His research focuses on foundation models for decision making, controllable video generation, and reinforcement learning.

Research Highlights

Motivated by the goal of creating general-purpose agents that can interact with the digital world, my work primarily focused on using the Internet-scale knowledge within foundation models to generalize beyond available data and to unseen situations. I am also interested in training agents and world models from videos.

Publications ( show selected / show all by date / show all by topic )

Topics: Foundation Models for Decision Making / Controllable Video Generation
Past topics: Multi-Agent Reinforcement Learning / Others (*/†: indicates equal contribution.)

MEMO: Memory-Guided Diffusion for Expressive Talking Video Generation
Longtao Zheng*, Yifan Zhang*, Hanzhong Guo, Jiachun Pan, Zhenxiong Tan, Jiahao Lu, Chuanxin Tang, Bo An, Shuicheng Yan

arXiv Preprint Paper / Project Page / Code / Model  Stars

A SOTA and open-weight model for audio-driven talking video generation

AgentStudio: A Toolkit for Building General Virtual Agents
Longtao Zheng*, Zhiyuan Huang*, Zhenghai Xue, Xinrun Wang, Bo An, Shuicheng Yan

ICLR 2025 Paper / Project Page / Code / Data  Stars

A trinity of environments, tools, and benchmarks for general virtual agents

Cradle: Empowering Foundation Agents Towards General Computer Control
Cradle Team

NeurIPS 2024 Workshop on Open-World Agents Paper / Project Page / Code  Stars

An agent that can play AAA video games

A Multimodal Foundation Agent for Financial Trading: Tool-Augmented, Diversified, and Generalist
Wentao Zhang, Lingxuan Zhao, Haochong Xia, Shuo Sun, Jiaze Sun, Molei Qin, Xinyi Li, Yuqing Zhao, Yilei Zhao, Xinyu Cai, Longtao Zheng, Xinrun Wang, Bo An

KDD 2024 Paper

The first multimodal agent for financial trading

Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control
Longtao Zheng, Rundong Wang, Xinrun Wang, Bo An

ICLR 2024 Paper / Project Page / Code

A computer agent with state abstraction, trajectory prompting, and memory

True Knowledge Comes from Practice: Aligning Large Language Models with Embodied Environments via Reinforcement Learning
Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, Bo An

ICLR 2024 Paper / Code

Train LLMs with online RL in embodied environments

Controlling Type Confounding in Ad Hoc Teamwork with Instance-wise Teammate Feedback Rectification
Dong Xing, Pengjie Gu, Qian Zheng, Xinrun Wang, Shanqi Liu, Longtao Zheng, Bo An, Gang Pan

ICML 2023 Paper

A causality-based solution to deal with type confounding in ad hoc teamwork

Multi-Agent Multi-Game Entity Transformer: Towards Generalist Models in MARL
Rundong Wang, Weixuan Wang, Xianhan Zeng, Liang Wang, Zhengjie Lian, Yiming Gao, Feiyu Liu, Siqin Li, Xianliang Wang, Qiang Fu, Wei Yang, Lanxiao Huang, Longtao Zheng, Zinovi Rabinovich, Bo An

DAI 2024 Paper

A generalist transformer for Honor of Kings, Starcraft II, and Neural MMO

Towards Skilled Population Curriculum for Multi-Agent Reinforcement Learning
Rundong Wang*, Longtao Zheng*, Wei Qiu, Bowei He, Bo An, Zinovi Rabinovich, Yujing Hu, Yingfeng Chen, Tangjie Lv, Changjie Fan

arXiv Preprint Paper / Code

Autocurricula for MARL in complex sparse-reward environments like Google Football

Contextual Data Cleaning with Ontology Functional Dependencies
Zheng Zheng, Longtao Zheng, Morteza Alipourlangouri, Fei Chiang, Lukasz Golab, Jaroslaw Szlichta, Sridevi Baskaran

ACM Journal of Data and Information Quality (JDIQ) Paper