Siteng Huang (黄思腾)

Siteng Huang (黄思腾)

Algorithm Expert in Alibaba DAMO Academy

👋 Hi! I am Siteng Huang (黄思腾 in Chinese). I work at DAMO Academy as an Algorithm Expert in Hangzhou. I received my Ph.D. degree from Zhejiang University in June 2024, affiliated with a joint program with Westlake University at Machine Intelligence Laboratory (MiLAB) and advised by Prof. Donglin Wang. In my Ph.D. study, I also spent wonderful internship time at TongYi Lab, Alibaba Group. Before that, I received my B.Eng. Degree from School of Computer Science, Wuhan University in June 2019.

🔬 My research has centered on the perception, understanding, reasoning, and generation of multimodal (including images, videos, language, dynamics, etc.) data from both the internet and the physical world. I also focus on efficientAI (in terms of data, time, parameters, memory, etc.) for multimodal applications. I have published 20+ papers on the above topics at the top-tier international AI conferences and journals. Recently, I devote myself to the development of multi-modal generative, embodied, and unified foundation models.

🌟 I am honored to have supervised several self-motivated visiting students and research assistants in their research and publications. If you are seeking any form of academic cooperation, please feel free to email me at siteng.huang[AT]gmail.com (replace [AT] with @). Additionly, I maintain close cooperation with MiLAB from Westlake University. This top-tier robot learning lab is actively looking for visiting students and RAs (please refer to Recruitment). Specially, if you are willing to cooperate with me there, please also send me a copy when sending your CV to the lab.

📢 News

  • 2025/05/22 [Preprint] We released VARD, a novel RL fine-tuning method on diffusion-based generative models for both protein structure and text-to-image synthesis, enhancing sample quality with improved efficiency, effective mitigation of reward hacking, and broad applicability.
  • 2025/05/21 [Preprint] We released SSR, raw depth data -> structured, interpretable textual rationales -> compact latent embeddings, enhancing spatial reasoning capabilities of MLLMs. See Project page and Github!
  • 2025/05/07 [Preprint] We released OpenHelix, a low-cost open-source dual-system VLA with systematic empirical evaluations on the core design elements. Code and List of papers have been available!
  • 2025/03/31 [Preprint] We released Unicorn to explore the question: can high-quality multimodal training data be synthesized purely from text?
  • 2025/03/28 [Survey Preprint] We released Exploring the Evolution of Physics Cognition in Video Generation: A Survey, which dives deep into the development of physics cognition in video generation, from basic perception to active cognition! List of papers has been available!
  • 2025/03/11 [TCSVT’25] M2IST, a novel Multi-Modal Interactive Side-Tuning method that effectively addresses the challenges of insufficient multi-modal interaction and high GPU memory consumption, got accepted for IEEE Transactions on Circuits and Systems for Video Technology! Code has been available!
  • 2025/02/24 [Preprint] We released Humanoid-VLA, a novel framework that integrates language understanding, egocentric scene perception, and motion control, enabling universal humanoid control!
  • 2025/01/28 [ICRA’25] QUART-Online, a novel latency-free quadruped MLLM model that achieves real-time inference while boosting the success rate across various tasks by 65%, got accepted for ICRA 2025! See Project page.
  • 2025/01/23 [ICLR’25] ToCa, a token-wise feature caching method that achieves a 2x acceleration for PixArt-α, OpenSora, and DiT while maintaining nearly lossless generation quality, got accepted for ICLR 2025! Code has been available!
  • 2025/01/10 [Preprint] We released GlobalCom2, a “global-to-local” approach for training-free acceleration of high-resolution MLLMs with AnyRes strategy. Code has been available!
  • 2024/12/13 [Preprint] We released Score and Distribution Matching Policy, which transforms diffusion-based policies into single-step generators for light-weight deployment, fast reasoning and precise action generation. Project page has been available.
  • 2024/12/10 [Preprint] We released CARP, Coarse-to-fine AutoRegressive Prediction for visuomotor policy learning. The approach produces highly accurate and smooth robot actions, achieving up to a 10% improvement of success rates, and delivers 10x faster inference compared to state-of-the-art policies. Project page with cool videos has been available. Code will be available soon!
  • 2024/12/10 [AAAI’25] Cobra, the first Mamba-based MLLM with efficient inference, got accepted for AAAI 2025! See Project page.
  • 2024/11/27 [Preprint] We released a new work on token reduction for MLLM inference acceleration, which proposes a unified paradigm to demystify the popular works and guide the future designs, and further offers a suite of methods FiCoCo grounded in the paradigm. Project page and code have been available!
  • 2024/09/09 [New Start] Joined Alibaba DAMO Academy as an Algorithm Expert!
  • 2024/07/16 [MM’24] One paper (ProFD) got accepted for ACM MM 2024. Congratulations to all collaborators!
  • 2024/07/09 [Scholar’24] 2024 Scholar Metrics was released by Google Scholar. Our paper “DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting” ranked 7th of the CIKM 2019 conference according to the citations, and 13th within five years.
  • 2024/07/01 [ECCV’24] Two papers (PiTe and QUAR-VLA) got accepted for ECCV 2024. 2024/08/12 PiTe got accepted as an Oral paper!
  • 2024/06/04 [Graduation] I successfully defended my dissertation. So many thanks to my Ph.D. committee (Prof. Xiaogang Jin, Prof. Mai Xu, Prof. Changxin Gao, Prof. Fajie Yuan, Prof. Peidong Liu, Prof. Xiaofei Li) and my advisor!
  • 2024/03/29 [VALSE’24] Troika got accepted as VALSE 2024 Poster! 2024/05/05 Our Cobra was selected for VALSE 2024 Annual Progress Representation. Thanks to all the committee for the approval!
  • 2024/03/13 [ICME’24] One paper (DARA) about parameter-efficient tuning for visual grounding got accepted for ICME 2024 (Oral).
  • 2024/02/27 [Award] Awarded as Zhejiang University 2024 Outstanding Graduates!
  • 2024/02/27 [CVPR’24] Three papers (ADI, Troika, SimM) as first/co-first author got accepted for CVPR 2024. Congratulations to all collaborators!

📝 Publications

†: Equal contribution ✉: Corresponding author

Peer-reviewed Conference

Xinyang Tong, Pengxiang Ding, Donglin Wang, Wenjie Zhang, Can Cui, Mingyang Sun, Yiguo Fan, Han Zhao, Hongyin Zhang, Yonghao Dang, Siteng Huang, Shangke Lyu, "QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning". In Proceedings of the 2025 IEEE International Conference on Robotics and Automation. [arXiv] [project page]

Chang Zou†, Xuyang Liu†, Ting Liu, Siteng Huang, Linfeng Zhang, "Accelerating Diffusion Transformers with Token-wise Feature Caching". In Proceedings of the 13th International Conference on Learning Representations. [arXiv] [github]

Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, Donglin Wang, "Cobra: Extending Mamba to Multi-Modal Large Language Model for Efficient Inference". In Proceedings of the 39th AAAI Conference on Artificial Intelligence. [arXiv] [pdf] [project page] [Chinese intro (Zhihu)] [github] [demo] [video (Youtube)] [机器之心] [Twitter@AK]

Can Cui†, Siteng Huang†, Wenxuan Song, Pengxiang Ding, Zhang Min, Donglin Wang, "ProFD: Prompt-Guided Feature Disentangling for Occluded Person Re-Identification". In Proceedings of the 32nd ACM International Conference on Multimedia. [arXiv] [github] [OpenReview]

(Oral) Yang Liu†, Pengxiang Ding†, Siteng Huang, Min Zhang, Han Zhao, Donglin Wang, "PiTe: Pixel-Temporal Alignment for Large Video-Language Model". In Proceedings of the European Conference on Computer Vision 2024. [arXiv] [github] [dataset]

Pengxiang Ding, Han Zhao, Wenxuan Song, Wenjie Zhang, Min Zhang, Siteng Huang, Ningxi Yang, Donglin Wang, "QUAR-VLA: Vision-Language-Action Model for Quadruped Robots". In Proceedings of the European Conference on Computer Vision 2024. [arXiv]

(Oral) Ting Liu†, Xuyang Liu†, Siteng Huang, Honggang Chen, Quanjun Yin, Long Qin, Donglin Wang, Yue Hu, "DARA: Domain- and Relation-aware Adapters Make Parameter-efficient Tuning for Visual Grounding". In Proceedings of the IEEE Conference on Multimedia Expo 2024. [arXiv] [github]

Siteng Huang, Biao Gong, Yutong Feng, Xi Chen, Yuqian Fu, Yu Liu, Donglin Wang, "Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation". In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024. [arXiv] [dataset] [project page] [poster (CVPR 2024)]

Biao Gong†, Siteng Huang†, Yutong Feng, Shiwei Zhang, Yuyuan Li, Yu Liu, "Check, Locate, Rectify: A Training-Free Layout Calibration System for Text-to-Image Generation". In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024. [arXiv] [project page] [poster (CVPR 2024)]

Siteng Huang, Biao Gong, Yutong Feng, Min Zhang, Yiliang Lv, Donglin Wang, "Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning". In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2024. [arXiv] [project page] [github] [poster (CVPR 2024)] [poster (VALSE 2024)]

Xuyang Liu†, Siteng Huang†, Yachen Kang, Honggang Chen, Donglin Wang, "VGDiffZero: Text-to-image Diffusion Models Can Be Zero-shot Visual Grounders". In Proceedings of the 2024 IEEE International Conference on Acoustics, Speech and Signal Processing. [arXiv] [code] [poster]

Shuanghao Bai, Min Zhang, Wanqi Zhou, Siteng Huang, Zhirong Luan, Donglin Wang, Badong Chen, "Prompt-based Distribution Alignment for Unsupervised Domain Adaptation". In Proceedings of the 38th AAAI Conference on Artificial Intelligence. [arXiv]

Siteng Huang, Biao Gong, Yulin Pan, Jianwen Jiang, Yiliang Lv, Yuyuan Li, Donglin Wang, "VoP: Text-Video Co-operative Prompt Tuning for Cross-Modal Retrieval". In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2023. [project page] [arXiv] [open access] [video (Youtube)] [github] [ModelScope] [poster] [slide]

Siteng Huang, Qiyao Wei, Donglin Wang, "Reference-Limited Compositional Zero-Shot Learning". In Proceedings of the 2023 ACM International Conference on Multimedia Retrieval. [project page] [arXiv] [video (Google Drive)] [github] [slide]

Min Zhang, Siteng Huang, Wenbin Li, Donglin Wang, "Tree Structure-Aware Few-Shot Image Classification via Hierarchical Aggregation". In Proceedings of the European Conference on Computer Vision 2022. [arXiv] [Chinese intro] [github]

Min Zhang, Siteng Huang, Donglin Wang, "Domain Generalized Few-shot Image Classification via Meta Regularization Network". In Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing. [pdf] [github]

Zifeng Zhuang, Xintao Xiang, Siteng Huang, Donglin Wang, "HINFShot: A Challenge Dataset for Few-Shot Node Classification in Heterogeneous Information Network". In Proceedings of the 2021 ACM International Conference on Multimedia Retrieval. [pdf]

Zhengyu Chen, Jixie Ge, Heshen Zhan, Siteng Huang, Donglin Wang, "Pareto Self-Supervised Training for Few-Shot Learning". In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. [arXiv] [open access]

Siteng Huang, Min Zhang, Yachen Kang, Donglin Wang, "Attributes-Guided and Pure-Visual Attention Alignment for Few-Shot Recognition". In Proceedings of the 35th AAAI Conference on Artificial Intelligence. [project page] [arXiv] [code] [poster] [slide]

Siteng Huang, Donglin Wang, Xuehan Wu, Ao Tang, "DSANet: Dual Self-Attention Network for Multivariate Time Series Forecasting". In Proceedings of the 28th ACM International Conference on Information and Knowledge Management. [project page] [pdf] [code] [poster] [slide]

Peer-reviewed Journal

Xuyang Liu†, Ting Liu†, Siteng Huang✉, Yi Xin, Yue Hu, Long Qin, Donglin Wang, Honggang Chen✉, "M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension". IEEE Transactions on Circuits and Systems for Video Technology, 2025. [arXiv] [github]

Preprints & Under Submission

Fengyuan Dai†, Zifeng Zhuang†, Yufei Huang, Siteng Huang, Bangyan Liao, Donglin Wang, Fajie Yuan, "VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RL". arXiv preprint arXiv:2505.15791. [pdf] [huggingface paper]

Yang Liu, Ming Ma, Xiaomin Yu, Pengxiang Ding, Han Zhao, Mingyang Sun, Siteng Huang, Donglin Wang, "SSR: Enhancing Depth Perception in Vision-Language Models via Rationale-Guided Spatial Reasoning". arXiv preprint arXiv:2505.12448. [pdf] [project page] [github] [huggingface paper]

Can Cui, Pengxiang Ding, Wenxuan Song, Shuanghao Bai, Xinyang Tong, Zirui Ge, Runze Suo, Wanqi Zhou, Yang Liu, Bofang Jia, Han Zhao, Siteng Huang, Donglin Wang, "OpenHelix: A Short Survey, Empirical Analysis, and Open-Source Dual-System VLA Model for Robotic Manipulation". arXiv preprint arXiv:2505.03912. [pdf] [project page] [github] [Awesome List] [huggingface paper]

Xiaomin Yu, Pengxiang Ding, Wenjie Zhang, Siteng Huang, Songyang Gao, Chengwei Qin, Kejian Wu, Zhaoxin Fan, Ziyue Qiao, Donglin Wang, "Unicorn: Text-Only Data Synthesis for Vision Language Model Training". arXiv preprint arXiv:2503.22655. [pdf] [github]

Minghui Lin, Xiang Wang, Yishan Wang, Shu Wang, Fengqi Dai, Pengxiang Ding, Cunxiang Wang, Zhengrong Zuo, Nong Sang, Siteng Huang✉, and Donglin Wang✉, "Exploring the Evolution of Physics Cognition in Video Generation: A Survey". arXiv preprint arXiv:2503.21765. [pdf] [github] [huggingface paper]

Pengxiang Ding, Jianfei Ma, Xinyang Tong, Binghong Zou, Xinxin Luo, Yiguo Fan, Ting Wang, Hongchao Lu, Panzhong Mo, Jinxin Liu, Yuefan Wang, Huaicheng Zhou, Wenshuo Feng, Jiacheng Liu, Siteng Huang, Donglin Wang, "Humanoid-VLA: Towards Universal Humanoid Control with Visual Integration". arXiv preprint arXiv:2502.14795. [pdf]

Xuyang Liu, Ziming Wang, Yuhang Han, Yingyao Wang, Jiale Yuan, Jun Song, Bo Zheng, Linfeng Zhang, Siteng Huang, Honggang Chen, "Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Models". arXiv preprint arXiv:2501.05179. [pdf] [github]

Bofang Jia, Pengxiang Ding, Can Cui, Mingyang Sun, Pengfang Qian, Siteng Huang, Zhaoxin Fan, Donglin Wang, "Score and Distribution Matching Policy: Advanced Accelerated Visuomotor Policies via Matched Distillation". arXiv preprint arXiv:2412.09265. [pdf] [project page]

Zhefei Gong, Pengxiang Ding, Shangke Lyu, Siteng Huang, Mingyang Sun, Wei Zhao, Zhaoxin Fan, Donglin Wang, "CARP: Visuomotor Policy Learning via Coarse-to-Fine Autoregressive Prediction". arXiv preprint arXiv:2412.06782. [pdf] [project page] [huggingface paper]

Yuhang Han†, Xuyang Liu†, Zihan Zhang, Pengxiang Ding, Donglin Wang, Honggang Chen, Qingsen Yan, Siteng Huang✉, "Filter, Correlate, Compress: Training-Free Token Reduction for MLLM Acceleration". arXiv preprint arXiv:2411.17686. [pdf] [project page] [huggingface paper] [github]

Fengyuan Dai, Siteng Huang, Min Zhang, Biao Gong, Donglin Wang, "Focus-Consistent Multi-Level Aggregation for Compositional Zero-Shot Learning". arXiv preprint arXiv:2408.17083. [pdf]

Ting Liu†, Xuyang Liu†, Siteng Huang, Liangtao Shi, Zunnan Xu, Yi Xin, Quanjun Yin, Xiaohong Liu, "Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inference". arXiv preprint arXiv:2405.14700. [pdf] [github]

Thesis

Siteng Huang, "Model Transfer for Multimodal Understanding and Generation". Zhejiang University, 2024.

💻 Internship Experience

💼 Services

Conference Reviewer

  • Annual Conference on Neural Information Processing Systems (NeurIPS)
  • IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • IEEE/CVF International Conference on Computer Vision (ICCV)
  • European Conference on Computer Vision (ECCV)
  • AAAI Conference on Artificial Intelligence (AAAI)
  • International Joint Conference on Artificial Intelligence (IJCAI)
  • ACM International Conference on Multimedia (ACMMM)
  • Conference on Robot Learning (CoRL)
  • IEEE International Conference on Multimedia and Expo (ICME)
  • ACM International Conference on Multimedia Retrieval (ICMR)
  • Asian Conference on Computer Vision (ACCV)
  • International Conference on Pattern Recognition (ICPR)

Journal Reviewer

  • IEEE Transactions on Neural Networks and Learning Systems (TNNLS)
  • IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)
  • ACM Transactions on Intelligent Systems and Technology (ACM TIST)
  • Journal of Visual Communication and Image Representation (JVCI)
  • Concurrency and Computation: Practice and Experience (CPE)

Program Committee for Conferences and Workshops

  • Session Chair, The First Westlake Robot Learning Symposium

😉 Misc

Welcome to follow my Zhihu and XiaoHongShu.