I am an Assistant Professor in the Department of Performing Arts Technology at the University of Michigan. I am also affiliated with Computer Science and Engineering and Electrical and Computer Engineering.

My research centers around Human-Centered AI for Creativity. I develop human-centered AI methodology that can be integrated into creative and scholarly workflows. My work spans multimodal generative models for music, audio, and video creation, self-supervised audiovisual learning paradigms, and interactive systems for music education and digital humanities. My long-term goal is to build collaborative AI partners that preserve human agency and democratize both creative expression and scholarly discovery.

My research is driven by two fundamental questions: (1) How can we develop multimodal AI frameworks that accurately capture nuanced human intent to enable seamless, collaborative human-AI communication? and (2) How can generative AI systems integrate deeply into professional workflows while preserving human agency and creativity?

• Controllable generative AI for music and audio: multitrack music generation (AAAI 2018, ISMIR 2018, ISMIR 2020, ICASSP 2023, ISMIR 2024), text-to-music generation (ISMIR 2025), expressive violin performance synthesis (ICASSP 2022, ICASSP 2025), music instrumentation (ISMIR 2021), and symbolic music processing tools (ISMIR LBD 2019, ISMIR 2020, AI4Music 2025)
• Multimodal retrieval-embedded generation for video editing: long-to-short video editing (ICLR 2025, NeurIPS 2025), video-to-music generation (ISMIR 2025), text-queried sound separation (ICLR 2023), and text-to-audio synthesis (WASPAA 2023)
• Collaborative AI systems for human creative and scholarly workflows: interactive music generation (info), a cappella coaching interface (UIST Posters 2025, AI4Music 2025), multi-pitch vocal transcription (arXiv 2026), and optical music recognition and score-audio alignment for guqin music (info)

---

 Recent Talks

---

 Currently Teaching

• Music and AI (PAT 463/563)
• Creative Coding (PAT 204/504)

---

 News

• 🤝 Our AI-assisted guqin studies project has been awarded the Humanities and AI Virtual Institute (HAVI) grant from Schmidt Sciences (learn more)!
• 🏅 Our a cappella research team won the Empowering Research with AI Award! Congrats to Ting-Yu Pan, Phyllis Ju, Yuna Lee, and Christian Li!
• 🏅 Selected for the AAAI New Faculty Highlights at AAAI 2026
• 📜 Paper on long-to-short video editing accepted to NeurIPS 2025 (demo)
• 💡 Organized the AI for Music Workshop at NeurIPS 2025
• 📜 Paper on text-to-symbolic music generation accepted to ISMIR 2025 (demo)
• 📜 Paper on video-guided text-to-music generation accepted to ISMIR 2025 (demo)
• 📜 Paper on documentary teaser generation accepted to ICLR 2025 (demo)

Older news

• 📜 Paper on expressive violin synthesis (demo) accepted to ICASSP 2025
• 📜 Paper on music LLMs (demo) accepted to ICASSP 2025
• 🏅 Awarded the Doctoral Award for Excellence in Research by UCSD CSE Department
• 🚀 I will be joining University of Michigan School of Music, Theatre & Dance as an Assistant Professor in the Department of Performing Arts Technology in Fall 2024!

---

 Team

Please read this if you’re interested in working with me.

Graduate Students

• Ming Xu, MS in Applied Statistics, University of Michigan
• Junyoung Koh, Visiting PhD Student, Yonsei University
• Junnuo Wang, Recent MS Graduate, New York University
• Sangheon Park, Recent MS Graduate, Georgia Institute of Technology

Alumni

• Ting-Yu Pan, MS in Electrical & Computer Engineering, UMich ➜ PhD at UIUC
• Kexin (Phyllis) Ju, MS in Information, UMich ➜ PhD at BU
• Weihan Xu, MS in Computer Science, Duke ➜ PhD at UW
• Erfun Ackley, MA in Performing Arts Technology, UMich

---

 Funding

My research is partly sponsored by:

Schmidt Sciences

NVIDIA

U-M Arts Initiative

U-M OVPR

---

 Selected Honors & Awards

• AAAI New Faculty Highlights AAAI

  2026
• Doctoral Award for Excellence in Research UCSD CSE Department

  2024
• Rising Stars in AI KAUST AI Initiative

  2024
• Rising Stars in Data Science UChicago DSI and UCSD HDSI

  2023
• IEEE SPS Scholarship IEEE Signal Processing Society

  2023
• Friends of International Center Fellowship UCSD GEPA

  2023
• Rising Stars in Signal Processing ICASSP

  2023
• Interdisciplinary Research Award UCSD GPSA

  2023
• Government Scholarship to Study Abroad Taiwan Ministry of Education

  2022
• J. Yang Scholarship J. Yang & Family Foundation

  2021
• ECE Department Fellowship UCSD ECE Department

  2019

---

 Selected Publications

See the full list of publications here (Google Scholar).

• REGen: Multimodal Retrieval-Embedded Generation for Long-to-Short Video Editing
  Weihan Xu, Yimeng Ma, Jingyue Huang, Yang Li, Wenye Ma, Taylor Berg-Kirkpatrick, Julian McAuley, Paul Pu Liang, and Hao-Wen Dong
  Advances in Neural Information Processing Systems (NeurIPS), 2025
  paper  demo  poster  code  reviews

• Generating Symbolic Music from Natural Language Prompts using an LLM-Enhanced Dataset
  Weihan Xu, Julian McAuley, Taylor Berg-Kirkpatrick, Shlomo Dubnov, and Hao-Wen Dong
  International Society for Music Information Retrieval Conference (ISMIR), 2025
  paper  demo  poster  code  reviews

• Video-Guided Text-to-Music Generation Using Public Domain Movie Collections
  Haven Kim, Zachary Novack, Weihan Xu, Julian McAuley, and Hao-Wen Dong
  International Society for Music Information Retrieval Conference (ISMIR), 2025
  paper  demo  poster  code  reviews

• TeaserGen: Generating Teasers for Long Documentaries
  Weihan Xu, Paul Pu Liang, Haven Kim, Julian McAuley, Taylor Berg-Kirkpatrick, and Hao-Wen Dong
  International Conference on Learning Representations (ICLR), 2025
  paper  demo  poster  code  reviews

• CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Models
  Hao-Wen Dong, Xiaoyu Liu, Jordi Pons, Gautam Bhattacharya, Santiago Pascual, Joan Serrà, Taylor Berg-Kirkpatrick, and Julian McAuley
  IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2023
  paper  demo  video  slides  reviews

• Multitrack Music Transformer
  Hao-Wen Dong, Ke Chen, Shlomo Dubnov, Julian McAuley, and Taylor Berg-Kirkpatrick
  IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023
  paper  demo  video  slides  code  reviews

• CLIPSep: Learning Text-queried Sound Separation with Noisy Unlabeled Videos
  Hao-Wen Dong, Naoya Takahashi, Yuki Mitsufuji, Julian McAuley, and Taylor Berg-Kirkpatrick
  International Conference on Learning Representations (ICLR), 2023
  paper  demo  video  slides  poster  code  reviews

• Deep Performer: Score-to-Audio Music Performance Synthesis
  Hao-Wen Dong, Cong Zhou, Taylor Berg-Kirkpatrick, and Julian McAuley
  IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022
  paper  demo  video  slides  poster  reviews

• Towards Automatic Instrumentation by Learning to Separate Parts in Symbolic Multitrack Music
  Hao-Wen Dong, Chris Donahue, Taylor Berg-Kirkpatrick, and Julian McAuley
  International Society for Music Information Retrieval Conference (ISMIR), 2021
  paper  demo  video  slides  code  reviews

• MusPy: A Toolkit for Symbolic Music Generation
  Hao-Wen Dong, Ke Chen, Julian McAuley, and Taylor Berg-Kirkpatrick
  International Society for Music Information Retrieval Conference (ISMIR), 2020
  paper  video  slides  poster  code  documentation  reviews

• Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation
  Hao-Wen Dong and Yi-Hsuan Yang
  International Society for Music Information Retrieval Conference (ISMIR), 2018
  paper  demo  video  slides  poster  code  reviews

• MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment
  Hao-Wen Dong,* Wen-Yi Hsiao,* Li-Chia Yang, and Yi-Hsuan Yang (*equal contribution)
  AAAI Conference on Artificial Intelligence (AAAI), 2018
  paper  demo  slides  code

---

 Education

• University of California San Diego
  Ph.D. in Computer Science
  Advisors: Julian McAuley and Taylor Berg-Kirkpatrick
  Dissertation: Generative AI for Music and Audio

  Sep 2019 – Jun 2024

  
• University of California San Diego
  M.S. in Computer Science

  Sep 2019 – Jun 2021

  
• National Taiwan Normal University
  Digital Video and Audio Arts Program

  Sep 2019 – Jun 2017

  
• National Taiwan University
  B.S. in Electrical Engineering

  Sep 2013 – Jun 2017

  

---

 Professional Experience

• NVIDIA
  Research Intern
  Deep Imagination Research Group, NVIDIA Research
  Advisors: Siddharth Gururani and Ming-Yu Liu
  Topic: Controllable audio generation

  Sep 2023 – Dec 2023

  
• Adobe
  Research Scientist/Engineer Intern
  Audio Research Group, Adobe Research
  Advisors: Justin Salamon and Oriol Nieto
  Topic: Text-to-audio retrieval

  May 2023 – Sep 2023

  
• Dolby
  Speech/Audio Deep Learning Intern
  Applied AI Team, Advanced Technology
  Advisor: Xiaoyu Liu
  Topic: Text-to-audio synthesis

  Jan 2023 – Apr 2023

  
• Amazon
  Applied Scientist Intern
  Natural Understanding Team, Alexa AI
  Advisors: Wenbo Zhao and Gunnar Sigurdsson
  Topic: Text-to-audio synthesis

  Sep 2022 – Jan 2023

  
• Sony
  Student Intern
  Tokyo Laboratory 30, R&D Center
  Advisor: Naoya Takahashi
  Topic: Universal sound separation

  May 2022 – Sep 2022

  
• Dolby
  Deep Learning Audio Intern
  Applied AI Team, Advanced Technology Group
  Advisor: Cong Zhou
  Topic: Music performance synthesis

  Jun 2021 – Sep 2021

  
• Yamaha
  Research Intern
  AI Group, Research and Development Division
  Advisor: Keijiro Saino
  Topic: Deep learning based synthesizer

  May 2019 – Aug 2019

  
• Academia Sinica
  Research Assistant
  Music and AI Lab, Research Center for IT Innovation
  Advisor: Yi-Hsuan Yang
  Topics: Music generation and deep generative models

  Jul 2017 – Apr 2019

  

---

Researchers create knowledge.
Teachers organize knowledge.
Engineers apply knowledge.
Artists challenge knowledge.

---

For more information, please see my CV.