Qi Zhu

My current and past work focuses on the following themes:

1. Data-Driven LLM Optimization – Mining massive LLM invocations to guide strategic decisions in LLM deployment—determining when to leverage multimodal representations for long-context processing, and how to efficiently select training data based on real-world usage patterns, etc.
2. LLMs with Structured Knowledge – Harnessing explicit and implicit data structures to to enhance LLM reasoning capabilities and mitigate hallucinations
3. Graph Representation Learning – Representing objects in heterogenous text-attributed graph with heterogenous learning, and robust to distribution shift.

We aim to make LLMs more efficient and resilient against hallucinations by harnessing structured knowledge. A key challenge lies in making the language model structure-aware while mitigating performance bottlenecks, such as the lost-in-the-middle phenomenon, To address this, we explore post-training, fine-tuning, and pre-training techniques on graph structured data. [Structured Knowledge for LLMs, KDD Workshop]

• Graph Retrieval Augmented Generation: We propose HYBGRAG, an agentic system for hybrid question answering over semi-structured knowledge bases. Unlike prior RAG systems that handle only textual or relational information, HYBGRAG synergizes both through a retriever bank with adaptive module selection and a critic module that provides corrective feedback for iterative refinement. This structure-aware approach addresses the lost-in-the-middle phenomenon by precisely routing questions to appropriate retrieval modules and self-correcting extraction errors.
  • Hybgrag: Hybrid retrieval-augmented generation on textual and relational knowledge bases , ACL 25

• Supervised Fine-tuning LLMs on graphs: We propose parameter-efficient fine-tuning of billion-scale GNN-LLM architecture to align the latent space between structure and text. The goal is to better adapt LLMs on graph representation learning with small of computation resources.
  • Can GNN be Good Adapter for LLMs? , WWW'24
  • Parameter-Efficient Tuning Large Language Models for Graph Representation Learning, preprint

• Pre-training Cascading GNN-LM: Numerous real-world application can be modeled as a text-attributed graph such as citation network and social network, where nodes or edges contains useful text information. We pre-train million-scale language model with GNN layers interleaved in transformers.
  • Patton: Language Model Pre-training on Text-Rich Networks, ACL'23
  • Heterformer: Transformer-based deep node representation learning on heterogeneous text-rich networks, KDD'23

My research aims to make graph representation learning adapt to distribution shift and data heterogeneity.

• GNN Out-of-distribution Generalization: Graph neural networks are notoriously poor at generalizing to out-of-distribution target data. We conduct theoretical analysis of their generalization properties and introduce unsupervised loss between training and target distributions to enhance robustness and performance on unseen data.
  • Fine-Tuning Graph Neural Networks by Preserving Graph Generative Patterns, AAAI'24
  • May the Force be with You: Unified Force-Centric Pre-Training for 3D Molecular Conformations, NeurIPS'23
  • Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training Data, NeurIPS'21
  • Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization, NeurIPS'21

• Heterogenous Graph Representation Learning:
  • Easing embedding learning by comprehensive transcription of heterogeneous information networks, KDD'18
  • Collective multi-type entity alignment between knowledge graphs, WWW'20
  • Unsupervised differentiable multi-aspect network embedding, KDD'20

• 2020 Amazon AWS Machine Learning Research Award
• 2018 ACM WWW Best Poster Honorable Mention