Qi Zhu

At AWS, I design AI systems (GraphStorm) utilizing structured knowledge for applications in retrieval-augmented generation (RAG), graph machine learning, and beyond. As a founding member of the GraphRAG Team, I help drive the launch of structure-aware features across various RAG services.

My current and past work focuses on the following themes:

1. LLMs with Structured Knowledge – Harnessing explicit and implicit data structures to enhance the long-context performance and parameter efficiency of large language models.
2. Graph Representation Learning – Representing objects in heterogenous text-attributed graph with heterogenous learning, and robust to distribution shift.

We aim to make LLMs more efficient and resilient against hallucinations by harnessing structured knowledge. A key challenge lies in making the language model structure-aware while mitigating performance bottlenecks, such as the lost-in-the-middle phenomenon, To address this, we explore post-training, fine-tuning, and pre-training techniques on graph structured data.

• Graph Retrieval Augmented Generation: We develop structure-aware algorithms for pre-retrieval, retrieval, and inference stages of RAG.
  • AGENT-G: An Agentic Framework for Graph Retrieval Augmented Generation, Under Review

• Supervised Fine-tuning LLMs on graphs: We propose parameter-efficient fine-tuning of billion-scale GNN-LLM architecture to align the latent space between structure and text. The goal is to better adapt LLMs on graph representation learning with small of computation resources.
  • Can GNN be Good Adapter for LLMs? , WWW'24
  • Parameter-Efficient Tuning Large Language Models for Graph Representation Learning, preprint

• Pre-training Cascading GNN-LM: Numerous real-world application can be modeled as a text-attributed graph such as citation network and social network, where nodes or edges contains useful text information. We pre-train million-scale language model with GNN layers interleaved in transformers.
  • Patton: Language Model Pre-training on Text-Rich Networks, ACL'23
  • Heterformer: Transformer-based deep node representation learning on heterogeneous text-rich networks, KDD'23

My research aims to make graph representation learning adapt to distribution shift and data heterogeneity.

• GNN Out-of-distribution Generalization: Graph neural networks are notoriously poor at generalizing to out-of-distribution target data. We conduct theoretical analysis of their generalization properties and introduce unsupervised loss between training and target distributions to enhance robustness and performance on unseen data.
  • Fine-Tuning Graph Neural Networks by Preserving Graph Generative Patterns, AAAI'24
  • May the Force be with You: Unified Force-Centric Pre-Training for 3D Molecular Conformations, NeurIPS'23
  • Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training Data, NeurIPS'21
  • Transfer Learning of Graph Neural Networks with Ego-graph Information Maximization, NeurIPS'21

• Heterogenous Graph Representation Learning:
  • Easing embedding learning by comprehensive transcription of heterogeneous information networks, KDD'18
  • Collective multi-type entity alignment between knowledge graphs, WWW'20
  • Unsupervised differentiable multi-aspect network embedding, KDD'20

• 2020 Amazon AWS Machine Learning Research Award
• 2018 ACM WWW Best Poster Honorable Mention