Large language models (LLMs) like Claude Sonnet 3.7, GPT-4, Llama 3.1, and Gemini 2.5 have shown promise in AI-assisted coding, transforming natural language descriptions into code. While LLMs perform well on simple benchmarks like LeetCode, MBPP, and HumanEval, their success rate remains low in real-world software and hardware development. Complex dependencies, mixed languages, and hardware/software constraints present significant challenges. To bridge this gap, we are exploring LLM multi-agent frameworks to assist the development, validation, and optimization of software and hardware programs. This talk will present our recent progress on two sets of problems. First, I will introduce an LLM agent framework for software issue localization, called OrcaLoca. It addresses the challenge of precisely identifying software problems within large codebases by integrating priority-based action scheduling, action decomposition with relevance scoring, and distance-aware context pruning. Through these mechanisms, OrcaLoca improves the accuracy and efficiency of navigating and understanding complex software repositories to locate bugs. Second, I will introduce our LLM agentic workflow for Verilog RTL design: VeriAssist enables the LLM to self-correct and self-verify the generated RTL code by adopting an automatic prompting system and integrating RTL simulator in the code generation loop; MAGE is a multi-agent system designed for robust and accurate Verilog RTL code generation. I will also discuss insights gained through our studies about LLM's ability in addressing complex real-world programming problems.
Jishen Zhao is a Professor in the Computer Science and Engineering Department at University of California, San Diego. Her research is at the boundary of computer systems and machine learning, particularly on memory systems, machine learning and systems co-design, and reliability. Before joining academia, she was a research scientist at HP Labs. She is a recipient of NSF CAREER award, AWS AI Amazon Research Award (ARA), many best paper awards, and a member of MICRO Hall of Fame.