Si2 Announces Creation of the Si2 LLM Benchmarking Coalition

AUSTIN, Texas--(BUSINESS WIRE)--The Silicon Integration Initiative (Si2) announced today the creation of the LLM benchmarking coalition (LBC) to expedite the development of high-quality large language models (LLMs) for semiconductor design problems. Si2 also announced that NVIDIA is joining as a founding member of the new coalition and that the coalition will build on the company’s open-source Comprehensive Verilog Design Problems (CVDP) benchmark that provides an LLM evaluation framework for LLMs used for Register-transfer level (RTL) design and verification problems.

The Si2 Large Language Model Benchmarking Coalition is one of four Coalitions supported by Si2, a not-for-profit member organization whose members innovate on trusted standards and share solutions that lower development costs and increase design productivity.

The Si2 LLM Benchmarking Coalition’s corporate and academic members will build upon the RTL design and verification benchmarks provided in CVDP by extending problems to cover new categories and design domains, monitoring a leaderboard of results, refining benchmarking methods and metrics, and providing an interpretation of the results.

Leigh Anne Clevenger, Si2 vice president of technology, said, “The motivation for LBC came from the 2024 IEEE International Workshop on LLM-Aided Design (LAD’24) which highlighted a lack of benchmarks to evaluate the quality-of-results for Verilog for hardware design. Many current Si2 AI/ML in EDA SIG member companies and universities plan to participate, with experts already providing important input for enhancing and expanding the benchmarks in CVDP to make it an industry and academia-wide effort.”

Robert Aslett, Si2 CEO, stated, “We are excited about the very positive reaction received during initial discussions with representatives from several semiconductor companies and academic institutions. We are now in the process of signing up members of the coalition.”

Learn more about Si2 and becoming a member of the LLM Benchmarking Coalition by visiting Si2 or Si2 contact.