Specialization with Benchmarks for Emerging Applications

October 2, 2022, Co-located with MICRO 2022


We have entered the era of application-driven architecture specialization, where significant power/energy/performance benefits are obtained by leveraging specialization of compute units, memory and interconnect - broadly referred to as accelerator fabrics. The majority of recent publications in architecture conferences addresses various aspects of the design, evaluation and deployment of these fabrics. However, the shift in research from general purpose computing to accelerator fabrics has not yet coalesced the architecture community around specific sets of benchmarks for key emerging application domains. There is some maturation of benchmarks in a few specific domains such as machine learning, but many others such as security/privacy, recommendation systems, graph-based and web-based applications, etc., are still lacking a commonly accepted set of benchmarks, thus slowing research progress.

The workshop will bring together researchers who have embarked upon new benchmarking efforts over the past several years. The goal of the workshop is to allow benchmarking researchers to share lessons learned and best practices for identifying, developing, and gaining adoption of benchmarks, and the broader community to learn about recent offerings that support architecture research. Ultimately, we seek to promulgate best practices in engaging with the prospective user communities for new benchmarks.

The program consists of presentations on individual benchmark suites selected among the submitted proposals followed by an opening panel discussing the topics above.


Workshop Program

Download Program PDF

(All times are CDT)

8:00-8:10am Opening remarks
8:10-8:30am ML Metrics: The Past, Present, and Future of Benchmarking ML Systems, Datasets, and Use Cases
Vijay Janapa Reddi, Associate Professor in SEAS (Harvard University)
8:30-8:50am   Genomics Bench: A benchmark suite of modern genomics workloads
Reetu Das, Associate Professor of Electrical Engineering and Computer Science (University of Michigan)
8:50-9:10am DeepRecSys: A System for Optimizing End-To-End At-scale Neural   
Recommendation Inference

Udit Gupta, Ph.D. Student (Harvard University)
9:10-9:30am VIP-Bench: A Benchmark Suite for Evaluating Privacy-Enhanced Computation Frameworks
Todd Austin, Professor of Electrical Engineering and Computer Science (University of Michigan) 
9:30-9:50am Datacenter Applications: Past, Present, Future
Baris Kasikci, Assistant Professor of Electrical Engineering and Computer Science and ADA PI (University of Michigan)
9:50-10:10am DyGraph: A Dynamic Graph Generator and Benchmark Suite
Andrew McCrabb, Ph.D. Student (University of Michigan)
10:10-10:30am Benchmarking HPC systems for Mixed-Analytic Workloads
Antonino Tumeo, HPC Team B Computer Scientist (Pacific Northwest National Laboratory)
10:30-11am Break

Benchmarks Panel
Moderator: Sharad Malik, George Van Ness Lothrop Professor of Engineering (Princeton University)
Panelists: Todd Austin (University of Michigan), Reetu Das (University of Michigan), Udit Gupta (Harvard University), Baris Kasikci (University of Michigan), Andrew McCrabb (University of Michigan), Antonino Tumeo (Pacific Northwest National Laboratory), Vijay Janapa Reddi (Harvard University)


Valeria Bertacco is a Mary Lou Dorf Professor of Computer Science and Engineering at the University of Michigan. Her research interests are in the area of computer systems design. Throughout her career she has contributed novel solutions in design validation and reliability, hardware-security assurance, and the design of specialized architectures for graph algorithms and machine learning. Prof. Bertacco is the Director of the Applications Driving Architectures (ADA) Research Center, which investigates computing architectures and design flows for the 2040s decade by exploring specialized heterogeneity, domain-specific languages, and new silicon devices. Prof. Bertacco is a Fellow of the IEEE. She currently serves as the Vice Provost for Engaged Learning at the University of Michigan, supporting domestic and international exchanges and partnerships for the institution.

David Brooks is the Haley Family Professor of Computer Science in the School of Engineering and Applied Sciences at Harvard University. During the 2021-22 academic year he is on sabbatical as a visiting research scientist at Meta AI Research (formerly, Facebook AI Research). Prof. Brooks' research interests include resilient and power-efficient computer hardware and software design for high-performance and embedded systems. Prof. Brooks is a Fellow of the IEEE and ACM, and he has received several honors and awards including the ACM Maurice Wilkes Award, ISCA Influential Paper Award, and two HPCA Test of Time Awards.

Todd Austin is the Jack Hu Professor of Electrical Engineering and Computer Science at the University of Michigan in Ann Arbor. His research interests include computer architecture, robust and secure system design, hardware and software verification, and performance analysis tools and techniques. From 2012-2017, Todd was the director of C-FAR, the Center for Future Architectures Research, a multi-university SRC/DARPA funded center that was seeking technologies to scale the performance and efficiency of future computing systems. Prior to joining academia, Todd was a Senior Computer Architect in Intel's Microcomputer Research Labs, a product-oriented research laboratory in Hillsboro, Oregon. In addition to his work in academia, Todd is founder of SimpleScalar LLC, and co-founder of Agita Labs Inc. and InTempo Design LLC. In 2002, Todd was a Sloan Research Fellow, and in 2007 he received the ACM Maurice Wilkes Award for "innovative contributions in Computer Architecture including the SimpleScalar Toolkit and the DIVA and Razor architectures." Todd is an IEEE Fellow, and he received his PhD in Computer Science from the University of Wisconsin in 1996.

Sharad Malik is the George Van Ness Lothrop Professor of Engineering at Princeton University. He has served as the Director of the multi-university MARCO Gigascale Systems Research Center (GSRC, 2009-2012), and as the Associate Director of the Center for Future Architectures Research (C-FAR, 2013-2016). His current research focuses on design methodology for formal functional and security verification of hardware and hardware-software systems. His research in functional timing analysis and propositional satisfiability has been widely used in industrial electronic design automation tools. He has received the IEEE/ACM Design Automation Conference (DAC) Award for the most cited paper in the 50-year history of the conference (2013), the Computer-Aided Verification (CAV) Award for fundamental contributions to the development of high-performance Boolean satisfiability solvers (2009), the IEEE CEDA A. Richard Newton Technical Impact Award in Electronic Design Automation (2017), the Princeton University President’s Award for Distinguished Teaching (2009), as well as several other research and teaching awards. He has also received the UC Berkeley Electrical Engineering and Computer Science Distinguished Alumni Award (2019) and the IIT Delhi Distinguished Alumni Award (2009). He is a fellow of the IEEE and ACM.