.

.

.

Benchmark Suites

.

.

.

 

ADA-Contributed Benchmark Suites

 
Benchmark Suite Scope URL Projects that Adopted the Suite
BayesSuite Bayesian Inference benchmark suite. https://github.com/Emma926/BayesSuite  
DeepRecSys A System for Optimizing End-To-End At-scale Neural Recommendation Inference https://github.com/harvard-acc/DeepRecSys  
DyGraph Dynamic Graph Datasets https://adacenter.org/dygraph OMEGA/ MessageFusion/ Centaur/ DREDGE
FPBench Benchmarks, compilers, and standards for the floating-point research community. https://github.com/FPBench  
Frontsuite Suite of data center applications used for i-spy and ripple evaluation https://github.com/efeslab/frontsuite Intel, Arm, Qualcomm, University of Wisconsin, Texas A&M University, UPC Barcelona, EPFL, and the University of Edinburgh
GBBS Graph-based algorithms: connectivity, covering, eigenvector, substructure, and shortest path https://github.com/ldhulipala/gbbs OMEGA/ MessageFusion/ Centaur/ DREDGE

PI Shun's group is using the parallel primitives and graph representations in GBBS to implement new parallel static and dynamic graph algorithms.

Google is adopting GBBS for their in-memory graph computations.
GenomicsBench Benchmark suite of recent genomics applications https://genomicsbench.eecs.umich.edu/ PI Batten's group is using Genomics Bench for their Ephemeral Vector Engine (EVE) project and big.VLITTLE project

PI Das's group is using GenomicBench for evaluating accelerator architectures in this domain.
ILLIXR Illinois Extended Reality testbed: a fully open-source end-to-end Virtual and Augmented Reality system and research testbed (including standalone constituent components). https://illixr.github.io/ PIs Adve, Brooks and Wei are using ILLIXR for their work on accelerator design space exploration.

PIs Adve and Gavrilovska are using ILLIXR for research on computation off-loading and multiparty AR.
Mobile Edge Computing (MEC) A suite of application benchmarks and software stack components for a mobile edge computing testbed and different application use cases. https://github.com/GTkernel/edge-benchmark-suite   
MilSpec Suite of end-to-end open-source benchmarks relevant in national defense applications, employing key applications in the embedded systems space. https://gitlab.eecs.umich.edu/wpehrett/milspec  
MLPerf Machine-learning applications benchmark suite https://github.com/mlcommons/ PI Kasikci's group is using MLPerf for profile-driven optimizations for ML systems.
ParaDNN Parametric neural network benchmarks
https://github.com/Emma926/paradnn  
Parapoly A Massively Parallel Polymorphic Benchmark Suite https://github.com/purdue-aalp/Parapoly  
RecPipe HW/SW  models for multi-stage recommendation systems’ performance https://github.com/harvard-acc/RecPipe harvard-acc/RecPipe
TinyMLPerf ML benchmarks for embedded devices https://github.com/mlcommons/tiny ML Systems
VIP Bench Privacy-enhanced computation benchmarks https://bitbucket.org/vip-benchmarks/vip-bench/ PI Austin’s group is using VIP Bench for Austin's startup, Agita Labs

PI Reagan's group is using VIP Bench for accelerator evaluation research.

PI Batten's group is to using VIP Bench and GenomicBench to explore hardware acceleration opportunities.
       
       
       

JUMP-Contributed Benchmark Suites

 
SAGA bench Streaming graph benchmarks https://github.com/abasak24/SAGA-Bench  

Community-Contributed Benchmark Suites

 
ANMLZoo Benchmark repository for automata-based applications and automata engines for both von-Neumann and reconfigurable dataflow architectures. https://github.com/jackwadden/ANMLZoo  
CEP - Lincoln Lab Publicly available SoC including several hardware blocks https://www.ll.mit.edu/r-d/projects/common-evaluation-platform  
CloudSuite Benchmark suite for cloud services. https://github.com/parsa-epfl/cloudsuite  
CORAL-2 Benchmarks Kernels inspired by natural science applications and core CS algorithms, including data science and ML https://asc.llnl.gov/coral-2-benchmarks/  
DIMACS implementation challenge - graph datasets Road networks http://users.diag.uniroma1.it/challenge9/download.shtml  
GAP Benchmark Suite Static graph processing suite https://github.com/sbeamer/gapbs  
HPC Challenge Benchmark Suite for range memory access patterns. http://icl.cs.utk.edu/hpcc/  
MachSuite Suite for accelerator design and customized architectures. https://github.com/breagen/MachSuite  
MAVBench Simulator and benchmark suite for micro aerial vehicle design https://github.com/harvard-edge/MAVBench  
MEVBench Mobile computer vision benchmarking suite. http://vhosts.eecs.umich.edu/mevbench//  
MiBench and MiBench2 Commercially representative embedded benchmark suite. MiBench2 is ported for IoT devices. http://vhosts.eecs.umich.edu/mibench//  
MicroSuite Benchmark suite for microservices. https://github.com/wenischlab/MicroSuite  
NAS Parallel Small set of programs designed to help evaluate the performance of parallel supercomputers. https://www.nas.nasa.gov/publications/npb.html  
PARSEC Benchmark suite composed of multithreaded programs, targeting emerging workloads for shared-memory programs for chip-multiprocessors. https://parsec.cs.princeton.edu/index.htm  
Rodinia Kernel suite of compute-intensive applications https://rodinia.cs.virginia.edu/doku.php  
SNAP - Static Graph Datasets Large graph datasets http://snap.stanford.edu/data/index.html  
STINGER Dynamic graph algorithms http://www.stingergraph.com/  
OpenNMT Attention-based speech-to-text and machine translation https://opennmt.net  
ERASER Interpretable NLP models http://www.eraserbenchmark.com  
Transformer Transformer-based NLP models https://github.com/huggingface/transformer  
X