|
Benchmarks
Tools
Order Benchmarks
SPEC
Mirror Sites
Resources
|
SPEC's Benchmarks
Cloud
- SPEC Cloud_IaaS 2016
[benchmark info] [published
results] [order benchmark]
SPEC's first benchmark suite to measure cloud performance SPEC Cloud_IaaS 2016's
use is targeted at cloud providers, cloud consumers, hardware vendors, virtualization
software vendors, application software vendors, and academic researchers. The benchmark
addresses the performance of infrastructure-as-a-service (IaaS) public or private cloud
platforms. The benchmark is designed to stress provisioning as well as runtime aspects
of a cloud using I/O and CPU intensive cloud computing workloads. SPEC selected the
social media NoSQL database transaction and K-Means clustering using map/reduce as two
significant and representative workload types within cloud computing.
CPU
- SPEC CPU2017
[benchmark info] [published
results] [support] [order
benchmark]
Designed to provide performance measurements that can be used to
compare compute-intensive workloads on different computer systems,
SPEC CPU2017 contains 43 benchmarks organized into four suites:
SPECspeed 2017 Integer, SPECspeed 2017 Floating Point, SPECrate 2017
Integer, and SPECrate 2017 Floating Point.
- SPEC CPU2006
[benchmark info] [published
results] [support] [order
benchmark]
Designed to provide performance measurements that can be used to
compare compute-intensive workloads on different computer systems,
SPEC CPU2006 contains two benchmark suites: CINT2006 for measuring
and comparing compute-intensive integer performance, and CFP2006
for measuring and comparing compute-intensive floating point performance.
- SPEC CPU2000
[Retired]
- SPEC CPU95
[Retired]
- SPEC CPU92
[Retired]
Graphics and Workstation
Performance
Handheld
- Handheld
[info]
In 2013 SPEC formed a committee chartered for the development of, and support for, a compute intensive benchmark suite for handheld devices. In 2017, recognizing the reduced interest in this work, this committee was dissolved.
High Performance Computing,
OpenMP, MPI, OpenACC, OpenCL
- SPEC ACCEL
[benchmark info] [published
results] [support] [order
benchmark]
SPEC ACCEL tests performance with a suite of computationally intensive
parallel applications running under the OpenCL 1.1, OpenACC 1.0, and
OpenMP 4.5 APIs. The suite exercises the performance of the accelerator,
host CPU, memory transfer between host and accelerator, support libraries
and drivers, and compilers.
- SPEC MPI2007
[benchmark info] [published
results] [support] [order
benchmark]
MPI2007 is SPEC's benchmark suite for evaluating MPI-parallel, floating
point, compute intensive performance across a wide range of cluster
and SMP hardware. The suite consists of the intial MPIM2007 suite
and MPIL2007, which contains larger working sets and longer run
times than MPIM2007. All benchmarks in the suite are developed in
compliance with MPI 2.1 standard.
- SPEC OMP2012
[benchmark info] [published
results] [support] [order
benchmark]
The successor to the OMP2001, designed for measuring performance
using applications based on the OpenMP 3.1 standard for shared-memory
parallel processing. OMP2012 also includes an optional metric for
measuring energy consumption.
- SPEC OMP2001
[Retired]
- SPEC HPC2002
[Retired]
- SPEC HPC96
[Retired]
Java Client/Server
- SPECjAppServer2004
[Retired]
- SPECjAppServer2002
[Retired]
- SPECjAppServer2001
[Retired]
- SPECjbb2015
[benchmark info] [published
results] [support]
[order benchmark]
The SPECjbb2015 benchmark has been developed from the ground up to measure
performance based on the latest Java application features. It is relevant to all audiences
who are interested in Java server performance, including JVM vendors, hardware developers,
Java application developers, researchers and members of the academic community.
- SPECjbb2013
[Retired]
- SPECjbb2005
[Retired]
- SPECjbb2000
[Retired]
- SPECjEnterprise2010
[benchmark info] [published
results] [support]
[order benchmark]
SPECjEnterprise2010 measures full-system performance for Java Enterprise
Edition (Java EE) 5 or later application servers, databases and
supporting infrastructure and expands the scope of the SPECjAppServer2004 benchmark.
- SPECjms2007
[benchmark info] [published
results] [support] [order
benchmark]
SPECjms2007 is the first industry-standard benchmark for evaluating
the performance of enterprise message-oriented middleware servers
based on JMS (Java Message Service). It provides a standard workload
and performance metrics for competitive product comparisons, as
well as a framework for indepth performance analysis of enterprise
messaging platforms.
- SPECjvm2008
[benchmark info] [published
results] [support]
[download benchmark]
SPECjvm2008 is a benchmark suite for measuring the performance of
a Java Runtime Environment (JRE), containing several real life applications
and benchmarks focusing on core java functionality. The SPECjvm2008
workload mimics a variety of common general purpose application
computations.
- SPEC JVM98
[Retired]
Mail Servers
Storage
Power
- SPECpower_ssj2008
[benchmark info] [published
results] [support]
[order benchmark]
SPECpower_ssj2008 is the first industry-standard SPEC benchmark
that evaluates the power and performance characteristics of volume
server class computers. The initial benchmark addresses the performance
of server-side Java, and additional workloads are planned.
- Other SPEC benchmarks incorporating power measurement
Virtualization
- SPEC virt_sc 2013
[benchmark info] [published
results] [support] [order
benchmark]
SPEC's updated benchmark addressing performance evaluation of datacenter
servers used in virtualized server consolidation. SPEC virt_sc 2013
measures the end-to-end performance of all system components including
the hardware, virtualization platform, and the virtualized guest
operating system and application software. The benchmark supports
hardware virtualization, operating system virtualization, and hardware
partitioning schemes.
- SPEC virt_sc 2010
[Retired]
Web Servers
SPEC Tools
- Server Efficiency Rating Tool (SERT) 2.0
[benchmark info] [support]
[order software]
The Server Efficiency Rating Tool (SERT) was created by Standard
Performance Evaluation Corporation (SPEC) at the request of the
US Environmental Protection Agency. The SERT 2.0 adds a single-value
metric, reduces runtime, improves automation and testing, and broadens
device and platform support. Designed to be simple to configure and use
via a comprehensive graphical user interface, the SERT suite uses a set
of synthetic worklets to test discrete system components such as processors,
memory and storage, providing detailed power consumption data at different
load levels.
The SERT metric, created with the support of the RG Power Working Group,
rates the server efficiency of single- and multi-node servers across a broad
span of configurations.
- Server Efficiency Rating Tool (SERT) 1.1.1
[benchmark info] [support]
[order software]
The Server Efficiency Rating Tool (SERT) was created by Standard Performance
Evaluation Corporation (SPEC) at the request of the US Environmental Protection
Agency. The SERT 1.1.1 is the most current SERT version supported by the U.S. EPA
Energy Star v2.0 program. Designed to be simple to configure and use via a comprehensive
graphical user interface, the SERT suite uses a set of synthetic worklets to test
discrete system components such as processors, memory and storage, providing detailed
power consumption data at different load levels.
- Chauffeur Worklet Development Kit (WDK)
[kit info] [forum]
[order software]
Chauffeur was designed to simplify the development of workloads
for measuring both performance and energy efficiency. Because Chauffeur
contains functions that are common to most workloads, developers
of new workloads can focus on the actual business logic of the application,
and take advantage of Chauffeur's capabilities for configuration,
run-time, data collection, validation, and reporting.
Chauffeur was initially designed to meet the requirements of the
SERT. However, SPEC recognized that the framework would also be
useful for research and development purposes. The Chauffeur framework
is now being made available as the Chauffeur Worklet Development
Kit (WDK). This kit can be used to develop new workloads (or "worklets"
in Chauffeur terminology). Researchers can also use the WDK to configure
worklets to run in different ways, in order to mimic the behavior
of different types of applications. These features can be used in
the development and assessment of new technologies such as power
management capabilities.
Version 2.0 is based on the SERT 2.0 infrastructure
and includes significant enhancements to the hardware detection,
customization options of generating HTML reports, and developer
documentation. It has now reduced memory requirements for the Director
when signing results files and reduced the size of the result output
for large systems or clusters.
Chauffeur WDK 2.0 added worklet-specific normalization of results
and an updated list of supported operating systems including Ubuntu
(14.04 LTS and 16.04 LTS) as well as current versions of Windows Server,
RHEL, SLES, AIX, and Solaris. The WDK also includes the latest PTDaemon
integration for power analyzers and temperature sensors, along with data
collection, validation and reporting.
- PTDaemon
[info]
The power temperature daemon (also known as PTDaemon) is used to offload
the work of controlling a power analyzer or temperature sensor during
measurement intervals to a system other than the SUT. It hides the details
of different power analyzer interface protocols and behaviors from the
benchmark software, presenting a common TCP/IP-based interface that can
be readily integrated into different benchmark harnesses. Benchmarks already
using PTDaemon include SPECpower_ssj2008,
SPEC CPU2017, and SPEC virt_sc 2013,
the SERT suite, and the Chauffeur WDK.
|
