intel / compute-benchmarks

Compute Benchmarks for oneAPI Level Zero and OpenCL™ Driver

Compute Benchmarks

A benchmark suite developed and maintained by NEO driver development team in order to provide a stable environment, which allows checking and reproducing performance for various parts of the driver. The benchmark suite is divided into multiple executables, each of which tests a different aspect of the driver. Each executable contained in ComputeBechmarks consists of multiple test cases with a set of predefined configurations (parameters like buffer sizes, workgroup sizes, etc.) along with a possibility to run them with any desired parameters. They support both OpenCL and Level Zero APIs, although some tests may be available in only one API.

Current list of tests can be found in TESTS.md.

How to run benchmarks

For specific information about how to run the benchmarks, please run the binary with "--help" parameter.

FAQ

Answers to frequently asked questions and common problems can be found in FAQ.md.

Building

Run CMake and compile

Build the benchmark as any other CMake project. Example command sequence:

git clone --recurse-submodules https://github.com/intel/compute-benchmarks
cd compute-benchmarks
mkdir build
cd build
cmake ..
cmake --build . --config Release

The last line can be substituted with platform-specific command, for example:

make -j`nproc`

or:

make memory_benchmark_ocl -j`nproc`

Building with SYCL support

SYCL implementations of benchmarks will be built if the following conditions are met:

• oneAPI is installed with support for DPC++;
• the user has sourced the setvars.sh script prior to running CMake;
• the CMake option BUILD_SYCL is set to ON;

Note that building SYCL benchmarks is currently only supported on Linux.

Example

git clone --recurse-submodules https://github.com/intel/compute-benchmarks
cd compute-benchmarks
mkdir build
cd build
. /opt/intel/oneapi/setvars.sh
cmake .. -DBUILD_SYCL=ON
cmake --build . --config Release

Building with MPI support

Hybrid MPI + X implementations of the benchmarks will be built if:

• the CMake option BUILD_MPI is set to ON;
• CMake is able to locate the MPI compilers/headers/libraries through its FindMPI module;

For Intel MPI, setting I_MPI_OFFLOAD=1 or I_MPI_OFFLOAD=2 is required; for other MPI implementations please refer to their manuals.

The MPI benchmarks are only supported on Linux.

Binary types

Each benchmark suite can be built as a single-api binary or as a an all-api binary.

• Single-api binaries are named like ulls_benchmark_ocl and do not load libraries from not used APIs. They are built by default and can be disabled by passing -DBUILD_SINGLE_API_BINARIES=OFF to CMake.
• All-api binaries are named like ulls_benchamrk and contain every API supported by the benchmark, which makes them more convenient, but at a risk of introducing some overhead caused by loading unnecessary library. They are not built by default and can be enabled by passing -DBUILD_ALL_API_BINARIES=ON to CMake.

By default all benchmarks are compiled in all configurations they support. This can be changed with CMake arguments. For example, to disable all-api binaries, run:

cmake .. -DBUILD_ALL_API_BINARIES=OFF -DBUILD_SINGLE_API_BINARIES=ON

SDK

ComputeBenchmarks will try to find SDKs for the APIs used. In case of inability to find those, it will use libraries contained in third_party/opencl-sdk and third_party/level-zero-sdk directories. The libraries were compiled on Ubuntu 20.04 LTS. Using a different setup may result in build failures due to ABI incompatibility, so it's safest to have the SDK installed in your system.

Contributing

Information on how to contribute to ComputeBenchmarks can be found in CONTRIBUTING.md

compute-benchmarks's Issues

error: explicit comparison with infinity in fast floating point mode [-Werror,-Wtautological-constant-compare]

When building the repo with icpx the test_case/test_case_statistics.cpp compile fail with

 error: explicit comparison with infinity in fast floating point mode [-Werror,-Wtautological-constant-compare]

Possible solution:

• Disable tautological-constant-compare
• Set -00 flags globally
• Disable fast-math for the particulars files

‘zeCommandListHostSynchronize’ was not declared in this scope

Following the instruction and running the cmake command shows the error:

/path/to/compute-benchmarks/source/benchmarks/api_overhead_benchmark/implementations/l0/command_list_host_synchronize_l0.cpp:56:9: error: ‘zeCommandListHostSynchronize’ was not declared in this scope; did you mean ‘CommandListHostSynchronize’?
56 | zeCommandListHostSynchronize(cmdList, std::numeric_limits<uint64_t>::max());
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
| CommandListHostSynchronize

Timer Overflows in ulls l0 `copy_submission_events_l0`

Hi,

Thank you very much for this repo. I just tried the ulls tests and got a "strange" output:

CopySubmissionEvents(api=l0 engine=CCS0)  167089998.336  167088431.200        100.00%        -23.280  334183367.920  [GPU]           [us]

It's caused by the timer overflow here:

compute-benchmarks/source/benchmarks/ulls_benchmark/implementations/l0/copy_submission_events_l0.cpp

Line 89 in 4426ddc

const auto submissionTime = std::chrono::nanoseconds((deviceStartTimestamp.global.kernelStart - truncatedDeviceEnqueueTimestamp) * timerResolution);

Assuming only one overflow one can use the "adding max value" back trick. For example,

https://github.com/intel/pti-gpu/blob/fffbe6d142fda7c1776de46cfc8e3d01e5daa34d/tools/ze_tracer/ze_kernel_collector.h#L590

Hope this helps,
Thomas

Typo in message

FindSYCL.cmake has a typo
It should read
" set(SYCL_INCLUDE_DIR "${SYCL_INCLUDE_DIR}" CACHE PATH "SYCL Include Directory")"

compute-benchmarks/cmake/FindSYCL.cmake

Line 34 in b049326

set(SYCL_INCLUDE_DIR "${SYCL_INCLUDE_DIR}" CACHE PATH "SYCL Inlcude Directory")

scripts/dpcpp-proxy.sh

Everywhere I am familiar with, it is /usr/bin/env rather than /bin/env. After changing this I could get compute-benchmarks to compile

diff --git a/scripts/dpcpp-proxy.sh b/scripts/dpcpp-proxy.sh
index d0bc740..f179fc5 100755
--- a/scripts/dpcpp-proxy.sh
+++ b/scripts/dpcpp-proxy.sh
@@ -1,4 +1,4 @@
-#!/bin/env bash
+#!/usr/bin/env bash

Copyright (C) 2022 Intel Corporation

@@ -16,4 +16,4 @@ if [[ " $@ " =~ .\ -fsycl\ . ]]; then
dpcpp "${@:2}"
else
"$@"
-fi
\ No newline at end of file
+fi

‘ZE_COMMAND_QUEUE_FLAG_IN_ORDER’ was not declared in this scope

Hello, I am facing an issue during "cmake --build . --config Release" on Ubuntu 22.04. Am I missing dependencies or is it referencing a missing parameter?