mlcommons / ck-mlops

A collection of portable workflows, automation recipes and components for MLOps in a unified CK format. Note that this repository is outdated - please check the 2nd generation of the CK workflow automation meta-framework with portable MLOps and DevOps components here:

Note that this repository is outdated: we are now using the next generation of the MLCommons CK workflow automation meta-framework (Collective Mind aka CM) developed by the open working group. Feel free to join this community effort to learn how to modularize ML Systems and automate their benchmarking, optimization and deployment in the real world!

CK repository with automation workflows for MLPerf and MLOps

CK compatibility

This repository is compatible with the MLCommons CK framework v2.5.8 (Apache 2.0 license):

Documentation

• CK-powered MLPerf™ benchmark automation and design space exploration
• CK-powered MLPerf™ inference submission automation

Overview

This repository contains a collection of stable CK components (automation recipes and workflows) to automate benchmarking, optimization and deployment of ML Systems across diverse platforms, environments, frameworks, models and data sets:

• CK automation recipes for MLOps: [inside CK framework] [in this repo]
• CK portable program workflows: [list]
• CK portable meta packages: [list]
• CK environment detection (software, models, data sets): [list]
• CK OS descriptions: [list]
• CK adaptive containers: [list]

Current projects

• Developing a platform to automate SW/HW co-design for ML Systems across diverse models, data sets, frameworks and platforms based on user constraints in terms of speed, accuracy, energy and costs: OctoML.ai & cKnowledge.io
• Automating MLPerf(tm) inference benchmark and packing ML models, data sets and frameworks as CK components with a unified API and meta description
• Providing a common format to share artifacts at ML, systems and other conferences: video, Artifact Evaluation
• Redesigning CK together with the community based on user feedback

Use cases

• Real-world use cases from our partners: overview

Motivation

• ACM TechTalk on YouTube
• White paper and extended journal article

Coordinator

• Grigori Fursin (OctoML/cTuning foundation)

Problems

Don't hesitate to report issues or submit feature requests here.

Public discussions

Contact Grigori Fursin to join our MLCommons Design Space Exploration Workgroup (subgroup of Best Practices)!

ck-mlops's Issues

loadgen install failed on fedora 34

I followed this page: https://colab.research.google.com/drive/1ij1rWoqje5-Sn6UsdFj1OzYakudI2RIS?usp=sharing#scrollTo=lt7aU3scRLZ4
Please help and thanks!

##################################################################################
steps:

Pull CK MLOps repo

ck pull repo:mlcommons@ck-mlops

Initialize platform

ck setup kernel --var.install_to_env=yes
ck detect platform.os --platform_init_uoa=generic-linux-dummy

Add Python, GCC, CMake to CK

ck detect soft:compiler.python --full_path={sys.executable}
ck detect soft:compiler.gcc --full_path=which gcc
ck detect soft:tool.cmake --full_path=which cmake

Install Python dependencies

python3 -m pip install protobuf

Download mlcommons/inference repo

ck install package --quiet --tags=mlperf,inference,src,r1.1

Install dependencies

ck install package --tags=lib,python-package,absl
ck install package --tags=lib,python-package,numpy
ck install package --tags=lib,python-package,scipy
ck install package --tags=lib,python-package,matplotlib
ck install package --tags=lib,python-package,cython
ck install package --tags=lib,python-package,opencv-python-headless
ck install package --tags=lib,python-package,mlperf,loadgen
##################################################################################

(mlperf-inference) bash-5.1$ ck install package --tags=lib,python-package,mlperf,loadgen

Package found: lib-python-mlperf-loadgen (6b744d888c45962b)

---

Resolving software dependencies ...

*** Dependency 1 = mlperf-inference (MLPerf Inference source):

Resolved. CK environment UID = 7801b88d489cdd48 (version r1.1-c5b8918)

*** Dependency 2 = compiler (C++ compiler):

Resolved. CK environment UID = 8667f4ffd932b2f9 (detected version 11.2.1)

*** Dependency 3 = python (Python interpreter):

Resolved. CK environment UID = a84239c2f9b63d72 (detected version 3.9.12)

*** Dependency 4 = lib-python-absl (Python Abseil library):

Resolved. CK environment UID = abf3a6e97cff1960 (version unversioned)

*** Dependency 5 = lib-python-numpy (Python NumPy library):

Resolved. CK environment UID = 7458c1283d2f59e8 (version 1.23.0)

---

Searching if CK environment for this package already exists using:

• Tags: compiled-by-gcc,compiled-by-gcc-11.2.1,needs-python,needs-python-3.9.12,lib,python-package,mlperf,loadgen,mlperf_loadgen,mlperf-loadgen,vmaster,host-os-linux-64,target-os-linux-64,64bits
• Dependency: mlperf-inference=7801b88d489cdd48
• Dependency: compiler=8667f4ffd932b2f9
• Dependency: python=a84239c2f9b63d72
• Dependency: lib-python-absl=abf3a6e97cff1960
• Dependency: lib-python-numpy=7458c1283d2f59e8

CK environment found for this package: 01ce5c2d548e0975

OVERWRITING AND PROCESSING AGAIN!

---

Resolving software dependencies ...

*** Dependency 1 = mlperf-inference (MLPerf Inference source):

Resolved. CK environment UID = 7801b88d489cdd48 (version r1.1-c5b8918)

*** Dependency 2 = compiler (C++ compiler):

Resolved. CK environment UID = 8667f4ffd932b2f9 (detected version 11.2.1)

*** Dependency 3 = python (Python interpreter):

Resolved. CK environment UID = a84239c2f9b63d72 (detected version 3.9.12)

*** Dependency 4 = lib-python-absl (Python Abseil library):

Resolved. CK environment UID = abf3a6e97cff1960 (version unversioned)

*** Dependency 5 = lib-python-numpy (Python NumPy library):

Resolved. CK environment UID = 7458c1283d2f59e8 (version 1.23.0)

---

Installing to /home/wxsun/CK/local/venv/mlperf-inference/CK/local/env/01ce5c2d548e0975

---

Cleanup: removing /home/wxsun/CK/local/venv/mlperf-inference/CK/local/env/01ce5c2d548e0975/python_deps_site
Building 'mlperf_loadgen' from source ...
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help

error: invalid command 'bdist_wheel'
Installing 'mlperf_loadgen' and its dependencies to '/home/wxsun/CK/local/venv/mlperf-inference/CK/local/env/01ce5c2d548e0975/python_deps_site/lib64/python3.9/site-packages' or '/home/wxsun/CK/local/venv/mlperf-inference/CK/local/env/01ce5c2d548e0975/python_deps_site/lib/python3.9/site-packages' ...
ls: cannot access 'dist/mlperf_loadgen*.whl': No such file or directory
ERROR: You must give at least one requirement to install (see "pip help install")
find: ‘/home/wxsun/CK/local/venv/mlperf-inference/CK/local/env/01ce5c2d548e0975/python_deps_site’: No such file or directory
touch: cannot touch '/init.py': Permission denied
Error: installation failed!

CK error: [package] package installation failed!

ck install package --tags=lib,armnn,tflite,neon,opencl,rel.20.08

More than one package or version found:

0. lib-armnn (f517b24b03b2b61f), Variations: rel.20.08,tflite,neon,opencl
1. lib-armnn (f517b24b03b2b61f), Variations: rel.20.08,tflite,neon,opencl
2. lib-armnn (f517b24b03b2b61f), Variations: rel.20.08,tflite,neon,opencl

Please select the package to install [ hit return for "0" ]: 0

Package found: lib-armnn (f517b24b03b2b61f)

---

Resolving software dependencies ...

*** Dependency 1 = lib-boost (Boost library):

---

WARNING: no registered CK environment was found for "Boost library" dependency with tags="lib,boost,for-armnn" and setup={"host_os_uoa": "linux-64", "target_os_uoa": "linux-64", "target_os_bits": "64"} and version constraints ([1, 64, 0] <= v <= [1, 66, 0])

Searching and installing CK software packages ...
* tags: lib,boost,for-armnn
* or tags:
* no tags:

More than one package or version found:

0. lib-boost-1.66.0 Version 1.66.0 (0417ba98aa5ca4ad), Variations: for-armnn
1. lib-boost-1.66.0 Version 1.66.0 (0417ba98aa5ca4ad), Variations: for-armnn
2. lib-boost-1.66.0 Version 1.66.0 (0417ba98aa5ca4ad), Variations: for-armnn
3. lib-boost-1.65.1 Version 1.65.1 (8bb8a5b19b48120c), Variations: for-armnn
4. lib-boost-1.65.1 Version 1.65.1 (8bb8a5b19b48120c), Variations: for-armnn
5. lib-boost-1.65.1 Version 1.65.1 (8bb8a5b19b48120c), Variations: for-armnn
6. lib-boost-1.64.0 Version 1.64.0 (c6a6991d5c951d20), Variations: for-armnn
7. lib-boost-1.64.0 Version 1.64.0 (c6a6991d5c951d20), Variations: for-armnn
8. lib-boost-1.64.0 Version 1.64.0 (c6a6991d5c951d20), Variations: for-armnn

Please select the package to install [ hit return for "0" ]: 0

Package found: lib-boost-1.66.0 (0417ba98aa5ca4ad)

---

Resolving software dependencies ...

*** Dependency 1 = cmake (cmake):

Resolved. CK environment UID = b269ae509c47a2a7 (detected version 3.14.3)

*** Dependency 2 = compiler (C++ compiler):

More than one environment found for "C++ compiler" with tags="compiler,lang-cpp" and setup={"host_os_uoa": "linux-64", "target_os_uoa": "linux-64", "target_os_bits": "64"}:

0. GNU C compiler - v9.3.0 (64bits,compiler,gcc,host-os-linux-64,lang-c,lang-cpp,target-os-linux-64,v9,v9.3,v9.3.0 (093326cb3ccb990f))

1. LLVM compiler - v10.0.0-4ubuntu1 (64bits,compiler,host-os-linux-64,lang-c,lang-cpp,llvm,target-os-linux-64,v10,v10.0,v10.0.0,v10.0.0.0 (0688b7d5dc0fa909))

Select one of the options for "C++ compiler" with tags="compiler,lang-cpp" and setup={"host_os_uoa": "linux-64", "target_os_uoa": "linux-64", "target_os_bits": "64"} [ hit return for "0" ]: 0

Resolved. CK environment UID = 093326cb3ccb990f (detected version 9.3.0)

---

Searching if CK environment for this package already exists using:

• Tags: compiled-by-gcc,compiled-by-gcc-9.3.0,lib,boost,v1.66.0,v1.66,v1,channel-stable,for-armnn,host-os-linux-64,target-os-linux-64,64bits
• Dependency: cmake=b269ae509c47a2a7
• Dependency: compiler=093326cb3ccb990f

CK environment not found for this package ...

*** Installation path used: /root/CK-TOOLS/lib-boost-1.66.0-gcc-9.3.0-for-armnn-linux-64

---

Resolving software dependencies ...

*** Dependency 1 = cmake (cmake):

Resolved. CK environment UID = b269ae509c47a2a7 (detected version 3.14.3)

*** Dependency 2 = compiler (C++ compiler):

Resolved. CK environment UID = 093326cb3ccb990f (detected version 9.3.0)

---

Installing to /root/CK-TOOLS/lib-boost-1.66.0-gcc-9.3.0-for-armnn-linux-64

Using scripts from /root/CK/mlcommons@ck-mlops/package/lib-boost-1.62.0

Downloading package from 'https://dl.bintray.com/boostorg/release/1.66.0/source/boost_1_66_0.zip' ...
--2021-12-08 11:19:17-- https://dl.bintray.com/boostorg/release/1.66.0/source/boost_1_66_0.zip
Resolving dl.bintray.com (dl.bintray.com)... failed: Temporary failure in name resolution.
wget: unable to resolve host address ‘dl.bintray.com’
Error: downloading package failed!

CK detected a PROBLEM in the third-party CK package:

CK package:           lib-boost-1.66.0
Native package URL:   https://dl.bintray.com/boostorg/release/1.66.0/source
CK repo:              mlcommons@ck-mlops

CK repo URL:          https://github.com/mlcommons/ck-mlops
CK package URL:       https://github.com/mlcommons/ck-mlops/tree/master/package/lib-boost-1.66.0

CodeReef stable package URL: https://cKnowledge.io/c/package/lib-boost-1.66.0

Please, submit the full log to the authors of this external CK package
at https://github.com/mlcommons/ck-mlops/issues to collaboratively fix this problem!

CK error: [package] package installation failed!

Unable to install a version of boost library

Unifying CK components for ML Systems

We started working with the community and MLCommons to test and unify CK components for ML Systems from this repository: https://github.com/ctuning/ck/blob/master/docs/mlperf-automation/components/README.md

We want to unify CK packages, workflows, adaptive Docker containers, scripts and other relevant components:

• https://github.com/ctuning/ai/tree/main/package
• https://github.com/ctuning/ai/tree/main/program
• https://github.com/ctuning/ai/tree/main/docker
• https://github.com/ctuning/ai/tree/main/jnotebook
• https://github.com/ctuning/ai/tree/main/script

The goal is to develop portable workflows for the MLPerf benchmark, automate software/hardware co-design of efficient ML Systems across diverse models, frameworks, datasets, compilers, run-time systems and platforms and help end-users select the most efficient ML/SW/HW stacks with the best performance/accuracy/energy/size/cost trade off on the Pareto frontier and based on their requirements and constraints.

This project is supported by OctoML.ai and MLCommons.

Add OneDNN package

CC @apeskov

Loadgen installation failure on MacBook Pro

ck install package --tags=lib,python-package,mlperf,loadgen

is failing on MacBook Pro with Apple M1 chip.

running install_scripts
creating build/bdist.macosx-10.14-arm64/wheel/mlperf_loadgen-1.1.dist-info/WHEEL
creating 'dist/mlperf_loadgen-1.1-cp38-cp38-macosx_10_14_arm64.whl' and adding 'build/bdist.macosx-10.14-arm64/wheel' to it
adding 'mlperf_loadgen.cpython-38-darwin.so'
adding 'mlperf_loadgen-1.1.dist-info/METADATA'
adding 'mlperf_loadgen-1.1.dist-info/WHEEL'
adding 'mlperf_loadgen-1.1.dist-info/top_level.txt'
adding 'mlperf_loadgen-1.1.dist-info/RECORD'
removing build/bdist.macosx-10.14-arm64/wheel
Installing 'mlperf_loadgen' and its dependencies to '/Users/arjun/CK/local/env/363eebbe64f6aaa4/python_deps_site/lib64/python3.8/site-packages' or '/Users/arjun/CK/local/env/363eebbe64f6aaa4/python_deps_site/lib/python3.8/site-packages' ...
ERROR: mlperf_loadgen-1.1-cp38-cp38-macosx_10_14_arm64.whl is not a supported wheel on this platform.
find: /Users/arjun/CK/local/env/363eebbe64f6aaa4/python_deps_site: No such file or directory
touch: /__init__.py: Read-only file system
Error: installation failed!