Specifically regarding the test on function @test_doubly_nested_loop: should line 23 be as well preceded by a // CHECK-NOT: affine.for? IIUC, the point of the test is to make sure that nested instances of affine.for are as well transformed or it is not really necessary?

Hello,

Firstly thanks very much for the tutorial which is very informative for ones who has little knowledge on LLVM.

I'm following charters one by one, and in the section 3'writing first pass', I met with problem in the test using command 'bazel test tests:all" :
'
//tests:ctlz.mlir.test PASSED in 0.4s
//tests:ctlz_runner.mlir.test PASSED in 0.9s
//tests:ctlz_simple.mlir.test PASSED in 0.4s
//tests:affine_loop_unroll.mlir.test FAILED in 0.3s
/home/user/.cache/bazel/_bazel_user/0d9e230fafe092ac5e7f4f9c63722bf8/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/affine_loop_unroll.mlir.test/test.log
//tests:mul_to_add.mlir.test FAILED in 0.3s
/home/user/.cache/bazel/_bazel_user/0d9e230fafe092ac5e7f4f9c63722bf8/execroot/_main/bazel-out/k8-fastbuild/testlogs/tests/mul_to_add.mlir.test/test.log'

The issue is that '

.---command stderr------------

| 'tutorial-opt': command not found

`-----------------------------

error: command failed with exit status: 127

'
However, I could use 'bazel run tools:tutorial-opt -- -affine-full-unroll ${PWD}/tests/affine_loop_unroll.mlir' to run this test successfully.

I checked the directory tree for bazel-bin/tests/affine_loop_unroll.mlir.test.runfiles/, it doesn't have '/mlir_tutorial/' as suggested in lit.cfg.py. And here are the output when I enable commented code in lit.cfg.py. I'm on Ubuntu 20.04, can you please help me to check what could be wrong in my setup ？

Xiang

================================================================================
INFO: From Testing //tests:affine_loop_unroll.mlir.test:
==================== Test output for //tests:affine_loop_unroll.mlir.test:
/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.runfiles/_main
total 16
-rw-rw-r-- 1 user user 0 May 2 09:38 init.py
drwxr-xr-x 5 user user 4096 May 2 09:17 _main
lrwxrwxrwx 1 user user 100 May 2 09:38 _repo_mapping -> /tmp/bazel-execroot/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.repo_mapping
drwxr-xr-x 3 user user 4096 May 2 09:17 bazel_tools
drwxr-xr-x 4 user user 4096 May 2 09:17 llvm-project
TEST_TMPDIR=/tmp/bazel-working-directory/_main/_tmp/885145028d2433fde581bc0498024cb7
JAVA_RUNFILES=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.runfiles
USER=user
TEST_LOGSPLITTER_OUTPUT_FILE=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/testlogs/tests/affine_loop_unroll.mlir.test/test.raw_splitlogs/test.splitlogs
BAZEL_TEST=1
SHLVL=0
LD_LIBRARY_PATH=:/usr/local/cuda-11.6/lib64
OLDPWD=/tmp/bazel-working-directory/_main
TEST_PREMATURE_EXIT_FILE=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/testlogs/tests/affine_loop_unroll.mlir.test/test.exited_prematurely
TEST_SIZE=small
TEST_UNDECLARED_OUTPUTS_DIR=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/testlogs/tests/affine_loop_unroll.mlir.test/test.outputs
TEST_TIMEOUT=60
LC_CTYPE=C.UTF-8
PYTHONSAFEPATH=1
XML_OUTPUT_FILE=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/testlogs/tests/affine_loop_unroll.mlir.test/test.xml
TEST_UNDECLARED_OUTPUTS_ANNOTATIONS_DIR=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/testlogs/tests/affine_loop_unroll.mlir.test/test.outputs_manifest
TMPDIR=/tmp
TEST_UNUSED_RUNFILES_LOG_FILE=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/testlogs/tests/affine_loop_unroll.mlir.test/test.unused_runfiles_log
RUN_UNDER_RUNFILES=1
PYTHON_RUNFILES=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.runfiles
_=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.runfiles/_main/tests/affine_loop_unroll.mlir.test
EXPERIMENTAL_SPLIT_XML_GENERATION=1
BAZEL_CXXOPTS=-std=c++17
PATH=.:/home/user/anaconda3/bin:/home/user/anaconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda-11.6/bin:/home/user/tools/
RUNFILES_DIR=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.runfiles
TEST_INFRASTRUCTURE_FAILURE_FILE=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/testlogs/tests/affine_loop_unroll.mlir.test/test.infrastructure_failure
TEST_BINARY=tests/affine_loop_unroll.mlir.test
TEST_WORKSPACE=_main
TEST_SRCDIR=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.runfiles
TEST_TARGET=//tests:affine_loop_unroll.mlir.test
PWD=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.runfiles/_main
PYTHONPATH=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.runfiles:/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.runfiles/:/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.runfiles/_main:/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.runfiles/bazel_tools:/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/bin/tests/affine_loop_unroll.mlir.test.runfiles/llvm-project
TZ=UTC
GTEST_TMP_DIR=/tmp/bazel-working-directory/_main/_tmp/885145028d2433fde581bc0498024cb7
TEST_WARNINGS_OUTPUT_FILE=/tmp/bazel-working-directory/_main/bazel-out/k8-fastbuild/testlogs/tests/affine_loop_unroll.mlir.test/test.warnings
None
None
None
-- Testing: 1 tests, 1 workers --
FAIL: mlir_tutorial :: affine_loop_unroll.mlir (1 of 1)
******************** TEST 'mlir_tutorial :: affine_loop_unroll.mlir' FAILED ********************
Exit Code: 127

This only affects the CMake build after article 12 (A Global Optimization and Dataflow Analysis) and before #31

See the discussion in #30 for details and #30 (comment) for the solution and links to further discussion.

Hi! Unfortunately, I'm still on the first page of this great guide. I'm trying to install this build on common systems (Ubuntu 22.04, Debian 12, FreeBSD 13.2). My actions (Ubuntu 22.04):

# apt install bazel-bootstrap bazel-skylib
# git clone https://github.com/j2kun/mlir-tutorial.git
# cd mlir-tutorial
# bazel build @llvm-project//mlir:IR

After this I get the following errors:

root@ubuntu:~/mlir-tutorial# bazel build @llvm-project//mlir:IR
INFO: Repository rules_python instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule new_git_repository defined at:
  /root/.cache/bazel/_bazel_root/fac3d405565bb727227c10fb24b5c557/external/bazel_tools/tools/build_defs/repo/git.bzl:182:37: in <toplevel>
ERROR: An error occurred during the fetch of repository 'rules_python':
   Traceback (most recent call last):
        File "/root/.cache/bazel/_bazel_root/fac3d405565bb727227c10fb24b5c557/external/bazel_tools/tools/build_defs/repo/git.bzl", line 169, column 13, in _new_git_repository_implementation
                fail("Exactly one of build_file and build_file_content must be provided.")
Error in fail: Exactly one of build_file and build_file_content must be provided.
ERROR: no such package '@rules_python//python': Exactly one of build_file and build_file_content must be provided.
INFO: Elapsed time: 0.056s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (0 packages loaded)

I tried to compile this tutorial on Ubuntu 22.04, Debian 12 and FreeBSD 13.2, but no luck so far. What makes debugging Bazel even more difficult is that the first step (cloning the monorepository) is quite time-consuming. Given your excellent example code, I might even find it easier to write my own CMakeLists.txt files.

Thank you for prudently indicating the commit number from which the compiler infrastructure should be built. Could you provide the version of the operating system and package versions that you use to successfully build this tutorial, as well as the commit version of the mlir-tutorial repository itself?

See https://discourse.llvm.org/t/mlir-dead-code-analysis/67568/8

I tried out the data flow analysis framework, and also realized the DeadCodeAnalysis was required for pretty much every analysis I wanted to do. So I’m sympathetic to making this easier to get started with, but I’m also curious what making it a “built-in” feature would entail. Maybe rather than built-in in the sense of “always included in DataFlowSolver” is too strong, but we could provide some helper that includes some sensible defaults (e.g. DeadCodeAnalysis and SparseConstantPropagation), which users could extend.

Otherwise it won't properly handle control flow, which I didn't test enough in my tutorial, but ran into when I was working on this in HEIR.

Commit e1ffe38 pins to bazel version 6.4.0, but users who use bazel via bazelisk before that commit will automatically get bazel version 7 (as of this writing) which breaks the build.

Hi~, I tried to build this tutorial as below commit version and also build corresponding correct external mlir project
I am encountering linking errors when building a shared library (lib/Dialect/Poly/libMLIRPoly.dylib). The error occurs during the linking phase and appears to be related to undefined symbols in the MLIR framework.
For example:

Undefined symbols:
mlir::Diagnostic::operator<<(llvm::Twine&&), referenced from: ...
mlir::ValueRange::dereference_iterator(llvm::PointerUnion<mlir::Value const*, mlir::OpOperand*, mlir::detail::OpResultImpl*> const&, long), referenced from: 

First off, thank you very much for this tutorial! I've been meaning to pick up both bazel and MLIR, so this is extremely helpful.

One thing I stumbled over in getting set up: apparently, bazel automatically detects XCode and picks a different toolchain, which led to it ignoring the -std=c++17 setting. Exporting BAZEL_USE_CPP_ONLY_TOOLCHAIN=1 in my shell ensured that the setting went through. I thought I'd share to save others the trouble I went through.

Thanks again!

Hi, Thank you so much for the tutorial series. It is really great and helped me a lot to understand a few core concepts of MLIR framework.

I was wondering if you can also include a tutorial where you show the way to build the passes as standalone shared library and show the process to call load the dialect and the pass library from command line using mlir-opt. I think this would be a great addition to the already great tutorial series.

Thanks in advance.

I just wanted to give a small guide for people looking to build this tutorial using CMake:

Prerequisites

• Make sure you have installed everything needed to build LLVM https://llvm.org/docs/GettingStarted.html#software
• For this recipe Ninja is used so be sure to have it as well installed https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages

Checking out the code

Checkout the tutorial including the LLVM dependency (submodules)

git clone --recurse-submodules https://github.com/j2kun/mlir-tutorial.git
cd mlir-tutorial

Building dependencies

Note: The following steps are suitable for macOs and use ninja as building system, they should not be hard to adapt for your environment.

Build LLVM/MLIR

#!/bin/sh

BUILD_SYSTEM=Ninja
BUILD_TAG=ninja
THIRDPARTY_LLVM_DIR=$PWD/externals/llvm-project
BUILD_DIR=$THIRDPARTY_LLVM_DIR/build
INSTALL_DIR=$THIRDPARTY_LLVM_DIR/install

mkdir -p $BUILD_DIR
mkdir -p $INSTALL_DIR

pushd $BUILD_DIR

cmake ../llvm -G $BUILD_SYSTEM \
      -DCMAKE_CXX_COMPILER="$(xcrun --find clang++)" \
      -DCMAKE_C_COMPILER="$(xcrun --find clang)" \
      -DCMAKE_INSTALL_PREFIX=$INSTALL_DIR \
      -DLLVM_LOCAL_RPATH=$INSTALL_DIR/lib \
      -DLLVM_PARALLEL_COMPILE_JOBS=7 \
      -DLLVM_PARALLEL_LINK_JOBS=1 \
      -DLLVM_BUILD_EXAMPLES=OFF \
      -DLLVM_INSTALL_UTILS=ON \
      -DCMAKE_OSX_ARCHITECTURES="$(uname -m)" \
      -DCMAKE_BUILD_TYPE=Release \
      -DLLVM_ENABLE_ASSERTIONS=ON \
      -DLLVM_CCACHE_BUILD=ON \
      -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
      -DLLVM_ENABLE_PROJECTS='mlir' \
      -DDEFAULT_SYSROOT="$(xcrun --show-sdk-path)" \
      -DCMAKE_OSX_SYSROOT="$(xcrun --show-sdk-path)"

cmake --build . --target check-mlir

popd

Building the tutorial and running tests

#!/bin/sh

BUILD_SYSTEM="Ninja"
BUILD_DIR=./build-`echo ${BUILD_SYSTEM}| tr '[:upper:]' '[:lower:]'`

rm -rf $BUILD_DIR
mkdir $BUILD_DIR
pushd $BUILD_DIR

LLVM_BUILD_DIR=externals/llvm-project/build
cmake -G $BUILD_SYSTEM .. \
    -DLLVM_DIR="$LLVM_BUILD_DIR/lib/cmake/llvm" \
    -DMLIR_DIR="$LLVM_BUILD_DIR/lib/cmake/mlir" \
    -DCMAKE_BUILD_TYPE=Debug

popd

cmake --build $BUILD_DIR --target MLIRAffineFullUnrollPasses
cmake --build $BUILD_DIR --target MLIRMulToAddPasses
cmake --build $BUILD_DIR --target mlir-headers
cmake --build $BUILD_DIR --target mlir-doc
cmake --build $BUILD_DIR --target tutorial-opt
cmake --build $BUILD_DIR --target check-mlir-tutorial

who uses C++ under 20 (17 or 13) might get error on build with cmake.

As ortools(stable or maybe main) uses C++20 (see here) should install C++20 or use ex-version of ortools.
in my case, ortools with v9.0 worked

message(STATUS "Fetching or-tools...")
include(FetchContent)
FetchContent_Declare(
  or-tools
  GIT_REPOSITORY https://github.com/google/or-tools.git
  GIT_TAG        v9.0     <== change here main -> v9.0
  )
FetchContent_MakeAvailable(or-tools)
message(STATUS "Done fetching or-tools")

I hope this issue helps someone with same issue.

• Build system
• Running mlir-opt and testing
• Writing a first pass (trivial wrapper around fullUnroll, use rewrite pattern framework)
• Converting passes to tablegen
• Defining a new dialect (poly)
• Writing a dialect conversion pass to tensor+affine (and compiling down through LLVM to machine code?)
• Adding folders/canonicalizers
• Interfaces and Traits?

I am trying to bazel build @llvm-project//mlir/...:all but am getting errors with the minimum macos version.

external/llvm-project/llvm/utils/TableGen/DFAEmitter.cpp:367:18: error: 'get<unsigned int, llvm::Record *, unsigned int, std::string>' is unavailable: introduced in macOS 10.13 OS << std::get<unsigned>(SingleAction); ^

/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/v1/variant:1544:22: note: 'get<unsigned int, llvm::Record *, unsigned int, std::string>' has been explicitly marked unavailable here constexpr const _Tp& get(const variant<_Types...>& __v) {

I have added --macos_minimum_os=10.13 in .bazelrc but it seems to get ignored - building with verbose_failure flag shows '-mmacosx-version-min=10.11.

sw_vers ProductName: macOS ProductVersion: 13.2.1 BuildVersion: 22D68
clang --version Apple clang version 14.0.0 (clang-1400.0.29.202) Target: arm64-apple-darwin22.3.0 Thread model: posix InstalledDir: /Library/Developer/CommandLineTools/usr/bin

Any help is appreciated, thanks!

Hi again! I hope that my issue raising comes across as intending to help rather than complain; please let me know if I get too noisy/annoying!

In order to get everything to build on my M1 Mac, I needed to add "AArch64" to the _LLVM_TARGETS list in bazel/setup_llvm.bzl. Perhaps this step is obvious to those more in the know, but I wasn't sure of the exact string to put in to get things to build at first ("darwin-arm64"?).

Thanks again for this tutorial!

Eventually want to write this article, and seeing some interesting people talking about it in the MLIR chat room. Posting links here for future reference

https://discord.com/channels/636084430946959380/642426447167881246/1164751356679172137

If you're following along PR by PR and checking out individual commits to run examples, you may encounter python complaining it can't find the lit module.

In the fourth article in the series I realized bazel was using the system Python, and aac8490 migrates to a bazel-managed python runtime.

A simple fix is to run pip install lit in your system Python. Or else cherry pick aac84908f7b09ec1b14489bbc0837e697b191630 to your step in the article series.