openmathlib / blas-benchmarks Goto Github PK

ATM cron runs hourly to assess the CI jitter. Need to flip it to a more civilized pace.

I'm trying to recreate the workflow of OpenMathLib/OpenBLAS#4751 (comment) , only with a gh-pages branch instead of a separate publisher repository.
The key point is for a workflow to push benchmark results to a branch --- and this needs a personal access token.

@martin-frbg could you please create and add a PAT? In my fork, the following permissions worked: actions r/w; commit statuses r/w; contents r/w; workflows r/w

Currently we benchmark:

BLAS level 1

• ddot
• ?nrm2
• ?axpy

BLAS level 3

• ?gemm
• ?syrk

LAPACK:

• ?gesv (linalg.solve)
• ?gesdd (linalg.svd)
• ?syev (linalg.eigh)

@martin-frbg suggested it'd be useful to add

• ?gemv
• ?gbmv

for the banded matrix-vector multiply, what is more interesting to scale: matrix size or the number of bands?

• cdotc -- complex-valued BLAS level 1

Text output is currently hidden in the Actions tab --- click Actions, click on a benchmark run, expand the "Run benchmarks" part, scroll down to the output.

There is a time limit on how long the logs are stored by github, so do we want to save them somewhere?

Note that all benchmark results from all runs are stored in the results/machine-name/commit-hash.json files in the gh-pages branch, so they are not lost either way.

codspeed benchmarks (https://codspeed.io/OpenMathLib/OpenBLAS) run on each PR to the main OpenBLAS repository (https://github.com/OpenMathLib/OpenBLAS/)

The openblas_wrap f2py setup is very similar to the one used here: the main difference is that we either build openblas manually (for codspeed) or reuse the scipy_openblas32 nightly wheel (here). Would be nice to share the f2py wrappers and benchmarks themselves.

The blocker is that meson.build and pyproject.toml for openblas_wrap hardcode the dependency on scipy_openblas32. This also blocks #8.

ARM-specific lib has been suggested in OpenMathLib/OpenBLAS#4751 (comment)

Currently, we only run for an arbitrary pair of problem sizes. Running for all sizes from 100 to 1000 with a step of 1 is probably too much resource-wise, so some middle ground needs to be found.

graviton jobs run on the main OpenBLAS repo, but not on this one: https://github.com/OpenMathLib/BLAS-Benchmarks/actions/runs/9549517327/job/26319258806 is stuck on Waiting for a runner to pick up this job...

I asked Amit https://github.com/aktech of cirun.io, and his reply is

If you're adding a new runner in the same account ... install cirun for that repository and create a .cirun.yml in the repository to trigger runs.

So I copy-pasted the .cirun.yml from the OpenBLAS repository, and.. it's not enough, so I guess the I'd need to ask @martin-frbg to install cirun for the repository?

Since asv works off the main branch only, I'm committing and force-pushing to it, not in feature branches.

ATM, we only run d variants, so other kernels are not benchmarked.