Invalid NUMBA_NUM_THREADS while running with Numpy about qibojit

[AleCandido]

https://github.com/numba/numba/blob/28573bb985cf4816e1c6f54ce2f4bd1e44617bcd/numba/np/ufunc/parallel.py#L563-L573

def gen_snt_check():
    from numba.core.config import NUMBA_NUM_THREADS
    msg = "The number of threads must be between 1 and %s" % NUMBA_NUM_THREADS

    def snt_check(n):
        if n > NUMBA_NUM_THREADS or n < 1:
            raise ValueError(msg)
    return snt_check

snt_check = gen_snt_check()

So someone is setting NUMBA_NUM_THREADS. I checked it is not available in my environment as an env var. Nor it is specified anywhere in qibotn.

[AleCandido]

And also, when inside IPython in the same environment, I get:

[ins] In [1]: import numba

[ins] In [2]: numba.config
Out[2]: <module 'numba.core.config' from '/media/alessandro/moneybin/Projects/Qibo/qibotn/env/lib/python3.10/site-packages/numba/core/config.py'>

[ins] In [3]: numba.config.NUMBA_NUM_THREADS
Out[3]: 12

while clearly Numba is resolving it to 6 during pytest run...

[scarrazza]

Did you check quimb import call?

[AleCandido]

The quimb import in qibotn is trivial:

import quimb as qu
import quimb.tensor as qtn

I'm going to check immediately inside quimb package itself.

[AleCandido]

You were right, culprit found:
https://github.com/jcmgray/quimb/blob/8ecf9d6ef7d2055ebbae5f7dbb5aeb0e65c0d422/quimb/core.py#L37-L51

    import psutil

    _NUM_THREAD_WORKERS = psutil.cpu_count(logical=False)

if "NUMBA_NUM_THREADS" in os.environ:
    if int(os.environ["NUMBA_NUM_THREADS"]) != _NUM_THREAD_WORKERS:
        import warnings

        warnings.warn(
            "'NUMBA_NUM_THREADS' has been set elsewhere and doesn't match the "
            "value 'quimb' has tried to set - "
            f"{os.environ['NUMBA_NUM_THREADS']} vs {_NUM_THREAD_WORKERS}."
        )
else:
    os.environ["NUMBA_NUM_THREADS"] = str(_NUM_THREAD_WORKERS)

[AleCandido]

Indeed, this is working:

❯ QUIMB_NUM_PROCS=12 pytest
===================================================================== test session starts =====================================================================
platform linux -- Python 3.10.7, pytest-7.2.1, pluggy-1.0.0
rootdir: /media/alessandro/moneybin/Projects/Qibo/qibotn, configfile: pyproject.toml, testpaths: tests/
plugins: env-0.8.1, cov-4.0.0
collected 4 items

tests/test_qasm_quimb_backend.py ....                                                                                                                   [100%]

The explanation is a few lines above those cited in the previous comment.

[AleCandido]

@scarrazza then I guess I shoul move this issue to qibotn, and set the correct variable there.

If you agree, I will "transfer" the issue there (i.e. open a new one, since being a private repo I can't actually transfer).

[scarrazza]

Sure, please go ahead.