Rlock - with limit? about portalocker

[wolph]

I've added a BoundedSemaphore class that uses the method with the multiple files I suggested earlier. That should be completely safe from race conditions and everything

Usage:

>>> import portalocker
>>> n = 2
>>> timeout = 0.1

>>> semaphore_a = portalocker.BoundedSemaphore(n, timeout=timeout)
>>> semaphore_b = portalocker.BoundedSemaphore(n, timeout=timeout)
>>> semaphore_c = portalocker.BoundedSemaphore(n, timeout=timeout)

>>> semaphore_a.acquire()
<portalocker.utils.Lock object at ...>
>>> semaphore_b.acquire()
<portalocker.utils.Lock object at ...>
>>> semaphore_c.acquire()
Traceback (most recent call last):
  ...
portalocker.exceptions.AlreadyLocked

And that can be done from separate processes of course :)

[wolph]

The RLock already keeps track of the lock count, but I don't think it's useful for your case actually. The RLock only works for multiple threads within a single process because (as far as I am aware) there is no system call that allows for lock counting.

Naturally this can be hacked around if needed. One option would be to create 3 separate lockfiles and try the lockfiles until an available one has been found. The library doesn't currently support that but it would be fairly easy to implement I think.

Alternatively, if your library is using the multiprocessing library you could use a bounded semaphore: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.BoundedSemaphore
But that suffers from the same restrictions that not every operating system has native support for those. So on OS X it's implemented with a loop/sleep combination.

[pliablepixels]

Hi, thanks. The multiple lock file approach seems kludgy but simple. I was considering an approach where we could use a single lock file with a lock value written into it, but I think your suggestion maps better into file open locking. I'll experiment with your suggestion. Thanks.

[wolph]

A single lock file with a value written to it would be possible too and not too hard to implement. The only risk involved is during a crash of your application that the count won't be decreased. Beyond that it could be as simple as this:

import portalocker

max_locks = 3


def lock(increment=1):
    with portalocker.Lock('lockfile', 'r+', timeout=1) as fh:
        lock_count = int(fh.read() or '0')
        if lock_count >= max_locks:
            raise RuntimeError('No GPU available')
        else:
            fh.seek(0)
            fh.write(str(lock_count + increment))


# To lock:
lock(1)

# do your thing here

# To unlock
lock(-1)

To alleviate crash handling the pid could be written to the file and checked, but that becomes a bit harder already.

[pliablepixels]

I am trying out your example with some small mods, but it looks like the lock write isn't really working

During unlock, it reads 0 and reduces to -1 which means lock did not write. Can you see what I am doing wrong?
Once I get the basics working, I'll implement pid

import portalocker
from time import sleep
import os

class Lock:
    def __init__(self, name='lockfile', max_locks=1):
        self.max_locks = max_locks
        self.name = name

    def _lock(self,increment=1):
        with portalocker.Lock(self.name, 'w+', timeout=1) as fh:
            x = fh.read()
            print ('Got read: {}'.format(x))
            lock_count = int(x or '0')
            if lock_count >= self.max_locks:
                raise RuntimeError('No locks available')
            else:
                fh.seek(0)
                fh.write(str(lock_count + increment))
                fh.flush()
                os.fsync(fh.fileno())
                print ('Writing '+str(lock_count + increment))

    def lock(self):
        self._lock(1)

    def unlock(self):
        self._lock(-1)

# To lock:

lock = Lock()
print ('Getting lock')
lock.lock()
print ('Got lock, sleep')
sleep(10)
lock.unlock()
print ('Released lock')

[pliablepixels]

thank you! works perfectly, even if I interrupt a running process that has acquired a lock.