Comments (6)
I've added a BoundedSemaphore
class that uses the method with the multiple files I suggested earlier. That should be completely safe from race conditions and everything
Usage:
>>> import portalocker
>>> n = 2
>>> timeout = 0.1
>>> semaphore_a = portalocker.BoundedSemaphore(n, timeout=timeout)
>>> semaphore_b = portalocker.BoundedSemaphore(n, timeout=timeout)
>>> semaphore_c = portalocker.BoundedSemaphore(n, timeout=timeout)
>>> semaphore_a.acquire()
<portalocker.utils.Lock object at ...>
>>> semaphore_b.acquire()
<portalocker.utils.Lock object at ...>
>>> semaphore_c.acquire()
Traceback (most recent call last):
...
portalocker.exceptions.AlreadyLocked
And that can be done from separate processes of course :)
from portalocker.
The RLock
already keeps track of the lock count, but I don't think it's useful for your case actually. The RLock
only works for multiple threads within a single process because (as far as I am aware) there is no system call that allows for lock counting.
Naturally this can be hacked around if needed. One option would be to create 3 separate lockfiles and try the lockfiles until an available one has been found. The library doesn't currently support that but it would be fairly easy to implement I think.
Alternatively, if your library is using the multiprocessing
library you could use a bounded semaphore: https://docs.python.org/3/library/multiprocessing.html#multiprocessing.BoundedSemaphore
But that suffers from the same restrictions that not every operating system has native support for those. So on OS X it's implemented with a loop/sleep combination.
from portalocker.
Hi, thanks. The multiple lock file approach seems kludgy but simple. I was considering an approach where we could use a single lock file with a lock value written into it, but I think your suggestion maps better into file open locking. I'll experiment with your suggestion. Thanks.
from portalocker.
A single lock file with a value written to it would be possible too and not too hard to implement. The only risk involved is during a crash of your application that the count won't be decreased. Beyond that it could be as simple as this:
import portalocker
max_locks = 3
def lock(increment=1):
with portalocker.Lock('lockfile', 'r+', timeout=1) as fh:
lock_count = int(fh.read() or '0')
if lock_count >= max_locks:
raise RuntimeError('No GPU available')
else:
fh.seek(0)
fh.write(str(lock_count + increment))
# To lock:
lock(1)
# do your thing here
# To unlock
lock(-1)
To alleviate crash handling the pid could be written to the file and checked, but that becomes a bit harder already.
from portalocker.
I am trying out your example with some small mods, but it looks like the lock write isn't really working
During unlock, it reads 0 and reduces to -1 which means lock did not write. Can you see what I am doing wrong?
Once I get the basics working, I'll implement pid
import portalocker
from time import sleep
import os
class Lock:
def __init__(self, name='lockfile', max_locks=1):
self.max_locks = max_locks
self.name = name
def _lock(self,increment=1):
with portalocker.Lock(self.name, 'w+', timeout=1) as fh:
x = fh.read()
print ('Got read: {}'.format(x))
lock_count = int(x or '0')
if lock_count >= self.max_locks:
raise RuntimeError('No locks available')
else:
fh.seek(0)
fh.write(str(lock_count + increment))
fh.flush()
os.fsync(fh.fileno())
print ('Writing '+str(lock_count + increment))
def lock(self):
self._lock(1)
def unlock(self):
self._lock(-1)
# To lock:
lock = Lock()
print ('Getting lock')
lock.lock()
print ('Got lock, sleep')
sleep(10)
lock.unlock()
print ('Released lock')
from portalocker.
thank you! works perfectly, even if I interrupt a running process that has acquired a lock.
from portalocker.
Related Issues (20)
- redis submodule cannot be imported due to circular import HOT 4
- race condition for⦠file metadata? HOT 4
- Setting the DenyMode against a file HOT 2
- is lock reenterable? HOT 2
- distutils was depreacted in Python 3.10 HOT 2
- Timeout not working with SHARED flag HOT 4
- Better Examples Needed HOT 3
- LockException(OSError(22, 'Invalid argument')) when using NON_BLOCKING flag on Linux HOT 2
- DLL load failed while importing win32file HOT 1
- What is the actual license for this project? HOT 1
- fileno support HOT 2
- Mismatch between documentation and implementation HOT 3
- How to lock a file but be able to read it HOT 4
- Python 3.12: ModuleNotFoundError: No module named 'distutils' HOT 2
- Require explicit name for BoundedSemaphore()? HOT 9
- docs files are installed in the wrong location HOT 5
- Version 2.7.0 Released in January 2023 was updated recently to remove support for Python 3.7 HOT 13
- 'portalocker' has no attribute 'LockFlags'
- lock on NFS mount not working HOT 10
- Is this 2016 StackOverflow answer still accurate? (`fh = open(p); Lock(fh)` vs. `with Lock(p) as fh`) HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from portalocker.