Comments (2)
Proof of concept with fcntl:
from multiprocessing.pool import Pool
import os
import json
import contextlib
import fcntl
import time
RUNS_PER_WORKER = 30_000
WORKERS = 10
def writer():
for _ in range(RUNS_PER_WORKER):
with open("test.json", "r+b") as f:
fcntl.lockf(f, fcntl.LOCK_EX)
counter = json.loads(f.read())
counter["counter"] += 1
f.seek(0)
f.write(json.dumps(counter).encode())
f.truncate()
fcntl.lockf(f, fcntl.LOCK_UN)
def reader():
for _ in range(RUNS_PER_WORKER):
with open("test.json", "r+") as f:
fcntl.lockf(f, fcntl.LOCK_SH)
counter = json.loads(f.read())
fcntl.lockf(f, fcntl.LOCK_UN)
if __name__ == "__main__":
t1 = time.time()
with contextlib.suppress(FileExistsError):
fd = os.open("test.json", os.O_CREAT | os.O_RDWR | os.O_EXCL)
os.write(fd, json.dumps({"counter": 0}).encode())
os.close(fd)
pool = Pool(WORKERS)
for _ in range(WORKERS):
pool.apply_async(writer)
pool.apply_async(reader)
pool.close()
pool.join()
td = time.time() - t1
print(f"Time: {td:.2f} seconds, per second: {WORKERS * RUNS_PER_WORKER / td:.2f}")
-> 19076 op/s
from dictdatabase.
Current solution:
from multiprocessing.pool import Pool
import os
import json
import contextlib
import time
import dictdatabase as DDB
RUNS_PER_WORKER = 3_000
WORKERS = 10
def writer():
DDB.config.storage_directory = "."
for _ in range(RUNS_PER_WORKER):
with DDB.at("test").session() as (session, t):
t["counter"] = t["counter"] + 1
session.write()
def reader():
DDB.config.storage_directory = "."
for _ in range(RUNS_PER_WORKER):
DDB.at("test").read()
if __name__ == "__main__":
t1 = time.time()
with contextlib.suppress(FileExistsError):
fd = os.open("test.json", os.O_CREAT | os.O_RDWR | os.O_EXCL)
os.write(fd, json.dumps({"counter": 0}).encode())
os.close(fd)
pool = Pool(WORKERS)
for _ in range(WORKERS):
pool.apply_async(writer)
pool.apply_async(reader)
pool.close()
pool.join()
td = time.time() - t1
print(f"Time: {td:.2f} seconds, per second: {WORKERS * RUNS_PER_WORKER / td:.2f}")
-> 538 op/s
from dictdatabase.
Related Issues (20)
- .at() filter capabilities
- Design decisions
- Partial read and write on bytes instead of decoded str
- Add config.use_indexing? HOT 1
- Use .ddb folder? HOT 1
- Drop json in favor of orjson? HOT 3
- Question about how `key` matches HOT 3
- Drop config.sort_keys?
- Improve where selector for file
- Change .session() Error behavior
- Evaluate usage of mmap HOT 1
- Use __slots__ and dataclass(slots=true) everywhere
- Batched partial reading
- Add XPath-like searching for keys HOT 1
- Indexer behavior
- Option not to sort the keys HOT 4
- Api improvement HOT 1
- Use clock with shared baseline HOT 1
- Use glob match for dictionary keys HOT 2
- Improve behavior to prevent data loss HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dictdatabase.