benw1 / wings Goto Github PK
View Code? Open in Web Editor NEWWFIRST Infrared Nearby Galaxies Survey helpful code for making and processing WFIRST simulations
WFIRST Infrared Nearby Galaxies Survey helpful code for making and processing WFIRST simulations
With concurrent sessions we are likely to read data and have it become stale later on. When we have highly independent and isolated sessions then we can't easily notify these sessions when their data has gone stale. One way to solve this, is to simply add a version to the database tables. When you go to update a value we can check the table version and if there is a mismatch then read the new value and update again. Each update then gives a new table version. This can just be done for the tables where you expect you'll will need to read and update concurrently and avoid putting this extra work on tables you won't concurrently update.
Here is an example of how to do this in SQLAlchemy.
https://docs.sqlalchemy.org/en/13/orm/versioning.html
When I attempt importing wpipe in 2 distinct python instances, the second attempt hold, on a Lock wait timeout exceeded
until I either close the first python instance or call a wpipe.si.commit()
or wpipe.si.rollback()
in it. That second attempt appears to be blocked when it tries to create the DefaultUser
, specifically when retrying repeatedly to SELECT FOR UPDATE the users table. Somehow, it fails to obtain the lock from the FOR UPDATE clause, and this seems to be affected by the import attempt of the first python instance. As far as my understanding of MySQL goes, this should mean that the session in the first python instance is keeping a lock on that table. However, running the query SHOW OPEN TABLES IN wpipe;
from within a MySQL interactive console between the 2 imports does not return that the users table is being used or locked.
I have sqlalchemy 1.3.19 installed and tried with 1.3.17, but the error is still there. I also have MySQL 5.7.31, although I haven't investigated whether or not the version of MySQL is impacting this. I have tried adding an explicit session.execute("UNLOCK TABLES")
at the end of the wpipe.si.commit
function, without any success.
The current implementation of wpipe
keeps track of a job workload as long as it is part of the python process that corresponds to that job. It remains ignorant of any workload handled by a sub-process that this python process may submit. It would be handy to equip the Job object with methods that could help to keep track of these sub-processes.
The instance method _ending_todo of a Job object handles the exiting procedure, whether or not the Job completed or exited with an error.
Lines 512 to 519 in e3d7519
In the latter case, it only updates the state column in the database to reflect the type of error encountered. Additionally, it should also force-terminates the jobs of any child event it generated and fired.
At the moment there is a functional set of data to test the functionalities of wingspipe
in the corresponding directory. This can already be used to add a small test of wingspipe init
, wingspipe run
and wingspipe delete
to the github workflow.
Ideally, the testing unit needs to be more thorough: notably, it would need to check:
wingspipe run
did process correctlyLines 507 to 511 in e3d7519
This method is meant to do a thorough diagnostics of the pipeline overall state, in particular, if it is currently active or inactive - i.e. if it has currently running jobs or not. It is mostly meant to be used via the wingspipe diagnose
command that calls the explicitly corresponding case in the wpipe.wingspipe
function:
Lines 296 to 297 in e3d7519
As such, its implementation shall print a report of the current status of the pipeline tree of jobs. One such report should give a clear overview of whether or not the pipeline is currently running, and if not if it has completed or terminated prematurely due to an error.
The current implementation of wingspipe run
goes iteratively through all child events and fired jobs, skipping or re-running jobs based on a case by case assessment described in the fire
method of every child events:
Lines 311 to 346 in e3d7519
It should be implemented in such a way that it re-run jobs:
All wingspipe
subcommands could have a -v/--verbose feature to describe in detail what it does. It is most important for wingspipe run
to indicate when it meets a job that has completed and it routes through its child events, and when it meets a job that didn't complete and attempts to re-run it if it had exited with an error.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.