Comments (7)
Filing this as a bug. The SQL handling is all pretty crappy and could be made better. One thing I will mention here is that any fixes must work for the Mongo, SQL, and Base backends, so the canonical definitions of the models will have to be stored in either the core classes or in a new centralized module. I think the best way would be for from_backend
to do its best to figure out what it wants from the core classes themselves.
To your last question, yes, data should be able to be preserved through schema migrations. I'm guessing SQLAlchemy has some functionality that should help us automate this.
from dagobah.
This is blocking the next release. Everyone that uses SQLite is going to have a shitty time upgrading unless the migration handles itself.
from dagobah.
Closed by f917990
from dagobah.
Whoops, I fixed one issue but not the other. Reopening this until changing the model is less of a pain.
from dagobah.
@utkarsh2012 Would appreciate your input here, as I'm having a hard time seeing an easy solution to the model definition problems in core.py. Here's my understanding of what changes in core.py when the Task schema is updated. Let's add a new variable called yoda
.
- Go to
Task.__init__
and setyoda
to its starting value. If it needs its own setters or special defaults (e.g.soft_timeout
), then handle that now, too. - If
yoda
needs to be persisted in the backend, add it toTask._serialize
. Importantly, this does not necessarily need to be done for everything in the Task definition, though it will be done for every variable included in the backend models. - If
yoda
can be set on a new Task at creation time, add it toDagobah._add_job_from_spec
(this used to be infrom_backend
when this issue was originally posted). Again, importantly, this does not necessarily need to be done for every new variable. It also does not necessarily need to be done if you did the previous step.
Given that these three steps aren't necessarily related, I think the only way to consolidate the process would be to add what would probably be a fairly complex config object that is capable of assigning variables to any subset of these three steps, as well as applying formatters to each individual step (e.g. soft_timeout
gets a default value of 0 in step 2).
Do you agree? If not, how do you think we could do this? If you do agree, do you think this path is worth it?
from dagobah.
I think addition of Alembic fixes a major issue of schema migration. Existing users will see a flawless migration.
Regarding the changes needed (step 1 to 3) in the app after addition of a variable to a table, as you mentioned it looks like a hard problem to solve which is not really worth tackling. New variables are not really added frequently to a table, when they are added we can go through this exercise.
Although it should be clearly documented some where (in developer doc rather than user doc) so that someone can hack around it quickly.
from dagobah.
Added a bit of developer documentation in the wiki: https://github.com/tthieman/dagobah/wiki/Changing-the-Core-Models
from dagobah.
Related Issues (20)
- error when start dagobah
- Installation issue, missing gpost HOT 1
- dagobahd not working on Mac OS X HOT 1
- get_status method for job HOT 2
- Changing code to use py-dag as a package instead of the internal file HOT 2
- RuntimeError: can't start new thread
- How to run and debug the whole project? HOT 1
- Is it distributed?
- Timezone cron / scheduler
- Updating config file, changes not reflected HOT 6
- Run Dagobah from folder HOT 1
- Address jump HOT 6
- [CLOSED] reformat code
- [CLOSED] Reformat code
- [CLOSED] Reformat code
- [CLOSED] update setup requirements
- [CLOSED] add amazon ses service in emails
- [CLOSED] Close SSH connection for remote tasks
- [CLOSED] UnicodeEncodeError while sending basic email
- Fail to install dagobah HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dagobah.