Comments (15)
Hey @liamphmurphy wanted to bump and see how you were doing on this? I understand if you are busy and haven't worked on it haha just trying to track various projects states is all
Hey @JacksonMaxfield, thanks for reaching out! My bad for not communicating. I've been moving the last few weeks and most of my communication went out the wayside.
As a general update, I explored the inspect lib and think that's a great way to go. I'm planning on using a Python class to represent each TS model, just to make things tidy / hopefully more readable even if it's a bit overkill.
Needless to say if it needs to be done quicker than I am able to commit feel free to let me know and someone else can jump in, but if not, I'm looking forward to continuing on it.
from cdp-backend.
Just a quick update. still chipping away at this! I've been dealing with COVID over the last week.
I've gotten some basic TypeScript models generated, I'm now working on a lot of those edge cases that require extra logic in the TS constructors, such as ReferenceField's.
from cdp-backend.
At long last (and after a lot of trial and error), I have MatterFile outputting:
import { ResponseData } from "../networking/NetworkResponse";
import { Model } from "./Model";
import { DocumentReference } from "firebase/firestore"
import Matter from "./Matter"
export default class MatterFile implements Model {
id?: string;
matter?: Matter;
matter_ref: string;
name: string;
uri: string;
external_source_id?: string;
constructor(jsonData: ResponseData) {
if (jsonData["id"]) {
this.id = jsonData["id"];
}
if (
typeof jsonData["matter_ref"] == "object" &&
!jsonData["matter_ref"] instanceof DocumentReference)
) {
this.matter = new Matter(jsonData["matter_ref"]);
}
this.matter_ref = jsonData["matter_ref"].id;
this.name = jsonData["name"];
this.uri = jsonData["uri"];
if (jsonData["external_source_id"]) {
this.external_source_id = jsonData["external_source_id"];
}
}
}
I'm going to prepare a PR soon, just want to check some of the other models and check for any touchups first.
The thing I know I'll change is that the id
field doesn't show up as required here, as the Python models don't seem to set the required=True
flag like on the other fields. I'm thinking of enforcing that any field named id
is required no matter what.
from cdp-backend.
In addition to creating the models themselves that have some interaction with the database, it would be great to generate the constants.
class FileFields:
name = "name"
uri = "uri"
So that we can use these constants during filtering and querying
from cdp-backend.
This is where the Python database models live: https://github.com/CouncilDataProject/cdp-backend/blob/main/cdp_backend/database/models.py#L26
This is where the TypeScript database models (and their unpackers) live: https://github.com/CouncilDataProject/cdp-frontend/tree/main/src/models
I think the best way todo this would be to use a jinja template potentially. But really this is all about code generation so whatever method works best.
from cdp-backend.
I've done a bit of initial research on parsing the Python models. It's going to take a bit of puzzle-solving / trial and error, but I'm inclined to go with Python's ast
module for doing the heavy lifting of parsing the Python code, and then using the grammar there as a means of generating the TypeScript with Jinja. For example, if ast
returns that a line is a ClassDef
, then we can get the name of the class from that tree node and pass it into the Jinja template.
Another approach is creating a set of regex's or substring conditionals to replicate the grammar of what goes into a Python class and any expressions / statements within it, but that seems a bit more hacky and error prone.
Or perhaps another approach that doesn't involve such heavy parsing I'm not thinking of...? Not sure what that would be though.
from cdp-backend.
Will give a more thorough look at your comment tomorrow but you may be interested to see how we are parsing this info in the database diagram generator: https://github.com/CouncilDataProject/cdp-backend/blob/main/cdp_backend/bin/create_cdp_database_uml.py
from cdp-backend.
And here is how that DATABASE_MODELS list is generated: https://github.com/CouncilDataProject/cdp-backend/blob/main/cdp_backend/database/__init__.py
Using the inspect
lib
from cdp-backend.
Learning many things about Python's built in modules... ty for the link.
inspect
seems to serve a somewhat similar solution as ast
but in a more user friendly way, that seems like a good way to go.
from cdp-backend.
Hey @liamphmurphy wanted to bump and see how you were doing on this? I understand if you are busy and haven't worked on it haha just trying to track various projects states is all
from cdp-backend.
No worries! Good luck with the move.
As a general update, I explored the inspect lib and think that's a great way to go. I'm planning on using a Python class to represent each TS model, just to make things tidy / hopefully more readable even if it's a bit overkill.
Hmmmm does that mean we will need to update both the Python model and the TS model when we make changes? I don't really care the structure of the class the goal is really just "get database changes and constant additions or changes down to a single point of truth"
Needless to say if it needs to be done quicker than I am able to commit feel free to let me know and someone else can jump in, but if not, I'm looking forward to continuing on it.
No timeline crunch! Please keep working on it whenever move is wrapped up ❤️
from cdp-backend.
Hmmmm does that mean we will need to update both the Python model and the TS model when we make changes? I don't really care the structure of the class the goal is really just "get database changes and constant additions or changes down to a single point of truth"
Good clarifying question, I just misspoke; I'm making a generic "Generator" class that'll have separate attributes, methods etc. to hopefully make it a bit easier to read what this new script is parsing in the Python models, to be used in the TS models. It won't make any assumptions about the models that currently exist.
e.g.:
class Generator:
"""
Generator will contain metadata and support utilities for capturing the data needed
from a single Python model to generate an equivalent TypeScript model
"""
name = ""
docstring = ""
attributes = []
references = [] # list of other models that the TS model needs to import
source_tree = {}
def set_attributes():
....
def generate():
....
....
The actual logic for this isn't super complex, just fiddlying with the inspect
library (and maybe ast
) to make some of the decisions needed, such as which attributes are required.
from cdp-backend.
Ahhhhh cool! Thanks! I am already learning :)
from cdp-backend.
Ahhhh sorry to hear COVID hit you!! Hopefully you are resting well!
Yea, ReferenceFields are always great hahaha. Please let me know if you need anything or any input or anything!
Thanks!!!
from cdp-backend.
I think it's because IDField
types automatically assume required=True
? I can check on that, maybe I am wrong.
This looks great though!
from cdp-backend.
Related Issues (20)
- Investigate / fix m3u8 processing...
- Google Speech-to-Text SR Model raises a confusing attr error instead of a defined error when Google runs into an issue
- Parse closed caption files for Oakland better HOT 13
- Hackathon Cleanup: Audio/Video Clipping
- Accept timestamps and only process a subset of a video as an event HOT 13
- Allow ability to flag Events as "try to scrape again next time" and do so on the next CRON run
- Filter out bad caption files
- Issue with deploy-infra action on 3.2.4 due to missing ffmpeg HOT 1
- Issue with deploy-infra action on 3.2.5 due to missing webvtt HOT 2
- Allow meeting minutes to be processed like a transcript
- Re-enable m3u8 and vimeo file utils resource copy tests HOT 1
- Duplicate Persons in cdp-seattle instance HOT 7
- Docker Images
- Break up Event Gather Pipeline HOT 1
- Cannot store and transcribe multiple videos that are clipped via the video_start_time and video_end_time event params
- Reduce complexity of event gather pipeline HOT 4
- MP4 Videos Unnecessarily Encoded During Trim HOT 3
- Inefficient usage of requests.get for very large videos causes event gather to fail
- deploy-infra - problem when Firestore database created in deprecated Datastore Mode HOT 3
- 403 error on resource_copy HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cdp-backend.