eltonlaw / gooserocket Goto Github PK
View Code? Open in Web Editor NEWBioinformatics framework
Bioinformatics framework
ServiceError(ServiceError { source: Unhandled(Unhandled { source: ErrorMetadata { code: Some("ValidationError"), message: Some("Stack:arn:aws:cloudformation:us-east-1:208577793574:stack/gr-jupyter-nb-1/0dcb6520-e6c6-11ed-bad7-0ada475864a5 is in ROLLBACK_COMPLETE state and can not be updated."), extras: Some({"aws_request_id": "94cfc515-a1fe-4b5f-b24b-461b658e954e"}) }, meta: ErrorMetadata { code: Some("ValidationError"), message: Some("Stack:arn:aws:cloudformation:us-east-1:208577793574:stack/gr-jupyter-nb-1/0dcb6520-e6c6-11ed-bad7-0ada475864a5 is in ROLLBACK_COMPLETE state and can not be updated."), extras: Some({"aws_request_id": "94cfc515-a1fe-4b5f-b24b-461b658e954e"}) } }), raw: Response { inner: Response { status: 400, version: HTTP/1.1, headers: {"x-amzn-requestid": "94cfc515-a1fe-4b5f-b24b-461b658e954e", "date": "Sat, 29 Apr 2023 19:50:51 GMT", "content-type": "text/xml", "content-length": "421"}, body: SdkBody { inner: Once(Some(b"<ErrorResponse xmlns="http://cloudformation.amazonaws.com/doc/2010-05-15/\">\n \n Sender\n ValidationError
\n Stack:arn:aws:cloudformation:us-east-1:208577793574:stack/gr-jupyter-nb-1/0dcb6520-e6c6-11ed-bad7-0ada475864a5 is in ROLLBACK_COMPLETE state and can not be updated.\n \n 94cfc515-a1fe-4b5f-b24b-461b658e954e\n\n")), retryable: true } }, properties: SharedPropertyBag(Mutex { data: PropertyBag, poisoned: false, .. }) } })
Bake the deps install into cloud formation
Either ssh tunnelling or openvpn instance
OpenVPN is probably better for connecting to multiple instances
For data that isn't necessary to keep long term
Setup s3://gr-data/tmp with object lifecycle policy
Replace all println's with calls to tracing. Should be mostly drag and drop.
Using println
will be tough to work with soon as we start increasing the number of instances that are up at any given time,. Need a place to aggregate all emitted telemetry.
Datadog, newrelic, cloudwatch all work but Honeycomb is nice cause it's free and you can run SQL queries on the emitted data right out of the box
Tasks:
tracing
callsShould be able to restart on sporadic failure or to handles ooms etc.
Putting this into a systemd service might be solution, would need to figure out how to bake the service file into the image (s3)
Either writing a shell call or setting up a lambda/ecs service to run something that already exists:
Or handrolling one of the simpler algorithms:
Given some person's genome/sequencing reads
This way each user can deploy their own stack, at the moment it just unconditionally deploys a gr-jupyter-nb-1
. Probably want to fix this by appending a username so the stack becomes gr-jupyter-nb-<username>
. Username can be parsed from aws sts get-caller-identity
Things that need to be implemented:
./cli shutdown all
: Should delete all gr-jupyter-nb-* stacks./cli shutdown
: Should just delete user deployed stacksAC
Need to be able to run aws api calls from inside the notebook instance. Configure aws credentials in jupyter notebook ec2 instance via https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-metadata.html
AC:
Create a gr-infra/src/aws/s3.rs and add fns to be able to read and write objects from s3. Take a look at how gr-infra/src/aws/cloudformation.rs is implemented. You will want to bring in this crate https://crates.io/crates/aws-sdk-s3
At the moment, create/update just always pass in capabilities for creating IAM resources. Not a security concern but it would probably be more correct to have it passed in only for the stacks that need it.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.