Comments (4)
@ramesh-maddegoda completed the trade study and asked questions to Mike Gangle to complete that.
from nucleus.
The trade study was updated at https://docs.google.com/spreadsheets/d/17Yxf1m6f4VvLd6QFuqJADAhwKPhqQnQemtMb9MqWyDI/edit#gid=2008405426 with Cumulus and the following list if questions were sent to Michael Gangl.
ETL Modes
- Can Cumulus execute ETL jobs in batch mode where data is extracted, transformed and loaded as batches?
- Can Cumulus execute ETL processes in real-time mode where data is extracted, transformed and loaded in real-time as soon as data is received (this can also be considered as streaming ETL)?
Pipeline Execution Control
- Can Cumulus deactivate a data pipeline (so it will not get executed in future)?
- Can Cumulus activate a deactivated data pipeline?
- Can Cumulus suspend a data pipeline (in runtime)?
- Can Cumulus resume a suspended data pipeline (in runtime)??
- Can Cumulus terminate a data pipeline (in runtime)?
Pipeline Triggers
- Can Cumulus trigger the execution of a data pipeline automatically based on a data file receive event (preferably on an S3 bucket?
- Can Cumulus trigger the execution of a data pipeline automatically based on a change in a database table (Change Data Capture)?
Interfaces
- Does Cumulus provide a Representative State Transfer (REST) based Application Programming Interface (API) to import and delete predefined data pipelines?
- Does Cumulus provide a Command Line Interface (CLI) based Application Programming Interface (API) to import and delete predefined data pipelines?
- Does Cumulus provide a python language wrapper for data pipeline monitoring (with ability to check progress and statistics on both current and historical pipelines)?
Security
- Can Cumulus log each user access attempt to the system with the details of date and time of the event, outcome of the access attempt (allowed/denied), user name and user roles?
- Does Cumulus provide role-based permissions to restrict data pipeline authoring, data pipeline management and data pipeline execution? If so, are these restrictions applicable to the operations invoked through other interfaces such as RESTful API and CLI?
- Are data passes through Cumulus only accessible by authorized users/services?
Cost Optimization
- Can Cumulus execute data pipelines on cost optimized infrastructure wherever possible?
- Can Cumulus execute data pipelines in cost optimized timeframes whenever possible?
User Community
- Does Cumulus have an external user community outside JPL and NASA?
from nucleus.
Waiting for answers from Mike Gangle
from nucleus.
Had a meeting with Michael Gangl and got answers to the question we had on Cumulus. Based on that the following Nucleus trade study was updated with Cumulus.
https://docs.google.com/spreadsheets/d/17Yxf1m6f4VvLd6QFuqJADAhwKPhqQnQemtMb9MqWyDI/edit#gid=1611047106
from nucleus.
Related Issues (20)
- Nucleus lambda function pds-nucleus-datasync-completion sometimes fails to write copied data file names to Nucleus database HOT 3
- Nucleus Airflow ECEOperator failed to launch ECS task with error "capacity is unavailable at this time" HOT 1
- Nucleus Airflow Validate task fails with error while reading data file: For input string: "Infinity" HOT 1
- Nucleus: Track the DAG processing status of each PDS product for monitoring and troubleshooting HOT 4
- Deploy Nucleus in Test env HOT 14
- Deploy Nucleus in Production env
- Implement Warm Backup Storage Solution (Data Backup Plan) HOT 5
- Pilot project with IMG processing using Nucleus HOT 2
- CSS MVP: Deploy Baseline Nucleus for Catalina Sky Survey in Test and Production Environments HOT 9
- Remove data from staging S3 bucket at the end of PDS basic use case
- Remove data from EFS at the end of PDS basic use case HOT 1
- Minimize EFS use of Nucleus to reduce the cost and to avoid DataSync related complications HOT 4
- Support processing data coming from multiple staging buckets with one Airflow environment
- Read harvest auth credentials from AWS secret manager HOT 2
- Nucleus MWAA DAG tasks are unable to read remote logs from Cloudwatch HOT 1
- Setup role based authentication and authorization for Airflow UI HOT 9
- Add unstable/stable deployments
- As a tenant, I want to test cross-account DAG execution in Airflow
- As a node operator, I want Nucleus to take care of the versioning of my products
- Find a solution to execute Terraform resources that take long time to deploy HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nucleus.