Comments (14)
📆 May status: Delay of 1 sprint expected due to delays in onboarding.
from nucleus.
📆 June status: In progress. On schedule.
from nucleus.
📆 July status: In progress. Task is taking much longer than expected due to large data volumes being worked. Plan to process subset by end of build. POC task that does not impact build delivery.
from nucleus.
📆 August status: Deferring to B14.1. Issues with AWS changing permissions on us in NGAP. Completed a workaround to load the MESSENGER data without Nucleus for the time being. No impact on delivery to I&T.
from nucleus.
@ramesh-maddegoda Here is your parent Epic for the Nucleus deployment. Happy to change the scope if we want to work with a different data set. Or maybe an additional sub-task will be to first just test out whether or not this works in MCP, then the overall pilot is to deploy this for MESSENGER.
from nucleus.
@ramesh-maddegoda Here is your parent Epic for the Nucleus deployment. Happy to change the scope if we want to work with a different data set. Or maybe an additional sub-task will be to first just test out whether or not this works in MCP, then the overall pilot is to deploy this for MESSENGER.
from nucleus.
📆 October status: Demo for SBN and CSS completed on 10/27. Work ongoing to load more data operationally and prepare for beta production usage
from nucleus.
📆 November status: Delayed 2 sprints due to issues with cost associated with initial design of CSS processing. #74 . No impact on delivery expected.
from nucleus.
📆 December status: Delayed 1 additional sprint. Need to process data through Nucleus, but software is ready. No impact on delivery.
from nucleus.
@viviant100 , what would be the next step on that ?
from nucleus.
@ramesh-maddegoda have we loaded all MESSENGER data?
from nucleus.
📆 01/2024 status: delayed 2 sprints. no impact on other tasks
from nucleus.
@ramesh-maddegoda have we loaded all MESSENGER data?
@jordanpadams , when loading the messenger data I found the following bug.
#83
I have loaded the following messenger data directories to MCP OpenSearch. There were few missing products due to the bug above. I will load following directories to JPL AWS too.
MESSDEM_1001/
MSGRMDS_1001/
MSGRMDS_4001/
MSGRMDS_5001/
MSGRMDS_6001/
MSGRMDS_7001/
MSGRMDS_7101/
MSGRMDS_7201/
MSGRMDS_7301/
MSGRMDS_8001/
Also, the following 2 directories are very large and I get timeouts due to AWS credential expiry. So , I use aws s3 sync command to copy data time to time (when there are timeouts).due to .
MSGRMDS_2001/ - 1.4 TiB
MSGRMDS_3001/ - 3.3 TiB
from nucleus.
📆 02/2024 status: In work. On schedule for completion by end of sprint.
from nucleus.
Related Issues (20)
- Nucleus lambda function pds-nucleus-datasync-completion sometimes fails to write copied data file names to Nucleus database HOT 3
- Nucleus Airflow ECEOperator failed to launch ECS task with error "capacity is unavailable at this time" HOT 1
- Nucleus Airflow Validate task fails with error while reading data file: For input string: "Infinity" HOT 1
- Nucleus: Track the DAG processing status of each PDS product for monitoring and troubleshooting HOT 4
- Deploy Nucleus in Test env HOT 14
- Deploy Nucleus in Production env
- Implement Warm Backup Storage Solution (Data Backup Plan) HOT 5
- Pilot project with IMG processing using Nucleus HOT 2
- CSS MVP: Deploy Baseline Nucleus for Catalina Sky Survey in Test and Production Environments HOT 9
- Remove data from staging S3 bucket at the end of PDS basic use case
- Remove data from EFS at the end of PDS basic use case HOT 1
- Minimize EFS use of Nucleus to reduce the cost and to avoid DataSync related complications HOT 4
- Support processing data coming from multiple staging buckets with one Airflow environment
- Read harvest auth credentials from AWS secret manager HOT 2
- Nucleus MWAA DAG tasks are unable to read remote logs from Cloudwatch HOT 1
- Setup role based authentication and authorization for Airflow UI HOT 9
- Add unstable/stable deployments
- As a tenant, I want to test cross-account DAG execution in Airflow
- As a node operator, I want Nucleus to take care of the versioning of my products
- Find a solution to execute Terraform resources that take long time to deploy HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nucleus.