Comments (20)
@ramesh-maddegoda focuses on MSGRMDS_4001 and MESSDEM_1001 that need to be loaded in the registry first so that they can be used in ticket NASA-PDS/search-api-notebook#24
from nucleus.
Blocked because AWS Airflow is unavailable on NGAP
from nucleus.
Unblocked since Ramesh work on MCP. He is now testing the ECS task called by the nucleus workflow.
from nucleus.
Status: @ramesh-maddegoda working on improving Terraform deployments
from nucleus.
@ramesh-maddegoda is deploying everything needed on MCP, from scratch.
from nucleus.
@ramesh-maddegoda will test nucleus to validate its robustness with a bigger dataset.
from nucleus.
Some of the files in the s3://asc-pds-messenger failed to copy to the PDS Nucleus staging bucket with a permission issue.
aws s3 cp s3://asc-pds-messenger/MSGRMDS_8001/RTM/MDIS_RTM_N01/2013_228/MDIS_RTM_N01_006974_4644396_1.IMG s3://pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_N01/2013_228/
copy failed: s3://asc-pds-messenger/MSGRMDS_8001/RTM/MDIS_RTM_N01/2013_228/MDIS_RTM_N01_006974_4644396_1.IMG to s3://pds-nucleus-staging/messenger-data/MSGRMDS_8001/RTM/MDIS_RTM_N01/2013_228/MDIS_RTM_N01_006974_4644396_1.IMG An error occurred (AccessDenied) when calling the GetObjectTagging operation: Access Denied
from nucleus.
A new parameter enable to copy all the metadata .
from nucleus.
@ramesh-maddegoda identified a bug while doing that test. The lambda reading the data sync report is now taking more that 15 minutes. Now there will be a single lambda call per report.
from nucleus.
The upgrade worked on a small dataset and @ramesh-maddegoda is now testing on the messenger dataset.
from nucleus.
Now Ramesh is loading data to the registry on JPL AWS. Last step for this task.
from nucleus.
20,000 processed! 8 directories ran! Found 2 errors:
- "Capacity not available at the moment", related to ECS, seems to have run out; need to have "capacity providers" but cost is a concern; see related bug in Nucleus
- Role error: have seen it before, error occured by reading data file. An error in validateβ½ Trying to isolate it with the same Docker image and reproduce it without Nucleus. Is it Airflow or validate?
from nucleus.
- capacity issue is solved with a retry configuration.
- validate error is a bug in validate: NASA-PDS/validate#844
from nucleus.
@ramesh-maddegoda is experimenting with SQS to send to new records to the mysql database and avoid the time out he was experiencing with direct insertion.
from nucleus.
SQS now mostly works, but another lambda had a time out.
from nucleus.
We now integrate the copy from S3 to EFS as a nucleus step in the DAGs. We give up DataSync which comes with risks of overlapping copies and complication to remove files from EFS.
@ramesh-maddegoda will also write a note in a wiki for a future design where we don't need to use EFS at all.
from nucleus.
This work has been paused as we focus on Catalina Sky Survey. Will move to B15.0 release plan to complete work.
from nucleus.
π 05/2024 status: Delayed several sprints due to delays in #93. This is an operations activity. No impact on build.
from nucleus.
π 06/2024 status: Delayed several sprints due to delays in #93. This is an operations activity. No impact on build.
from nucleus.
π 07/2024 status: Delayed several sprints due to delays in #93. This is an operations activity. No impact on build.
from nucleus.
Related Issues (20)
- Nucleus lambda function pds-nucleus-datasync-completion sometimes fails to write copied data file names to Nucleus database HOT 3
- Nucleus Airflow ECEOperator failed to launch ECS task with error "capacity is unavailable at this time" HOT 1
- Nucleus Airflow Validate task fails with error while reading data file: For input string: "Infinity" HOT 1
- Nucleus: Track the DAG processing status of each PDS product for monitoring and troubleshooting HOT 4
- Deploy Nucleus in Test env HOT 14
- Deploy Nucleus in Production env
- Implement Warm Backup Storage Solution (Data Backup Plan) HOT 5
- Pilot project with IMG processing using Nucleus HOT 2
- CSS MVP: Deploy Baseline Nucleus for Catalina Sky Survey in Test and Production Environments HOT 9
- Remove data from staging S3 bucket at the end of PDS basic use case
- Remove data from EFS at the end of PDS basic use case HOT 1
- Minimize EFS use of Nucleus to reduce the cost and to avoid DataSync related complications HOT 4
- Support processing data coming from multiple staging buckets with one Airflow environment
- Read harvest auth credentials from AWS secret manager HOT 2
- Nucleus MWAA DAG tasks are unable to read remote logs from Cloudwatch HOT 1
- Setup role based authentication and authorization for Airflow UI HOT 9
- Add unstable/stable deployments
- As a tenant, I want to test cross-account DAG execution in Airflow
- As a node operator, I want Nucleus to take care of the versioning of my products
- Find a solution to execute Terraform resources that take long time to deploy HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nucleus.