seagate / cortx-experiments Goto Github PK

• Setup CORTX in single node VM
• Setup DAOS in single node VM
• Create bucket in Cortx using aws s3 cli
• Create pool and containers in DAOS
• Create test objects in Cortx and DAOS
• Mount bucket using s3fs
• Copy objects from S3 bucket in Cortx to DAOS using mpifilutils (cortx-s3fs <--> daos-dfuse)
• Copy objects from DAOS source container to S3 bucket in Cortx using mpifilutils (cortx-s3fs <--> daos-dfuse)
• Verify Target bucket and DAOS containers for copied objects in each step
• Document the process in quickstart

@rajkumarpatel2602 

Our main github page listing our integrations has a link for the DAOS organization which points to:

https://github.com/Seagate/cortx-experiments/blob/main/daos-cortx/docs/datamovment_with_s3.md

Is that the correct page that best describes the DAOS integration and enables the community to reproduce what you've done?

The reason that I ask is that I just approved a PR where you added a video link to this page:

https://github.com/Seagate/cortx-experiments/blob/main/daos-cortx/docs/setup_robinhood_policy_engine.md

The problem is that I cannot find any links to that second page from anywhere else in GitHub.  Do we need to add a link to the second page from the first page?  Or do we just need to change the link from the integration page to point to the second page?

By the way, our general rule for github is to avoid orphan pages.  An orphan page is a page that has useful information but which is not reachable via links from other existing documents.  So, if we ever add a new page we have to simultaneously add a link to that new page into an existing page.  In this case, I cannot find any links to this second page so I fear we accidentally created an orphan page.  How can we fix this?

Instead of using:

--daos-src-pool
--daos-src-cont
--daos-dst-pool
--daos-dst-cont

daos paths are now represented by:
daos://<pool_uuid>/<cont_uuid>[/<path>]

A new optional parameter allows explicitly switching between the "DAOS" API and "DFS" API:
--daos-api {DAOS, DFS}
By default, POSIX type containers use he "DFS" API, and other containers use the "DAOS" API.
The difference is that the "DFS" API treats objects as files, whereas the "DAOS" API simply copies all objects in the container.

Is this the normal/acceptable programming by Seagate Employees?

I know this is a POC code, but at the same time it's on github.

Please make github a better place when you add your code, if that's not possible, refrain from making it more poor.

Make sure your code quality does not bring down the average code quality of cpp files on github.

All, I noticed that the default permission that the cortx-community team had on this permission was Write. That seemed wrong so I changed it to Read. If a group of folks need Write permission to this repo, and those folks are already in their own Team, please add Write permission to just that small Team. If you need a new Team, @venkuppu-chn can help. Thanks! Feel free to close this Issue; I just thought this would be an easy way to share this info with you. :)

• Install robinhood policy engine
• Register dfuse mount point as a file system in robinhood
• Verify du, info, find command on existing source container
• Create a policy to move objects greater/lesser than 100MB to s3 bucket
• Documentation for above procedure

• Setup CORTX in single node VM
• Setup DAOS in single node VM
• Create bucket in Cortx using aws s3 cli
• Create pool and containers in DAOS
• Create test objects in Cortx and DAOS
• Copy objects from S3 bucket in Cortx to DAOS using aws s3 cli
• Copy objects from DAOS source container to S3 bucket in Cortx using aws s3 cli
• Verify Target bucket and DAOS containers for copied objects in each step
• Document the process in quickstart

• Setup DAOS in single node VM
• Create pool and source and destination containers
• Create objects in source container
• Copy objects from source container to destination containers using mpifiltuils (with daos support)
• Verify Target container for copied objects
• Document the process in quickstart

https://github.com/Seagate/cortx-experiments/tree/b4447e179bf87ca9a5b6b50b50c803ec0adcda38/S3/Full-Fake-TTFB/objective

Add support to mpifileutils for S3 backend and enable DAOS to CORTX-S3 data movement.

Add support to mpifileutils for motr-client backend and enable DAOS to CORTX data/KVS movement.

• Setup 2 DAOS clusters with single node VM each
• Create pool and source and destination containers in both clusters
• Create objects in source container in cluster 1
• Copy objects from source container in cluster 1 to destination containers in cluster 2 using mpifiltuils (with daos support)
• Verify Target container in cluster 2 for copied objects
• Document the process in quickstart

Please rename the default branch from 'master' to 'main'.

All,

We should change the visibility of this repo to public so we can share this excellent work with the community and embrace the fact that we are 100% open source.

Thanks,

John

Makefile is not broken.

@rajkumarpatel2602 please work with Rachel Novak to get it uploaded into youtube and then link it.

@TechWriter-Mayur please handle this

• Build DAOS from source
• Install built DAOS and configure single node cluster
• Create/Write/Read/Delete DAOS Objects
• Prepare document with steps/references for quickstart daos for CORTX team