How to export data? about deeplake

[mikayelh]

Hey @Bit0r thanks for reaching out!

MySQL is a non AI-native database. Creating a format that is optimized for ML workflows comes at a tradeoff such as the need for the specific format that is hyper performant in those use cases. Since data in Deep Lake format is also very efficiently stored (depends on the datatype, you may actually save up to 30% of storage costs), we haven't seen any need in converting data to original format.

While we currently do not offer export functionality, we will certainly flag this as a feature request. Could you specify a reason why this would be important for your daily work?

Thanks!

[Bit0r]

Because our GPU servers for training ML models cannot connect to the external network. We need to download the data to the local machine first, and then transfer it using SSH. If Deeplake has data import and export functions, I can use Deeplake on the GPU server.

[activesoull]

Because our GPU servers for training ML models cannot connect to the external network. We need to download the data to the local machine first, and then transfer it using SSH. If Deeplake has data import and export functions, I can use Deeplake on the GPU server.

hi @Bit0r,
Can you please check out functionality here? It helps to copy the source dataset to the given destination path

[Bit0r]

So can I first use deeplake.deep_copy to copy the downloaded dataset to save/to/dst_dataset, then directly use tar to package the entire save/to/dst_dataset directory, and finally upload it via ssh?

[activesoull]

yes save/to/dst_dataset directory contains a complete dataset structure so you can archive and move it wherever you want

[mikayelh]

hey @Bit0r , please let us know if this answers your questions. As for now, i'm closing this issue given that our current capabilities address your use case without a need to export the data. I'm however looping in @istranic / @istranical to log this as a feature enhancement.