Comments (4)
@jinserk
I think all bugs in this issue has been resolved, so I'll close it. If you need more, please re-open. Thanks for your reporting!
from matorage.
It's embarrassing but I forgot to make all numpy arrays as batch-like (expand_dims
). It worked if I changed:
data_saver({
"id": np.asarray([key]),
"fp": np.expand_dims(feat, axis=0),
"target": np.expand_dims(target, axis=0),
})
But one thing I'd like to suggest is, it will be the best if we have any option to save all data element-wise: so if the attribute has the dims of (20, 20), then will be better to put not (B, 20, 20) but (20, 20) array with some additional option like elementwise=True
in DataSaver.__call__()
or separated function DataSaver.save_element()
.
from matorage.
@jinserk
Thanks for the detailed bug reporting. The first issue with filetype
is my mistake.
The filetype is the content added to 0.2.0, and it is an option to save the corresponding file when you enter the file path here.
This is a boolean option called filetype. Therefore, I set an error to occur when the filetype is False
and the attribute type is string, and I will remove it.
fixed
- remove raise error(TypeError: I suspect you need to set the filetype.)
from matorage.
It's embarrassing but I forgot to make all numpy arrays as batch-like (
expand_dims
). It worked if I changed:data_saver({ "id": np.asarray([key]), "fp": np.expand_dims(feat, axis=0), "target": np.expand_dims(target, axis=0), })
But one thing I'd like to suggest is, it will be the best if we have any option to save all data element-wise: so if the attribute has the dims of (20, 20), then will be better to put not (B, 20, 20) but (20, 20) array with some additional option like
elementwise=True
inDataSaver.__call__()
or separated functionDataSaver.save_element()
.
The unittest related to this works without problems. Could you remove the raise TypeError("I suspect you need to set the filetype.")
and try again?
from matorage.
Related Issues (14)
- A few questions for the usage HOT 7
- Feature for storing not only 'numpy arrays' but also file format required for training
- Save & load for scheduler
- show bucket list each type(dataset, model and optimizer) HOT 1
- support inference large models such as gpt-3 in storage calculation. HOT 1
- no metadata dir in a compressed bucket HOT 6
- dataloader raises pickle error when it is used in a process HOT 3
- AssertionError(assert len(self._object_file_mapper) == (len(self.merged_indexer) + len(self.merged_filetype))) HOT 13
- Error when I change the attributes if DataSaver with refresh=True HOT 1
- Createt context manager for `DataSaver`
- metadata file change from FileStorage to Database HOT 5
- When there are many cores during multi-processing, the metadata and the actually uploaded file are not synced.
- How to save and load .pt pytorch model (to and from Minio) using your library?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from matorage.