Comments (8)
Thank you @msamogh, this makes perfect sense.
from nonechucks.
Hi @sannawag,
Can you tell me a bit more of what you are trying to do? It would help me understand your situation better so that I can help you.
from nonechucks.
@msamogh, given the large size of my training dataset, I wish to validate more often than once per epoch. For this reason, when enumerating the dataloader, I check whether the index equals the length of the dataloader - 1. I do not directly have access to the dataset length because I initialize the dataloaders in a separate function.
from nonechucks.
So if I understand you correctly, you wish to enumerate through a single DataLoader in a nested fashion?
from nonechucks.
I do wish to enumerate through DataLoaders in a nested fashion, but one is a built from a training set, the other from a validation set.
from nonechucks.
So what's preventing you from enumerating through the validation set in the usual way (using enumerate()
)? You can reinitialize your validation set DataLoader inside the loop as many times you want.
from nonechucks.
Here is the basic structure I am trying to obtain:
for i, sample in enumerate(training_dataloader):
# process the training sample
if i % step == 0 or i == len(training_dataloader) - 1:
validate_and_report_loss(validation_dataloader)
The catch is computing len(training_dataloader)
.
Thanks!
from nonechucks.
Ah, I see. The bad news is you can't call len
on your DataLoader (unless you're okay with setting eager_eval
to True
on your dataset). The good news is, in this case, is that you can simply move the part where you check if it's the last iteration to outside the loop (once it has ended).
This is because without actually checking every element, SafeDataLoader
has no way of telling what the effective number of valid samples in your dataset is going to be. So one of the things you will have to give up with SafeDataLoader
is the ability to call len()
.
Hope that helps!
from nonechucks.
Related Issues (20)
- Memory leak HOT 10
- Override some functions of the original dataset HOT 2
- batch size reduction HOT 9
- Not compatible with PyTorch 1.2.0 HOT 1
- Not compatible with PyTorch v1.2 any more HOT 2
- KeyError:(<function SafeDataset.__getitem__ at ...>) HOT 2
- KeyError: (<function SafeDataset.__getitem__ at 0x0000025B61310B70>, (1527,), frozenset()) HOT 3
- nonechucks breaks with pytorch's revision not in int HOT 1
- dataset attribute should not be set after SafeDataLoader is initialized HOT 5
- Pytorch's IterableDataset HOT 2
- import nonechucks fails for torch1.4.0+cu100 HOT 3
- Skip filtering step for safe samples
- ValueError: dataset attribute should not be set after SafeDataLoader is initialized HOT 4
- Should SafeDataset drop __getitem__ and inherrit IterableDataset? HOT 1
- Potential bug in `_reset_index`
- _get_pytorch_version can't deal with +cuda versions
- Colab issue HOT 2
- Skip Samples with High Confidence after each epoch
- Generator keyword error HOT 1
- support newer versions of pytorch
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nonechucks.