Image.open(image) chokes memory when preparing custom dataset about transformers-tutorials

[NielsRogge]

Why is this memory consuming?

Normally, in PyTorch, dataloaders are Python generators, meaning they will only consume RAM when iterating over them.

[sankalptambe]

Why is this memory consuming?

Normally, in PyTorch, dataloaders are Python generators, meaning they will only consume RAM when iterating over them.

Yeah @NielsRogge , I thought so, and I tried that but it caused the below error.

ValueError: Images must of type `PIL.Image.Image`, `np.ndarray` or `torch.Tensor` (single example),`List[PIL.Image.Image]`, `List[np.ndarray]` or `List[torch.Tensor]` (batch of examples), but is of type <class 'generator'>

The processor() below expects Images as a List[PIL.Image.Image].I am new to all this. Could you help me here?

Below is the code.

class LMDataset(torch.utils.data.Dataset):
    def __init__(self, encodings):
        self.encodings = encodings

    def __getitem__(self, idx):
        item = {key: torch.tensor(val[idx]) for key, val in self.encodings.items()}
        return item

    def __len__(self):
        return len(self.encodings)

processor = LayoutLMv2Processor.from_pretrained("microsoft/layoutlmv2-base-uncased", revision="no_ocr")

train_encodings = processor(
        [Image.open(image).convert('RGB') for image in tqdm(train_df['image'])], # <--- this is causing memory issues.
        train_df['words'].tolist(),
        boxes=train_df['boxes'].tolist(),
        word_labels=[[label2id[label] for label in labels] for labels in train_df['labels'].tolist()],
        return_tensors="pt", padding='max_length', truncation=True)

train_dataset = LMDataset(train_encodings)

[sankalptambe]

Yes my mistake. Missed the 'yield'. Fixed that and working now.