The neuralmachinetranslation from gugundi

neuralmachinetranslation's People

Contributors

Stargazers

Watchers

neuralmachinetranslation's Issues

About pad with window

I noticed pad with window in the encoder section,

def pad_with_window_size(self, batch):
    size = batch.size()
    n = len(size)
    if n == 2:
        length, batch_size = size
        padded_length = length + (2 * self.window_size + 1)
        padded = torch.empty((padded_length, batch_size), dtype=torch.long, device=self.device)
        padded[:self.window_size, :] = self.pad
        padded[self.window_size:self.window_size + length, :] = batch
        padded[-(self.window_size + 1):, :] = self.pad
    elif n == 3:
        length, batch_size, hidden = size
        padded_length = length + (2 * self.window_size + 1)
        padded = torch.empty((padded_length, batch_size, hidden), dtype=torch.long, device=self.device)
        padded[:self.window_size, :, :] = self.pad
        padded[self.window_size:self.window_size + length, :, :] = batch
        padded[-(self.window_size + 1):, :, :] = self.pad
    else:
        raise Exception(f'Cannot pad batch with {n} dimensions.')
    return padded

When calculating the attention, pad with window on the output of the encoder, that is when n==3, can this achieve the same effect? But this will seriously hurt my results when I try to do this.

Hi, I want to ask some about your code.
In 176-188 lines in model.py, why to do this:
if start < self.window_size:
d = self.window_size - start
score[i, j, :d] = epsilon
if end > li + self.window_size:
d = (li + self.window_size) - end
score[i, j, d:] = epsilon

Shouldn’t it judge whether the selected window is beyond length?

Recommend Projects