Segfault after the program runs for a while about marian-dev

[emjotde]

Hi,
That looks more like a driver error to me? What machine is this? Also does this happen regularly, only once, only with your branch? Too little information to guess anything.

[afaji]

this is from master, original code without modifying anything.
Happened once, just today.

running on azure

[emjotde]

Hm, difficult to say. What's the driver version? On some machines the nvidia drivers are highly unstable, not something we can fix really.

[afaji]

Tesla M60

command used: `./marian -t corpus.ro corpus.en -d 0 1 2 3 --disp-freq=100 --layer-normalization``

Actually, just happened again right now, with freshly pulled code:

[2017-03-22 16:49:00] Ep. 1 : Up. 5600 : Sen. 358400 : Cost 56.34 : Time 48.07s : 3026.32 words/s
[2017-03-22 16:49:52] Ep. 1 : Up. 5700 : Sen. 364800 : Cost 55.40 : Time 51.16s : 2809.11 words/s
[2017-03-22 16:50:42] Ep. 1 : Up. 5800 : Sen. 371200 : Cost 56.01 : Time 50.92s : 2893.21 words/s
GPUassert: an illegal memory access was encountered /home/alfikri/ori/marian/src/kernels/tensor_operators.cu 567
GPUassert: an illegal memory access was encountered /home/alfikri/ori/marian/src/tensors/tensor.cu 46
Segmentation fault (core dumped)

[emjotde]

Can you reset the machine? Maybe the driver went out of whack. W dniu 22.03.2017 o 17:57, afaji pisze:

…

Tesla M60 Actually, just happened again right now, with freshly pulled code: |[2017-03-22 16:49:00] Ep. 1 : Up. 5600 : Sen. 358400 : Cost 56.34 : Time 48.07s : 3026.32 words/s [2017-03-22 16:49:52] Ep. 1 : Up. 5700 : Sen. 364800 : Cost 55.40 : Time 51.16s : 2809.11 words/s [2017-03-22 16:50:42] Ep. 1 : Up. 5800 : Sen. 371200 : Cost 56.01 : Time 50.92s : 2893.21 words/s GPUassert: an illegal memory access was encountered /home/alfikri/ori/marian/src/kernels/tensor_operators.cu 567 GPUassert: an illegal memory access was encountered /home/alfikri/ori/marian/src/tensors/tensor.cu 46 Segmentation fault (core dumped) | — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#45 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABCG9ahVVtae5y93BIML-bjlMhw7wcmUks5roVLegaJpZM4Mlbg1>.

[afaji]

Still happened after reset.
I just deallocate and re-start the azure server, not sure if that counts as resetting the machine.

[2017-03-22 18:33:09] Ep. 2 : Up. 7100 : Sen. 81408 : Cost 47.13 : Time 49.01s : 3035.36 words/s
[2017-03-22 18:33:53] Ep. 2 : Up. 7200 : Sen. 87808 : Cost 43.82 : Time 44.38s : 3113.88 words/s
[2017-03-22 18:34:46] Ep. 2 : Up. 7300 : Sen. 94208 : Cost 48.74 : Time 52.93s : 2899.93 words/s
GPUassert: an illegal memory access was encountered /home/alfikri/ori/marian/src/kernels/tensor_operators.cu 567
GPUassert: an illegal memory access was encountered /home/alfikri/ori/marian/src/tensors/tensor.cu 46
Segmentation fault (core dumped)

So this is 3 errors in 3 attempts.

[emjotde]

OK, very weird. Which driver version is nvidia-smi reporting? Can you make the data and a config with your parameters avalable for download? W dniu 23.03.2017 o 08:37, afaji pisze:

…

Still happened after reset. I just deallocate and re-start the azure server, not sure if that counts as resetting the machine. |[2017-03-22 18:33:09] Ep. 2 : Up. 7100 : Sen. 81408 : Cost 47.13 : Time 49.01s : 3035.36 words/s [2017-03-22 18:33:53] Ep. 2 : Up. 7200 : Sen. 87808 : Cost 43.82 : Time 44.38s : 3113.88 words/s [2017-03-22 18:34:46] Ep. 2 : Up. 7300 : Sen. 94208 : Cost 48.74 : Time 52.93s : 2899.93 words/s GPUassert: an illegal memory access was encountered /home/alfikri/ori/marian/src/kernels/tensor_operators.cu 567 GPUassert: an illegal memory access was encountered /home/alfikri/ori/marian/src/tensors/tensor.cu 46 Segmentation fault (core dumped) | So this is 3 errors in 3 attempts. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#45 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABCG9dNeOk6XswNS0KCpluH2pW4eVzLdks5roiEvgaJpZM4Mlbg1>.

[afaji]

Driver version: 375.39

config: ./marian -t corpus.ro corpus.en -d 0 1 2 3 --disp-freq=100 --layer-normalization --log=data_baseline.txt

data:
I ran the this wmt script and used corpus.ro and corpus.en

[emjotde]

I have been having problems with those new driver versions, especially with this version (weird errors, random freezes), and downgraded to 367.48 on all my machines. This made all errors go away. You can try this and I can experiment with the data in the meantime.

[emjotde]

You did not do any preprocessing? Notice that the data is not even tokenized. But I guess it might be empty lines or something, so maybe a good test. Good chance this is a case of garbage in/garbage out.

[afaji]

I always thought Marian did the preprocessing...

but I don't think unprocessed data causes the error. You might get a crappy result, though.

Now that I retry with preprocessed data, I found something interesting:

[2017-03-23 09:22:39] [config] train-sets:
[2017-03-23 09:22:39] [config]   - corpusPrep.ro
[2017-03-23 09:22:39] [config]   - corpusPrep.en
[2017-03-23 09:22:39] [config] type: dl4mt
[2017-03-23 09:22:39] [config] valid-freq: 10000
[2017-03-23 09:22:39] [config] valid-metrics:
[2017-03-23 09:22:39] [config]   - cross-entropy
[2017-03-23 09:22:39] [config] vocabs:
[2017-03-23 09:22:39] [config]   - corpus.ro.yml
[2017-03-23 09:22:39] [config]   - corpus.en.yml
[2017-03-23 09:22:39] [config] workspace: 2048

see that vocab file is corpus.ro.yml, even though my trainfile is corpusPrep.ro. In fact no matter what my train file name is, the vocab is always corpus.ro.yml. Is this intended? Maybe we load wrong vocab thus causing the crash.

[emjotde]

No, why should marian do preprocessing? No such magic yet.
I suppose you have a model.npz and a model.npz.yml file in that folder. Marian automatically reloads that.

[afaji]

I see..

so that's not the issue. Currently training the preprocessed files.

[kpu]

@afaji can you make a minimal test case (tokenized or not) that causes the segfault then check it in, intentionally breaking the build?

[emjotde]

I am running it now on the unprocessed data. So far no problems in the 3rd epoch (different machine though). I still suspect it's the driver version.

[emjotde]

So, six full epochs. no problem. You might want to check downgrading that driver...

[emjotde]

Two times 10 epochs, no segfault.