can not run cntk binary about cntk

[frankseide]

Could you share the full command line, and any log output before the crash? Or does it crash immediately?

From: iamkkk [mailto:[email protected]]
Sent: Thursday, January 28, 2016 16:14
To: Microsoft/CNTK [email protected]
Subject: [CNTK] can not run cntk binary (#52)

hi, I am trying the cntk binary file and when I try to run it on gpu, I encounter the the following error message:" CNTK: Win32 exception caught (such an access violation or a stack overflow"
I am using nvidia k5000 and cuda70 on Windows 10
Does anyone know what is the problem?
thanks a lot

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/52.

I run"cntk configFile=../Config/FeedForward.config" and the program crashes immediatly. no log output either. The command prompt only print build infom,then "MPIWrapper: initializing MPI" then"CNTK: Win32 exception caught (such an access violation or a stack overflow)"
I tried to run simple2d on cpu and It worked fine.

[frankseide]

Could you try with a non-existent config file? (change the path to a file that does not exist) and let me know whether it still crashes? In that case, it is a setup error, I would bet a missing delay-loaded DLL.

From: iamkkk [mailto:[email protected]]
Sent: Thursday, January 28, 2016 16:45
To: Microsoft/CNTK [email protected]
Cc: Frank Seide [email protected]
Subject: Re: [CNTK] can not run cntk binary (#52)

I run"cntk configFile=../Config/FeedForward.config" and the program crashes immediatly. no log output either.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/52#issuecomment-176498646.

the following is erroe message when i try a non-existent config file

Build info:

            Built time: Jan 26 2016 04:34:07
            Last modified date: Mon Jan 25 09:01:43 2016
            CUDA_PATH: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v7.0
            CUB_PATH: C:\src\cub-1.4.1
            CUDNN_PATH: C:\NVIDIA\cudnn-4.0\cuda
            Build Branch: HEAD
            Build SHA1: 56a2a15f64676ea4c0e0a0a681a57b19a46f64c6
            Built by svcphil on DPHAIM-22
            Build Path: c:\jenkins\workspace\CNTK-Build-Windows\Source\CNTK\

---

[CALL STACK]
>Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::BaseMatrix::GetSizeAllocated
-BaseThreadInitThunk
-RtlUserThreadStart

attempt: error opening file '../Config/FeedForwar1d.config': No such file or directory, retrying 2-th time out of 5...

[CALL STACK]
>Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::BaseMatrix::GetSizeAllocated
-BaseThreadInitThunk
-RtlUserThreadStart

attempt: error opening file '../Config/FeedForwar1d.config': No such file or directory, retrying 3-th time out of 5...

[CALL STACK]
>Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::BaseMatrix::GetSizeAllocated
-BaseThreadInitThunk
-RtlUserThreadStart

attempt: error opening file '../Config/FeedForwar1d.config': No such file or directory, retrying 4-th time out of 5...

[CALL STACK]
>Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::BaseMatrix::GetSizeAllocated
-BaseThreadInitThunk
-RtlUserThreadStart

attempt: error opening file '../Config/FeedForwar1d.config': No such file or directory, retrying 5-th time out of 5...

[CALL STACK]
>Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::GPUMatrix::AssignNCEUnnormalizedEval
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::MatrixBase::operator=
-Microsoft::MSR::CNTK::BaseMatrix::GetSizeAllocated
-BaseThreadInitThunk
-RtlUserThreadStart

EXCEPTION occurred: error opening file '../Config/FeedForwar1d.config': No such file or directory

Usage: cntk configFile=yourConfigFile
For detailed information please consult the CNTK book
"An Introduction to Computational Networks and the Computational Network Toolkit"

[haixinguan]

I have the same problem, but some differences.
I run the samples for speech as the readme and get the info,"MPIWrapper: initializing MPI, CNTK: Win32 exception caught (such an access violation or a stack overflow)", the enviroment is Win 7 64bit, only CPU, and had installed the cuda7.0 toolkit, and also vs2013.
Are there some other setup should be do?
Thanks.

[frankseide]

I will see if I can add a more clear error message. I suspect this is due to a missing DLL that is being delay-loaded, likely a missing MPI runtime.

From: haixinguan [mailto:[email protected]]
Sent: Wednesday, February 3, 2016 19:07
To: Microsoft/CNTK [email protected]
Cc: Frank Seide [email protected]
Subject: Re: [CNTK] can not run cntk binary (#52)

I have the same problem, but some differences.
I run the samples for speech as the readme and get the info,"MPIWrapper: initializing MPI, CNTK: Win32 exception caught (such an access violation or a stack overflow)", the enviroment is Win 7 64bit, only GPU, and had installed the cuda7.0 toolkit, and also vs2013.
Are there some other setup should be do?
Thanks.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/52#issuecomment-179592364.

[frankseide]

A fix for MPIWrapper has been put into master. Haixin, iamkkk, would you mind trying again?

Haixin should get a proper error message saying that MPI is not installed. For iamkkk, I would like to know whether this is the same issue.

I am using the binary file(https://github.com/Microsoft/CNTK/releases). it seems that that file has not been updated yet.
p.s.I am pretty sure that I installed MPI

[frankseide]

I just added more detailed error reporting for Win32 exceptions. Will land soon, but only in source for now.

From: iamkkk [mailto:[email protected]]
Sent: Thursday, February 4, 2016 15:45
To: Microsoft/CNTK [email protected]
Cc: Frank Seide [email protected]
Subject: Re: [CNTK] can not run cntk binary (#52)

I am using the binary file(https://github.com/Microsoft/CNTK/releases). it seems that that file has not been updated yet.
p.s.I am pretty sure that I installed MPI

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/52#issuecomment-180106222.

[raminia]

I get the same error and I noticed that MSMPI.DLL is not accessible by the binary even though MS MPI installation is finished successfully.

[frankseide]

I fixed something a few days ago—the actual initialization of MPI was guarded by a catch of a failed delay-load, but there was a check outside whether MPI was already initialized, which was not caught.

If your version is older than that commit, would you mind trying whether latest master still have this problem?

commit 1e65a41
Author: Frank Seide [email protected]
Date: Thu Feb 4 08:51:08 2016 -0800

fixed bug in catching when MPI runtime is missing

From: raminia [mailto:[email protected]]
Sent: Monday, February 8, 2016 15:13
To: Microsoft/CNTK [email protected]
Cc: Frank Seide [email protected]
Subject: Re: [CNTK] can not run cntk binary (#52)

I get the same error and I noticed that MSMPI.DLL is not accessible by the binary even though MS MPI installation is finished successfully.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/52#issuecomment-181614628.

[raminia]

Will do. However, in the process of getting dependencies ready to build CNTK I realized that I've missed an important step which is installing both _SDK _*and * _runtime _ versions of the MS-MPI from the link. That means installing both .exe and .msi files. After doing so CNTK started normally.
However, when testing AN4 dataset example, I landed on another exception (below) which I believe is a separate issue but it emphasizes that I need to build CNTK for better customization.

EXCEPTION occurred: Gradient quantization is unsupported in CNTK binaries built without quantized gradient aggregation support!

[frankseide]

This is related to the 1-bit SGD method. You will need to get that through a separate submodule.

https://github.com/Microsoft/CNTK/wiki/Enabling-1bit-SGD

From: raminia [mailto:[email protected]]
Sent: Monday, February 8, 2016 16:23
To: Microsoft/CNTK [email protected]
Cc: Frank Seide [email protected]
Subject: Re: [CNTK] can not run cntk binary (#52)

Will do. However, in the process of getting dependencies ready to build CNTK I realized that I've missed an important step which is installing both _SDK _*and * _runtime _ versions of the MS-MPI from the linkhttps://na01.safelinks.protection.outlook.com/?url=https%3a%2f%2fmsdn.microsoft.com%2fen-us%2flibrary%2fbb524831(v%3dvs.85).aspx&data=01%7c01%7cfseide%40microsoft.com%7c209132d7e71840146adc08d330e724bf%7c72f988bf86f141af91ab2d7cd011db47%7c1&sdata=xryYx%2bziUnS%2fkoquK0j8x4tUMvxcVuAea8iNKn%2ftZcs%3d. That means installing both .exe and .msi files. After doing so CNTK started normally.
However, when testing AN4 dataset example, I landed on another exception (below) which I believe is a separate issue but it emphasizes that I need to build CNTK for better customization.

EXCEPTION occurred: Gradient quantization is unsupported in CNTK binaries built without quantized gradient aggregation support!

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/52#issuecomment-181637731.

[mahilleb-msft]

@iamkkk - are you experiencing issues to run the CNTK binary? Please let us know, otherwise we'll close this isuse...

I downloaded CNTK-2016-02-08-Windows-64bit-GPU.zip and got the error "mpihelper: msmpi.dll missing
mpiaggregator: MPI_Init, MPI error 16

[alexeyo26]

Hello!

You need to have MPI installed: https://msdn.microsoft.com/en-us/library/bb524831(v=vs.85).aspx

Tell us, whether it fixes it.
Tank you,
Alexey

hi, the following is the command I typed:D:\cntk\Examples\Other\Simple2d\Data>cntk configFile=../Config/Multigpu.cntk

Build info:

            Built time: Feb  8 2016 00:54:07
            Last modified date: Sun Feb  7 16:51:01 2016
            Build type: Unknown
            Build target: Unknown
            With 1bit-SGD: no
            CUDA_PATH: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v7.0
            CUB_PATH: C:\src\cub-1.4.1
            CUDNN_PATH: C:\NVIDIA\cudnn-4.0\cuda
            Build Branch: HEAD
            Build SHA1: 2f9a48c71dc0a6097498cb7e90ac3b151ab536dd
            Built by svcphil on LIANA-09-w
            Build Path: c:\jenkins\workspace\CNTK-Build-Windows\Source\CNTK\

---

MPIWrapper: initializing MPI
mpihelper: msmpi.dll missing
mpiaggregator: MPI_Init, MPI error 16

[CALL STACK]
>Microsoft::MSR::CNTK::TracingGPUMemoryAllocator::operator=
-Microsoft::MSR::CNTK::TracingGPUMemoryAllocator::operator=
-Microsoft::MSR::CNTK::TracingGPUMemoryAllocator::operator=
-Microsoft::MSR::CNTK::TracingGPUMemoryAllocator::operator=
-Microsoft::MSR::CNTK::TracingGPUMemoryAllocator::operator=
-Microsoft::MSR::CNTK::TracingGPUMemoryAllocator::operator=
-Microsoft::MSR::CNTK::TracingGPUMemoryAllocator::operator=
-BaseThreadInitThunk
-RtlUserThreadStart

EXCEPTION occurred: mpiaggregator: MPI_Init

Usage: cntk configFile=yourConfigFile
For detailed information please consult the CNTK book
"An Introduction to Computational Networks and the Computational Network Toolkit"

[alexeyo26]

Sorry for asking again - do you have MS-MPI installed? (see my previous post)

I installed Microsoft MPI SDK(7.0.12437.6). there are 2 files in the download website. I downloaded and installed the .msi file, is it ok? Or should I install the MSMPISetup.exe

[alexeyo26]

Ah, you need run time. (Same download page, file MSMpiSetup.exe)

it's working! thanks a lot!

[alexeyo26]

Great! I will make binary setup instructions clearer. Thank you. I will close the request now.