I am facing the following issue: The system cannot find the path specified but my pyspark opens up. about spark

[dbeavon]

Hi @saumyatiwari03 Sorry for the trouble.
This community is focused on .net language bindings for spark (c#) and not on general-purpose spark problems with python. It isn't a good place for normal pyspark Q&A.

I think you are on the right track with the use of "process monitor". You can see that it is making incremental searches to find the files.

Please note that there is lots of noise in process monitor. While you may be looking at something that can't be found, but it is possible that it isn't actually related to the error message. (There could be other things that cant be found as well).

Have you reviewed the installation instructions to see if there are PATH changes need in your environment? I would recommend making sure you have introduced all those changes as well. There are generally changes for both hadoop and spark.

Good luck. Please close the case, since it is not directly related to .net. It is something you could also ask on stackoverflow, perhaps.

[dbeavon]

@saumyatiwari03 What is the problem you experience? Just the error message?

Is this related to .net programming, or are you a pyspark developer? it would help to have a bit more context.

Also I'd highly recommend using something like process monitor from sysinternals to give us more information about what file is missing. The reason it shows a message saying it cannot find a file, is because it is trying to find a file with a specific name, and those details would appear if you are running process monitor as administrator.

[saumyatiwari03]

@dbeavon this issue is related to Pyspark. I tried running the process monitor and captured the events while initiating Pyspark.

[saumyatiwari03]

Hi @dbeavon, thanks for your help. Yeah, I have certainly verified the environment variables. Thanks for your suggestion.