tensorboard_neuron seems to not take --run_neuron_profile about aws-neuron-sdk

[micwade-aws]

Hi, mbuckler.

Tensorboard needs the logdir arg passed on each run. AWS Neuron is going to fill the logdir directory with results from profiling a single inference call. We will update the tutorial to be more explicit about this flow and clarify documentation around installation.

Regarding the error you’re seeing:
tensorboard: error: unrecognized arguments: --run_neuron_profile

We’re only able to reproduce it when installing tensorboard over tensorboard-neuron, like this:

pip install tensorboard-neuron
pip install tensorboard --force-reinstall
tensorboard_neuron --run_neuron_profile

Are you doing something similar? If not, would you mind sending some more details on how you’re installing tensorboard-neuron and tensorboard to help us get it recreated?

Thanks,
Mike

[mbuckler]

Good to know about the need for logdir, I've added that in to my command but unfortunately the error persists. Since you've asked for my installation setup I've attached a full log of my install and run process to this message. Below is also a summary of my process:

0. Start with a fresh Ubuntu 18.04 DLAMI on an inf1.6xlarge instance
1. Source the appropriate file to use the DLAMI's python environment
2. Install tensorflow-neuron & aws-neuron-tools
3. Create directories for logdir and neuron_profile
4. Run inference (in this case for tensorflow Resnet50 as in the getting started tutorial) and observe output inside the neuron_profile directory
5. Attempt to run tensorboard_neuron and receive the unrecognized arguments error

tensorflow-neuron_log.txt

[micwade-aws]

Thanks for providing the logs to help us reproduce this issue.

When using the DLAMI v26.0 Conda python environment (aws_neuron_tensorflow_p36), tensorflow-neuron and tensorboard-neuron are already installed. We have identified that the version of tensorflow-neuron in this environment incorrectly has a dependency on tensorboard rather than tensorboard-neuron, and has been fixed in newer released versions.

We’ve updated the DLAMI release notes (https://github.com/aws/private-aws-neuron-sdk-staging/blob/master/release-notes/dlami-release-notes.md) and TensorBoard-Neuron guide (https://github.com/aws/aws-neuron-sdk/blob/master/docs/neuron-tools/getting-started-tensorboard-neuron.md) to be explicit about install and updates of TensorBoard-Neuron for DLAMI v26.0.

The quickest way to get a working TensorBoard-Neuron in the DLAMI v26.0 Conda python environment (aws_neuron_tensorflow_p36) is to run these commands from inside the environment:

conda install numpy=1.17.2 --yes --quiet
conda update tensorflow-neuron