Insufficient documentation about profiler-ui

[ChrisAntaki]

Installation step 3 has been rewritten based on what you shared in this bug, I hope it's clearer now, thanks

[ChrisAntaki]

So what if you ran this command instead?

python ui.py --profile_context_path=/path/to/your/profile.context

Where ui.py is the file from this repository (tensorflow/profiler-ui).

[ChrisAntaki]

Hey Antonio, sorry to hear you're running into issues. Are you running into any specific errors that could be documented?

[AntonioSerrano]

No, unfortunately no. My problem is that I do not know how to run this line: bazel-bin/tensorflow/core/profiler/profiler
--profile_path=/tmp/train_dir/profile_xx`. More specifically, I do not know how to create a profile context with the provided doc. Any clue?

[ChrisAntaki]

Ah, I see how those instructions could be confusing. In the instructions you linked, check out the code underneath this comment: "# Create options to profile the time and memory information." That will show you how to create a profile context that can be displayed in the UI.

[AntonioSerrano]

I already did it. If I execute my testing script from the terminal (python my_script.py --profile_context_path=my/path/to/my/profile.context), I get the expected report. In addition, a file called "profile_20" is generated in that path. But how do enter in the tfprof mode in the terminal? I cannot find executable bazel-bin/tensorflow/core/profiler/profiler. Please, assume that I have not idea about Bazel and that I am on a Mac.

[ChrisAntaki]

Now that you have a profile file, you can continue to Installation step 4: python ui.py --profile_context_path=/path/to/your/profile.context

[AntonioSerrano]

Yes, it is clearer now. Thanks Chris. Following those instructions, however, I only get the report in the CLI. But how can I visualize the profiler on my web browser?

[ChrisAntaki]

Glad to hear that, Antonio. Now that this repository's (tensorflow/profiler-ui) instructions have been clarified, let's put the other repository's (tensorflow/tensorflow) instructions aside for now, as they focus on the CLI.

The next step should be running this command from Step 4: python ui.py --profile_context_path=/path/to/your/profile.context. When you run that command, after updating the profile_context_path parameter, does a browser window open?

[AntonioSerrano]

No, it doesn't. Should it open automatically? When using TensorBoard, I have to open it manually from Chrome via localhost. I am using Chrome on a Mac, BTW.

[ChrisAntaki]

Yes, it should be opening automatically. Could I ask what's printing in your terminal after you run that command?

[AntonioSerrano]

Sure. There you go:

25 ops no flops stats due to incomplete shapes.
Parsing Inputs...
Incomplete shape.
Incomplete shape.
Incomplete shape.
Incomplete shape.
Incomplete shape.
Incomplete shape.

=========================Options=============================
-max_depth 10000
-min_bytes 1
-min_peak_bytes 0
-min_residual_bytes 0
-min_output_bytes 0
-min_micros 1
-min_accelerator_micros 0
-min_cpu_micros 0
-min_params 0
-min_float_ops 0
-min_occurrence 0
-step -1
-order_by micros
-account_type_regexes .*
-start_name_regexes .*
-trim_name_regexes
-show_name_regexes .*
-hide_name_regexes
-account_displayed_op_only true
-select bytes,micros
-output stdout:

==================Model Analysis Report======================
Incomplete shape.
Incomplete shape.
Incomplete shape.
Incomplete shape.
Incomplete shape.
Incomplete shape.

Doc:
op: The nodes are operation kernel type, such as MatMul, Conv2D. Graph nodes belonging to the same type are aggregated together.
requested bytes: The memory requested by the operation, accumulatively.
total execution time: Sum of accelerator execution time and cpu execution time.
cpu execution time: The time from the start to the end of the operation. It's the sum of actual cpu run time plus the time that it spends waiting if part of computation is launched asynchronously.
accelerator execution time: Time spent executing on the accelerator. This is normally measured by the actual hardware library.

Profile:
node name | requested bytes | total execution time | accelerator execution time | cpu execution time
MatMul 348.96KB (100.00%, 86.01%), 335us (100.00%, 63.57%), 0us (0.00%, 0.00%), 335us (100.00%, 63.57%)
Tile 4.40KB (13.99%, 1.08%), 35us (36.43%, 6.64%), 0us (0.00%, 0.00%), 35us (36.43%, 6.64%)
Softmax 0B (0.00%, 0.00%), 24us (29.79%, 4.55%), 0us (0.00%, 0.00%), 24us (29.79%, 4.55%)
Sum 840B (12.90%, 0.21%), 15us (25.24%, 2.85%), 0us (0.00%, 0.00%), 15us (25.24%, 2.85%)
VariableV2 31.40KB (12.70%, 7.74%), 12us (22.39%, 2.28%), 0us (0.00%, 0.00%), 12us (22.39%, 2.28%)
Mul 12.00KB (4.96%, 2.96%), 11us (20.11%, 2.09%), 0us (0.00%, 0.00%), 11us (20.11%, 2.09%)
Sub 0B (0.00%, 0.00%), 11us (18.03%, 2.09%), 0us (0.00%, 0.00%), 11us (18.03%, 2.09%)
Add 0B (0.00%, 0.00%), 10us (15.94%, 1.90%), 0us (0.00%, 0.00%), 10us (15.94%, 1.90%)
Shape 36B (2.00%, 0.01%), 10us (14.04%, 1.90%), 0us (0.00%, 0.00%), 10us (14.04%, 1.90%)
ApplyGradientDescent 0B (0.00%, 0.00%), 8us (12.14%, 1.52%), 0us (0.00%, 0.00%), 8us (12.14%, 1.52%)
Const 24B (1.99%, 0.01%), 7us (10.63%, 1.33%), 0us (0.00%, 0.00%), 7us (10.63%, 1.33%)
Identity 0B (0.00%, 0.00%), 6us (9.30%, 1.14%), 0us (0.00%, 0.00%), 6us (9.30%, 1.14%)
Reshape 0B (0.00%, 0.00%), 6us (8.16%, 1.14%), 0us (0.00%, 0.00%), 6us (8.16%, 1.14%)
BroadcastGradientArgs 4B (1.98%, 0.00%), 4us (7.02%, 0.76%), 0us (0.00%, 0.00%), 4us (7.02%, 0.76%)
DynamicStitch 8B (1.98%, 0.00%), 4us (6.26%, 0.76%), 0us (0.00%, 0.00%), 4us (6.26%, 0.76%)
NoOp 0B (0.00%, 0.00%), 4us (5.50%, 0.76%), 0us (0.00%, 0.00%), 4us (5.50%, 0.76%)
Cast 4B (1.98%, 0.00%), 3us (4.74%, 0.57%), 0us (0.00%, 0.00%), 3us (4.74%, 0.57%)
FloorDiv 3B (1.98%, 0.00%), 3us (4.17%, 0.57%), 0us (0.00%, 0.00%), 3us (4.17%, 0.57%)
Log 4.00KB (1.98%, 0.99%), 3us (3.61%, 0.57%), 0us (0.00%, 0.00%), 3us (3.61%, 0.57%)
_arg_Placeholder_1_0_1 0B (0.00%, 0.00%), 2us (3.04%, 0.38%), 0us (0.00%, 0.00%), 2us (3.04%, 0.38%)
Maximum 8B (0.99%, 0.00%), 2us (2.66%, 0.38%), 0us (0.00%, 0.00%), 2us (2.66%, 0.38%)
Neg 0B (0.00%, 0.00%), 2us (2.28%, 0.38%), 0us (0.00%, 0.00%), 2us (2.28%, 0.38%)
Prod 0B (0.00%, 0.00%), 2us (1.90%, 0.38%), 0us (0.00%, 0.00%), 2us (1.90%, 0.38%)
Reciprocal 4.00KB (0.99%, 0.99%), 2us (1.52%, 0.38%), 0us (0.00%, 0.00%), 2us (1.52%, 0.38%)
RealDiv 0B (0.00%, 0.00%), 1us (1.14%, 0.19%), 0us (0.00%, 0.00%), 1us (1.14%, 0.19%)
gradients/Sum_grad/range/_4__cf__5 8B (0.00%, 0.00%), 1us (0.95%, 0.19%), 0us (0.00%, 0.00%), 1us (0.95%, 0.19%)
_arg_Placeholder_0_0 0B (0.00%, 0.00%), 1us (0.76%, 0.19%), 0us (0.00%, 0.00%), 1us (0.76%, 0.19%)
gradients/Mean_grad/Maximum/_0__cf__1 4B (0.00%, 0.00%), 1us (0.57%, 0.19%), 0us (0.00%, 0.00%), 1us (0.57%, 0.19%)
gradients/Mean_grad/Reshape/_1__cf__2 4B (0.00%, 0.00%), 1us (0.38%, 0.19%), 0us (0.00%, 0.00%), 1us (0.38%, 0.19%)
gradients/Sum_grad/mod/_3__cf__4 4B (0.00%, 0.00%), 1us (0.19%, 0.19%), 0us (0.00%, 0.00%), 1us (0.19%, 0.19%)

======================End of Report==========================

=========================Options=============================
-max_depth 10000
-min_bytes 1
-min_peak_bytes 0
-min_residual_bytes 0
-min_output_bytes 0
-min_micros 1
-min_accelerator_micros 0
-min_cpu_micros 0
-min_params 0
-min_float_ops 0
-min_occurrence 0
-step -1
-order_by micros
-account_type_regexes .*
-start_name_regexes .*
-trim_name_regexes
-show_name_regexes .*
-hide_name_regexes
-account_displayed_op_only true
-select bytes,micros
-output stdout:

==================Model Analysis Report======================
Incomplete shape.
Incomplete shape.
Incomplete shape.
Incomplete shape.
Incomplete shape.
Incomplete shape.

Doc:
op: The nodes are operation kernel type, such as MatMul, Conv2D. Graph nodes belonging to the same type are aggregated together.
requested bytes: The memory requested by the operation, accumulatively.
total execution time: Sum of accelerator execution time and cpu execution time.
cpu execution time: The time from the start to the end of the operation. It's the sum of actual cpu run time plus the time that it spends waiting if part of computation is launched asynchronously.
accelerator execution time: Time spent executing on the accelerator. This is normally measured by the actual hardware library.

Profile:
node name | requested bytes | total execution time | accelerator execution time | cpu execution time
MatMul 348.96KB (100.00%, 86.01%), 343us (100.00%, 63.64%), 0us (0.00%, 0.00%), 343us (100.00%, 63.64%)
Tile 4.40KB (13.99%, 1.08%), 38us (36.36%, 7.05%), 0us (0.00%, 0.00%), 38us (36.36%, 7.05%)
Softmax 0B (0.00%, 0.00%), 24us (29.31%, 4.45%), 0us (0.00%, 0.00%), 24us (29.31%, 4.45%)
Sum 840B (12.90%, 0.21%), 15us (24.86%, 2.78%), 0us (0.00%, 0.00%), 15us (24.86%, 2.78%)
VariableV2 31.40KB (12.70%, 7.74%), 12us (22.08%, 2.23%), 0us (0.00%, 0.00%), 12us (22.08%, 2.23%)
Mul 12.00KB (4.96%, 2.96%), 11us (19.85%, 2.04%), 0us (0.00%, 0.00%), 11us (19.85%, 2.04%)
Sub 0B (0.00%, 0.00%), 11us (17.81%, 2.04%), 0us (0.00%, 0.00%), 11us (17.81%, 2.04%)
Add 0B (0.00%, 0.00%), 10us (15.77%, 1.86%), 0us (0.00%, 0.00%), 10us (15.77%, 1.86%)
Shape 36B (2.00%, 0.01%), 9us (13.91%, 1.67%), 0us (0.00%, 0.00%), 9us (13.91%, 1.67%)
ApplyGradientDescent 0B (0.00%, 0.00%), 7us (12.24%, 1.30%), 0us (0.00%, 0.00%), 7us (12.24%, 1.30%)
Cast 4B (1.99%, 0.00%), 6us (10.95%, 1.11%), 0us (0.00%, 0.00%), 6us (10.95%, 1.11%)
Const 24B (1.99%, 0.01%), 6us (9.83%, 1.11%), 0us (0.00%, 0.00%), 6us (9.83%, 1.11%)
Identity 0B (0.00%, 0.00%), 6us (8.72%, 1.11%), 0us (0.00%, 0.00%), 6us (8.72%, 1.11%)
Reshape 0B (0.00%, 0.00%), 6us (7.61%, 1.11%), 0us (0.00%, 0.00%), 6us (7.61%, 1.11%)
NoOp 0B (0.00%, 0.00%), 5us (6.49%, 0.93%), 0us (0.00%, 0.00%), 5us (6.49%, 0.93%)
BroadcastGradientArgs 4B (1.98%, 0.00%), 4us (5.57%, 0.74%), 0us (0.00%, 0.00%), 4us (5.57%, 0.74%)
DynamicStitch 8B (1.98%, 0.00%), 4us (4.82%, 0.74%), 0us (0.00%, 0.00%), 4us (4.82%, 0.74%)
FloorDiv 3B (1.98%, 0.00%), 3us (4.08%, 0.56%), 0us (0.00%, 0.00%), 3us (4.08%, 0.56%)
Log 4.00KB (1.98%, 0.99%), 3us (3.53%, 0.56%), 0us (0.00%, 0.00%), 3us (3.53%, 0.56%)
_arg_Placeholder_1_0_1 0B (0.00%, 0.00%), 2us (2.97%, 0.37%), 0us (0.00%, 0.00%), 2us (2.97%, 0.37%)
Maximum 8B (0.99%, 0.00%), 2us (2.60%, 0.37%), 0us (0.00%, 0.00%), 2us (2.60%, 0.37%)
Neg 0B (0.00%, 0.00%), 2us (2.23%, 0.37%), 0us (0.00%, 0.00%), 2us (2.23%, 0.37%)
RealDiv 0B (0.00%, 0.00%), 2us (1.86%, 0.37%), 0us (0.00%, 0.00%), 2us (1.86%, 0.37%)
Reciprocal 4.00KB (0.99%, 0.99%), 2us (1.48%, 0.37%), 0us (0.00%, 0.00%), 2us (1.48%, 0.37%)
gradients/Sum_grad/range/_4__cf__5 8B (0.00%, 0.00%), 1us (1.11%, 0.19%), 0us (0.00%, 0.00%), 1us (1.11%, 0.19%)
Prod 0B (0.00%, 0.00%), 1us (0.93%, 0.19%), 0us (0.00%, 0.00%), 1us (0.93%, 0.19%)
_arg_Placeholder_0_0 0B (0.00%, 0.00%), 1us (0.74%, 0.19%), 0us (0.00%, 0.00%), 1us (0.74%, 0.19%)
gradients/Mean_grad/Maximum/_0__cf__1 4B (0.00%, 0.00%), 1us (0.56%, 0.19%), 0us (0.00%, 0.00%), 1us (0.56%, 0.19%)
gradients/Mean_grad/Reshape/_1__cf__2 4B (0.00%, 0.00%), 1us (0.37%, 0.19%), 0us (0.00%, 0.00%), 1us (0.37%, 0.19%)
gradients/Sum_grad/mod/_3__cf__4 4B (0.00%, 0.00%), 1us (0.19%, 0.19%), 0us (0.00%, 0.00%), 1us (0.19%, 0.19%)

======================End of Report==========================

=========================Options=============================
-max_depth 10000
-min_bytes 1
-min_peak_bytes 0
-min_residual_bytes 0
-min_output_bytes 0
-min_micros 1
-min_accelerator_micros 0
-min_cpu_micros 0
-min_params 0
-min_float_ops 0
-min_occurrence 0
-step -1
-order_by micros
-account_type_regexes .*
-start_name_regexes .*
-trim_name_regexes
-show_name_regexes .*
-hide_name_regexes
-account_displayed_op_only true
-select bytes,micros
-output stdout:

==================Model Analysis Report======================
Incomplete shape.
Incomplete shape.
Incomplete shape.
Incomplete shape.
Incomplete shape.
Incomplete shape.

Doc:
op: The nodes are operation kernel type, such as MatMul, Conv2D. Graph nodes belonging to the same type are aggregated together.
requested bytes: The memory requested by the operation, accumulatively.
total execution time: Sum of accelerator execution time and cpu execution time.
cpu execution time: The time from the start to the end of the operation. It's the sum of actual cpu run time plus the time that it spends waiting if part of computation is launched asynchronously.
accelerator execution time: Time spent executing on the accelerator. This is normally measured by the actual hardware library.

Profile:
node name | requested bytes | total execution time | accelerator execution time | cpu execution time
MatMul 348.96KB (100.00%, 86.01%), 337us (100.00%, 62.87%), 0us (0.00%, 0.00%), 337us (100.00%, 62.87%)
Tile 4.40KB (13.99%, 1.08%), 38us (37.13%, 7.09%), 0us (0.00%, 0.00%), 38us (37.13%, 7.09%)
Softmax 0B (0.00%, 0.00%), 23us (30.04%, 4.29%), 0us (0.00%, 0.00%), 23us (30.04%, 4.29%)
Sum 840B (12.90%, 0.21%), 16us (25.75%, 2.99%), 0us (0.00%, 0.00%), 16us (25.75%, 2.99%)
VariableV2 31.40KB (12.70%, 7.74%), 14us (22.76%, 2.61%), 0us (0.00%, 0.00%), 14us (22.76%, 2.61%)
Mul 12.00KB (4.96%, 2.96%), 11us (20.15%, 2.05%), 0us (0.00%, 0.00%), 11us (20.15%, 2.05%)
Sub 0B (0.00%, 0.00%), 11us (18.10%, 2.05%), 0us (0.00%, 0.00%), 11us (18.10%, 2.05%)
Add 0B (0.00%, 0.00%), 10us (16.04%, 1.87%), 0us (0.00%, 0.00%), 10us (16.04%, 1.87%)
Shape 36B (2.00%, 0.01%), 9us (14.18%, 1.68%), 0us (0.00%, 0.00%), 9us (14.18%, 1.68%)
ApplyGradientDescent 0B (0.00%, 0.00%), 7us (12.50%, 1.31%), 0us (0.00%, 0.00%), 7us (12.50%, 1.31%)
Const 24B (1.99%, 0.01%), 6us (11.19%, 1.12%), 0us (0.00%, 0.00%), 6us (11.19%, 1.12%)
Identity 0B (0.00%, 0.00%), 6us (10.07%, 1.12%), 0us (0.00%, 0.00%), 6us (10.07%, 1.12%)
Reshape 0B (0.00%, 0.00%), 6us (8.96%, 1.12%), 0us (0.00%, 0.00%), 6us (8.96%, 1.12%)
BroadcastGradientArgs 4B (1.98%, 0.00%), 5us (7.84%, 0.93%), 0us (0.00%, 0.00%), 5us (7.84%, 0.93%)
Cast 4B (1.98%, 0.00%), 5us (6.90%, 0.93%), 0us (0.00%, 0.00%), 5us (6.90%, 0.93%)
DynamicStitch 8B (1.98%, 0.00%), 5us (5.97%, 0.93%), 0us (0.00%, 0.00%), 5us (5.97%, 0.93%)
NoOp 0B (0.00%, 0.00%), 5us (5.04%, 0.93%), 0us (0.00%, 0.00%), 5us (5.04%, 0.93%)
FloorDiv 3B (1.98%, 0.00%), 3us (4.10%, 0.56%), 0us (0.00%, 0.00%), 3us (4.10%, 0.56%)
Log 4.00KB (1.98%, 0.99%), 3us (3.54%, 0.56%), 0us (0.00%, 0.00%), 3us (3.54%, 0.56%)
_arg_Placeholder_1_0_1 0B (0.00%, 0.00%), 2us (2.99%, 0.37%), 0us (0.00%, 0.00%), 2us (2.99%, 0.37%)
Maximum 8B (0.99%, 0.00%), 2us (2.61%, 0.37%), 0us (0.00%, 0.00%), 2us (2.61%, 0.37%)
Neg 0B (0.00%, 0.00%), 2us (2.24%, 0.37%), 0us (0.00%, 0.00%), 2us (2.24%, 0.37%)
RealDiv 0B (0.00%, 0.00%), 2us (1.87%, 0.37%), 0us (0.00%, 0.00%), 2us (1.87%, 0.37%)
Reciprocal 4.00KB (0.99%, 0.99%), 2us (1.49%, 0.37%), 0us (0.00%, 0.00%), 2us (1.49%, 0.37%)
gradients/Sum_grad/range/_4__cf__5 8B (0.00%, 0.00%), 1us (1.12%, 0.19%), 0us (0.00%, 0.00%), 1us (1.12%, 0.19%)
Prod 0B (0.00%, 0.00%), 1us (0.93%, 0.19%), 0us (0.00%, 0.00%), 1us (0.93%, 0.19%)
_arg_Placeholder_0_0 0B (0.00%, 0.00%), 1us (0.75%, 0.19%), 0us (0.00%, 0.00%), 1us (0.75%, 0.19%)
gradients/Mean_grad/Maximum/_0__cf__1 4B (0.00%, 0.00%), 1us (0.56%, 0.19%), 0us (0.00%, 0.00%), 1us (0.56%, 0.19%)
gradients/Mean_grad/Reshape/_1__cf__2 4B (0.00%, 0.00%), 1us (0.37%, 0.19%), 0us (0.00%, 0.00%), 1us (0.37%, 0.19%)
gradients/Sum_grad/mod/_3__cf__4 4B (0.00%, 0.00%), 1us (0.19%, 0.19%), 0us (0.00%, 0.00%), 1us (0.19%, 0.19%)

======================End of Report==========================
Incomplete shape.
Incomplete shape.

=========================Options=============================
-max_depth 10000
-min_bytes 0
-min_peak_bytes 0
-min_residual_bytes 0
-min_output_bytes 0
-min_micros 0
-min_accelerator_micros 0
-min_cpu_micros 0
-min_params 0
-min_float_ops 0
-min_occurrence 0
-step -1
-order_by name
-account_type_regexes _trainable_variables
-start_name_regexes .*
-trim_name_regexes
-show_name_regexes .*
-hide_name_regexes
-account_displayed_op_only true
-select params
-output stdout:

==================Model Analysis Report======================
Incomplete shape.
Incomplete shape.

Doc:
scope: The nodes in the model graph are organized by their names, which is hierarchical like filesystem.
param: Number of parameters (in the Variable).

Profile:
node name | # parameters
_TFProfRoot (--/7.85k params)
Variable (784x10, 7.84k/7.84k params)
Variable_1 (10, 10/10 params)

======================End of Report==========================

Model accuracy is: 0.917

[ChrisAntaki]

Interesting, could you send the command that generated this?

[AntonioSerrano]

python ts_tutorial_2_profiler_TFProf.py

[AntonioSerrano]

Finally! Now I get it. So this is procedure (please, correct me if I am wrong):

1. Obviously, you should have your tensorflow model in a Python script e.g. my_model.py.
2. Configure the profile context by adding the tf.contrib.tfprof.ProfileContext class to my_model.py before the training loop or when session object is available, as described in here.
3. Generate the profile context file by executing python my_model.py. When configuring the profile context in the previous step, you should have indicated a path to save the profile context file (this file has no extension).
4. Assuming you downloaded this repository (tensorflow/profiler-ui) and installed all dependencies (i.e. tensorflow, flask, and pprof), now go to the repository path and run python ui.py --profile_context_path=/path/to/your/profile.context. Do not forget to change the path to the one where you saved the file in step 3. Then, the graphical interface on your web browser should open automatically.

Thank you very much ChrisAntak. Nevertheless, I suggest that you add a brief model (e.g. MNIST for beginners) with the profile context already configured in it and a thorough step-by-step description of how to enter into the GUI of the profiler. It would help a lot.

[ChrisAntaki]

Glad to hear it's working for you! And thanks for those ideas.