GithubHelp home page GithubHelp logo

fai-pep's People

Contributors

agarla avatar amyreese avatar asankaran avatar ashutosh1919 avatar axitkhurana avatar bfrommfb avatar bharat123rox avatar bstocks101 avatar danthe3rd avatar digantdesai avatar haomiaoliu730 avatar harouwu avatar hl475 avatar hlu1 avatar jason-cooke avatar jspark1105 avatar jteller avatar kevinbchen avatar kimishpatel avatar lly-zero-one avatar pritamdamania avatar pritamdamania87 avatar rainbowman1 avatar rashirungta avatar sf-wind avatar stanislavglebik avatar virtan avatar xta0 avatar zhizhenqin avatar zpao avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fai-pep's Issues

Checking if the key exists in dictionary instead of using dict.get(key, None)

if "shared_libs" in info:
minfo["shared_libs"] = info["shared_libs"]

if "shared_libs" in info:
cinfo["shared_libs"] = info["shared_libs"]

Some places in this file FAI-PEP/benchmarking/driver/benchmark_driver.py has check if a key is in the dictionary and then tries to access it. It may be inefficient and less clean to go through
the dictionary twice. Can I raise a PR to change these to

minfo["shared_libs"] = info.get("shared_libs", "") 
cinfo["shared_libs"] = info.get("shared_libs", "")

What is meant by "without a battery" in top level README.md ?

In top level README one of the bullets about performance metrics says,

"energy/power : the energy per inference and average power of running the the ML model on a phone without battery"

I assume the "the the" is a typing error that should be "the".

But what about "without battery" ? If inference engine hardware in mobile phones could truly run without a battery that would be very low power indeed !! Is this a typo or if not, what is meant by this ?

-- jS

General tutorial on running FAI-PEP

It would be great if you could add a general tutorial which allows practitioners to benchmark all sorts of models. The idea behind FAI-PEP is really good but all tutorials are geared towards image-classification models.

Contribution

Sorry for being creating an issue in this repo. Are you guys working on Django ?

What should be the directory the framework repo resides for run_bench.py

I tried benchmarking/run_bench.py -b specifications/models/tflite/mobilenet_v2/mobilenet_v2_1.4_224.json but don't know what should I enter at the prompt "Please enter the directory the framework repo resides" - should it be the local PyTorch or TF Lite repo directory? An example of this?

Problems when running the experiment with docker

When trying to run the experiment with docker for TFLite example, I encountered the following error:

+ python /tmp/FAI-PEP/benchmarking/run_bench.py -b /tmp/FAI-PEP/specifications/models/tflite/mobilenet_v2/mobilenet_v2_0.35_96.json --config_dir /tmp/config
usage: run_bench.py [-h] [--app_id APP_ID] [-b BENCHMARK_FILE] [--lab]
                    [--logger_level {info,warning,error}] [--remote]
                    --root_model_dir ROOT_MODEL_DIR [--token TOKEN]
                    [-c CUSTOM_BINARY] [--pre_built_binary PRE_BUILT_BINARY]
                    [--user_string USER_STRING]
run_bench.py: error: argument --root_model_dir is require

Does anyone know how to resolve this? Did I missed anything before running the example?

Change dictionary syntax to use get

if 'x_is_date' not in kw_extra:
kw_extra['x_is_date'] = False
if 'x_axis_format' not in kw_extra:
kw_extra['x_axis_format'] = "%d %b %Y"
if 'color_category' not in kw_extra:
kw_extra['color_category'] = "category20"
if 'tag_script_js' not in kw_extra:
kw_extra['tag_script_js'] = True
if 'chart_attr' not in kw_extra:
kw_extra['chart_attr'] = {}
# set the container name

The above syntax can be changed to the following.

kw_extra['x_is_date'] = kw_extra.get('x_is_date', False)
kw_extra['x_axis_format'] = kw_extra.get('x_axis_format', "%d %b %Y")
kw_extra['color_category'] = kw_extra.get('color_category', "category20")
kw_extra['tag_script_js'] = kw_extra.get('tag_script_js', True)
kw_extra['chart_attr'] = kw_extra.get('chart_attr', {})

Failed to run benchmark scripts in Android

Hi there,

I just new to here. So I typed by following the tutorial:

benchmarking/run_bench.py -b specifications/models/caffe2/shufflenet/shufflenet.json --platforms android

After a long time compiling, all of compile and link tasks are finished in the build_android folder in my pytorch repo. But it throws an error:

cmake unknown rule to install xxxx

It looks the caffe2_benchmark executable has been generated by failed to copied to the install folder, so I manually copied to the folder, namely:
/home/new/.aibench/git/exec/caffe2/android/2019/4/5/fefa6d305ea3e820afe64cec015d2f6746d9ca88
Then I modified repo_driver.py to avoid compiling again and run function _runBenchmarkSuites
But failed :

In file included from ../third_party/zstd/lib/common/pool.h:20:0,
from ../third_party/zstd/lib/common/pool.c:14:
../third_party/zstd/lib/common/zstd_internal.h:382:37: error: unknown type name ‘ZSTD_dictMode_e’; did you mean ‘FSE_decode_t’?
ZSTD_dictMode_e dictMode,
^~~~~~~~~~~~~~~
FSE_decode_t

Questions:

  1. Any suggestions on how to run the tutorial correctly?
  2. How to avoid the long time compiling for each time running

benchmarking/run_bench.py -b specifications/models/caffe2/shufflenet/shufflenet.json --platforms android

thanks!

Failed to run shuffleNet on host

I have tried to run shufflenet and modified inputs to gpu_0/data getting following issue
INFO 12:24:56 subprocess_with_logger.py: 24: Running: /tmp/FAI-PEP/libraries/python/imagenet_test_map.py --image-dir /tmp/imagenet/val --label-file /tmp/FAI-PEP/libraries/python/labels.txt --output-image-file /tmp/tmpLwDpGk/caffe2/host/images.txt --output-label-file /tmp/tmpLwDpGk/caffe2/host/labels.txt --shuffle
INFO 12:24:56 subprocess_with_logger.py: 24: Running: awk (NR>050000/1000)&&(NR<=050000/1000+50000/1000) {print > "/tmp/tmpLwDpGk/caffe2/host/inputs/labels_0.txt"} /tmp/tmpLwDpGk/caffe2/host/labels.txt
INFO 12:24:56 subprocess_with_logger.py: 24: Running: /tmp/config/exec/caffe2/host/incremental/2019/5/23/90182a7332997fb0edf666abc4b554b83a1670d1/convert_image_to_tensor --input_image_file /tmp/tmpLwDpGk/caffe2/host/inputs/labels_0.txt --output_tensor /tmp/tmpLwDpGk/caffe2/host/images_tensor.pb --batch_size 1 --scale 256,-1 --crop 224,224 --preprocess normalize,mean,std --report_time json|Caffe2Observer
INFO 12:24:57 hdb.py: 27: push /tmp/tmpLwDpGk/caffe2/host/images_tensor.pb to /tmp/tmpJ2TNu5/6ea951fe0a41/images_tensor.pb
INFO 12:24:57 hdb.py: 27: push /tmp/config/model_cache/caffe2/shufflenet/model.pb to /tmp/tmpJ2TNu5/6ea951fe0a41/model.pb
INFO 12:24:57 hdb.py: 27: push /tmp/config/model_cache/caffe2/shufflenet/model_init.pb to /tmp/tmpJ2TNu5/6ea951fe0a41/model_init.pb
{u'softmax': u'/tmp/tmpJ2TNu5/6ea951fe0a41/output/softmax.txt'}
INFO 12:24:57 subprocess_with_logger.py: 24: Running: /tmp/tmpJ2TNu5/6ea951fe0a41/caffe2_benchmark --net /tmp/tmpJ2TNu5/6ea951fe0a41/model.pb --init_net /tmp/tmpJ2TNu5/6ea951fe0a41/model_init.pb --warmup 0 --iter 50 --input gpu_0/data --input_file /tmp/tmpJ2TNu5/6ea951fe0a41/images_tensor.pb --input_type float --output gpu_0/softmax --text_output true --output_folder /tmp/tmpJ2TNu5/6ea951fe0a41/output

/tmp/tmpLwDpGk/caffe2

/tmp/tmpLwDpGk/caffe2/output
INFO 12:25:00 hdb.py: 38: pull /tmp/tmpJ2TNu5/6ea951fe0a41/output/softmax.txt to /tmp/tmpLwDpGk/caffe2/output/softmax.txt
INFO 12:25:00 hdb.py: 40: directory /tmp/tmpJ2TNu5/6ea951fe0a41/output
INFO 12:25:00 hdb.py: 46: filenames /tmp/tmpJ2TNu5/6ea951fe0a41/output/
INFO 12:25:00 benchmark_driver.py: 64: Exception caught when running benchmark
INFO 12:25:00 benchmark_driver.py: 65: [Errno 2] No such file or directory: u'/tmp/tmpJ2TNu5/6ea951fe0a41/output/softmax.txt'
ERROR 12:25:00 benchmark_driver.py: 69: Traceback (most recent call last):

As gpu_0 is prefixed in every node in shufflenet checkpoint.

Example usage for iOS

Can you show an example on how to use the system with iOS?
Does the system compiles the .ipa that get sent to the iOS device or do we have to provide it?

UnboundLocalError: local variable 'abs_name' referenced before assignment

When I first run the command,

python ${FAI_PEP_DIR}/benchmarking/run_bench.py -b "${BENCHMARK_FILE}" --config_dir "${CONFIG_DIR}"

I encounter this message.
"UnboundLocalError: local variable 'abs_name' referenced before assignment"

However, when I ran the same command again, it disappears. I doubt .md5 autogeneration somehow broke at the first time.

[Proposal]Replace bazel build command with the binary provided from tensorflow document

bazel build command in specifications/frameworks/tflite/android/build.sh requires a lot of dependencies, like appropriate version of bazel, Android SDK and NDK. It's burdensome.

I found that the resulted binary is provided in here which is form https://www.tensorflow.org/lite/performance/measurement.

I commented out

#   --config=android_arm \
#   --cxxopt='--std=c++11' \
#   tensorflow/lite/tools/benchmark:benchmark_model

these lines and saved the downloaded binary in {tensorflow_dir}/bazel-bin/tensorflow/lite/tools/benchmark/benchmark_model

Then,

python ${FAI_PEP_DIR}/benchmarking/run_bench.py -b "${BENCHMARK_FILE}" --config_dir "${CONFIG_DIR}"

executed without problems.

Below is my configuration.

{
  \"--commit\": \"master\",
  \"--exec_dir\": \"${CONFIG_DIR}/exec\",
  \"--framework\": \"tflite\",
  \"--local_reporter\": \"${CONFIG_DIR}/reporter\",
  \"--model_cache\": \"${CONFIG_DIR}/model_cache\",
  \"--platforms\": \"android\",
  \"--remote_repository\": \"origin\",
  \"--repo\": \"git\",
  \"--repo_dir\": \"${REPO_DIR}\",
  \"--tmp_model_dir\": \"${CONFIG_DIR}/tmp_model_dir\",
  \"--root_model_dir\": \"${CONFIG_DIR}/root_model_dir\",
  \"--screen_reporter\": null
}

What's the magic behind "--platform android"

Hello, I'm trying to get benchmarks from my phone running MobilenetV2.

I was reading this document and got one question.

Just passing "--platform=android" is enough to run benchmark binaries? How does this tell where's my phone to the executed binary file? I thought I should set up some "adb" kind of things, but I couldn't find any mention of "adb". Now I am wondering if FAI-PEP has some android simulators that runs binary..

Could you tell me where should I refer to if I want to run FAI-PEP on the phone I connect?

When running multiple tests, failures can be "swallowed"

We run multiple tests from a single config, and introduced an error in the second test in the group. This failure was not noticed, because each test reuses the same temp directory so report.json from the previous run is reused. That means that even though the run fails, we report data for the previous run.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.