Support for TensorFlow about deepdetect

[kyrs]

@beniz with the new modification in dd , I am able to run graph for .jpg and .jpeg images. It is also able to process multiple data.
but there is still some issue:

• Presently It doesn't support prediction on image url.
• although it is able to run graph for multiple images serially. But still it doesn't store the images in a single tensor and call Run only once.

[kyrs]

For connectingImgInputFileConn with tfinputconn we need to convert CV::Mat file into tensor but unlike caffe there isn't any direct method for that in Tensorflow. Discussion in this link might come handy in this regard.
http://stackoverflow.com/questions/36044197/tensorflow-c-pass-an-opencv-mat-into-the-graph

[kyrs]

yep!! Once it is done we can plan the same for text and csv format too..

[kyrs]

hi @beniz, One good news .. Just made the first prediction in dd using inception model. :) :)

[beniz]

We're moving on with this, see beniz@77eca1b

I aim to get it to work nicely and merge over the next two weeks.

[kyrs]

found few relevant document related to the problem:

http://stackoverflow.com/questions/33620794/how-to-build-and-use-google-tensorflow-c-api
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/cc/tutorials/example_trainer.cc

[kyrs]

For loading the tensorflow graph in c++, we need to know the name of the input and output variable in the graph. It would be good option to have common connection point in dd framework.

http://stackoverflow.com/questions/35508866/tensorflow-different-ways-to-export-and-run-graph-in-c
http://stackoverflow.com/questions/35120102/executing-a-model-trained-in-skflow-using-the-tensorflow-c-api/35139892

[kyrs]

found something similar to dd and with support for tensorflow. It would be really interesting to look how they are doing this stuff.
http://mldb.ai/v1/plugins/tensorflow/doc/TensorflowGraph.md.html

[beniz]

AFAIK mldb is not open source (EDIT: there's a community edition, here: https://github.com/mldbai/mldb). Their plugin code appears to be here: https://github.com/mldbai/mldb/blob/a7ef31f30c155d6f65b77165920695d5a7ddd9ab/tensorflow/tensorflow_plugin.cc

You should be able to work with a TF inception graph and connect the inputs and outputs.

Let me know what your current difficulty is and I'll be able to help. Typically I can find the time to put a tf branch up, with the basic C++ classes in which to operate the connection to TF graphs.

[kyrs]

Found this really interesting tutorial, which talks about the input and output connectivity of a graph using c++ . I think best approach would be to first focus on running inception model using dd architecture and then we will scale its features.
https://www.tensorflow.org/versions/r0.7/tutorials/image_recognition/index.html

[beniz]

I think best approach would be to first focus on running inception model using dd architecture

I agree 100% and had had the same thought the other day. Again, let me know how it goes and if you need help.

[kyrs]

It seems I need you help.. I am not a big C++ fan and understanding everything in your code is difficult for me. I might need your help in integrating codes with dd. By now, I have understand that there needs to be three file just like xgblib, xgbinputconns, xgbmodel. Can you please let me know functionality of all these file and there important classes and function. I am developing similar stuff for tensorflow focusing on inception model. so what type of classes should I have for proper integration with dd..

[beniz]

@kyrs yes the C++ and templates withvariant are not easy to get at first glance, compared to other languages. First, to answer your question about the files:

• a tflib file would handle the train() and predict() mais functions. For the inception test, only filling out predict would do the trick
• a tfinputconns file would convert the input data from the image connector (imginputfileconn.h) into whatever (tensor?) TF takes as input
• a tfmodel file would handle the possibly various files required by the TF model on disk.

Second, if you agree with this plan, I can write the embryo of these files and put them up onto a tf branch in the repository. At this stage there would be no connection to TF yet, and you'll be able to move forward by connecting the code you already have. I will be available to provide further help in connecting the various parts.

Let me know what you think, and I propose you post the TF code you already have either here or in a gist so that we can connect the dots.

[beniz]

@kyrs you'll find the C++ shells for Tensorflow integration in branch tf. This should compile as usual. Note that there's no integration of any TF header yet.

[kyrs]

@beniz thanks for creating the branch. I will upload the code soon.

[kyrs]

I am following this file https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/label_image/main.cc
It answers a lot about the connectivity and other important features of tensorflow.

[beniz]

@kyrs excellent. Various parts of this code should go into tflib.cc, starting with ::predict which should be the easiest. Let me know if you have issues with input image conversion to TF input format. FYI, images are already read from file by the dd image in put connector and are readily available as a vector of cv::Mat elements.

[beniz]

OK, good pointer. I suggest to test the conversion outside of dd for a start, i.e. take a cv::Mat and try to get it into the in-memory array. dd takes images from many formats, and even from in-json base64, so I think we really need to do the conversion (as opposed to let TF read the images on file for instance).

I'd be interested if you can post the conversion code here, and I'd be happy to test it as well.

[kyrs]

I am using bazel for compilation of c++ files developed using tensorflow library. Any idea on how to use library of both tensorflow and Opencv for compilation of code.

[beniz]

I m not familiar with bazel enough. But you may want to locate the TF headers and use dd cmake. Or share your existing code and setup and I ll try to help.

[kyrs]

I am running this code https://gist.github.com/kyrs/9adf86366e9e4f04addb and have setup as is mentioned in https://medium.com/jim-fleming/loading-a-tensorflow-graph-with-the-c-api-4caaff88463f#.y1r3jiryb for using bazel using tensorflow.

[beniz]

Can you see or extract the g++ call when using bazel ? This would be a way to work the dependencies. Because in the end it needs to be compiled with cmake, not bazel, and to do this, we'll need to list the dependencies. Tell me if it is not clear. Look at http://stackoverflow.com/questions/32823625/bazel-build-verbose-compiler-commands-logging it seems to be the right command to see the underlying calls.

[kyrs]

Found this file https://github.com/mldbai/mldb/blob/master/ext/tensorflow.mk I am looking into it for compilation of code..

[beniz]

I'm putting some time on it directly into dd, and it is not going to be easy, notably because of conflicting protocol-buffers version and more. However, you should be able to compile it outside dd with less difficulties. Let me know.

[beniz]

FIY https://github.com/mldbai/mldb/blob/master/ext/tensorflow.mk is to replace the use of bazel, which would be the right thing to do, but looks like hell. Lot of work ahead I fear...

[kyrs]

seem so.. Is it possible to compile codes of dd using bazel instead of cmake ?? . Atleast, I can start focusing on writing rest of codes first and then on using cmake for rest of stuff.

[beniz]

Not really. The way to go is build TF with bazel, then to refer the headers to dd for compilation. But maybe you could compile the opencv_tensor.cc file with bazel and make sure everything works fine ?

[beniz]

OK, I'm getting there, in your script https://gist.github.com/kyrs/9adf86366e9e4f04addb, input_tensor_mapped is undefined, could you check your code and see if you can fix it ?

[kyrs]

Oops!! I missed that line. Did the modification..

[beniz]

OK I got things to compile, I'll have more time tomorrow, but for now the sketch of what to do is as follows:

• get tensorflow sources via github
• compile tensorflow with

bazel build //tensorflow:libtensorflow.so

This builds a share library for TF, otherwise it is complicated to link against TF code.

• apply this diff to your deepdetect repo, https://gist.github.com/beniz/dd35f1d6933814006f24f2585b18dac1
• modify the paths to replace /home/beniz/... with the paths to the similar files on your system
• add opencv_tensor.cc to deepdetect/tests
• run make in build/tests
• run ./opencv_tensor

Now, the last step crashes on me, but the steps above unlock the ability to move forward.

[beniz]

Also, I forgot to add that I had to install protocol-buffers from sources into my home repository, as follows:

git clone https://github.com/google/protobuf.git
./autogen.sh
./configure --prefix=/home/yourhome
make
make install

This takes a while.

[beniz]

@kyrs getting the source image pointer in your code is wrong I think. Here is what seems to not crash for me:

 std::vector<float> array;
  if (Image.isContinuous()) {
    array.assign((float*)Image.datastart, (float*)Image.dataend);
  } else {
    for (int i = 0; i < Image.rows; ++i) {
      array.insert(array.end(), (float*)Image.ptr<uchar>(i), (float*)Image.ptr<uchar>(i)+Image.cols);
    }
  }                                                      
  const float * source_data = &array[0];

Let me know how it goes.

[kyrs]

@beniz Well the compilation of tensorflow library took a lot of time on my system and now on building the dd I am getting this error.

In file included from .build_release/src/caffe/proto/caffe.pb.cc:5:0:
.build_release/src/caffe/proto/caffe.pb.h:12:2: error: #error This file was generated by a newer version of protoc which is
#error This file was generated by a newer version of protoc which is
^
.build_release/src/caffe/proto/caffe.pb.h:13:2: error: #error incompatible with your Protocol Buffer headers. Please update
#error incompatible with your Protocol Buffer headers. Please update
^
.build_release/src/caffe/proto/caffe.pb.h:14:2: error: #error your headers.
#error your headers.
^
.build_release/src/caffe/proto/caffe.pb.h:22:35: fatal error: google/protobuf/arena.h: No such file or directory
#include <google/protobuf/arena.h>
^
compilation terminated.
make[3]: *** [.build_release/src/caffe/proto/caffe.pb.o] Error 1
make[2]: *** [caffe_dd/src/caffe_dd-stamp/caffe_dd-configure] Error 2
make[1]: *** [CMakeFiles/caffe_dd.dir/all] Error 2
make: *** [all] Error 2

[beniz]

Yes, you need to install the protocol-buffer 3.0.0 lib as mentioned above.

But first, I'd suggest you try compiling opencv_tensor by going into build/tests and doing make. Then you should be able to debug the OpenCV to TF tensor convertor. If you can do this part, this'd be great.

Though if you'd like to compile dd, you'd need the protocol-buffer as indicated, and you'd need to hack build/caffe_dd/src/caffe_dd/Makefile.config by adding /home/yourhome/lib to LIBRARY_DIRS and /home/yourhome/include to INCLUDE_DIRS

I'm in the process of trying to make all this simple, not there yet.

[beniz]

The branch tf now has the first automated build with Caffe + TF! It is still rough on the edges, most especially:

• TF + Caffe appear to conflict on CUDA, so no GPU on the current build
• Beware: Build automatically installs protocol-buffers v3.0.0 into the $HOME repository from where it is started. Not a good practice, but no way around for now.

To build:

mkdir build_tf
cd build_tf
cmake .. -DBUILD_TESTS=ON
make

Keep an eye on it, TF configuration asks two questions, just type Enter and proceed. Also, TF builds may take a while depending on configuration.

@kyrs to test the modified opencv_tensor code:

cd build_tf/tests
./opencv_tensor ../examples/caffe/mnist/sample_digit.png

I have not tested the conversion from OpenCV to TF tensor yet, and the internals for TF are still empty shells. Though this is a good step forward.

[kyrs]

Cool I will look into it. It would really be helpful, atleast now I can compile the code.

[kyrs]

here is the final code for running the inception model using opencv mat file https://gist.github.com/kyrs/9adf86366e9e4f04addb . Although it is working perfectly fine with a png images, but code fails to read images of other format like jpg or jpeg might be some tensorflow issue.

[kyrs]

@beniz, I tried to compile the code under dd architecture but got some weird error. I am sharing the log file generated during compilation.
https://gist.github.com/kyrs/7c9e7cc4ecbc95a6a7763390378d83e6

[beniz]

@kyrs here is a temporary way to get to the compilation of your code, patching tensorflow with:
https://gist.github.com/828e7c6979b0604cda00b947a14b02f9
Copy the patch into logging.h.patch

Way to do it:

cd build/tensorflow/src/tensorflow
patch -p1 < logging.h.patch

Resume compilation of dd with tf.

[kyrs]

@beniz I have removed the error in the tf files. but the error in logging still persist.
following is the log file of the error generated https://gist.github.com/kyrs/5e0516bc2ef103cf98d404c873bf712f

[beniz]

Yes, this is the linker complaining. Can you update the PR with your changes so I can try to fix the linking part. Thanks!

[beniz]

@kyrs linking is fixed, and the md5 hash from bazel is no longer an issue as well. You should be able to go into happy debug mode.

One remark, in your code at https://github.com/kyrs/deepdetect/blob/tensorflow/src/tflib.cc#L150, you may want to store the images in a single tensor and call Run only once.

Let me know how it goes, and don't hesitate to report difficulties. Thanks again for the long work.

[kyrs]

@beniz, I am planning to get things working once with present code, then will move forward to store images in a single tensor, btw thanks for making the modification.

[beniz]

@kyrs understood, this is the right way to go. TF in predict mode is a great addition, considering that TF model can be created from several other libs (keras, ...).

[beniz]

So at the moment, we're still partly stuck because of tensorflow/tensorflow#1924 (comment)

We can move forward by letting images (all jpeg and some png) be rejected. It seems that TF has its own libjpeg and libpng and they may interact badly with OpenCV.

[kyrs]

@beniz can't we use libjpeg and libpng files specifically while interacting with tensorflow graphs . Instead of using opencv to load images.

[beniz]

@kyrs yes, that's a good idea indeed, it'd have to go into the TFInputConn class. Maybe one way to do this would be to use TF way of reading images, from their snippet of code. Are you willing to test this ? (and do you have enough free time to do it ?)

However, keep in mind that if the libs are the true problem here, since OpenCV is already being linked into dd, you may still experience the issue.

[kyrs]

@beniz sure I will try to do that. Will update you with the progress.

[kyrs]

@beniz, I am trying to compile ReadTensorFromImageFile function in opencv_tensor.cc code as is mentioned in file https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/label_image/main.cc
but for some reason I am not able to call #include "tensorflow/cc/ops/image_ops.h" and
#include "tensorflow/cc/ops/standard_ops.h". Both of these files are important for reading an image using tf library. Interestingly, I was able to build main.cc using https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/label_image/BUILD . I guess there might be some problem in compilation of tensorflow in dd.

[beniz]

@kyrs I can't find image_ops.h so it might have been removed from TF. standard_ops.h is there, you can modify TF_INC_DIR in deepdetect/CMakeLists.txt to include ${CMAKE_BINARY_DIR}/tensorflow/src/tensorflow/. Let me know how this goes.

[kyrs]

@beniz tried that but got following error
In file included from /home/ks/OpenSource/tf/shubham/deepdetect/tests/opencv_tensor.cc:28:0: /home/ks/OpenSource/tf/shubham/deepdetect/build/tensorflow/src/tensorflow/dbuild/tensorflow/tensorflow/cc/ops/standard_ops.h:22:41: fatal error: tensorflow/cc/ops/array_ops.h: No such file or directory #include "tensorflow/cc/ops/array_ops.h" ^ compilation terminated. make[2]: *** [tests/CMakeFiles/opencv_tensor.dir/opencv_tensor.cc.o] Error 1 make[1]: *** [tests/CMakeFiles/opencv_tensor.dir/all] Error 2 make: *** [all] Error 2

[beniz]

@kyrs obviously array_ops.h does not exist by default: https://github.com/tensorflow/tensorflow/tree/master/tensorflow/cc/ops

Now, maybe it is generated at some point, we will need to investigate.

[beniz]

So, some files appear to be generated, see https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tensorflow.bzl#L142
EDIT: probably called from https://github.com/tensorflow/tensorflow/blob/e39d8feebb9666a331345cd8d960f5ade4652bba/tensorflow/cc/BUILD#L28

[kyrs]

@beniz I will need these files for loading image and its modification using libjpeg. Can you tell me, how to integrate these files with dd.

[beniz]

I'm not sure yet, one way to start would be to check whether building TF as indicated on their install page does generate these files, or not.

[beniz]

@kyrs commit beniz@b1b520a modifies the TF build so that the missing headers are auto-generated.

[kyrs]

@beniz, I have tried to run the code which your last commit. But during compilation, I am getting an error of undefined reference to few function used in it. I guess there is a bug in linking of new library to the executable file which is getting generated. following is the code i am using https://gist.github.com/kyrs/a20c691624e007f4b769f4a1894af61e

[beniz]

It never ends :) can you post the linking error ? I ll try the build a bit later today

[kyrs]

Let's hope that using tf lib solve the problem of opencv. Following is the error file generated during compilation. https://gist.github.com/kyrs/5e0516bc2ef103cf98d404c873bf712f

[beniz]

@kyrs see the new commit to tf branch and that fixes your linking issue by adding the standard ops to TF link.

[kyrs]

@beniz, I was able to run inception model for both .png and .jpeg even with opencv linker . Will Soon make the necessary changes in dd. :)

[AlexJoz]

Hi there! Is it possible to connect to this work somehow? I like your project, but i'm not really interested in caffe or boost) Only Tensorflow =)

[beniz]

Due to the lack of true C++ API within TF this PR has limited functionality. This is why this PR hasn't been merged in yet.

What models are willing to run ?

[AlexJoz]

I have c++ restservice for TF backend, but it's specific model in private production. I mean, TF api is really pure, but its working, there were no significant issues to implement something. I was just thinking that you already have some kind of plan for this and i can join =)

[beniz]

What I meant is, are you classifying images, text, using RNNs, or feedforward deep nets ?

[AlexJoz]

image processing mostly

[beniz]

This TF issue + existing PR supports image classification, not training.

[beniz]

FTR, just updated the tf branch to accomodate changes by TF to both their API and building system. Successfully tested on prediction with Inception pre-trained net.

[beniz]

Notes on build:

• requires cmake >=3
• not compatible with ccache

[beniz]

Merged.