bendangnuksung / mrcnn_serving_ready Goto Github PK

hey @bendangnuksung ! Wow, this repo seriously saved my life, thank you so much. So using your repo, I have successfully deployed a mask rcnn model to gcp ai platform with no issues. But, for a couple weeks now, I have been hitting a road block on getting a prediction back. In other words, what's an example JSON object i can send that will work?
here is the code i used to create the serving model:

def make_serving_ready(model_path, save_serve_path, version_number):
    import tensorflow as tf

    export_dir = os.path.join(save_serve_path, str(version_number))
    graph_pb = model_path

    builder = tf.saved_model.builder.SavedModelBuilder(export_dir)

    with tf.gfile.GFile(graph_pb, "rb") as f:
        graph_def = tf.GraphDef()
        graph_def.ParseFromString(f.read())

    sigs = {}
    
    # tf.import_graph_def(graph_model_def, name='', input_map={"input_image": img_uint8})
    with tf.Session(graph=tf.Graph()) as sess:
        # name="" is important to ensure we don't get spurious prefixing
        tf.import_graph_def(graph_def, name="")
        g = tf.get_default_graph()
        input_image = g.get_tensor_by_name("input_image:0")
        input_image_meta = g.get_tensor_by_name("input_image_meta:0")
        input_anchors = g.get_tensor_by_name("input_anchors:0")

        output_detection = g.get_tensor_by_name("mrcnn_detection/Reshape_1:0")
        output_mask = g.get_tensor_by_name("mrcnn_mask/Reshape_1:0")

        sigs[signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY] = \
            tf.saved_model.signature_def_utils.predict_signature_def(
                {"input_image": input_image},
#                 {"input_image": input_image, 'input_image_meta': input_image_meta, 'input_anchors': input_anchors},
#                 {"image_bytes": img_uint8, 'input_image_meta': input_image_meta, 'input_anchors': input_anchors},
                {"mrcnn_detection/Reshape_1": output_detection, 'mrcnn_mask/Reshape_1': output_mask})

        builder.add_meta_graph_and_variables(sess,
                                             [tag_constants.SERVING],
                                             signature_def_map=sigs)

    builder.save()
    print("*" * 80)
    print("FINISH CONVERTING FROZEN PB TO SERVING READY")
    print("PATH:", PATH_TO_SAVE_TENSORFLOW_SERVING_MODEL)
    print("*" * 80)

for example, i tried the JSON input below to just get any type of response with no luck:

{"instances":[
{"input_image":[[[[0.0],[0.5],[0.8]]]]},
{"input_image_meta":[[[1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0]]]
}
]}

please help!!

p.s. Going the extra mile: How would we be able to adjust the above function to accept b64 encoded images?? :)

$ python3 main.py

Using TensorFlow backend.
2019-07-01 11:52:10.385753: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-07-01 11:52:10.408767: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
2019-07-01 11:52:10.409344: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x22d6a50 executing computations on platform Host. Devices:
2019-07-01 11:52:10.409356: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:From /home/deepedge/mask_rcnn-master/mrcnn_serving_ready-master/model.py:771: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
Traceback (most recent call last):
File "main.py", line 108, in
model.load_weights(H5_WEIGHT_PATH, by_name=True)
File "/home/deepedge/mask_rcnn-master/mrcnn_serving_ready-master/model.py", line 2131, in load_weights
saving.load_weights_from_hdf5_group_by_name(f, layers)
File "/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py", line 1149, in load_weights_from_hdf5_group_by_name
str(weight_values[i].shape) + '.')
ValueError: Layer #391 (named "mrcnn_bbox_fc"), weight <tf.Variable 'mrcnn_bbox_fc/kernel:0' shape=(1024, 4) dtype=float32_ref> has shape (1024, 4), but the saved weight has shape (1024, 8).
@bendangnuksung

Hello,

Below are the steps I followed:

Docker image was used using Tensorflow 1.14 version

1. python main.py (COCO weights were used) ---> Successfully returns frozen and serving_model

Using Docker image Tensorflow 2.1 version

2. install tensorflow_server_model (https://medium.com/@noone7791/how-to-install-tensorflow-serving-load-a-saved-tf-model-and-connect-it-to-a-rest-api-in-ubuntu-48e2a27b8c2a)

3. python3 inferencing/saved_model_inference.py -t restapi -p test_image/monalisa.jpg

Error 1: mrcnn not found.
Solution: Copied mrcnn folder from matterport to inferencing

3. python3 inferencing/saved_model_inference.py -t restapi -p test_image/monalisa.jpg

Error :
File "inferencing/saved_model_inference.py", line 106, in
result = detect_mask_single_image_using_restapi(image)
File "inferencing/saved_model_inference.py", line 78, in detect_mask_single_image_using_restapi
response = requests.post(RESTAPI_URL, data=json.dumps(data), headers={"content-type":"application/json"})
File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 116, in post
return request('post', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/api.py", line 60, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError:

However I run your code in CPU mode. but it gives the following error:

KeyError: "The name 'input_image:0' refers to a Tensor which does not exist. The operation, 'input_image', does not exist in the graph."

Would you help me please?

Thanks

I want to use this MaskRCNN model to inference on android how to do that. Thank you for your kind consideration.

In the coco model, when I use resnet100 and resnet50 to generate h5 files to pb files respectively, why can both be used?

I have converted the .h5 file to .pb file using

https://github.com/bendangnuksung/mrcnn_serving_ready

and converted .pb file to .pbtxt file using

https://gist.github.com/Arafatk/c063bddb9b8d17a037695d748db4f592

after then I have a .pb file and .pbtxt file yes, so I used your blogpost

https://www.pyimagesearch.com/2018/11/19/mask-r-cnn-with-opencv/

I have used above generated .pb and .pbtxt to produce masked result image so I am getting the error like this :

python3 mask_rcnn.py --mask-rcnn mask-rcnn-coco --image /home/deepedge/mask_rcnn-master/mask-rcnn-bottle-training-master/dataset/val/car39.jpg

[INFO] loading Mask R-CNN from disk...
[libprotobuf ERROR /io/opencv/3rdparty/protobuf/src/google/protobuf/text_format.cc:288] Error parsing text-format opencv_tensorflow.GraphDef: 60540:5: Unknown enumeration value of "DT_RESOURCE" for field "type".
Traceback (most recent call last):
File "mask_rcnn.py", line 48, in
net = cv2.dnn.readNetFromTensorflow(weightsPath, configPath)
cv2.error: OpenCV(4.1.0) /io/opencv/modules/dnn/src/tensorflow/tf_io.cpp:54: error: (-2:Unspecified error) FAILED: ReadProtoFromTextFile(param_file, param). Failed to parse GraphDef file: mask-rcnn-coco/dent.pbtxt in function 'ReadTFNetParamsFromTextFileOrDie'

How to solve?

please help
@bendangnuksung

when I use curl -X POST http://localhost:8501/v1/models/123456:predict
{ "error": "JSON Parse error: The document is empty" }

any directions or pointers please why this is coming

Hey, I converted the h5 model into pb with the given instructions, hoping that the inference speed would improve. It does take less time(~6s) for inference compared to h5(~13s) on CPU. Both are taking around 30s for the first inference and then speeds up to the time mentioned above. However, there is no improvement in speed on GPU, and sometimes it is even slower than CPU. How can I serve my model with the lowest inference speed possible? What speed is expected from the inference for mask RCNN in general?

Hi there,

There are two different .pb file in both tensorflow_serving_model and the frozen_pb. If I want to inference the model, which .pb file should I use? Thanks.

Dear bendangnuksung,

First of all thanks for the code it was really helpful.
but,I have a question,how can I use the pb file to inference?

When I run your code (a little modified for config) it gives the following error:

InvalidArgumentError: Cannot assign a device for operation ROI/rpn_non_max_suppression/NonMaxSuppressionV3: Could not satisfy explicit device specification '/device:GPU:0' because no supported kernel for GPU devices is available. Registered kernels: device='CPU'; T in [DT_FLOAT] device='CPU'; T in [DT_HALF]

Maybe the problem arise due these lines:

sess = tf.Session()
K.set_session(sess)

How can I solved?

I created .pb file using your code. I would like to use the .pb file in opencv-python dnn . How can I use?

I have the frozen graph, I added in my own config class. When I try to convert the frozen graph into a model usable for the movidius stick I get a series of errors. The one that's got me really stumped is:

KeyError: "The name 'input:0' refers to a Tensor which does not exist. The operation, 'input', does not exist in the graph."

This is what I'm inputting:
sudo mvNCCompile -s 12 mask_frozen_graph_bottle.pb -in=input -on=/mnt/d/Users/Asher/Research/Mask_RCNN/frozen_model

Does anyone know how to solve this?

Line 50 of main.py:

def freeze_model(model, name):
    frozen_graph = freeze_session(
        sess,
        output_names=[out.op.name for out in model.outputs][:4])

Correct me if I'm wrong, but using the first 4 output names [:4] of your output_names is not guaranteed to get the correct output node names.

What we really want are the 5 output nodes regarding Mask RCNN loss:

rpn_class_loss
rpn_bbox_loss
mrcnn_class_loss
mrcnn_bbox_loss
mrcnn_mask_loss

When I run [out.op.name for out in keras_model.model.outputs]
I get 14 output nodes:

['rpn_class_logits/concat',
'rpn_class/concat',
'rpn_bbox/concat',
'mrcnn_class_logits/Reshape_1',
'mrcnn_class/Reshape_1',
'mrcnn_bbox/Reshape',
'mrcnn_mask/Reshape_1',
'ROI/packed_2',
'output_rois/mul',
'rpn_class_loss/cond/Merge',
'rpn_bbox_loss/cond/Merge',
'mrcnn_class_loss/truediv',
'mrcnn_bbox_loss/Mean',
'mrcnn_mask_loss/Mean']

So, at least for this particular model, the desired output nodes are actually the last 5 of these nodes: `output_names=[out.op.name for out in model.outputs][-5:])

Am I missing something?

File url:
https://nomeroff.net.ua/models/mrcnn/mask_rcnn_numberplate_1000_2019_10_07.h5
Error:

Instructions for updating:
Use tf.cast instead.
Traceback (most recent call last):
File "main.py", line 108, in
model.load_weights(H5_WEIGHT_PATH, by_name=True)
File "C:\WWW\convert\convert_all\mrcnn_serving_ready-master\mrcnn_serving_ready-master\model.py", line 2131, in load_weights
saving.load_weights_from_hdf5_group_by_name(f, layers)
File "C:\Program Files\Python37\lib\site-packages\keras\engine\saving.py", line 1328, in load_weights_from_hdf5_group_by_name
str(weight_values[i].shape) + '.')
ValueError: Layer #391 (named "mrcnn_bbox_fc"), weight <tf.Variable 'mrcnn_bbox_fc/kernel:0' shape=(1024, 24) dtype=float32> has shape (1024, 24), but the saved weight has shape (1024, 8).

I tried to deploy this model on google ai platform, I get this error:

WARNING: Using endpoint [https://ml.googleapis.com/]
Creating version (this might take a few minutes)......failed.
ERROR: (gcloud.ai-platform.versions.create) Create Version failed. Model validation failed: Outer dimension for outputs must be unknown, outer dimension of 'mrcnn_detection/Reshape_1:0' is 1 

Any Idea how to solve it, here's the signature def of my model:

signature_def['serving_default']:
  The given SavedModel SignatureDef contains the following input(s):
    inputs['input_anchors'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, -1, 4)
        name: input_anchors:0
    inputs['input_image'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, -1, -1, 3)
        name: input_image:0
    inputs['input_image_meta'] tensor_info:
        dtype: DT_FLOAT
        shape: (-1, 14)
        name: input_image_meta:0
  The given SavedModel SignatureDef contains the following output(s):
    outputs['mrcnn_detection/Reshape_1'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 100, 6)
        name: mrcnn_detection/Reshape_1:0
    outputs['mrcnn_mask/Reshape_1'] tensor_info:
        dtype: DT_FLOAT
        shape: (1, 100, 28, 28, 2)
        name: mrcnn_mask/Reshape_1:0
  Method name is: tensorflow/serving/predict

How can this error be fixed ?

hello,this saved_model.pd has been loaded into tensorflow_model server, but I do not know how to inference in client, could you help me ? thanks!

I've converted my h5 file to pb using mrcnn_serving_ready seemingly successfully. However when I try to convert my saved_model.pb it to a tflite lite, using tflite_convert, it crashes.

user_config.py

is_coco = False

MODEL_DIR = '../export/'

H5_WEIGHT_PATH = '../export/mask_rcnn_me_cfg_0113.h5'

PATH_TO_SAVE_FROZEN_PB = '../export/frozen_model/'

FROZEN_NAME = 'mask_frozen_graph.pb'

PATH_TO_SAVE_TENSORFLOW_SERVING_MODEL = '../export/serving_model'

VERSION_NUMBER = 1

NUMBER_OF_CLASSES = 1

Crash

(keras-gpu) PS C:\src> tflite_convert --output_file .\export\foo.tflite --graph_def_file=saved_model.pb --input_arrays=input_image --output_arrays=mrcnn_bbox/Reshape
2019-09-02 11:00:15.382200: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library nvcuda.dll
2019-09-02 11:00:15.496843: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2019-09-02 11:00:15.502449: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2019-09-02 11:00:15.505062: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-09-02 11:00:15.510887: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2019-09-02 11:00:15.514936: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2019-09-02 11:00:15.527747: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2019-09-02 11:00:15.532083: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-09-02 11:00:16.162723: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-02 11:00:16.166335: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-09-02 11:00:16.168587: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-09-02 11:00:16.170862: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4637 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
Ignore 'tcmalloc: large alloc' warnings.
Traceback (most recent call last):
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\tensorflow\lite\python\lite.py", line 608, in from_frozen_graph
graph_def.ParseFromString(file_content)
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\google\protobuf\message.py", line 187, in ParseFromString
return self.MergeFromString(serialized)
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\python_message.py", line 1124, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\python_message.py", line 1189, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\decoder.py", line 723, in DecodeField
if value._InternalParse(buffer, pos, new_pos) != new_pos:
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\python_message.py", line 1189, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\decoder.py", line 702, in DecodeRepeatedField
if value.add()._InternalParse(buffer, pos, new_pos) != new_pos:
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\python_message.py", line 1189, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\decoder.py", line 847, in DecodeMap
if submsg._InternalParse(buffer, pos, new_pos) != new_pos:
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\python_message.py", line 1176, in InternalParse
buffer, new_pos, wire_type) # pylint: disable=protected-access
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\decoder.py", line 948, in _DecodeUnknownField
(data, pos) = _DecodeUnknownFieldSet(buffer, pos)
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\decoder.py", line 927, in _DecodeUnknownFieldSet
(data, pos) = _DecodeUnknownField(buffer, pos, wire_type)
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\decoder.py", line 952, in _DecodeUnknownField
raise _DecodeError('Wrong wire type in tag.')
google.protobuf.message.DecodeError: Wrong wire type in tag.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\me.conda\envs\keras-gpu\Scripts\tflite_convert-script.py", line 10, in
sys.exit(main())
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\tensorflow\lite\python\tflite_convert.py", line 503, in main
app.run(main=run_main, argv=sys.argv[:1])
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\absl\app.py", line 300, in run
_run_main(main, args)
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\absl\app.py", line 251, in _run_main
sys.exit(main(argv))
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\tensorflow\lite\python\tflite_convert.py", line 499, in run_main
_convert_tf1_model(tflite_flags)
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\tensorflow\lite\python\tflite_convert.py", line 124, in _convert_tf1_model
converter = _get_toco_converter(flags)
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\tensorflow\lite\python\tflite_convert.py", line 111, in _get_toco_converter
return converter_fn(**converter_kwargs)
File "C:\Users\me.conda\envs\keras-gpu\lib\site-packages\tensorflow\lite\python\lite.py", line 615, in from_frozen_graph
file_content = file_content.decode("utf-8")
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbb in position 3: invalid start byte
(keras-gpu) PS C:\src>

saved_model.pb

https://drive.google.com/open?id=1_3RcuuTy1dXHmdVyYJU5RQLWG9Dg1KWc

I also tried opening the file with tensorboard but that also failed:

(keras-gpu) PS C:\Users\oixx\src\parkshark> python .\import_pb_to_tensorboard.py --model_dir ./export/serving_model/1/saved_model.pb --log_dir .
2019-09-02 14:28:07.270170: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library nvcuda.dll
2019-09-02 14:28:07.368085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2019-09-02 14:28:07.372248: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2019-09-02 14:28:07.374859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-09-02 14:28:07.376887: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX AVX2
2019-09-02 14:28:07.385897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1660 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.8
pciBusID: 0000:01:00.0
2019-09-02 14:28:07.389397: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2019-09-02 14:28:07.396890: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2019-09-02 14:28:08.051391: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-09-02 14:28:08.053953: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2019-09-02 14:28:08.056163: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2019-09-02 14:28:08.058340: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4637 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1660 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
Traceback (most recent call last):
File ".\import_pb_to_tensorboard.py", line 86, in
app.run(main=main, argv=[sys.argv[0]] + unparsed)
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\absl\app.py", line 300, in run
_run_main(main, args)
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\absl\app.py", line 251, in _run_main
sys.exit(main(argv))
File ".\import_pb_to_tensorboard.py", line 67, in main
import_to_tensorboard(FLAGS.model_dir, FLAGS.log_dir)
File ".\import_pb_to_tensorboard.py", line 57, in import_to_tensorboard
graph_def.ParseFromString(f.read())
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\google\protobuf\message.py", line 187, in ParseFromString
return self.MergeFromString(serialized)
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\python_message.py", line 1124, in MergeFromString
if self._InternalParse(serialized, 0, length) != length:
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\python_message.py", line 1189, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\decoder.py", line 723, in DecodeField
if value._InternalParse(buffer, pos, new_pos) != new_pos:
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\python_message.py", line 1189, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\decoder.py", line 702, in DecodeRepeatedField
if value.add()._InternalParse(buffer, pos, new_pos) != new_pos:
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\python_message.py", line 1189, in InternalParse
pos = field_decoder(buffer, new_pos, end, self, field_dict)
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\decoder.py", line 847, in DecodeMap
if submsg._InternalParse(buffer, pos, new_pos) != new_pos:
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\python_message.py", line 1176, in InternalParse
buffer, new_pos, wire_type) # pylint: disable=protected-access
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\decoder.py", line 948, in _DecodeUnknownField
(data, pos) = _DecodeUnknownFieldSet(buffer, pos)
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\decoder.py", line 927, in _DecodeUnknownFieldSet
(data, pos) = _DecodeUnknownField(buffer, pos, wire_type)
File "C:\Users\oixx.conda\envs\keras-gpu\lib\site-packages\google\protobuf\internal\decoder.py", line 952, in _DecodeUnknownField
raise _DecodeError('Wrong wire type in tag.')
google.protobuf.message.DecodeError: Wrong wire type in tag.

Any ideas what could be wrong here?

@bendangnuksung
Thanks for your great repository. it helps me lot.

Would you suggest me please How can I read input and output node/tensor from frozen inference graph (.pb)?
Thanks again

I have the mask_frozen_graph.pb, now how to run on command line and use it?

@bendangnuksung @bendangnuksung

It is unclear precisely what the SavedModel inputs and outputs should look like for the Mask RCNN model, once frozen and exported in a Servable format.

What should the inputs and outputs of the SavedModel SignatureDef look like?

I'm seeing these 3 input tensors and these two output tensors. Is this correct?

The given SavedModel SignatureDef contains the following input(s):
inputs['input_anchors'] tensor_info:
dtype: DT_FLOAT
shape: (-1, -1, 4)
name: input_anchors:0
inputs['input_image'] tensor_info:
dtype: DT_FLOAT
shape: (-1, -1, -1, 3)
name: input_image:0
inputs['input_image_meta'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 33)
name: input_image_meta:0
The given SavedModel SignatureDef contains the following output(s):
outputs['mrcnn_detection/Reshape_1'] tensor_info:
dtype: DT_INT64
shape: (-1)
name: mrcnn_detection/Reshape_1:0
outputs['mrcnn_mask/Reshape_1'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 100, 28, 28, 21)
name: mrcnn_mask/Reshape_1:0
Method name is: tensorflow/serving/predict

Dear bendangnuksung,

First of all thanks for the code it was really helpful.
I used your code to create serving ready model. It was successfully done but there is an issue after running the code I was able to see

Finish converting keras model to Frozen PB
PATH: ./frozen_model/

FINISH CONVERTING FROZEN PB TO SERVING READY
PATH: ./serving_model

COMPLETED

But when I checked variables folder it was empty.
Would you please help me with this issue?

i am getting results like this, but how to display the result of image?

[{'rois': array([[244, 287, 590, 721],
[216, 737, 426, 925]], dtype=int32), 'class': array([1, 1], dtype=int32), 'scores': array([0.9834276 , 0.76979005], dtype=float32), 'mask': array([[[False, False],
[False, False],
[False, False],
...,
[False, False],
[False, False],
[False, False]],

   [[False, False],
    [False, False],
    [False, False],
    ...,
    [False, False],
    [False, False],
    [False, False]],

   [[False, False],
    [False, False],
    [False, False],
    ...,
    [False, False],
    [False, False],
    [False, False]],

   ...,

   [[False, False],
    [False, False],
    [False, False],
    ...,
    [False, False],
    [False, False],
    [False, False]],

   [[False, False],
    [False, False],
    [False, False],
    ...,
    [False, False],
    [False, False],
    [False, False]],

   [[False, False],
    [False, False],
    [False, False],
    ...,
    [False, False],
    [False, False],
    [False, False]]])}]

@bendangnuksung @buaacarzp

i got weights from wget https://github.com/matterport/Mask_RCNN/releases/download/v2.0/mask_rcnn_coco.h5

getting error as File "main.py", line 113, in
model.load_weights(H5_WEIGHT_PATH, by_name=True)
File "/home/ubuntu/Downloads/Blaize/mrcnn_serving_ready-master/model.py", line 2131, in load_weights
saving.load_weights_from_hdf5_group_by_name(f, layers)
File "/home/ubuntu/Downloads/venv/lib/python3.6/site-packages/keras/engine/saving.py", line 1328, in load_weights_from_hdf5_group_by_name
str(weight_values[i].shape) + '.')
ValueError: Layer #389 (named "mrcnn_bbox_fc"), weight <tf.Variable 'mrcnn_bbox_fc/kernel:0' shape=(1024, 24) dtype=float32, numpy=
array([[-0.05486 , -0.03290214, 0.05897582, ..., -0.05898178,
-0.06868616, 0.05374715],
[-0.06710163, -0.03682471, -0.03057443, ..., -0.05611433,
-0.04561458, 0.05178914],
[-0.0041154 , -0.07344876, -0.06137543, ..., 0.0011842 ,
0.04365869, -0.05199062],
...,
[ 0.06231805, -0.02443966, -0.00532094, ..., -0.01833269,
-0.02245103, -0.01552512],
[-0.04047406, -0.06753345, 0.02390008, ..., 0.01883602,
-0.04362615, -0.05265519],
[ 0.00530255, 0.04341973, 0.03085093, ..., -0.07011634,
0.01440722, 0.02777647]], dtype=float32)> has shape (1024, 24), but the saved weight has shape (1024, 324).

After I generated the frozen graph, and I loaded it to run the inference with a picture, the detection result is different than I directly run the model

For example, I run directly from the model I can have detentions as the following.

	detections[:,4] [1. 1. 1. 3. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
	 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
	 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
	 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
	 0. 0. 0. 0.]

But when I run my frozen graph I will have the following results

	detections[:,4] [1. 1. 1. 3. **3. 9.** 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
	 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
	 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
	 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.
	 0. 0. 0. 0.]

I checked the first 4 detentions are similar, but the frozen graph has 2 different ones, any ideas why frozen graph will generate different detection results, and how to solve it?

Thanks

I got an error while runing main.py file

Hi , bendangnuksung, I try to run with mask_rcnn_coco.h5, but I got the error message as below:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.721 pciBusID: 0000:01:00.0 totalMemory: 11.00GiB freeMemory: 9.10GiB 2019-04-15 08:51:08.395483: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0 2019-04-15 08:51:09.217262: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix: 2019-04-15 08:51:09.222426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988] 0 2019-04-15 08:51:09.225392: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0: N 2019-04-15 08:51:09.228476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8788 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1) WARNING:tensorflow:From E:\Anaconda\lib\site-packages\tensorflow\python\ops\sparse_ops.py:1165: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version. Instructions for updating: Create a tf.sparse.SparseTensorand usetf.sparse.to_dense instead. Traceback (most recent call last): File "main.py", line 88, in <module> model.load_weights(H5_WEIGHT_PATH, by_name=True) File "F:\Code\mrcnn_serving_ready-master\model.py", line 2131, in load_weights saving.load_weights_from_hdf5_group_by_name(f, layers) File "E:\Anaconda\lib\site-packages\keras\engine\saving.py", line 1149, in load_weights_from_hdf5_group_by_name str(weight_values[i].shape) + '.') ValueError: Layer #389 (named "mrcnn_bbox_fc"), weight <tf.Variable 'mrcnn_bbox_fc/kernel:0' shape=(1024, 24) dtype=float32_ref> has shape (1024, 24), but the saved weight has shape (1024, 324).

Received gRPC Error: StatusCode.INVALID_ARGUMENT - input size does not match signature: 1!=3 len({inputs}) != len({input_anchors,input_image,input_image_meta}). Sent extra: {inputs}. Missing but required: {input_anchors,input_image,input_image_meta}.

I have got an error like this while serving. So is there any method to avoid this error without having any preprocessing on clientAPI

Hi,
I've deployed the model to the server, but I don't know how to import data from the client. Could you help me? Thank you

I put my custom config in the beginning main.py

class ShapesConfig(mask_config):
    """Configuration for training on the toy shapes dataset.
    Derives from the base Config class and overrides values specific
    to the toy shapes dataset.
    """
    # Give the configuration a recognizable name
    NAME = "Newton Ring"
    BACKBONE = "resnet50"
    # Train on 1 GPU and 8 images per GPU. We can put multiple images on each
    # GPU because the images are small. Batch size is 8 (GPUs * images/GPU).
    GPU_COUNT = 1
    IMAGES_PER_GPU = 1

    # Number of classes (including background)
    NUM_CLASSES = 1 + 1  # background + 3 shapes

    # Use small images for faster training. Set the limits of the small side
    # the large side, and that determines the image shape.
    IMAGE_MIN_DIM = 1024
    IMAGE_MAX_DIM = 1024

    qq = 4
    # Use smaller anchors because our image and objects are small
    RPN_ANCHOR_SCALES = (8 * qq, 16 * qq, 32 * qq, 64 * qq, 128 * qq) #(32, 64, 128, 256, 512

    # Reduce training ROIs per image because the images are small and have
    # few objects. Aim to allow ROI sampling to pick 33% positive ROIs.
    TRAIN_ROIS_PER_IMAGE = 64

    # Use a small epoch since the data is simple
    STEPS_PER_EPOCH = 890

    # use small validation steps since the epoch is small
    VALIDATION_STEPS = 111    
    # iter_num 
    iter_num = 0
    DETECTION_MIN_CONFIDENCE = 0.81

then
config = ShapesConfig()

the error is like :

    config = ShapesConfig()
TypeError: __init__() missing 1 required positional argument: 'NUMBER_OF_CLASSES'

what is wrong with it?

hello, I executed main.py, but I didn't find any files in /serving_model/1/variables.
Is it correct？

Hi I am following steps mentioned in https://github.com/fizyr/keras-maskrcnn. The model that I use is resnet50_coco_v0.2.0.h5 from (https://github.com/fizyr/keras-maskrcnn/releases).

when I run

python3 main.py

The error I get is:

ValueError: Layer #2 (named "conv1") expects 2 weight(s), but the saved weights have 1 element(s).

complete error stack trace:

FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/ajinkya/anaconda3/envs/tensorflowGPU_p36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/ajinkya/anaconda3/envs/tensorflowGPU_p36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/ajinkya/anaconda3/envs/tensorflowGPU_p36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/ajinkya/anaconda3/envs/tensorflowGPU_p36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/ajinkya/anaconda3/envs/tensorflowGPU_p36/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/ajinkya/anaconda3/envs/tensorflowGPU_p36/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/ajinkya/anaconda3/envs/tensorflowGPU_p36/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/ajinkya/anaconda3/envs/tensorflowGPU_p36/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/ajinkya/anaconda3/envs/tensorflowGPU_p36/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/ajinkya/anaconda3/envs/tensorflowGPU_p36/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/ajinkya/anaconda3/envs/tensorflowGPU_p36/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  np_resource = np.dtype([("resource", np.ubyte, 1)])
Using TensorFlow backend.
WARNING:tensorflow:From main.py:10: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-05-14 10:40:17.583273: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-05-14 10:40:17.623709: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3193675000 Hz
2020-05-14 10:40:17.624878: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x562e8e163e60 executing computations on platform Host. Devices:
2020-05-14 10:40:17.624924: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
WARNING:tensorflow:From /home/ajinkya/.local/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:4070: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.

WARNING:tensorflow:From /home/ajinkya/Downloads/mrcnn_serving_ready-master/model.py:337: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From /home/ajinkya/Downloads/mrcnn_serving_ready-master/model.py:395: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From /home/ajinkya/Downloads/mrcnn_serving_ready-master/model.py:419: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From /home/ajinkya/Downloads/mrcnn_serving_ready-master/model.py:719: The name tf.sets.set_intersection is deprecated. Please use tf.sets.intersection instead.

WARNING:tensorflow:From /home/ajinkya/Downloads/mrcnn_serving_ready-master/model.py:721: The name tf.sparse_tensor_to_dense is deprecated. Please use tf.sparse.to_dense instead.

WARNING:tensorflow:From /home/ajinkya/Downloads/mrcnn_serving_ready-master/model.py:771: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
Traceback (most recent call last):
  File "main.py", line 108, in <module>
    model.load_weights(H5_WEIGHT_PATH, by_name=True)
  File "/home/ajinkya/Downloads/mrcnn_serving_ready-master/model.py", line 2131, in load_weights
    saving.load_weights_from_hdf5_group_by_name(f, layers)
  File "/home/ajinkya/.local/lib/python3.6/site-packages/keras/engine/saving.py", line 1309, in load_weights_from_hdf5_group_by_name
    ' element(s).')
ValueError: Layer #2 (named "conv1") expects 2 weight(s), but the saved weights have 1 element(s).

What is happening here ? any thoughts to get serving model from this script?

Hi There.
Have you got some sample code to predict from the pb model using the coco sample h5?