cloud-annotations / custom-training

👷‍♀️ Build and train your own model

Cloud Annotations

Important Note: Due to unforseen circumstances, the hosted version of Cloud Annotations (cloud.annotations.ai) is no longer available.

For a similar hosted experience, but focused on education/learning, see Computer Vision Learning Studio.

For instructions on how to run Cloud Annotations using the local file system, see the experimental Iris project.

custom-training's Issues

Training Failed

Hi,

I'm trying to train model as per guideline of cloud annotation cli.
Following steps has been done

1. Created object storage and bucket
2. Created ML instance
3. using cloud.annotation.ai
   3.a. --> Uploaded Images
   3.b. --> Labeled Images
4. Using Cloud Annotation Cli
   4.a --> Initialised cli with credentials and training configuration (gpu=v100x2 & steps=500/1000/5000/10000)
   4.b --> Started training using cacli train trainer.zip

Once all above steps done --> Progress shows with messages
Preparing to train (this may take a while)...
message shows around 30 min and then done.

But whenever i started the training it is failing.
I have my company account as well as personal account too.
On the both account training get failed with following error. (Error extracted from IBM Cloud storage bucket)

Gone through training-status.json on cloud bucket and following error return

"status": {
      "state": "error",
      "finished_at": "2019-07-23T06:31:31.903Z",
      "submitted_at": "2019-07-23T06:26:57.293Z",
      "error": {
        "trace": "",
        "errors": [{
          "code": "dl_job_failed (S100)",
          "message": "Internal error (S100)",
          "more_info": "http://watson-ml-api.mybluemix.net/"
        }]
      },
      "message": "training-a072AJHZg: INSUFFICIENT_RESOURCES",
      "metrics": [],
      "current_at": "2019-07-23T06:31:32.784Z",
      "error_cause": "system"
    }

Error Received through ibm cli

Training with training/test data at:
DATA_DIR: /mnt/data/prod.bucket
MODEL_DIR: /job/model-code
TRAINING_JOB: 
TRAINING_COMMAND: cd "$(dirname "$(find . -name "start.sh" -maxdepth 2 | head -1)")" && ./start.sh 1000
Storing trained model at:
RESULT_DIR: /mnt/results/prod.bucket.train/training-Sw9aGBNZR
Wed Jul 24 06:58:31 UTC 2019: Running Tensorflow job
/usr/local/bin/train.sh: line 38: ./start.sh: Permission denied
Training exited with error code 126
Failed: learner_exit_code: 126

I did tried searching error in IBM docs but there is no luck and not getting any details about it.

Please respond as we are using IBM platform for production object detection purpose.

Thanks

Convert & Download Web Model is missing

Hi,
While doing custom training.
Training done properly however, Convert to web / tflite part is missing in script.
Is there any way to convert model after training. I have trained model and waited for almost 12 hrs for 200K Steps using faster r-cnn.

Please help. Otherwise i have to start training again.
Thanks