RemoveToxic: HTTP 503: Unexpected response code, expected 204 about toxiproxy

[sergey-kurenkov]

/tp # ./toxiproxy-cli --version
toxiproxy-cli version 2.5.0
/tp # ./toxiproxy --version
toxiproxy-server version 2.5.0
/tp # 

[sergey-kurenkov]

This is a config file:

[
  {
    "name": "gengw_svc",
    "listen": "0.0.0.0:19102",
    "upstream" : "localhost:9102",
    "enabled": true
  },
  {
    "name": "gengw_mdt",
    "listen": "0.0.0.0:19103",
    "upstream" : "localhost:9103",
    "enabled": true
  }
]

[miry]

@sergey-kurenkov It is not clear how toxiproxy-server is deployed from your example. I see those only client logs. Would like to provide a example how I could reproduce the problem on local machine?

[miry]

hm, actually it could be related to https://github.com/Shopify/toxiproxy/pull/422/files
It means there is a resource locked, that prevent to remove resource.

Context:

In 2.4 it was a report when using of Java client the toxiproxy was stuck to remove specific proxy. It was some sort race condition, when mutating the proxy and traffic is going on.
It was introduced few PRs: return api response as fast as possible 503 and force drop connection after some time.

[miry]

cc: @dianadevasia