wearefarmgeek / diplomat Goto Github PK
View Code? Open in Web Editor NEWA HTTP Ruby API for Consul
License: BSD 3-Clause "New" or "Revised" License
A HTTP Ruby API for Consul
License: BSD 3-Clause "New" or "Revised" License
I am wanting to use this library with Puppet, but we don't want to have to compile native extensions on our production machines before puppet can run. This is currently required for us to use diplomat, because of the dependency on the json
gem.
Would it be possible to move to using json_pure
instead?
I have a single test server. When using Diplomat::Status.leader(), it raises an exception.
Consul 0.7.0
Diplomat 2.0.1
$ curl -XGET http://localhost:8500/v1/status/leader
"10.0.2.15:8300"
irb(main):001:0> require 'diplomat'
=> true
irb(main):007:0> Diplomat::Status.leader()
JSON::ParserError: 784: unexpected token at '"10.0.2.15:8300"'
from /usr/lib/ruby/2.3.0/json/common.rb:156:in `parse'
from /usr/lib/ruby/2.3.0/json/common.rb:156:in `parse'
from /var/lib/gems/2.3.0/gems/diplomat-2.0.1/lib/diplomat/status.rb:11:in `leader'
from /var/lib/gems/2.3.0/gems/diplomat-2.0.1/lib/diplomat/rest_client.rb:43:in `method_missing'
from (irb):7
from /usr/bin/irb:11:in `<main>'
Consul returns http service calls in a consistent order, when 'usually' one doesn't want all clients to use the same instance. When requesting just one instance of a service, return a random instance from the list of results.
In the same way consul DNS round robins services by IP address, it would be nice if diplomat would rotate all the available services for one service name (in this case, round robining w/ both IP and port rather than just IP as consul's DNS works).
Diplomat::Service.next('myservice')
When running Diplomat::Service.get() only the first result is returned.
It would be handy if there was a way to return all services as provided by the returned JSON from the API.
Maybe in the interests of backward compatibility an additional "all" parameter could be required in order to do this? Thoughts?
My code tests fail with:
expected no Exception, got #<NoMethodError: undefined method `empty?' for nil:NilClass> with backtrace:
# ./.bundle/ruby/2.3.0/bundler/gems/diplomat-dd4eb143e598/lib/diplomat/kv.rb:242:in `decode_transaction'
# ./.bundle/ruby/2.3.0/bundler/gems/diplomat-dd4eb143e598/lib/diplomat/kv.rb:237:in `transaction_return'
# ./.bundle/ruby/2.3.0/bundler/gems/diplomat-dd4eb143e598/lib/diplomat/kv.rb:174:in `txn'
# ./.bundle/ruby/2.3.0/bundler/gems/diplomat-dd4eb143e598/lib/diplomat/rest_client.rb:43:in `method_missing'
I instrumented the failing line:
237: def transaction_return(raw_return, options)
=> 238: binding.pry
239: decoded_return =
240: options && options[:decode_values] == false ? raw_return : decode_transaction(raw_return)
241: OpenStruct.new decoded_return
242: end
and saw this:
[1] pry(#<Diplomat::Kv>)> raw_return
=> {"Results"=>nil,
"Errors"=>
[{"OpIndex"=>1, "What"=>"failed index check for key \"our_path\", current modify index 8 != 7"}]}
Suggested fix is #133.
Hello John! Thanks so much for your nicely done diplomat gem. I've begun using it, along with consul and consul-template, and am very happy. I wrote a new class, and a few new methods, that I'd like you to review for inclusion. Sorry - my workstation is old, so I'm not setup to do pull requests, and haven't begun writing tests yet... that's the next hurdle I hope to attack shortly.
In the meantime, you can find my proposed files here:
http://speak.smashrun.com/assets/images/dip.tgz
I tried to keep your general sense of things. You'll immediately see the new "key_exist" method, and the new "Agent" class, which I added due to the /v1/agent/ API structure.
let me know what you think! Thanks again.
Steve
@cixelsyd
Trying to fetch a non existent key from Consul's Key/Value store using the following:
Diplomat::Kv.get("unexistent_key", nil, :wait, :wait)
Raises the following error:
Faraday::ResourceNotFound: the server responded with status 404
From the Diplomat::Kv.get
documentation:
@param not_found [Symbol] behaviour if the key doesn't exist; # :reject with exception, :return degenerate value, or :wait for it to appear
The behaviour should've been to keep blocking until the key exists.
Checking out the code, it seems like the problem originates from Diplomat::RestClient#start_connection
(lib/diplomat/rest_client.rb
):
def start_connection api_connection=nil
@conn = build_connection(api_connection)
@conn_no_err = build_connection(api_connection, true)
end
def build_connection(api_connection, raise_error=false)
@conn_no_error
is being set to a Faraday client which DOES raise an error (notice the second argument which is set to true
for build_connection
) as opposed to @conn
.
This is directly connected to:
lib/diplomat/kv.rb
def get key, options=nil, not_found=:reject, found=:return
@key = key
@options = options
url = ["/v1/kv/#{@key}"]
url += check_acl_token
url += use_consistency(@options)
# 404s OK using this connection
raw = @conn_no_err.get concat_url url # <= This statement raises the error
if raw.status == 404
@conn_no_err.get concat_url url
in the above raises the error since the Faraday client is set to raise errors.
@cablehead (who is the author of python-consul) mentioned on twitter that adding ModifyIndex & X-Consul-Index should be considered.
But first we need to decide how best to add this functionality in without breaking things too much.
@cablehead - any comments? ๐
Currently there are no tests for the Service class.
I don't have consol setup in development environment, so I want to use remote URL .. but seems like it doesn't work ..
Cannot load Rails.application.database_configuration
: (Faraday::ConnectionFailed)
Connection refused - connect(2) for "localhost" port 8500
Provide methods to wrap /v1/status/leader and /v1/status/peers endpoints.
Could it be possible to add the support for the /v1/agent endpoint ?
https://www.consul.io/docs/agent/http/agent.html
Regards
Using diplomat v0.15.0 and Rails v4.2.5.2 in a console session running:
pry(main)> Diplomat::Service.get('api')
LoadError: cannot load such file -- response/raise_error
from /Users/jfilipe/.rbenv/versions/2.3.0/lib/ruby/gems/2.3.0/gems/backports-3.6.8/lib/backports/std_lib.rb:9:in `require'
I've traced the exception to this line
diplomat/lib/diplomat/rest_client.rb
Line 64 in a5b5756
RaiseError
exception class.
If I update the above line to the following, everything works properly:
faraday.use Faraday::Response::RaiseError unless raise_error
I am trying to do a blocking Kv query and I get the following exception.
changed = Diplomat::Kv.get(prefix, { recurse: true }, :wait, :wait)
<NoMethodError> undefined method `first' for nil:NilClass
/Users/athompson/code/daemon_runner/vendor/gems/diplomat-1.0.0/lib/diplomat/rest_client.rb:96:in `return_value'
/Users/athompson/code/daemon_runner/vendor/gems/diplomat-1.0.0/lib/diplomat/kv.rb:99:in `get'
/Users/athompson/code/daemon_runner/vendor/gems/diplomat-1.0.0/lib/diplomat/rest_client.rb:46:in `method_missing'
At the moment a Diplomat.get('foo')
raise an exception if the 'foo' key does not exist in the key/value store.
This looks like an odd behaviour as I was expecting a nil
value (as would Redis or Memcached do for example).
I understand #16 do some work around this very thing but does not seem to change the end behavior. Is there any reason why the exception is raised rather than returning a nil value ?
When you have a consul ACL in place, the exceptions your get from diplomat are inconsistent:
E.g. Diplomat::Acl.list
raises
Diplomat::UnknownStatus: status 403
while Diplomat::Acl.create({})
raises
Faraday::ClientError: the server responded with status 403
which makes it messy to catch the exceptions and determine if authentication is needed.
This calling convention is painful:
Diplomat::Kv.get(KEY_PREFIX, {recurse: true}, :return)
The ruby interpreter expects to find a hash as the last option. It would be far better to roll the not_found
and found
options into the options hash itself and grab them with Hash#delete
.
Example call site:
Diplomat::Kv.get(KEY_PREFIX, recurse: true, not_found: :return)
It's also surprising that get
returns an empty string instead of nil
when the key is not found in not_found = :return
mode.
I'm not sure what the way forward is since there is a ton of existing code that relies on the existing behavior (including mine).
I am trying to use the service health api and noticed it doesn't support any of the query args[1]. I'm trying to use this to get healthy instances of a service. Or is there a better way to do this with diplomat?
I'd be happy to help out with this. It looks like this might end up looking like Node.get[2] or Service.get[3].
[1] https://consul.io/docs/agent/http/health.html
[2]
Lines 9 to 26 in de8a5b6
diplomat/lib/diplomat/service.rb
Lines 9 to 42 in 5f6563d
Current Diplomat::Kv.get
implementation does not allow to specify an ?index
param for fetching a key value if index
's value is smaller than the current key's ModifyIndex
and "wait" (block) if it equals to it.
This is extremely useful for implementing a watch
like functionality which avoids polling.
When getting the value of a key it is not possible to get all attributes.
It might be useful to get the session for example in the case that it is needed to discover a leader.
For my use case I want to treat the consul events like an event stream and get the newest one or sleep until a new event arrives. However, both .get
and .get_all
appear to drop the current newest event if run with :wait, :wait
. This means that I can either get the latest event or choose to possibly drop an event and wait for a new one.
Here's some ruby code that does what I want:
response = Excon.get(
URL,
query: {name: EVENT},
expects: [200],
connect_timeout: 5,
read_timeout: 5,
write_timeout: 5,
tcp_nodelay: true
)
handle_events(response.body)
loop do
index = response.headers['X-Consul-Index']
response = Excon.get(
URL,
query: {name: EVENT, index: index},
expects: [200],
connect_timeout: 5,
read_timeout: 86400,
write_timeout: 5,
tcp_nodelay: true
)
handle_events(response.body)
end
I then sift the newest event against a consul key that stores the last event LTime I worked off.
I expected to be able to use Diplomat to solve this problem.
If you have something like /this/is/my/path/value1 = 1
and you use diplomat with
values = Diplomat::Kv.get('/this/is/my/path', {recurse: true}, :return)
it will return a string value, value1, not and array
[{'/this/is/my/path/value1' => 1}]
Hey guys, first I got to say that it's a great gem!
Unfortunately I found a bug that I can not pinpoint exactly what causes it, in my opinion it's related to Faraday but I couldn't find the commit that updated the Faraday gem.
What basically happens is that when I'm using version 0.5.1 I am able to use Diplomat with a middleware to talk to consul over SSL but with 0.6.1 I can not.
I'm using a self-signed ssl using a CA I created, used this tutorial to get it running,
https://www.digitalocean.com/community/tutorials/how-to-secure-consul-with-tls-encryption-on-ubuntu-14-04
Steps to reproduce:
# config.json
{
"server": true,
"ports": {
"dns": -1,
"https": 8500,
"http": -1
},
"encrypt": "NH4Ykhc4aqzdsSO8Rduchg==",
"ca_file": "/src/research/diplomat/ca.cert",
"cert_file": "/src/research/diplomat/consul.cert",
"key_file": "/src/research/diplomat/consul.key",
"verify_incoming": true,
"verify_outgoing": true
}
# test.rb
require 'rubygems'
require 'diplomat'
class DiplomatSSLMiddleware
def initialize(app)
@app = app
end
def call(env)
env.ssl = {
ca_file: "/src/research/diplomat/ca.cert",
client_cert: OpenSSL::X509::Certificate.new(File.read("/src/research/diplomat/consul.cert")),
client_key: OpenSSL::PKey::RSA.new(File.read("/src/research/diplomat/consul.key"))
}
@app.call(env)
end
end
Diplomat.configure do |config|
config.url = "https://local.example.com:8500"
config.middleware = DiplomatSSLMiddleware
end
Diplomat.get('test')
I created a SSL certificate for local.exampe.com
and edited my local /etc/hosts
to pinpoint it to localhost.
Running in a terminal window the following command to have a consul server running
consul agent -data-dir=/src/research/diplomat -server -bootstrap -config-file=/src/research/diplomat/config.json
Running bundle exec ruby test.rb
On 0.6.1
:
SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (Faraday::SSLError)
On 0.5.1
:
the server responded with status 404 (Faraday::ResourceNotFound)
The session can't be renewed. I looks like a typo on line 47 of session.rb.
Currently:
req.url = "/v1/session/renew/#{id}"
and should be:
req.url "/v1/session/renew/#{id}"
When trying to get a list of services for an explicit datacenter, I found if the DC wasn't found, or otherwise invalid, the service request throws a 500 error and doesn't give any useful feedback.
Could this be setup to be more explicit Diplomat/Consul related exception ?
irb(main):006:0> Diplomat::Service.get('redis?dc=no_such_dc')
Faraday::ClientError: the server responded with status 500
from /var/lib/gems/1.9.1/gems/faraday-0.9.1/lib/faraday/response/raise_error.rb:13:in `on_complete'
from /var/lib/gems/1.9.1/gems/faraday-0.9.1/lib/faraday/response.rb:9:in `block in call'
from /var/lib/gems/1.9.1/gems/faraday-0.9.1/lib/faraday/response.rb:57:in `on_complete'
from /var/lib/gems/1.9.1/gems/faraday-0.9.1/lib/faraday/response.rb:8:in `call'
from /var/lib/gems/1.9.1/gems/faraday-0.9.1/lib/faraday/request/url_encoded.rb:15:in `call'
from /var/lib/gems/1.9.1/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:56:in `call'
from /var/lib/gems/1.9.1/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:139:in `build_response'
from /var/lib/gems/1.9.1/gems/faraday-0.9.1/lib/faraday/connection.rb:377:in `run_request'
from /var/lib/gems/1.9.1/gems/faraday-0.9.1/lib/faraday/connection.rb:140:in `get'
from /var/lib/gems/1.9.1/gems/diplomat-0.12.0/lib/diplomat/service.rb:22:in `get'
from /var/lib/gems/1.9.1/gems/diplomat-0.12.0/lib/diplomat/rest_client.rb:46:in `method_missing'
from (irb):6
from /usr/bin/irb:12:in `<main>'
We have a weird situation in that we need to use a proxy for some traffic and not for other traffic.
Diplomat (or farady in this case?) seems to pick up my ENV["HTTP_PROXY"] variables and is using that; I want diplomat to NOT use the proxy.
Is there a way to easily disable the proxy?
The README is currently very out of date and missing several of the now available endpoints.
This seems to get cause by diplomat nowdas:
WARNING: Unexpected middleware set after the adapter. This won't be supported from Faraday 1.0.
Hi -
Starting with v0.7.0, the following breaks:
require 'diplomat'
Diplomat::Service.get('mysql').Address
You get the following error:
ArgumentError: wrong number of arguments (0 for 1..4)
from /var/lib/gems/2.0.0/gems/diplomat-0.7.0/lib/diplomat/service.rb:13:in `get'
from /var/lib/gems/2.0.0/gems/diplomat-0.7.0/lib/diplomat/service.rb:47:in `get'
After downgrading to v0.6.1 the error disappeared.
Would be awesome if you'd like to add a change log:
http://keepachangelog.com/
Just tried out Diplomat, it's awesome! Also a big fan of the photos! On top of everything else I noticed @PeterFCaswell have an excellent taste in avatar photos ๐
The maintenance class is missing some of the available calls as well as should be updated to use the agent API endpoint.
42 files inspected, 1849 offenses detected
As per https://www.consul.io/docs/agent/http/kv.html, the recurse
option is allowed to delete all keys with the given prefix.
I read the service.rb code, It seems that It does send http request to get server list every time
foo_service = Diplomat::Service.get('foo')
is invoked.
If I have a application with high traffic. does it will cause high pressure to consul cluster?
why don't cache the service url and its server list and start a daemon thread to update it?
When I use the recurse version and there is only one value set in any of the child paths, e.g.
Diplomat::Kv.put('a/b/c', 'test')
Diplomat::Kv.get('a/b', {recurse: true}, :return, :return)
The the output is just
test
and I am missing the information from which childpath/key the value came from. This behaviour is implemented at https://github.com/WeAreFarmGeek/diplomat/blob/master/lib/diplomat/rest_client.rb#L79-L81
Currently, a service lookup goes like this:
mail_service = Diplomat::Service.get('mail')
mail_service.inspect # <OpenStruct Node="i-0c9693e4", Address="10.10.20.220", ServiceID="mail", ServiceName="mail", ServiceTags=[], ServiceAddress="", ServicePort=8080, ServiceEnableTagOverride=false, CreateIndex=4398409, ModifyIndex=4549606>
host, port = sage_service.Address, sage_service.ServicePort
This syntax is unlike anything else in ruby, where properties and methods are snake_cased
and only classes are CamelCased
.
One would expect something like this:
mail_service = Diplomat::Service.get('mail')
mail_service.inspect # <OpenStruct node="i-0c9693e4", address="10.10.20.220", service_id="mail", service_name="mail", service_tags=[], service_address="", service_port=8080, service_enable_tag_override=false, create_index=4398409, modify_index=4549606>
host, port = sage_service.address, sage_service.service_port
Overall Diplomat is excellent! โต๏ธ
But this part could use some adjustments to make it align with all of Rubys standard-library plus basically all other gems.
A related method naming issue aside: since Service.get
is called it feels a bit tautological to then call the method service_port
instead of just port
(same for service_tags
, service_name
, etc).
I think we need to add require 'ostruct'
. Weird that no one ran into this issue. Thoughts?
Kv.get
with recurse: true is not consistent if there is only single key were returned.
I am expecting array of key/values, but if there is only single key it would return just its value.
Diplomat::Kv.put('hello/world', 'test')
Diplomat::Kv.get('hello', recurse: true)
# current:
# => 'test'
# expected
# => [{ key: 'hello/world', value: 'test'}]
Current workaround is specifying decode_values: true
:
Diplomat::Kv.get('hello', recurse: true, decode_values: true)
# => {[ 'Key' => 'hello/world', 'Value' => 'test', ... }]
But there is no way to make convert_to_hash
to work with single key atm.
It seems that a few Diplomat functions do not use the acl_token that is set during configuration.
Looking at the register method definition from service.rb
def register(definition, path='/v1/agent/service/register')
json_definition = JSON.dump(definition)
register = @conn.put path, json_definition
return register.status == 200
end
I see that the acl_token definition is missing because it is neither part of query string not set as a http header. This means that service and node registrations/deregistartions can not happen when acls are in play using Diplomat.
I have temporarily fixed this by modifying the register definition per below - but this may not be the most optimal way given the rest of the code flow
def register(definition, path='/v1/agent/service/register')
path += "?#{check_acl_token[0]}" unless check_acl_token.empty?
json_definition = JSON.dump(definition)
register = @conn.put path, json_definition
return register.status == 200
end
Most of the other put action use different syntax that adds the acl_token if it exists.
I was hoping this is something that you may want to fix and push push out a new gem. Thanks!
According to https://www.consul.io/docs/agent/http/health.html the possible parameters for /v1/health/service/:service
are:
dc
(implemented correctly)near
(missing)passing
(implemented as state
, which does not seem to work)I am trying to filter only healthy nodes for a service, which does not seem to work without client-side post-processing.
Am I missing something? Is this version specific?
Is there a way to get ModifyIndex value from Diplomat::Kv.get
?
In Diplomat::Kv.put
, it accept the :cas
to do atomic update, but I can't find a way to get the ModifyIndex value..
When calling Diplomat::Acl.info(a_uuid)
, the method raise an exception if the acl does not exist.
The expected output is to return nil
instead.
With consul 0.5.2:
JSON::ParserError: 757: unexpected token at 'null'
'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/json-1.8.3/lib/json/common.rb:155:inparse' from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/json-1.8.3/lib/json/common.rb:155:in
parse'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/diplomat-0.15.0/lib/diplomat/rest_client.rb:76:inparse_body' from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/diplomat-0.15.0/lib/diplomat/acl.rb:30:in
info'
from /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/diplomat-0.15.0/lib/diplomat/rest_client.rb:46:inmethod_missing' from (irb):9 from /opt/chef/embedded/bin/irb:11:in
In this case, the body is the string "null" instead of being empty/nil.
It would be very helpful if tags/releases were created when a new version of the gem was cut.
The following line:
https://github.com/WeAreFarmGeek/diplomat/blob/master/lib/diplomat/rest_client.rb#L18
sends all output to STDOUT.
That's all well and good in a Rails app, but when running from a commandline Ruby script this pollutes the output to the terminal. It would be great if there was a way to specify the log output to go elsewhere, or at the very least suppress it entirely.
Please release current master to rubygems :).
This is required to register external services
https://www.consul.io/docs/agent/http.html under 'Consistency Modes'
related to #120 (introduced by this in 2.0.1) .. probably related to the concerns here #119 (comment)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.