GithubHelp home page GithubHelp logo

jeffmccune / ncio Goto Github PK

View Code? Open in Web Editor NEW
4.0 4.0 2.0 69 KB

Puppet Node Classifier backup / restore and transformation of hostnames

Home Page: https://www.openinfrastructure.co

License: MIT License

Ruby 98.23% Shell 1.77%

ncio's People

Contributors

jeffmccune avatar

Stargazers

avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar

Forkers

geoffwilliams dylanratcliffe

ncio's Issues

error running on older rubies

If user accidentally installed gem and ran inside an older ruby, the attached error will occur. The workaround is of course to use puppet's supplied modern version of ruby:

/opt/puppetlabs/puppet/bin/gem install ncio
/opt/puppetlabs/puppet/bin/ncio backup
/usr/share/ruby/syslog/logger.rb:177:in `initialize': wrong number of arguments (2 for 0..1) (ArgumentError)
	from /usr/local/share/gems/gems/ncio-2.0.0/lib/ncio/support.rb:28:in `new'
	from /usr/local/share/gems/gems/ncio-2.0.0/lib/ncio/support.rb:28:in `syslog_logger'
	from /usr/local/share/gems/gems/ncio-2.0.0/lib/ncio/support.rb:20:in `reset_logging!'
	from /usr/local/share/gems/gems/ncio-2.0.0/lib/ncio/support.rb:80:in `reset_logging!'
	from /usr/local/share/gems/gems/ncio-2.0.0/lib/ncio/app.rb:41:in `reset!'
	from /usr/local/share/gems/gems/ncio-2.0.0/lib/ncio/app.rb:34:in `initialize'
	from /usr/local/share/gems/gems/ncio-2.0.0/exe/ncio:4:in `new'
	from /usr/local/share/gems/gems/ncio-2.0.0/exe/ncio:4:in `<top (required)>'
	from /usr/local/bin/ncio:23:in `load'
	from /usr/local/bin/ncio:23:in `<main>'

[feature] retry-wait with timeout if puppetmaster is not up yet

Overview

If puppetserver is still loading, ncio will fail with an error. It would be cool if ncio could automatically retry the command while the server is booting until it either succeeds or times out.

Current error

If puppet server is down, the error looks like this at present

root@pe-puppet:/# ncio backup
/opt/puppetlabs/puppet/lib/ruby/2.1.0/net/http.rb:879:in `initialize': Connection refused - connect(2) for "pe-puppet.localdomain" port 4433 (Errno::ECONNREFUSED)

Improvement

We could catch Errno::ECONNREFUSED add a switch --connect-timeout and retry say every 5 seconds until we either hit the connection timeout or get a hard fail/success from the classifier

[feature/idea] customer groups

I had an idea earlier about using NCIO to upgrade between different versions of PE by dumping just customer's rules, omitting the build in ones.

This might let us do something like dump the customer-owned rules from say PE 2016.4.2 and then loading them into PE 2017.3.0 when its becomes available, without having to worry about importing 'old' built-in rules and breaking things.

Would this be a useful feature? I this might work as long as customers have another way to get any change they need into the puppet-owned rules - say through puppet code or some other tool that just deals with upgrades and installations

ncio hangs backups hang with PE 2017.3.5

Overview

Using latest PE 2017.3.5 on CentOS 7.4 I'm experiencing hangs 100% of the time when running ncio backup. Also occurs on PE 2017.3.2.

Environment

Running PE inside a docker container - I've done this many times before so don't think this is the problem

[root@pe-puppet /]# ncio --version
ncio 2.0.0 (c) 2016 Jeff McCune

Expected result

Expected to be able to dump the classifier database

Actual result

ncio backup hangs until killed (ctrl +c). Equivalent curl request before and after failure works so the server is operational

[root@pe-puppet /]# ncio --debug --no-syslog backup
W, [2018-04-19T05:55:51.031613 #11796]  WARN -- : Starting Node Classification Backup using GET https://pe-puppet.localdomain:4433/classifier-api/v1/groups
^CF, [2018-04-19T05:55:52.390937 #11796] FATAL -- : ERROR Obtaining backup: {
  "error": "Interrupt",
  "message": "",
  "backtrace": [
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/protocol.rb:176:in `wait_readable'",
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/protocol.rb:176:in `rbuf_fill'",
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/protocol.rb:154:in `readuntil'",
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/protocol.rb:164:in `readline'",
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http/response.rb:40:in `read_status_line'",
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http/response.rb:29:in `read_new'",
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:1446:in `block in transport_request'",
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:1443:in `catch'",
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:1443:in `transport_request'",
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:1416:in `request'",
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:1409:in `block in request'",
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:877:in `start'",
    "/opt/puppetlabs/puppet/lib/ruby/2.4.0/net/http.rb:1407:in `request'",
    "/opt/puppetlabs/puppet/lib/ruby/gems/2.4.0/gems/ncio-2.0.0/lib/ncio/http_client.rb:77:in `request'",
    "/opt/puppetlabs/puppet/lib/ruby/gems/2.4.0/gems/ncio-2.0.0/lib/ncio/api/v1.rb:77:in `request_without_timeout'",
    "/opt/puppetlabs/puppet/lib/ruby/gems/2.4.0/gems/ncio-2.0.0/lib/ncio/api/v1.rb:86:in `request'",
    "/opt/puppetlabs/puppet/lib/ruby/gems/2.4.0/gems/ncio-2.0.0/lib/ncio/api/v1.rb:101:in `groups'",
    "/opt/puppetlabs/puppet/lib/ruby/gems/2.4.0/gems/ncio-2.0.0/lib/ncio/app.rb:81:in `backup_groups'",
    "/opt/puppetlabs/puppet/lib/ruby/gems/2.4.0/gems/ncio-2.0.0/lib/ncio/app.rb:57:in `run'",
    "/opt/puppetlabs/puppet/lib/ruby/gems/2.4.0/gems/ncio-2.0.0/exe/ncio:5:in `<top (required)>'",
    "/opt/puppetlabs/puppet/bin/ncio:23:in `load'",
    "/opt/puppetlabs/puppet/bin/ncio:23:in `<main>'"
  ]

Analysis

Spent some time debugging this today and have verified the cause of this as the use of chunked encoding to make the REST calls.

It's possible this is related to the container based PE instance I'm using although I don't know why this would be the case. Anyone else seeing this issue?

Workaround

File: /lib/ncio/api/v1.rb
Comment line 26, eg:

      DEFAULT_HEADERS = {
        'Content-Type' => 'application/json',
#        'Transfer-Encoding' => 'chunked'
      }.freeze

After this change, backup command works instantly, as expected

[improvement] half-installed puppet masters give un-authenticated error

Overview

In the current version of PE (2016.2.0), the certificate whitelist is not updated until the initial puppet run has been completed. If puppet orchestrator has not been enabled on the master, then any puppet code using its new language features will generate a syntax error (there is a ticket on this ...somewhere), preventing puppet from running and thus preventing us from being able to run NCIO at all.

It took me a while to figure this out so it would be great to express the above to the user somehow.

Message

Currently, users encountering the above situation receive the message:

[root@pupper-sbxr101 vagrant]# ncio backup
/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/ncio-1.0.1/lib/ncio/api/v1.rb:68:in `groups': Expected 200 response, got 401 body: {"kind":"puppetlabs.rbac/user-unauthenticated","msg":"Route requires authentication","redirect":"/classifier-api/v1/groups?inherited=false"} (Ncio::Api::V1::ApiError)
    from /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/ncio-1.0.1/lib/ncio/app.rb:75:in `backup_groups'
    from /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/ncio-1.0.1/lib/ncio/app.rb:56:in `run'
    from /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/gems/ncio-1.0.1/exe/ncio:5:in `<top (required)>'
    from /opt/puppetlabs/puppet/bin/ncio:23:in `load'
    from /opt/puppetlabs/puppet/bin/ncio:23:in `<main>'

Improvement

Perhaps catch the exception (being careful not to mask errors from bad file permissions) and give a message like:

Error:  Route requires authentication
Make sure the certificate from file #{CERTFILE} is in the certificate whitelist and that you are able to run puppet on the master

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.