ytoolshed / pogo

Pogo is an agent-based system for running interruptive commands safely on thousands of machines in parallel

NAME

Pogo - Run commands on many hosts in a controlled manner

DESCRIPTION

Pogo is a highly scalable system for running arbitrary commands on many hosts in a controlled manner.

It is mostly used for quick mass software deployments on server farms while making sure only an allowed number of nodes are upgraded in parallel to ensure business continuity.

Project Setup

Pogo is hosted on Github at

https://github.com/ytoolshed/pogo

The latest stable version can be found on the master branch. The project is automatically being tested on every commit, using travis-ci's service:

http://travis-ci.org/#!/ytoolshed/pogo

Architecture

Pogo consists of several components, which can be all running on the same system, or, in order to scale it, be replicated and even be installed on many distributed hosts. Those components are

Client

Users submit jobs to pogo using the client, which in turn contacts the API.

API

Takes requests via HTTP from the client and forwards them to a dispatcher.

Dispatcher

Takes job requests from the API, figures out constraints, and determines single tasks the job consists of. It then assigns tasks to workers, watches their individual completion and keeps track of overall job completion. Dispatchers can be queried by the API to determine the status of a given job.

Worker

Takes a task (like "ssh to a host and run this command") from the dispatcher, executes it and reports back the result. Can handle many tasks concurrently.

Security

To make sure dispatchers and workers communicate over secure channels, and enable them to authenticate each other (is a connecting worker really an authorized worker, or is the dispatcher it's connecting to really an authorized dispatcher?), Pogo uses SSL server and client certs. See Pogo::Security for details.

NAME

pogo-one - All-In-One Pogo

SYNOPSIS

$ cat t/cfgs/one.cfg
tag:
  # no tags
sequence:
  - host3
  - host2
  - host1

  # specify two hosts 1 and 2, but config says to run them in 
  # reverse order
$ perl -Ilib bin/pogo-one --password --targets host1,host2 \
    --config t/cfgs/one.cfg --run-command 'date'
Password: ****
host2: Running target
host2: Success
host1: Running target
host1: Success

DESCRIPTION

pogo-one is a simple utility that loads all the usually distributed Pogo components into one single process and runs it from the command line. In this way, you can test Pogo functions like constraints and sequences without having to set up the different components of an entire production system. It is mainly used for testing, but also helps with small deployment jobs.

PASSWORDS

By default, pogo-one assumes that you have access to the target boxes without having to type your password or your ssh keyring's passphrase. To have pogo-one collect the password and use it to sign in on the target, use the --password options

$ pogo-one --password ...
Password: ****
...

OPTIONS

--config file

Location of configuration file that defineds Pogo tags, sequences, and constraints.

--password

If specified, c<pogo-one> will collect the user's target password at a command line prompt, and use it to log into the target machines.

--targets

All hosts to run the command on. Order and parallelism is determined by the configuration file.

--run-command

The command to run on all hosts.

NAME

pogo-schedule - Print out the schedule for a job

SYNOPSIS

$ cat t/cfgs/triple.cfg
# sample configuration file

tag:
  colo:
    one:
      - host1
      - host2
      - host3
    two:
      - host4
      - host5
      - host6
    three:
      - host7
      - host8
      - host9
sequence:
  foo:
    - $colo.one
    - $colo.two
  bar:
    - $colo.three

$ perl -Ilib bin/pogo-schedule t/cfgs/triple.cfg
.--------------------------------------------------------------.
|                           Schedule                           |
+------------------+---------------------+---------------------+
| Thread           | slot-1              | slot-2              |
+------------------+---------------------+---------------------+
| thread-000000000 | [foo.$colo.one]     | [foo.$colo.two]     |
|                  | host3, host2, host1 | host6, host5, host4 |
| thread-000000001 | [bar.$colo.three]   |                     |
|                  | host9, host8, host7 |                     |
| unconstrained    | [unconstrained]     |                     |
'------------------+---------------------+---------------------'

DESCRIPTION

This utility pretty-prints a schedule for a job, as it would be executed based on a Pogo configuration file. It uses the actual Pogo scheduler in Pogo::Scheduler::Classic and asks it for an execution plan just as if we submitted a Pogo job for all hosts present in the configuration.

LICENSE

Copyright (c) 2010-2012 Yahoo! Inc. All rights reserved.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and imitations under the License.

AUTHORS

Mike Schilli <[email protected]> Ian Bettinger <[email protected]>

Many thanks to the following folks for implementing the original version of Pogo:

Andrew Sloane <[email protected]>, Michael Fischer <[email protected]>, Nicholas Harteau <[email protected]>, Nick Purvis <[email protected]>, Robert Phan <[email protected]>, Srini Singanallur <[email protected]>, Yogesh Natarajan <[email protected]>

pogo's Issues

Restore arrays as arrays after json-decoding Zookeeper configs

Config data is stored in Zookeeper with its arrays converted to hashes, so ["a", "b"] becomes { 0 => "a", 1 => "b" }.

To make sure the config data gets restored in its original format after getting it back from Zookeeper, I've added numhash_to_array_r() in mschilli/pogo@1f59cee.

bring back 'offline' and 'exclude'

these were removed in the 2.0-3.0 rewrite, but there's still a good use case. bring them back!

new ui code can't display output.

attempting to fetch gives me:

URLs: [http://localhost:8081/static/p0000000001/foo11.example.com.0.txt]
Proxying 0-0 from http://localhost:8081/static/p0000000001/foo11.example.com.0.txt
Status: 0
HEADERS:
Non-200 response!

in the yui log.

Support headless (multiple pluggable?) authentication scheme

Pogo/Limo currently use expect'ed passwords as an authentication mechanism. I'd like to see a mechanism that enabled headless secure actions to be taken so that automation can be built around pogo.

retrying a task on an expired job fails silently in the UI

I retried a task from last week, and now pogo is stuck forever retrying that task and failing:

2011-01-10 11:21:17,091 DEBUG pogo-dispatcher [Pogo::Dispatcher::poll@133] skipping task for p0000000004, no secrets found

This has continued well past the job timeout.

Safe eval() for stub code

The stub code used eval("...") to read in the data structure in the DATA
section, which is unsafe, as someone could slip in executable code. Replaced
by Safe::reval. mschilli/pogo@aacf766

host metadata ui element.

create a ui for the hostinfo api, to help with debugging host metadata issues.

possibly also create a ui element in the output window showing the output host's metadata - do we write that into zookeeper someplace? it could change between jobs.

Include copyright info

job log entries are sometimes truncated, resulting in incorrect jobsnapshots

smoke tests have some nondeterministic behaviors

for example, smoke_job tests if there are idle, connected workers, but on slow systems, the worker process spawned by PogoTester.pm may not yet be connected and marked idle by the dispatcher before this test runs.

jobstatus header is screwy.

DEBUG Client.pm:57 AUTOLOAD() http://localhost:8080/v3 request: ["jobstatus","p0000000002"]
DEBUG Client.pm:63 AUTOLOAD() response: [
   {
      "hosts" :
"Pogo::Engine::Response=HASH(0x10202b728),Pogo::Engine::Response=HASH(0x10202b728),Pogo::Engine::Response=HASH(0x10202b728),Pogo::Engine::Response=HASH(0x10202b728),Pogo::Engine::Response=HASH(0x10202b728),Pogo::Engine::Response=HASH(0x10202b728),Pogo::Engine::Response=HASH(0x10202b728),Pogo::Engine::Response=HASH(0x10202b728),Pogo::Engine::Response=HASH(0x10202b728),Pogo::Engine::Response=HASH(0x10202b728)",
      "status" : "OK",
      "version" : "V3",
      "action" : "jobstatus",
      "hostname" : "toki.local"
   }

bundle zookeeper with Store::ZooKeeper

zookeeper itself is pretty self-contained, rather than requiring that users install and configure zookeeper-server (which can be a PITA), we could use our experience from the test suite to bundle/configure/start/stop zookeeper in the main pogo product.

I don't believe this creates any licensing issues since both zookeeper and pogo are apache2.0

add detailed worker stats

to go along with the dispatcher stats

honor sequences for apps with no constraints.

from Pogo/Engine/Namespace.pm filter_apps():
142 # FIXME: if you have sequences for which there are no constraints, we might
143 # improperly ignore them here. Can't think of a straightforward fix and
144 # can't tell whether it will ever be a problem, so leaving for now.

The future of pogo

There are many sysadmin automation tools.
CFengine(C)
Puppet, Chef, capistrano (Ruby)
Fabric, Cobbler (Python)
etc..

I'm sorry Perl was originally born to be a sysadmin language, but Perl don't seem to have a resonable equivalent to them.
Despite I've found http://code.google.com/p/slack/ and http://www.steve.org.uk/Software/slaughter/ are in Perl. but They are not well organized and advertized.

I think pogo could be a resonable Perl equivalent to cfengine,puppet,fabric..etc.
What do you think about that?

ps.
Do you have any plans to upload pogo to CPAN?
There is already http://search.cpan.org/dist/Pogo/ module :(

make crap work

import initial pogo stuff, de-funked.

resume broken due to differing behavior in $slot->reserve()