GithubHelp home page GithubHelp logo

bosh-workspace's Introduction

Bosh workspace

Build Status Test Coverage Code Climate Dependency Status Stories in Ready

Depreciation Notice

BOSH workspace is intimately related to how deployment manifests were handled with BOSH v1. At the time, most projects were providing Bash-based ad-hoc toolchains for building large YAML deployment manifests out of somewhat smaller YAML pieces called Spiff templates and stubs.

In this landscape, BOSH Workspace introduced an effort for providing a standard toolchain around Spiff, giving a chance for better organizing Spiff templates and stubs into what could be called “infrastructure-as-code“ git repositories, that could precisely describe staging and production environments managed by BOSH.

Later, the development of BOSH Workspace has been abandoned in favor of other tools like Genesis v1, then v2. And on the BOSH side, the v2 CLI has deprecated Spiff and thus the way BOSH Workspace works. Plus, reference deployment manifests are progressively being distributed as *-deployment (example: bosh, cf) git repos containing a base BOSH v2 deployment along with operation files that implement variants around the base deployment. So, fewer and fewer projects are shipping Spiff templates anymore.

So, newer ways of organizing BOSH deployment manifests have emerged like Genesis that has pivoted to Genesis v2 to support BOSH v2 or Gstack BOSH Environment (or GBE for intimates) that is natively built around the BOSH v2 CLI.

Introduction

BOSH Workspace is in essence a bosh v1 CLI plugin for managing an organized layout of Spiff stubs and templates that are the basis of typical BOSH v1 deployments. Such organized layout helps in managing consistently the various infrastructure-as-code environments you might have, like “sandbox”, “pre-prod” or ”production”, all made of BOSH deployments that are similar but not exactly the same.

BOSH Workspace aslo noticeably introduces new handly verbs in the bosh v1 CLI that make it easier to deploy things. Indeed, running bosh prepare deployment automates the uploading of BOSH Releases and BOSH Stemcells (the required bits for a BOSH deployment) to the BOSH server.

For a good introduciton on the initial goals of the project (back in 2015), see the Introducing bosh-workspace: how we deploy all things BOSH video.

bws-intro

Getting started

Before you start make sure ruby, bundler and spiff are available on your system. Instructions for installing spiff can found here.

Creating a workspace repository

First you will have to create a new repo for our company called Foo Group (short FG).

git init fg-boshworkspace
cd fg-boshworkspace

Lets create the initial files & directories.

mkdir deployments templates
echo -e 'source "https://rubygems.org"\n\ngem "bosh-workspace"' > Gemfile
echo "2.1.0" > .ruby-version
echo -e '.stemcells*\n.deployments*\n.releases*\n.stubs*\n' > .gitignore

Now install the gems by running bundler.

bundle install

Lets finish by making an initial commit.

git add .
git commit -m "Initial commit"

Creating a first deployment

For demonstration purposes we will deploy Cloud Foundry on bosh-lite. The steps below will show the bosh-workspace equivalent of bosh-lite manual deploy instructions.

Before we start make sure you have access to properly installed bosh-lite.

We will start by targetting our bosh-lite.

bosh target 192.168.50.4
bosh login admin admin

Now lets create our deployment file.

cat >deployments/cf-warden.yml <<EOL
---
name: cf-warden
director_uuid: current

releases:
  - name: cf
    version: latest
    git: https://github.com/cloudfoundry/cf-release.git

stemcells:
  - name: bosh-warden-boshlite-ubuntu-lucid-go_agent
    version: 60

templates:
  - cf/cf-deployment.yml
  - cf/cf-jobs.yml
  - cf/cf-properties.yml
  - cf/cf-resource-pools.yml
  - cf/cf-infrastructure-warden.yml
  - cf/cf-minimal-dev.yml

meta:
  default_quota_definitions:
    default:
      memory_limit: 102400 # Increased limit for demonstration purposes
EOL

Now lets use this deployment and upload it's dependencies.

bosh deployment cf-warden
bosh prepare deployment

Lets make sure to above template paths exist.

ln -s ../.releases/cf/templates templates/cf

To finish we only have to start the deployment process and commit our changes.

bosh deploy
git add . && git commit -m "Added cf-warden deployment"

Congratulations you should now have a running Cloud Foundry. For further reference on how to start using it go to the bosh-lite documentation.

Managing sandbox and production environments

The suggested way of doing this is to create many similar deployments in the deployments/ folder. They typically have tha same radix like cf or mysql as prefix of their name, and be distinguished by a -sandbox.yml suffix or -prod.yml depending on your environments names.

Then each of those similar deployment can express variants by including environment-specific Spiff stubs, or specifying specific configuration in the meta root YAML node. Easy as that. Examples of deployment variants can be found in cf-boshworkspace.

Using private boshreleases

When using a boshrelease from a location which requires authentication a .credentials.yml file is required, located at the root of your boshworkspace. Two types of authentication are supported: username/password and sshkey.

Example .credentials.yml file:

- url: https://github.com/example/top-secret-boshrelease.git
  username: foo
  password: bar
- url: ssh://[email protected]/example/super-secret-boshrelease.git
  private_key: |
    -----BEGIN RSA PRIVATE KEY-----
    MIICXAIBAAKBgQDHFr+KICms+tuT1OXJwhCUmR2dKVy7psa8xzElSyzqx7oJyfJ1
    JZyOzToj9T5SfTIq396agbHJWVfYphNahvZ/7uMXqHxf+ZH9BL1gk9Y6kCnbM5R6
    0gfwjyW1/dQPjOzn9N394zd2FJoFHwdq9Qs0wBugspULZVNRxq7veq/fzwIDAQAB
    AoGBAJ8dRTQFhIllbHx4GLbpTQsWXJ6w4hZvskJKCLM/o8R4n+0W45pQ1xEiYKdA
    Z/DRcnjltylRImBD8XuLL8iYOQSZXNMb1h3g5/UGbUXLmCgQLOUUlnYt34QOQm+0
    KvUqfMSFBbKMsYBAoQmNdTHBaz3dZa8ON9hh/f5TT8u0OWNRAkEA5opzsIXv+52J
    duc1VGyX3SwlxiE2dStW8wZqGiuLH142n6MKnkLU4ctNLiclw6BZePXFZYIK+AkE
    xQ+k16je5QJBAN0TIKMPWIbbHVr5rkdUqOyezlFFWYOwnMmw/BKa1d3zp54VP/P8
    +5aQ2d4sMoKEOfdWH7UqMe3FszfYFvSu5KMCQFMYeFaaEEP7Jn8rGzfQ5HQd44ek
    lQJqmq6CE2BXbY/i34FuvPcKU70HEEygY6Y9d8J3o6zQ0K9SYNu+pcXt4lkCQA3h
    jJQQe5uEGJTExqed7jllQ0khFJzLMx0K6tj0NeeIzAaGCQz13oo2sCdeGRHO4aDh
    HH6Qlq/6UOV5wP8+GAcCQFgRCcB+hrje8hfEEefHcFpyKH+5g1Eu1k0mLrxK2zd+
    4SlotYRHgPCEubokb2S1zfZDWIXW3HmggnGgM949TlY=
    -----END RSA PRIVATE KEY-----

Install Notes

OSX

cmake isneeded and libssh2 is optionally (only needed when using cloning over ssh)

brew install cmake libssh2 pkg-config

Ubuntu

cmake and libcurl4-openssl-dev is needed for rugged install

sudo apt-get install cmake libcurl4-openssl-dev libssh2-1-dev

Experimental

dns support

Dns support can be enabled by adding a domain_name property to your deployment. For example: domain_name: microbosh or if you are using a normal bosh just use bosh. When enabled, a transformation step will be executed after the spiff merge. Which will transform all the static ip references into domain names.

Contributing

  1. Fork it
  2. Create your feature branch (git checkout -b my-new-feature)
  3. Commit your changes (git commit -am 'Add some feature')
  4. Push to the branch (git push origin my-new-feature)
  5. Create new Pull Request

List of Contributors

bosh-workspace's People

Contributors

allomov avatar bgandon avatar fearoffish avatar funcode avatar geofffranks avatar ivan-sap avatar jamesclonk avatar landesherr avatar lnguyen avatar marwahaha avatar matthiaswinzeler avatar pronoiac avatar rkoster avatar sklevenz avatar teancom avatar tushar-dadlani avatar voelzmo avatar waffle-iron avatar wayneeseguin avatar weberstephanhd avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

bosh-workspace's Issues

Support release upload from URL / local file

Would it be possible to add the capability of uploading a given release by URL (and maybe also by local file) during prepare deployment?

Maybe something similar to the new bosh-init manifest format:

releases:
- name: bosh
  url: http://company_internal_host/github.com/cloudfoundry/bosh?v=155
  sha1: 875ac9a0df0bdcd449d612f07459b17c28304156
  path: release
- name: logsearch-shipper
  url: file://./releases/logsearch-shipper.tgz
  sha1: b6889630717ac4d9b25b407f887a01efe064b52c

The reasoning behind this is as you might have already guessed by looking at the first example http://company_internal_host/..., to be able to use bosh-workspace in an environment where you don't have internet access to download releases.
They are provided either via http through an internal-access-only host, or maybe even as a tgz file already available on the filesystem.

In such cases bosh-workspace should basically do a bosh upload release <URL|file> for these.

bosh deployment <deployment> - wrong director UUID

It seems that when running a bosh deployment <deployment>, bosh workspace is reading from .deployments/* and if it finds a file there it's using the director UUID from there, instead of the one configured in deployments/*.

In the sample case below the correct UUID would be 51e8dfeb-1de4-4cf5-abb1-b39e08673f56, as configured in deployments/cf.yml.
But because it found a (presumably old) file .deployments/cf.yml it picked the UUID from there, which was wrong.

$ cat deployments/cf.yml

---
name: my-cf
director_uuid: 51e8dfeb-1de4-4cf5-abb1-b39e08673f56
...
$ bundle exec bosh deployment cf
This manifest references director with UUID 8a88decc-b7b9-4533-975d-9cd8a20c9c3c.
You've never targeted it before.
Please find your director IP or hostname and target it first.
$ cat .deployments/cf.yml

---
director_uuid: 8a88decc-b7b9-4533-975d-9cd8a20c9c3c.
# Don't edit; placeholder deployment for: /home/user/bosh-workspace/deployments/cf.yml

It would be nice to handle the case where ssh is not supported.

Fetching release 'identity_service_broker' to satisfy template references
/var/lib/gems/2.0.0/gems/rugged-0.24.0b11/lib/rugged/repository.rb:210:in `fetch': Unsupported URL protocol (Rugged::NetworkError)
        from /var/lib/gems/2.0.0/gems/rugged-0.24.0b11/lib/rugged/repository.rb:210:in `fetch'
        from /var/lib/gems/2.0.0/gems/bosh-workspace-0.9.7/lib/bosh/workspace/release.rb:81:in `fetch_repo'
        from /var/lib/gems/2.0.0/gems/bosh-workspace-0.9.7/lib/bosh/workspace/release.rb:19:in `update_repo'
        from /var/lib/gems/2.0.0/gems/bosh-workspace-0.9.7/lib/bosh/cli/commands/prepare.rb:32:in `block in prepare_release_repos'
        from /var/lib/gems/2.0.0/gems/bosh-workspace-0.9.7/lib/bosh/cli/commands/prepare.rb:29:in `each'
        from /var/lib/gems/2.0.0/gems/bosh-workspace-0.9.7/lib/bosh/cli/commands/prepare.rb:29:in `prepare_release_repos'
        from /var/lib/gems/2.0.0/gems/bosh-workspace-0.9.7/lib/bosh/cli/commands/prepare.rb:19:in `prepare'
        from /var/lib/gems/2.0.0/gems/bosh_cli-1.3160.0/lib/cli/command_handler.rb:57:in `run'
        from /var/lib/gems/2.0.0/gems/bosh_cli-1.3160.0/lib/cli/runner.rb:59:in `run'
        from /var/lib/gems/2.0.0/gems/bosh_cli-1.3160.0/bin/bosh:19:in `<top (required)>'
        from /usr/local/bin/bosh:23:in `load'
        from /usr/local/bin/bosh:23:in `<main>'
staging@jumpbox:~/workspace/staging-azure$ less .credentials.yml
staging@jumpbox:~/workspace/staging-azure$ irb
irb(main):001:0> require 'rugged'
=> true
irb(main):002:0> Rugged.features
=> [:threads, :https]

Support director_uuid current

As a boshdeployment repo maintainer I want to be able to set director_uuid to current. Which would use the currently targeted director. This would simplify the getting started experience for new users.

To discuss:

  • Show warning when using current, about it not being recommended for production use.
  • Or don't allow the use of current when target is not warden (bosh-lite).
  • Or combination no warning when using warden

[deployment-patch branch] cannot prepare logsearch-boshrelease (develop branch)

https://github.com/logsearch/logsearch-boshrelease does not have a master branch; only a develop branch. Perhaps this is why I'm getting this error (using deployment-patch branch of bosh-workspace):

$ bosh prepare deployment

Fetching release 'logsearch' to satisfy template references
/Users/drnic/.rvm/gems/ruby-2.1.5/gems/rugged-0.22.0b5/lib/rugged/repository.rb:37:in `checkout_tree': Expected Rugged::Commit, Rugged::Tag or Rugged::Tree (TypeError)
    from /Users/drnic/.rvm/gems/ruby-2.1.5/gems/rugged-0.22.0b5/lib/rugged/repository.rb:37:in `checkout'
    from /Users/drnic/.rvm/gems/ruby-2.1.5/bundler/gems/bosh-workspace-4e49de6b1d42/lib/bosh/workspace/helpers/git_credentials_helper.rb:19:in `fetch_and_checkout'
    from /Users/drnic/.rvm/gems/ruby-2.1.5/bundler/gems/bosh-workspace-4e49de6b1d42/lib/bosh/workspace/helpers/git_credentials_helper.rb:7:in `fetch_or_clone_repo'
    from /Users/drnic/.rvm/gems/ruby-2.1.5/bundler/gems/bosh-workspace-4e49de6b1d42/lib/bosh/cli/commands/prepare.rb:30:in `block in prepare_release_repos'

[idea] Support parent deployments

Implementation:

  • generate deployment manifest for parent deployments
  • spiff merge [current deployment stub] [parent deployment manifest]
  • normal spiff merge with result from above

cf/templates/cf-lamb.yml links to file in cf-release submodule

Hi,

It seems that bosh-workspace does not do a "git submodule update --init" when populating the .releases folder with the release's versions.

As described in the BWS-README.md the templates folder of the boshworkspace contains a link with name "cf" to ".releases/cf/templates templates/cf"
In cf-release's templates folder is a link ".../.releases/cf/templates/cf-lamb.yml" to a cf submodule's file ".../.releases/cf/src/loggregator/manifest-templates/cf-lamb.yml". This file does not exist if the submodules were not updated and initialized after checkout.

Will 0.9.0 solve this "uninitialized releases' submodules" problem?
My deployment file looks like https://gist.github.com/weberstephanhd/4c31c6690c02fa6bbb26

Best regards, Stephan

Only first release get uploaded when multiple BOSH releases are referenced in deployment manifest

I’m using a BOSH workspace and inside the deployment manifest it references two BOSH releases . When I call bosh prepare deployment everything works OK with the first release but the second one doesn't get uploaded.
What I see in the logs is:

Uploading 'release/2'

Copying packages
----------------
release (ffa8618912f373724b8f4394aaedbe57dd07065c) MISSING

I'm quite new to Ruby, BOSH and CF but I think I figure out why this happens. The issue seems to be in release_helper.rb.

def release_cmd
   @release_cmd ||= Bosh::Cli::Command::Release::UploadRelease.new
end

ReleaseHelper has an instance variable release_cmd. This variable is instantiated only once and used for the upload. The variable has state and it still points to the first release when the second one is performed. A fix could be to instantiate UploadRelease for each release. Is there a reason to avoid multiple instances of UploadRelease?

TypeError when running prepare deployment

bosh prepare deployment

Fetching release 'diego' to satisfy template references
Version '0.1057.0' has been checkout into:
- /home/ubuntu/workspace/deployments/cf-boshworkspace/.releases/diego
Fetching release 'cf' to satisfy template references
/home/ubuntu/.rvm/gems/ruby-2.1.5/gems/rugged-0.23.0b1/lib/rugged/repository.rb:37:in `checkout_tree': Expected Rugged::Commit, Rugged::Tag or Rugged::Tree (TypeError)
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/rugged-0.23.0b1/lib/rugged/repository.rb:37:in `checkout'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.0.rc3/lib/bosh/workspace/helpers/git_credentials_helper.rb:19:in `fetch_and_checkout'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.0.rc3/lib/bosh/workspace/helpers/git_credentials_helper.rb:7:in `fetch_or_clone_repo'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.0.rc3/lib/bosh/cli/commands/prepare.rb:31:in `block in prepare_release_repos'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.0.rc3/lib/bosh/cli/commands/prepare.rb:28:in `each'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.0.rc3/lib/bosh/cli/commands/prepare.rb:28:in `prepare_release_repos'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh-workspace-0.9.0.rc3/lib/bosh/cli/commands/prepare.rb:18:in `prepare'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh_cli-1.2905.0/lib/cli/command_handler.rb:57:in `run'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh_cli-1.2905.0/lib/cli/runner.rb:56:in `run'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/gems/bosh_cli-1.2905.0/bin/bosh:16:in `<top (required)>'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/bin/bosh:23:in `load'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/bin/bosh:23:in `<main>'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in `eval'
        from /home/ubuntu/.rvm/gems/ruby-2.1.5/bin/ruby_executable_hooks:15:in `<main>'

workaround rm -rf .releases/cf
Happens when using 0.9.x with a workspace which has releases which have been cloned with 0.8.x bosh-workspace.

bosh prepare deployment raises error in project_deployment_helper.rb:18

$ bosh prepare deployment
/Users/sw/.rvm/gems/ruby-2.1.4/gems/bosh-workspace-0.8.5/lib/bosh/workspace/helpers/project_deployment_helper.rb:18:in 'project_deployment_file?': undefined method 'has_key?' for "<%\nrequire_relative "./lib/helper"":String (NoMethodError)
from /Users/sw/.rvm/gems/ruby-2.1.4/gems/bosh-workspace-0.8.5/lib/bosh/workspace/helpers/project_deployment_helper.rb:6:in 'project_deployment?'
from /Users/sw/.rvm/gems/ruby-2.1.4/gems/bosh-workspace-0.8.5/lib/bosh/workspace/helpers/project_deployment_helper.rb:22:in 'require_project_deployment'
from /Users/sw/.rvm/gems/ruby-2.1.4/gems/bosh-workspace-0.8.5/lib/bosh/cli/commands/prepare.rb:14:in 'prepare'
from /Users/sw/.rvm/gems/ruby-2.1.4/gems/bosh_cli-1.2922.0/lib/cli/command_handler.rb:57:in 'run'
from /Users/sw/.rvm/gems/ruby-2.1.4/gems/bosh_cli-1.2922.0/lib/cli/runner.rb:56:in 'run'
from /Users/sw/.rvm/gems/ruby-2.1.4/gems/bosh_cli-1.2922.0/bin/bosh:16:in '<top (required)>'
from /Users/sw/.rvm/gems/ruby-2.1.4/bin/bosh:23:in 'load'
from /Users/sw/.rvm/gems/ruby-2.1.4/bin/bosh:23:in 'main'
from /Users/sw/.rvm/gems/ruby-2.1.4/bin/ruby_executable_hooks:15:in 'eval'
from /Users/sw/.rvm/gems/ruby-2.1.4/bin/ruby_executable_hooks:15:in 'main'

I used a clean ruby 2.1.4 and only installed the following:
gem install bundler --no-rdoc --no-ri
gem install net-ssh --no-rdoc --no-ri
gem install bosh_cli --no-ri --no-rdoc
gem install bosh_cli_plugin_micro --no-ri --no-rdoc
gem install nats cf-uaac --no-ri --no-rdoc
gem install bosh-workspace --no-ri --no-rdoc

What am I missing?
Best regards, Stephan

using bosh-stemcell version 1.2.3 not allowed?

Hi,

we use stemcells and releases with versions like "1.2.3" - having two dots.
We use this syntax like 'our third patch of our fork which is based on the official 1.2 version'.
Bosh-workspace should be able to cope with that and even versions like "1.2-RC4"

Best regards, Stephan

Outdated assumption about releases folder breaks working with newer boshreleases

Since version 2682 the BOSH CLI creates a subfolder with with release's name in the releases folder: https://groups.google.com/a/cloudfoundry.org/forum/#!search/final$20release$20directory/bosh-users/W3A3OIeYN_E/yL0dIS0_V7gJ
This results in a structure where the actual foo-<version>.yml files are sitting in releases/foo/foo-<version>.yml instead of just releases/foo-<version>.yml

Currently, bosh-workspace only supports the old structure and cannot find the desired final version for boshreleases built with the new structure.

Some boshreleases, have worked around this by making releases/foo a symlink pointing to releases, see e.g. cf-release. However, if you don't control the boshrelease yourself, you cannot use bosh-workspace with it right now.

[idea] Colocation helper

Add a post spiff merge task which automatically colocates jobs based on the schema defined in the deployment file:

colocation:
  - name: data_z1
    jobs: [postgresql_z1, etcd_z1]
  - name: api_z1
    jobs: [api_z1, gorouter_z1, haproxy_z1]

Should also take care of updating static_ips used in properties.

Failed to set deployment when no previous deployment was set

bundle exec bosh deployment dev-pg-cf
/home/cf-internet/.gem/ruby/1.9.1/bundler/gems/bosh-manifests-7013017ce08e/lib/bosh/manifests/helpers/project_deployment_helper.rb:83:in `dirname': can't convert nil into String (TypeError)
    from /home/cf-internet/.gem/ruby/1.9.1/bundler/gems/bosh-manifests-7013017ce08e/lib/bosh/manifests/helpers/project_deployment_helper.rb:83:in `deployment_dir'
    from /home/cf-internet/.gem/ruby/1.9.1/bundler/gems/bosh-manifests-7013017ce08e/lib/bosh/manifests/helpers/project_deployment_helper.rb:77:in `project_deployment_file'
    from /home/cf-internet/.gem/ruby/1.9.1/bundler/gems/bosh-manifests-7013017ce08e/lib/bosh/manifests/helpers/project_deployment_helper.rb:5:in `project_deployment?'
    from /home/cf-internet/.gem/ruby/1.9.1/bundler/gems/bosh-manifests-7013017ce08e/lib/bosh/manifests/helpers/project_deployment_helper.rb:22:in `require_project_deployment'
    from /home/cf-internet/.gem/ruby/1.9.1/bundler/gems/bosh-manifests-7013017ce08e/lib/bosh/cli/commands/01_make_manifest.rb:20:in `set_current'
    from /home/cf-internet/.gem/ruby/1.9.1/gems/bosh_cli-1.1782.0/lib/cli/command_handler.rb:57:in `run'
    from /home/cf-internet/.gem/ruby/1.9.1/gems/bosh_cli-1.1782.0/lib/cli/runner.rb:56:in `run'
    from /home/cf-internet/.gem/ruby/1.9.1/gems/bosh_cli-1.1782.0/lib/cli/runner.rb:16:in `run'
    from /home/cf-internet/.gem/ruby/1.9.1/gems/bosh_cli-1.1782.0/bin/bosh:7:in `<top (required)>'
    from /home/cf-internet/bin/bosh:23:in `load'
    from /home/cf-internet/bin/bosh:23:in `<main>'

Support uploading stemcells

As a user I expect bosh prepare deployment to resolve and upload the specified stemcells. In oder to simplify the getting started experience.

Undefined method 'has_key?'

Hola,

after installing the bosh-workspace gem and using it with

$ bosh deployment logsearch-warden

I get the following error

/Library/Ruby/Gems/2.0.0/gems/bosh-workspace-0.9.0/lib/bosh/workspace/helpers/project_deployment_helper.rb:18:in `project_deployment_file?': undefined method `has_key?' for "<%\nrequire_relative \"./lib/helper\"":String (NoMethodError)
    from /Library/Ruby/Gems/2.0.0/gems/bosh-workspace-0.9.0/lib/bosh/cli/commands/project_deployment.rb:18:in `set_current'
    from /Library/Ruby/Gems/2.0.0/gems/bosh_cli-1.3042.0/lib/cli/command_handler.rb:57:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/bosh_cli-1.3042.0/lib/cli/runner.rb:56:in `run'
    from /Library/Ruby/Gems/2.0.0/gems/bosh_cli-1.3042.0/bin/bosh:19:in `<top (required)>'
    from /usr/bin/bosh:23:in `load'
    from /usr/bin/bosh:23:in `<main>'

Any suggestions what I am did wrong when installing the gem?

Cheers,
Fabian

Support multiline yaml in meta

Generating deployment manifest
Command failed: 'spiff merge /Users/rubenkoster/workspace/cf-boshdeployment/templates/cf-release/cf-deployment.yml /Users/rubenkoster/workspace/cf-boshdeployment/templates/cf-release/cf-jobs.yml /Users/rubenkoster/workspace/cf-boshdeployment/templates/cf-release/cf-properties.yml /Users/rubenkoster/workspace/cf-boshdeployment/templates/cf-release/cf-infrastructure-warden.yml /Users/rubenkoster/workspace/cf-boshdeployment/templates/warden-networks.yml /Users/rubenkoster/workspace/cf-boshdeployment/templates/cf-release/cf-minimal-dev.yml /Users/rubenkoster/workspace/cf-boshdeployment/.stubs/cf-warden.yml 2>&1'
2014/02/25 14:20:48 error parsing stub: YAML error: line 7: did not find URI escaped octet

Support uploading releases

As a user I expect bosh prepare deployment to resolve and upload the specified releases. In oder to simplify the getting started experience.

Referencing a local BOSH (dev) release in deployment file (manifest)

I am trying to reference a local BOSH (dev) release in my deployment file (manifest), however, I either don't completely understand how to do this, or I might just be doing it wrong.

Relevant information and versions:

bosh 1.2950.0
bosh-workspace 0.9.0

Let's say I started out with something like this:

deployment file (manifest):

---
name: xxx
director_uuid: xxx

releases:
  - name: cf
    version: 212
    git: https://github.com/cloudfoundry/cf-release.git

However, I needed to customize some things within the cf-release, and after doing so, I created a development release on my BOSH box.

bosh releases:

+------+------------+-------------+
| Name | Versions   | Commit Hash |
+------+------------+-------------+
| cf   | 212        | ae2ec7a5+   |
|      | 212+dev.1* | b30e0fd5+   |
+------+------------+-------------+
(*) Currently deployed
(+) Uncommitted changes

But, now I want to be able to run bosh prepare deployment while referencing my (own, local) BOSH (dev) release in my deployment file (manifest), which (I think) should look something like this:

deployment file (manifest):

---
name: xxx
director_uuid: xxx

releases:
  - name: cf
    version: 212+dev.1
    #git: https://github.com/cloudfoundry/cf-release.git
    #there should probably be some other reference here, to tell bosh-workspace where it can find the 212+dev.1 release?

Can you help me out here on how to get this working?
I don't have a (public or private) git repository to point to, so that might be an issue.
However, the BOSH director does hold this local BOSH (dev) release somewhere, and it is also on my local disk (cf/dev_releases/cf/212+dev.1.yml).

Ssh authentication not working on ubuntu

staging@jumpbox:~/workspace/staging-azure$ bosh prepare deployment
Fetching release 'identity_service_broker' to satisfy template references
/var/lib/gems/2.0.0/gems/rugged-0.24.0b11/lib/rugged/repository.rb:210:in `fetch': Failed to authenticate SSH session: Unable to extract public key from private key file: Method unimplemented in libgcrypt backend (Rugged::SshError)
        from /var/lib/gems/2.0.0/gems/rugged-0.24.0b11/lib/rugged/repository.rb:210:in `fetch'
        from /var/lib/gems/2.0.0/gems/bosh-workspace-0.9.7/lib/bosh/workspace/release.rb:81:in `fetch_repo'
        from /var/lib/gems/2.0.0/gems/bosh-workspace-0.9.7/lib/bosh/workspace/release.rb:19:in `update_repo'
        from /var/lib/gems/2.0.0/gems/bosh-workspace-0.9.7/lib/bosh/cli/commands/prepare.rb:32:in `block in prepare_release_repos'
        from /var/lib/gems/2.0.0/gems/bosh-workspace-0.9.7/lib/bosh/cli/commands/prepare.rb:29:in `each'
        from /var/lib/gems/2.0.0/gems/bosh-workspace-0.9.7/lib/bosh/cli/commands/prepare.rb:29:in `prepare_release_repos'
        from /var/lib/gems/2.0.0/gems/bosh-workspace-0.9.7/lib/bosh/cli/commands/prepare.rb:19:in `prepare'
        from /var/lib/gems/2.0.0/gems/bosh_cli-1.3160.0/lib/cli/command_handler.rb:57:in `run'
        from /var/lib/gems/2.0.0/gems/bosh_cli-1.3160.0/lib/cli/runner.rb:59:in `run'
        from /var/lib/gems/2.0.0/gems/bosh_cli-1.3160.0/bin/bosh:19:in `<top (required)>'
        from /usr/local/bin/bosh:23:in `load'
        from /usr/local/bin/bosh:23:in `<main>'

staging@jumpbox:~/workspace/staging-azure$ apt-show-versions libssh2-1-dev
libssh2-1-dev:amd64/trusty 1.4.3-2 uptodate

stemcell version: 2719.1 not supported

Today a patch of 2719 came out called 2719.1; but bosh deploy is failing.

$ bosh deploy
Validation errors:
- { stemcells => At index 0: { version => Object 2719.1 doesn't validate against any of #<Membrane::Schemas::Class:0x00000001ef4cb8>, #<Membrane::Schemas::Value:0x00000001ef4c90> } }
'/home/ubuntu/workspace/deployments/cf-boshworkspace/deployments/cf-aws-vpc.yml' is not valid

If deployment not targeted, then menu to select one from deployments folder

Currently, it is ugly if you try bosh prepare deployment but haven't targeted yet:

ubuntu@ip-10-10-0-37:~/workspace/deployments/cf-boshworkspace$ bosh prepare deployment
/usr/bin/traveling-bosh/lib/vendor/ruby/2.1.0/gems/bosh-workspace-0.8.5/lib/bosh/workspace/helpers/project_deployment_helper.rb:88:in `dirname': no implicit conversion of nil into String (TypeError)
        from /usr/bin/traveling-bosh/lib/vendor/ruby/2.1.0/gems/bosh-workspace-0.8.5/lib/bosh/workspace/helpers/project_deployment_helper.rb:88:in `deployment_dir'
        from /usr/bin/traveling-bosh/lib/vendor/ruby/2.1.0/gems/bosh-workspace-0.8.5/lib/bosh/workspace/helpers/project_deployment_helper.rb:82:in `project_deployment_file'
        from /usr/bin/traveling-bosh/lib/vendor/ruby/2.1.0/gems/bosh-workspace-0.8.5/lib/bosh/workspace/helpers/project_deployment_helper.rb:5:in `project_deployment?'
        from /usr/bin/traveling-bosh/lib/vendor/ruby/2.1.0/gems/bosh-workspace-0.8.5/lib/bosh/workspace/helpers/project_deployment_helper.rb:22:in `require_project_deployment'
        from /usr/bin/traveling-bosh/lib/vendor/ruby/2.1.0/gems/bosh-workspace-0.8.5/lib/bosh/cli/commands/prepare.rb:14:in `prepare'
        from /usr/bin/traveling-bosh/lib/vendor/ruby/2.1.0/gems/bosh_cli-1.2809.0/lib/cli/command_handler.rb:57:in `run'
        from /usr/bin/traveling-bosh/lib/vendor/ruby/2.1.0/gems/bosh_cli-1.2809.0/lib/cli/runner.rb:56:in `run'
        from /usr/bin/traveling-bosh/lib/vendor/ruby/2.1.0/gems/bosh_cli-1.2809.0/lib/cli/runner.rb:16:in `run'
        from /usr/bin/traveling-bosh/lib/vendor/ruby/2.1.0/gems/bosh_cli-1.2809.0/bin/bosh:7:in `<top (required)>'
        from /usr/bin/traveling-bosh/lib/vendor/ruby/2.1.0/bin/bosh:23:in `load'
        from /usr/bin/traveling-bosh/lib/vendor/ruby/2.1.0/bin/bosh:23:in `<main>'

use http_proxy/https_proxy if env vars are defined

bosh-workspace should use the env vars http_proxy and https_proxy during prepare deployment when fetching a release or a stemcell, so it can be used in an enviroment without direct internet access, where everything goes through an http proxy.

Doesn't support releases with non-integer versions

I have a release with version v0.1 but this fails for bosh-workspace:

$ bosh prepare deployment
Validation errors:
- { releases => At index 1: { version => Object 0.1 doesn't validate against any of #<Membrane::Schemas::Class:0x007f12be53c190>, #<Membrane::Schemas::Value:0x007f12be53c0f0> } }
'/home/ubuntu/workspace/deployments/cf-boshworkspace/deployments/cf-aws-tiny.yml' is not valid

prepare deployment fails because of bosh-release gocd/docker symlink

bosh-workspace freaks out during prepare deployment on bosh-release because of a symlink (gocd/docker)

Cleaning up .releases/ doesn't help.
Manually git cloning the bosh release into .releases/ doesn't help either.
It seems bosh-workspace is trying to switch to some other commithash where the gocd/docker->ci/docker link then becomes broken and trips up because of this.

$ bundle exec bosh prepare deployment

Fetching release 'bosh' to satisfy template references
/home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/rugged-0.23.0b1/lib/rugged/repository.rb:47:in `checkout_tree': Failed to make directory '/home/user/bosh-workspace/.releases/bosh/gocd/docker': No such file or directory (Rugged::OSError)
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/rugged-0.23.0b1/lib/rugged/repository.rb:47:in `checkout'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh-workspace-0.9.0.rc4/lib/bosh/workspace/release.rb:15:in `update_repo'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh-workspace-0.9.0.rc4/lib/bosh/cli/commands/prepare.rb:32:in `block in prepare_release_repos'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh-workspace-0.9.0.rc4/lib/bosh/cli/commands/prepare.rb:28:in `each'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh-workspace-0.9.0.rc4/lib/bosh/cli/commands/prepare.rb:28:in `prepare_release_repos'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh-workspace-0.9.0.rc4/lib/bosh/cli/commands/prepare.rb:18:in `prepare'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh_cli-1.2922.0/lib/cli/command_handler.rb:57:in `run'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh_cli-1.2922.0/lib/cli/runner.rb:56:in `run'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh_cli-1.2922.0/bin/bosh:16:in `<top (required)>'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/bin/bosh:23:in `load'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/bin/bosh:23:in `<main>'

git error when doing `bosh prepare deployment`

I tried to follow the example in the readme with the cf-warden deployment but I always get the git error:

$ bosh prepare deployment

Cloning release 'cf' to satisfy template references
~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/git-1.2.9.1/lib/git/lib.rb:918:in `command': git '--git-dir=~/GitHub/bosh-workspace/.releases/cf/.git' '--work-tree=~/GitHub/bosh-workspace/.releases/cf' log '-30' '--no-color' '--pretty=raw' 'releases/cf-195.yml'  2>&1:fatal: ambiguous argument 'releases/cf-195.yml': unknown revision or path not in the working tree. (Git::GitExecuteError)
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/git-1.2.9.1/lib/git/lib.rb:847:in `command_lines'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/git-1.2.9.1/lib/git/lib.rb:145:in `full_log_commits'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/git-1.2.9.1/lib/git/log.rb:119:in `run_log'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/git-1.2.9.1/lib/git/log.rb:112:in `check_log'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/git-1.2.9.1/lib/git/log.rb:89:in `first'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/bosh-workspace-0.8.5/lib/bosh/workspace/release.rb:59:in `version_ref'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/bosh-workspace-0.8.5/lib/bosh/workspace/release.rb:18:in `update_repo'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/bosh-workspace-0.8.5/lib/bosh/cli/commands/prepare.rb:29:in `block in prepare_release_repos'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/bosh-workspace-0.8.5/lib/bosh/cli/commands/prepare.rb:27:in `each'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/bosh-workspace-0.8.5/lib/bosh/cli/commands/prepare.rb:27:in `prepare_release_repos'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/bosh-workspace-0.8.5/lib/bosh/cli/commands/prepare.rb:17:in `prepare'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/bosh_cli-1.2811.0/lib/cli/command_handler.rb:57:in `run'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/bosh_cli-1.2811.0/lib/cli/runner.rb:56:in `run'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/bosh_cli-1.2811.0/lib/cli/runner.rb:16:in `run'
  from ~/.rbenv/versions/2.0.0/lib/ruby/gems/2.0.0/gems/bosh_cli-1.2811.0/bin/bosh:7:in `<top (required)>'
  from ~/.rbenv/versions/2.0.0/bin/bosh:23:in `load'
  from ~/.rbenv/versions/2.0.0/bin/bosh:23:in `<main>'

If execute the git command in the shell i get the same error. If I add -- before the path to the yml file it works.

The following change in lib/bosh/workspace/release.rb:59 for me solved the problem.

    def version_ref
-     @repo.log.object(manifest).first
+     @repo.log.path(manifest).first
    end

missing credentials file error, though no credentials are needed

Our grafana-boshworkspace uses a grafana-boshrelease from a public git repo. No credentials are needed for cloning/fetching and with bosh-workspace 0.8.5 everything works fine.

Using bosh-workspace 0.9.0 on the same bws I get the following error with bosh prepare deployment:

"Authentication is required for: https://github.wdf.sap.corp/cloudfoundry/grafana-boshrelease.git
Credentials file does not exist: /home/vcap/landscape-dev06/bosh-workspaces/grafana-boshworkspace/.credentials.yml".

best regards, Stephan

apply deployment patch error because of git submodule

bosh-workspace aborts with an error when applying a deployment patch in a bosh-workspace where there's a git submodule (the submodule is unrelated to bosh-workspace).

$ bundle exec bosh apply deployment patch patch/bosh.yml
/home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh-workspace-0.9.0.rc4/lib/bosh/cli/commands/deployment_patch.rb:69:in `add_all': 'ci/apcctl.rb' appears as both a file and a directory (Rugged::IndexError)
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh-workspace-0.9.0.rc4/lib/bosh/cli/commands/deployment_patch.rb:69:in `commit_all'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh-workspace-0.9.0.rc4/lib/bosh/cli/commands/deployment_patch.rb:29:in `apply'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh_cli-1.2922.0/lib/cli/command_handler.rb:57:in `run'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh_cli-1.2922.0/lib/cli/runner.rb:56:in `run'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/gems/bosh_cli-1.2922.0/bin/bosh:16:in `<top (required)>'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/bin/bosh:23:in `load'
        from /home/user/bosh-workspace/.vendor/bundle/ruby/2.2.0/bin/bosh:23:in `<main>'

In this example I had a submodule ci in my bosh-workspace directory.

It seems though it actually applied the patch yml.
Upon rerunning the exact same command the error does not occur anymore.

I think bosh-workspace should not care about any possible submodules, except for the templates.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.