GithubHelp home page GithubHelp logo

facebook / grocery-delivery Goto Github PK

View Code? Open in Web Editor NEW
153.0 46.0 55.0 88 KB

The Grocery Delivery utility for managing cookbook uploads to distributed Chef backends.

License: Apache License 2.0

Ruby 84.62% Shell 15.38%

grocery-delivery's Introduction

Grocery Delivery

Continuous Integration

Intro

Ohai!

Welcome to Grocery Delivery, software to keep cookbooks, roles, and databags in sync between a VCS repo and a chef server. The idea is that if you have multiple, distinct Chef server instances that should all be identical, they can all run this script in cron. The script uses proper locking, so you should be able to run it every minute.

However, there are several things to know:

  • It assumes you don't leverage versions or environments.
  • It assumes you want anything committed to HEAD to be uploaded immediately.

Grocery Delivery is pretty customizable. Many things can be tuned from a simple config file, and it's pluggable so you can extend it as well.

Prerequisites

Grocery Delivery is a particular way of managing your Chef infrastructure, and it assumes you follow that model consistently. Here are the basic principals:

  • Checkins are live immediately (which implies code review before merge)
  • Versions are meaningless (ideally, never change them)
  • You want all your chef-servers in sync
  • Everything you care about comes from version control.

We recommend enabling attributes whitelisting to prevent node attributes being saved back to the server. In recent versions of Chef (past Chef 11), this feature is built-in. For older versions, we recommend using the whitelist_node_attrs cookbook.

Dependencies

  • Mixlib::Config
  • BetweenMeals

Config file

The default config file is /etc/gd-config.rb but you may use -c to specify another. The config file works the same as client.rb does for Chef - there are a series of keywords that take an argument and anything else is just standard Ruby.

The following configuration options are available:

  • berks

    Description: Determines if we should use Berkshelf to resolve deps, upload cookbooks

    CLI Argument(s): '-b', '--berks'

    Value data type: Boolean

    Default: false

    Notes: Existence of CLI argument equates to true

  • berks_bin

    Description: Path to Berkshelf binary.

    CLI Argument(s): '-B', '--berks-bin FILE'

    Value data type: String

    Default: /opt/chefdk/bin/berks

  • berks_config

    Description: Path to Berkshelf config.

    CLI Argument(s): '--berks-config FILE'

    Value data type: String

    Default: nil

  • config_file

    Description: Path to Grocery Delivery config

    CLI Argument(s): -c, --config-file FILE

    Value data type: String

    Default: /etc/gd-config.rb

  • cookbook_paths

    Description: Space-delimited array of dirs that contain cookbooks relative to reponame

    CLI Argument(s): '--cookbook-paths DIRECTORY1 DIRECTORY2 DIRECTORY3 ...'

    Value data type: String

    Default: ['chef/cookbooks']

  • databag_path

    Description: A directory to find databags in relative to reponame.

    CLI Argument(s): '-d', '--databag-path FILE'

    Value data type: String

    Default: 'chef/databags'

  • dry_run

    Description: Enable dry-run mode.

    CLI Argument(s): '-n', '--dry-run'

    Value data type: Boolean

    Default: false

    Notes: Existence of CLI argument equates to true

  • knife_bin

    Description: Path to knife binary.

    CLI Argument(s): '-k', '--knife-bin FILE'

    Value data type: String

    Default: /opt/chef/bin/knife

    Notes: e.g. /var/chef/grocery_delivery_work/ops/chef/cookbooks

  • knife_config

    Description: Knife config to use for uploads.

    CLI Argument(s): '-K', '--knife-config FILE'

    Value data type: String

    Default: /root/.chef/knife.rb

    Notes: knife.rb will need to set cookbook_path pointing to the cookbook path in the work directory

  • lockfile

    Description: Path to lockfile.

    CLI Argument(s): '-l', '--lockfile FILE'

    Value data type: String

    Default: /var/lock/subsys/grocery_delivery

  • master_path

    Description: The top-level path for Grocery Delivery's work (most other paths are relative to this).

    CLI Argument(s): '-m', '--master-path FILE'

    Value data type: String

    Default: /var/chef/grocery_delivery_work

  • pidfile

    Description: Path to pidfile.

    CLI Argument(s): -p, --pidfile FILE

    Value data type: String

    Default: /var/run/grocery_delivery.pid

  • plugin_path

    Description: Path to plugin file.

    CLI Argument(s): '-P', '--plugin-path FILE'

    Value data type: String

    Default: /etc/gd-plugin.rb

  • repo_update

    Description: Enables/disables cookbook repo update before run.

    CLI Argument(s): '-U', '--no-repo-update'

    Value data type: Boolean

    Default: true

    Note: Existence of CLI argument equates to false. This might be beneficial for those wanting to run Grocery Delivery in CI, within a container, or some other non-standard environment where there is no need to update the cookbook repository before running Grocery Delivery.

  • repo_url

    Description: The URL to clone/checkout if it doesn't exist.

    CLI Argument(s): '-u', '--repo-url URL'

    Value data type: String

    Default: nil

  • reponame

    Description: The relative directory to check the repo out to, inside of master_path.

    CLI Argument(s): '-N', '--repo-name'

    Value data type: String

    Default: ops

  • rev_checkpoint

    Description: File with last-uploaded revision, relative to reponame.

    CLI Argument(s): '-C', '--revision-checkpoint FILE'

    Value data type: String

    Default: gd_revision

  • role_path

    Description: A directory to find roles in relative to reponame.

    CLI Argument(s): '-r', '--role-path FILE'

    Value data type: String

    Default: 'chef/roles'

  • role_type

    Description: RB or JSON roles?

    CLI Argument(s): '-R', '--role-type TYPE'

    Value data type: String

    Default: rb

  • stdout

    Description: Log to stdout as well.

    CLI Argument(s): '--stdout'

    Value data type: Boolean

    Default: false

    Notes: Existence of CLI argument equates to true

  • track_symlinks

    Description: Whether or not to track symlinks.

    CLI Argument(s): '--track-symlinks'

    Value data type: Boolean

    Default: false

    Notes: Existence of CLI argument equates to true

  • vcs_path

    Description: Path to git or svn binary.

    CLI Argument(s): '--vcs-path FILE'

    Value data type: String

    Default: nil

    Notes: If not given, just uses 'git' or 'svn'

  • vcs_type

    Description: Git or SVN?

    CLI Argument(s): '--vcs-type TYPE'

    Value data type: String

    Default: svn

  • verbosity

    Description: Verbosity level.

    CLI Argument(s): '-v', '--verbosity'

    Value data type: N/A

    Default: WARN

    Notes: Specify twice via CLI for debug.

Plugin

The plugin should be a ruby file which defines several class methods. It is class_eval()d into a Hooks class.

The following functions can optionally be defined:

  • self.preflight_checks(dryrun)

This code will run once we've read our config and loaded our plugins but before anything else. We don't even have a lock yet. Dryrun is a bool which indicates if we are in dryrun mode.

  • self.prerun(dryrun)

This is run after we've gotten a lock, written a pidfile and initialized our repo object (but not touched the repo yet)

  • self.post_repo_up(dryrun)

This is code to run after we've updated the repo, but before we've done any work to parse it.

  • self.postrun(dryrun, success, msg)

After we've parsed the updates to the repo and uploaded/deleted the relevant items from the local server. Success is a bool for whether we succeeded, and msg is the status message - either the revision we sync'd or an error.

  • self.atexit(dryrun, success, msg)

Same as postrun, but is registered as an atexit function so it happens even if we crash.

License

See the LICENSE file in this repo.

grocery-delivery's People

Contributors

bwann avatar chefaustin avatar dafyddcrosby avatar davide125 avatar dbagaev avatar djakobsson avatar ifel avatar jason-cooke avatar jaymzh avatar joshuamiller01 avatar krzkowalczyk avatar lmaghran avatar malmond77 avatar mikedodge04 avatar naomireeves avatar odcinek avatar pavlo-fb avatar promisedlandt avatar wjimenez5271 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

grocery-delivery's Issues

Install Instructions for Chef Automate

The days of chef server are gone.

Chef Automate bundles Infra and /opt/opscode/ is no longer part of the installation.

Version: 2
  CLI Build: 20221006204445
  Server Build: 4.3.0

I tried looking through the new DIR's and found that chef automate \ infra is now installed under /hab/*

For the life of me, i cannot find a relative directory or /embedded/ to install GD. Hoping there's something I don't know and that GD is supported on the latest version.

The closes thing i found was...

root@chef:/hab/pkgs/core/ruby27/2.7.5/20220312100031/bin# ls
bundle  bundler  erb  gem  irb  racc  racc2y  rake  rdoc  ri  ruby  update_rubygems  y2racc

Undefined method 'level' for GroceryDelivery::Log::Module

It might be something I'm doing wrong in my latest setup but I'm getting this repeated error on the latest version of chefdk and chef-workstation.

OS Version: Ubuntu 16.04.6 LTS
grocery_delivery Gem: 0.0.8

Chef-DK Version: 4.8.38

WARN: Dryrun mode activated, no changes will be made.
WARN: Attempting to acquire lock
WARN: Lock acquired
WARN: [DRYRUN] Would do: Updating repo
ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.6.0/gems/between_meals-0.0.11/lib/between_meals/knife.rb:44:in `initialize'
ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.6.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery:151:in `new'
ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.6.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery:151:in `upload_changed'
ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.6.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery:281:in `'
ERROR: /opt/chefdk/embedded/bin/grocery-delivery:23:in `load'
ERROR: /opt/chefdk/embedded/bin/grocery-delivery:23:in `'
ERROR: /opt/chefdk/embedded/lib/ruby/2.6.0/bundler/cli/exec.rb:74:in `load'
ERROR: /opt/chefdk/embedded/lib/ruby/2.6.0/bundler/cli/exec.rb:74:in `kernel_load'
ERROR: /opt/chefdk/embedded/lib/ruby/2.6.0/bundler/cli/exec.rb:28:in `run'
ERROR: /opt/chefdk/embedded/lib/ruby/2.6.0/bundler/cli.rb:463:in `exec'
ERROR: /opt/chefdk/embedded/lib/ruby/2.6.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
ERROR: /opt/chefdk/embedded/lib/ruby/2.6.0/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
ERROR: /opt/chefdk/embedded/lib/ruby/2.6.0/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
ERROR: /opt/chefdk/embedded/lib/ruby/2.6.0/bundler/cli.rb:27:in `dispatch'
ERROR: /opt/chefdk/embedded/lib/ruby/2.6.0/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
ERROR: /opt/chefdk/embedded/lib/ruby/2.6.0/bundler/cli.rb:18:in `start'
ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.2/exe/bundle:30:in `block in '
ERROR: /opt/chefdk/embedded/lib/ruby/2.6.0/bundler/friendly_errors.rb:124:in `with_friendly_errors'
ERROR: /opt/chefdk/embedded/lib/ruby/gems/2.6.0/gems/bundler-1.17.2/exe/bundle:22:in `'
ERROR: /opt/chefdk/embedded/bin/bundle:23:in `load'
ERROR: /opt/chefdk/embedded/bin/bundle:23:in `'
WARN: undefined method `level' for GroceryDelivery::Log:Module

Chef Workstation Version: 20.6.62

WARN: Dryrun mode activated, no changes will be made.
WARN: Attempting to acquire lock
WARN: Lock acquired
WARN: [DRYRUN] Would do: Updating repo
ERROR: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/between_meals-0.0.11/lib/between_meals/knife.rb:44:in `initialize'
ERROR: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery:151:in `new'
ERROR: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery:151:in `upload_changed'
ERROR: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery:281:in `'
ERROR: /opt/chef-workstation/embedded/bin/grocery-delivery:23:in `load'
ERROR: /opt/chef-workstation/embedded/bin/grocery-delivery:23:in `'
ERROR: /opt/chef-workstation/embedded/lib/ruby/2.7.0/bundler/cli/exec.rb:63:in `load'
ERROR: /opt/chef-workstation/embedded/lib/ruby/2.7.0/bundler/cli/exec.rb:63:in `kernel_load'
ERROR: /opt/chef-workstation/embedded/lib/ruby/2.7.0/bundler/cli/exec.rb:28:in `run'
ERROR: /opt/chef-workstation/embedded/lib/ruby/2.7.0/bundler/cli.rb:476:in `exec'
ERROR: /opt/chef-workstation/embedded/lib/ruby/2.7.0/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
ERROR: /opt/chef-workstation/embedded/lib/ruby/2.7.0/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
ERROR: /opt/chef-workstation/embedded/lib/ruby/2.7.0/bundler/vendor/thor/lib/thor.rb:399:in `dispatch'
ERROR: /opt/chef-workstation/embedded/lib/ruby/2.7.0/bundler/cli.rb:30:in `dispatch'
ERROR: /opt/chef-workstation/embedded/lib/ruby/2.7.0/bundler/vendor/thor/lib/thor/base.rb:476:in `start'
ERROR: /opt/chef-workstation/embedded/lib/ruby/2.7.0/bundler/cli.rb:24:in `start'
ERROR: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/libexec/bundle:46:in `block in '
ERROR: /opt/chef-workstation/embedded/lib/ruby/2.7.0/bundler/friendly_errors.rb:123:in `with_friendly_errors'
ERROR: /opt/chef-workstation/embedded/lib/ruby/gems/2.7.0/gems/bundler-2.1.4/libexec/bundle:34:in `'
ERROR: /opt/chef-workstation/embedded/bin/bundle:23:in `load'
ERROR: /opt/chef-workstation/embedded/bin/bundle:23:in `'
WARN: undefined method `level' for GroceryDelivery::Log:Module

support git submodules

Hi,

It would be awesome if GD supported git submodules. This way various teams could have repositories with their own databags / cookbooks.
Also, some kind of whitelisting would be useful in that case:

  • git submodule -> list of databags or cookbooks allowed to be synchronized from this submodule.

Please consider, thanks.

Changes not reflected in chef-manage/knife cookbook list

/var/chef/grocery_delivery_work/ops/cookbooks has the up-to-date chef data after a grocery delivery run but chef manage doesn’t reflect the changes. I don't think the knife upload is occurring correctly.

How should I investigate this discrepancy?

grocery fails to delete cookbooks when you have multiple versions

Deleting one of the two versions allowed the process to resume

Here's a trace :

grocery-delivery[8434]: WARN: Attempting to acquire lock
grocery-delivery[8434]: WARN: Lock acquired
grocery-delivery[8434]: WARN: Updating repo
grocery-delivery[8434]: WARN: Determing changes... from a to b
grocery-delivery[8434]: WARN: Added cookbooks: [“xxxxxx”]
grocery-delivery[8434]: WARN: Deleted cookbooks: ["simple_iptables"]
grocery-delivery[8434]: WARN: Added roles: []
grocery-delivery[8434]: WARN: Deleted roles: []
grocery-delivery[8434]: WARN: Added databags: []
grocery-delivery[8434]: WARN: Deleted databags: []
grocery-delivery[8434]: ERROR: /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/mixlib-shellout-2.0.0/lib/mixlib/shellout.rb:267:in `invalid!'
grocery-delivery[8434]: ERROR: /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/mixlib-shellout-2.0.0/lib/mixlib/shellout.rb:254:in `error!'
grocery-delivery[8434]: ERROR: /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/between_meals-0.0.4/lib/between_meals/util.rb:42:in `exec!'
grocery-delivery[8434]: ERROR: /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/between_meals-0.0.4/lib/between_meals/knife.rb:86:in `block in cookbook_delete'
grocery-delivery[8434]: ERROR: /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/between_meals-0.0.4/lib/between_meals/knife.rb:85:in `each'
grocery-delivery[8434]: ERROR: /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/between_meals-0.0.4/lib/between_meals/knife.rb:85:in `cookbook_delete'
grocery-delivery[8434]: ERROR: /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/grocery_delivery-0.0.4/bin/grocery-delivery:122:in `partial_upload'
grocery-delivery[8434]: ERROR: /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/grocery_delivery-0.0.4/bin/grocery-delivery:149:in `upload_changed'
grocery-delivery[8434]: ERROR: /opt/chef/embedded/lib/ruby/gems/2.1.0/gems/grocery_delivery-0.0.4/bin/grocery-delivery:258:in `<top (required)>'
grocery-delivery[8434]: ERROR: /opt/chef/embedded/bin/grocery-delivery:23:in `load'
grocery-delivery[8434]: ERROR: /opt/chef/embedded/bin/grocery-delivery:23:in `<main>'
rocery-delivery[8434]: WARN: Expected process to exit with [0], but received '100'#012---- Begin output of /opt/chef/bin/knife cookbook delete simple_iptables --purge --yes -c /root/.chef/knife.rb ----#012STDOUT: Which version(s) do you want to delete?#0121. simple_iptables 0.6.0#0122. simple_iptables 0.2.3#0123. All versions#012STDERR: ERROR: EOFError: end of file reached#012---- End output of /opt/chef/bin/knife cookbook delete simple_iptables --purge --yes -c /root/.chef/knife.rb ----#012Ran /opt/chef/bin/knife cookbook delete simple_iptables --purge --yes -c /root/.chef/knife.rb returned 100

GD returns false positive exit status

Hi,

Summary

Due to rescue block used for upload_changed GD writes errors to stdout, but always returns success on exit status. $success variable stores exit code, but it's not used. This affects validating GD execution.

Details

False positive example

Proposed solution

I've been using GD triggered by commit hook, so I fixed it by adding simple exit($success) at the end, in order to be able to evaluate the run. You can find it here

Therefore, I'm not 100% sure if it should go there, as we have post hooks as well. The question is if the postrun hook should be executed when there was no success with knife upload, and if atexit hook shouldn't be the place to return status code.

Regards,
Krzysztof

Roles upload error

Summary

grocery-delivery errors on upload if at least one rb style role isn't present in the chef repo.

Details

I tried to find the code responsible for this issue but couldn't quite locate it. I have a roles directory that only contains json files. When I run grocery-delivery with this chef repo I get the following error.

root@chef:~# /opt/opscode/embedded/bin/grocery-delivery
WARN: Attempting to acquire lock
WARN: Lock acquired
WARN: Updating repo
WARN: Uploading all cookbooks
WARN: Uploading all roles
ERROR: /opt/opscode/embedded/lib/ruby/gems/2.2.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout.rb:289:in `invalid!'
ERROR: /opt/opscode/embedded/lib/ruby/gems/2.2.0/gems/mixlib-shellout-2.2.7/lib/mixlib/shellout.rb:276:in `error!'
ERROR: /opt/opscode/embedded/lib/ruby/gems/2.2.0/gems/between_meals-0.0.7/lib/between_meals/util.rb:42:in `exec!'
ERROR: /opt/opscode/embedded/lib/ruby/gems/2.2.0/gems/between_meals-0.0.7/lib/between_meals/knife.rb:56:in `role_upload_all'
ERROR: /opt/opscode/embedded/lib/ruby/gems/2.2.0/gems/grocery_delivery-0.0.7/bin/grocery-delivery:80:in `full_upload'
ERROR: /opt/opscode/embedded/lib/ruby/gems/2.2.0/gems/grocery_delivery-0.0.7/bin/grocery-delivery:166:in `upload_changed'
ERROR: /opt/opscode/embedded/lib/ruby/gems/2.2.0/gems/grocery_delivery-0.0.7/bin/grocery-delivery:277:in `<top (required)>'
ERROR: /opt/opscode/embedded/bin/grocery-delivery:22:in `load'
ERROR: /opt/opscode/embedded/bin/grocery-delivery:22:in `<main>'
WARN: Expected process to exit with [0], but received '1'
---- Begin output of /opt/opscode/bin/knife role from file /var/chef/grocery_delivery_work/cpe/roles/*.rb -c /root/.chef/knife.rb ----
STDOUT: 
STDERR: ERROR: Could not find or open file '/var/chef/grocery_delivery_work/cpe/roles/*.rb' in current directory or in 'roles//var/chef/grocery_delivery_work/cpe/roles/*.rb'
---- End output of /opt/opscode/bin/knife role from file /var/chef/grocery_delivery_work/cpe/roles/*.rb -c /root/.chef/knife.rb ----
Ran /opt/opscode/bin/knife role from file /var/chef/grocery_delivery_work/cpe/roles/*.rb -c /root/.chef/knife.rb returned 1

A simple solution is to create a test.rb roles with minimal content

name "test"
description "A test role so gd doesn't error."

which allows grocery-delivery to finish successfully. However it would be great if the above rb role limitation was removed.

Missing install instructions

There is no guidance on how to install gd. Ideally it would be a section in README.md or an INSTALL.md file could be added to the repo.

I'm willing to write the instructions once I figure out how to get it running myself 😄

Incompatible OpenSSL library prevents GD from running on F28

Trying to run grocery-delivery results in an unhandled exception:

# /opt/opscode/embedded/bin/grocery-delivery --help
Traceback (most recent call last):
	11: from /opt/opscode/embedded/bin/grocery-delivery:23:in `<main>'
	10: from /opt/opscode/embedded/bin/grocery-delivery:23:in `load'
	 9: from /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/grocery_delivery-0.0.8/bin/grocery-delivery:21:in `<top (required)>'
	 8: from /opt/opscode/embedded/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
	 7: from /opt/opscode/embedded/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
	 6: from /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/between_meals-0.0.9/lib/between_meals/repo/git.rb:20:in `<top (required)>'
	 5: from /opt/opscode/embedded/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
	 4: from /opt/opscode/embedded/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
	 3: from /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/rugged-0.27.7/lib/rugged.rb:6:in `<top (required)>'
	 2: from /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/rugged-0.27.7/lib/rugged.rb:10:in `rescue in <top (required)>'
	 1: from /opt/opscode/embedded/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
/opt/opscode/embedded/lib/ruby/site_ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require': /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/rugged-0.27.7/lib/rugged/rugged.so: undefined symbol: OPENSSL_sk_num - /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/rugged-0.27.7/lib/rugged/rugged.so (LoadError)

It looks like rugged depends on libssl but is expecting a newer version than what is vendored in /opt/opscode/.

# ldd /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/rugged-0.27.7/lib/rugged/rugged.so
	linux-vdso.so.1 (0x00007ffcbf2f0000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f2768887000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f2768668000)
	libssl.so.1.0.0 => /opt/opscode/embedded/lib/libssl.so.1.0.0 (0x00007f27683fc000)
	libz.so.1 => /opt/opscode/embedded/lib/libz.so.1 (0x00007f27681e5000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f2767fe1000)
	libcrypto.so.1.0.0 => /opt/opscode/embedded/lib/libcrypto.so.1.0.0 (0x00007f2767b32000)
	libruby.so.2.5 => /opt/opscode/embedded/lib/libruby.so.2.5 (0x00007f27675fa000)
	libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f27673c1000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f276702d000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f2766c6e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f2768db6000)
# objdump -T /opt/opscode/embedded/lib/ruby/gems/2.5.0/gems/rugged-0.27.7/lib/rugged/rugged.so | grep OPENSSL_sk_num
0000000000000000      D  *UND*	0000000000000000              OPENSSL_sk_num

If I try overwriting /opt/opscode/embedded/lib/libssl.so.1.0.0 with the system's /usr/lib64/libssl.so.1.1.0i that LoadError exception goes away (another exception prevents gd from working) indicating it's expecting a newer libssl than what's in opscode.

I tried a /opt/opscode/embedded/bin/gem update as well as uninstalling openssl, rugged, grocery_delivery and installing them again in order to rebuild the .so files, but that didn't help.

This is a fresh install of chef workstation and server on Fedora 28. I installed grocery_delivery via /opt/opscode/embedded/bin/gem install grocery_delivery from https://clburlison.com/chef-grocery-delivery-setup/.

# dnf list | grep chef
chef-server-core.x86_64                   12.18.14-1.el7                @@commandline
chef-workstation.x86_64                   0.2.43-1.el6                  @@commandline
rubygem-chef-zero.noarch                  2.2-6.fc28                    fedora
rubygem-chef-zero-doc.noarch              2.2-6.fc28                    fedora
# chef --version
Chef Workstation: 0.2.43
  chef-run: 0.2.4
  chef-client: 14.8.12
  delivery-cli: master (5fb4908da53579c9dcf4894d4acf94d2d9ee6475)
  berks: 7.0.7
  test-kitchen: 1.24.0
  inspec: 3.2.6

Unable to sync json role definitions

grocery-delivery script assumes definition files are ruby (*.rb). We have defined all our roles, nodes and databags using JSON.

Is there a config switch to allow uploading of json files?

Timestamp option (--t or -T) does not work

I was testing out GD and notice that the timestamp option doesn't produce timestamps in the output, even with upping the verbosity level. Would be handy when debugging when GD fails. Examples below:

root@localhost:~# /opt/chefdk/embedded/bin/grocery-delivery --timestamp -vv
WARN: Attempting to acquire lock
WARN: Lock acquired
WARN: Updating repo
DEBUG: Reading /var/chef/grocery_delivery_work/gd_revision
WARN: Repo has not changed, nothing to do...
WARN: Success at 14a40f45c0685b17005355004284163e546d9686
root@localhost:~# /opt/chefdk/embedded/bin/grocery-delivery --T -vv
WARN: Attempting to acquire lock
WARN: Lock acquired
WARN: Updating repo
DEBUG: Reading /var/chef/grocery_delivery_work/gd_revision
WARN: Repo has not changed, nothing to do...
WARN: Success at 14a40f45c0685b17005355004284163e546d9686

Question: Error when using GD to upload with Berkshelf

I am attempting to setup GD to upload cookbooks from a chef-repo to self-hosted chef server. We use bershelf to manage cookbook dependencies so I also have berks set to true in the GD config.

But I continue to hit an error on the berks upload that I can't work through. I am wondering if anyone has any pointers?

The error I am hitting is as follows. When I run the command printed at the bottom in shell everything succeeds.

> bundler exec grocery-delivery -c gd-config.rb -v
WARN: Attempting to acquire lock
WARN: Lock acquired
WARN: Updating repo
INFO: Running "git pull --rebase"
WARN: Uploading all cookbooks with berkshelf
WARN: Running berkshelf on cookbook: /Users/bwallace/github/grocery-delivery/grocery_delivery_work/chef-repo/cookbooks/mycookbook
INFO: Running: cd /Users/bwallace/github/grocery-delivery/grocery_delivery_work/chef-repo/cookbooks/mycookbook && /opt/chefdk/bin/berks install --config=/Users/bwallace/github/grocery-delivery/config.json && /opt/chefdk/bin/berks upload --config=/Users/bwallace/github/grocery-delivery/config.json
INFO: STDERR: /Users/bwallace/.rvm/gems/ruby-2.2.1/gems/bundler-1.11.2/lib/bundler/rubygems_integration.rb:304:in `block in replace_gem': addressable is not part of the bundle. Add it to Gemfile. (Gem::LoadError)
INFO: STDERR: from /opt/chefdk/bin/berks:4:in `<main>'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/gems/mixlib-shellout-2.2.6/lib/mixlib/shellout.rb:289:in `invalid!'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/gems/mixlib-shellout-2.2.6/lib/mixlib/shellout.rb:276:in `error!'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bundler/gems/between-meals-20e855de5e5d/lib/between_meals/util.rb:42:in `exec!'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bundler/gems/between-meals-20e855de5e5d/lib/between_meals/knife.rb:88:in `block (2 levels) in berks_cookbook_upload_all'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bundler/gems/between-meals-20e855de5e5d/lib/between_meals/knife.rb:86:in `each'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bundler/gems/between-meals-20e855de5e5d/lib/between_meals/knife.rb:86:in `block in berks_cookbook_upload_all'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bundler/gems/between-meals-20e855de5e5d/lib/between_meals/knife.rb:84:in `each'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bundler/gems/between-meals-20e855de5e5d/lib/between_meals/knife.rb:84:in `berks_cookbook_upload_all'
ERROR: /Users/bwallace/github/grocery-delivery/bin/grocery-delivery:74:in `full_upload'
ERROR: /Users/bwallace/github/grocery-delivery/bin/grocery-delivery:166:in `upload_changed'
ERROR: /Users/bwallace/github/grocery-delivery/bin/grocery-delivery:277:in `<top (required)>'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bin/grocery-delivery:23:in `load'
ERROR: /Users/bwallace/.rvm/gems/ruby-2.2.1/bin/grocery-delivery:23:in `<main>'
WARN: Expected process to exit with [0], but received '1'
---- Begin output of cd /Users/bwallace/github/grocery-delivery/grocery_delivery_work/chef-repo/cookbooks/mycookbook && /opt/chefdk/bin/berks install --config=/Users/bwallace/github/grocery-delivery/config.json && /opt/chefdk/bin/berks upload --config=/Users/bwallace/github/grocery-delivery/config.json ----

I should note that I am uploading to a chef-server with a self-signed cert. I have turned ssl-validation off by using the [undocumented] berks_config parameter to give berkshelf the following custom config:

{
  "ssl": {
    "verify": false
  }
}

Berkshelf leaves modified files after run causing git merge issues

When using the berkshelf option with grocery-delivery, berkshelf will sometimes make changes to the Berksfile.lock on disk as a result of the berks update command. This causes a merge error for git next time grocery delivery runs. I've been working around it by doing a git reset after each run, but a more integrated solution would be better:

cd [repo-dir]
git reset --hard

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.