facebook / chef-cookbooks Goto Github PK

View Code? Open in Web Editor NEW

567.0 567.0 139.0 2.46 MB

Open source chef cookbooks.

License: Apache License 2.0

Ruby 92.51% Shell 1.66% HTML 4.33% PowerShell 1.49%

chef-cookbooks's People

Contributors

Stargazers

Watchers

Forkers

jaymzh davide125 miteshshah moandcompany cloudxtreme devopsbox eheydrick patcox shakespears nukepuppy simonvik ram-devsecops sundar85 jjam3774 akpatil darktul bigdatalyn parthmonga jasonwbarnett rlugojr knyy liangyu misheska qiushenglu runt18 tomchomiak gauravtayal0 rameshshihora elthariel daijingwu skywalk7 azaugg daniellawrence jaga6040 polinakud alexxnica kryndex dilippanwar1 barthclem nakul-0071 cwonrails jamsheer vishnupanati rstadeu hankshz dhirenshumsher mareksapota zealws seanpandrew shyam2205 vincentpanqi opportunitylivetv brnout mjsorribas ashi-austin aekondratiev wagnerpinheiro ykankaya jaketux8 ascdso2020 shantanupanda dnuang sagar2366 heliocampos es jjustice6 neuralnoise natewalck bhanditz chrisrlong codermaze warrelis paulyc roblox zhengqin djakobsson facebook-gad srinivasadhu greeneg mjabuz bwann johngidt vinayasathyanarayana kcbraunschweig vinted oko taskset sasg baisilg claretnnamocha rb2k chefaustin dreweasland joshuamiller01 akimdi holladay-io chmodawk naomireeves southgate isabella232

chef-cookbooks's Issues

Small copy-pasta in the fb_ethtool README.md

References fb_screen when it should be fb_ethtool:

"You can opt out of package management by settings node['fb_screen']['manage_packages'] to false."

validate the config in fb_apache

We should run httpd -t in a validator when rendering config files in templates

Regression with recent log output change

This change appears to cause a regression if node['fb_apt']['apt_update_log_path'] is not set.
9218cc5

It seems like the code should have curly braces instead:

execute 'apt-get update' do
  command lazy {
    log_path = node['fb_apt']['apt_update_log_path']
    cmd_suffix = " >>#{Shellwords.shellescape(log_path)} 2>&1" if log_path
    "apt-get update#{cmd_suffix}"
  }
  action :nothing
end

I'm getting the following error with this code when I try to use chef 18:
https://github.com/boxcutter/boxcutter-chef-cookbooks/actions/runs/7862940576/job/21452971460

================================================================================
       Recipe Compile Error in /opt/kitchen/cache/cookbooks/boxcutter_init/recipes/default.rb
       ================================================================================
       
       ArgumentError
       -------------
       tried to create Proc object without a block
       
       Cookbook Trace: (most recent call first)
       ----------------------------------------
         /opt/kitchen/cache/cookbooks/fb_apt/recipes/default.rb:99:in `block in from_file'
         /opt/kitchen/cache/cookbooks/fb_apt/recipes/default.rb:98:in `from_file'
         /opt/kitchen/cache/cookbooks/boxcutter_init/recipes/default.rb:35:in `from_file'
       
       Relevant File Content:
       ----------------------
       /opt/kitchen/cache/cookbooks/fb_apt/recipes/default.rb:
       
        92:  end
        93:  
        94:  fb_apt_sources_list 'populate sources list' do
        95:    notifies :run, 'execute[apt-get update]', :immediately
        96:  end
        97:  
        98:  execute 'apt-get update' do
        99>>   command lazy do
       100:      log_path = node['fb_apt']['apt_update_log_path']
       101:      cmd_suffix = " >>#{Shellwords.shellescape(log_path)} 2>&1" if log_path
       102:      "apt-get update#{cmd_suffix}"
       103:    end
       104:    action :nothing
       105:  end
       106:  
       107:  if Chef::VERSION.to_i >= 16
       108:    notify_group 'periodic package cache update' do
       
       System Info:
       ------------
       chef_version=18.2.7
       platform=ubuntu
       platform_version=20.04
       ruby=ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
       program_name=/opt/cinc/bin/cinc-client
       executable=/opt/cinc/bin/cinc-client

vs_ftpd service does not respect disable setting

https://github.com/facebook/chef-cookbooks/blob/master/cookbooks/fb_vsftpd/recipes/default.rb#L69

The disable vsftpd service resource does not include service_name vsftpd so it errors when node['fb_vsftpd']['enable'] == false

fb_postfix should not default mydomain to fb.com

We should use localhost instead

apt-key does not output short key IDs in debian 9

Hi,

The provider code:

if keys && keyring
installed_keys = []
if ::File.exist?(keyring)
cmd = Mixlib::ShellOut.new("LANG=C apt-key --keyring #{keyring} list")
cmd.run_command
cmd.error!
output = cmd.stdout.split("\n")
Chef::Log.debug("apt-key output: #{output.join("\n")}")
installed_keys = output.select { |x| x.start_with?('pub') }.map do |x|
x[%r{pub./(?[A-Z0-9])}, 'keyid']
end
end

Does not detect the debian 9 apt-key output:

pub rsa4096 2016-10-05 [SC]
72EC F46A 56B4 AD39 C907 BBB7 1646 B01B 86E5 0310
uid [ unknown] Yarn Packaging [email protected]

Every chef pass reinstalls all the keys.

Compound API interactions for cookbooks included by `fb_init` are difficult to implement

Context

At Etsy, we've gone all-in on using this repository and the philosophy it represents (which we call the "Facebook pattern"), and have been working on refactoring our Chef code repository in a greenfield project.

As such, we've written an etsy_init (i.e. fb_init) cookbook that follows the same pattern as fb_init_sample - we have a bunch of settings we configure for our "base", and then a number of include_recipe 'fb_foo' statements to match.

We also apply our cookbooks with the run list as suggested by the README - recipe[fb_init], recipe[$SERVICE] - where in our case $SERVICE is hostgroup_foo, i.e. a cookbook that configures everything necessary for a foo cluster.

These "hostgroup" cookbooks often configure and include a cookbook that wraps some "Facebook-style" (our term) cookbooks, like fb_apache, into a reusable building block that can be shared by nearly similar hostgroups. We refer to these as stack cookbooks, but the naming isn't particularly important - what is important is that these cookbooks will almost always have "compound API interactions".

To make this more concrete, as an example we may have a stack_etsyweb cookbook that:

Sets up fb_apache
Installs php (imagine an fb_php cookbook)
Configures crons
Etc.

...all for running "Etsyweb", our monolithic PHP application.

While the application itself is a monolith, we do run the application on separate clusters, and these clusters may be configured slightly differently. Due to this, our stack_etsyweb cookbook will expose an API that a hostgroup_foo can configure differently than a hostgroup_bar would, but otherwise they are exactly the same.

The API exposed by stack_etsyweb is, as mentioned above, a compound API interaction, and so it will set fb attributes at runtime, like so:

whyrun_safe_ruby_block 'foo' do
  block do
    # ...
  end
end

The Problem

We've run into an issue where we cannot configure Facebook cookbooks (using the compound API interaction pattern above) that have been included in fb_init, because fb_init converges before our "tier" cookbooks converge.

Building on the previous example, if our fb_init cookbook looked like:

# ...

include_recipe 'fb_cron'

# ...

...and our stack_etsyweb cookbook looked like:

whyrun_safe_ruby_block 'foo' do
  block do
    node.default['fb_cron']['jobs']['foo'] = {
      'time' => '4 5 * * *',
      'user' => 'apache',
      'command' => "foo #{node['stack_etsyweb']['foo_cron_param']}",
    }
  end
end

...and our hostgroup_bar cookbook looked like:

node.default['stack_etsyweb']['foo_cron_param'] = 'baz'

include_recipe 'stack_etsyweb'

...and finally, our run list looks like (as described in the README):

recipe[fb_init], recipe[hostgroup_bar]

...then we would unfortunately not see a foo cron in the resulting crontab, because again, fb_cron has already converged by the time that the stack_etsyweb cookbook is converging.

While the example above demonstrates it with a cron, I'd like to emphasize that this happens for any compound API interaction involving a cookbook included in fb_init.

This would also happen if we were to have two stack cookbooks with a common dependency included in a single hostgroup, like if hostgroup_foobar included stack_foo and stack_bar, and if stack_foo and stack_bar both used fb_apache. In this case, stack_bar's compound API interactions with fb_apache would never show up, since stack_foo would converge fb_apache before stack_bar converged.

Discussion

First off, I realize the above has a lot of our own terminology in it, but hopefully the underlying problem is clear: compound API interactions require careful ordering to ensure that attributes written the converge phase are converged before the cookbooks that use them are converged.

It seems that you all have considered this issue on some level, as there are comments in the fb_init_sample that mention ordering:

chef-cookbooks/cookbooks/fb_init_sample/recipes/default.rb

Lines 143 to 148 in e17c840

 # we recommend you put this as late in the list as possible - it's one of the 

 # few places where APIs need to use another API directly... other cookbooks 

 # often want to setup cronjobs at runtime based on user attributes... they can 

 # do that in a ruby_block or provider if this is at the end of the 'base 

 # runlist' 

 include_recipe 'fb_cron'

We've brainstormed this a bit internally, so I'd like to share some of the ideas we've had so far about how to handle this, in no particular order:

Somehow implement a "post-compile, pre-any-cookbook-convergence" phase, and set attributes there.

Since the issue boils down to wanting to set an attribute in the convergence phase so that it happens after the compile phase, but that causes ordering issues with the things that are being converged, we could implement some way of setting the attributes before any resources actually converge.

This could be something like a whyrun_safe_ruby_block-esque custom resource that accumulates blocks at compile time and then have something at the top of fb_init that executes all of them, or something like a pattern where cookbooks can register a recipes/converged-attributes.rb file that we dynamically execute at the top of fb_init somehow, etc.

Make an "`fb_init` sandwich", and change the run list to something like `['fb_init::part_one', 'hostgroup_foo', 'fb_init::part_two']`.

I'm not a huge fan of this as we could have situations where we want something to live in "part one" and "part two", and so we'd have to pick whether we wanted to have it be set up early, or if we wanted to allow compound API interactions with it. It also doesn't solve the issue of having a hostgroup_foobar that includes stack_foo and stack_bar.

Use `lazy` node attributes via `Chef::DelayedEvaluator`.

This one may require more explanation, but since this issue is already pretty long, I'll try to keep it brief. As of chef/chef#10345, Chef has some support for Chef::DelayedEvaluator instances inside the node attribute tree. This means that instead of using a whyrun_safe_ruby_block for a compound API interaction, you could write:

node.default['fb_cron']['jobs']['foo_cron'] = {
  'time' => '4 5 * * *',
  'user' => 'apache',
  'command' => Chef::DelayedEvaluator.new { "foo #{node['stack_etsyweb']['foo_cron_param']}" },
}

This reads a lot better than the whyrun_safe_ruby_block approach in my opinion, and also has the nice side-effect of still allowing for a "last-write wins" scenario where a later cookbook decides to overwrite the node.default['fb_cron']['jobs']['foo_cron'] entry entirely at compile time.

Unfortunately, this only works if the fb cookbook in question accesses the node attributes directly via [], and not via something like map, each_pair, to_h, etc. As far as I can tell, most or all fb cookbooks, like the aforementioned fb_cron, do not access the attributes via [], so this won't work.

I've documented this as a bug in the Chef repository in chef/chef#14159, because I believe that accessing the node attributes without [] should still evaluate any Chef::DelayedEvaluators anyway, but alternatively Facebook cookbooks could be changed to expect Chef::DelayedEvaluator instances until the behavior is changed (if ever). This would potentially be something like walking the node attribute tree to evaluate Chef::DelayedEvaluators manually before iterating over the node attributes, or always using [], or something else.

We've considered doing that, but it means that we'd need to make a lot of changes to our fork of this repository. We wanted to get your opinion on the matter first before we more seriously considered this approach, since we'd ideally like to contribute these changes here.

Closing

Off the cuff, I think that being able to use Chef::DelayedEvaluator instances in the attribute tree is pretty attractive, but the "post-compile, pre-any-cookbook-convergence" phase idea is probably more tractable.

In any case, we're interested in getting your thoughts on this issue, and we're curious if you all have implemented anything or follow any patterns internally in order to better allow for compound API interactions in all situations.

@jaymzh in particular, I'm curious what your take is on either this issue or on the one I filed with Chef at chef/chef#14159, since I saw you participated in chef/chef#10861.

Thanks for giving this a read, and let me know if I can clarify anything! I'm also happy to take this conversation elsewhere.

fb_tmpclean doesn't include tmpreaper defaults on debian/ubuntu - breaks /tmp cleanup

default file doesn't run tmpreaper for anything by default and only adds new calls based on what people add to the API. The default API does not include anything in directories (presumably because tmpclean file always does /tmp hardcoded), but the template for tmpreaper drops that. The default file on Ubuntu btw has:

TMPREAPER_TIME=${TMPREAPER_TIME:-7d}
TMPREAPER_PROTECT_EXTRA=${TMPREAPER_PROTECT_EXTRA:-''}
TMPREAPER_DIRS=${TMPREAPER_DIRS:-'/tmp/.'}

nice -n10 tmpreaper --delay=$TMPREAPER_DELAY --mtime-dir --symlinks $TMPREAPER_TIME  \
  $TMPREAPER_ADDITIONALOPTIONS \
  --ctime \
  --protect '/tmp/.X*-{lock,unix,unix/*}' \
  --protect '/tmp/.ICE-{unix,unix/*}' \
  --protect '/tmp/.iroha_{unix,unix/*}' \
  --protect '/tmp/.ki2-{unix,unix/*}' \
  --protect '/tmp/lost+found' \
  --protect '/tmp/journal.dat' \
  --protect '/tmp/quota.{user,group}' \
  `for i in $TMPREAPER_PROTECT_EXTRA; do echo --protect "$i"; done` \
  $TMPREAPER_DIRS

in it.

It's probably better to move away from the templating of calls and instead template /etc/tmpreaper.conf and just populate the right variables.

fb_vsftpd is broken on debian 9

https://github.com/facebook/chef-cookbooks/pull/140/checks?check_run_id=828494452#step:4:3518

NoMethodError using fb_fstab on Ubuntu 18.04

Hi there,

I'm having an issue using fb_fstab on Ubuntu 18.04.

My Node looks like this:

{
  "name": "somethingsomething",
  "chef_environment": "prod",
  "normal": {
    "fb_fstab" : {
      "enable_remount": true,
      "enable_unmount": true,
      "mounts" : { 
        "varlog" : { 
          "enable_remount": true,
          "device": "logs-tmpfs",
          "type": "tmpfs",
          "opts": "size=1G",
          "mount_point": "/var/log"
        }
      }
    },
    "tags": [ ]
  },
  "policy_name": null,
  "policy_group": null,
  "run_list": [
  "recipe[fb_fstab::default]"
  ]

}

Running fb_fstab::default gives me a NoMethodError.
(the version Numbers are bumped, because we also use a version of this cookbook from like 2 years ago, this run used the cookbooks from d1e06db )

Starting Chef Client, version 14.10.9
resolving cookbooks for run list: ["fb_fstab::default"]
Synchronizing Cookbooks:
  - fb_helpers (0.1.1)
  - fb_fstab (0.0.2)
Installing Cookbook Gems:
Compiling Cookbooks...
Converging 5 resources
Recipe: fb_fstab::default
  * file[/etc/.fstab.chef] action create
    - change mode from '0644' to '0444'
  * whyrun_safe_ruby_block[validate data] action run
    
    ================================================================================
    Error executing action `run` on resource 'whyrun_safe_ruby_block[validate data]'
    ================================================================================
    
    NoMethodError
    -------------
    undefined method `[]' for nil:NilClass
    
    Cookbook Trace:
    ---------------
    /var/chef/cache/cookbooks/fb_fstab/libraries/default.rb:98:in `get_autofs_points'
    /var/chef/cache/cookbooks/fb_fstab/libraries/default.rb:105:in `autofs_parent'
    /var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:83:in `block (3 levels) in from_file'
    /var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:31:in `each'
    /var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:31:in `block (2 levels) in from_file'
    
    Resource Declaration:
    ---------------------
    # In /var/chef/cache/cookbooks/fb_fstab/recipes/default.rb
    
     28: whyrun_safe_ruby_block 'validate data' do
     29:   block do
     30:     uniq_devs = {}
     31:     node['fb_fstab']['mounts'].to_hash.each do |name, data|
     32:       # Handle only_if
     33:       if data['only_if']
     34:         unless data['only_if'].class == Proc
     35:           fail 'fb_fstab\'s only_if requires a Proc'
     36:         end
     37:         unless data['only_if'].call
     38:           Chef::Log.debug("fb_fstab: Not including #{name} due to only_if")
     39:           node.rm('fb_fstab', 'mounts', name)
     40:           next
     41:         end
     42:       end
     43: 
    
    Compiled Resource:
    ------------------
    # Declared in /var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:28:in `from_file'
    
    whyrun_safe_ruby_block("validate data") do
      action [:run]
      default_guard_interpreter :default
      declared_type :whyrun_safe_ruby_block
      cookbook_name "fb_fstab"
      recipe_name "default"
      block #<Proc:0x0000000003c16fa8@/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:29>
      block_name "validate data"
    end
    
    System Info:
    ------------
    chef_version=14.10.9
    platform=ubuntu
    platform_version=18.04
    ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
    program_name=/usr/bin/chef-client
    executable=/opt/chef/bin/chef-client
    

Running handlers:
[2019-02-20T09:21:18+00:00] ERROR: Running exception handlers
Running handlers complete
[2019-02-20T09:21:18+00:00] ERROR: Exception handlers complete
Chef Client failed. 1 resources updated in 04 seconds
[2019-02-20T09:21:18+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out
[2019-02-20T09:21:18+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2019-02-20T09:21:18+00:00] FATAL: NoMethodError: whyrun_safe_ruby_block[validate data] (fb_fstab::default line 28) had an error: NoMethodError: undefined method `[]' for nil:NilClass

this is the generated stacktrace:

Generated at 2019-02-20 09:21:52 +0000
NoMethodError: whyrun_safe_ruby_block[validate data] (fb_fstab::default line 28) had an error: NoMethodError: undefined method `[]' for nil:NilClass
/var/chef/cache/cookbooks/fb_fstab/libraries/default.rb:98:in `get_autofs_points'
/var/chef/cache/cookbooks/fb_fstab/libraries/default.rb:105:in `autofs_parent'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:83:in `block (3 levels) in from_file'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:31:in `each'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:31:in `block (2 levels) in from_file'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider/whyrun_safe_ruby_block.rb:25:in `action_run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider.rb:182:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource.rb:578:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:70:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `each'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:97:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:720:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `catch'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:754:in `converge_and_save'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:286:in `run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:321:in `block in fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:309:in `fork'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:309:in `fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:274:in `block in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:261:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:479:in `sleep_then_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:468:in `block in interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:467:in `loop'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:467:in `interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:451:in `run_application'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:66:in `run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/bin/chef-client:25:in `<top (required)>'
/usr/bin/chef-client:74:in `load'
/usr/bin/chef-client:74:in `<main>'

>>>> Caused by NoMethodError: undefined method `[]' for nil:NilClass
/var/chef/cache/cookbooks/fb_fstab/libraries/default.rb:98:in `get_autofs_points'
/var/chef/cache/cookbooks/fb_fstab/libraries/default.rb:105:in `autofs_parent'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:83:in `block (3 levels) in from_file'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:31:in `each'
/var/chef/cache/cookbooks/fb_fstab/recipes/default.rb:31:in `block (2 levels) in from_file'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider/whyrun_safe_ruby_block.rb:25:in `action_run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider.rb:182:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource.rb:578:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:70:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `each'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:97:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:720:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `catch'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:754:in `converge_and_save'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:286:in `run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:321:in `block in fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:309:in `fork'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:309:in `fork_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:274:in `block in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:261:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:479:in `sleep_then_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:468:in `block in interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:467:in `loop'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:467:in `interval_run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:451:in `run_application'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:66:in `run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/bin/chef-client:25:in `<top (required)>'
/usr/bin/chef-client:74:in `load'
/usr/bin/chef-client:74:in `<main>

It seems like I'm missing something. Any Idea how I can get this cookbook to work?

Thanks,
-Markus

codemod internal library methods in fb_fstab to make it clear they're not part of the API

Stuff like FB::Fstab.get_unmasked_base_mounts isn't safe to call from random cookbooks as it relies on node data that may not necessarily be there or be correct. Audit all methods, codemod the internal ones with a _ and document the rest properly in README.

fb_helpers_reboot lies about :now

:now is suppsoed to reboot now (if allowed). But it has an undocumented FB-ism, that if you're in firstboot, it switches to managed reboots:

action :now do
  # TODO (t15830562) - this action should observe required and override the
  # same way as the :deferred action
  if node['fb_helpers']['reboot_allowed']
    if node.firstboot_any_phase?
      set_reboot_override('immediate')
      do_managed_reboot
    else

That's... not cool.

Enablement of `unified_mode` for v17+ Chef client compatibility

Describe the Enhancement:

Converging the cookbooks in this repo on a v17+ Chef client result in the following deprecation warning being thrown at the end of the run (using fb_systemd as an example):

  The  resource in the fb_systemd cookbook should declare `unified_mode true` at 2 locations:
    - /var/chef/cache/cookbooks/fb_systemd/resources/loader_entries.rb
    - /var/chef/cache/cookbooks/fb_systemd/resources/reload.rb
   See https://docs.chef.io/deprecations_unified_mode/ for further details.

See: https://docs.chef.io/unified_mode/

Describe the Need:

Anyone using Chef client v17+ will face this deprecation warning until unified_mode true is appended to the first line of each custom resource file.

Current Alternative

Ignore the deprecation warning.

Can We Help You Implement This?:

Should be a relatively easy PR (or series of PR's) to craft.

Cleanup the fb_systemd rollout

Its sitting at in_shard?(99)... wanna clean that up?

fix shellcheck issues in fb_ipset and fb_less

https://github.com/facebook/chef-cookbooks/pull/140/checks?check_run_id=828534769#step:4:5

fb_network_scripts changes should be reflected in /var/chef/backup

fb_network_scripts does some trickery under the hood to in-place modify configs to try and avoid network restarts at much as possible. This is great, but the changes don't get reflected in /var/chef/backups, making troubleshooting harder at times. I'm not entirely sure if this is something we can sanely improve, but filing this for tracking and visibility.

node.antlir_build? not defined in open source

Hi all, it looks like the node.antlir_build? function is not defined that is referenced in the fb_syslog cookbook:

chef-cookbooks/cookbooks/fb_syslog/recipes/packages.rb

Line 25 in 7cbfb03

if node.centos9? && node.antlir_build?

Any chance of providing a definition for this in the open source fb_helpers so we can use this cookbook unmodified, if it is needed?

fb_apt doesn't take into account /etc/apt/trusted.gpg.d/

Debian and ubuntu both allow keys in /etc/apt/trusted.gpg.d

Currently if a key listed in chef is in a file in that directory, which for Ubuntu 18.04 all of the older keys are, then Chef won't find them, it tries to add them, this "succeeds" because it's already there and it tries to do this every time.

To make matters worse, apt-key does not accept more than one --keyring option.

I'm not sure of the best solution, but I suspect it's:

Chef should use a keyring like /etc/apt/trusted.gpg.d/fb_keyring.gpg
Chef should, by default, cleanup keyrings that dont' match the format of {ubuntu,debian}-keyring-\d{4}-.*.gpg but should leave that and the main trusted.gpg alone as "the package's pervue"
Users should then only specify non-OS keys in chef.

fb_systemd::udevd fails on Ubuntu 20.04 LTS due to missing symlink

On Ubuntu 16 and 18 the fb_systemd::udevd recipe succeeds without issue but on Ubuntu 20 the recipe will fail because the symlink from /sbin/udevadm to /bin/udevadm has been removed.

From the udev filelists provided by Canonical for each release, one can validate the existence of /sbin/udevadm in Xenial and Bionic Beaver, as well as its non-existence on Focal Fossa.

fb_apache: hardcoded /etc/httpd path for server-status

1002544

This diff added a thing to write out status.conf, but it's hardcoded with /etc/httpd/ which breaks on non-CentOS systems. PR incoming eventually

fb_reprepro needs love

I'm leaving it out of my testing PR, but it needs defaults for all it's values in attributes...

Thought to be honest, I would actually pull all that configurability of what it's directories are out to match the rest of the cookbooks that let you put whatever you want in the config, but are strongly opinionated about where that config is laid out.

Fix CI failures

several shellcheck failures for fb_network_scripts/files/default/ifup-pre-local

kitchen failures for unknown instance:

/usr/bin/kitchen test default-debian-10
/usr/bin/kitchen test default-ubuntu-2004
/usr/bin/kitchen test default-centos-8

kitchen failures for grub:

template[grub2_config_efi] action create
Parent directory /boot/efi/EFI/debian does not exist.

various rubocop failures

Publishing to supermarket.chef.io

Can we publish these to supermarket.chef.io ?

fb_ntp shouldn't default to facebook timeservers

fb_storage always ignores override files when '_clowntown_override_file_method' not defined

According to both the docs and the logs, '_clowntown_override_file_method' allows you to add additional restrictions to when override files take effect, but if it's not defined, then they always take effect.

Turns out, this isn't true, we always return false. This is a simple bug, but fixing it it probably requires a slow rollout to FB so I haven't sent a PR.

Docs:

For additional safety, you can define a check method in
`node['fb_storage']['_clowntown_override_file_method']`
that will be invoked whenever override files are evaluated.

Actual code where we say we're ignore the method, but then returnf alse.

      def override_file_applies?(verb, fname, quiet = false)
        if File.exist?(fname)
          base_msg = "fb_storage: System has #{fname} file present"
          if node['fb_storage']['_clowntown_override_file_method']
            ...
          end
          unless quiet
            Chef::Log.warn(
              "#{base_msg} but the override check method is not defined, " +
              'therefore we are ignoring it.',
            )
          end
          return false   # this should be `true`
        end
        return false
      end

fb_zfs fails on every single supported distro

all centos and all debian/ubuntus fail for something equivalent to:

No candidate version available for spl, spl-dkms, zfs-dkms, zfsutils-linux

Definitions for antlir2 fix are not exposed in open source

Hi all - this commit seems to have definitions that aren't in the open source exposed to the public:
6baf374

I don't see a fb_util or a definition for antlir2_build? and this commit prevents use of fb_systemd due to missing definitions.

fb_helpers contains namespace collisions with official chef node objects

In chef v17, chef added a lot of built-ins and we are seeing instability with at least one of them

https://github.com/facebook/chef-cookbooks/blob/main/cookbooks/fb_helpers/libraries/node_methods.rb#L146

when we call debian? within a library function, it returns false, so we are now having to do debian? || ubuntu? to workaround this as we consume fb_helpers.

To put this nicely, is Facebook willing to stop doing this? Or at the very least, stop putting so many of these things in node and put it to a FB namespace?

fb_apt update

Hello, we've written a version of fb_apt that replaces use of apt-key and /etc/apt/trusted.gpg.d¹ with per-repo keyrings managed by gpg and specified via Signed-By. It also uses the newer DEB822 format which we find easier to read and more concise in some cases. The cookbook API is similar except for the repos and keys attributes which are replaced with a sources hash:

node.default['etsy_apt']['sources']['repo_name'] = {
  'URIs' => 'https://repo.url.here',
  'Suites' => 'noble',
  'Components' => 'main',
  '_public_key' => read_file('repo_name-signing-key'),
}
# or `_fingerprint` instead of `_public_key` to fetch from a keyserver

Presently it doesn't try to manage anything in /etc/apt/trusted* nor accommodate older versions of Debian/Ubuntu or Chef.

We figure other users may want this functionality so contributing it back to this repo came up. Curious if the maintainers have a minimum level of back-compatibility they'd want before considering merging something like this, or if they'd be open to an fb_apt2 or something.

Happy to share code of course.

"The certificate MUST NOT be placed in /etc/apt/trusted.gpg.d or loaded by apt-key add." https://wiki.debian.org/DebianRepository/UseThirdParty ↩

get_seeded_flexible_shard in fb_cron appears to be missing from public repo

When you set splaysecs the fb_cron cookbook references the node.get_seeded_flexible_shard(). This method does not appear to be publicly available.

	# we recommend you put this as late in the list as possible - it's one of the
	# few places where APIs need to use another API directly... other cookbooks
	# often want to setup cronjobs at runtime based on user attributes... they can
	# do that in a ruby_block or provider if this is at the end of the 'base
	# runlist'
	include_recipe 'fb_cron'