Configuration settings: bind to about netdata-cookbook

[nictrix]

@sergiopena I would also like netdata to bind to localhost, but you definitely take on a lot more maintenance once you start managing the /etc/netdata/netdata.conf file. One way to punt on managing the configuration and take care of @robbat2 request would be to edit the service file with the line cookbook. Such as:

replace_or_add 'bind to' do
  path '/etc/systemd/system/netdata.service'
  pattern "ExecStart.*"
  line "ExecStart=/usr/sbin/netdata -D -i 127.0.0.1"
  replace_only true
end

And do the same with /etc/init.d/netdata - I can create a PR with this functionality or we can discuss how to go about managing the main configuration file.

[jmadureira]

I've been looking through the netdata.conf file and as far as I can tell the only configuration available on a vanilla installation is the registry to announce (which btw make perfect sense to be configurable). Everything else is commented by default.

I also couldn't find anything that would allow us to affect the content of the netdata.conf during installation.

In my opinion we could try to manage the conf file iff netdata allows the groups to be left out i.e.; if for instance remove the [users.mem] the metric remains available. This way we can have something like:

default['netdata']['conf']['global'] = { } # default is empty

node['netdata']['conf']['global'] = {
  'bind to' => 'localhost'
} # specify only the attributes we want to change

[nictrix]

@jmadureira I left comments on your PR, but also wanted to add something I found. It seems if we delete everything except bind to we get fewer metrics, but I couldn't quantify which ones they were as browsing the dashboard side-by-side and I couldn't see the difference. I think I'll submit this as a bug report to Netdata.

Although, if I did the conf change to bind to = localhost then did the wget to get the updated conf from Netdata itself and then restarted, it came back with those extra metrics (whatever they were as I didn't see any differences other than the count from the dashboard)

[nictrix]

Also, we may want to look into offering a similar setup as the HAproxy cookbook: https://github.com/sous-chefs/haproxy (using resources to define the blocks in the large config file.

[jmadureira]

@nictrix Thank you for your comments.
First I was wrong and the registry to announce isn't configured on a vanilla netdata installation so as far as I can tell the netdata.conf file starts with only default attributes.

I cannot reproduce the issue you found with missing metrics but I did find out that since the page is loaded dynamically the information may not be accurate during a restart. For instance in my installation I have 736 metrics being collected and 164 charts. After restart the number of metrics drops to ~50 metrics but after a few refreshes is eventually reaches the previous numbers. Can you check if you also have this behaviour?

Regarding your suggestion to use the same behaviour as HAproxy are your referring to the use of a chef resource coupled with edit_resource so that users can configure the netdata.conf from other cookbooks? At the moment I do not know if it will make things simpler but I'll see.
Unfortunately this approach won't solve the problem of replacing the original netdata.conf (and subsequent updates from netdata itself). I do not believe it can actually be solved since that would break one of Chef's design principles.

[ktsaou]

Hi folks,

Unfortunately this approach won't solve the problem of replacing the original netdata.conf (and subsequent updates from netdata itself)

netdata never updates its config files by itself. Also, netdata updates do not overwrite files that are not stock versions.

[jmadureira]

Hello @ktsaou

Thank you for the feedback. Yes I was aware of that so I probably didn't explain myself well enough.
I only have two concerns here:

1. If there's any difference in having an empty netdata.conf or a stock version of that file since @nictrix noticed a difference in the number of metrics being collected;
2. What' the best way to manage the netdata.conf file through Chef:
   1. node attributes (what's on this PR)
   2. Chef resource (what's being used on the HAproxy cookbook)

[ktsaou]

An empty netdata.conf is the stock netdata.conf

1. netdata has 2 functions that influence the number of metrics:

   • auto-detection, meaning that metrics are collected only when they are found available. To save CPU cycles netdata does not check all the possible sources all the time. For example it checks for new mount points, new cgroups (containers, systemd services) every 10 seconds.

   • delay visualization of zero valued metrics. This is an attempt to lower the memory requirements of netdata. Without it the dashboard would have a few thousand more metrics with just zeros. So, netdata collects all metrics, but as long as the metrics are zero, it ignores them. It needs just a non-zero collection to enable the metric/chart, and after that it remains active until the next restart of netdata. So, for example, until your network interface drops a packet, you will not see the dropped packets chart. It will appear and stay there until the next restart, on the first packet dropped.

2. I am sorry, I don't know. I don't have any experience with Chef so I can't tell the difference of the 2 methods.

[nictrix]

Thanks @ktsaou for clearing up our questions on an empty netdata config.

@jmadureira I did the same steps as before and only saw 2 metric +/- difference, so I think I'll chalk this up as dynamic metric gathering (though I would like to get a simple listing of all collected metrics to see a difference, maybe I'll work on that sometime soon)

If we went with custom resources managing the netdata.conf we could then easily default and combine different parts of the configuration with multiple resources provided by the cookbook.

For example:

# this edits the [global] section of /etc/netdata/netdata.conf
netdata_config 'global' do
  log_directory: '/var/log/netdata'
end

# this edits the [plugin:apps] section of /etc/netdata/netdata.conf
netdata_config 'plugin:apps' do
  update_every: 1
end

# this edits the [stream] section of /etc/netdata/stream.conf file
netdata_stream 'stream' do
  enabled: 'no'
end

....

These types of resources can be used in simple setups, however, they can also scale to larger setups as they are all resources managing smaller parts of the configuration vs. a very large attribute file to handle everything. This will also allow the use case of this cookbook being used on 1 server but by multiple cookbooks such as a nginx cookbook adding itself to netdata, plus docker cookbook adding itself to netdata. All configuration would be handled without doing difficult search and replace, appending content in particular places, etc.. (I know a lot of netdata monitoring is dynamic, however dropping in users, passwords, specific sockets, database names, etc.. would have to come from a cookbook that originally created that resource.

For example:

mongodb::default # installs mongodb
mongodb::users # adds authentication
mongodb::netdata # enables netdata, by configuring the mongodb plugin to use a mongodb database user setup in the previous recipe: users

I can take a stab at this and submit a separate PR, if this makes the most sense.

Also, I just realized I never clicked submit on the review comments, I have now, please see the PR comments for additional information.

[jmadureira]

Thanks @ktsaou for the clarification as well.

@nictrix Yes you are correct. Using resources for this configuration makes more sense. In fact considering that so far the only mechanisms for updating conf files on this cookbook are based on resources it doesn't make much sense to handle this one in particular any differently.

If you like to try to implement that approach please do otherwise let me know and I'll submit a separate PR with a proposal.

Also thank you for the comments as well but I'll be closing the #13 since there's no point in merging that solution.