fb_ntp shouldn't default to facebook timeservers about chef-cookbooks

[davide125]

time*.facebook.com are public NTP servers managed by Facebook, but meant for public usage: https://engineering.fb.com/2020/03/18/production-engineering/ntp-service/

I think it's a fine default, and I'd rather use these that put even more load on pool.ntp.org.

[malmond77]

Most folks don't have enough visibility into their infrastructure to see the leap second issues they have. That's not the same as not having them.

If each OS has it's own time source: they could differ by how they handle leap seconds, or they might just be different, and we can't reason about them. The only ones Facebook can reason about are the ones already listed. I'm equally concerned that you might end up in a worse state when comparing time stamps in heterogeneous infrastructure.

Lastly, I don't know who is using this stuff today, outside of you and Facebook. I'm concerned about coordinating this change in concert with firewall policy.

I don't think you're wrong to bring this up: it could definitely have gone a different way at the beginning. Now the defaults are out there, is there a strong enough use case for changing them in light of these concerns?

[malmond77]

Thanks for the report. Is this a question of policy: the cookbook shouldn't have any defaults, or concerns over access (I believe these are public now)? For comparison: systemd's code embeds other public NTP and DNS server addresses, so this isn't without some precedent. Which way do you think this should go?

[jaymzh]

You mean how we have always tried to keep close OS defaults?

@yarikk - you don't need to explain FB cookbooks to me, I designed the model and wrote the vast majority of them.

The Facebook cookbooks are not tools to drive people to Facebook's products - that's not why we open-sourced them.

The FB cookbook model has always been to either provide blank defaults where that is the most logical or to keep close to the OS defaults as is manageable. In this case there's a small set of defaults that's easy to provide.

[yarikk]

With all due respect,

The FB cookbook model has always been to either provide blank defaults where that is the most logical or to keep close to the OS defaults as is manageable. In this case there's a small set of defaults that's easy to provide.

Right, but at cost of added complexity which can be avoided if the defaults work for all while stay uniform across supported platforms.

You mean how we have always tried to keep close OS defaults?

Exactly that kind of clutter. It's a compromise to make. We'd have keep up with every supported platform. There must be good reasons to do that. Even if we go for it, what value is this going to add here?

@yarikk - you don't need to explain FB cookbooks to me, I designed the model and wrote the vast majority of them.

By all means, your legacy lives on!

[leoleovich]

@malmond77 I'm looking to do what we do everywhere else. We can fall back to FB if we don't know. Observe this "complicated" code:

value_for_platform_family(
  ['debian', 'rhel', 'fedora'] => 4.times.map { |i| "#{i}.#{node['platform']}.pool.ntp.org" },
  'windows' => ['time.windows.com'],
  'mac_os_x' => ['time.apple.com', 'time.euro.apple.com'],
  'default' => 5.times.map { |i| "time#{i}.facebook.com" },
)

What you suggest here has potentially very dangerous implications. This code easily leads to homogeneous infra using different time sources with different precision, reachability, leap second smearing strategy and a lot more.
This puts an extra burden on all the users to make sure the default is actually never used. Which makes it a pretty useless and not solving any actual problem.

[malmond77]

It's been quite a while since this was discussed. Facebook is providing both the service and config management together. If there are any technical problems with how this works, let's follow up on those specifically.

[yarikk]

fb_ntp shouldn't default to facebook timeservers

Why not? Please elaborate your concerns.

[jaymzh]

In general, we have kept our default to be OS defaults, and then set FB-specific settings in fb_base_settings.

I would suggest throwing these at ntp[1-4].{debian,ubuntu} etc.

[yarikk]

Since we overwrite the OS's /etc/ntp.conf there's no easy way to loop back its original contents into the cookbook. And since the contents happens to be OS and Distribution-specific, an attempt to mimic their exact values would add lots of clutter.

Instead, we supply universal defaults which should work for everybody: the Facebook Public NTP pool is, as one may grasp from the article referenced above, is a world-class service. After all, deployments are free to make their own choices.

[leoleovich]

One thing to mention here - the cookbook is called fb_ntp and it's fair to have facebook public time servers by default.

[jaymzh]

One thing to mention here - the cookbook is called fb_ntp and it's fair to have facebook public time servers by default.

No, that's not what the fb_ means, that means it follows the FB Attribute model.

None of our cookbooks default people to FB infra. Never have.

[malmond77]

@jaymzh are you looking for no defaults in this cookbook, and the expectation that everyone using this will need to explicitly pick values for their environment, or are you looking to establish a different default, the most extreme being (trying) to model distro specific values so that those that employ this cookbook have minimal divergence from upstream?

[jaymzh]

@malmond77 I'm looking to do what we do everywhere else. We can fall back to FB if we don't know. Observe this "complicated" code:

value_for_platform_family(
  ['debian', 'rhel', 'fedora'] => 4.times.map { |i| "#{i}.#{node['platform']}.pool.ntp.org" },
  'windows' => ['time.windows.com'],
  'mac_os_x' => ['time.apple.com', 'time.euro.apple.com'],
  'default' => 5.times.map { |i| "time#{i}.facebook.com" },
)

[jaymzh]

(I can write a PR for fb_ntp and fb_chrony, I don't mind. I just didn't have the time when I wrote this issue and did it to remind myself)

[davide125]

IMHO this isn't a big deal either way. NTP is one of those things that's effectively fungible, and it doesn't even have the privacy implications of say DNS. I think fb is a good default as any other, especially on Linux where pretty much every distro just uses a dedicated pool.ntp.org. With that said, if you feel strongly about this, feel free to send a PR. We should still use the fb server as a fallback if there's isn't a more specific one though.

[jaymzh]

I'll send a PR when I get a moment. Thanks all!

[jaymzh]

I think that's true... for places that do leap second smearing... which is like... < 10 organizations. The vast majority of folks use NTP to make sure their logs are readable and that auth (i.e kerb) work. They don't care about sub-second drift, and they definitely don't care about leap seconds. I know the hell that FB went through because of it, but it's not a common concern. And for those who do have such a concern, they are likely very carefully picking their NTP servers anyway.... and probably not using NTP, but instead using Chrony.