Support FreeBSD about opensearch-build

[rmuir]

I use #!/usr/bin/env bash for my bash scripts. It should be a bit more portable than using #!/bin/bash and may solve the issue here.

[smortex]

Hello!

Overview of FreeBSD ports and packages

For some years now, FreeBSD provide packages for the ports: users do not need to compile all software they want to use anymore, they can just install packages using the pkg command.

The ports are basically a recipe to handle build dependencies, download the source code of a program, compile it, install it, and generate a package. The FreeBSD packages are built using the ports in the FreeBSD ports tree. New packages are built regularly (weekly), FreeBSD basically follow a rolling release pattern: new software is published ⇒ the port is updated ⇒ a new package is available.

For this reason, I would recommend not to try to ship FreeBSD packages on the OpenSearch website, but make it easier for FreeBSD contributors to maintain a port of OpenSearch which will make OpenSearch available as a regular package for all FreeBSD users.

Packaging of OpenSearch

For Java applications, the process described above is often a bit different: build dependencies are sometimes tedious to manage at the port level, and since java code is portable (can run on any platform with a JVM), a lot of FreeBSD ports for Java application download an artifact from the vendor and not the source code. The port maintainer has then just to manage dependencies, ensure the right files are placed at the right place, manage rc.d scripts and it is basically all: no "compilation" step is required. A drawback is of course if some modification of the source is required, then the maintainer has no real other choice than integrate the change with upstream and wait for the next release.

The FreeBSD packages shipped by FreeBSD are built using a set of "default" options which a user can override by compiling their ports (in the old-school way, before we had packages), or building their own packages. This can be useful for example to use a specific version of say python, ruby, PostgreSQL, java and so on.

This mean that an OpenSearch FreeBSD package will not ship with a JVM. The port will say "You need at least Java X", the FreeBSD port will be built with Java Y so the package will have a dependency on Java Y, and user will be able to build their packages using Java Z if that is what they want/need.

So, the tarball made available for 1.0.0 basically fit as a FreeBSD "source" tarball, even if it bundle a lot of thing we do not care about (mainly the non-portable stuff that lead you to ship multiple packages for different architectures).

Maybe a "lite" / "portable" tarball which does not bundle java and sample certificates would be a better fit, avoiding to download unneeded data?

Maybe this can be generalized with 2 sets of packages:

1. a bunch of all-in-one tarball that contain everything needed to try out OpenSearch quickly (basically what is the 1.0.0 tarball);
2. a "production" tarball that only contain OpenSearch itself: no java, no sample certificates, and so on.

Packaging of OpenSearch-Dashboards

Just like OpenSearch, a FreeBSD package would not bundle node.

Currently, as far as I understand, the kibana package is built using the kibana-7.14.0-darwin-x86_64.tar.gz archive as a source, applying some local patches on top of it, and rebuilding the shipped node libraries (which contain some C/C++ that has to be rebuilt).

This look quite ugly, but I am not aware of a cleaner way for projects in JS which include some C/C++ code (I do not hack in JS at all).

We can probably apply a similar strategy for opensearch-dashboards?

I am available for chatting about this 😄 Thanks!
(I try to attend the Community Meeting)

[smortex]

@peternied if I am right, the minimal archives do not include plugins such as the security which is definitively something we want to have support for on FreeBSD.

For now, the WIP port is based on the "full" Linux distribution and ignore the non-portable content. The port is being tested by a few people who are working on it, we still have a few minor things to tweak but I think that we will commit it to the FreeBSD ports tree soon™ 😉

PR: smortex/freebsd-ports#3

I can follow-up here when we have committed the port and it is available to all FreeBSD users and request the issue to be closed?

[dblock]

Let's close this! @smortex thank you & everyone else who contributed.

[hackacad]

Not sure about the roadmap (separate packages for very os/distro? rpm, pkg, deb etc.), but imho the shebang fixes should be done during packaging. RC scripts can be added anyway.

[hackacad]

I added some basic checks and env in the following PRs.
This would help to create a FreeBSD port.

opensearch-project/OpenSearch#1013
opensearch-project/OpenSearch#1014

opensearch-project/OpenSearch-Dashboards#678

[smortex]

@dblock, @peternied the ports have just been committed to the FreeBSD ports tree:

https://cgit.freebsd.org/ports/commit/?id=7fe660d7479c2a88f1c3aac267470f2a507757a0
https://cgit.freebsd.org/ports/commit/?id=19e156d5142115dcd542d47ee329a2656798d789

User of FreeBSD can now build OpenSearch and OpenSearch-Dashboards, and in a few days packages should be available for them.

Do you want we add some kind of installation instructions for FreeBSD on the OpenSearch website?

And I think this issue can now be closed.

[smortex]

Yes, we should! @smortex those docs are opensearch-project/project-website, and I think we should spell out how to install on FreeBSD.

Here we go: opensearch-project/project-website#321

Also, after 1.1 is released, what's there to do to make a FreeBSD release? We should have those instructions somewhere (maybe here)?

We have setup a FreeBSD OpenSearch Team composed of both committers and external contributors (and this group will be happy to see more contributors). So we will try to keep the port updated in a timely fashion when a new release is available. The process being open, if people think we are too slow, they can work themself on the update and propose the change. This whole process is documented in the FreeBSD Porter's Handbook, the most relevant sections being:

• Upgrading a Port
• Submitting the New Port

Thanks!

[dblock]

I'm going to un-assign myself and @nknize because we're not working on this, but I think we do have most bits and pieces to make this happen. I would start by trying to building the bundle from the top on FreeBSD and opening specific issues, such as "remove JDK in opensearch-min on FreeBSD". If anyone wants to take it on, I'd be happy to help/review PRs.

[dblock]

Note that the OpenSearch PRs @hackacad made (thanks!) were merged. The one on dashboards is still open, but hopefully soon.

[smortex]

We are working with @hackacad in a fork of the FreeBSD ports repo to complete the opensearch and opensearch-dashboards ports (opensearch branch):

https://github.com/smortex/freebsd-ports/tree/opensearch

We probably still have a few minor issues, but while I am typing, I could pkg install opensearch opensearch-dashboard in a jail and ingest data. So I expect we will be able to commit this soon in the official FreeBSD ports tree.

[dblock]

We are working with @hackacad in a fork of the FreeBSD ports repo to complete the opensearch and opensearch-dashboards ports (opensearch branch):

https://github.com/smortex/freebsd-ports/tree/opensearch

We probably still have a few minor issues, but while I am typing, I could pkg install opensearch opensearch-dashboard in a jail and ingest data. So I expect we will be able to commit this soon in the official FreeBSD ports tree.

Well, this is ... rad.

[peternied]

[Triage] @dblock What should we require to resolve this issue, do we need the whole distribution or just min?

[dblock]

Yes, we should! @smortex those docs are https://github.com/opensearch-project/project-website, and I think we should spell out how to install on FreeBSD.

Also, after 1.1 is released, what's there to do to make a FreeBSD release? We should have those instructions somewhere (maybe here)?

[dblock]

@smortex 🚀 I've made a placeholder for releasing OpenSearch here. We'll be populating this with a ton of things that we do before a release. Care to add a "Releasing for FreeBSD" section?

[peternied]

[Triage] @stockholmux Could you update when we have next steps?

[stockholmux]

@peternied Based on the update to #546, I think we're good to go on this.