linbit / saas

Spatch As A Service

saas - Spatch As A Service

Starting from DRBD kernel module version 9.0.20 we use spatch to generate compatibility patches that are required to build the kernel module on a host machine. Older kernels need different and more patches than new ones as we try to use latest Linux upstream features as our base line in DRBD.

While using spatch makes maintenance a lot easier, it slightly complicates things for users just trying to compile one of our release-tarballs on their machine.

Our tarballs ship with a wide range of pre-generated patches, but we can never build for everybody (e.g., locally patched kernels, distribution kernels of distributions we don't build for in the first place,...).

Additionally, spatch is a beast on its own. As of now we require a very current version that is not packaged in any distribution, and even if, we actually don't like to force a ton of OCaml dependencies on everybody trying to build a release-tarball.

This is where saas comes into the picture. It is a web service, hosted by LINBIT, that takes as input a "DRBD tarball version", and a "compat.h" (which describes the capabilities of your kernel). From that, it generates a patch with spatch, and returns it.

API

For the regular user, this is handled transparently by our Makefiles. A manual call looks similar to this:

curl -d "$(cat compat.h | base64 -)" -X POST http://localhost:8080/api/v1/spatch/9.0.20-0rc2

saas's Issues

Libpython-dev is required for Ubuntu 22.04

When I trying to build drbd for Ubuntu 22.04. I see the errors:

root@spaas-77fbcff5cc-qf5jc:/# make -C /tmp/saas839853613/drbd-9.1.11/drbd compat
make: Entering directory '/tmp/saas839853613/drbd-9.1.11/drbd'
  GENPATCHNAMES   _
drbd-kernel-compat/gen_patch_names.c: In function 'main':
drbd-kernel-compat/gen_patch_names.c:303:3: warning: #warning "Unknown bio rw flags, check compat layer" [-Wcpp]
  303 | # warning "Unknown bio rw flags, check compat layer"
      |   ^~~~~~~
  SPATCH   _  _
    drbd-kernel-compat/cocci_cache/_/.compat.cocci
    : warning: line 24: should peer_device be a metavariable?
    : warning: rule starting on line 80: metavariable bioset not used in the - or context code
    : warning: line 489: bio, previously declared as a metavariable, is used as an identifier
    : warning: line 489: bio, previously declared as a metavariable, is used as an identifier
    : warning: rule starting on line 482: metavariable other_q not used in the - or context code
    : warning: line 552: bio, previously declared as a metavariable, is used as an identifier
    : warning: line 559: bio, previously declared as a metavariable, is used as an identifier
    : warning: rule starting on line 599: metavariable err not used in the - or context code
    : warning: line 1164: op, previously declared as a metavariable, is used as an identifier
    : warning: line 1167: op, previously declared as a metavariable, is used as an identifier
    : warning: line 1892: should nla_strscpy be a metavariable?
    : Cannot infer Python version
make: *** [Makefile:180: drbd-kernel-compat/cocci_cache/_/compat.patch] Error 255
make: Leaving directory '/tmp/saas839853613/drbd-9.1.11/drbd'

If I switch default python to python3, then python3-dev is still required

root@spaas-77fbcff5cc-qf5jc:/# make -C /tmp/saas839853613/drbd-9.1.11/drbd compat
make: Entering directory '/tmp/saas839853613/drbd-9.1.11/drbd'
  GENPATCHNAMES   _
drbd-kernel-compat/gen_patch_names.c: In function 'main':
drbd-kernel-compat/gen_patch_names.c:303:3: warning: #warning "Unknown bio rw flags, check compat layer" [-Wcpp]
  303 | # warning "Unknown bio rw flags, check compat layer"
      |   ^~~~~~~
  SPATCH   _  _
    drbd-kernel-compat/cocci_cache/_/.compat.cocci
    : warning: line 24: should peer_device be a metavariable?
    : warning: rule starting on line 80: metavariable bioset not used in the - or context code
    : warning: line 489: bio, previously declared as a metavariable, is used as an identifier
    : warning: line 489: bio, previously declared as a metavariable, is used as an identifier
    : warning: rule starting on line 482: metavariable other_q not used in the - or context code
    : warning: line 552: bio, previously declared as a metavariable, is used as an identifier
    : warning: line 559: bio, previously declared as a metavariable, is used as an identifier
    : warning: rule starting on line 599: metavariable err not used in the - or context code
    : warning: line 1164: op, previously declared as a metavariable, is used as an identifier
    : warning: line 1167: op, previously declared as a metavariable, is used as an identifier
    : warning: line 1892: should nla_strscpy be a metavariable?
    : Py.find_library: unable to find the Python library [libpython2.7m.so returned Library not found] [/usr/bin/../lib/libpython2.7m.so returned Library not found] [libpython2.7.so returned Library not found] [/usr/bin/../lib/libpython2.7.so returned Library not found]
make: *** [Makefile:180: drbd-kernel-compat/cocci_cache/_/compat.patch] Error 255
make: Leaving directory '/tmp/saas839853613/drbd-9.1.11/drbd'

gcc v12 can't write to /dev/null

Not sure if this is a bug, but better I'll report this

Just tried to run spaas on debian bookworm (gcc v12)
spaas is not able to generate patches, because it can't write to /dev/stdout

error:

2022-09-26T09:40:02.249Z	ERROR	spaas/main.go:399	Could not generate patch: Could not precompile compat.h, looks like it is invalid: exit status 1	{"type": "error", "remoteAddr": "192.168.199.108:31771", "code": 400}
main.(*server).errorf
	/usr/local/go/spaas/main.go:399
main.(*server).spatchCreate.func1
	/usr/local/go/spaas/main.go:189
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2084
github.com/gorilla/mux.(*Router).ServeHTTP
	/go/pkg/mod/github.com/gorilla/[email protected]/mux.go:212
main.(*server).ServeHTTP
	/usr/local/go/spaas/main.go:124
net/http.serverHandler.ServeHTTP
	/usr/local/go/src/net/http/server.go:2916
net/http.initALPNRequest.ServeHTTP
	/usr/local/go/src/net/http/server.go:3523
net/http.(*http2serverConn).runHandler
	/usr/local/go/src/net/http/h2_bundle.go:5903

debug:

# gcc -o /dev/null ./drbd-kernel-compat/cocci_cache/_/compat.h
./drbd-kernel-compat/cocci_cache/_/compat.h:1: fatal error: cannot write PCH file: required memory segment unavailable
    1 | #define COMPAT___VMALLOC_HAS_2_PARAMS
      |
compilation terminated.

without -o /dev/null compilation is successfully finishes:

# gcc compat.h
# ls
compat.h  compat.h.gch