notqmail / notqmail

The collaborative Open Source successor to qmail and netqmail

notqmail 1.08

20200520

notqmail 1.08 was produced by this motley krewe:

Rolf Eike Beer <[email protected]>
Manvendra Bhangui <[email protected]>
Alan Post <[email protected]>
Amitai Schleier <[email protected]>

qmail is a secure, reliable, efficient, simple message transfer agent. It is meant as a replacement for the entire sendmail-binmail system on typical Internet-connected UNIX hosts. See BLURB.md, BLURB2.md, BLURB3.md, and BLURB4.md for more detailed advertisements.

INSTALL.md says how to set up and test qmail. If you're upgrading from a previous version, read UPGRADE.md instead.

See PIC.* for some "end-to-end" pictures of mail flowing through the qmail system.

See https://cr.yp.to/qmail.html for other qmail-related software and a pointer to the qmail mailing list.

Other documentation: https://cr.yp.to/proto.html shows solutions to several Internet mail problems; many of these solutions are implemented in qmail. CHANGES.md and THANKS.md show how qmail has changed since it was first released. SECURITY.md, INTERNALS.md, and THOUGHTS.md record many of the qmail design decisions.

notqmail's Issues

Solaris make does not support "make -C"

Tested it on a Solaris CI (coming soon). I'm not sure if we want to fix it, but I think it should not get lost.

Check if we can get rid of queue/lock/trigger on some platforms

qmail-queue uses the pipe in queue/lock/trigger to signal qmail-send that a new message is ready for processing. This can be avoided on newer platforms, where one could use e.g. inotify on Linux to watch the todo directory for new files. This would get the notification automatically when new files are created, and avoids scanning the directory afterwards, as the event would already return the filename of the new file.

Feature request: TLS and SMTP AUTH support in qmail-remote

As a notqmail user, I want to relay mail securely to other hosts. Currently qmail-remote does not support SMTP AUTH at all, but e.g. the smtp-auth branch does. qmail-remote also needs to support TLS to not send the credentials in plain text.

For current best practices, at least PLAIN and LOGIN should be supported.

Make it easy to configure (the build), build, and install

It would be great to make it easy to configure (the build), build, and install notqmail. Doing all that by hand is a pain right now. Amitai Schleier has done an excellent job of automating all of that in pkgsrc with the qmail-run package, but obviously that requires using pkgsrc, and I worry that that is a big enough barrier to entry for some people that they won't consider notqmail.

Even though I'm not a big fan of Autotools, being able to do ./configure && make && make install is pretty convenient, and a lot of package management systems support it. If not actually Autotools, it could be an Autotools workalike. (Or something else, I suppose, even though not as standard as the Autotools interface, assuming it requires running just a few commands, would really be a big improvement.)

This would also increase the chance of notqmail being included in other OS package repositories (e.g., Homebrew, RHEL, Ubuntu, Arch, Gentoo, Alpine, etc.) which would also be a big plus. It's certainly convenient for someone intending a production deployment, but it's also convenient for someone wishing to try it out.

Lastly, this would make it easier for someone writing a tutorial. There are a number of tutorials on the Internet on how to install and configure software X on operating system Y. For example, DigitalOcean seems to host a number of these (not surprisingly since I assume they hope people will use their services for hosting), and a Google search (e.g., how install postfix rhel) will often find them. When a tutorial author can write, e.g., "First, install Postfix: # yum install postfix," that's really easy compared to what they'd have to write without a binary package being available in the OS's package repository.

add const to function parameters and variables

[ENHANCEMENT] Expand the "Why not Postfix?" Section

Currently the 'Why not Postfix?' section simply says:

Postfix is very good. We also really like qmail.

For those trying to chose between the two, this doesn't inspire people (or at least me personally) to use the product. Perhaps a more in depth comparison could be used, or even what features are available in NotQmail over Postfix.

Add manpage for qmail-todo

The branch netqmail-ext-todo contains ext_todo-20030105.patch, amended for consistency with the rest of the netqmail codebase.

This patch fixes the "silly qmail syndrome" by moving queue todo processing out of qmail-send in to it's own program, qmail-todo. Here it joins qmail-clean, qmail-lspawn, and qmail-rspawn.

A manpage for qmail-todo was not provided in this patch. Using qmail-clean.8, qmail-lspawn.8, and qmail-rspawn.8 as examples, write a qmail-todo.8 manpage for inclusion along with this patch.

Note also that qmail-start.9 and qmail-send.9 must also mention qmail-todo.

Missing closing parenthesis in 1.07 release description

The last item (Remove vfork(), fixing macOS runtime) in the notqmail 1.07 release description is missing a closing parenthesis for the referenced PR.

I know this is trivial, and I know that I can edit it myself, but I don't know if doing so would mess up the official release. Will undesirable things happen if I fix it (i.e., click Edit, fix the description, and click "Update release")?

Include support for slashpackage-format packages

Background

qmail services, and by extension notqmail services, are expected to be run under daemontools-style process supervision. While this is not required, the legacy of public opinion on the qmail mailing list and in the netqmail/Life with qmail documentation encourages it.

daemontools-0.76 is distributed in "slashpackage format", a djb-devised source code packaging format with a number of advantages over other packaging conventions. The slashpackage format and its design goals are documented on cr.yp.to.

While "slashpackaging" has not seen widespread adoption, a number of people including myself have taken to porting their software into slashpackage format for easy deployment, installation, and upgrading with package-provided scripts called ./package/compile, ./package/upgrade, and ./package/install (usually ./package/install just calls ./package/compile and ./package/upgrade directly).

What this could mean for notqmail

A slashpackage package has a strict path that it must follow because package names are globally allocated. Underneath, for example, "./mail/notqmail-v.ers.i.on/", are a ./package directory and a ./src directory.

The ./src directory contains the source code for the package. To simulate this structure on a local git repo:

mkdir -p ./mail/notqmail-version/src
chmod +t ./mail
ls -1 | while read f; do git mv $f ./mail/notqmail-version/src/; done

While this is simple enough for a user to do this when building a notqmail slashpackage, the user would also need ./package/compile and ./package/upgrade scripts that can build the software and install it locally.

notqmail could have an --enable-slashpackage option, similar to skarnet's skalibs/execline/s6 software which allows for slashpackage-style building and deployment. To avoid forcing slashpackage format on everyone, the compile and upgrade scripts could probably live in a subdirectory of the repository. For example, we could move the notqmail source into a ./src subdirectory, which is commonly seen in software projects, and put slashpackage details and script into, for example, ./contrib/slashpackage.

If we intend to make notqmail compliant with the usual "./configure && make && sudo make install" mantra, it could also allow for "./configure --enable-slashpackage && ./package/compile && sudo ./package/upgrade".

slashpackaging qmail - experimental notes

I've converted netqmail-1.06 into an experimental slashpackage that seems to work as expected. Binaries are still installed with a lightly edited hier.c that puts them into "/package/mail/netqmail-1.06/commands" with their necessary ownership and modes. In the same way auto_qmail.h reads ./conf-qmail, I added an auto_slashpackage.h that reads ./conf-slashpackage at compile time. (This is a chicken and egg problem. If your /package dir is really in, say, /var/package, you would need to edit this file after extracting the source under /var/package before you can successfully compile it.)

Platform note: On OpenBSD the /var partition is by default mounted with the "nosuid" option, which breaks qmail-queue. If I do not mount /package under /var, this saves me a deployment gotcha that I've only been tripping over for about 15 years. If I can avoid a dedicated /var/qmail partition, and I don't have to guesstimate the maximum allowable size of my mail queue, and I don't need to reserve that much disk space on the machine for no other purpose -- forever -- then that makes me happy.

With the qmail-* binaries installed under /package, I then symlink them into /var/qmail/bin so that the system can still find them as needed. Unlike a normal slashpackage deployment, I do not symlink these commands into /command and then symlink those links into /usr/local/bin for the same reason I don't put /var/qmail/bin into the default user path. This is personal preference and may go against the spirit and letter of the slashpackage law.

syncdir, or something like it

According to DJB's reliability FAQ:

qmail's queue, except for bounce message contents, is crashproof on the BSD FFS and most of its variants.

Do not use async or softupdates filesystems. If you do, and if your system crashes at the wrong moment, you will lose mail. Under Linux, make sure that all mail-handling filesystems are mounted sync. The same comments apply to many other popular MTAs. (However, some MTAs are unreliable no matter what filesystem you use.)

It is safe to put qmail's queue on a noatime filesystem.

According to qmail.org:

Bruce Guenter's syncdir gives qmail bsd fsync semantics on a Linux filesystem.

syncdir is a library providing replacement implementations of open, link, unlink, and rename that wrap the real system calls and also invoke fsync.

In pkgsrc, which targets a couple dozen Unix platforms, mail/qmail simply takes an unconditional dependency on devel/syncdir. (For portability, I've libtoolized syncdir and provided an alternate implementation using dlsym() for platforms like macOS where syscall() is deprecated or gone.) Some platforms might not need this, but I chose to err on the side of safety.

For notqmail, since people will install on a wide variety of systems, I also think we ought to default to safety. Some options for how to do that:

1. Make syncdir an optional prerequisite for the build, defaulting to failing the build if it's not present
2. Vendor a copy of syncdir into our tree, enabled by default
3. Provide our own fsync-calling wrapper functions that don't shadow the syscall names, and adjust all callers to go through our wrappers

It's probably reasonable to provide a compile-time option for folks who are sure they don't want all the extra fsyncs.

Caveat: syncdir is GPL. This is the first bit of GPL code we might want to include, but it won't be the last (I'm aware of at least qmail-spp, slated for 1.9). I don't know how GPL interacts with public domain. If need be, we can ask Bruce Guenter whether he'll grant us different conditions for use of his code.

Use mess822

DJB's TODO said:

- use mess822 in qmail-inject
- use mess822 in qreceipt
- use mess822 in qbiff
- use mess822 in maildirwatch

So we might want to do these things.

LWQ (Life with qmail): update or fork

LWQ is the community's agreed best documentation for installing and running qmail. As we continue to diverge from qmail and netqmail, LWQ will become less accurate -- unless we are able to update it, or otherwise get it updated.

For instance, with notqmail 1.07, LWQ already doesn't describe installation from binary packages.

It's possible to fork LWQ:

Life with qmail is covered by the OpenContent License, version 1.0. See http://www.opencontent.org/opl.shtml for the full license. Basically, you can copy, redistribute, or modify Life with qmail provided that modified versions, if redistributed, are also covered by the OpenContent License.

But of course we'd rather not need to. And it's not at all inconceivable that Dave Sill would still be willing to update it and/or share write access with one or more of us.

GitHub using ./SECURITY as project security policy

GitHub is using the file in the repo at ./SECURITY as the project's security policy, but that file is not really the security policy for the project. If you click on the project's Security tab and then Policy, it displays the contents of ./SECURITY.

The project's security policy is also referenced in a notification the first time a user submits an issue to this project:

Ideally, I think it would be best if the security policy could point to a different file. I found Adding a security policy to your repository, but it wasn't immediately clear to me that it is possible to set the security policy to a different file. If not, then I think the best thing would be to move ./SECURITY somewhere else, and create the project's security policy at ./SECURITY or ./SECURITY.md (or not have a security policy at all).

"Use of a standard library function that is not thread-safe" (CodeQL #1 and #2)

CodeQL wants us to know that two places in predate.c are not thread-safe:

• Line 60 gmtime()
• Line 69 localtime()

predate is a tiny client program, and seems quite unlikely to me to be ever made multi-threaded or copied for use in some other multi-threaded program.

OTOH, if it's portable to just use the _r variants, maybe we should just do that.

On the third hand, is this a program we can announce intent to delete? In-tree, it's used only by datemail, a tiny wrapper around sendmail -- and then datemail is used in-tree only as the answer to FAQ 6.1 about time zones with BSD mail(1).

Two somewhat coupled decisions here:

1. Fix CodeQL report by dismissing these reports, or by switching to _r?
2. Plan to remove predate and/or datemail after 1.09, or plan to keep them?

Check if "char*" must equal "signed char*"

There are some places where unsigned char is used to get unsigned semantics. Check if char really means "I don't care" everywhere, or if it expects them to be signed at some places. There are platforms like ARM where char's are unsigned by default.

is the variable firstwhen used uninitialized in tcpto.c?

If one were to compile the qmail code with -Wall -Werror, the following error is produced:

./compile tcpto.c
tcpto.c: In function ‘tcpto_err’:
tcpto.c:139:34: error: ‘firstwhen’ may be used uninitialized in this
function [-Werror=maybe-uninitialized]
      if ((firstpos < 0) || (when < firstwhen))
                            ~~~~~~^~~~~~~~~~~~

Is there a condition where firstwhen is actually used uninitialized, or is the compiler issuing a spurious report about a condition that cannot in truth happen? If someone is interested in tracing this code out and reporting back, this investigation makes a good first issue.

get rid of str_len()

strlen() is one of the functions that a compiler usually optimizes away, given much more efficient implementations can be done on todays SIMD processors. The old implementation uses loop unrolling, a common 90's optimization technique that has usually lost it's benefit today.

configuration for systemd?

Is there a recommended configuraton for running it purely under systemd (e.g. no daemontools)?

Specify public key for verifying signatures for downloaded tarball releases

I've not submitted a PR because I don't know how the signer would like this to be specified.

I downloaded https://github.com/notqmail/notqmail/releases/download/notqmail-1.08/notqmail-1.08.tar.gz and https://github.com/notqmail/notqmail/releases/download/notqmail-1.08/notqmail-1.08.tar.gz.sig recently. I wanted to verify the gzipped tarball according to the signature but there was no link to the public key so that I can use gpg --verify to use public-key cryptography to verify the download.

I ended up asking around in IRC, then doing a Google search for a username that I don't know anything about. I'm new to the qmail world. I ended up finding the (apparently) real name of the person who signed the tarball. I then searched for their name on Google to find their public key fingerprint, and finally used gpg --search-keys to download the public key from a keyserver while hoping that the public keyserver system is presently working.

I know how to use GPG. The problem was that I didn't know which public key to install to verify the signature. It would be very useful to include this in the installation instructions.

Like I mentioned, I would have written the PR myself, but it's not my key and after a recent conversation in #qmail on Freenode, it seems to me that it would be better to leave this to the signer. I would be happy to help write this piece of documentation, however, so if I can help, let me know.

No support for IPv6-address-literal

Just dropping this here, addrparse in qmail-smtpd.c doesn't support IPv6-address-literals ala root@[IPv6:2001:db8::25].

https://tools.ietf.org/html/rfc5321#section-4.1.3

Relook at /var/qmail for everthing

Most linux distribution follow FHS standards. Look at http://refspecs.linuxfoundation.org/FHS_3.0/fhs-3.0.html

We should relook at having binaries in /var/qmail/bin, man pages in /var/qmail/man, etc. These directories also cause warnings/errors when you use a packing build service like openSUSE build service. rpmlint also lists such violation of FHS as error instead of warning. One of our goalis should be to make notqmail instlallation trivial for many class of users, something as easy as

yum install dotqmail
or
apt-get install dotqmail

The change to FHS is not easy. Many of the qmail programs have
chdir("/var/qmail");
....
args[0] = "bin/qmail-local";
args[1] = "--";
args[2] = x;

If the goal of dotqmail will never be seeing it installed as a standard (like postfix, sendmail, exim), then this is not required. But if it needs to be made poplular with all class of users, then we should take a relook at /var/qmail for everhing. Even something like /opt/qmail will serve better.

License, unlicense, etc.

notqmail's direct ancestor qmail is public-domain, but that may not be legally valid everywhere on the planet. I just encountered https://unlicense.org and wonder if that's what we might want for notqmail.

Make regular use of code analysis tools

Very generally, there are things people do in 2019 to get earlier and more automated help finding bugs in C code. I certainly don’t know all of them, or which ones are worth the costs for the risks they mitigate.

One idea I’m suggesting specifically here: starting to run some sort of static analysis tool, and some sort of memory-misuse checker, in our own working trees. Perhaps we can require certain kinds of results from these tools before merging a PR. Later, when we routinely get clean results for the whole tree, we can include these checkers in CI.

Investigate generating Makefile

Hand-editing Makefile is error-prone. DJB probably used his redo. Since that's not available to us, maybe Avery Pennarun's redo would help?

Reduce duplication in Makefile

Noted by @DerDakon: we've inherited lots of what look like repetitively open-coded Makefile rules. For instance,

env.a: makelib env.o envread.o
        ./makelib env.a env.o envread.o

It's possible that they were generated by redo, in which case we might like to start doing the same (see #28).

Alternatively, we might like to make greater use of suffix rules, taking care to preserve compatibility across BSD and GNU make. If we go this route, later is probably better, to minimize the risk of breaking patches while users are still depending on them.

Convert the code to Rust

Rust language offers many things compared to C:

• Rust is memory safe. By design, Rust code can't have dangling pointers, buffer overflows or other types of memory-related errors.
• Rust is as fast as C/C++ while it is far safer.
• Rust is general-purpose language that can be used in any purpose.
• Rust is great at concurrent programming.
• Rust has an inbuilt dependency and builds management tool known as Cargo.

It will simplify documentation generation, build system, Windows support, which will help to avoid some annoying bugs and still be able to provide the interface as a C library for any external user.

There is an available automated tool called c2rust that offers automated conversion from C to Rust, along with the refactoring tool which allows scripting in Lua.

There is a successful example of the ongoing rewriting in Rust of the LaTeX engine - Tectonic, all while it is still able to pass the tests (building arXiv papers to PDF) - see crlf0710/tectonic fork.

Default to -Wall (but not -Werror)

1. Get to warning-free builds in our usual dev environments

2. Submit a PR adding -Wall -Werror to conf-cc (and maybe shrinking our CI build matrix)

3. Test it on every platform we can get our hands on

4. Fix whatever comes up

5. Merge

6. Rejoice!

7. Proceed directly to #48

notqmail.org should redirect with a 307, not 301.

Inspired by #39, where we publish our homepage in our SMTP protocol negotiation. Here are the HTTP headers for that request:

$ curl -D- https://notqmail.org/
HTTP/1.1 301 Moved Permanently
Location: https://github.com/notqmail/notqmail/wiki
Content-Length: 0
Date: Mon, 22 Jul 2019 00:45:48 GMT
Server: lighttpd/1.4.54

Note we redirect to our GitHub wiki using a 301 status code. I'd like this to be changed to 307 Temporary Redirect.

It's existentially possible, if not likely, that we'll want a landing page in the future that isn't permanently tied to our GitHub wiki page. Using 307 instead of 301 will change the way a browser caches the response, in favor of making it easier to change where the site redirects to.

share responsibility for notqmail.org

For the notqmail.org domain, I'm currently the only person who can modify its

• DNS registration
• DNS entries
• HTTP{,S} config
• MTA and mailing-list config

@alanpost can provide a Virtual Private Server dedicated to notqmail.org hosting, such that all this stuff can be hosted there and we can share administrative control. As for the DNS registration, I'll gladly add two of you as contacts, unless we have a better idea for that.

Stop redeclaring system functions

We should really include <stdlib.h> and friends and get rid of all the handwritten forward declarations.

I don't think anyone remotely sane should care about SunOS or equally outdated system compatibility anymore. My rule of thumb would be: if it hasn't seen a decently complete C99 implementation we should not care. These machines should be kept away from the internet as far as possible. Oh, wait, the 80's have called and want their rsh security holes back…

fix "silly qmail syndrome"

The silly qmail syndrome means you don't deliver as fast as you could if you have a high rate of new mail being injected.

There are 2 versions of a fix, one simply restricts todo processing to only run once every few seconds, the other one moves todo processing to an external process called qmail-todo. Both patches can be found on nrg4u.com.

Add new project members

I hereby propose to add 2 new project members:

• @leahneukirchen
• @josuah

Both have shown a good understanding of the code, valueable feedback and first PRs. What more can we expect?

So, poll for the already existing project members: what do you think?

qmail-pop3d might get run as root

While developing acceptutils, I found a surprising behavior in POP3 service: checkpassword will authenticate root, and then qmail-pop3d will run as root. Substituting id for qmail-pop3d:

$ pgrep -flu 0 pop3
2834 /opt/pkg/bin/sslserver -ne -vRl0 -x /etc/qmail/control/tcprules/pop3.cdb -c 20 127.0.0.1 110 /opt/pkg/bin/qmail-popup miracle-dirt.schmonz.com /opt/pkg/bin/nbcheckpassword /usr/bin/id

$ telnet 127.0.0.1 110
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
+OK <[email protected]>
USER root
+OK
PASS <root-password-here>
uid=0(root) gid=0(wheel) groups=0(wheel)
Connection closed by foreign host.

This was almost certainly unintended. It's inconsistent with qmail's design, because qmail-local will never deliver anything to root's Maildir for qmail-pop3d to serve. And it's inconsistent with qmail's security focus, because it lets abusers dictionary-attack the root password over the network.

In acceptutils, I addressed this by adding checknotroot (.c, .8), a tiny program that sits between checkpassword and qmail-pop3d. If getuid() returns 0, it writes loudly to the logs, then exits 1. With checknotroot in the chain, when someone manages to guess the root password, they won’t know they did: it looks exactly like any other failed checkpassword login.

This may or may not be the right fix for notqmail, but we'll want to do something about it for 1.08.

qmail-remote: remove CNAME handling in addrmangle

Currently, when qmail-remote sends a mail (not to a RELAYHOST) to [email protected], and bar.org is a CNAME to quux.org, addrmangle will rewrite the envelope address to [email protected]. Aeons ago, this was deemed a good idea. Nowadays, at least Postfix (since version 2 from 2002) and OpenSMTPD (never) don't do it. Sendmail can do it, but apparently doesn't by default.

RFC 5321 (from 2008), Section 5.1 clearly specifies what MTA should do today:

The lookup first attempts to locate an MX record associated with the
name. If a CNAME record is found, the resulting name is processed as
if it were the initial name.

djb wrote in https://cr.yp.to/im/cname.html:

Most implementors agree that this feature is not worth the implementation costs. I recommend that, before January 2001, sites stop relying on client-side CNAME translation, so that implementors can safely remove the feature at that time.

It is easy to patch out by considering flagcname to be constant zero.

Remove the custom allocator (alloc.c)

What does this do at all? It has a 4096 byte static allocation and serves small memory blocks from there as long as they fit, probably to avoid crappy slow system allocators. I say: kill it. Here are some things I have in mind why we should do it:

• it's as simple as possible, probably intentional. This makes things like alloc_re() (i.e. realloc()) inefficient as it does not look if the thing it reallocates is already the last thing in the batch, and simply extends that, or if alloc_free() has the last block and adds things back to the pool (ideally with zeroing for security reasons).
• it makes it basically impossible to detect overruns on early small allocations with tools like valgrind, address sanitizer or friends as those allocations never hit malloc(), so are never traced by any custom tools. Any overrun will just go into the next variable, untracable.
• allocators do their own buffering, alignment, and so on. If they are broken, your system is severely screwed. And if we need more money^H^H^H^Hemory we will hit that buggy allocatator anyway, making things harder to detect as it will only happen on "heavy" memory usage. This is not TruOS, SunOS 1.x or something like that anymore. We expect a reasonably sane OS. Everything else will get us screwed in much more severe ways.
• it has shown issues when coming close to 4GiB allocations (#37, #109, older reports). DJB correctly says that an allocation of that size in your mailer shows that you have already done something wrong, but still: 64 bit platforms are the default, so using a 32 bit type to pass allocation sizes is a bug of it's own.
• why bother at all with writing an allocator?

Regression in 1.08: recipient-address qualification

The NetBSD script /etc/daily generates two emails to the sysadmin. It's called from cron like so:

22 3 * * * /bin/sh /etc/daily 2>&1 | tee /var/log/daily.out | sendmail -t

In notqmail 1.07 (and netqmail before that), both emails were being addressed To: [email protected].

As of 1.08, one of the emails is instead addressed just To: root. Replacing qmail-inject with 1.07’s is sufficient to restore the previous behavior.

ipme does not detect multiple network addresses on the same interface

Test setup: have multiple ip addresses configured on the same interface:

$ ip addr list | grep -e 'inet ' -e ^[0-9]
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
3: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    inet 172.16.198.3/16 brd 172.16.255.255 scope global noprefixroute dynamic wlan0
    inet 192.168.18.19/24 scope global wlan0

Ask ipmeprint about this:

$ ./ipmeprint 
0.0.0.0
127.0.0.1
172.16.198.3
172.16.198.3

Found here: https://bugs.gentoo.org/566826

I'm working on a patch.

Configurable validation of SMTP recipients

In 1.9 or so, we'll add Paweł Foremski's qmail-spp to qmail-smtpd. It's a very general-purpose API. Programs can very easily influence just about any SMTP behavior. Admins control which programs run for which events in control/smtpplugins.

qmail-spp is not the only patch that adds enough API to do recipient checking. A much simpler API, specific for this purpose, is Jay Soffian's RCPTCHECK.

Since RCPTCHECK is a subset of qmail-spp, we can easily support both APIs. In my rejectutils, qmail-rcptcheck runs the sequence of RCPTCHECK-compatible programs in control/rcptchecks, rejecting if any of them reject. (It runs under qmail-smtpd with either the RCPTCHECK patch or the qmail-spp patch, though only the latter is important for us here.)

rejectutils also includes several RCPTCHECK-compatible programs, repackaged from familiar SMTP recipient validation patches:

• qmail-rcptcheck-realrcptto is derived from Paul Jarc's realrcptto
• qmail-rcptcheck-badrcptto is derived from Ward Vandewege's badrcptto
• qmail-rcptcheck-qregex is derived from Andrew St. Jean's qregex

(There are a couple other programs in rejectutils that may be interesting for notqmail, but probably not for this precise issue.)

rejectutils does not include every SMTP recipient validator anyone might want. That's fine, because qmail-spp provides hooks for every SMTP everything anyone might want, and we plan to freely accept extensions.

when we broke the TLS patch, we didn't notice immediately

Sometime after 1.08, the TLS patch stopped applying to our tree. It's good we noticed this now so we can repair the situation before our next release. Some options:

1. Prepare a TLS patch rebased onto 1.09
2. Roll back our breaking changes and submit them as new PR(s) that document our known preconditions for merging
3. Coordinate with the TLS patch maintainer to publish a timely new version for notqmail 1.09 when we're ready to release

As usual, I'm extremely not excited about (1). We haven't maintained our rebased patches particularly well, and I don't blame us: they're easy to forget about, and if we do remember, the work isn't any fun. And users who are considering notqmail -- or already using notqmail and trying to track updates -- will much prefer being able to apply the patches they already have and know well. (I'm sure there'll be cases where we can't avoid providing rebased versions of known patches. Still, the fewer the better.)

(3) would be a pretty good outcome: minimal work for us, and users have already been trained to seek official updates to that patch over the years. If I were maintainer of a popular qmail patch, I'm not sure I'd be willing to make the effort to do this. But we can certainly ask. If he is willing, the tradeoff here is that we've made the usability of our release depend on a third party. What if he changes his mind, or simply has no free time when we need him to?

My preference is for (2), especially since our breaking changes in this case appear to be small. For instance, we need to roll back calls to the safe{read,write} generator macros in qmail-{remote,smtpd}.c and restore the open-coded functions so that the TLS patch can modify them.

---

Once we've preserved TLS for 1.09, we need to solve the meta-problem: in our current dev process, important patches can break and we might not know until too late.

@josuah's auto-patcherator is awesome, but we have to remember to look, and it seems to be the case that we haven't remembered. Again, I don't blame us. Our PR flow is already somewhat complicated.

Before we release 1.09, we need an automated way to be notified promptly that we broke a patch we intended not to break, so that we're sure not to release 1.09 with unintended breakage.

The cheapest time to avoid unintentional breakage, of course, is before merging a PR. But we already have to wait a while for the CI builds we've got. If that gets a lot longer, it'll be really tedious. The next-cheapest time to find out we broke something, then, is shortly after the merge.

The proper response to a patch-breakage notification is not necessarily "block the merge" (or "roll it back"), just like it might not be in the present TLS case. My goal here is not that we never decide to break an agreed-important patch. It's that when we make such a decision we're aware of it, we've considered our options, and we think breaking that patch is our smartest available move.

Some ways this notification could be implemented:

• Pre-merge check and/or post-merge "nightly build"
• Running at GitHub and/or on OS package autobuilders (like Gentoo, which is how we discovered the current breakage)
• Email to the dev list

We'll also need to agree on a short list of patches we want to never be surprised about.

Email Address Internationalization

Russ Nelson has made very few updates to qmail.org (as of this writing, site's been down for a few weeks) in the last several years. One was pretty significant, though: Russ added Arnt Gulbrandsen's Email Address Internationalization (or EAI, or SMTPUTF8) patch to the very small "Recommended patches" section.

Coming from Russ, this is a strong recommendation. Should we include EAI in 1.08? I played with this patch a year or two ago, have integrated it into pkgsrc qmail, and would suggest from foggy memory that this is not an easy "yes":

• I found and fixed at least one bug that prevented other qmail-smtpds from accepting mail from an EAI-enabled qmail-remote
• IIRC, the patch claims it applies atop TLS but actually the SMTP AUTH patch is what it expects to be present
• I don't remember the code being as clear and expressive as I'd wish for (possibly related to why it had at least one bug)

For reference:

• Arnt's repo with various branches
• The EAI patch I provide in pkgsrc

Mark functions and variables static where possible

Many functions and variables are only of interest in the file they are declared in. Check which of those can be marked static to allow better code locality checking e.g. by the compiler.

One must take care that e.g. some of these items may be used by external patches.

Remove code for QMTP.

According to DJB in 1997, QMTP is a replacement for SMTP. This doesn't seem to have happened.

Our next release announcement (1.09, as of this writing) can include our intent to remove this code. After that release goes out, barring persuasive objections, we can remove these files and references thereto:

• qmail-qmtpd.8
• qmail-qmtpd.c

Test that popular patches still apply cleanly

The wiki says "Expect your patchset to mostly continue to apply".

To automatically keep paying attention to this, I'm thinking we need an automated regression test:

1. Select a baseline set of popular patches
2. Verify that each patch individually applies and builds
3. Maybe verify certain combinations too

This can be the first of a suite of automated tests that will grow to protect us against more and more mistakes we want to avoid making. (Later, when we have a build farm, we can run them on every commit.)

Feature request: relaying to different smarthosts

As a notqmail user, I want to relay mail via different smarthosts according to the Sender or From address, e.g. to send work mail through the work SMTP server and private mail over a different host.

Together with #174, this also needs to support different credentials per smarthost.

Add support for out-of-tree build

Something like

mkdir build
cd build
make -f ../Makefile

This helps having parallel things and avoids things like TARGETS alltogether.

Consider queue safety on case-insensitive filesystems

Mac OS X's default HFS+ filesystem defaults to case-insensitive (and case-preserving).

With a couple small fixes, notqmail will build and installs on a typical OS X system, and casually appears to run fine. But it seems unlikely that DJB was thinking about case-insensitive filesystems when he designed qmail, in particular its guarantees about how the queue behaves.

In pkgsrc, the qmail package warns at install time if the queue is on a case-insensitive filesystem. If we can evaluate the queue code and draw a convincing conclusion, we can either not bother warning or not bother installing.

Add support for restricting access permissions

Many platforms support adding additional restrictions from the binary to itself or it's childs. Examples:

• OpenBSD has unveil
• FreeBSD has capsicum
• Linux has things from SELinux and Apparmor policies to something else I have forgotten

It would be nice if we could e.g. restrict qmail-send just because we can: after it's startup it only needs to modify the queue, and read only few things outside of it (and all should be in /var/qmail).

*BSD blocklist support

NetBSD and FreeBSD have a blocklist daemon and associated library (formerly blacklist) that provide a sensible alternative to fail2ban. Given a network service that's been adapted to invoke the library on both successful and failed logins, the sysadmin can configure blocklistd policy to dynamically block source-IP/dest-port combinations at the OS firewall when enough failed logins have happened quickly enough, and then unblock when it's been long enough.

It's a very easy API to integrate iff you can pass it the listening network socket. This probably works under tcpserver and probably doesn't work under TLS-enabled analogues, especially if they support delayed encryption, because the file descriptors they provide are pipe ends.

Talked it through with the blocklist author. The API needs to be this way for security reasons. If we want to integrate blocklist support for e.g. authenticated mail submission, two options (both involving patching the various UCSPI server programs and trying to get the changes upstreamed) are:

• Share the listening socket with the child program, as a predictably numbered fd or otherwise
• Link with libblocklist themselves and provide some small API for the child program to invoke it

Use CLOEXEC flag where possible

This flag prevents that descriptors are shared beyond an exec*() call. Setting it via fcntl() on pipes or requesting it directly in the wrappers in open.h would remove the burden to close a lot of them. We can not use it unconditionally, as sometimes the descriptors actually should be passed down.

Changes required for RPM Build

So I spent some time setting up opensuse build service (obs) for notqmail. The RPM builds are available at

https://software.opensuse.org//download.html?project=home%3Anotqmail&package=notqmail

This is work in progress and I still have to start creating the necessary files needed for debian builds (rules, changelog, notqmail.dsc, etc).

The good thing is that notqmail will build on all the known linux distros. So far notqmail was found to build successfully on the following distros
centos7, centos6, rhel7, rhel6, FC30, opensuse Tumbleweed, opensuse leap 15.1, opensuse leap 15.0, suse linux enterprise 15 (& SP1), suse linux enterprise 12 (sp1, sp2, sp3, sp4)

To build binary packages, following are the issues (all suppressed by creating rpmlintrc file) and by modifying instchown.c,

• non fhs compliance
• instchown needs the same destdir changes as was done in instpackage.c. I will raise a separate pull request for all changes after I complete the debian build
• all calls to chown to be skipped when instchown is run with non-root uid. This is because rpm and debian builds is built using non-privileged user
• The license file COPYRIGHT is termed as illegal for suse builds. Somehow opensuse wants a known license like GPLv3, BSD, etc. So I have to figure out this license problem
• creating a systemd unit file to start qmail-send. I have modified hier.c to install notqmail.service in /var/qmail/boot
• The binary builds shouldn't be shipping /var/qmail/queue. It should be created during rpm/debian installation. This will require modification to hier.c and a seperate binary like queue-fix (e.g. /var/qmail/bin/queue-fix /var/qmail/queue) which will create the queue when invoked.