clearlinux / tallow

Block hosts that attempt to bruteforce SSH using the journald API.

tallow

Tallow is a fail2ban/lard replacement that uses systemd's native journal API to scan for attempted ssh logins, and issues temporary IP bans for clients that violate certain login patterns.

Author: Auke Kok [email protected]

How it works

Tallow attaches to the journal and subscribes to messages from /usr/sbin/sshd. The messages are matched against rules and the IP address is extracted from the message. For each IP address that is extracted, the last timestamp and count is kept. Once the count exceeds a threshold, the offending IP address is added to an ipset and blocked with a corresponding firewall rule. It will use firewalld or iptables / ip6tables.

The timestamp is kept for pruning. Records are pruned from the list if the IP address hasn't been seen by tallow for longer than the threshold. If the IP was blocked and the threshold was exceeded, the IP is unblocked. If the threshold was never reached, the record is removed as well.

Pruning is done automatically after incoming messages are processed, so there is a chance that if no messages arrive, that IP addresses remain blocked for longer than the default blocking period.

Motivation

This program was originally written to demonstrate the journal API. One of the typical use cases for journal (or syslog) readers was to act dynamically on certain syslog messages, and many types of actions can be imagined. This is trivial to implement on systems that use the journal API, and often doesn't take much code at all.

The journal is attached to and forwarder to the end. We place a simple message filter, and then process each incoming message. For more information check out the sd-journal manual pages, which contain example code that demonstrates almost the exact same code flow.

Security

DISCLAIMER: THIS IS NOT A SECURITY APPLICATION !!!

Tallow is meant to reduce log clutter and system resource usage at the cost of denying access to potentially valid users.

Even if you reduce the threshold at which clients are blocked to 1, an attacker may still gain access to your server if the attacker uses the correct credentials.

By itself, tallow is an application that creates a Denial of Service. It's sole purpose and function is to block IP addresses. Therefore, with tallow running on a service, you could potentially deny valid users access to your systems if you deploy tallow.

Be very careful if you deploy tallow on systems that expect valid users to log on from many random source addresses. If your user mistypes their username, they could find themselves denied access.

tallow's Issues

ssh-copy-id results in immediate lockout

I can pull detailed logs if it would help, but the basic use case seems consistent with each of the ~half-dozen CL hosts I've deployed in the cloud:

1. SSH to a remote host using password auth for the normal installer-created user, then log out: ✅
2. Run ssh-copy-id to install the local client (ed25519) SSH public key into authorized_keys on the same host
3. ...watch the copy hang at INFO: 1 key(s) remain to be installed..., after which tallow has banned the client IP: 😢

I'm guessing there's a logfile parsing issue where the handshake used to query existing keys appears as a failed auth and results in the ban, but this exact workflow is part of how I bootstrap a new server or workstation so it's honestly a PITA that tallow locks me out 100% of the time when I do it, after which I have to log in via the actual server console and whitelist client IPs and flush firewall rules to unlock my access.

Possible security issue: any IP can be blocked

Usernames in SSH log entries can actually contain spaces. If someone tries to login as x x x 1.2.3.4, the log will contain:

Invalid user x x x 1.2.3.4 from <attacker ip>

This will cause "1.2.3.4" to be blocked instead of .

Today I finished systemd support on https://github.com/Lekensteyn/ssh-blocker. Note that it does not have regexes for the "Failed password for invalid user" since I use keys only and use messages from sshd, not PAM. It is trivial to add more matches though (regex.c)

v17 doesn't block anything (sans `firewalld`)

Confirmed on one system - loads of login failures:

• v17 no blocks
• v16 blocks right away almost

Block happened on my local home network 192.168...

Just feedback:

Butter fingered the password a few times on ssh to my Clear laptop from another machine at home and was surprised to find tallow was running after a fresh install and it blocked my other machine.

I didn't realize Clear Linux enables this by default
I didn't realize it operates on my local network

Sean

RFE: add IPv6 LLA and ULA to default whitelist

This is an RFE for adding IPv6 Link-Local Address (LLA) range and Unique local address (ULA) to the default whitelist.

So essentially add fd00::/8 and FE80::/10 prefixes.
Maybe even IPv6 localhost adrress

tallow not receiving a single message from journal - no IP is getting blocked

I have noticed on my server (Arch) that not a single malicious IP has been blocked.

I debugged the code and notices that the sd_journal_next call always returns 0. So the while loop is always skipped (and no journal message will ever get parsed). I found this discussion systemd/systemd#26577 which describes that a sd_journal_previous call directly after sd_journal_seek_tail is necessary to pull out journal messages with sd_journal_next.

So I applied this patch and tallow started working again.

diff --git a/src/tallow.c b/src/tallow.c
index 58e0fb4..2c9fc85 100644
--- a/src/tallow.c
+++ b/src/tallow.c
@@ -371,6 +371,7 @@ int main(void)
 
 	/* go to the tail and wait */
 	r = sd_journal_seek_tail(j);
+	sd_journal_previous(j);
 	sd_journal_wait(j, (uint64_t) 0);
 	dbg("sd_journal_seek_tail() returned %d\n", r);
 	while (sd_journal_next(j) != 0)
@@ -387,6 +388,7 @@ int main(void)
 		if (r == SD_JOURNAL_INVALIDATE) {
 			fprintf(stderr, "Journal was rotated, resetting\n");
 			sd_journal_seek_tail(j);
+			sd_journal_previous(j);
 		} else if (r == SD_JOURNAL_NOP) {
 			dbg("Timeout reached, waiting again\n");
 			continue;

Don't know if Clear Linux is also affected by this strange journal behavior.

tallow eating 100% of 1 CPU thread

I noticed in top that tallow was consuming 100% of a cpu thread.

A gdb attach shows its stack as:

(gdb) where
#0  0x00007fa3ae02fd4c in ?? () from /usr/lib64/libsystemd.so.0
#1  0x00007fa3ae0301be in ?? () from /usr/lib64/libsystemd.so.0
#2  0x00007fa3ae03e057 in sd_journal_get_data () from /usr/lib64/libsystemd.so.0
#3  0x0000555ba651b775 in ?? ()
#4  0x00007fa3ae1472c3 in __libc_start_main () from /usr/lib64/haswell/libc.so.6
#5  0x0000555ba651baee in ?? ()

A continue/stop then showed it as:

(gdb) where
#0  0x00007fa3ae0ca578 in ?? () from /usr/lib64/libpcre.so.1
#1  0x00007fa3ae0db08b in pcre_exec () from /usr/lib64/libpcre.so.1
#2  0x0000555ba651b801 in ?? ()
#3  0x00007fa3ae1472c3 in __libc_start_main () from /usr/lib64/haswell/libc.so.6
#4  0x0000555ba651baee in ?? ()

The tallow journal looks like:

 # journalctl -u tallow
-- Logs begin at Thu 2019-10-31 10:18:08 GMT, end at Fri 2019-11-01 17:28:18 GMT. --
Oct 31 13:57:04 skull tallow[216312]: Journal was rotated, resetting
Oct 31 18:00:40 skull systemd[1]: Stopping Tallow Service...
Oct 31 18:00:40 skull systemd[1]: tallow.service: Succeeded.
Oct 31 18:00:40 skull systemd[1]: Stopped Tallow Service.
-- Reboot --
Nov 01 09:53:49 skull systemd[1]: Started Tallow Service.
Nov 01 09:53:49 skull tallow[397]: /usr/share/tallow/sshd.json: 10 patterns
Nov 01 09:53:49 skull tallow[397]: Skipped reading /etc/tallow: No such file or directory
Nov 01 09:53:49 skull tallow[397]: Loaded 10 patterns total
Nov 01 09:53:49 skull tallow[397]: tallow 18 Started
Nov 01 10:06:34 skull tallow[397]: Journal was rotated, resetting
Nov 01 10:46:00 skull systemd[1]: Stopping Tallow Service...
Nov 01 10:46:00 skull systemd[1]: tallow.service: Succeeded.
Nov 01 10:46:00 skull systemd[1]: Stopped Tallow Service.
Nov 01 10:46:00 skull systemd[1]: Started Tallow Service.
Nov 01 10:46:00 skull tallow[134447]: /usr/share/tallow/sshd.json: 10 patterns
Nov 01 10:46:00 skull tallow[134447]: Skipped reading /etc/tallow: No such file or directory
Nov 01 10:46:00 skull tallow[134447]: Loaded 10 patterns total
Nov 01 10:46:00 skull tallow[134447]: tallow 18 Started
Nov 01 14:44:35 skull tallow[134447]: Journal was rotated, resetting

The only 'interesting' thing on this machine is that it is running a single node k8s cluster.

The machine is:

# cat /etc/os-release
NAME="Clear Linux OS"
VERSION=1
ID=clear-linux-os
ID_LIKE=clear-linux-os
VERSION_ID=31460
PRETTY_NAME="Clear Linux OS"
ANSI_COLOR="1;35"
HOME_URL="https://clearlinux.org"
SUPPORT_URL="https://clearlinux.org"
BUG_REPORT_URL="mailto:[email protected]"
PRIVACY_POLICY_URL="http://www.intel.com/privacy"

iptables rules order

iptables -t filter -A INPUT -m set --match-set tallow src -j DROP

Does not apply when rules already exists as it insert at bottom.

iptables -I INPUT 1 -m set --match-set tallow src -j DROP

Fix the issue.

Add/replace in source code iptables' commands with nft's ones

Thanks

Block occurs after apparent journal rotation

In course of debugging an issue with the clr_debug_daemon today, tallow suddenly blocked my IP address. From the evidence in the journal, I see these two log messages (IP address and hostname masked):

May 25 20:20:05 HOSTNAME tallow[8811]: Journal was rotated, resetting
May 25 20:20:05 HOSTNAME tallow[8811]: Blocked xx.xx.xx.xx

The previous ~50 journal log messages were all from clr_debug_daemon, and all have the same timestamp as above (May 25 20:20:05).

I was locked out of my ssh session a couple of days ago as well, and from the evidence in the journal, I suspect a similar sequence of events: clr_debug_daemon was spamming the journal, and then tallow blocked my IP. Except there were two "journal rotated" log messages that time...

May 23 22:42:11 HOSTNAME tallow[302]: Journal was rotated, resetting
May 23 22:42:11 HOSTNAME tallow[302]: Blocked xx.xx.xx.xx
May 23 22:42:11 HOSTNAME tallow[302]: Journal was rotated, resetting

Pretty serious signal race condition in tallow, remote root

Since we have multiple vectors being handled by the same function, it can be re-entered multiple times at an attackers discretion since they may have control over various signals that can be sent by terminal sequences, packet flags and so forth. As the signal handler here is managing heap data, it can be groomed and exploited to facilitate remote command execution, for instance as the various free() calls are made with pointers that have already been free'd by another signal that jumped in while processing.

• Do not have multiple vectors calling the same handler.

	memset(&s, 0, sizeof(struct sigaction));
	s.sa_handler = sig;
	sigaction(SIGHUP, &s, NULL);
	sigaction(SIGTERM, &s, NULL);
	sigaction(SIGINT, &s, NULL);

• Be very careful with state, there's no locking here and lots of heap interactions that are remotely controllable by attackers.

tallow/tallow.c

Lines 287 to 302 in 36946de

	static void sig(int u __attribute__ ((unused)))
	{
	fprintf(stderr, "Exiting on request.\n");
	sd_journal_close(j);
	struct tallow_struct *s = head;
	while (s) {
	struct tallow_struct *n = NULL;
	free(s->ip);
	n = s;
	s = s->next;
	free(n);
	}
	exit(EXIT_SUCCESS);
	}

OWASP has an excerpt, pretty much see every UNIX daemon from 2006 (ssh, sendmail, etc...).

https://www.owasp.org/index.php/Race_condition_in_signal_handler

add firewalld support

A feature request to add firewalld as an option for the ipt_tool and use it to configure ipsets when it is in use.

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/sec-setting_and_controlling_ip_sets_using_firewalld

https://firewalld.org/documentation/man-pages/firewalld.ipset.html