Should outbound traffic also be blocked to make a node unavailable? about akkluster

[hseeberger]

Thanks.

I think blocking either input or outout should be enough to interrupt the exchange of heartbeats.

Simply give it a try 😉

[glammers1]

Hi,

Sorry if I did something wrong, I'll try to explain my tests.

Using the commands before your last commit I had weird behaviors.

Using this command to make a node unavailable (the 172.17.0.7 for example):

iptables -A INPUT -p tcp --dport 25520 -j DROP

More than one node appears as unreachable:

HTTP/1.1 200 OK
Content-Length: 1179
Content-Type: application/json
Date: Fri, 21 Dec 2018 08:56:06 GMT
Server: akka-http/10.1.5

{
    "leader": "akka://[email protected]:25520", 
    "members": [
        {
            "node": "akka://[email protected]:25520", 
            "nodeUid": "7654717377692625970", 
            "roles": [
                "static", 
                "dc-default"
            ], 
            "status": "Up"
        }, 
        {
            "node": "akka://[email protected]:25520", 
            "nodeUid": "-3330086966741012345", 
            "roles": [
                "dynamic", 
                "dc-default"
            ], 
            "status": "Up"
        }, 
        {
            "node": "akka://[email protected]:25520", 
            "nodeUid": "3552516451594050943", 
            "roles": [
                "dynamic", 
                "dc-default"
            ], 
            "status": "Up"
        }, 
        {
            "node": "akka://[email protected]:25520", 
            "nodeUid": "421422723911626559", 
            "roles": [
                "dynamic", 
                "dc-default"
            ], 
            "status": "Up"
        }
    ], 
    "oldest": "akka://[email protected]:25520", 
    "oldestPerRole": {
        "dc-default": "akka://[email protected]:25520", 
        "dynamic": "akka://[email protected]:25520", 
        "static": "akka://[email protected]:25520"
    }, 
    "selfNode": "akka://[email protected]:25520", 
    "unreachable": [
        {
            "node": "akka://[email protected]:25520", 
            "observedBy": [
                "akka://[email protected]:25520"
            ]
        }, 
        {
            "node": "akka://[email protected]:25520", 
            "observedBy": [
                "akka://[email protected]:25520", 
                "akka://[email protected]:25520", 
                "akka://[email protected]:25520"
            ]
        }, 
        {
            "node": "akka://[email protected]:25520", 
            "observedBy": [
                "akka://[email protected]:25520"
            ]
        }
    ]
}

However using the command of your last commit without specifying the port:

iptables -A INPUT -p tcp -j DROP

Everything seems right:

HTTP/1.1 200 OK
Content-Length: 987
Content-Type: application/json
Date: Fri, 21 Dec 2018 09:00:05 GMT
Server: akka-http/10.1.5

{
    "leader": "akka://[email protected]:25520", 
    "members": [
        {
            "node": "akka://[email protected]:25520", 
            "nodeUid": "7654717377692625970", 
            "roles": [
                "static", 
                "dc-default"
            ], 
            "status": "Up"
        }, 
        {
            "node": "akka://[email protected]:25520", 
            "nodeUid": "-3330086966741012345", 
            "roles": [
                "dynamic", 
                "dc-default"
            ], 
            "status": "Up"
        }, 
        {
            "node": "akka://[email protected]:25520", 
            "nodeUid": "3552516451594050943", 
            "roles": [
                "dynamic", 
                "dc-default"
            ], 
            "status": "Up"
        }, 
        {
            "node": "akka://[email protected]:25520", 
            "nodeUid": "421422723911626559", 
            "roles": [
                "dynamic", 
                "dc-default"
            ], 
            "status": "Up"
        }
    ], 
    "oldest": "akka://[email protected]:25520", 
    "oldestPerRole": {
        "dc-default": "akka://[email protected]:25520", 
        "dynamic": "akka://[email protected]:25520", 
        "static": "akka://[email protected]:25520"
    }, 
    "selfNode": "akka://[email protected]:25520", 
    "unreachable": [
        {
            "node": "akka://[email protected]:25520", 
            "observedBy": [
                "akka://[email protected]:25520", 
                "akka://[email protected]:25520", 
                "akka://[email protected]:25520"
            ]
        }
    ]

That's why I also had to block the outbound traffic to have the same behavior (the correct one) when the port is specified, i.e.

iptables -A INPUT -p tcp --dport 25520 -j DROP
iptables -A OUTPUT -p tcp --dport 25520 -j DROP

[hseeberger]

No, you are drawing the wrong conclusions ;-)

Like I described in my talk, if threre is a network partition, there is no single truth anymore. E.g. if the cluster is split into two like in your example (using iptables with or without dport) each side observes that the other side is unreachable.

In the first example above you blocked traffic on node 182.18.0.7. Then you asked this exact node for the cluster status. Hence it reports all other nodes unreachable. Please see the "selfNode" and "obervedBy" fields.

In the second example you ask some other node (182.18.0.8) for the cluster status and hence get the view from this partition, i.e. all available except for 182.18.0.7 which is observed unreachable by all others.