Log pushes for Kubernetes Audit Events to Loki are failing about falcosidekick

[fjogeleit]

@Issif just as verification, its working now with the latest image

[CXYALEX]

@fjogeleit I have the same problem. how did you solve it.

[epcim]

I see it either, 50% events to loki.

Very simple log on sidekick side.

Could \u003cNA\u003e (without ") be the cause. .I received <NA> without any problem on fd.name for example in other cases.. but in "

2022/09/30 08:48:58 [ERROR] : Loki - header missing (400)
2022/09/30 08:48:58 [ERROR] : Loki - header missing
2022/09/30 08:49:32 [DEBUG] : Falco's payload : {"output":"08:49:32.780838175: Warning Privilege escalation activity (user=root auser=\u003cNA\u003e command=kubectl get po -n monitoring ppid=31150 apid=23500proc.apid pid=23500 gparent=su ggparent=sudo gggparent=bash user_loginuid=2201 parent=bash pcmdline=bash ) k8s.ns=\u003cNA\u003e k8s.pod=\u003cNA\u003e container=host","priority":"Warning","rule":"Detect su or sudo","time":"2022-09-30T08:49:32.780838175Z","output_fields":{"":23500,"cluster_name":"sv10-sjc-dev-int-xxx.xx","container.id":"host","evt.time":1664527772780838175,"group":"Ves-Internal","identifier":"falco.sv10-sjc-dev.int.xxx-xx","k8s.ns.name":null,"k8s.pod.name":null,"proc.aname[2]":"su","proc.aname[3]":"sudo","proc.aname[4]":"bash","proc.cmdline":"kubectl get po -n monitoring","proc.pcmdline":"bash","proc.pid":23500,"proc.pname":"bash","proc.ppid":31150,"tenant":"ves-sre","user.loginname":"\u003cNA\u003e","user.loginuid":2201,"user.name":"root"},"source":"syscall","tags":["process","su","sudo"],"origin_host":""}
2022/09/30 08:49:32 [DEBUG] : Loki payload : {"streams":[{"labels":"{=\"23500\",procaname3=\"sudo\",procpcmdline=\"bash\",userloginuid=\"2201\",identifier=\"falco.sv10-sjc-dev.int.xxx.xx\",userloginname=\"\u003cNA\u003e\",cluster_name=\"sv10-sjc-dev-int-xxx-xx\",tenant=\"ves-sre\",containerid=\"host\",procaname2=\"su\",proccmdline=\"kubectl get po -n monitoring\",group=\"Ves-Internal\",evttime=\"1664527772780838175\",procaname4=\"bash\",procpid=\"23500\",procpname=\"bash\",procppid=\"31150\",username=\"root\",tags=\"process,su,sudo\",rule=\"Detect su or sudo\",source=\"syscall\",priority=\"Warning\",app=\"falco\",type=\"event\",severity=\"minor\",origin_host=\"10.62.53.10\"}","entries":[{"ts":"2022-09-30T08:49:32Z","line":"08:49:32.780838175: Warning Privilege escalation activity (user=root auser=\u003cNA\u003e command=kubectl get po -n monitoring ppid=31150 apid=23500proc.apid pid=23500 gparent=su ggparent=sudo gggparent=bash user_loginuid=2201 parent=bash pcmdline=bash ) k8s.ns=\u003cNA\u003e k8s.pod=\u003cNA\u003e container=host"}]}]}

2022/09/30 08:49:32 [ERROR] : Loki - header missing (400)
2022/09/30 08:49:32 [ERROR] : Loki - header missing

mind

"labels":"{=\"23500\"

and

apid=23500proc.apid 

I do have some local modifications (but aware of these .. ) going to try to skip value with missing key.. now

[epcim]

possibly related https://githubhelp.com/falcosecurity/falcosidekick/issues/77

[Issif]

The Loki output for next release 2.27 will be upgraded, hope it will help you #356

[Issif]

The 2.27.0 is out, and the helm charts are updated falco.org/blog/falcosidekick-2-27-0-ui-2-1-0

[sreejithsoman-mc]

@Issif I get this issue in falcosidekick 2.27v version as well ? Do we have any fix for it?

2023/06/12 09:00:48 [ERROR] : Loki - header missing (400): entry with timestamp 2023-06-11 20:50:23.331614 +0000 UTC ignored, reason: 'entry too far behind' for stream: {cluster="devstage", from="falcosidekick", hostname="falco-z2rwf", priority="Informational", rule="K8s Serviceaccount Created", source="k8s_audit", tags="k8s"}, total ignored: 1 out of 1

[Issif]

First time I see this issue.

Are you sure your hosts' times are correct? I see a big diff between the log line of falcosidekick (2023/06/12 09:00:48) and the timestamp of the event 2023-06-11 20:50:23.331614 +0000 UTC).