eldadru / ksniff Goto Github PK
View Code? Open in Web Editor NEWKubectl plugin to ease sniffing on kubernetes pods using tcpdump and wireshark
License: Apache License 2.0
Kubectl plugin to ease sniffing on kubernetes pods using tcpdump and wireshark
License: Apache License 2.0
The privileged option ( -p ) requires that the K8S cluster has access to the public docker repository (to pull the required images)
In cases where the cluster does not have this access for security reasons, it will be useful to be able to provide the private container registry hostname that has all required images.
Currently we can not specify context by --context parameter to sniff. We can use it with native kubectl parameters like:
kubectl get pods -n foo --context some-context
I am volunteer to implement this feature.
ksniff should respect the namespace set in the user's kubectl context
k sniff customer-7f8d477894-9gl96 -c istio-proxy -p -f '((tcp) and (net $PREF_POD_IP))' -o ~/temp/foo.pcap
INFO[0000] sniffing method: privileged pod
INFO[0000] using tcpdump path at: '/Users/ceposta/.krew/store/sniff/v1.3.1/static-tcpdump'
INFO[0000] sniffing on pod: 'customer-7f8d477894-9gl96' [namespace: 'default', container: 'istio-proxy', filter: '((tcp)', interface: 'any']
INFO[0000] creating privileged pod on node: 'gke-ceposta-customer-gloo-gateway-poo-8111e7e4-fc87'
INFO[0000] pod created: &Pod{ObjectMeta:k8s_io_apimachinery_pkg_apis_meta_v1.ObjectMeta{Name:ksniff-4zwpm,GenerateName:ksniff-,Namespace:default,SelfLink:/api/v1/namespaces/default/pods/ksniff-4zwpm,UID:b128687c-415c-11ea-b17e-42010a8a0137,ResourceVersion:44002068,Generation:0,CreationTimestamp:2020-01-27 16:28:15 -0700 MST,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container ksniff-privileged,},OwnerReferences:[],Finalizers:[],ClusterName:,Initializers:nil,},Spec:PodSpec{Volumes:[{docker-sock {HostPathVolumeSource{Path:/var/run/docker.sock,Type:*File,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}} {default-token-jgf24 {nil nil nil nil nil &SecretVolumeSource{SecretName:default-token-jgf24,Items:[],DefaultMode:*420,Optional:nil,} nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil}}],Containers:[{ksniff-privileged docker [sh -c sleep 10000000] [] [] [] [] {map[] map[cpu:{{100 -3} {<nil>} 100m DecimalSI}]} [{docker-sock true /var/run/docker.sock <nil>} {default-token-jgf24 true /var/run/secrets/kubernetes.io/serviceaccount <nil>}] [] nil nil nil /dev/termination-log File Always SecurityContext{Capabilities:nil,Privileged:*true,SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,ReadOnlyRootFilesystem:nil,AllowPrivilegeEscalation:nil,RunAsGroup:nil,ProcMount:nil,} false false false}],RestartPolicy:Never,TerminationGracePeriodSeconds:*30,ActiveDeadlineSeconds:nil,DNSPolicy:ClusterFirst,NodeSelector:map[string]string{},ServiceAccountName:default,DeprecatedServiceAccount:default,NodeName:gke-ceposta-customer-gloo-gateway-poo-8111e7e4-fc87,HostNetwork:false,HostPID:false,HostIPC:false,SecurityContext:&PodSecurityContext{SELinuxOptions:nil,RunAsUser:nil,RunAsNonRoot:nil,SupplementalGroups:[],FSGroup:nil,RunAsGroup:nil,Sysctls:[],},ImagePullSecrets:[],Hostname:,Subdomain:,Affinity:nil,SchedulerName:default-scheduler,InitContainers:[],AutomountServiceAccountToken:nil,Tolerations:[{node.kubernetes.io/not-ready Exists NoExecute 0xc000366330} {node.kubernetes.io/unreachable Exists NoExecute 0xc000366350}],HostAliases:[],PriorityClassName:,Priority:*0,DNSConfig:nil,ShareProcessNamespace:nil,ReadinessGates:[],RuntimeClassName:nil,EnableServiceLinks:*true,},Status:PodStatus{Phase:Pending,Conditions:[],Message:,Reason:,HostIP:,PodIP:,StartTime:<nil>,ContainerStatuses:[],QOSClass:Burstable,InitContainerStatuses:[],NominatedNodeName:,},}
INFO[0000] waiting for pod successful startup
INFO[0002] pod: 'ksniff-4zwpm' created successfully on node: 'gke-ceposta-customer-gloo-gateway-poo-8111e7e4-fc87'
INFO[0002] output file option specified, storing output in: '/Users/ceposta/temp/foo.pcap'
INFO[0002] starting remote sniffing using privileged pod
INFO[0002] executing command: '[docker run --rm --name=ksniff-container-KCIooubl --net=container:b167089f8e6a8966551b7db278640703ddf9a92415dd68e863f2ee87305b72c1 corfr/tcpdump -i any -U -w - ((tcp)]' on container: 'ksniff-privileged', pod: 'ksniff-4zwpm', namespace: 'default'
INFO[0003] command: '[docker run --rm --name=ksniff-container-KCIooubl --net=container:b167089f8e6a8966551b7db278640703ddf9a92415dd68e863f2ee87305b72c1 corfr/tcpdump -i any -U -w - ((tcp)]' executing successfully exitCode: '1', stdErr :'tcpdump: syntax error in filter expression: syntax error
'
INFO[0003] remote sniffing using privileged pod completed
INFO[0003] starting sniffer cleanup
INFO[0003] removing privileged container: 'ksniff-container-KCIooubl'
INFO[0003] executing command: '[docker rm -f ksniff-container-KCIooubl]' on container: 'ksniff-privileged', pod: 'ksniff-4zwpm', namespace: 'default'
INFO[0004] command: '[docker rm -f ksniff-container-KCIooubl]' executing successfully exitCode: '1', stdErr :'Error: No such container: ksniff-container-KCIooubl
'
INFO[0004] privileged container: 'ksniff-container-KCIooubl' removed successfully
INFO[0004] removing pod: 'ksniff-4zwpm'
INFO[0004] removing privileged pod: 'ksniff-4zwpm'
INFO[0004] privileged pod: 'ksniff-4zwpm' removed
INFO[0004] pod: 'ksniff-4zwpm' removed successfully
INFO[0004] sniffer cleanup completed successfully ```
Tcpdump execution requires root, some containers runs as non-root user, support those use-cases as well.
In kubectl 1.12, the plugin subsystem changed completely, new model introduced to support more use-cases and more powerful plugins.
Refactor ksniff, move to golang based native client. use new kubectl capabilities to improve plugin UX
ngpe72mgtb26 ~# kubectl sniff -n mvnr-paas istio-pilot-5cc98fd795-bczx6
INFO[0000] sniffing method: upload static tcpdump
INFO[0000] using tcpdump path at: '/root/.krew/store/sniff/v1.4.2/static-tcpdump'
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'discovery'
INFO[0000] sniffing on pod: 'istio-pilot-5cc98fd795-bczx6' [namespace: 'mvnr-paas', container: 'discovery', filter: '', interface: 'any']
INFO[0000] uploading static tcpdump binary from: '/root/.krew/store/sniff/v1.4.2/static-tcpdump' to: '/tmp/static-tcpdump'
INFO[0000] uploading file: '/root/.krew/store/sniff/v1.4.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'discovery'
INFO[0000] executing command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' on container: 'discovery', pod: 'istio-pilot-5cc98fd795-bczx6', namespace: 'mvnr-paas'
INFO[0000] command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :''
INFO[0000] file found: '-rwxr-xr-x 1 root root 2696368 Jan 1 1970 /tmp/static-tcpdump
'
INFO[0000] file was already found on remote pod
INFO[0000] tcpdump uploaded successfully
INFO[0000] spawning wireshark!
INFO[0000] start sniffing on remote container
INFO[0000] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'discovery', pod: 'istio-pilot-5cc98fd795-bczx6', namespace: 'mvnr-paas'
INFO[0000] starting sniffer cleanup
INFO[0000] sniffer cleanup completed successfully
Error: exit status 1
ngpe72mgtb26 ~#
I am getting Error: exit status 1 at the end. where i can see logs to troubleshoot.
@eldadru please suggest
also where i can see the tcpdump on worker node
I just did brew install wireshark
on my macOS, it doesn't seem to be installing a Wireshark binary, so the plugin is not finding it:
$ brew link wireshark -v
Linking /Users/ahmetb/.homebrew/Cellar/wireshark/3.0.3...
ln -s ../Cellar/wireshark/3.0.3/bin/capinfos capinfos
ln -s ../Cellar/wireshark/3.0.3/bin/captype captype
ln -s ../Cellar/wireshark/3.0.3/bin/dumpcap dumpcap
ln -s ../Cellar/wireshark/3.0.3/bin/editcap editcap
ln -s ../Cellar/wireshark/3.0.3/bin/idl2wrs idl2wrs
ln -s ../Cellar/wireshark/3.0.3/bin/mergecap mergecap
ln -s ../Cellar/wireshark/3.0.3/bin/mmdbresolve mmdbresolve
ln -s ../Cellar/wireshark/3.0.3/bin/randpkt randpkt
ln -s ../Cellar/wireshark/3.0.3/bin/rawshark rawshark
ln -s ../Cellar/wireshark/3.0.3/bin/reordercap reordercap
ln -s ../Cellar/wireshark/3.0.3/bin/sharkd sharkd
ln -s ../Cellar/wireshark/3.0.3/bin/text2pcap text2pcap
ln -s ../Cellar/wireshark/3.0.3/bin/tshark tshark
ln -s ../Cellar/wireshark/3.0.3/include/wireshark wireshark
ln -s ../../Cellar/wireshark/3.0.3/share/doc/wireshark wireshark
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/androiddump.1 androiddump.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/capinfos.1 capinfos.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/captype.1 captype.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/ciscodump.1 ciscodump.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/dftest.1 dftest.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/dumpcap.1 dumpcap.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/editcap.1 editcap.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/mergecap.1 mergecap.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/mmdbresolve.1 mmdbresolve.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/randpkt.1 randpkt.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/randpktdump.1 randpktdump.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/rawshark.1 rawshark.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/reordercap.1 reordercap.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/sshdump.1 sshdump.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/text2pcap.1 text2pcap.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/tshark.1 tshark.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/udpdump.1 udpdump.1
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man1/wireshark.1 wireshark.1
mkdir -p /Users/ahmetb/.homebrew/share/man/man4
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man4/extcap.4 extcap.4
ln -s ../../../Cellar/wireshark/3.0.3/share/man/man4/wireshark-filter.4 wireshark-filter.4
ln -s ../Cellar/wireshark/3.0.3/share/wireshark wireshark
ln -s ../Cellar/wireshark/3.0.3/lib/libwireshark.12.0.3.dylib libwireshark.12.0.3.dylib
ln -s ../Cellar/wireshark/3.0.3/lib/libwireshark.12.dylib libwireshark.12.dylib
ln -s ../Cellar/wireshark/3.0.3/lib/libwireshark.dylib libwireshark.dylib
ln -s ../Cellar/wireshark/3.0.3/lib/libwiretap.9.0.3.dylib libwiretap.9.0.3.dylib
ln -s ../Cellar/wireshark/3.0.3/lib/libwiretap.9.dylib libwiretap.9.dylib
ln -s ../Cellar/wireshark/3.0.3/lib/libwiretap.dylib libwiretap.dylib
ln -s ../Cellar/wireshark/3.0.3/lib/libwscodecs.2.0.0.dylib libwscodecs.2.0.0.dylib
ln -s ../Cellar/wireshark/3.0.3/lib/libwscodecs.2.dylib libwscodecs.2.dylib
ln -s ../Cellar/wireshark/3.0.3/lib/libwscodecs.dylib libwscodecs.dylib
ln -s ../Cellar/wireshark/3.0.3/lib/libwsutil.10.0.0.dylib libwsutil.10.0.0.dylib
ln -s ../Cellar/wireshark/3.0.3/lib/libwsutil.10.dylib libwsutil.10.dylib
ln -s ../Cellar/wireshark/3.0.3/lib/libwsutil.dylib libwsutil.dylib
ln -s ../../Cellar/wireshark/3.0.3/lib/pkgconfig/wireshark.pc wireshark.pc
ln -s ../Cellar/wireshark/3.0.3/lib/wireshark wireshark
50 symlinks created
I think it would be good to have some instructions on how to get it up and running on macOS.
CI should run tests. This could be added explicitly in the yaml or as a new dependency in the Makefile
In some test scripts I sometimes use ksniff to sniff a container and save to output to a capture file by use the "-o" flag to ksniff, for example:
kubectl sniff -c -o
I also tried to use the "-p" flag for priviledge mode that starts a new pod.
But when I try to stop/quit the ksniff, either by "ctrl-c" or let my script kill the ksniff process, then the "ksniff"-pod is kept aswell as the "static-tcpdump" process in the container (if I launch without the "-p" flag).
If I do not use the "-o" flag, and insteasd let it launch Wireshark, then everything seem to be safely stoped and clean up after quit the Wireshark application. But if I use the "-o" option and send the stream to a file, I have not yeat found a nice way to stop/quit the ksniff application.
After running kubectl sniff -p <POD> -c <CONTAINER_NAME> -n <NAMESPACE>
.
I'm using AKS Kubernetes v1.17.11.
ksniff version:
sniff v1.5.0
INFO[0000] waiting for pod successful startup
INFO[0004] pod: 'ksniff-5pbv6' created successfully on node: 'aks-d8sv3-38575711-vmss000001'
INFO[0004] spawning wireshark!
INFO[0004] starting remote sniffing using privileged pod
INFO[0004] executing command: '[docker --host unix:///host/var/run/docker.sock run --rm --name=ksniff-container-fLuezHME --net=container:602167f3ac7de5f5156763f8ad765ea15e7b9ed1cfeb242392fe0330c6762aaa maintained/tcpdump -i any -U -w - ]' on container: 'ksniff-privileged', pod: 'ksniff-5pbv6', namespace: 'global'
INFO[0005] command: '[docker --host unix:///host/var/run/docker.sock run --rm --name=ksniff-container-fLuezHME --net=container:602167f3ac7de5f5156763f8ad765ea15e7b9ed1cfeb242392fe0330c6762aaa maintained/tcpdump -i any -U -w - ]' executing successfully exitCode: '125', stdErr :'docker: Cannot connect to the Docker daemon at unix:///host/var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.
INFO[0005] remote sniffing using privileged pod completed
β― go test ./...
# ksniff/pkg/cmd [ksniff/pkg/cmd.test]
pkg/cmd/sniff_test.go:14:11: undefined: NewSniffOptions
pkg/cmd/sniff_test.go:28:11: undefined: NewSniffOptions
pkg/cmd/sniff_test.go:42:11: undefined: NewSniffOptions
When i run kubectl sniff i get this, no documentation to show how to fix this.
make darwin
works fine, but make static-tcpdump
gives the following output:
cd tcpdump-4.9.2 && CFLAGS=-static ./configure --without-crypto && make
checking build system type... x86_64-apple-darwin18.2.0
checking host system type... x86_64-apple-darwin18.2.0
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/Users/jordan/git/ksniff/tcpdump-4.9.2':
configure: error: C compiler cannot create executables
See `config.log' for more details
make: *** [static-tcpdump] Error 77
The config.log output: config.log
Also note that running the plugin via Krew results in another error: kubernetes-sigs/krew-index#48
The error I see when running via krew seems related to the other open issue here: #21
When I run k plugin sniff my-product-api-c97484d9b-bn6d6 -n my-product-api
I get
[+] Sniffing on pod: my-product-api-c97484d9b-bn6d6 container: namespace:
[+] Verifying pod status
Error from server (NotFound): pods "my-product-api-c97484d9b-bn6d6" not found
[-] Pod is not existing or on different namespace
error: exit status 1
But, when I run k describe pod my-product-api-c97484d9b-bn6d6 -n my-product-api
then I am getting the expected result.
Client Version: v1.10.4
Server Version: v1.11.1
Hi,
I'm trying to use ksniff with Istio 1.7.3 with config option values.global.proxy.holdApplicationUntilProxyStart
enabled, which causes the sidecar injector to inject the sidecar at the start of the podβs container list.
I'm now getting the following error.
INFO[0010] executing command: '[docker --host unix:///host/var/run/docker.sock run --rm --name=ksniff-container-dKHBrsxq --net=container:ab652c801a2616a5f47603054cdfc579eb4a403f4597c4b08f146f58d0c71ee9 maintained/tcpdump -i any -U -w - ]' on container: 'istio-proxy', pod: 'ksniff-zzk4m', namespace: 'global'
INFO[0011] command: '[docker --host unix:///host/var/run/docker.sock run --rm --name=ksniff-container-dKHBrsxq --net=container:ab652c801a2616a5f47603054cdfc579eb4a403f4597c4b08f146f58d0c71ee9 maintained/tcpdump -i any -U -w - ]' executing successfully exitCode: '126', stdErr :''
I try with this temporary workaround and everything is working fine (sorry but I'm not a developer).
https://github.com/ffais/ksniff/blob/62b535a9cc068d25eb6d31bde8ad20308490b42a/pkg/service/sniffer/privileged_pod_sniffer_service.go
After make install
, kubectl stops working and gives below error
panic: shorthand redefinition
goroutine 1 [running]:
k8s.io/kubernetes/vendor/github.com/spf13/pflag.(*FlagSet).AddFlag(0xc4208d4b40, 0xc4208d2d20)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/pflag/flag.go:747 +0x584
k8s.io/kubernetes/vendor/github.com/spf13/pflag.(*FlagSet).VarPF(0xc4208d4b40, 0x2d959a0, 0xc4203917d0, 0xc4203916a0, 0xa, 0xc420391698, 0x1, 0xc42019bc80, 0x71, 0x335c4b0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/pflag/flag.go:706 +0x120
k8s.io/kubernetes/vendor/github.com/spf13/pflag.(*FlagSet).VarP(0xc4208d4b40, 0x2d959a0, 0xc4203917d0, 0xc4203916a0, 0xa, 0xc420391698, 0x1, 0xc42019bc80, 0x71)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/pflag/flag.go:712 +0x8e
k8s.io/kubernetes/vendor/github.com/spf13/pflag.(*FlagSet).StringVarP(0xc4208d4b40, 0xc4203917d0, 0xc4203916a0, 0xa, 0xc420391698, 0x1, 0x0, 0x0, 0xc42019bc80, 0x71)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/pflag/string.go:42 +0xab
k8s.io/kubernetes/vendor/github.com/spf13/pflag.(*FlagSet).StringP(0xc4208d4b40, 0xc4203916a0, 0xa, 0xc420391698, 0x1, 0x0, 0x0, 0xc42019bc80, 0x71, 0xc4203917b0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/pflag/string.go:67 +0xb0
k8s.io/kubernetes/pkg/kubectl/cmd.NewCmdForPlugin(0x2daf040, 0xc42024e960, 0xc42019f680, 0x2d804e0, 0x2f1aed8, 0x2d84e20, 0xc42008a000, 0x2d84e60, 0xc42008a008, 0x2d84e60, ...)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/plugin.go:118 +0x3db
k8s.io/kubernetes/pkg/kubectl/cmd.NewCmdPlugin(0x2daf040, 0xc42024e960, 0x2d84e20, 0xc42008a000, 0x2d84e60, 0xc42008a008, 0x2d84e60, 0xc42008a010, 0xc420741e60)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/plugin.go:64 +0x305
k8s.io/kubernetes/pkg/kubectl/cmd.NewKubectlCommand(0x2daf040, 0xc42024e960, 0x2d84e20, 0xc42008a000, 0x2d84e60, 0xc42008a008, 0x2d84e60, 0xc42008a010, 0x202426e)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/kubectl/cmd/cmd.go:379 +0x1810
k8s.io/kubernetes/cmd/kubectl/app.Run(0x0, 0x0)
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/app/kubectl.go:38 +0xc7
main.main()
/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:26 +0x22
kubectl version:
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd", GitTreeState:"clean", BuildDate:"2017-11-08T18:39:33Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"darwin/amd64"}
Ksniff can be installed as part of the krew kubectl plugin manager.
krew use krew-index (https://github.com/GoogleContainerTools/krew-index) a repository that used as krew plugin store, each plugin has it's own yaml file with the plugin description that includes its version, its binary package location and its hash.
In order to upgrade a plugin in krew one has to update the plugin yaml to have the latest information and open a pull request for the krew-index project.
I'd like this process to be automatic on each time we merge ksniff to master and build a plugin release
$ kubectl-sniff -v at-atlantis-0 -c atlantis
INFO[0000] running in verbose mode
Error: lstat kubectl-sniff: no such file or directory
Using kubectl sniff
works as expected.
ksniff fails when running tar if it does not exist. This is the verbose logs when running on an image that has a statically compiled binary in a Docker 'scratch' image.
INFO[0000] running in verbose mode
DEBU[0000] searching for tcpdump binary using lookup list: '[ static-tcpdump /usr/local/bin/static-tcpdump /home/oyiu/.kube/plugin/sniff/static-tcpdump]'
DEBU[0000] tcpdump binary was not found at: ''
DEBU[0000] tcpdump binary was not found at: 'static-tcpdump'
DEBU[0000] tcpdump binary found at: '/usr/local/bin/static-tcpdump'
INFO[0000] using tcpdump path at: '/usr/local/bin/static-tcpdump'
DEBU[0000] pod 'goldpinger-8kwlm' status: 'Running'
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'goldpinger'
INFO[0000] sniffing on pod: 'goldpinger-8kwlm' [namespace: 'oyiu-test', container: 'goldpinger', filter: '', interface: 'eth0']
INFO[0000] checking for static tcpdump binary on: '/tmp/static-tcpdump'
DEBU[0000] checked for tcpdump on remote pod: exit-code: '126', stdout: 'OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"/bin/sh\": stat /bin/sh: no such file or directory": unknown
', stderr: ''
INFO[0000] couldn't find static tcpdump binary on: '/tmp/static-tcpdump', starting to upload
DEBU[0000] uploading file from: '/usr/local/bin/static-tcpdump' to '/tmp/static-tcpdump'
DEBU[0000] read '/usr/local/bin/static-tcpdump' to memory, file size: '2642872'
DEBU[0000] formatted '/usr/local/bin/static-tcpdump' as tar, tar size: '2644480'
DEBU[0000] executing tar: '[tar -xf - -C /tmp]'
DEBU[0001] done uploading file, exitCode: '126', stdOut: 'OCI runtime exec failed: exec failed: container_linux.go:348: starting container process caused "exec: \"tar\": executable file not found in $PATH": unknown
', stdErr: ''
INFO[0001] spawning wireshark!
ERROR: logging before flag.Parse: E0111 17:03:52.436667 10386 v2.go:105] write tcp 10.49.16.74:52046->10.49.252.10:443: use of closed network connection
DEBU[0001] executing tcpdump on remote pod
DEBU[0001] tcpdump executed, exitCode: '126', stdErr: '&{}' error="<nil>"
^C
Error: failed to create pod within timeout (1m0s)
Would be nice if we could change it because my pod takes 70s to get ready^^
Can it be updated please?
Running on EKS v 1.13 and getting this when trying to capture pod traffic:
INFO[0000] running in verbose mode INFO[0000] sniffing method: upload static tcpdump DEBU[0000] searching for tcpdump binary using lookup list: '[ /Users/mgrimshaw/.krew/store/sniff/v1.4.1/static-tcpdump /usr/local/bin/static-tcpdump /Users/mgrimshaw/.kube/plugin/sniff/static-tcpdump]' DEBU[0000] tcpdump binary was not found at: '' DEBU[0000] tcpdump binary found at: '/Users/mgrimshaw/.krew/store/sniff/v1.4.1/static-tcpdump' INFO[0000] using tcpdump path at: '/Users/mgrimshaw/.krew/store/sniff/v1.4.1/static-tcpdump' DEBU[0000] pod 'XXX-5b9b475fbf-etcs9' status: 'Running' INFO[0000] no container specified, taking first container we found in pod. INFO[0000] selected container: 'XXX' INFO[0000] sniffing on pod: 'XXX-5b9b475fbf-etcs9' [namespace: 'default', container: 'XXX', filter: '', interface: 'any'] INFO[0000] uploading static tcpdump binary from: '/Users/mgrimshaw/.krew/store/sniff/v1.4.1/static-tcpdump' to: '/tmp/static-tcpdump' INFO[0000] uploading file: '/Users/mgrimshaw/.krew/store/sniff/v1.4.1/static-tcpdump' to '/tmp/static-tcpdump' on container: 'XXX' INFO[0000] executing command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' on container: 'XXX', pod: 'XXX-5b9b475fbf-etcs9', namespace: 'default' INFO[0001] command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :'' INFO[0001] file found: '-rwxr-xr-x 1 XXX XXX 2696368 Jan 1 1970 /tmp/static-tcpdump ' INFO[0001] file was already found on remote pod INFO[0001] tcpdump uploaded successfully INFO[0001] spawning wireshark! INFO[0001] start sniffing on remote container INFO[0001] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'XXX', pod: 'XXX-5b9b475fbf-etcs9', namespace: 'default' INFO[0002] command: '[/tmp/static-tcpdump -i any -U -w - ]' executing successfully exitCode: '1', stdErr :'static-tcpdump: any: You don't have permission to capture on that device (socket: Operation not permitted) ' ERRO[0002] failed to start remote sniffing, stopping wireshark error="executing sniffer failed, exit code: '1'" INFO[0002] starting sniffer cleanup INFO[0002] sniffer cleanup completed successfully Error: signal: killed
Yet when I exec into the pod I see that the file /tmp/static-tcpdump
is owned correctly.
Am I missing something here?
Thanks
Hello,
Is there an option to use the tcpdump
already installed in the target container in cases where the container file system is mounted read-only?
Thanks!
Ralf
The plugin support changed in kubectl 1.12, support it:
https://kubernetes.io/docs/tasks/extend-kubectl/kubectl-plugins/
π Hello, maintainer of the kubectl plugin manager krew here.
Thank you for your commitment to open source by making this plugin available via krew
!
Krew wants to give credit where credit is due by installing the proper license file for the plugins it distributes. However, your plugin was found to not contain any license file. We wanted to remind you that if you're using a license such as Apache 2.0, you should be bundling your LICENSE file with your pluginβs distributions.
What do you have to do?
files:
section to copy the file to the installation directory. Have a look at this PR for an example: https://github.com/kubernetes-sigs/krew-index/pull/314/filesIf you need further assistance, don't hesitate to ask for help.
After running kubectl sniff <POD> -n <NAMESPACE>
I get the following output with error:
INFO[0000] sniffing method: upload static tcpdump
INFO[0000] using tcpdump path at: '/Users/jason/.krew/store/sniff/v1.4.2/static-tcpdump'
INFO[0001] no container specified, taking first container we found in pod.
INFO[0001] selected container: 'container1'
INFO[0001] sniffing on pod: '<POD>' [namespace: '<NAMESPACE>', container: 'container1', filter: '', interface: 'any']
INFO[0001] uploading static tcpdump binary from: '/Users/jason/.krew/store/sniff/v1.4.2/static-tcpdump' to: '/tmp/static-tcpdump'
INFO[0001] uploading file: '/Users/jason/.krew/store/sniff/v1.4.2/static-tcpdump' to '/tmp/static-tcpdump' on container: 'container1'
INFO[0001] executing command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' on container: 'container1', pod: '<POD>', namespace: '<NAMESPACE>'
INFO[0002] command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' executing successfully exitCode: '2', stdErr :'ls: cannot access /tmp/static-tcpdump: No such file or directory
'
INFO[0002] file not found on: '/tmp/static-tcpdump', starting to upload
INFO[0003] tcpdump uploaded successfully
INFO[0003] spawning wireshark!
E0717 12:37:48.123789 28353 v2.go:105] write tcp 192.168.1.9:53321->23.20.184.138:443: write: broken pipe
INFO[0003] start sniffing on remote container
INFO[0003] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'container1', pod: '<POD>', namespace: '<NAMESPACE>'
INFO[0004] command: '[/tmp/static-tcpdump -i any -U -w - ]' executing successfully exitCode: '126', stdErr :''
ERRO[0004] failed to start remote sniffing, stopping wireshark error="executing sniffer failed, exit code: '126'"
INFO[0004] starting sniffer cleanup
INFO[0004] sniffer cleanup completed successfully
Error: signal: killed
Any thoughts on what might be causing this?
I am wondering if you've done any tuning on figuring out what the minimum required RBAC permissions for a user would been to be to get a successful sniff.
Would you consider an option to allow for specifying an alternative to the exec'ing the wireshark GUI?
For example, I think it would be interesting to run sniff and connect the pcap output to tshark or https://github.com/gcla/termshark
I'd be happy to work on a PR for this if it is something you would be OK including.
Perhaps via a flag like -w/--wireshark-bin=/path
I think it'd be really cool if one could pass a selector and have ksniff run tcpdump on all matching pods
To get packets sniffed on the remote pod displayed on our local wireshark as soon as possible, we should disable tcpdump packet buffering default feature.
"Use the -U flag to cause packets to be written as soon as they are received."
Hi,
That is a great project.
Please allow the script to accept namespace as a parameter
Hello there!
We are using OpenShift with default security in place so we have no option not to use ksniff privileged mode. But also we're using CRI-O instead of Docker runtime. Currently ksniff supports only Docker runtime for privileged mode. Do you have any plans to support CRI-O runtime for privileged mode?
PS: Thanks for a great tool!
Getting this from wireshark after trying to use ksniff for the first time "kubectl sniff ":
Couldn't run /usr/sbin/dumpcap in child process: Permission denied
Are you a member of the 'wireshark' group? Try running
'usermod -a -G wireshark your_username' as root.
Interfaces on the wireshark screen appear to be "null".
When executing "kubectl exec -it id" I am getting root.
Hello,
is it possible to specify a docker repository from where to download the privileged container? We don't have access to external repositories from the cluster. Instead we need to pull all images through an internal mirror.
Thanks!
Ralf
It would be great to have a label to the ksniff pod so I can do filtering and whitelisting of resources in my k8s cluster.
For example, adding a label app=ksniff
would be great so I can then identify the resource using
labelSelector:
matchExpressions:
- {key: app, operator: NotIn, values: ["ksniff"]}
Current ksniff pod
$ (master) kubectl describe pod ksniff-xxxx
Name: ksniff-xxxx
Namespace: connectivity
Priority: 0
Node: ip-xxxxxx.xxxxxxcompute.internal/xxxxxxx
Start Time: Wed, 12 Feb 2020 09:39:22 +0100
Labels: <none>
Annotations: cni.projectcalico.org/podIP: xxxx/32
Status: Running
IP: 100.108.216.43
Containers:
ksniff-privileged:
Each new feature merged into master should advance the plugin semver.
To guarantee that, add a check into the CI pipeline that validates this commit has newer version that the master has.
Installed ksniff in Ubuntu on WSL (Windows subsystem for Linux) but it does not run, or give any useful information:
$ k sniff shell-7f7f4fb5cb-bpc7p
cannot allocate memory
$ k sniff shell-7f7f4fb5cb-bpc7p -o myfile
cannot allocate memory
It's not an issue at all, but I wonder what are the options rather than change the image if I need to sniff the traffic on distroless image
When the pod is created, it mounts the whole /
host path under the /host
path of the pod.
ksniff/kube/kubernetes_api_service.go
Line 149 in 798b1c7
However, when the Service Account Admission Manager came into the play, it will mount the token under /var/run/secrets/kubernetes.io/serviceaccount
, and both the /var/run
and /host/var/run
is a symlink to /run
, the /host/var/run/docker.sock
is gone and docker won't be able to connect to the docker daemon.
May I ask what's the purpose of mounting the whole /
directory instead of just the .sock
file? The current approach seems a bit dangerous and might have unexpected side effects.
Hi,
I've had ksniff working with a minikube deployment with no issues. I've switched to microk8s and am now getting the following error.
ERRO[0000] failed to create privileged pod on node: 'mudged-laptop' error="container runtime on node: 'mudged-laptop' isn't docker"
From what I can see this is down to microk8s using containerd as the container runtime.
Firstly am I right in my assumption? And are there any plans to support microk8s/containerd in the future?
In privileged mode we create a new privileged container on the remote kubernetes node that attaches to the target container network namespace to perform the network capture.
ksniff doesn't remove that container when it finish, solve that
I have configured my kubectl to control remote cluster, so this operation is on my laptop, I am trying to use sniff to dump packet traffic go through K8S Pod.
π kc -n epc1 get pods
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 2h
hss-0 1/1 Running 0 2h
mme-0 1/1 Running 0 2h
π kc sniff mme-0 -n epc1
INFO[0000] using tcpdump path at: '/Users/aweimeow/.krew/store/sniff/afb1a2e2cd093f1c8f8fff511f48cc5a290d2c6ecd18d9f51f9c66500710297b/static-tcpdump'
INFO[0000] no container specified, taking first container we found in pod.
INFO[0000] selected container: 'mme'
INFO[0000] sniffing on pod: 'mme-0' [namespace: 'epc1', container: 'mme', filter: '']
INFO[0000] checking for static tcpdump binary on: '/tmp/static-tcpdump'
INFO[0000] couldn't find static tcpdump binary on: '/tmp/static-tcpdump', starting to upload
INFO[0000] tcpdump uploaded successfully
INFO[0000] spawning wireshark!
And wireshark shows the following message:
If you need more information for helping debug, please let me know :)
To improve project reliability, we would like to have e2e testing:
$ uname -a
Darwin bhutwala-mac 18.7.0 Darwin Kernel Version 18.7.0: Thu Jan 23 06:52:12 PST 2020; root:xnu-4903.278.25~1/RELEASE_X86_64 x86_64
$
$ which -a wireshark
wireshark: aliased to open /Applications/Wireshark.app
INFO[0000] sniffing method: upload static tcpdump
INFO[0000] using tcpdump path at: '/Users/bhutwala/.krew/store/sniff/v1.4.1/static-tcpdump'
INFO[0000] sniffing on pod: 'kube-apiserver-kind-control-plane' [namespace: 'kube-system', container: 'kube-apiserver', filter: '', interface: 'any']
INFO[0000] uploading static tcpdump binary from: '/Users/bhutwala/.krew/store/sniff/v1.4.1/static-tcpdump' to: '/tmp/static-tcpdump'
INFO[0000] uploading file: '/Users/bhutwala/.krew/store/sniff/v1.4.1/static-tcpdump' to '/tmp/static-tcpdump' on container: 'kube-apiserver'
INFO[0000] executing command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' on container: 'kube-apiserver', pod: 'kube-apiserver-kind-control-plane', namespace: 'kube-system'
INFO[0000] command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' executing successfully exitCode: '0', stdErr :''
INFO[0000] file found: '-rwxr-xr-x 1 root root 2696368 Jan 1 1970 /tmp/static-tcpdump
'
INFO[0000] file was already found on remote pod
INFO[0000] tcpdump uploaded successfully
INFO[0000] spawning wireshark!
INFO[0000] starting sniffer cleanup
INFO[0000] sniffer cleanup completed successfully
Error: exec: "wireshark": executable file not found in $PATH
When I try to install I'm getting the following error:
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking rpc/rpc.h usability... yes
checking rpc/rpc.h presence... yes
checking for rpc/rpc.h... yes
checking rpc/rpcent.h usability... no
checking rpc/rpcent.h presence... no
checking for rpc/rpcent.h... no
checking netdnet/dnetdb.h usability... no
checking netdnet/dnetdb.h presence... no
checking for netdnet/dnetdb.h... no
checking for net/pfvar.h... no
checking for netinet/if_ether.h... yes
checking whether time.h and sys/time.h may both be included... yes
checking smi.h usability... no
checking smi.h presence... no
checking for smi.h... no
checking whether to enable the possibly-buggy SMB printer... yes
configure: WARNING: The SMB printer may have exploitable buffer overflows!!!
checking whether to drop root privileges by default... no
checking whether to chroot... no
checking for cap_enter... no
checking for cap_rights_limit... no
checking for cap_ioctls_limit... no
checking for openat... yes
checking whether to sandbox using capsicum... no
checking for library containing gethostbyname... none required
checking for library containing socket... none required
checking for library containing putmsg... none required
checking whether the operating system supports IPv6... yes
checking ipv6 stack type... linux-glibc
checking for dnet_htoa declaration in netdnet/dnetdb.h... no
checking for vfprintf... yes
checking for strlcat... no
checking for strlcpy... no
checking for strdup... yes
checking for strsep... yes
checking for getopt_long... yes
checking for fork... yes
checking for vfork... yes
checking for strftime... yes
checking for setlinebuf... yes
checking for alarm... yes
checking for vsnprintf... yes
checking for snprintf... yes
checking return type of signal handlers... void
checking for sigaction... yes
checking for library containing dnet_htoa... no
checking for main in -lrpc... no
checking for library containing getrpcbynumber... none required
checking for local pcap library... not found
checking for pcap-config... no
checking for main in -lpcap... no
configure: error: see the INSTALL doc for more info
Makefile:17: recipe for target 'static-tcpdump' failed
make: *** [static-tcpdump] Error 1
I get this error:
(β stage:kube-system)β kube-system git:(master) k sniff coredns-77594c5b55-ck7kb -v
INFO[0000] running in verbose mode
Error: No Auth Provider found for name "oidc"
(β stage:kube-system)β kube-system git:(master) k version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.1", GitCommit:"eec55b9ba98609a46fee712359c7b5b365bdd920", GitTreeState:"clean", BuildDate:"2018-12-13T10:39:04Z", GoVersion:"go1.11.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.5", GitCommit:"753b2dbc622f5cc417845f0ff8a77f539a4213ea", GitTreeState:"clean", BuildDate:"2018-11-26T14:31:35Z", GoVersion:"go1.10.3", Compiler:"gc", Platform:"linux/amd64"}
Any idea was happening here?
See also #9 - I just hardcoded the namespace value at the top of ksniff.sh
.
Now, when I run ./ksniff.sh my-product-8d7599548-vk24f
I get the following:
namespace is: my-product
[+] Sniffing on pod: my-product-8d7599548-vk24f container: namespace: my-product
[+] Verifying pod status
NAME READY STATUS RESTARTS AGE
my-product-8d7599548-vk24f 2/2 Running 1 3h
[+] checking if tcpdump already exist
Defaulting container name to my-product.
Use 'kubectl describe pod/my-product-8d7599548-vk24f -n my-product' to see all of the containers in this pod.
ls: cannot access /static-tcpdump: No such file or directory
command terminated with exit code 2
[+] couldn't find static tcpdump binary, uploading static tcpdump to container
Defaulting container name to my-product.
error: lstat static-tcpdump: no such file or directory
Defaulting container name to my-product.
Use 'kubectl describe pod/my-product-8d7599548-vk24f -n my-product' to see all of the containers in this pod.
chmod: cannot access '/static-tcpdump': No such file or directory
command terminated with exit code 1
[+] Starting remote sniffing!
./ksniff.sh: line 48: wireshark: command not found
Defaulting container name to my-product.
Use 'kubectl describe pod/my-product-8d7599548-vk24f -n my-product' to see all of the containers in this pod.
Running sniff in a privileged, or unprivileged pod results in the wireshark client posting an error dialog box, "Couldn't run /usr/bin/dumpcap in child process: Permission denied". Examining the target pod I find this executable is not present in both the priv and unpriv cases.
I think modified the Dockerfile to pull in wireshark-common and now get 'operation not permitted'.
These containers are running in a Kubernetes 1.11.1 cluster, using on AWS EC2 nodes provisioned using kops 1.11.1.
Any thoughts ?
$ k krew list
PLUGIN VERSION
krew v0.4.0
modify-secret v0.0.36
sniff v1.5.0
$ k sniff --context kind-istio-mc-demo-1 -n istio-system -c istio-proxy istio-ingressgateway-8d9cc9b86-hxgxf
INFO[0000] using tcpdump path at: '/Users/pnovotnak/.krew/store/sniff/v1.5.0/static-tcpdump'
INFO[0000] sniffing method: upload static tcpdump
INFO[0000] sniffing on pod: 'istio-ingressgateway-8d9cc9b86-hxgxf' [namespace: 'istio-system', container: 'istio-proxy', filter: '', interface: 'any']
INFO[0000] uploading static tcpdump binary from: '/Users/pnovotnak/.krew/store/sniff/v1.5.0/static-tcpdump' to: '/tmp/static-tcpdump'
INFO[0000] uploading file: '/Users/pnovotnak/.krew/store/sniff/v1.5.0/static-tcpdump' to '/tmp/static-tcpdump' on container: 'istio-proxy'
INFO[0000] executing command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' on container: 'istio-proxy', pod: 'istio-ingressgateway-8d9cc9b86-hxgxf', namespace: 'istio-system'
INFO[0000] command: '[/bin/sh -c ls -alt /tmp/static-tcpdump]' executing successfully exitCode: '2', stdErr :'ls: cannot access '/tmp/static-tcpdump': No such file or directory
'
INFO[0000] file not found on: '/tmp/static-tcpdump', starting to upload
INFO[0000] tcpdump uploaded successfully
INFO[0000] spawning wireshark!
INFO[0000] start sniffing on remote container
INFO[0000] executing command: '[/tmp/static-tcpdump -i any -U -w - ]' on container: 'istio-proxy', pod: 'istio-ingressgateway-8d9cc9b86-hxgxf', namespace: 'istio-system'
ERRO[0000] failed executing command: '[/tmp/static-tcpdump -i any -U -w - ]', exitCode: '0', stdErr: '' error="Internal error occurred: error executing command in container: failed to exec in container: failed to start exec \"1fe509373caceebfbaca3ddc7bbc745d7b0ed348c6afe50ad76b07f3950c6d0c\": OCI runtime exec failed: exec failed: container_linux.go:370: starting container process caused: exec: \"/tmp/static-tcpdump\": stat /tmp/static-tcpdump: no such file or directory: unknown"
ERRO[0000] failed to start remote sniffing, stopping wireshark error="executing sniffer failed, exit code: '0'"
INFO[0000] starting sniffer cleanup
INFO[0000] sniffer cleanup completed successfully
Error: signal: killed
$ k config get-contexts | grep kind
kind-istio-mc-demo-1 kind-istio-mc-demo-1 kind-istio-mc-demo-1 default
* kind-istio-mc-demo-2 kind-istio-mc-demo-2 kind-istio-mc-demo-2 default
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.