What happened:
Prerequisites
make a ubuntu-test image from ubuntu:21.04, and install with telnet, net-tools, dnsutils, mosquitto, mosquitto-clients, inetutils-ping etc.
run it to a container from yaml:
apiVersion: v1
kind: Pod
metadata:
name: edge-ubuntu-test
spec:
nodeSelector:
node-role.kubernetes.io/edge: ""
containers:
- name: edge-ubuntu-test
image: ubuntu-test:21.04
# Just spin & wait forever
command: [ "/bin/bash", "-c", "--" ]
args: [ "while true; do sleep 30; done;" ]
Success case
在边缘端pod层面创建一个mosquitto容器，供边缘端edge-ubuntu-test访问。
Create a mosquitto container at the edge pod for access by edge-ubuntu-test.
service -> endpoints
- configmap-mosquitto.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: mosquitto-config
labels:
app: chirpstack-mosquitto
data:
mosquitto.conf: |
persistence true
persistence_location /mosquitto/data/
# per_listener_settings false
log_dest stdout
# listener 1886
listener 31883 0.0.0.0
protocol mqtt
listener 32901 0.0.0.0
protocol websockets
# Defaults to false, unless there are no listeners defined in the configuration
# file, in which case it is set to true, but connections are only allowed from
# the local machine.
allow_anonymous true
# password_file /.config/mosquitto/auth/password_file.txt
# cafile: /mosquitto/config/certs/ca.crt
# certfile: /mosquitto/config/certs/server.crt
# keyfile: /mosquitto/config/certs/server.key
require_certificate false
use_identity_as_username false
- deployment-chirpstack-mosquitto.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: chirpstack-mosquitto
spec:
replicas: 1
selector:
matchLabels:
app: chirpstack-mosquitto
template:
metadata:
labels:
app: chirpstack-mosquitto
spec:
nodeSelector:
node-role.kubernetes.io/edge: ""
containers:
- name: chirpstack-mosquitto
image: eclipse-mosquitto:2.0.10
ports:
- containerPort: 31883
name: mqtt
hostPort: 31883
volumeMounts:
- name: mosquitto-data
mountPath: /mosquitto/data
- name: mosquitto-log
mountPath: /mosquitto/log
- name: config-file
mountPath: /mosquitto/config/mosquitto.conf
subPath: mosquitto.conf
securityContext:
runAsNonRoot: true
fsGroup: 1
runAsGroup: 1000
runAsUser: 1000
supplementalGroups:
- 1
volumes:
- name: config-file
configMap:
name: mosquitto-config
- name: password-file
configMap:
name: mosquitto-password
- name: mosquitto-data
emptyDir: {}
- name: mosquitto-log
emptyDir: {}
- svc-mosquitto.yaml
apiVersion: v1
kind: Service
metadata:
name: chirpstack-mosquitto
labels:
app: chirpstack-mosquitto
spec:
type: ClusterIP
ports:
- name: tcp-mqtt
port: 31883
targetPort: 31883
protocol: TCP
selector:
app: chirpstack-mosquittovalidate the dns
root@ubuntu:/# dig -t A chirpstack-mosquitto
;; Warning: Message parser reports malformed message packet.
; <<>> DiG 9.16.8-Ubuntu <<>> -t A chirpstack-mosquitto
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61934
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;chirpstack-mosquitto. IN A
;; ANSWER SECTION:
chirpstack-mosquitto. 64 IN A 10.10.5.202
;; Query time: 0 msec
;; SERVER: 172.17.0.1#53(172.17.0.1)
;; WHEN: Thu Jun 10 13:45:27 CST 2021
;; MSG SIZE rcvd: 54
pub and sub the broker of service -> endpoints
mosquitto_sub -h chirpstack-mosquitto -p 31883 -t test
mosquitto_pub -h chirpstack-mosquitto -p 31883 -t test -m 'hello'
edgemesh's log
I0610 05:27:48.565550 1 dns.go:298] [EdgeMesh] dns server parse chirpstack-mosquitto ip 10.10.5.202
W0610 05:27:48.565628 1 dns.go:136] [EdgeMesh] failed to resolve dns: get from real dns
I0610 05:27:48.571146 1 serviceproxy.go:53] ip: 10.10.5.202 port: 31883
I0610 05:27:48.571192 1 serviceproxy.go:125] newProtocolFromSock().svcPorts:tcp,31883,31883|http,32901,32901|default.chirpstack-mosquitto
W0610 05:27:48.571259 1 strategy.go:21] failed to find default.chirpstack-mosquitto DestinationRule, use default strategy RoundRobin from config file
I0610 05:27:48.571267 1 tcp.go:84] MicroServiceName:chirpstack-mosquitto.default.svc.cluster.local:31883, Endpoint:, Port:, SourceServiceID:, Protocol:tcp, Strategy:RoundRobin, Args:[]
I0610 05:27:48.571284 1 log.go:181] DEBUG: add [2] handlers for chain [tcp]
I0610 05:27:48.571440 1 tcp.go:120] responseCallback:ep:192.168.241.181:31883
I0610 05:27:48.571455 1 tcp.go:144] l4 proxy get server address: 192.168.241.181:31883
I0610 05:27:48.572029 1 tcp.go:187] l4 Proxy start a proxy to server 192.168.241.181:31883
I0610 05:27:48.623079 1 tcp.go:212] processClientProxy read err: EOF
I0610 05:27:48.623282 1 tcp.go:257] processServerProxy read err: EOF
E0610 05:27:48.623295 1 tcp.go:260] rconn close err: close tcp 192.168.241.180:49800->192.168.241.181:31883: use of closed network connection
E0610 05:27:48.623306 1 tcp.go:264] lconn close err: close tcp 172.17.0.1:40001->172.17.0.3:40780: use of closed network connection
Failed case
将宿主机的mosquitto当作外部资源引入到Service层，这种方式需要先创建Endpoints，然后创建同名的Service与之对应。
Mapping the host's mosquitto as an external resource to the Service layer. In this way, you need to create Endpoints first, and then create a Service with the same name to correspond to it.
endpoints -> service
- endpoint-edge1.yaml
kind: Endpoints
apiVersion: v1
metadata:
name: mosquitto-host-edge1
namespace: default
subsets:
- addresses:
- ip: 192.168.241.180
ports:
- port: 1883
name: tcp-mqtt-edge1
- service-edge1.yaml
apiVersion: v1
kind: Service
metadata:
name: mosquitto-host-edge1
spec:
ports:
- port: 1883
name: tcp-mqtt-edge1
protocol: TCPvalidate the dns
root@ubuntu:/# dig -t A mosquitto-host-edge1
;; Warning: Message parser reports malformed message packet.
; <<>> DiG 9.16.8-Ubuntu <<>> -t A mosquitto-host-edge1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58529
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;mosquitto-host-edge1. IN A
;; ANSWER SECTION:
mosquitto-host-edge1. 64 IN A 10.10.179.131
;; Query time: 0 msec
;; SERVER: 172.17.0.1#53(172.17.0.1)
;; WHEN: Thu Jun 10 13:30:08 CST 2021
;; MSG SIZE rcvd: 54
sub the broker of endpoint -> service
mosquitto_sub -h mosquitto-host-edge1 -p 1883 -t test
NOTE! Start the mosquitto_sub process, the edgemesh continue to report errors.
edgemesh's log
I0610 05:31:43.840156 1 dns.go:298] [EdgeMesh] dns server parse mosquitto-host-edge1 ip 10.10.179.131
W0610 05:31:43.840216 1 dns.go:136] [EdgeMesh] failed to resolve dns: get from real dns
I0610 05:31:43.845059 1 serviceproxy.go:53] ip: 10.10.179.131 port: 1883
I0610 05:31:43.845071 1 serviceproxy.go:125] newProtocolFromSock().svcPorts:tcp,1883,1883|default.mosquitto-host-edge1
W0610 05:31:43.845138 1 strategy.go:21] failed to find default.mosquitto-host-edge1 DestinationRule, use default strategy RoundRobin from config file
I0610 05:31:43.845144 1 tcp.go:84] MicroServiceName:mosquitto-host-edge1.default.svc.cluster.local:1883, Endpoint:, Port:, SourceServiceID:, Protocol:tcp, Strategy:RoundRobin, Args:[]
I0610 05:31:43.845152 1 log.go:181] DEBUG: add [2] handlers for chain [tcp]
I0610 05:31:43.845184 1 tcp.go:120] responseCallback:ep:192.168.241.181:0
I0610 05:31:43.845188 1 tcp.go:144] l4 proxy get server address: 192.168.241.181:0
E0610 05:31:43.845766 1 tcp.go:156] l4 proxy dial server error: dial tcp 192.168.241.181:0: connect: connection refused
I0610 05:31:44.847295 1 dns.go:298] [EdgeMesh] dns server parse mosquitto-host-edge1 ip 10.10.179.131
W0610 05:31:44.847334 1 dns.go:136] [EdgeMesh] failed to resolve dns: get from real dns
I0610 05:31:44.852629 1 serviceproxy.go:53] ip: 10.10.179.131 port: 1883
I0610 05:31:44.852655 1 serviceproxy.go:125] newProtocolFromSock().svcPorts:tcp,1883,1883|default.mosquitto-host-edge1
W0610 05:31:44.852688 1 strategy.go:21] failed to find default.mosquitto-host-edge1 DestinationRule, use default strategy RoundRobin from config file
I0610 05:31:44.852693 1 tcp.go:84] MicroServiceName:mosquitto-host-edge1.default.svc.cluster.local:1883, Endpoint:, Port:, SourceServiceID:, Protocol:tcp, Strategy:RoundRobin, Args:[]
I0610 05:31:44.852718 1 log.go:181] DEBUG: add [2] handlers for chain [tcp]
I0610 05:31:44.852765 1 tcp.go:120] responseCallback:ep:192.168.241.181:0
I0610 05:31:44.852771 1 tcp.go:144] l4 proxy get server address: 192.168.241.181:0
E0610 05:31:44.853449 1 tcp.go:156] l4 proxy dial server error: dial tcp 192.168.241.181:0: connect: connection refused
......
pub the broker of endpoint -> service
- sub the topic from edge1 host
[root@ke-edge1 ~]# mosquitto_sub -h 127.0.0.1 -p 1883 -t test
- pub the topic from pod in the edge
root@ubuntu:/# mosquitto_pub -h mosquitto-host-edge1 -p 1883 -t test -m 'hello'
Error: The connection was lost.
- NOTE! Start the mosquitto_pub process, the edgemesh report error, and the mosquitto_sub 127.0.0.1 is no message is received.
edgemesh's log
I0610 05:33:14.925329 1 dns.go:298] [EdgeMesh] dns server parse mosquitto-host-edge1 ip 10.10.179.131
W0610 05:33:14.925426 1 dns.go:136] [EdgeMesh] failed to resolve dns: get from real dns
I0610 05:33:14.930465 1 serviceproxy.go:53] ip: 10.10.179.131 port: 1883
I0610 05:33:14.930477 1 serviceproxy.go:125] newProtocolFromSock().svcPorts:tcp,1883,1883|default.mosquitto-host-edge1
W0610 05:33:14.930507 1 strategy.go:21] failed to find default.mosquitto-host-edge1 DestinationRule, use default strategy RoundRobin from config file
I0610 05:33:14.930511 1 tcp.go:84] MicroServiceName:mosquitto-host-edge1.default.svc.cluster.local:1883, Endpoint:, Port:, SourceServiceID:, Protocol:tcp, Strategy:RoundRobin, Args:[]
I0610 05:33:14.930519 1 log.go:181] DEBUG: add [2] handlers for chain [tcp]
I0610 05:33:14.930588 1 tcp.go:120] responseCallback:ep:192.168.241.170:0
I0610 05:33:14.930593 1 tcp.go:144] l4 proxy get server address: 192.168.241.170:0
E0610 05:33:14.931273 1 tcp.go:156] l4 proxy dial server error: dial tcp 192.168.241.170:0: connect: connection refused
I modified part of the log output, trying to find the cause of the error, but I suffered from the lack of knowledge of the principle;
What you expected to happen:
It is passable to run the second test case in the k8s cloud.
Environment:
-
EdgeMesh version: branch release-1.7
-
Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.10", GitCommit:"98d5dc5d36d34a7ee13368a7893dcb400ec4e566", GitTreeState:"clean", BuildDate:"2021-04-15T03:28:42Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.10", GitCommit:"98d5dc5d36d34a7ee13368a7893dcb400ec4e566", GitTreeState:"clean", BuildDate:"2021-04-15T03:20:25Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}
-
KubeEdge version(e.g. cloudcore --version and edgecore --version):
branch release-1.7
-
Cloud nodes Environment:
- Hardware configuration (e.g.
lscpu):
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 2
Socket(s): 2
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 142
Model name: Intel(R) Core(TM) i5-10210U CPU @ 1.60GHz
Stepping: 12
CPU MHz: 2112.048
BogoMIPS: 4224.00
Hypervisor vendor: VMware
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 6144K
NUMA node0 CPU(s): 0-3
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 arat
- OS (e.g.
cat /etc/os-release):
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
- Kernel (e.g. `uname -a`):
Linux ke-master1 3.10.0-693.el7.x86_64 #1 SMP Tue Aug 22 21:09:27 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
- Go version (e.g. go version):
go version go1.14.15 linux/amd64
- Others:
-
Edge nodes Environment:
Same to cloud
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent