Comments (11)
Sorry for the delay. I do not usually use docker these days.
Docker works a bit differently in this area but it does work intuitively, at least for me.
- Docker seems to only allow creating a container with one network attached. After creating you can separately run
docker network connect $network_name $container_name
to add the second network. I was not able to find another way with a quick look. - Docker provides only one dns server in
/etc/resolv.conf
: even when multiple networks are attached. This is the same across all containers
/ # cat /etc/resolv.conf
search mydomain.com vagrant.mydomain.com
nameserver 127.0.0.11
options edns0 trust-ad ndots:0
- The original container automatically gets its short name and a long name that is specific to its network. Not a generic name like podman provides. So
container1
from this example resolves atcontainer1
andcontainer1.network1
. - Since
container1
is only connected to one network I created bothcontainer2
andcontainer3
that are both connected to bothnetwork1
andnetwork2
and tested resolvingcontainer1
andcontainer2
fromcontainer3
. As there is only one dns server this is simpler and avoids having the OS figure out which nameserver is responsible for what as well as any timeouts associated with that.
From container3
: the following names resolve:
container1 -> 192.168.55.2
container1.network1 -> 192.168.55.2
container1.network2 -> bad address (as it should be as container1
is not connected to network2
container2 -> 192.168.55.3
container2.network1 -> 192.168.55.3
contaner2.network2 -> 192.168.56.2
From container1
things also resolve as expected. Namely only other containers in network1 with both shortnames and fqdns resolve successfully.
- Every name resolved to one and only one IP. A short name never returned both network1 and network2 names from any container.
Here is the list of commands I ran for setup.
docker network create --subnet 192.168.55.0/24 network1
docker network create --subnet 192.168.56.0/24 network2
docker run --detach --rm -ti --name container1 --network network1 alpine sleep 9000
docker run --rm -ti --name container2 -d --network network1 alpine sleep 9000
docker network connect network2 container2
docker run --rm -ti --name container3 -d --network network1 alpine sleep 9000
docker network connect network2 container3
From there I used the following commands to test:
docker exec -ti container1 apk add bind-tools
docker exec -ti container2 apk add bind-tools
docker exec -ti container3 apk add bind-tools
docker exec -ti container1 cat /etc/resolv.conf #can see above
docker exec -ti container2 cat /etc/resolv.conf #can see above
docker exec -ti container3 cat /etc/resolv.conf #can see above
docker exec -ti container1 dig +short container1 #-> 192.168.55.2
docker exec -ti container1 dig +short container1.network1 #-> 192.168.55.2
docker exec -ti container1 dig +short container1.network2 #-> blank (entry should not exist)
docker exec -ti container1 dig +short container3.network2 #-> blank (container1 is not in network2 so access is denied)
docker exec -ti container3 dig +short container1 #-> 192.168.55.2
docker exec -ti container3 dig +short container1.network1 #-> 192.168.55.2
docker exec -ti container3 dig +short container1.network2 #-> blank (entry should not exist)
docker exec -ti container3 dig +short container2 #-> 192.168.55.3
docker exec -ti container3 dig +short container2.network1 #-> 192.168.55.3
docker exec -ti container3 dig +short container2.network2 #-> 192.168.56.2
docker exec -ti container3 dig +short container3 #-> 192.168.55.4
Hopefully this covers everything.
edit: my docker version: Docker version 20.10.16, build aa7e414
on fedora 36
edit2: change second sentence to be less presumptuous.
from aardvark-dns.
I move this to aardvark-dns, looks like this is still a problem
from aardvark-dns.
@flouthoc PTAL
from aardvark-dns.
@ykuksenko I think container1
is only connected to network1
please connect it to network2
as well in order to get it resolved from nameserver in network2
. So podman run --detach --rm -ti --name container1 --network network1,network2 alpine sleep 9000
should be the right thing here.
OTOH you could say that bug is that short name should not get resolved by nameserver if container is not connected to the network of nameserver.
from aardvark-dns.
container1
is intentionally connected to only network1
for isolation. For some additional context of where I ran into this issue, I use the following scenario so that the database
is more isolated from the load balancer
:
Load Balancer ------------> Application -------------> Database
(network1) (network2)
In this scenarios the application
container sometimes fails to resolve the database
when addressed with FQDN. I prefer using FQDNs for names because it makes it really obvious what the term refers to and that the name is meant to be internally resolved, and not by my main DNS server.
I suppose the biggest thing I would like to see is consistent behavior:
- If shortnames should be resolvable universally, then FQDNs should be too. (preferred solution)
- If FQDNs should not be resolvable universally then shortnames should not be too.
I do not see any reason for shortnames to resolve and FQDNs to fail or vise versa. If there is such a reason could you share it?
I prefer having both FQDNs and shortnames to be resolvable because:
- the autogenerated
/etc/resolv.conf
file randomly orders podman network name servers which causes unpredictability - FQDNs avoid name ambiguity if the short name is defined in the host DNS servers. Or if DNS suffixes are utilized (not sure if these are configurable in netavark/aardvark)
- it is convenient
- it avoids having tweaking /etc/resolv.conf to try multiple DNS servers in case the first one cannot respond and response durations. (With the alpine image, if the first server cannot respond pings simply fail saying bad address, same with Keycloak trying to connect to a database)
- Podman container names are unique so any server being able to respond on behalf of any attached network for internal resources seems sound. (This brings up the question of why have more than one server if they act as one anyway but this is not important)
- I am not aware of any down side to this. I am missing one?
I am under the impression that netavark and aardvark-dns should combine DNS results from any network a container is connected to according to @mheon (containers/podman#14262 (comment))
from aardvark-dns.
@ykuksenko Does the requested feature works on docker cause this seems a bug a too me I think we should fix this otherway around and make short names also not resolvable if they are not part of the network.
from aardvark-dns.
should this be moved to nv?
from aardvark-dns.
A friendly reminder that this issue had no activity for 30 days.
from aardvark-dns.
Bump. Happening to me and it's very annoying! @flouthoc I can confirm this works as expected in docker, been using this networking stuff for years.
In fact this issue was first reported early 2021 here: containers/podman#9492
from aardvark-dns.
@ykuksenko: The label(s) kind/bug
cannot be applied, because the repository doesn't have them.
In response to this:
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
When a container is connected to multiple networks only one network can resolve containers by fqdn.
Steps to reproduce the issue:
- create two networks
podman network create --subnet 192.168.55.0/24 network1 podman network create --subnet 192.168.56.0/24 network2
- start up a container that will be resolved in dns attached only to network1 and have it do nothing
podman run --detach --rm -ti --name container1 --network network1 alpine sleep 9000
- create a container attached to both networks and run dig to resolve the fqdn of the first container against both network dns servers in resolv.conf
podman run --rm -ti --name container2 --network network1,network2 alpine sh -c "cat /etc/resolv.conf; apk add bind-tools > /dev/null; echo '<<<<<<<<<<< network1 dns test'; dig container1.dns.podman @192.168.55.1; echo '<<<<<<<<<<< network2 dns test'; dig container1.dns.podman @192.168.56.1"
- repeat number 3 with short names
podman run --rm -ti --name container2 --network network1,network2 alpine sh -c "cat /etc/resolv.conf; apk add bind-tools > /dev/null; echo '<<<<<<<<<<< network1 dns test'; dig container1 @192.168.55.1; echo '<<<<<<<<<<< network2 dns test'; dig container1 @192.168.56.1"
Describe the results you received:
When resolving the fqdn of container1 only one name server responds correctly.search dns.podman dns.podman nameserver 192.168.55.1 nameserver 192.168.56.1 nameserver 192.168.121.1 <<<<<<<<<<< network1 dns test ... (clipped for clarity) ;; QUESTION SECTION: ;container1.dns.podman. IN A ;; ANSWER SECTION: container1.dns.podman. 86400 IN A 192.168.55.2 ;; Query time: 1 msec ;; SERVER: 192.168.55.1#53(192.168.55.1) ;; WHEN: Tue May 24 14:37:03 UTC 2022 ;; MSG SIZE rcvd: 78 <<<<<<<<<<< network2 dns test ... (clipped for clarity) ;; QUESTION SECTION: ;container1.dns.podman. IN A ;; Query time: 3 msec ;; SERVER: 192.168.56.1#53(192.168.56.1) ;; WHEN: Tue May 24 14:37:03 UTC 2022 ;; MSG SIZE rcvd: 62
When resolving the short name of the container one both name server respond correctly
search dns.podman dns.podman nameserver 192.168.56.1 nameserver 192.168.55.1 nameserver 192.168.121.1 <<<<<<<<<<< network1 dns test ... (clipped for clarity) ;; QUESTION SECTION: ;container1. IN A ;; ANSWER SECTION: container1. 86400 IN A 192.168.55.2 ;; Query time: 2 msec ;; SERVER: 192.168.55.1#53(192.168.55.1) ;; WHEN: Tue May 24 14:38:01 UTC 2022 ;; MSG SIZE rcvd: 67 <<<<<<<<<<< network2 dns test ... (clipped for clarity) ;; QUESTION SECTION: ;container1. IN A ;; ANSWER SECTION: container1. 86400 IN A 192.168.55.2 ;; Query time: 3 msec ;; SERVER: 192.168.56.1#53(192.168.56.1) ;; WHEN: Tue May 24 14:38:01 UTC 2022 ;; MSG SIZE rcvd: 67
Describe the results you expected:
Both name servers should respond to both the shortname and fqdn queries.Additional information you deem important (e.g. issue happens only occasionally):
- /etc/resolv.conf entries show up unsorted so if a user uses fqdns to reference containers they will have intermittent issues in applications. ( see containers/podman#14262 )
- It is also notable that /etc/resolv.conf lists the dns suffix dns.podman twice - this probably does not harm anything but should probably be deduplicated by podman.
Output of
podman version
:# podman version Client: Podman Engine Version: 4.1.0 API Version: 4.1.0 Go Version: go1.18 Built: Fri May 6 16:15:54 2022 OS/Arch: linux/amd64
Output of
podman info --debug
:# podman info --debug host: arch: amd64 buildahVersion: 1.26.1 cgroupControllers: - cpuset - cpu - io - memory - hugetlb - pids - misc cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon-2.1.0-2.fc36.x86_64 path: /usr/bin/conmon version: 'conmon version 2.1.0, commit: ' cpuUtilization: idlePercent: 97.24 systemPercent: 0.93 userPercent: 1.83 cpus: 2 distribution: distribution: fedora variant: cloud version: "36" eventLogger: journald hostname: container.redacted idMappings: gidmap: null uidmap: null kernel: 5.17.5-300.fc36.x86_64 linkmode: dynamic logDriver: journald memFree: 148381696 memTotal: 6217089024 networkBackend: netavark ociRuntime: name: crun package: crun-1.4.4-1.fc36.x86_64 path: /usr/bin/crun version: |- crun version 1.4.4 commit: 6521fcc5806f20f6187eb933f9f45130c86da230 spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL os: linux remoteSocket: path: /run/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: false seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.2.0-0.2.beta.0.fc36.x86_64 version: |- slirp4netns version 1.2.0-beta.0 commit: 477db14a24ff1a3de3a705e51ca2c4c1fe3dda64 libslirp: 4.6.1 SLIRP_CONFIG_VERSION_MAX: 3 libseccomp: 2.5.3 swapFree: 5851836416 swapTotal: 6217003008 uptime: 15h 16m 15.62s (Approximately 0.62 days) plugins: log: - k8s-file - none - passthrough - journald network: - bridge - macvlan volume: - local registries: search: - docker.io store: configFile: /usr/share/containers/storage.conf containerStore: number: 19 paused: 0 running: 19 stopped: 0 graphDriverName: overlay graphOptions: overlay.mountopt: nodev,metacopy=on graphRoot: /var/lib/containers/storage graphRootAllocated: 41788899328 graphRootUsed: 9318744064 graphStatus: Backing Filesystem: btrfs Native Overlay Diff: "false" Supports d_type: "true" Using metacopy: "true" imageCopyTmpDir: /var/tmp imageStore: number: 67 runRoot: /run/containers/storage volumePath: /var/lib/containers/storage/volumes version: APIVersion: 4.1.0 Built: 1651853754 BuiltTime: Fri May 6 16:15:54 2022 GitCommit: "" GoVersion: go1.18 Os: linux OsArch: linux/amd64 Version: 4.1.0
Package info (e.g. output of
rpm -q podman
orapt list podman
):# rpm -q netavark podman netavark-1.0.3-3.fc36.x86_64 podman-4.1.0-1.fc36.x86_64
Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
Yes
Additional environment details (AWS, VirtualBox, physical, etc.):
Libvirt VM
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.
from aardvark-dns.
@flouthoc @Luap99 @felixsanz is @ykuksenko Is this still an issue?
from aardvark-dns.
Related Issues (20)
- Recursion Available bit is not set in response header HOT 3
- Shall we lookup host's /etc/hosts before forwarding other request to host's /etc/resolv.conf? HOT 5
- Need way to tell aardvark DNS to refer to a particular DNS, and not host's configured DNS HOT 13
- dns request failed: request timed out HOT 22
- dns: inbuilt resolver should return both `IPv6` and `IPv4` records if request type is `ANY` HOT 2
- Add LICENSE file and COC to repoistory HOT 1
- Dependency Dashboard
- Disable Dependabot after renovate trial
- Need bidirectional communication channel between netavark and aardvark HOT 8
- Add host.containers.internal entry in aardvark-dns HOT 2
- [NOT UPSTREAM PROBLEM] test `packit propose-downstream` HOT 2
- [packit] Propose downstream failed for release v1.7.0
- test_backend_network_scoped_custom_dns_server fails HOT 3
- Updating trust-dns HOT 1
- DNS requests timeout HOT 24
- Is there a way to reserve or limit IP addresses when using DNS? HOT 1
- CI flake: three networks with a connect HOT 1
- When forward dns request to outside name server, `aardvark-dns` should check and ignore its own listening IPs or error out, to avoid infinite recursion. HOT 1
- Setting invalid options in /etc/resolv.conf makes dns unresponsive HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aardvark-dns.