GithubHelp home page GithubHelp logo

m13253 / dns-over-https Goto Github PK

View Code? Open in Web Editor NEW
1.9K 67.0 219.0 363 KB

High performance DNS over HTTPS client & server

Home Page: https://developers.google.com/speed/public-dns/docs/dns-over-https

License: MIT License

Go 91.83% Makefile 4.06% Shell 0.83% Swift 3.27%
dns-over-https doh dns google-dns ietf-doh rfc-8484

dns-over-https's Introduction

DNS-over-HTTPS

Client and server software to query DNS over HTTPS, using Google DNS-over-HTTPS protocol and IETF DNS-over-HTTPS (RFC 8484).

Guides

Installing

From Source

  • Install Go, at least version 1.20. The newer, the better.

Note for Debian/Ubuntu users: You need to set $GOROOT if you could not get your new version of Go selected by the Makefile.

  • First create an empty directory, used for $GOPATH:
mkdir ~/gopath
export GOPATH=~/gopath
  • To build the program, type:
make
  • To install DNS-over-HTTPS as Systemd services, type:
sudo make install
  • By default, Google DNS over HTTPS is used. It should work for most users (except for People's Republic of China). If you need to modify the default settings, type:
sudoedit /etc/dns-over-https/doh-client.conf
  • To automatically start DNS-over-HTTPS client as a system service, type:
sudo systemctl start doh-client.service
sudo systemctl enable doh-client.service
  • Then, modify your DNS settings (usually with NetworkManager) to 127.0.0.1.

  • To test your configuration, type:

dig www.google.com
Output:
;; SERVER: 127.0.0.1#53(127.0.0.1)

Uninstall

  • To uninstall, type:
sudo make uninstall

Note: The configuration files are kept at /etc/dns-over-https. Remove them manually if you want.

Using docker image

docker run -d --name doh-server \
  -p 8053:8053 \
  -e UPSTREAM_DNS_SERVER="udp:8.8.8.8:53" \
  -e DOH_HTTP_PREFIX="/dns-query" \
  -e DOH_SERVER_LISTEN=":8053" \
  -e DOH_SERVER_TIMEOUT="10" \
  -e DOH_SERVER_TRIES="3" \
  -e DOH_SERVER_VERBOSE="false" \
  satishweb/doh-server

Feeling adventurous? Try the latest build:

  • m13253/dns-over-https-server:latest
  • m13253/dns-over-https-client:latest

Logging

All log lines (by either doh-client or doh-server) are written into stderr; you can view them using your OS tool of choice (journalctl when using systemd).

Server Configuration

The following is a typical DNS-over-HTTPS architecture:

+--------------+                                +------------------------+
| Application  |                                |  Recursive DNS Server  |
+-------+------+                                +-----------+------------+
        |                                                   |
+-------+------+                                +-----------+------------+
| Client side  |                                |      doh-server        |
| cache (nscd) |                                +-----------+------------+
+-------+------+                                            |
        |         +--------------------------+  +-----------+------------+
+-------+------+  |    HTTP cache server /   |  |   HTTP service muxer   |
|  doh-client  +--+ Content Delivery Network +--+ (Apache, Nginx, Caddy) |
+--------------+  +--------------------------+  +------------------------+

Although DNS-over-HTTPS can work alone, an HTTP service muxer would be useful as you can host DNS-over-HTTPS along with other HTTPS services.

HTTP/2 with at least TLS v1.3 is recommended. OCSP stapling must be enabled, otherwise DNS recursion may happen.

Configuration file

The main configuration file is doh-client.conf.

Server selectors. If several upstream servers are set, one is selected according to upstream_selector for each request. With upstream_selector = "random", a random upstream server will be chosen for each request.

# available selector: random (default) or weighted_round_robin or lvs_weighted_round_robin
upstream_selector = "random"

Example configuration: Apache

SSLProtocol TLSv1.2
SSLHonorCipherOrder On
SSLCipherSuite ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+3DES:!aNULL:!MD5:!DSS:!eNULL:!EXP:!LOW:!MD5
SSLUseStapling on
SSLStaplingCache shmcb:/var/lib/apache2/stapling_cache(512000)

<VirtualHost *:443>
    ServerName MY_SERVER_NAME
    Protocols h2 http/1.1
    ProxyPass /dns-query http://[::1]:8053/dns-query
    ProxyPassReverse /dns-query http://[::1]:8053/dns-query
</VirtualHost>

(Credit: Joan Moreau)

Example configuration: Nginx

server {
  listen       443 ssl http2 default_server;
  listen       [::]:443 ssl http2 default_server;
  server_name  MY_SERVER_NAME;

  server_tokens off;

  ssl_protocols TLSv1.2 TLSv1.3;          # TLS 1.3 requires nginx >= 1.20.0
  ssl_prefer_server_ciphers on;
  ssl_dhparam /etc/nginx/dhparam.pem;     # openssl dhparam -dsaparam -out /etc/nginx/dhparam.pem 4096
  ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
  ssl_ecdh_curve secp384r1;               # Requires nginx >= 1.1.0
  ssl_session_timeout  10m;
  ssl_session_cache shared:SSL:10m;
  ssl_session_tickets off;                # Requires nginx >= 1.5.9
  ssl_stapling on;                        # Requires nginx >= 1.3.7
  ssl_stapling_verify on;                 # Requires nginx => 1.3.7
  ssl_early_data off;                     # 0-RTT, enable if desired - Requires nginx >= 1.15.4
  resolver 1.1.1.1 valid=300s;            # Replace with your local resolver
  resolver_timeout 5s;
  # HTTP Security Headers
  add_header X-Frame-Options DENY;
  add_header X-Content-Type-Options nosniff;
  add_header X-XSS-Protection "1; mode=block";
  add_header Strict-Transport-Security "max-age=63072000";
  ssl_certificate /path/to/your/server/certificates/fullchain.pem;
  ssl_certificate_key /path/to/your/server/certificates/privkey.pem;
  location /dns-query {
    proxy_pass       http://localhost:8053/dns-query;
    proxy_set_header Host      $host;
    proxy_set_header X-Real-IP $remote_addr;
  }
}

(Credit: Cipherli.st)

Example configuration: Caddy (v2)

my.server.name {
        reverse_proxy * localhost:8053
        tls [email protected]
        try_files {path} {path}/index.php /index.php?{query}
}

Example configuration: Docker Compose + Traefik + Unbound (Raspberry Pi/Linux/Mac) [linux/amd64,linux/arm64,linux/arm/v7]

version: '2.2'
networks:
  default:

services:
  proxy:
    # The official v2 Traefik docker image
    image: traefik:v2.3
    hostname: proxy
    networks:
      - default
    environment:
      TRAEFIK_ACCESSLOG: "true"
      TRAEFIK_API: "true"
      TRAEFIK_PROVIDERS_DOCKER: "true"
      TRAEFIK_API_INSECURE: "true"
      TRAEFIK_PROVIDERS_DOCKER_NETWORK: "${STACK}_default"
      # DNS provider specific environment variables for DNS Challenge using route53 (AWS)
      AWS_ACCESS_KEY_ID: ${AWS_ACCESS_KEY_ID}
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET_ACCESS_KEY}
      AWS_REGION: ${AWS_REGION}
      AWS_HOSTED_ZONE_ID: ${AWS_HOSTED_ZONE_ID}
    ports:
      # The HTTP port
      - "80:80"
      # The HTTPS port
      - "443:443"
      # The Web UI (enabled by --api.insecure=true)
      - "8080:8080"
    command:
      #- "--log.level=DEBUG"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge=true"
      # Providers list:
      #  https://docs.traefik.io/https/acme/#providers
      #  https://go-acme.github.io/lego/dns/
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=route53"
      # Enable below line to use staging letsencrypt server.
      #- "--certificatesresolvers.letsencrypt.acme.caserver=https://acme-staging-v02.api.letsencrypt.org/directory"
      - "--certificatesresolvers.letsencrypt.acme.email=${EMAIL}"
      - "--certificatesresolvers.letsencrypt.acme.storage=/certs/acme.json"
    volumes:
      # So that Traefik can listen to the Docker events
      - /var/run/docker.sock:/var/run/docker.sock
      - ./data/proxy/certs:/certs
  doh-server:
    image: satishweb/doh-server:latest
    hostname: doh-server
    networks:
      - default
    environment:
      # Enable below line to see more logs
      # DEBUG: "1"
      UPSTREAM_DNS_SERVER: "udp:unbound:53"
      DOH_HTTP_PREFIX: "${DOH_HTTP_PREFIX}"
      DOH_SERVER_LISTEN: ":${DOH_SERVER_LISTEN}"
      DOH_SERVER_TIMEOUT: "10"
      DOH_SERVER_TRIES: "3"
      DOH_SERVER_VERBOSE: "false"
    #volumes:
      # - ./doh-server.conf:/server/doh-server.conf
      # - ./app-config:/app-config
    depends_on:
      - unbound
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.doh-server.rule=Host(`${SUBDOMAIN}.${DOMAIN}`) && Path(`${DOH_HTTP_PREFIX}`)"
      - "traefik.http.services.doh-server.loadbalancer.server.port=${DOH_SERVER_LISTEN}"
      - "traefik.http.middlewares.mw-doh-compression.compress=true"
      - "traefik.http.routers.doh-server.tls=true"
      - "traefik.http.middlewares.mw-doh-tls.headers.sslredirect=true"
      - "traefik.http.middlewares.mw-doh-tls.headers.sslforcehost=true"
      - "traefik.http.routers.doh-server.tls.certresolver=letsencrypt"
      - "traefik.http.routers.doh-server.tls.domains[0].main=${DOMAIN}"
      - "traefik.http.routers.doh-server.tls.domains[0].sans=${SUBDOMAIN}.${DOMAIN}"
      # Protection from requests flood
      - "traefik.http.middlewares.mw-doh-ratelimit.ratelimit.average=100"
      - "traefik.http.middlewares.mw-doh-ratelimit.ratelimit.burst=50"
      - "traefik.http.middlewares.mw-doh-ratelimit.ratelimit.period=10s"
  unbound:
    image: satishweb/unbound:latest
    hostname: unbound
    networks:
      - default
    ports:
      # Disable these ports if DOH server is the only client
      - 53:53/tcp
      - 53:53/udp
    volumes:
      - ./unbound.sample.conf:/templates/unbound.sample.conf
      - ./data/unbound/custom:/etc/unbound/custom
      # Keep your custom.hosts file inside custom folder
    #environment:
    #  DEBUG: "1"

Complete Guide available at: https://github.com/satishweb/docker-doh

IPV6 Support for Docker Compose based configuration TBA

Example configuration: DNS-over-TLS

There is no native DNS-over-TLS support, but you can easily add it via nginx:

stream {
    server {
        listen                  *:853 ssl;
        proxy_pass              ipofyourdnsresolver:port  #127.0.0.1:53
    }

    ssl_certificate /etc/letsencrypt/live/site.yourdomain/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/site.yourdomain/privkey.pem;
}

The DoT service can also be provided by running a STunnel instance to wrap dnsmasq (or any other resolver of your choice, listening on a TCP port); this approach does not need a stand-alone daemon to provide the DoT service.

DNSSEC

DNS-over-HTTPS is compatible with DNSSEC, and requests DNSSEC signatures by default. However, signature validation is not built-in. It is highly recommended that you install unbound or bind and pass results for them to validate DNS records. An instance of Pi Hole could also be used to validate DNS signatures as well as provide other capabilities.

EDNS0-Client-Subnet (GeoDNS)

DNS-over-HTTPS supports EDNS0-Client-Subnet protocol, which submits part of the client's IP address (/24 for IPv4, /56 for IPv6 by default) to the upstream server. This is useful for GeoDNS and CDNs to work, and is exactly the same configuration as most public DNS servers.

Keep in mind that /24 is not enough to track a single user, although it is precise enough to know the city where the user is located. If you think EDNS0-Client-Subnet is affecting your privacy, you can set no_ecs = true in /etc/dns-over-https/doh-client.conf, with the cost of slower video streaming or software downloading speed.

To ultilize ECS, X-Forwarded-For or X-Real-IP should be enabled on your HTTP service muxer. If your server is backed by unbound or bind, you probably want to configure it to enable the EDNS0-Client-Subnet feature as well.

Protocol compatibility

Google DNS-over-HTTPS Protocol

DNS-over-HTTPS uses a protocol compatible to Google DNS-over-HTTPS, except for absolute expire time is preferred to relative TTL value. Refer to json-dns/response.go for a complete description of the API.

IETF DNS-over-HTTPS Protocol

DNS-over-HTTPS uses a protocol compatible to IETF DNS-over-HTTPS (RFC 8484).

Supported features

Currently supported features are:

  • IPv4 / IPv6
  • EDNS0 large UDP packet (4 KiB by default)
  • EDNS0-Client-Subnet (/24 for IPv4, /56 for IPv6 by default)

Known issues

  • it does not work well with dnscrypt-proxy, you might want to use either (or fix the compatibility bugs by submitting PRs)

The name of the project

This project is named "DNS-over-HTTPS" because it was written before the IETF DoH project. Although this project is compatible with IETF DoH, the project is not affiliated with IETF.

To avoid confusion, you may also call this project "m13253/DNS-over-HTTPS" or anything you like.

License

DNS-over-HTTPS is licensed under the MIT License. You are encouraged to embed DNS-over-HTTPS into your other projects, as long as the license permits.

You are also encouraged to disclose your improvements to the public, so that others may benefit from your modification, in the same way you receive benefits from this project.

dns-over-https's People

Contributors

1574242600 avatar achauvinhameau avatar buckaroogeek avatar chaz6 avatar dwoffinden avatar farseerfc avatar felixonmars avatar fuero avatar gdm85 avatar gontier-julien avatar greyxor avatar jamesits avatar jangrewe avatar joubin avatar leiless avatar m13253 avatar mghadam avatar mingaldrichgan avatar modib avatar monperrus avatar paulie-g avatar qyb avatar rwv avatar sanyo0714 avatar satishweb avatar sherlock-holo avatar takumin avatar testwill avatar usernamehyphen avatar wsquasher avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

dns-over-https's Issues

Active: activating (auto-restart)

root@iZwz9d1rjhrzzxa4lsoi93Z:~/build/dns-over-https-2.0.1# systemctl restart doh-server
root@iZwz9d1rjhrzzxa4lsoi93Z:~/build/dns-over-https-2.0.1# systemctl status doh-server
● doh-server.service - DNS-over-HTTPS Server
   Loaded: loaded (/usr/lib/systemd/system/doh-server.service; disabled; vendor preset: enabled)
   Active: activating (auto-restart) since Wed 2019-10-02 12:23:42 CST; 2s ago
     Docs: https://github.com/m13253/dns-over-https
  Process: 25954 ExecStart=/usr/local/bin/doh-server -conf /etc/dns-over-https/doh-server.conf (code=exited, status=0/SUC
 Main PID: 25954 (code=exited, status=0/SUCCESS)
lines 1-6/6 (END)

Unable to start the program, how can I solve it?

Add configure script

Please add a configure script so prefix options can be added. Such as prefix=/opt/dns-over-https and a server only install option so client pieces are not installed.

post queries?

Hello,
Is it possible to do post queries instead of get?

question: how to use random resolving?

Hi!

the code suggests that one can randomize the upstream servers:

Random   = "random"
NginxWRR = "weighted_round_robin"
LVSWRR   = "lvs_weighted_round_robin"

How to configure dns-over-https to use it?

Macbook becomes open resolver

I configured (on OSX):

listen = [
"127.0.0.1:53",
"[::1]:53",
## To listen on both 0.0.0.0:53 and [::]:53, use the following line
# ":53",
]

I then ran it and got a firewall warning which I accepted, but which also raised some suspicion.

So, I checked with

lsof -i ":53"

To my surprise doh-client is listening to 0.0.0.0:53 and [::]:53 and has therefore become an open resolver when I am on an unprotected network (such as the one where I am right now).

Am I missing something here?

configurable ECS addresses to defeat NAT / IPv6 tunnels inefficiency

I need to configure ECS addresses used for the following scenarios:

NAT

Some ISPs will NAT to other ISP addresses (e.g. 长城宽带). So When I use 长城宽带, usually another outgoing address is used instead to access the DNS server (e.g. NATed to **联通 or **电信) but some CDNs do have nodes inside my ISP's network (长城宽带).

IPv6 tunnels

I don't have IPv6 from my ISP but it's fun so I come up with a tunnel instead. When requesting DNS via this tunnel however, I receive A records optimized for my IPv6 tunnel.

BTW there seems to be a cache that ignores manually specified ECS addresses (via dig +subnet=xxx).

Support for URI Template

IETF DoH requires a URI Template while configuring the client.

For example: /query{?dns}

This implementation does not support URI Template yet, and still uses the obsolete ct= parameter to negotiate protocols. We need to eventually switch to URI Template in the future.

But since it does not affect compatibility (either client or server side), and I do not have a clear understanding of it until the next extension of the protocol is published, I plan to leave this issue open and postpone this.

And as always, pull request is welcome.

Need conditional forwarders

For hybrid setups, an option for conditional forwarders would be appreciated. In this case, I am testing with a bind server listening on 127.0.0.1:54. In the interim, I've just used the bind server as the forwarder, with it's own forwarders in tact. While an acceptable workaround, this is probably overkill.

Unsupported protocol scheme

Minimal issue reproduction

Go Version

1.15

OS and Arch

Ubuntu/Bionic Beaver - amd64

Steps

  1. add 127.0.0.53:53 in doh-client.conf to listen array (or if your dns is configured to listen on 127.0.0.1:53, then skip this step)
  2. stop systemd-resolved service (if it's running in your system)
  3. start doh-client with
./doh-client -conf doh-client.conf 
  1. send 4-5 dns requests using
host -v google.com

Output

Getting: Host google.com not found: 2(SERVFAIL) every 2-3 request using host tool

Also, on doh-client getting following output:

Get "?ct=application/dns-message&dns=AAABAAABAAAAAAABBnZvcnRleARkYXRhCW1pY3Jvc29mdANjb20AABwAAQAAKQSwAAAAAAAA": unsupported protocol scheme ""

Remarks

I dived into code and found that unsupported protocol scheme is from roundTrip function in net std library, which is called by (t *Transport) RoundTrip(r *Request).

Also, I see that this triggered in file doh-client/ietf.go:

if err != nil {
log.Println(err)
reply := jsondns.PrepareReply(r)
reply.Rcode = dns.RcodeServerFailure
w.WriteMsg(reply)
return &DNSRequest{
err: err,
}
}

My opinion issues is somewhere in client.go file func (c *Client) newHTTPClient() error

doh-server data race

Minimal issue reproduction

Go version

1.15

OS and Architecture

Ubuntu/Bionic Beaver - amd64

Steps

Run following test with -race flag on doh-server/server.go file. I've created a server_test.go file inside doh-server folder, copying here just a part of test that outputs data race and fails. Command

// loadWebsitesFromCSV loads and returns list of 1000 websites from .csv file (used for high load testing)
func loadWebsitesFromCSV() ([]string, error) {
	f, err := os.Open("websites.csv")
	if err != nil {
		return nil, errors.Wrap(err, "failed to load websites csv")
	}
	defer f.Close()

	csvR := csv.NewReader(f)
	var websites []string
	for {
		rec, err := csvR.Read()
		if err == io.EOF {
			break
		} else if err != nil {
			return nil, errors.Wrap(err, "failed to read record from websites csv")
		}

		websites = append(websites, rec[1]) // 1 index is website's name
	}

	return websites, nil
}

// worker sends a request to dns server
func worker(errChan chan<- error, websiteChan <-chan string, wg *sync.WaitGroup) {
	defer wg.Done()
	for {
		// check if websites are finished
		w, ok := <-websiteChan
		if !ok {
			return
		}

		ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()

		// prapare dns json request
		reqURL := fmt.Sprintf("http://127.0.0.1:8053/dns-query?name=%s&type=1", w)
		req, err := http.NewRequestWithContext(ctx, "GET", reqURL, nil)
		if err != nil {
			errChan <- errors.Errorf("website: %s, failed to create new GET request: %s", w, err)
			return
		}
		req.Close = true

		// add headers for google request
		req.Header.Add("Content-Type", "application/dns-json")
		req.Header.Add("Accept", "application/json")

		// send it to dns server
		resp, err := http.DefaultClient.Do(req)
		if err != nil {
			errChan <- errors.Errorf("website: %s, failed to make GET request: %s", w, err)
			return
		}
		defer resp.Body.Close()

		if resp.StatusCode != 200 {
			errChan <- errors.Errorf("website: %s, expected 200 response status code, got: %d", w, resp.StatusCode)
		}
	}
}

// TestHTTPServerWithLoad performs high load test on HTTP DNS server using 1000 domains from .csv file.
//
// Command for testing: go test -race -v doh-server/*.go -run ^TestHTTPServerWithLoad$
func TestHTTPServerWithLoad(t *testing.T) {
	conf := prepareConfig()
	conf.Listen = []string{
		"127.0.0.1:8053",
		"[::1]:8053",
	}

	// initialize a new server
	srv, err := NewServer(conf)
	if err != nil {
		t.Fatalf("failed to initialize new server: %v", err)
	}

	// start a server
	srvErr := make(chan error)
	go func(srvErr chan<- error) {
		if err := srv.Start(); err != nil {
			srvErr <- err
			return
		}
	}(srvErr)

	select {
	case <-srvErr:
		t.Fatalf("failed to start a server: %v", <-srvErr)
		break
	default:
		time.Sleep(500 * time.Millisecond) // sleep for some mills, so server has time to start

		// load 1000 websites from csv
		websites, err := loadWebsitesFromCSV()
		if err != nil {
			t.Fatal(err)
		}

		// run 11 workers, so requests would be faster
		websiteChan := make(chan string, 999)
		errChan := make(chan error, 10)
		wg := sync.WaitGroup{}
		for i := 0; i < 12; i++ {
			go worker(errChan, websiteChan, &wg)
			wg.Add(1)
		}

		// sends 1000 website to workers
		for _, w := range websites {
			websiteChan <- w
		}
		close(websiteChan)

		// if any error happens while sending request, just log it
		go func() {
			for err := range errChan {
				t.Log(err)
			}
			close(errChan)
		}()

		wg.Wait()
	}
}

Output

WARNING DATA RACE
github.com/miekg/dns.(*Client).ExchangeContext()
command-line-arguments.(*Server).doDNSQuery()
command-line-arguments.(*Server).handlerFunc()

Previous read at 0x00c00018ab80 by goroutine 31:
github.com/miekg/dns.(*Client).getTimeoutForRequest()
github.com/miekg/dns.(*Client).exchange()
github.com/miekg/dns.(*Client).ExchangeWithConn()
github.com/miekg/dns.(*Client).Exchange()
github.com/miekg/dns.(*Client).ExchangeContext()
command-line-arguments.(*Server).doDNSQuery()
command-line-arguments.(*Server).handlerFunc()

Goroutine 52 (running) created at:
net/http.(*Server).Serve()
net/http.(*Server).ListenAndServe()

Previous write at 0x00c000125620 by goroutine 62:
github.com/miekg/dns.(*Client).ExchangeContext()
command-line-arguments.(*Server).doDNSQuery()
command-line-arguments.(*Server).handlerFunc()

and so on ...

Remarks

I've changed private config struct in server.go file to public in order to perform test. Link to websites.csv file. So, as I see there is data race specifically in doDnsQuery and ExchangeContext functions. I hope I'm not doing wrong tests.

Can you give an example of how to configure nginx dot

Example configuration: Nginx doh
server {
listen 443 ssl http2 default_server;
listen [::]:443 ssl http2 default_server;
server_name MY_SERVER_NAME;

server_tokens off;

ssl_protocols TLSv1.2 TLSv1.3; # TLS 1.3 requires nginx >= 1.13.0
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/nginx/dhparam.pem; # openssl dhparam -dsaparam -out /etc/nginx/dhparam.pem 4096
ssl_ciphers EECDH+AESGCM:EDH+AESGCM;
ssl_ecdh_curve secp384r1; # Requires nginx >= 1.1.0
ssl_session_timeout 10m;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off; # Requires nginx >= 1.5.9
ssl_stapling on; # Requires nginx >= 1.3.7
ssl_stapling_verify on; # Requires nginx => 1.3.7
ssl_early_data off; # 0-RTT, enable if desired - Requires nginx >= 1.15.4
resolver 1.1.1.1 valid=300s; # Replace with your local resolver
resolver_timeout 5s;

HTTP Security Headers

add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header Strict-Transport-Security "max-age=63072000";
ssl_certificate /path/to/your/server/certificates/fullchain.pem;
ssl_certificate_key /path/to/your/server/certificates/privkey.pem;
location /dns-query {
proxy_pass http://localhost:8053/dns-query;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
(Credit: Cipherli.st)

How to configure nginx dot

curl 127.0.0.1:8053/dns-query returns json response status 2 comment invalid argument value and doh server doesn't seem to work

Hello ,

I appreciate you have put lot of effort in put this up together, I have been trying to setup DoH on my home network from more than 2 weeks now. going through your GitHub page I see some hope.

I pulled your repository followed steps make and then sudo make install on ubuntu running on arm64 architecture
now when I try curl 127.0.0.1:8053/dns-query I get response saying:
{"Status":2,"Comment":"Invalid argument value: \"ct\" = \"\""}

I have setup this with apache2, I already had domain name self hosted on ssl so I had cert I can use it .

so even when I tried https://selfhosted.ddns.com/dns-query I got the same json result.
I think the problem is on the doh server end to fix this json response.

I used this self hosted.ddns.com/dns-query in chrome://settings > advanced > Security > custom DNS and tried from google chrome by visiting 1.1.1.1/help but I still do not get DoH .

Any help will be greatly appreciated thank you

Add PIDfile support

In order to be compatible with "monit", is it possible to add the possibility to add the pidfile support in the doh-server ?

toml: cannot load TOML value of type string into a Go slice

Config file seems to fail to load with the error message on the title. Aparently it's related to data format.

In the .conf file I changed:
listen = "127.1.2.3:123"
to
listen = ["127.1.2.3:123"]

Not sure if that's correct, but apparetnly that got rid of the error message and doh-client started to work.
Go version is 1.10.1 linux/arm.

macOS: doh-logger crash after Swift 4.2 upgrade

dyld: Symbol not found: __T0SSMa
  Referenced from: /usr/local/bin/doh-logger
  Expected in: /Library/Developer/CommandLineTools/usr/lib/swift/macosx/libswiftCore.dylib
 in /usr/local/bin/doh-logger
fish: 'doh-logger' terminated by signal SIGABRT (Abort)

Solution:

  1. Start doh-client manually, to make sure you have a working temporary Internet access.
  2. Open Xcode, you will be asked to update command line tools.
  3. After the update, close Xcode
  4. Recompile (make clean all) and reinstall (sudo make install) DNS-over-HTTPS client.

Could you tag 2.0.0?

I see a commit about

Bump version 2.0.0

If you tag and release it, I will upgrade it on AUR

Is it working with Firefox?

Hello, I have installed and configured doh-server. Below is a redacted config file.

# HTTP listen port
listen = [
    "127.0.0.1:443",
    "[::1]:443",
    "IPV4:443",
    "[IPV6]:443",
]

# TLS certification file
cert = "/usr/local/doh-server/etc/dns_over_https.pem"

# TLS key file
key = "/usr/local/doh-server/etc/dns_over_https.key"

# HTTP path for resolve application
path = "/dns-query"

# Upstream DNS resolver
# If multiple servers are specified, a random one will be chosen each time.
upstream = [
    "127.0.0.1:53",
]

# Upstream timeout
timeout = 10

# Number of tries if upstream DNS fails
tries = 3

# Only use TCP for DNS query
tcp_only = false

# Enable logging
verbose = true

I use Unbound on the same server as upstream.

It seems to be working when I try with curl/doh (https://github.com/curl/doh) but not when enabling DNS-over-HTTPS in Firefox (61.0):

$ ./doh www.github.com https://SERVER/dns-query
www.github.com from https://SERVER/dns-query
TTL: 60 seconds
A: 192.30.253.112
A: 192.30.253.113
CNAME: github.com
CNAME: github.com
$ tail -f /var/log/doh-server/doh-server.log
[MYIP]:59738 - - [29/Jun/2018:14:06:10 +0200] "www.github.com. IN AAAA"
MYIP - - [29/Jun/2018:14:06:10 +0200] "POST /dns-query HTTP/2.0" 200 166 "" "curl-doh/1.0"

This is how I setup Firefox in about:config:

network.trr.bootstrapAddress;IPV4
network.trr.mode;3
network.trr.uri;https://SERVER/dns-query

I see some probes arriving but resolution doesn't seem to work in Firefox.

$ tail -f /var/log/doh-server/doh-server.log
[MYIP]:60282 - - [29/Jun/2018:13:34:04 +0200] "example.com. IN NS"
2001:a18:1:8::163 - - [29/Jun/2018:13:34:04 +0200] "POST /dns-query HTTP/2.0" 200 126 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/61.0"
[MYIP]:60282 - - [29/Jun/2018:13:35:00 +0200] "example.com. IN NS"
2001:a18:1:8::163 - - [29/Jun/2018:13:35:00 +0200] "POST /dns-query HTTP/2.0" 200 126 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/61.0"

If I set network.trr.mode to 2, then it falls back to not using DoH (as it seems not working).

Do you have any idea about this? What further tests can I make?

Regards,

Edit: I also tried with Firefox 62.0b3 and ended up with the same results.

X-Real-IP for private address not respected if ecs_allow_non_global_ip=true

I found that I had to set ecs_allow_non_global_ip=false in order for doh-server to respect the X-Real-IP header and send private addresses to upstream, which is the opposite of what that config variable is apparently intended to mean.

The problem seems to be that config file variable ecs_allow_non_global_ip is read to in-memory variable LocalIPFilter as-is, but the logical meaning of the two values is different. i.e. LocalIPFilter=true is treated as meaning "filter out local IPs", while ecs_allow_non_global_ip=true is supposed to mean "do NOT filter out local IPs".

local import "../json-dns" in non-local package

Try by downloading this package using go get github.com/m13253/dns-over-https.

I'm on MacOSX 10.13.3 and using Go 1.10.

mactriphe:src matriphe$ go get github.com/m13253/dns-over-https
package github.com/m13253/dns-over-https: no Go files in /Volumes/data/Development/golang/src/github.com/m13253/dns-over-https
mactriphe:src matriphe$ cd github.com/m13253/dns-over-https/
mactriphe:dns-over-https matriphe$ make
go get -d -v ./doh-client ./doh-server
github.com/miekg/dns (download)
github.com/BurntSushi/toml (download)
github.com/gorilla/handlers (download)
doh-client/client.go:37:2: local import "../json-dns" in non-local package
doh-server/google.go:36:2: local import "../json-dns" in non-local package
make: *** [deps] Error 1
mactriphe:dns-over-https matriphe$

doh-server not starting when key file is only readable by root

Hello

In doh-server.conf you can set a cert an key file. However, when the key file is only readable by root, the service won't start:

[root@testserver ~]# ls -l test.key
-r-------- 1 root root 1675 21. Sep 17:52 test.key

systemctl status doh-server
● doh-server.service - DNS-over-HTTPS Server
Loaded: loaded (/usr/lib/systemd/system/doh-server.service; disabled; vendor preset: disabled)
Active: activating (auto-restart) since Mon 2018-09-24 10:08:34 CEST; 949ms ago
Docs: https://github.com/m13253/dns-over-https
Process: 152070 ExecStart=/usr/local/bin/doh-server -conf /etc/dns-over-https/doh-server.conf (code=exited, status=0/SUCCESS)
Main PID: 152070 (code=exited, status=0/SUCCESS)

It works when permissions are set to 444.
I checked the service which drops its privileges and is then using AmbientCapabilities to bind to port 443; however I don't know how to read the file as user nobody with AmbientCapabilities.

doh-client has a serious memory leak problem under high concurrency

problem description

Background

  • BIND 9.16.1 (Stable Release)
  • dns-over-https (59f79fb)
  • Linux localhost 5.5.11-arch1-1 SMP PREEMPT Sun, x86_64 GNU/Linux

Architecture:

  • BIND forwards some DNS requests to doh-client.
  • BIND enables dnssec-validation.
  • Add NTA (negative trust anchor) for some domain names.

Phenomenon

The doh-client has high memory usage. When the memory usage reaches a certain level (greater than 5G), the program exits abnormally.

Screen capture at 3.7G memory
3 7G

doh-client.service keeps restarting
systemd-unit

Above, at the 6th time, 39208 process memory layout

# pmap 39208
39208:   /usr/bin/doh-client -conf /etc/dns-over-https/doh-client.conf
000000c000000000 1011712K rw---   [ anon ]
000000c03dc00000   2048K rw---   [ anon ]
000000c03de00000  20480K rw---   [ anon ]
000000c03f200000   2048K rw---   [ anon ]
000000c03f400000   4096K rw---   [ anon ]
000000c03f800000  18432K rw---   [ anon ]
000000c040a00000  12288K rw---   [ anon ]
000000c041600000   2048K rw---   [ anon ]
000000c041800000  81920K rw---   [ anon ]
000000c046800000   2048K rw---   [ anon ]
000000c046a00000   4096K rw---   [ anon ]
000000c046e00000   2048K rw---   [ anon ]
000000c047000000   6144K rw---   [ anon ]
000000c047600000   2048K rw---   [ anon ]
000000c047800000   4096K rw---   [ anon ]
000000c047c00000   2048K rw---   [ anon ]
000000c047e00000  12288K rw---   [ anon ]
000000c048a00000 2082816K rw---   [ anon ]
000000c0c7c00000   2048K rw---   [ anon ]
000000c0c7e00000 796672K rw---   [ anon ]
000000c0f8800000   2048K rw---   [ anon ]
000000c0f8a00000  55296K rw---   [ anon ]
000055ee746a3000   1652K r---- doh-client
000055ee74840000   3260K r-x-- doh-client
000055ee74b6f000    512K r---- doh-client
000055ee74bef000   2852K r---- doh-client
000055ee74eb8000    320K rw--- doh-client
000055ee74f08000    192K rw---   [ anon ]
000055ee757cf000    132K rw---   [ anon ]
00007f7e3f521000  68484K rw---   [ anon ]
00007f7e4380d000  35188K rw---   [ anon ]
00007f7e45a6c000  30316K rw---   [ anon ]
00007f7e4780e000   8136K rw---   [ anon ]
00007f7e48000000    132K rw---   [ anon ]
00007f7e48021000  65404K -----   [ anon ]
00007f7e4c000000    132K rw---   [ anon ]
00007f7e4c021000  65404K -----   [ anon ]
00007f7e50000000    132K rw---   [ anon ]
00007f7e50021000  65404K -----   [ anon ]
00007f7e54000000    132K rw---   [ anon ]
00007f7e54021000  65404K -----   [ anon ]
00007f7e58000000    132K rw---   [ anon ]
00007f7e58021000  65404K -----   [ anon ]
00007f7e5c000000    132K rw---   [ anon ]
00007f7e5c021000  65404K -----   [ anon ]
00007f7e60008000   9932K rw---   [ anon ]
00007f7e609c5000   9228K rw---   [ anon ]
00007f7e612c9000   5320K rw---   [ anon ]
00007f7e617fb000      4K -----   [ anon ]
00007f7e617fc000   8192K rw---   [ anon ]
00007f7e61ffc000      4K -----   [ anon ]
00007f7e61ffd000   8192K rw---   [ anon ]
00007f7e627fd000      4K -----   [ anon ]
00007f7e627fe000   8192K rw---   [ anon ]
00007f7e62ffe000      4K -----   [ anon ]
00007f7e62fff000   8192K rw---   [ anon ]
00007f7e637ff000      4K -----   [ anon ]
00007f7e63800000   8192K rw---   [ anon ]
00007f7e64000000    132K rw---   [ anon ]
00007f7e64021000  65404K -----   [ anon ]
00007f7e68000000    132K rw---   [ anon ]
00007f7e68021000  65404K -----   [ anon ]
00007f7e6c000000    132K rw---   [ anon ]
00007f7e6c021000  65404K -----   [ anon ]
00007f7e70000000    132K rw---   [ anon ]
00007f7e70021000  65404K -----   [ anon ]
00007f7e74000000    132K rw---   [ anon ]
00007f7e74021000  65404K -----   [ anon ]
00007f7e78000000    132K rw---   [ anon ]
00007f7e78021000  65404K -----   [ anon ]
00007f7e7c000000    132K rw---   [ anon ]
00007f7e7c021000  65404K -----   [ anon ]
00007f7e8000d000   1728K rw---   [ anon ]
00007f7e801c3000   3332K rw---   [ anon ]
00007f7e80508000   3012K rw---   [ anon ]
00007f7e807f9000      4K -----   [ anon ]
00007f7e807fa000   8192K rw---   [ anon ]
00007f7e80ffa000      4K -----   [ anon ]
00007f7e80ffb000   8192K rw---   [ anon ]
00007f7e817fb000      4K -----   [ anon ]
00007f7e817fc000   8192K rw---   [ anon ]
00007f7e81ffc000      4K -----   [ anon ]
00007f7e81ffd000   8192K rw---   [ anon ]
00007f7e827fd000      4K -----   [ anon ]
00007f7e827fe000   8192K rw---   [ anon ]
00007f7e82ffe000      4K -----   [ anon ]
00007f7e82fff000   8192K rw---   [ anon ]
00007f7e837ff000      4K -----   [ anon ]
00007f7e83800000   8192K rw---   [ anon ]
00007f7e84000000    132K rw---   [ anon ]
00007f7e84021000  65404K -----   [ anon ]
00007f7e88000000    132K rw---   [ anon ]
00007f7e88021000  65404K -----   [ anon ]
00007f7e8c000000    132K rw---   [ anon ]
00007f7e8c021000  65404K -----   [ anon ]
00007f7e90000000    132K rw---   [ anon ]
00007f7e90021000  65404K -----   [ anon ]
00007f7e94000000    132K rw---   [ anon ]
00007f7e94021000  65404K -----   [ anon ]
00007f7e98000000    132K rw---   [ anon ]
00007f7e98021000  65404K -----   [ anon ]
00007f7e9c000000    132K rw---   [ anon ]
00007f7e9c021000  65404K -----   [ anon ]
00007f7ea0001000    384K rw---   [ anon ]
00007f7ea0069000   5124K rw---   [ anon ]
00007f7ea056a000     16K r---- libresolv-2.31.so
00007f7ea056e000     52K r-x-- libresolv-2.31.so
00007f7ea057b000     12K r---- libresolv-2.31.so
00007f7ea057e000      4K ----- libresolv-2.31.so
00007f7ea057f000      4K r---- libresolv-2.31.so
00007f7ea0580000      4K rw--- libresolv-2.31.so
00007f7ea0581000      8K rw---   [ anon ]
00007f7ea0583000     28K r---- libnss_resolve.so.2
00007f7ea058a000    200K r-x-- libnss_resolve.so.2
00007f7ea05bc000     68K r---- libnss_resolve.so.2
00007f7ea05cd000      4K ----- libnss_resolve.so.2
00007f7ea05ce000     12K r---- libnss_resolve.so.2
00007f7ea05d1000      4K rw--- libnss_resolve.so.2
00007f7ea05d2000      4K rw---   [ anon ]
00007f7ea05d3000     12K r---- libnss_myhostname.so.2
00007f7ea05d6000     40K r-x-- libnss_myhostname.so.2
00007f7ea05e0000     32K r---- libnss_myhostname.so.2
00007f7ea05e8000     12K r---- libnss_myhostname.so.2
00007f7ea05eb000      4K rw--- libnss_myhostname.so.2
00007f7ea05ec000     28K r---- libnss_mymachines.so.2
00007f7ea05f3000    204K r-x-- libnss_mymachines.so.2
00007f7ea0626000     68K r---- libnss_mymachines.so.2
00007f7ea0637000      4K ----- libnss_mymachines.so.2
00007f7ea0638000     12K r---- libnss_mymachines.so.2
00007f7ea063b000      4K rw--- libnss_mymachines.so.2
00007f7ea063c000      4K rw---   [ anon ]
00007f7ea063d000     12K r---- libnss_files-2.31.so
00007f7ea0640000     28K r-x-- libnss_files-2.31.so
00007f7ea0647000      8K r---- libnss_files-2.31.so
00007f7ea0649000      4K r---- libnss_files-2.31.so
00007f7ea064a000      4K rw--- libnss_files-2.31.so
00007f7ea064b000    280K rw---   [ anon ]
00007f7ea069a000    256K rw---   [ anon ]
00007f7ea06da000      4K -----   [ anon ]
00007f7ea06db000   8192K rw---   [ anon ]
00007f7ea0edb000      4K -----   [ anon ]
00007f7ea0edc000   8448K rw---   [ anon ]
00007f7ea171c000      4K -----   [ anon ]
00007f7ea171d000   8448K rw---   [ anon ]
00007f7ea1f5d000      4K -----   [ anon ]
00007f7ea1f5e000   8192K rw---   [ anon ]
00007f7ea275e000      4K -----   [ anon ]
00007f7ea275f000   8448K rw---   [ anon ]
00007f7ea2f9f000      4K -----   [ anon ]
00007f7ea2fa0000   8192K rw---   [ anon ]
00007f7ea37a0000      4K -----   [ anon ]
00007f7ea37a1000   8192K rw---   [ anon ]
00007f7ea3fa1000      4K -----   [ anon ]
00007f7ea3fa2000  44228K rw---   [ anon ]
00007f7ea6ad3000 263680K -----   [ anon ]
00007f7eb6c53000      8K rw---   [ anon ]
00007f7eb6c55000 293560K -----   [ anon ]
00007f7ec8b03000      4K rw---   [ anon ]
00007f7ec8b04000  36692K -----   [ anon ]
00007f7ecaed9000      4K rw---   [ anon ]
00007f7ecaeda000   4580K -----   [ anon ]
00007f7ecb353000      4K rw---   [ anon ]
00007f7ecb354000    508K -----   [ anon ]
00007f7ecb3d3000    140K rw---   [ anon ]
00007f7ecb3f6000    148K r---- libc-2.31.so
00007f7ecb41b000   1328K r-x-- libc-2.31.so
00007f7ecb567000    300K r---- libc-2.31.so
00007f7ecb5b2000     12K r---- libc-2.31.so
00007f7ecb5b5000     12K rw--- libc-2.31.so
00007f7ecb5b8000     16K rw---   [ anon ]
00007f7ecb5bc000     28K r---- libpthread-2.31.so
00007f7ecb5c3000     64K r-x-- libpthread-2.31.so
00007f7ecb5d3000     20K r---- libpthread-2.31.so
00007f7ecb5d8000      4K r---- libpthread-2.31.so
00007f7ecb5d9000      4K rw--- libpthread-2.31.so
00007f7ecb5da000     24K rw---   [ anon ]
00007f7ecb5e0000      8K r---- libnss_dns-2.31.so
00007f7ecb5e2000     16K r-x-- libnss_dns-2.31.so
00007f7ecb5e6000      4K r---- libnss_dns-2.31.so
00007f7ecb5e7000      4K r---- libnss_dns-2.31.so
00007f7ecb5e8000      4K rw--- libnss_dns-2.31.so
00007f7ecb5e9000    256K rw---   [ anon ]
00007f7ecb629000      8K r---- ld-2.31.so
00007f7ecb62b000    128K r-x-- ld-2.31.so
00007f7ecb64b000     32K r---- ld-2.31.so
00007f7ecb654000      4K r---- ld-2.31.so
00007f7ecb655000      4K rw--- ld-2.31.so
00007f7ecb656000      4K rw---   [ anon ]
00007ffe1e804000    132K rw---   [ stack ]
00007ffe1e93b000     12K r----   [ anon ]
00007ffe1e93e000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total          6432516K

The log of doh-client exits abnormally

……
Mar 26 03:29:36 localhost doh-client[78140]: 2020/03/26 03:29:35 Get "https://jcdns.fun/dns-query?ct=application/dns-message&dns=AAABEAABAAAAAAABCWlkZXJlY2VzcwJjZgAALwABAAApAgAAAIAAABwACgAYRKLfWfFbztgBAAAAXnurZTKZzQnFoEZ1": write tcp 192.168.0.102:50464->178.62.214.105:443: write: connection timed out
Mar 26 03:29:36 localhost doh-client[78140]: 2020/03/26 03:29:35 Get "https://jcdns.fun/dns-query?ct=application/dns-message&dns=AAABEAABAAAAAAABCmZhbmdlcWlhbmcDY29tAAAvAAEAACkEgAAAgAAAHAAKABiYEf2UnpS0m6dBjxVee6uC7xfPqeRbI00": write tcp 192.168.0.102:50464->178.62.214.105:443: write: connection timed out
Mar 26 03:29:36 localhost doh-client[78140]: 2020/03/26 03:29:35 Get "https://jcdns.fun/dns-query?ct=application/dns-message&dns=AAABEAABAAAAAAABEnRpYmV0YW5jb21tdW5pdHl1awNuZXQAAC8AAQAAKQIAAACAAAAcAAoAGESi31nxW87YAQAAAF57q2Uymc0JxaBGdQ": write tcp 192.168.0.102:50464->178.62.214.105:443: write: connection timed out
Mar 26 03:29:36 localhost doh-client[78140]: panic: runtime error: racy use of timers
Mar 26 03:29:36 localhost doh-client[78140]: goroutine 1265166 [running]:
Mar 26 03:29:36 localhost doh-client[78140]: time.stopTimer(0xc044a6d278, 0x0)
Mar 26 03:29:36 localhost doh-client[78140]:         /usr/lib/go/src/runtime/time.go:224 +0x2d
Mar 26 03:29:36 localhost doh-client[78140]: time.(*Timer).Stop(...)
Mar 26 03:29:36 localhost doh-client[78140]:         /usr/lib/go/src/time/sleep.go:78
Mar 26 03:29:36 localhost doh-client[78140]: golang.org/x/net/http2.(*ClientConn).roundTrip(0xc01e378480, 0xc0025d7d00, 0x0, 0x0, 0x0, 0x0)
Mar 26 03:29:36 localhost doh-client[78140]:         /build/dns-over-https/src/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:968 +0xa1
Mar 26 03:29:36 localhost doh-client[78140]: golang.org/x/net/http2.(*Transport).RoundTripOpt(0xc000022780, 0xc0025d7d00, 0x56281cf7d400, 0xc00017d2c0, 0xc0025dc000, 0x5)
Mar 26 03:29:36 localhost doh-client[78140]:         /build/dns-over-https/src/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:447 +0x171
Mar 26 03:29:36 localhost doh-client[78140]: golang.org/x/net/http2.(*Transport).RoundTrip(...)
Mar 26 03:29:36 localhost doh-client[78140]:         /build/dns-over-https/src/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:408
Mar 26 03:29:36 localhost doh-client[78140]: golang.org/x/net/http2.noDialH2RoundTripper.RoundTrip(0xc000022780, 0xc0025d7d00, 0x56281cff3100, 0xc000022780, 0x0)
Mar 26 03:29:36 localhost doh-client[78140]:         /build/dns-over-https/src/go/pkg/mod/golang.org/x/[email protected]/http2/transport.go:2589 +0x40
Mar 26 03:29:36 localhost doh-client[78140]: net/http.(*Transport).roundTrip(0xc0001308c0, 0xc0025d7d00, 0xc020c99960, 0xc009858688, 0x56281cb95a8a)
Mar 26 03:29:36 localhost doh-client[78140]:         /usr/lib/go/src/net/http/transport.go:515 +0xd96
Mar 26 03:29:36 localhost doh-client[78140]: net/http.(*Transport).RoundTrip(0xc0001308c0, 0xc0025d7d00, 0xc0001308c0, 0x0, 0x0)
Mar 26 03:29:36 localhost doh-client[78140]:         /usr/lib/go/src/net/http/roundtrip.go:17 +0x37
Mar 26 03:29:36 localhost doh-client[78140]: net/http.send(0xc0025d7d00, 0x56281cff2860, 0xc0001308c0, 0x0, 0x0, 0x0, 0xc00a9bf9f8, 0x203000, 0x1, 0x0)
Mar 26 03:29:36 localhost doh-client[78140]:         /usr/lib/go/src/net/http/client.go:252 +0x440
Mar 26 03:29:36 localhost doh-client[78140]: net/http.(*Client).send(0xc00017d320, 0xc0025d7d00, 0x0, 0x0, 0x0, 0xc00a9bf9f8, 0x0, 0x1, 0xc0025d7c00)
Mar 26 03:29:36 localhost doh-client[78140]:         /usr/lib/go/src/net/http/client.go:176 +0xfc
Mar 26 03:29:36 localhost doh-client[78140]: net/http.(*Client).do(0xc00017d320, 0xc0025d7d00, 0x0, 0x0, 0x0)
Mar 26 03:29:36 localhost doh-client[78140]:         /usr/lib/go/src/net/http/client.go:699 +0x44c
Mar 26 03:29:36 localhost doh-client[78140]: net/http.(*Client).Do(...)
Mar 26 03:29:36 localhost doh-client[78140]:         /usr/lib/go/src/net/http/client.go:567
Mar 26 03:29:36 localhost doh-client[78140]: main.(*Client).generateRequestIETF(0xc00007bee0, 0x56281cff8940, 0xc013532cc0, 0x56281cffc460, 0xc013532c60, 0xc01460d7a0, 0x0, 0xc00002b300, 0xe)
Mar 26 03:29:36 localhost doh-client[78140]:         doh-client/ietf.go:138 +0x98a
Mar 26 03:29:36 localhost doh-client[78140]: main.(*Client).handlerFunc(0xc00007bee0, 0x56281cffc460, 0xc013532c60, 0xc01460d7a0, 0xc014c23000)
Mar 26 03:29:36 localhost doh-client[78140]:         doh-client/client.go:378 +0x40d
Mar 26 03:29:36 localhost doh-client[78140]: main.(*Client).udpHandlerFunc(...)
Mar 26 03:29:36 localhost doh-client[78140]:         doh-client/client.go:443
Mar 26 03:29:36 localhost doh-client[78140]: github.com/miekg/dns.HandlerFunc.ServeDNS(0xc0001821b0, 0x56281cffc460, 0xc013532c60, 0xc01460d7a0)
Mar 26 03:29:36 localhost doh-client[78140]:         /build/dns-over-https/src/go/pkg/mod/github.com/miekg/[email protected]/server.go:37 +0x46
Mar 26 03:29:36 localhost doh-client[78140]: github.com/miekg/dns.(*Server).serveDNS(0xc000119d40, 0xc000ccd000, 0x45, 0x1000, 0xc013532c60)
Mar 26 03:29:36 localhost doh-client[78140]:         /build/dns-over-https/src/go/pkg/mod/github.com/miekg/[email protected]/server.go:609 +0x2e2
Mar 26 03:29:36 localhost doh-client[78140]: github.com/miekg/dns.(*Server).serveUDPPacket(0xc000119d40, 0xc00028e040, 0xc000ccd000, 0x45, 0x1000, 0xc00029c000, 0xc021188f00)
Mar 26 03:29:36 localhost doh-client[78140]:         /build/dns-over-https/src/go/pkg/mod/github.com/miekg/[email protected]/server.go:549 +0xb4
Mar 26 03:29:36 localhost doh-client[78140]: created by github.com/miekg/dns.(*Server).serveUDP
Mar 26 03:29:36 localhost doh-client[78140]:         /build/dns-over-https/src/go/pkg/mod/github.com/miekg/[email protected]/server.go:479 +0x28c
Mar 26 03:29:36 localhost systemd[1]: doh-client.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Mar 26 03:29:36 localhost systemd[1]: doh-client.service: Failed with result 'exit-code'.
Mar 26 03:29:39 localhost systemd[1]: doh-client.service: Scheduled restart job, restart counter is at 2.
Mar 26 03:29:39 localhost systemd[1]: Stopped DNS-over-HTTPS Client.
Mar 26 03:29:39 localhost systemd[1]: Started DNS-over-HTTPS Client.
Mar 26 03:30:42 localhost doh-client[155656]: 2020/03/26 03:30:42 Get "https://dns.dns-over-https.com/dns-query?ct=application/dns-message&dns=AAABEAABAAAAAAABCXNlcnZlYmVlcgNjb20AAC8AAQAAKQIAAACAAAAcAAoAGESi31nxW87Y0wD45157q4kpiH4F_0KoRw": read tcp 192.168.0.102:48584->104.24.123.53:443: read: connection reset by peer
Mar 26 03:30:42 localhost doh-client[155656]: 2020/03/26 03:30:42 Get "https://dns.dns-over-https.com/dns-query?ct=application/dns-message&dns=AAABEAABAAAAAAABBGFsNGEDY29tAAAvAAEAACkCAAAAgAAAHAAKABhEot9Z8VvO2NMA-Odee6uJKYh-Bf9CqEc": read tcp 192.168.0.102:48584->104.24.123.53:443: read: connection reset by peer
……

Reproduction process

Note: The domain name doh.bgme.me in the doh-client2server.conf file has been resolved locally and the corresponding nginx has been set. Please modify it during testing.

./doh-client -conf ./doh-client2server.conf &> /dev/null &
./doh-server -conf ./doh-server.conf  &> /dev/null &
  • Use top to view doh-client, doh-server resource usage.
top -p $(ps -ef | grep doh-server | sed 2d | awk '{print $2}') -p $(ps -ef | grep doh-client | sed 2d | awk '{print $2}')
  • Use the send_nsec_dns_pkgs.py script for testing

Uncomment the corresponding line in the last part of send_nsec_dns_pkgs.py according to the required frequency of sending packets, and then run the script to send the packets.

sudo python send_nsec_dns_pkgs.py

Set inter to 0.1, 0.01 , 0.001, and 0, and the result is as follows:
send_nsec_pkgs

As the packet sending frequency increases, the memory usage of doh-client also increases.
When the test was over, the memory usage of doh-client dropped extremely slowly and still maintained a high memory usage.

Use wireshark to compare the packet sending frequency bewteen BIND and test script.
It can seen that the actual packet sending frequency of BIND is much higher than our test scipt (set inter to 0).

Test script
send_scipt_dns

BIND
dns_nsec_full
dns_nsec_full_12940

golang 1.10 required?

The documentation states that you need golang 1.9 as a minimum, but attempting to build fails with:

/home/you/gopath/src/github.com/miekg/dns/msg_helpers.go:271:8: undefined: strings.Builder
/home/you/gopath/src/github.com/miekg/dns/serve_mux.go:43:9: undefined: strings.Builder

A quick google shows that strings.Builder was added in golang 1.10. The only thing this issue probably needs is just to update the documentation to read 1.10.

dnscrypt tag

Hi,

And congrats for this great project making DNS-over-HTTPS more accessible.

Since this project is unrelated to DNSCrypt, and in order to avoid confusion, would it be possible to remove the "dnscrypt" label present in the Github description?

Thanks!

new issue for random mode

I am using this program. But I think I found a bug. It seems to be a bug related to a random query patched in version 2.0.0.

bug log:

172.19.0.240:54867 - - [23/Mar/2019:15:43:31 +0000] "BLUR FOR DOMAIN. IN TXT"
panic: invalid argument to Intn
goroutine 22 [running]:
math/rand.(*Rand).Intn(0xc00008a180, 0x0, 0x13)
	/usr/local/go/src/math/rand/rand.go:169 +0x9c math/rand.Intn(...)
	/usr/local/go/src/math/rand/rand.go:329 github.com/m13253/dns-over-https/doh-client/selector.(*RandomSelector).Get(0xc0000931e0, 0xc00009cc20)
	/tmp/dns-over-https-2.0.0/doh-client/selector/randomSelector.go:45 +0x3f main.(*Client).handlerFunc(0xc0000955f0, 0x85b660, 0xc0000a05a0, 0xc0000fa090, 0xc00004be00)
	/tmp/dns-over-https-2.0.0/doh-client/client.go:365 +0x322 main.(*Client).udpHandlerFunc(...)
	/tmp/dns-over-https-2.0.0/doh-client/client.go:443 github.com/miekg/dns.HandlerFunc.ServeDNS(0xc000114490, 0x85b660, 0xc0000a05a0, 0xc0000fa090)
	/go/pkg/mod/github.com/miekg/[email protected]/server.go:37 +0x44 github.com/miekg/dns.(*Server).serveDNS(0xc000112500, 0xc000142000, 0x36, 0x1000, 0xc0000a05a0)
	/go/pkg/mod/github.com/miekg/[email protected]/server.go:603 +0x2b8 github.com/miekg/dns.(*Server).serveUDPPacket(0xc000112500, 0xc000028020, 0xc000142000, 0x36, 0x1000, 0xc000010010, 0xc000093280)
	/go/pkg/mod/github.com/miekg/[email protected]/server.go:549 +0xb2 created by github.com/miekg/dns.(*Server).serveUDP
	/go/pkg/mod/github.com/miekg/[email protected]/server.go:479 +0x27a

I will wait for a quick reply.

Tag release 2.2.2

I saw there is a new release 2.2.2 but it has no tag so I only found out about it by accident. It would be great when releases would be tagged so they can properly discovered by GitHub.

Weird logs from doh-server

Hello,

Since latest release doh-server is making a lot of writings to syslog like below, despite verbose=false. Not very informative - at least for me. Can it be turned off?

Looks like number stations to me ;-)

Oct 28 08:15:08 warrior doh-server[24450]: [0 4 3 4]
Oct 28 08:15:17 warrior doh-server[24450]: [0 4 3 4]
Oct 28 08:15:23 warrior doh-server[24450]: [0 4 3 4]
Oct 28 08:15:38 warrior doh-server[24450]: [0 4 3 4]
Oct 28 08:15:53 warrior doh-server[24450]: [0 4 3 4]
Oct 28 08:16:00 warrior doh-server[24450]: [0 4 3 4]
Oct 28 08:16:08 warrior doh-server[24450]: [0 4 3 4]
Oct 28 08:16:10 warrior doh-server[24450]: [0 4 3 4]
Oct 28 08:16:24 warrior doh-server[24450]: [0 4 3 4]
Oct 28 08:16:39 warrior doh-server[24450]: [0 4 3 4]
Oct 28 08:16:54 warrior doh-server[24450]: [0 4 3 4]
Oct 28 08:17:09 warrior doh-server[24450]: [0 4 3 4]

Choose different upstream DNS by a domain list

Hi,

Thanks for creating such an amazing project!

I'm currently using it to establish a pollution-free DOH server as well as a proxy server together. I realised that for PRC domains, querying DNSPod with X-Forwarded-For that contains the client IP can get the best result. But for other domains, the best option is to query Google DNS with a specific IP (the IP of the proxy server) or omit the ECS entirely.

It would be really nice to have such a list filtering function.

X-Forwarded-For or X-Real-IP is not forwarding client ip to backend

After setting the mentioned header in the webserver 'nginx', the backend still receives DNS queries from nginx IP and not from the client IP.
Even in the logs, I only see the IP of the web server and not the client.
Is it bugged or I'm missing something here?

from nginx config:
...
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $remote_addr;
...
proxy_pass http://ungound-dns;

from DOH config:
...
log_guessed_client_ip = true

make error when build doh-client & doh-server

error messages:

sail@zqbi:~/go/src/github.com/m13253/dns-over-https$ make
go get -d -u -v github.com/m13253/dns-over-https/json-dns
github.com/m13253/dns-over-https (download)
github.com/miekg/dns (download)
go get -d -v ./doh-client ./doh-server
cd doh-client && go build
# crypto/rc4
/usr/local/go/src/crypto/rc4/rc4_asm.go:13:18: (*Cipher).XORKeyStream redeclared in this block
	previous declaration at /usr/local/go/src/crypto/rc4/rc4.go:61:6
# runtime
/usr/local/go/src/runtime/map.go:65:2: bucketCntBits redeclared in this block
	previous declaration at /usr/local/go/src/runtime/hashmap.go:64:18
/usr/local/go/src/runtime/map.go:66:2: bucketCnt redeclared in this block
	previous declaration at /usr/local/go/src/runtime/hashmap.go:65:23
/usr/local/go/src/runtime/map.go:70:2: loadFactorNum redeclared in this block
	previous declaration at /usr/local/go/src/runtime/hashmap.go:69:18
/usr/local/go/src/runtime/map.go:71:2: loadFactorDen redeclared in this block
	previous declaration at /usr/local/go/src/runtime/hashmap.go:70:18
/usr/local/go/src/runtime/map.go:77:2: maxKeySize redeclared in this block
	previous declaration at /usr/local/go/src/runtime/hashmap.go:76:17
/usr/local/go/src/runtime/map.go:78:2: maxValueSize redeclared in this block
	previous declaration at /usr/local/go/src/runtime/hashmap.go:77:17
/usr/local/go/src/runtime/map.go:83:2: dataOffset redeclared in this block
	previous declaration at /usr/local/go/src/runtime/hashmap.go:85:4
/usr/local/go/src/runtime/map.go:94:2: evacuatedX redeclared in this block
	previous declaration at /usr/local/go/src/runtime/hashmap.go:93:19
/usr/local/go/src/runtime/map.go:95:2: evacuatedY redeclared in this block
	previous declaration at /usr/local/go/src/runtime/hashmap.go:94:19
/usr/local/go/src/runtime/map.go:96:2: evacuatedEmpty redeclared in this block
	previous declaration at /usr/local/go/src/runtime/hashmap.go:92:19
/usr/local/go/src/runtime/map.go:96:2: too many errors
Makefile:66: recipe for target 'doh-client/doh-client' failed
make: *** [doh-client/doh-client] Error 2

My golang environment:

sail@zqbi:~/go/src/github.com/m13253/dns-over-https$ go version
go version go1.12 linux/amd64
sail@zqbi:~/go/src/github.com/m13253/dns-over-https$ go env
GOARCH="amd64"
GOBIN="/home/sail/go/bin"
GOCACHE="/home/sail/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/sail/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build290594187=/tmp/go-build -gno-record-gcc-switches"

My OS:

sail@zqbi:~/go/src/github.com/m13253/dns-over-https$ uname -a
Linux zqbi 4.4.0-142-generic #168-Ubuntu SMP Wed Jan 16 21:00:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
sail@zqbi:~/go/src/github.com/m13253/dns-over-https$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.6 LTS
Release:	16.04
Codename:	xenial

Use of loopback address as upstream DNS server fails

Hi @m13253 ,

I have noticed that using loopback address as DNS upstream server does not work with current version of doh-server (2.2.4).

DOH-Server Config:

072e6a06e523:/server# cat doh-server.conf
listen = [ ":8053" ]
local_addr = ""
cert = ""
key = ""
path = "/getnsrecord"
upstream = [ "udp:127.0.0.1:53" ]
timeout = 10
tries = 3
verbose = true
log_guessed_client_ip = false

Curl Test Logs:

072e6a06e523:/# curl -I 'http://localhost:8053/getnsrecord?name=google.com&type=A'
HTTP/1.1 503 Service Unavailable
Access-Control-Allow-Headers: Content-Type
Access-Control-Allow-Methods: GET, HEAD, OPTIONS, POST
Access-Control-Allow-Origin: *
Access-Control-Max-Age: 3600
Content-Type: application/json; charset=UTF-8
Server: DNS-over-HTTPS/2.2.4 (+https://github.com/m13253/dns-over-https)
X-Powered-By: DNS-over-HTTPS/2.2.4 (+https://github.com/m13253/dns-over-https)
Date: Sat, 26 Dec 2020 05:41:50 GMT
Content-Length: 114

DOH-Server console logs:

072e6a06e523:/server#  /server/doh-server -conf /server/doh-server.conf
2020/12/26 05:41:50 DNS error from upstream udp:127.0.0.1:53: read udp 127.0.0.1:53131->127.0.0.1:53: read: connection refused
2020/12/26 05:41:50 DNS error from upstream udp:127.0.0.1:53: read udp 127.0.0.1:55126->127.0.0.1:53: read: connection refused
2020/12/26 05:41:50 DNS error from upstream udp:127.0.0.1:53: read udp 127.0.0.1:55747->127.0.0.1:53: read: connection refused
127.0.0.1 - - [26/Dec/2020:05:41:50 +0000] "HEAD /getnsrecord?name=google.com&type=A HTTP/1.1" 503 114 "" "curl/7.64.0"

Simple nslookup test using loopback address was successful:

072e6a06e523:/server# nslookup google.com 127.0.0.1
Server:    127.0.0.1
Address 1: 127.0.0.1 localhost

Name:      google.com
Address 1: 172.217.14.78
Address 2: 2607:f8b0:4005:807::200e

Please let me know if this is expected behavior or there is something that needs to be done to allow use of loopback address as upstream DNS server in DOH configuration.

Steps to reproduce

  1. Run docker command to launch doh-server container
docker run -itd \
  --name doh-server \
  -e UPSTREAM_DNS_SERVER='udp:127.0.0.1:53' \
  -e DEBUG=1 \
  -e DOH_SERVER_VERBOSE=true \
  satishweb/doh-server \
sleep 36000
  1. Start doh-server inside the container in console mode
docker exec -it doh-server /server/doh-server -conf /server/doh-server.conf
  1. Add curl and execute curl test command
docker exec -it doh-server apk add curl
docker exec -it doh-server curl -I 'localhost:8053/getnsrecord?name=google.com&type=A'
  1. To open shell: docker exec -it doh-server bash
  2. Cleanup: docker rm -f doh-server

Fedora/CentOS package

I've created an RPM package for CentOS (should work on Fedora as well) + SELinux policy.
Are you interested in this?

Does dns-over-https have cache?

On my server, I use dnsmasq to cache to DNS resolve result, I want to use doh directly, I searched cache in this repo and it seems there are not codes about cache the result.

So I wonder if dns-over-https has the feature about cache the result.

Provide fallback semantics for upstream server groups

Currently, you can specify more than one upstream server in the google or ietf groups in doh-client.conf. The semantics for choosing one to query are currently random choice. It would be useful to provide an option to configure doh-client to use the upstream servers as an ordered list with fallback semantics. Something like a {google,ietf}_list_semantics = (random,fallback) configuration option would do. An additional fallback_timeout would be great as well, since it would be useful to have it set fairly low, unlike the overall timeout.

The use case for this is that I prefer using cloudflare over google (I distrust them less with query data than google), but I've had hiccups and would like google to be used automagically when that happens.

Ideally, this would include marking upstream servers as down when multiple queries time out in a short window, but in the absence of a full-blown solution like that, the fallback+timeout config options would suffice.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.