Hey Ryan,

I've read your blog post about any-proxy, but I'm quite not sure, if I can achieve my goal with your software.
Maybe you can help me out.

I have a server in a MZ which needs to send emails via SMTP and receive emails via IMAP. I have a satellite server in my DMZ which is able to talk to the correct MX server.

Can I simply forward any incoming traffic from the needed ports to my upstream server?

Cheers,
Matthias

Linux 3.5 provides a facility called tcp connection repair which could allow any_proxy to run on two machines (A and B) in a active/passive high availability situation. If this works the way I think it does, you would run VRRP (keepalived) on both machines, which provides auto failover, but does not provide a way to replicate socket state of running services. any_proxy on Machine A would replicate socket state to any_proxy Machine B, and if a failover occurred, Machine B would take over proxying without users noticing (no connections would be dropped).

See http://criu.org/TCP_connection

Hi!

I tried to use any_proxy in combination with wget (just for testing purpose), to redirect a simple HTTPS web request, but it always results in this (data anonymized):

2022/08/08 19:04:45 any_proxy.go:584: : DEBUG : Enter handleProxyConnection: clientConn=&{conn:{fd:0xc00008b000}} (*net.TCPConn)
2022/08/08 19:04:45 any_proxy.go:115: : DEBUG : lookup(): CACHE_MISS
2022/08/08 19:04:45 any_proxy.go:623: : DEBUG : PROXY|10.XX.XX.250:52728->10.XX.XX.15:8080->pd9535add.dip0.dest.adr.net.:443|Connected to proxy
2022/08/08 19:04:45 any_proxy.go:630: : DEBUG : SNI-PARSING|10.XX.XX.250:52728 via 10.XX.XX.15:8080 for req.url.com on destination pd9535add.dip0.dest.adr.net.:443
2022/08/08 19:04:45 any_proxy.go:637: : DEBUG : PROXY|10.XX.XX.250:52728->10.XX.XX.15:8080->pd9535add.dip0.dest.adr.net.:443|Sending to proxy: "CONNECT req.url.com:443 HTTP/1.1\r\nProxy-Authorization: Basic eWkwBLABLABLABLAODEwNg==\r\nX-Forwarded-For: 10.XX.XX.250\r\n\r\n"
2022/08/08 19:04:45 any_proxy.go:644: : DEBUG : PROXY|10.XX.XX.250:52728->10.XX.XX.15:8080->pd9535add.dip0.dest.adr.net.:443|Received from proxy: "HTTP/1.1 403 URLBlocked\r\n"
2022/08/08 19:04:45 any_proxy.go:665: : INFO : PROXY|10.XX.XX.250:52728->10.XX.XX.15:8080->pd9535add.dip0.dest.adr.net.:443|ERR: Proxy response to CONNECT was: "HTTP/1.1 403 URLBlocked\r\n". Trying next proxy.
2022/08/08 19:04:45 any_proxy.go:680: : INFO : PROXY|10.XX.XX.250:52728->UNAVAILABLE->pd9535add.dip0.dest.adr.net.:443|ERR: Tried all proxies, but could not establish connection. Giving up.

If I simply set the https_proxy environment variable and do not use any_proxy, everything works as expected - but that is not possible with the program I want to use any_proxy for... as I said, wget is for testing purpose only.

I am wondering what any_proxy is doing differently.

Regards,
Holger

Build a load test to simulate a clientConn that gets closed at random time intervals.
Do the same for proxyConn

We are trying to make sure that our testing of Conn!=nil is sufficient.

-incoming req/sec
-since start
-1min avg
-5min avg
-15min avg

-proxied req/sec
-since start
-1min avg
-5min avg
-15min avg

-direct req/sec
-since start
-1min avg
-5min avg
-15min avg

As suggested by Dave Cheney on gonuts, change net.Dial calls to net.DialTCP calls to avoid casting.

We will have to do name resolution manually beforehand, as DialTCP expects addresses, not strings.

Reference:
https://groups.google.com/forum/#!topic/golang-nuts/5FGZwAvpxSc

hi.

root@ar:~/go/gopath# go get -u -v github.com/ryanchapman/go-any-proxy
github.com/ryanchapman/go-any-proxy (download)
github.com/namsral/flag (download)
github.com/zdannar/flogger (download)
github.com/namsral/flag
github.com/zdannar/flogger
github.com/ryanchapman/go-any-proxy

github.com/ryanchapman/go-any-proxy

src/github.com/ryanchapman/go-any-proxy/any_proxy.go:200:51: undefined: BUILDTIMESTAMP
src/github.com/ryanchapman/go-any-proxy/any_proxy.go:201:28: undefined: BUILDTIMESTAMP
src/github.com/ryanchapman/go-any-proxy/any_proxy.go:202:91: undefined: BUILDUSER
src/github.com/ryanchapman/go-any-proxy/any_proxy.go:202:102: undefined: BUILDHOST
root@ar:/go/gopath# which any_proxy
root@ar:/go/gopath#

I want to monitor the traffic that the 'dnf' update tool generates on Fedora. My understanding is that if I have go-any-proxy running to intercept traffic on port 80 (which I've modified dnf to use), then go-any-proxy will receive the dnf traffic and then send it on the the actual destination repo servers. However, when I run ./any-proxy -l :80, I see the following messages in the error log:

2018/06/21 20:40:38 any_proxy.go:475: : INFO : GETORIGINALDST|[::1]:46350->?->FAILEDTOBEDETERMINED|ERR: getsocketopt(SO_ORIGINAL_DST) failed: protocol not available
2018/06/21 20:40:38 any_proxy.go:681: : INFO : handleConnection(): can not handle this connection, error occurred in getting original destination ip address/port: protocol not available

Is my understanding of what any_proxy can do correct? If not, any suggestions for doing what I described?

Thanks!
Jon Forrest

Not capable of HTTP (non TLS) requests? If so can this feature be added somehow?

panic:
concurrent map iteration and map write

type reverseLookupCache struct {
  hostnames map[string]*cacheEntry
  keys []string
  next int
  mu   sync.Mutex
}

lookup and store function add

c.mu.Lock()
defer c.mu.Unlock()

I'm forcing lots of data through this proxy.

Running dedicated hardware.
Dell 710 Xeon 5650 24gb ram

I'm getting hard crashes and a stack trace
As a workaround we've been rebooting the service every 5min.

Here's a small subset of the stack trace.

2016/03/07 16:30:05 any_proxy.go:509: : INFO : dial(): ERR: could not connect to 10.0.7.214:443: dial tcp 10.0.7.214:443: no route to host
2016/03/07 16:30:05 any_proxy.go:509: : INFO : dial(): ERR: could not connect to 10.0.7.214:443: dial tcp 10.0.7.214:443: no route to host
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x404b09]

goroutine 6 [running]:
runtime.panic(0x526ce0, 0x6ab6a8)
        /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
main.handleDirectConnection(0xc2100002c0, 0xc2100445a0, 0xa, 0x1bb)
        /home/ubuntu/anyproxy/go-any-proxy/any_proxy.go:533 +0x459
main.handleConnection(0xc2100002c0)
        /home/ubuntu/anyproxy/go-any-proxy/any_proxy.go:670 +0x27c
created by main.main
        /home/ubuntu/anyproxy/go-any-proxy/any_proxy.go:349 +0x452

goroutine 1 [IO wait]:
net.runtime_pollWait(0x7fde6ec7d340, 0x72, 0x0)
        /usr/lib/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc210050370, 0x72, 0x7fde6ec7c098, 0xb)
        /usr/lib/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc210050370, 0xb, 0x7fde6ec7c098)
        /usr/lib/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).accept(0xc210050310, 0x5950f0, 0x0, 0x7fde6ec7c098, 0xb)
        /usr/lib/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(*TCPListener).AcceptTCP(0xc210000138, 0xc210000230, 0x0, 0x0)
        /usr/lib/go/src/pkg/net/tcpsock_posix.go:233 +0x47
main.main()
        /home/ubuntu/anyproxy/go-any-proxy/any_proxy.go:342 +0x35d

goroutine 3 [syscall]:
os/signal.loop()
        /usr/lib/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init<C2><B7>1
        /usr/lib/go/src/pkg/os/signal/signal_unix.go:27 +0x31

goroutine 4 [chan receive]:
main.func<C2><B7>006()
        /home/ubuntu/anyproxy/go-any-proxy/stats.go:259 +0x4d
created by main.setupStats
        /home/ubuntu/anyproxy/go-any-proxy/stats.go:290 +0x101

goroutine 5 [running]:
        goroutine running on other thread; stack unavailable
created by main.main
        /home/ubuntu/anyproxy/go-any-proxy/any_proxy.go:349 +0x452
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x404b09]

goroutine 5 [running]:
runtime.panic(0x526ce0, 0x6ab6a8)
        /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
main.handleDirectConnection(0xc2100001f0, 0xc210044500, 0xa, 0x1bb)
        /home/ubuntu/anyproxy/go-any-proxy/any_proxy.go:533 +0x459
main.handleConnection(0xc2100001f0)
        /home/ubuntu/anyproxy/go-any-proxy/any_proxy.go:670 +0x27c
created by main.main
        /home/ubuntu/anyproxy/go-any-proxy/any_proxy.go:349 +0x452

2021/07/29 18:10:56 any_proxy.go:705: : INFO : handleConnection(): can not handle this connection, error occurred in getting original destination ip address/port: protocol not $
2021/07/29 18:10:56 any_proxy.go:485: : INFO : GETORIGINALDST|MyIpHere:36895->?->FAILEDTOBEDETERMINED|ERR: getsocketopt(SO_ORIGINAL_DST) failed: protocol not available

I tried to do minecraft proxy setup :/

About every 12hrs or so we'll get a disconnect from our proxy server.
x.x.x.x is our proxy server

2016/03/13 23:50:02 any_proxy.go:375: : INFO : Added proxy server x.x.x.x:80
2016/03/13 23:50:02 any_proxy.go:339: : INFO : Listening for connections on [::]:3129
2016/03/13 23:55:02 any_proxy.go:509: : INFO : dial(): ERR: could not connect to x.x.x.x:80: dial tcp x.x.x.x:80: connection refused
2016/03/13 23:55:02 any_proxy.go:368: : INFO : Test connection to x.x.x.x:80: failed. Removing from proxy server list
2016/03/13 23:55:02 any_proxy.go:380: : INFO : None of the proxy servers specified are available. 
Exiting.
None of the proxy servers specified are available. 
Exiting.
2016/03/14 00:00:02 any_proxy.go:375: : INFO : Added proxy server x.x.x.x:80
2016/03/14 00:00:02 any_proxy.go:339: : INFO : Listening for connections on [::]:3129
2016/03/14 00:05:02 any_proxy.go:375: : INFO : Added proxy server x.x.x.x:80
2016/03/14 00:05:02 any_proxy.go:339: : INFO : Listening for connections on [::]:3129

Again I can't thank you enough for this project. With the latest fix this really has been able to change how we do a lot of different things here for the better. This is not a huge issue, just thought you should know.

The result of following the installation instructions on impish:

any_proxy.go:47:5: no required module provides package github.com/namsral/flag: go.mod file not found in current directory or any parent directory; see 'go help modules'
any_proxy.go:50:5: no required module provides package github.com/zdannar/flogger: go.mod file not found in current directory or any parent directory; see 'go help modules'

The new LTS on this code base is just around the corner

Although we don't want to create a situation where we are decrypting the communications that flow through any proxy, users have requested a way to make proxy allow/deny decisions based on the SSL certificate, which we read during the initial handshake.

Additionally, if possible, add a command line option that will reconnect to the CN in the cert.

Users requesting this: 2 (send an email to Ryan if you'd would also like to see this implemented)

Provide a way to make routing decisions based on hostname.

One way I think might work:

any_proxy answers DNS queries.
Client directs all DNS traffic to any_proxy.
When DNS query comes in, any_proxy looks at ruleset and determines whether the connection should be:

1. denied (ACL deny)
2. allowed; if allowed, should the connection be sent to an upstream proxy or sent directly to the destination host
   The (client, destination, port, disposition) tuple is placed in an internal routing table
   Client attempts to connect to host
   Linux iptables redirects attempt to any_proxy
   any_proxy consults it's internal routing table to determine how to route the request

My code is at:
http://paste.ngtech.co.il/plseujxih

It's basically a sketch rather then full implementation since it doesn't have the tproxy dial bind option.
I have seen a tproxy outgoing connection example at:
https://github.com/andrew-d/demux/blob/master/transparent_linux.go
I will later write my own code that will use it.
Is it any of interest for you?

For now, any_proxy will use CPU at 100% if it exhausted file descriptors. When all file descriptors are used up, it is even hard to ssh to the server. I am wondering if it's possible to deny new connections or just restart itself, instead of infinite retrying, which at least remains the server operatable.

By the way, I love any_proxy a lot. It solves my real problems. Thank you!

It currently uses eth2 (why, I don't know). It should instead create an iptables rule for all interfaces.

Log the version of go in use on startup to aid in debugging

Any Proxy seems to be a good piece for the following puzzle.

In a datacenter there is a server S with a high-speed connection directly to the internet. Somewhere else is my notebook N with several UMTS/LTE usb modems connected and running (i.e. through each modem I can access the internet). On N and on S is a Linux kernel supporting MPTCP. I would like to stream all TCP traffic (that includes also all higher layers like HTTP, HTTPS, FTP, etc.) and if possible also DNS traffic (but that's just a nice-to-have as it could be solved using some tricks like DNS with multihomed-routing) between S and N (of course both ways).

Naive possible solution: To leverage MPTCP, we need something like a peer-to-peer connection (e.g. any tunnel like VPN or SOCKS or just some proxy protocol - e.g. the one squid uses) and then a corresponding configuration on S and N which directs all data through the "p2p tunnel".

What everything do you think would be needed for this use case from the any_proxy point of view?

When the program crashes, write the stack trace to the log and also append the stats to the log file so we can determine what load was at time of crash