GithubHelp home page GithubHelp logo

nfx / slrp Goto Github PK

View Code? Open in Web Editor NEW
150.0 3.0 23.0 3.76 MB

rotating open proxy multiplexer

License: MIT License

Makefile 0.25% Go 87.19% HTML 0.11% CSS 0.80% Yacc 0.48% Dockerfile 0.14% TypeScript 11.03%
crawler proxy-pool golang proxy proxy-checker proxy-list proxy-server

slrp's Introduction

SLRP - rotating open proxy multiplexer

slrp logo

codecov lines downloads

  • Searches for proxies in open sources
  • Intelligently stores state on disk across restarts
  • Validates via configurable speed thresholds and anonymity
  • Multiplexes HTTP/HTTPS MITM to HTTP, HTTPS, SOCKS4, and SOCKS5
  • Exposes REST API for refresh stats and pool health
  • Exposes minimal Query Language for filtering of History and Proxy Stats.
  • Records request history in-memory for further UI inspection
  • Real-time statistics display about available pool
  • Packaged as a single executable binary, that also includes Web UI

Usage

Download service, start it up, wait couple of minutes for the pool to pick up. Now run curl --proxy-insecure -D - -x http://127.0.0.1:8090 -k http://httpbin.org/get couple of times and see different origins and user agent headers.

Concepts

  • Source is an async process that looks at one or more pages for refreshed proxy list.
  • Refresher component does best effort on scheduling items.
  • Some sources perform better forwarded through a Pool, warming it up.
  • One proxy may be seen in multiple sources, so we keep exclusive proxies per source across refreshes, which are not found in other sources.
  • Proxy consists of protocol (HTTP, HTTPS, SOCKS4, or SOCKS5) and IP:PORT.
  • Proxy becomes Scheduled immediately after it's seen in the source.
  • Scheduled could transition into Probing queue if it's not Ignored (e.g. Timeouts or Blacklist).
  • Probing uses configurable pool of rotating anonymity checkers to check for liveliness.
  • Timeout items are re-added to Scheduled queue as Reverify source to probe item up to 5 times.
  • Blacklist hosts historical faulty proxies that should never be probed again.
  • Successful check results in Found queue and gets added to a Pool.
  • Pool subdivides its memory into shards for randomized rotation and minimal resource contention.
  • Pool uses configurable and backpressure-controlled workers to perform HTTP request forwarding.
  • Every forwarded request gets a serial number (returned in X-Proxy-Serial header) and picks a different shard for an attempt, which is reflected in response in X-Proxy-Attempt header.
  • Every forwarded request can later be inspected through GET /api/history or UI.
  • Every attempt picks first available working random proxy from a shard and marks it as Offered. Total number of offers per used proxy is returned in response in X-Proxy-Offered header.
  • In the event of no working proxies in a shard, proxy pool exhaustion errors can do backpressure and slow down issuing of serial numbers through simple leaky bucket algorithm.
  • Every succeeded attempt through a proxy increases it's Success Rate (Succeeded/Offered), which is also calculated per hour. Total number of succeded attempts of used proxy are returned via X-Proxy-Succeed header. Proxy used is returned in X-Proxy-Through header.
  • Every failed attempt marks proxy as not working and suspends offering it for 5 minutes.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

User Interface

Overview

overview

http://localhost:8089/ shows current source refresh status and stats.

Proxies

proxies

http://localhost:8089/proxies provides search interface over active pool of found proxies. By default, entries are sorted by last working on top. Query samples:

History

history

http://localhost:8089/history provides search interface over the last 1000 forwarding attempts (configurable). Sample queries:

Reverify

http://localhost:8089/reverify provides search interface over timed out probes. Sample queries:

Blacklist

http://localhost:8089/blacklist provides search interface over unsuccessful probes. Sample queries:

Configuration

Conf file is looked in the following paths:

  1. $PWD/slrp.yml
  2. $PWD/config.yml
  3. $HOME/.slrp/config.yml

Default configuration is approximately the following:

app:
  state: $HOME/.slrp/data
  sync: 1m
log:
  level: info
  format: pretty
server:
  addr: "localhost:8089"
  read_timeout: 15s
mitm:
  addr: "localhost:8090"
  read_timeout: 15s
  idle_timeout: 15s
  write_timeout: 15s
pprof:
  enable: false
  addr: "localhost:6060"
checker:
  timeout: 5s
  strategy: simple
history:
  limit: 1000

Every configuration property can be overridden through environment variable by using SLRP_ prefix followed by section name and key, divided by _. For example, in order to set log level to trace, do SLRP_LOG_LEVEL=TRACE slrp.

app

Fabric that holds application components together.

  • state - where data persists on disk through restarts of the application. Default is .slrp/data of your home directory.
  • sync - how often data is synchronised to disk, pending availability of any updates of component state. Default is every minute.

dialer

WireGuard userspace VPN dialer configuration. Embeds the official Go implementation. Disabled by default.

  • wireguard_config_file - configuration file from WireGuard. IPv6 address parsing is ignored at the moment.
  • wireguard_verbose - verbose logging mode for WireGuard tunnel.

Sample WireGuard configuration file:

[Interface]
PrivateKey = gI6EdUSYvn8ugXOt8QQD6Yc+JyiZxIhp3GInSWRfWGE=
Address = 1.2.3.4/24
DNS =  1.2.3.4

[Peer]
PublicKey = HIgo9xNzJMWLKASShiTqIybxZ0U3wGLiUeJ1PKf8ykw=
Endpoint = 1.2.3.4:51820
AllowedIPs = 0.0.0.0/0

log

Structured logging meta-components.

  • level - log level of application. Default is info. Possible values are trace, debug, info, warn, and error.
  • format - format of log lines printed. Default is pretty, though it's recommended for exploratory use only for performance reasons. Possible values are pretty, json, and file (experimental). file will create a $PWD/slrp.log, unless specified by log.file property.
  • file (experimental) - application logs in JSON format. Default value is $PWD/slrp.log.

server

API and UI serving component.

  • addr - address of listening HTTP server. Default is http://127.0.0.1:8089.
  • read_timeout - default is 15s.
  • enable_profiler - either or not enabling profiler endpoints. Default is false. Developer use only.

pool

Proxy pool maintenance.

  • request_workers - number of workers to perform outgoing HTTP requests. Defaults to 512.
  • request_timeout - outgoing HTTP request timeout. defaults to 10s.
  • shards - number of shards. Defaults to 1. This property may go away.
  • evict_span_minutes - number of minutes to identify the latest span of time for rolling counters. Defaults to 5.
  • short_timeout_sleep - time to remove a proxy from routing after the first timeout or error.
  • long_timeout_sleep - time to remove a proxy from routing after evict_threshold_timeouts within the last evict_span_minutes.
  • evict_threshold_timeouts - used with long_timeout_sleep. Defaults to 3.
  • evict_threshold_failures - number of failures within the last evict_span_minutes to evict proxy from the pool.
  • evict_threshold_reanimations - number of any proxy sleeps ever to evict proxy from the pool.

probe

Proxy probing component.

  • enable_http_rescue - experimental feature to enable rescuing HTTP proxies, that were presented as SOCKS5 or HTTPS. Detected based on protocol probe heuristics. Defaults to false.

refresher

Source refresh component.

  • enabled - run the refresher. Enabled by default.
  • max_scheduled - number of sources to refresh at the same time. Defaults to 5.

mitm

HTTP proxy frontend.

  • addr - address of listening HTTP proxy server. Default is http://127.0.0.1:8090.
  • read_timeout - default is 15s.
  • idle_timeout - default is 15s.
  • write_timeout - default is 15s.

checker

Component for verification of proxy liveliness and anonymity.

history

Component for recording forwarded requests through a pool of proxies.

  • limit - number of requests to keep in memory. Default is 1000.

ipinfo

You can optionally enable this feature. This product includes GeoLite2 Data created by MaxMind, available from https://www.maxmind.com.

  • license - your (free) license key for MaxMind downloads. You can skip specifying license key if mmdb_asn and mmdb_city are already downloaded in any other way and configured.
  • mmdb_asn - already (or automatically) downloaded snapshots of MaxMind database. Default is $HOME/.slrp/maxmind/GeoLite2-ASN.mmdb
  • mmdb_city - already (or automatically) downloaded snapshots of MaxMind database. Default is $HOME/.slrp/maxmind/GeoLite2-City.mmdb

API

GET /api

Retrieve last sync status for all components

GET /api/dashboard

Get information about refresh status for all sources

GET /api/pool

Get 20 last used proxies

POST /api/refresher/{source_name}

Start refreshing the source

DELETE /api/refresher/{source_name}

Stop refreshing the source

GET /api/history

Get 100 last forwarding attempts

GET /api/history/{id}

Get sanitized HTTP response from forwarding attempt

GET /api/reverify

Get first 20 timed out items that are in the reverify pool

GET /api/blacklist

Get first 20 blacklisted items sorted by proxy along with common error stats

Developing

UI development requires npm installed. Once you have it, please npm install vite typescript -g.

References

  • ProxyBroker is pretty similar project in nature. Requires couple of Python module dependencies and had the last commit in March 2019.
  • Scylla is pretty similar project in nature. Requires couple of Python module dependencies.
  • ProxyBuilder

Star History

Star History Chart

slrp's People

Contributors

ar-kan avatar dependabot[bot] avatar doomedraven avatar nfx avatar rc5hack avatar snyk-bot avatar sriramkeerthi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

slrp's Issues

Change how stats are shown

Add a more visual stats component.

The big numbers on proxies page was quite fine.

Add graphs to show statistics.

curl shows "Unsupported HTTP version (0.0) in response" on HTTP 403 responses

Test command:

curl -SsfL -D - --proxy-insecure -k -x "https://127.0.0.1:8090" "https://httpbin.org/status/403"

This will produce the following error:

HTTP/1.1 200 OK
Date: Tue, 27 Feb 2024 01:53:48 GMT
Transfer-Encoding: chunked

curl: (1) Unsupported HTTP version (0.0) in response

Btw. most requests through the proxy end up with this error message.
It's quite rare that a response gets back from the targeted URL.

If I run curl with trace enabled, I don't see any HTTP response from the targeted server at all.
See the attached curl_trace_ascii.txt for the output of the following command:

curl --trace-ascii - --proxy-insecure -k -x "https://127.0.0.1:8090" "https://httpbin.org/status/403" > curl_trace_ascii.txt 2>&1

Docker Compose set-up ?

Can I please request for a docker compose set up example as part of the documentation / readme ?

Can't run v0.3.0 binary in Docker: `panic: As4 called on IPv6 address`

Seems like some ipv6/ipv4 backward compatibility problem:

$ md5sum ./slrp
d5176f385f243708b347063e46f5c99c  ./slrp

$ cat Dockerfile
ARG ALPINE_VERSION=3.18
FROM alpine:${ALPINE_VERSION} AS runner
RUN apk add --no-cache ca-certificates tzdata
ENV PWD="/app"
ENV SLRP_APP_STATE="$PWD/.slrp/data"
ENV SLRP_APP_SYNC="1m"
ENV SLRP_LOG_LEVEL="info"
ENV SLRP_LOG_FORMAT="pretty"
ENV SLRP_SERVER_ADDR="0.0.0.0:8089"
ENV SLRP_SERVER_READ_TIMEOUT="15s"
ENV SLRP_MITM_ADDR="0.0.0.0:8090"
ENV SLRP_MITM_READ_TIMEOUT="15s"
ENV SLRP_MITM_IDLE_TIMEOUT="15s"
ENV SLRP_MITM_WRITE_TIMEOUT="15s"
ENV SLRP_CHECKER_TIMEOUT="5s"
ENV SLRP_CHECKER_STRATEGY="simple"
ENV SLRP_HISTORY_LIMIT="1000"
WORKDIR $PWD
RUN mkdir ./.slrp
COPY ./slrp $PWD
EXPOSE 8089 8090
CMD ["./slrp"]

$ docker build -t myslrp .
$ docker run myslrp
slrp v0.3.0
5:40AM WRN using clear dialer
5:40AM INF configured proxy checker ip=<my IPv4 address was here> strategy=simple timeout=5
panic: As4 called on IPv6 address

goroutine 1 [running]:
net/netip.Addr.As4({{0xc0001e4240?, 0xeca77d?}, 0xc000010090?})
        /opt/hostedtoolcache/go/1.20.6/x64/src/net/netip/netip.go:692 +0x8e
github.com/nfx/slrp/pmux.NewProxy({0xc0000b61c4?, 0x0?}, {0xeca77d, 0x5})
        /home/runner/work/slrp/slrp/pmux/proxy.go:259 +0x90
github.com/nfx/slrp/pmux.HttpsProxy(...)
        /home/runner/work/slrp/slrp/pmux/proxy.go:276
github.com/nfx/slrp/serve.(*HttpsProxyServer).Proxy(0x18803c0?)
        /home/runner/work/slrp/slrp/serve/https_proxy.go:43 +0x2c
github.com/nfx/slrp/serve.(*MitmProxyServer).Configure(0xc000474240, 0xc0003f7e60?)
        /home/runner/work/slrp/slrp/serve/mitm_proxy.go:45 +0x13c
github.com/nfx/slrp/app.(*Fabric).configureServices(0xc000410680)
        /home/runner/work/slrp/slrp/app/fabric.go:224 +0xdc
github.com/nfx/slrp/app.(*Fabric).Start(0xc000410680, {0x1100600?, 0xc000046038})
        /home/runner/work/slrp/slrp/app/fabric.go:107 +0x765
github.com/nfx/slrp/app.Run(...)
        /home/runner/work/slrp/slrp/app/fabric.go:58
main.main()
        /home/runner/work/slrp/slrp/main.go:34 +0x5d8

Invalid memory address or nil pointer dereference

Tested slrp_0.1.3_linux_amd64.tar.gz on two different Linux machines (Ubuntu 20.04.5 LTS and Linux Mint 21), and after running for a few minutes it crashed with a NULL pointer reference issue:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x8b86e9]

goroutine 38597 [running]:
github.com/bdandy/go-socks4.socks4.Dial.func1()
	/home/runner/work/slrp/slrp/vendor/github.com/bdandy/go-socks4/socks4.go:73 +0x29
github.com/bdandy/go-socks4.socks4.Dial({0xc0026ceab0?, {0xf1baa0?, 0x15359b0?}}, {0xc2a6d3?, 0x90?}, {0xc001de3350, 0x16})
	/home/runner/work/slrp/slrp/vendor/github.com/bdandy/go-socks4/socks4.go:85 +0x114d
github.com/nfx/slrp/pmux.dialProxiedConnection({0xf1f8b0, 0xc0024c9560}, {0xc2a6d3, 0x3}, {0xc001de3350, 0x16})
	/home/runner/work/slrp/slrp/pmux/proxy.go:175 +0x177
net/http.(*Transport).customDialTLS(0x0?, {0xf1f8b0?, 0xc0024c9560?}, {0xc2a6d3?, 0xf1f040?}, {0xc001de3350?, 0xc00271b358?})
	/opt/hostedtoolcache/go/1.18.8/x64/src/net/http/transport.go:1317 +0x50
net/http.(*Transport).dialConn(0xc00032e8c0, {0xf1f8b0, 0xc0024c9560}, {{}, 0x0, {0xc3f9a5, 0x5}, {0xc001de3350, 0x16}, 0x0})
	/opt/hostedtoolcache/go/1.18.8/x64/src/net/http/transport.go:1583 +0x3ff
net/http.(*Transport).dialConnFor(0xf19980?, 0xc002a10a50)
	/opt/hostedtoolcache/go/1.18.8/x64/src/net/http/transport.go:1449 +0xb0
created by net/http.(*Transport).queueForDial
	/opt/hostedtoolcache/go/1.18.8/x64/src/net/http/transport.go:1418 +0x3d2

Configuration from PWD/config.yml read, but not applied

Platform info

Hello. I'm using

OS:

$ cat /etc/lsb-release /etc/upstream-release/lsb-release 
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=18.1
DISTRIB_CODENAME=serena
DISTRIB_DESCRIPTION="Linux Mint 18.1 Serena"
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04 LTS"

Kernel:

$ uname -a
Linux linux 4.4.0-53-generic #74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Binary from: https://github.com/nfx/slrp/releases/download/v0.0.7/slrp_0.0.7_linux_amd64.tar.gz

Expected behaviour

  1. I place the following configuration (taken from README.md) in $PWD/config.yml:
$ cat config.yml 
app:
    state: $HOME/.slrp/data
    sync: 1m
log:
    level: warn
    format: pretty
server:
    addr: "localhost:8089"
    read_timeout: 15s
    enable_profiler: false
mitm:
    addr: "0.0.0.0:8090"
    read_timeout: 15s
    idle_timeout: 15s
    write_timeout: 15s
checker:
    timeout: 5s
    strategy: simple
history:
    limit: 1000

Log level and listen address are changed here.

  1. Configuration is read and applied.

Observed behaviour

  1. Configuration is being ignored by app.
$ strace -o debug -y -yy -yyy -s 2000 -vf ./slrp
slrp v0.0.7
6:39PM INF loaded service=probe
6:39PM INF loaded service=pool
6:39PM INF loaded service=stats
6:39PM INF starting service=server
6:39PM INF starting service=mitm
6:39PM INF started refresh source=checkerproxy.net
6:39PM INF Loading proxy checker database
6:39PM INF started refresh source=proxylists.net
^C
$ ag -v getdent debug | ag '\.yml|8090'
1619:29035 openat(AT_FDCWD, "/home/user/tmp/slrp/slrp.yml", O_RDONLY|O_CLOEXEC <unfinished ...>
1623:29035 openat(AT_FDCWD, "/home/user/tmp/slrp/config.yml", O_RDONLY|O_CLOEXEC <unfinished ...>
1625:29035 <... openat resumed> )            = 9</home/user/tmp/slrp/config.yml>
1626:29035 epoll_ctl(4<anon_inode:[eventpoll]>, EPOLL_CTL_ADD, 9</home/user/tmp/slrp/config.yml>, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2547521208, u64=140654136530616}}) = -1 EPERM (Operation not permitted)
1627:29035 fstat(9</home/user/tmp/slrp/config.yml>, {st_dev=makedev(8, 18), st_ino=8674353, st_mode=S_IFREG|0644, st_nlink=1, st_uid=1000, st_gid=1000, st_blksize=4096, st_blocks=8, st_size=339, st_atime=2022/08/19-18:38:41, st_mtime=2022/08/19-18:32:44, st_ctime=2022/08/19-18:38:16}) = 0
1628:29035 read(9</home/user/tmp/slrp/config.yml>, "app:\n    state: $HOME/.slrp/data\n    sync: 1m\nlog:\n    level: warn\n    format: pretty\nserver:\n    addr: \"localhost:8089\"\n    read_timeout: 15s\n    enable_profiler: false\nmitm:\n    addr: \"0.0.0.0:8090\"\n    read_timeout: 15s\n    idle_timeout: 15s\n    write_timeout: 15s\nchecker:\n    timeout: 5s\n    strategy: simple\nhistory:\n    limit: 1000\n", 512) = 339
1629:29035 read(9</home/user/tmp/slrp/config.yml>, "", 173) = 0
1630:29035 close(9</home/user/tmp/slrp/config.yml>) = 0
1631:29035 openat(AT_FDCWD, "/home/user/.slrp/config.yml", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
3535:29044 bind(3<TCP:[43385928]>, {sa_family=AF_INET, sin_port=htons(8090), sin_addr=inet_addr("127.0.0.1")}, 16 <unfinished ...>
3543:29044 epoll_ctl(4<anon_inode:[eventpoll]>, EPOLL_CTL_ADD, 3<TCP:[127.0.0.1:8090]>, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=2547521208, u64=140654136530616}} <unfinished ...>
3547:29044 getsockname(3<TCP:[127.0.0.1:8090]>,  <unfinished ...>
3549:29044 <... getsockname resumed> {sa_family=AF_INET, sin_port=htons(8090), sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
3851:29041 accept4(3<TCP:[127.0.0.1:8090]>,  <unfinished ...>

From the trace it's obvivous that

  1. App successfully finds the config.
  2. App successfully reads the config.
  3. But it got ignored: it start listening at default IP address, not one included in config.

When the same config file is moved to ~/.slrp/ it work like expected.

$ mv config.yml ~/.slrp/

$ strace -o debug -y -yy -yyy -s 2000 -vf ./slrp 
slrp v0.0.7
^C

$ ag -v getdent debug | ag '8090'
1443:2412  read(9</home/user/.slrp/config.yml>, "app:\n    state: $HOME/.slrp/data\n    sync: 1m\nlog:\n    level: warn\n    format: pretty\nserver:\n    addr: \"localhost:8089\"\n    read_timeout: 15s\n    enable_profiler: false\nmitm:\n    addr: \"0.0.0.0:8090\"\n    read_timeout: 15s\n    idle_timeout: 15s\n    write_timeout: 15s\nchecker:\n    timeout: 5s\n    strategy: simple\nhistory:\n    limit: 1000\n", 512) = 339
3617:2422  bind(3<TCPv6:[43449405]>, {sa_family=AF_INET6, sin6_port=htons(8090), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28 <unfinished ...>
3624:2422  epoll_ctl(4<anon_inode:[eventpoll]>, EPOLL_CTL_ADD, 3<TCPv6:[:::8090]>, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=4270608472, u64=140230657855576}} <unfinished ...>
3627:2422  getsockname(3<TCPv6:[:::8090]>,  <unfinished ...>
3629:2422  <... getsockname resumed> {sa_family=AF_INET6, sin6_port=htons(8090), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
3869:2422  accept4(3<TCPv6:[:::8090]>,  <unfinished ...>
8090:2412  read(111<TCP:[192.168.1.230:45866->103.117.192.14:80]>,  <unfinished ...>

Can't build

package github.com/nfx/slrp
	imports github.com/nfx/slrp/dialer
	imports golang.zx2c4.com/wireguard/tun/netstack
	imports gvisor.dev/gvisor/pkg/bufferv2
	imports gvisor.dev/gvisor/pkg/atomicbitops
	imports gvisor.dev/gvisor/pkg/state
	imports gvisor.dev/gvisor/pkg/state/wire
	imports gvisor.dev/gvisor/pkg/gohacks: build constraints exclude all Go files in /home/.../go/pkg/mod/gvisor.dev/[email protected]/pkg/gohacks

go version go1.22.0 linux/amd64

New sources

Sources to be added:

Obsolete:

Curl command in ReadMe no longer works

The curl command in the readme no longer works and I am not sure how to make it work with a self signed cert. I keep getting the following error when running the command:

curl -D - -k -x https://127.0.0.1:8090 -k http://httpbin.org/ip

curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

Downgrading from 0.3.0 fixed it.

Empty reply from server

root@sayrix1:~# curl -D - -x http://127.0.0.1:8090 -k http://httpbin.org/get
curl: (52) Empty reply from server
root@sayrix1:~#

Bug : All the proxies are put in Bucket 0.

The bug is basically stated in the title. I didn't have time to investigate further, but I the Proxy which is always represented as a uint64 is always divisible by 32. Probably using a prime number of shards will fix the error (e.g. 33 instead of 32).

Not listening on 0.0.0.0

Platform info

OS:

$ cat /etc/lsb-release /etc/upstream-release/lsb-release 
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=18.1
DISTRIB_CODENAME=serena
DISTRIB_DESCRIPTION="Linux Mint 18.1 Serena"
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04 LTS"

Kernel:

$ uname -a
Linux linux 4.4.0-53-generic #74-Ubuntu SMP Fri Dec 2 15:59:10 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Binary from: https://github.com/nfx/slrp/releases/download/v0.0.7/slrp_0.0.7_linux_amd64.tar.gz

Expected behaviour

When addr: "0.0.0.0:8090" is specified, listen on 0.0.0.0:8090 (ipv4).

Observed behaviour

It start listening on IPv6 address and breaks

1638:5821  read(9</home/user/.slrp/config.yml>, "app:\n    state: $HOME/.slrp/data\n    sync: 1m\nlog:\n    level: warn\n    format: pretty\nserver:\n    addr: \"localhost:8089\"\n    read_timeout: 15s\n    enable_profiler: false\nmitm:\n    addr: \"0.0.0.0:8090\"\n    read_timeout: 15s\n    idle_timeout: 15s\n    write_timeout: 15s\nchecker:\n    timeout: 5s\n    strategy: simple\nhistory:\n    limit: 1000\n", 512) = 339
3555:5817  bind(9<TCPv6:[43840488]>, {sa_family=AF_INET6, sin6_port=htons(8090), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28 <unfinished ...>
3563:5817  epoll_ctl(4<anon_inode:[eventpoll]>, EPOLL_CTL_ADD, 9<TCPv6:[:::8090]>, {EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, {u32=1644746904, u64=140567334410392}} <unfinished ...>
3567:5817  getsockname(9<TCPv6:[:::8090]>,  <unfinished ...>
3569:5817  <... getsockname resumed> {sa_family=AF_INET6, sin6_port=htons(8090), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 0
3787:5816  accept4(9<TCPv6:[:::8090]>,  <unfinished ...>
$ netstat -tpln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:8089          0.0.0.0:*               LISTEN      5814/slrp       
tcp6       0      0 :::8090                 :::*                    LISTEN      5814/slrp   
2022/08/19 19:43:01 http: panic serving 192.168.1.230:40620: As4 called on IPv6 address
goroutine 8668 [running]:
net/http.(*conn).serve.func1()
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:1825 +0xbf
panic({0xb0dd60, 0xeeab60})
	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/panic.go:844 +0x258
net/netip.Addr.As4({{0x9?, 0xc038d8?}, 0xc0001a40d8?})
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/netip/netip.go:699 +0x8e
github.com/nfx/slrp/pmux.NewProxy({0xc003469800?, 0x1f4?}, {0xc038d8, 0x4})
	/home/runner/work/slrp/slrp/pmux/proxy.go:224 +0x85
github.com/nfx/slrp/pmux.HttpProxy(...)
	/home/runner/work/slrp/slrp/pmux/proxy.go:237
github.com/nfx/slrp/serve.(*HttpProxyServer).Proxy(0xef08d8?)
	/home/runner/work/slrp/slrp/serve/http_proxy.go:64 +0x45
github.com/nfx/slrp/serve.(*HttpProxyServer).handleConnect(0xc0001cc480, {0xef2110, 0xc00339a000}, 0xc000d08200)
	/home/runner/work/slrp/slrp/serve/http_proxy.go:118 +0x3a5
github.com/nfx/slrp/serve.(*HttpProxyServer).ServeHTTP(0xef2840?, {0xef2110?, 0xc00339a000?}, 0x14f8930?)
	/home/runner/work/slrp/slrp/serve/http_proxy.go:84 +0x54
github.com/nfx/slrp/serve.(*MitmProxyServer).ServeHTTP(0xc0001cc480, {0xef2110, 0xc00339a000}, 0xc000d08100)
	/home/runner/work/slrp/slrp/serve/mitm_proxy.go:50 +0x52b
net/http.serverHandler.ServeHTTP({0xc00117aa20?}, {0xef2110, 0xc00339a000}, 0xc000d08100)
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc0027a65a0, {0xef28e8, 0xc00011d0e0})
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:3071 +0x4db
2022/08/19 19:43:01 http: panic serving 192.168.1.230:40624: As4 called on IPv6 address
goroutine 8618 [running]:
net/http.(*conn).serve.func1()
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:1825 +0xbf
panic({0xb0dd60, 0xeeab60})
	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/panic.go:844 +0x258
net/netip.Addr.As4({{0x9?, 0xc038d8?}, 0xc0001a40d8?})
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/netip/netip.go:699 +0x8e
github.com/nfx/slrp/pmux.NewProxy({0xc00375f120?, 0x1f4?}, {0xc038d8, 0x4})
	/home/runner/work/slrp/slrp/pmux/proxy.go:224 +0x85
github.com/nfx/slrp/pmux.HttpProxy(...)
	/home/runner/work/slrp/slrp/pmux/proxy.go:237
github.com/nfx/slrp/serve.(*HttpProxyServer).Proxy(0xef08d8?)
	/home/runner/work/slrp/slrp/serve/http_proxy.go:64 +0x45
github.com/nfx/slrp/serve.(*HttpProxyServer).handleConnect(0xc0001cc480, {0xef2110, 0xc0000c8000}, 0xc0014a9a00)
	/home/runner/work/slrp/slrp/serve/http_proxy.go:118 +0x3a5
github.com/nfx/slrp/serve.(*HttpProxyServer).ServeHTTP(0xef2840?, {0xef2110?, 0xc0000c8000?}, 0x14f8930?)
	/home/runner/work/slrp/slrp/serve/http_proxy.go:84 +0x54
github.com/nfx/slrp/serve.(*MitmProxyServer).ServeHTTP(0xc0001cc480, {0xef2110, 0xc0000c8000}, 0xc0014a9500)
	/home/runner/work/slrp/slrp/serve/mitm_proxy.go:50 +0x52b
net/http.serverHandler.ServeHTTP({0xc00132b320?}, {0xef2110, 0xc0000c8000}, 0xc0014a9500)
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc0001ad400, {0xef28e8, 0xc00011d0e0})
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:3071 +0x4db
2022/08/19 19:43:01 http: panic serving 192.168.1.230:40626: As4 called on IPv6 address
goroutine 8669 [running]:
net/http.(*conn).serve.func1()
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:1825 +0xbf
panic({0xb0dd60, 0xeeab60})
	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/panic.go:844 +0x258
net/netip.Addr.As4({{0x9?, 0xc038d8?}, 0xc0001a40d8?})
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/netip/netip.go:699 +0x8e
github.com/nfx/slrp/pmux.NewProxy({0xc00375f140?, 0x1f4?}, {0xc038d8, 0x4})
	/home/runner/work/slrp/slrp/pmux/proxy.go:224 +0x85
github.com/nfx/slrp/pmux.HttpProxy(...)
	/home/runner/work/slrp/slrp/pmux/proxy.go:237
github.com/nfx/slrp/serve.(*HttpProxyServer).Proxy(0xef08d8?)
	/home/runner/work/slrp/slrp/serve/http_proxy.go:64 +0x45
github.com/nfx/slrp/serve.(*HttpProxyServer).handleConnect(0xc0001cc480, {0xef2110, 0xc0000c80e0}, 0xc0014a9b00)
	/home/runner/work/slrp/slrp/serve/http_proxy.go:118 +0x3a5
github.com/nfx/slrp/serve.(*HttpProxyServer).ServeHTTP(0xef2840?, {0xef2110?, 0xc0000c80e0?}, 0x14f8930?)
	/home/runner/work/slrp/slrp/serve/http_proxy.go:84 +0x54
github.com/nfx/slrp/serve.(*MitmProxyServer).ServeHTTP(0xc0001cc480, {0xef2110, 0xc0000c80e0}, 0xc0000bf900)
	/home/runner/work/slrp/slrp/serve/mitm_proxy.go:50 +0x52b
net/http.serverHandler.ServeHTTP({0xc001ead8f0?}, {0xef2110, 0xc0000c80e0}, 0xc0000bf900)
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc0027a66e0, {0xef28e8, 0xc00011d0e0})
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:3071 +0x4db
2022/08/19 19:43:01 http: panic serving 192.168.1.230:40628: As4 called on IPv6 address
goroutine 8670 [running]:
net/http.(*conn).serve.func1()
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:1825 +0xbf
panic({0xb0dd60, 0xeeab60})
	/opt/hostedtoolcache/go/1.18.5/x64/src/runtime/panic.go:844 +0x258
net/netip.Addr.As4({{0x9?, 0xc038d8?}, 0xc0001a40d8?})
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/netip/netip.go:699 +0x8e
github.com/nfx/slrp/pmux.NewProxy({0xc00375f164?, 0x1f4?}, {0xc038d8, 0x4})
	/home/runner/work/slrp/slrp/pmux/proxy.go:224 +0x85
github.com/nfx/slrp/pmux.HttpProxy(...)
	/home/runner/work/slrp/slrp/pmux/proxy.go:237
github.com/nfx/slrp/serve.(*HttpProxyServer).Proxy(0xef08d8?)
	/home/runner/work/slrp/slrp/serve/http_proxy.go:64 +0x45
github.com/nfx/slrp/serve.(*HttpProxyServer).handleConnect(0xc0001cc480, {0xef2110, 0xc0000c8460}, 0xc0014a9d00)
	/home/runner/work/slrp/slrp/serve/http_proxy.go:118 +0x3a5
github.com/nfx/slrp/serve.(*HttpProxyServer).ServeHTTP(0xef2840?, {0xef2110?, 0xc0000c8460?}, 0x14f8930?)
	/home/runner/work/slrp/slrp/serve/http_proxy.go:84 +0x54
github.com/nfx/slrp/serve.(*MitmProxyServer).ServeHTTP(0xc0001cc480, {0xef2110, 0xc0000c8460}, 0xc0014a9c00)
	/home/runner/work/slrp/slrp/serve/mitm_proxy.go:50 +0x52b
net/http.serverHandler.ServeHTTP({0xc00132bad0?}, {0xef2110, 0xc0000c8460}, 0xc0014a9c00)
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:2916 +0x43b
net/http.(*conn).serve(0xc0027a6780, {0xef28e8, 0xc00011d0e0})
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:1966 +0x5d7
created by net/http.(*Server).Serve
	/opt/hostedtoolcache/go/1.18.5/x64/src/net/http/server.go:3071 +0x4db

Panic and crash when using filtering in Web UI

Version https://github.com/nfx/slrp/releases/download/v0.0.7/slrp_0.0.7_linux_amd64.tar.gz.

Entering Session>2 or even S>2 in Search field or navigating to http://localhost:8089/history?filter=Session%3E2 crashes entire application

panic: interface conversion: interface {} is float64, not string

goroutine 27 [running]:
github.com/nfx/slrp/ql.(*ConditionOperand).eval(0xc002cdd8f0, {{0xbd8be0?, 0xc0006d0780?, 0x0?}, 0xc0041ed380?})
	/home/runner/work/slrp/slrp/ql/ast.go:234 +0x7e8
github.com/nfx/slrp/ql.(*Condition).eval(0x203000?, {{0xbd8be0?, 0xc0006d0780?, 0x0?}, 0xc0041ed380?})
	/home/runner/work/slrp/slrp/ql/ast.go:194 +0x4a
github.com/nfx/slrp/ql.(*AndCondition).eval(0x7ff378952850?, {{0xbd8be0?, 0xc0006d0780?, 0xc0?}, 0xc0041ed380?})
	/home/runner/work/slrp/slrp/ql/ast.go:173 +0xb1
github.com/nfx/slrp/ql.(*Expression).eval(0xbd6de0?, {{0xbd8be0?, 0xc0006d0780?, 0x4b3e6f?}, 0xc0041ed380?})
	/home/runner/work/slrp/slrp/ql/ast.go:45 +0xbd
github.com/nfx/slrp/ql.applyFilter[...](0xc000548118, 0xc0001cb8a8, 0xc0041ecfc0, 0x0)
	/home/runner/work/slrp/slrp/ql/ast.go:421 +0x237
github.com/nfx/slrp/ql.Execute[...](0xc000548118, 0xc0001cb8a8, {0xc0039c5930, 0x9}, 0xc004699a80, {0xc004699a90, 0x2, 0xc00})
	/home/runner/work/slrp/slrp/ql/ast.go:385 +0x273
github.com/nfx/slrp/history.(*History).handleFilter(0xc000548100, {{0xc0039c5930?, 0x6787668f00500000?}, 0xc0030c4660?})
	/home/runner/work/slrp/slrp/history/history.go:202 +0x189
github.com/nfx/slrp/history.(*History).main(0xc000548100, {0xef1c90, 0xc0002be510})
	/home/runner/work/slrp/slrp/history/history.go:192 +0x350
created by github.com/nfx/slrp/history.(*History).Start
	/home/runner/work/slrp/slrp/history/history.go:111 +0x95

I don't think that invalid search query should crash the entire app, honestly...

v 0.12 panic: runtime error: invalid memory address or nil pointer dereference

fails on both linux and macosx

tried to run as and no extra logs:
SLRP_LOG_LEVEL=DEBUG ./slrp
SLRP_LOG_LEVEL=TRACE ./slrp

10:40AM INF blacklisted error="dial socks: socks4 dial: dial tcp addr:port: connect: connection refused" idx=20 proxy=socks4://217.198.181.50:49582 source=reverify
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x14b5909]

goroutine 258489 [running]:
github.com/bdandy/go-socks4.socks4.Dial.func1()
	/home/runner/work/slrp/slrp/vendor/github.com/bdandy/go-socks4/socks4.go:73 +0x29
github.com/bdandy/go-socks4.socks4.Dial({0xc004e22fc0?, {0x1b15d60?, 0x212f428?}}, {0x1825e92?, 0x90?}, {0xc000a3e550, 0xf})
	/home/runner/work/slrp/slrp/vendor/github.com/bdandy/go-socks4/socks4.go:85 +0x114d
github.com/nfx/slrp/pmux.dialProxiedConnection({0x1b19af0, 0xc001cf6d80}, {0x1825e92, 0x3}, {0xc000a3e550, 0xf})
	/home/runner/work/slrp/slrp/pmux/proxy.go:175 +0x177
net/http.(*Transport).customDialTLS(0xc006d8cbd0?, {0x1b19af0?, 0xc001cf6d80?}, {0x1825e92?, 0xc0000960f0?}, {0xc000a3e550?, 0x0?})
	/opt/hostedtoolcache/go/1.18.8/x64/src/net/http/transport.go:1317 +0x50
net/http.(*Transport).dialConn(0xc0003ca640, {0x1b19af0, 0xc001cf6d80}, {{}, 0x0, {0x18371af, 0x5}, {0xc000a3e550, 0xf}, 0x0})
	/opt/hostedtoolcache/go/1.18.8/x64/src/net/http/transport.go:1583 +0x3ff
net/http.(*Transport).dialConnFor(0x1174e8a?, 0xc0048b6370)
	/opt/hostedtoolcache/go/1.18.8/x64/src/net/http/transport.go:1449 +0xb0
created by net/http.(*Transport).queueForDial
	/opt/hostedtoolcache/go/1.18.8/x64/src/net/http/transport.go:1418 +0x3d2

running over proxy doesn't populate node's country

Hello

If I run SLRP over proxy like command below SLRP doesn't populate country code of each node and make it impossible to filter/query per country

  • HTTP_PROXY="socks5://X:X" HTTPS_PROXY="socks5://X:X" ./slrp

update fails if $CWD is not in $PATH

./slrp -update

[================================================] 100%
panic: looking up path of "slrp": exec: "slrp": executable file not found in $PATH

goroutine 1 [running]:
github.com/nfx/slrp/internal/updater.AutoUpdate({0xf14490, 0x5})
	/home/runner/work/slrp/slrp/internal/updater/auto.go:42 +0x207
main.main()
	/home/runner/work/slrp/slrp/main.go:30 +0xa8

This fixes it
PATH=.:$PATH ./slrp -update

Generate default config at startup

Please make it so that app generate a config file with default values and place it in one of locations.

With current behavior, it's somewhat hard to realize what user needs to do in order to change the defaults and what the defaults are.

Example from README.md may be outdated and so on...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.