barnacs / compy Goto Github PK
View Code? Open in Web Editor NEWHTTP/HTTPS compression proxy
License: ISC License
HTTP/HTTPS compression proxy
License: ISC License
compy should allow recompressing WebP images with a given quality setting.
compy could serve diagnostics like its configuration, statistics, and more importantly the CA certificate needed to enable MitM support. The latter would make installing the root certificate more convenient on mobile:
https://mtersch.wordpress.com/2015/03/17/certificate-import-in-firefox-on-android/
Sometimes after heavy loads; one of child processes continues 100% loading and does not come back to normal load! (even if i'm not using it).
using latest commit, i get this error.
2017/06/09 07:50:01 http: TLS handshake error from 41.66.203.167:41837: read tcp 45.62.242.187:3130->41.66.203.167:41837: read: connection reset by peer 2017/06/09 07:50:01 http: TLS handshake error from 41.66.203.167:57673: read tcp 45.62.242.187:3130->41.66.203.167:57673: read: connection reset by peer 2017/06/09 07:50:03 http: TLS handshake error from 41.66.203.167:41127: read tcp 45.62.242.187:3130->41.66.203.167:41127: read: connection reset by peer 2017/06/09 07:50:04 http: TLS handshake error from 41.66.203.167:47938: read tcp 45.62.242.187:3130->41.66.203.167:47938: read: connection reset by peer 2017/06/09 07:50:05 http: TLS handshake error from 41.66.203.167:56411: read tcp 45.62.242.187:3130->41.66.203.167:56411: read: connection reset by peer
compy should support a --verbose
flag so that it emits HTTP requests and responses like curl does. We should also attach some kind of request id since the current logging causes confusion with concurrent requests.
This will make compy easier for non-programmers to use.
Description:
Compy can't deal with the websocket
, which fails some interaction tasks like chatrooms and online services.
Suggested fix:
Compy can simply add a condition for websocket and directly forward the data in the websocket to the end users.
Hi
I tried setting up compy to serve compressed assets locally and I'm running into issues (probably with the way I'm using it).
2020/04/20 13:45:03 compy listening on :8080
2020/04/20 13:45:03 listen tcp :8080: bind: address already in use
james@james-pc:~/Desktop/testing-compy$
OS: Ubuntu 18.04
Go version: 1.14.1
This is by no means critical, but I guess it is better to document it. If I do:
compy -cert cert.crt -key cert.key -ca ca.crt -cakey ca.key -jpeg 40 -gif false -gzip 9 -brotli 11 -minify true -host :1234
compy ignores the -host
option:
2017/12/13 00:28:23 compy listening on :9999
But if I do
compy -cert cert.crt -key cert.key -ca ca.crt -cakey ca.key -host :1234 -jpeg 40 -gif false -gzip 9 -brotli 11 -minify true
The proxy now listens on port 1234
I think the command line is not correctly parsing the command line options on some situations.
I am very interested in this project! It sounds awesome! I have no idea why it isn't more popular.
Anyways, I am unfamiliar with Go, so how I could I install this on a OpenWRT/LEDE?
How resource intensive is it? My router is has a 1.3 GHz dual-core with 512 MB of RAM.
I have generated Let's Encrypt files on a test server, where I have these .pem files to work with:
$ cat ./README
This directory contains your keys and certificates.
`privkey.pem` : the private key for your certificate.
`fullchain.pem`: the certificate file used in most server software.
`chain.pem` : used for OCSP stapling in Nginx >=1.3.7.
`cert.pem` : will break many server configurations, and should not be used
without reading further documentation (see link below).
WARNING: DO NOT MOVE OR RENAME THESE FILES!
Certbot expects these files to remain in this location in order
to function properly!
We recommend not moving these files. For more information, see the Certbot
User Guide at https://certbot.eff.org/docs/using.html#where-are-my-certificates.
Which of these are compatible/should be used with the various cert/ca/key flags or the software itself of compy
?
Presently compy sorts all its command-line flags such that related flags like cert
and key
are not adjacent. We may need to switch parsers for this.
I have installed the compy on centos 6.8. Everything I did followed the instructions, but I cannot use MitM method.
The compy was launched by command:
compy -cert cert.crt -key cert.key -ca ca.crt -cakey ca.key
and the compy continues showing the following errors:
remote error: tls: unknown certificate authority
I have added the generated ca.cert to Firefox.
Please help me to solve this problem.
Are you interested in a better more up to date and slimer Dockerfile? I am currently building one and would contribute it if you like.
How to install it?
There are various sites I have found that cannot be accessed using this proxy with unhelpful error messages such as bad request or invalid reponse. I'm not a software developer so have had no luck tracking down what the issue is. If anyone gets 5 minuted can you look at
https://www.old-maps.co.uk/
or
https://www.realvnc.com/connect/download/vnc/windows/
Thanks, Jon.
Getting the error message when trying to open some HTTPS web pages.
one example is aao25.com page does not load on client and message in compy log x509: requested SignatureAlgorithm does not match private key type.
Anyone got any ideas how to resolve?
This format can reduce file size even more than WebP:
https://en.wikipedia.org/wiki/AV1#AV1_Image_File_Format_(AVIF)
Firefox 77 already has support behind a feature flag:
I tried to set up a Compy with HTTPS and MitM support, and I connected my iPhone to it but I keep getting these two errors whenever I try to load a page. I don't understand what I did wrong but i'm hoping you can help me.
http: TLS handshake error from 192.168.1.74:49989: tls: oversized record received with length 20037
http: TLS handshake error from 192.168.1.74:49994: tls: first record does not look like a TLS handshake
Description:
Some websites compress the images for transmission, yet Compy fails to recognize it can make an error when displaying the image.
Reason:
Compy can't recognize the compressed images.
Suggested fix:
Compy may use the ``content-encoding'' field in the response header to determine the compression methods and further decompress the images before display.
Hi,
For a reason unknown to me, compy triggers Content encoding errors on the client when both the browser and the target server support brotli, and when the minifier plugin is enabled.
Known failure cases which triggers Content encoding errors are:
One solution to circumvent the issue is to mask br
when passing the Accept-encoding
headers, if the minify plugin is enabled.
The more elegant one would be to check why the content isn't encoded back in brotli when processed by the minify plugin.
Kind regards,
The newest commit seems to be not working with https connections.
Errors outputed from compy:
2016/03/28 15:06:03 http: TLS handshake error from *IP*:54255: EOF
2016/03/28 15:06:03 http: TLS handshake error from *IP*:54256: EOF
2016/03/28 15:06:03 http: TLS handshake error from *IP*:54257: EOF
2016/03/28 15:06:04 http: TLS handshake error from *IP*:54258: EOF
2016/03/28 15:06:04 http: TLS handshake error from *IP*:54259: EOF
2016/03/28 15:06:04 http: TLS handshake error from *IP*:54260: EOF
IP is hereby my locals computer IP.
With 6dae9bd the same setup was working.
Since a4a691b it no longer works.
Chrome shows ERR_SSL_PROTOCOL_ERROR
Command used for running compy:
./compy -ca ca.crt -cakey ca.key
I think compy has not any feature to empty unsued memory. For example if i do not use compy for hours or days it would take hundreds of megabytes even in idle state!
Can create any examples e.g. to intercept on MITM and do something with the payload, like refusing a forward.
Before compy I used RabbIT with ffmpeg, and convert GIF more than 30kb to WebM format. This allows you to leave GIF animated when the size is significantly reduced. Can you add similar functional to compy?
Now that compy has unit tests via #12, we should set up Travis to prevent regressions.
I saw diagnostic endpoint in #22 but can't get it to work.
I keep getting following:
$ curl https://localhost:9999/ -kv
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9999 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* server certificate verification SKIPPED
* server certificate status verification SKIPPED
* common name: localhost (matched)
* server certificate expiration date OK
* server certificate activation date OK
* certificate public key: RSA
* certificate version: #3
* subject: CN=localhost
* start date: Wed, 18 Apr 2018 18:20:01 GMT
* expire date: Sat, 15 Apr 2028 18:20:01 GMT
* issuer: CN=localhost
* compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: localhost:9999
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 500 Internal Server Error
< Date: Wed, 18 Apr 2018 18:20:37 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
and in logs:
2018/04/18 18:20:37 serving request: /
2018/04/18 18:20:37 http: TLS handshake error from 127.0.0.1:47946: tls: first record does not look like a TLS handshake
2018/04/18 18:20:37 error forwarding request: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02" while serving request: http://localhost:9999/
I'm using docker version of compy without any special config.
One weird thing I've noticed is "while serving request: http://localhost:9999/" in logs even though I'm hitting proxy itself as https (because it works in HTTPS proxy mode).
Configuring the SSL certificates can confuse users. The README or wiki should have detailed instructions for how to establish both SSL trust to compy and the CA trust for MitM.
Some servers serve JPEG images with content type image/jpg
instead of image/jpeg
(the e
).
The transcoder should catch these as well (compy.go):
if *jpeg != 0 {
tcjpeg := tc.NewJpeg(*jpeg)
p.AddTranscoder("image/jpeg", tcjpeg)
p.AddTranscoder("image/jpg", tcjpeg)
}
@barnacs do you agree and if so do you want a pull-request for this change?
I tried to install compy to my Raspberry Pi 3B with Rasbian Stretch and Go 1.10.1 but got this error:
pi@BM-Raspi-Server:~ $ go get github.com/barnacs/compy
# github.com/pixiv/go-libjpeg/jpeg
go/src/github.com/pixiv/go-libjpeg/jpeg/compress.go:6:21: fatal error: jpeglib.h: No such file or directory
#include "jpeglib.h"
^
compilation terminated.
How to fix it?
Also tried to install libjpeg9 and libjpeg-turbo but neither works.
If I run compy with mitm support and supply username and password, on my remote computer browser and phone, whenever I try to access a site, a dialog pops up asking for username and password and then errors out. It say ERR_UNEXPECTED_PROXY_AUTH.
Note: authentication only works with http sites and not https
Hi there,
I would be super excited to test the project but I'm having real trouble to set it up.
What I did:
I get following log on the Docker side:
http2: server: error reading preface from client 37.167.90.155:49700: remote error: tls: bad certificate
Is there something obvious that I'm missing or what extra information can I provide you in order that you could help me?
I was testing Compy on some http websites and I am getting some strange numbers for the "amount transcoded" report. Example:
2017/10/03 07:53:27 transcoded: 53 -> 77 (145.3%)
2017/10/03 07:53:27 transcoded: 61 -> 74 (121.3%)
2017/10/03 07:52:56 transcoded: 61 -> 74 (121.3%)
2017/10/03 07:52:56 transcoded: 69 -> 93 (134.8%)
2017/10/03 07:52:28 transcoded: 0 -> 23 (+Inf%)
@gaul mention:
Very small HTML and images can increase in size due to overhead of compression metadata and possibly different formats have different overhead, e.g., GIF vs. WebP.
Perhaps a conditional statement could be included, such as "if x is this small or smaller then bypass transcoding/compression, else transcode/compress"?
Hi,
I'm trying to use compy with TLS and MitM support: (on Windows Server 2012 R2)
> compy -cert cert.crt -key cert.key -ca ca.crt -cakey ca.key
and:
not using mitm: crypto/x509: system root pool is not available on Windows
2018/03/04 08:01:11 compy listening on :9999
Compiler: go1.10.windows-amd64 & MinGW_x86_64-7.2.0-release-posix-seh-rt_v5-rev1
Hi,
I tried installing using the latest version of Go on a Raspberry Pi. The instructions now require "go install [url]@Version", which fails at:
github.com/chai2010/webp
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:22:9: undefined: webpGetInfo
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:26:20: undefined: webpDecodeGray
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:39:20: undefined: webpDecodeRGB
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:52:20: undefined: webpDecodeRGBA
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:68:14: undefined: webpDecodeGrayToSize
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:82:14: undefined: webpDecodeRGBToSize
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:96:14: undefined: webpDecodeRGBAToSize
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:109:7: undefined: toGrayImage
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:110:14: undefined: webpEncodeGray
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:119:14: undefined: webpEncodeRGB
go/pkg/mod/github.com/chai2010/[email protected]/webp.go:119:14: too many errors
github.com/pixiv/go-libjpeg/jpeg: build constraints exclude all Go files in /root/go/pkg/mod/github.com/pixiv/[email protected]/jpeg
gopkg.in/kothar/brotli-go.v0/dec: build constraints exclude all Go files in /root/go/pkg/mod/gopkg.in/kothar/[email protected]/dec
gopkg.in/kothar/brotli-go.v0/enc: build constraints exclude all Go files in /root/go/pkg/mod/gopkg.in/kothar/[email protected]/enc
Note that I had to install libjpeg-9 instead of libjpeg-8 as the latter is now unavailable - would that cause the above issues? I'd love to use this project, but I can't tell if it's being maintained or not.
Description:
Compy goes wrong when compressing some images, which makes the images unable to load.
Reason:
This is because Compy uses the content-type'' field in the HTTP response header to judge the type of the file. When some websites mistake the
content-type'', Compy goes wrong!
Suggested fix:
A typical solution to this big is that Compy should judge the file type from the content (e.g., the header of the file) rather than merely using the content type.
Description:
Compy may block the redirecting process of some websites
Reason:
Some websites may use user-agent
to present different contents to different users, yet Compy doesn't attempt to forge the user's request.
Suggested fix:
Compy can record the UA when forwarding the user's data packets if necessary.
Hello, I've successfully setup compy on my computer. But, does it offer data savings as seen by the ISP when run on a local server? I used 127.0.0.1:9999 on my browser. Or should I run compy on a cloud server to provide data savings?
Most of https sites work ok, but some of them fail with invalid certificate error. Here's an example list:
https://vk.com
https://youtube.com/
https://support.google.com/chrome/answer/6098869
Compy should allow clients to configure their compression settings via HTTP request headers, e.g., X-Compy-Quality
. This is similar to the implicit configuration that the Accept
and Accept-Encoding
headers provide. Firefox and Chrome extensions allow modification of headers:
It would be a nice feature to transcode video streams into lower resolutions or more efficient codecs.
I'm running compy on my LAN machine, which also hosts a website with nginx via HTTP and HTTPS. If I try to access compy's local webpage where the total savings are displayed, nginx interferes with compy and nothing is displayed.
I guess an option to change the port where the proxy is listinening to HTTP/S connections would be the solution.
Just my two cents.
Homebrew could provide a package to make installation and upgrades easier for macOS users. We might need to start tagging releases for user-friendliness.
While running compy I found that others were using my instance and generated 1.6 GB of traffic. compy should include HTTP authentication to prevent this.
Deploying in GAE would allow users to run compy without allocating an entire VM for this purpose.
Add option via http headers to configure compression levels (gzip and brotli) and separate minification toggles for html, css, json, js
When i run with mitm enabled, i get this:
# compy -ca ca.crt -cakey ca.key
not using mitm: open : no such file or directory
2018/03/11 12:11:32 compy listening on :9999
OS is Debian 9 x64 and go 1.10
when i try to deploy on heroku .
there error in line.
RUN /usr/local/go/bin/go get -d -v ./...
it returned a non-zero code: 1
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.