Comments (15)
Supporting other encodings would be a great addition, and additional dependencies are no big deal. As the dependency would probably require some external shared library, it would be good if it could be switched off at compile time for non-quicklisp users.
The hook would probably be installed in the wrap-stream (https://github.com/edicl/drakma/blob/master/request.lisp#L581), but I'm unsure whether a hook can be general enough, given that there may be chunking and encryption as well (i.e. does there need to be any flexibility with respect to where in the stack the hook gets to place its wrapper?).
Is a generic, public hook really useful, i.e. are there uses for it other than implementing decompression?
Would it be useful to be able to compress the request body as well? Is that even supported as per the HTTP standard?
from drakma.
The Content-Encoding and Transfer-Encoding headers only applies to responses.
Also ideally, it should be able to handle both the Content-Encoding and Transfer-Encoding headers.
As far as I can tell, decoding should always be done after encryption and chunking.
The only use case I can think of for having a general, public hook is that it allows users to add support for additional encoding types that are either non-standard or not implemented in drakma. That means that drakma could have support for gzip and deflate, but a user could add additional support for the less common pac200-gzip, or the un-official bzip2 if needed.
Another possible implementation would be to use a generic function that dispatches on the encoding type, then have methods for identity, gizp, defalte, etc. I think I prefer this method.
from drakma.
A generic function that dispatches on the encoding type sounds compelling.
Can you come up with a patch for that?
from drakma.
Yes, I've started working on it.
from drakma.
It is an unfortunate fix, because it breaks backward compatibility.
I've just spent 3 hours debugging my application, to locate this problem.
I am stroing data on Amazon S3 and gzip it to save space. I retrieve my data with zs3:get-vector
and ungzip it. Suddenly, ungzip started to signal an error. Turns out, with new drakma zs3:get-vector
returns not the data I posted earlier, but ungzipped version of it.
I suggest to rollback this fix, until it's refactored in backward compatible fashion.
from drakma.
Thayne, would you be able to take care of this?
2015-03-13 12:08 GMT+01:00 Anton Vodonosov [email protected]:
It is an unfortunate fix, because it breaks backward compatibility.
I've just spent 3 hours debugging my application, to locate this problem.
I am stroing data on Amazon S3 and gzip it to save space. I retrieve my
data with zs3:get-vector and ungzip it. Suddenly, ungzip started to
signal an error. Turns out, with new drakma zs3:get-vector returns not
the data I posted earlier, but ungzipped version of it.I suggest to rollback this fix, until it's refactored in backward
compatible fashion.Reply to this email directly or view it on GitHub
#45 (comment).
from drakma.
I can look at it tonight
from drakma.
Thanks!
2015-03-13 15:59 GMT+01:00 Thayne McCombs [email protected]:
I can look at it tonight
Reply to this email directly or view it on GitHub
#45 (comment).
from drakma.
I don't see an issue here, if S3 sets the Content-Encoding to deflate, then it's fair game. An option to disable this could be added, but no backwards compatibility is necessary. Incidentally, there should be an option for easily specifying Accept-Encoding, probably defaulting to gzip,deflate
from drakma.
Fuzzy concept of "fair game" can't justify breaking someone's existing code.
Support of gzip and deflate is convenient, but doing so in backward incompatible way ads to degradation of CL ecosystem, which has lot of bugs and instabilities as it is. I regularly monitor QL libraries and every month something breaks (https://common-lisp.net/project/cl-test-grid/ql/).
Who is affected - any code already dealing with gzip or deflate encoding. It's not a theoretical speculation, my case is a concrete example of this.
from drakma.
Then "tough luck" is a better justification. Sometimes things have to be broken. It's hard to get API right the first time and foresee any future needs. Not breaking anything will result in a shit API full of crutches.
from drakma.
Generally speaking, if you need to improve an API, you introduce new API, instead of breaking the existing one.
If I wanted to improve drakma API, I would introduce new method drakma:http-request2
with better default behaviour and simpler parameters. Or, if the changes are very significant, I would created new library, drakma2. So the old tested code remains working using drakma:http-request
, and the new code uses new function / library.
That way API evolves without and destructive network effect or efforts / time expenses. Almost always maintaining backward compatibility is easy and cheap (both for library author and the users).
About this particular case. Anyway, we need a parameter to disable / enable automatic gzip handling - I may want to download the gzipped version in order to back up it, or move to another location; in this case I don't want redundant ungzip / gzip processing.
And in my opinion the default should be backwards compatible. In the open source CL we don't have many resources, it takes years to fix even simple bugs, so deliberately breaking things is unacceptable.
Incidentally, there should be an option for easily specifying Accept-Encoding
It's easy to specify using :additional-headers
from drakma.
from drakma.
OK.
For the record, new API for new functionality, instead of breaking the old API.
from drakma.
from drakma.
Related Issues (20)
- Can not send nil values in multipart/form-data. HOT 2
- cl+ssl difficulties on Windows, possible alternatives HOT 2
- Wrong "Host:" header when accessing IPv6 resources by address HOT 7
- unidentified bug causing short read of chunked data HOT 10
- Request hangs despite :connection-timeout HOT 5
- Need to be able to set :content-type when using :method :post HOT 2
- Feature request: API to get the local and remote hosts and port.
- Feature request: API to obtain the results of the SSL/TLS negociation
- Requests using ":verify :required" eats RAM when error is signalled HOT 1
- Add connection pool? HOT 1
- Doesn't retrieve data from GET request HOT 1
- http-request broken in LispWorks 8 HOT 6
- Allow uploading directories HOT 4
- Possible to make Drakma less picky on put vs. post? HOT 12
- USOCKET:TIMEOUT-ERROR HOT 2
- failure to send octets to a :force-binary stream. HOT 5
- Not able to send different content-types on multipart/form-data requests
- ssl3_read_n:unexpected eof while reading HOT 5
- How to read response from stream returned by :want-stream t HOT 2
- http-request http://lisp.org example triggers socket error (A non-blocking socket operation could not be completed immediately.) HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from drakma.