GithubHelp home page GithubHelp logo

metosin / muuntaja Goto Github PK

View Code? Open in Web Editor NEW
444.0 25.0 51.0 1.25 MB

Clojure library for fast http api format negotiation, encoding and decoding.

Home Page: https://cljdoc.org/d/metosin/muuntaja

License: Eclipse Public License 2.0

Clojure 99.23% Shell 0.77%
clojure http ring middleware interceptor json transit edn content-negotiation metosin-stable

muuntaja's Introduction

Muuntaja Continuous Integration status cljdoc badge

Clojure library for fast HTTP format negotiation, encoding and decoding. Standalone library, but ships with adapters for ring (async) middleware & Pedestal-style interceptors. Explicit & extendable, supporting out-of-the-box JSON, EDN and Transit (both JSON & Msgpack). Ships with optional adapters for MessagePack and YAML.

Based on ring-middleware-format, but a complete rewrite (and up to 30x faster).

Rationale

  • explicit configuration
  • fast with good defaults
  • extendable & pluggable: new formats, behavior
  • typed exceptions - caught elsewhere
  • support runtime docs (like swagger) & inspection (negotiation results)
  • support runtime configuration (negotiation overrides)

Modules

  • metosin/muuntaja - the core abstractions + Jsonista JSON, EDN and Transit formats
  • metosin/muuntaja-cheshire - optional Cheshire JSON format
  • metosin/muuntaja-charred - optional Charred format
  • metosin/muuntaja-form - optional application/x-www-form-urlencoded formatter using ring-codec
  • metosin/muuntaja-msgpack - Messagepack format
  • metosin/muuntaja-yaml - YAML format

Posts

Check the docs on cljdoc.org for detailed API documentation as well as more guides on how to use Muuntaja.

Latest version

[metosin/muuntaja "0.6.10"]

Optionally, the parts can be required separately:

[metosin/muuntaja-form "0.6.10"]
[metosin/muuntaja-cheshire "0.6.10"]
[fi.metosin/muuntaja-charred "0.6.10"]
[metosin/muuntaja-msgpack "0.6.10"]
[metosin/muuntaja-yaml "0.6.10"]

Muuntaja requires Java 1.8+

Quickstart

Standalone

Use default Muuntaja instance to encode & decode JSON:

(require '[muuntaja.core :as m])

(->> {:kikka 42}
     (m/encode "application/json"))
; => #object[java.io.ByteArrayInputStream]

(->> {:kikka 42}
     (m/encode "application/json")
     slurp)
; => "{\"kikka\":42}"

(->> {:kikka 42}
     (m/encode "application/json")
     (m/decode "application/json"))
; => {:kikka 42}

Ring

Automatic decoding of request body and response body encoding based on Content-Type, Accept and Accept-Charset headers:

(require '[muuntaja.middleware :as middleware])

(defn echo [request]
  {:status 200
   :body (:body-params request)})

; with defaults
(def app (middleware/wrap-format echo))

(def request
  {:headers
   {"content-type" "application/edn"
    "accept" "application/transit+json"}
   :body "{:kikka 42}"})

(app request)
; {:status 200,
;  :body #object[java.io.ByteArrayInputStream]
;  :headers {"Content-Type" "application/transit+json; charset=utf-8"}}

Automatic decoding of response body based on Content-Type header:

(-> request app m/decode-response-body)
; {:kikka 42}

There is a more detailed Ring guide too. See also differences to ring-middleware-format & ring-json.

Interceptors

Muuntaja support Sieppari -style interceptors too. See muuntaja.interceptor for details.

Interceptors can be used with Pedestal too, all but the exception-interceptor which conforms to the simplified exception handling model of Sieppari.

Configuration

Explicit Muuntaja instance with custom EDN decoder options:

(def m
  (m/create
    (assoc-in
      m/default-options
      [:formats "application/edn" :decoder-opts]
      {:readers {'INC inc}})))

(->> "{:value #INC 41}"
     (m/decode m "application/edn"))
; => {:value 42}

Explicit Muuntaja instance with custom date formatter:

(def m
  (m/create
    (assoc-in
      m/default-options
      [:formats "application/json" :encoder-opts]
      {:date-format "yyyy-MM-dd"})))

(->> {:value (java.util.Date.)}
     (m/encode m "application/json")
     slurp)
; => "{\"value\":\"2019-10-15\"}"

Explicit Muuntaja instance with camelCase encode-key-fn:

(require '[camel-snake-kebab.core :as csk])

(def m
  (m/create
    (assoc-in
      m/default-options
      [:formats "application/json" :encoder-opts]
      {:encode-key-fn csk/->camelCase})))

(->> {:some-property "some-value"}
     (m/encode m "application/json")
     slurp)
; => "{\":someProperty\":\"some-value\"}"

Returning a function to encode transit-json:

(def encode-transit-json
  (m/encoder m "application/transit+json"))

(slurp (encode-transit-json {:kikka 42}))
; => "[\"^ \",\"~:kikka\",42]"

Encoding format

By default, encode writes value into a java.io.ByteArrayInputStream. This can be changed with a :return option, accepting the following values:

value description
:input-stream encodes into java.io.ByteArrayInputStream (default)
:bytes encodes into byte[]. Faster than Stream, enables NIO for servers supporting it
:output-stream encodes lazily into java.io.OutputStream via a callback function

All return types satisfy the following Protocols & Interfaces:

  • ring.protocols.StreamableResponseBody, Ring 1.6.0+ will stream these for you
  • clojure.io.IOFactory, so you can slurp the response

:input-stream

(def m (m/create (assoc m/default-options :return :input-stream)))

(->> {:kikka 42}
     (m/encode m "application/json"))
; #object[java.io.ByteArrayInputStream]

:bytes

(def m (m/create (assoc m/default-options :return :bytes)))

(->> {:kikka 42}
     (m/encode m "application/json"))
; #object["[B" 0x31f5d734 "[B@31f5d734"]

:output-stream

(def m (m/create (assoc m/default-options :return :output-stream)))

(->> {:kikka 42}
     (m/encode m "application/json"))
; <<StreamableResponse>>

Format-based return

(def m (m/create (assoc-in m/default-options [:formats "application/edn" :return] :output-stream)))

(->> {:kikka 42}
     (m/encode m "application/json"))
; #object[java.io.ByteArrayInputStream]

(->> {:kikka 42}
     (m/encode m "application/edn"))
; <<StreamableResponse>>

HTTP format negotiation

HTTP format negotiation is done using request headers for both request (content-type, including the charset) and response (accept and accept-charset). With the default options, a full match on the content-type is required, e.g. application/json. Adding a :matches regexp for formats enables more loose matching. See Configuration docs for more info.

Results of the negotiation are published into request & response under namespaced keys for introspection. These keys can also be set manually, overriding the content negotiation process.

Exceptions

When something bad happens, an typed exception is thrown. You should handle it elsewhere. Thrown exceptions have an ex-data with the following :type value (plus extra info to enable generating descriptive erros to clients):

  • :muuntaja/decode, input can't be decoded with the negotiated format & charset.
  • :muuntaja/request-charset-negotiation, request charset is illegal.
  • :muuntaja/response-charset-negotiation, could not negotiate a charset for the response.
  • :muuntaja/response-format-negotiation, could not negotiate a format for the response.

Server Spec

Request

  • :muuntaja/request, client-negotiated request format and charset as FormatAndCharset record. Will be used in the request pipeline.
  • :muuntaja/response, client-negotiated response format and charset as FormatAndCharset record. Will be used in the response pipeline.
  • :body-params contains the decoded body.

Response

  • :muuntaja/encode, if set to truthy value, the response body will be encoded regardles of the type (primitives!)
  • :muuntaja/content-type, handlers can use this to override the negotiated content-type for response encoding, e.g. setting it to application/edn will cause the response to be formatted in JSON.

Options

Default options

{:http {:extract-content-type extract-content-type-ring
        :extract-accept-charset extract-accept-charset-ring
        :extract-accept extract-accept-ring
        :decode-request-body? (constantly true)
        :encode-response-body? encode-collections}

 :allow-empty-input? true
 :return :input-stream

 :default-charset "utf-8"
 :charsets available-charsets

 :default-format "application/json"
 :formats {"application/json" json-format/json-format
           "application/edn" edn-format/edn-format
           "application/transit+json" transit-format/transit-json-format
           "application/transit+msgpack" transit-format/transit-msgpack-format}}

Profiling

YourKit supports open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of YourKit Java Profiler and YourKit .NET Profiler, innovative and intelligent tools for profiling Java and .NET applications.

License

By Unknown. The drawing is signed "E. Ducretet", indicating that the apparatus was made by Eugene Ducretet, a prominent Paris scientific instrument manufacturer and radio researcher. The drawing was undoubtedly originally from the Ducretet instrument catalog. [Public domain], via Wikimedia Commons.

Original Code (ring-middleware-format)

Copyright © 2011, 2012, 2013, 2014 Nils Grunwald
Copyright © 2015, 2016 Juho Teperi

This library

Copyright © 2016-2020 Metosin Oy

Distributed under the Eclipse Public License 2.0.

muuntaja's People

Contributors

bsless avatar cgore avatar cpjolicoeur avatar dadair-ca avatar deraen avatar hlship avatar ieure avatar ikitommi avatar kalekale avatar lamdor avatar lotuc avatar martinklepsch avatar michaelblume avatar miikka avatar mthl avatar mvarela avatar ngrunwald avatar nikolap avatar ninjudd avatar opqdonut avatar philipa avatar r0man avatar the-alchemist avatar tranchis avatar valerauko avatar viebel avatar vielmath avatar yayitswei avatar ykarikos avatar yogthos avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

muuntaja's Issues

0.2.0 doesn't compile

Problem

I found this while initiating a new luminus app. The app won't run or compile. Steps to reproduce and error message below:

lein new luminus guestbook +h2
cd guestbook
lein run migrate

Exception in thread "main" java.lang.ExceptionInInitializerError
	at clojure.main.<clinit>(main.java:20)
Caused by: java.io.FileNotFoundException: Could not locate msgpack/clojure_extensions__init.class or msgpack/clojure_extensions.clj on classpath. Please check that namespaces with dashes use underscores in the Clojure file name., compiling:(muuntaja/format/json.clj:1:1)
...

This is caused by the breaking change in the latest version.

Investigations

I went ahead and cloned muuntaja, then ran lein with-profile uberjar check and sure enough, I got

Exception in thread "main" java.io.FileNotFoundException: Could not locate msgpack/clojure_extensions__init.class or msgpack/clojure_extensions.clj on classpath

I'm pretty sure that's not expected behaviour? I would have submitted a fix but since you don't want the dep in the base dependencies, I'm not sure what I can do.

Cannot use muuntaja.core without depending on Ring

The namespace muuntaja.protocols requires ring.core.protocols, but there is no dependency on ring/ring-core in project.clj. Trying to compile a project depending on Muuntaja and just requiring muuntaja.core results in an error

java.io.FileNotFoundException: Could not locate ring/core/protocols__init.class or ring/core/protocols.clj on classpath., compiling:(protocols.clj:1:1)
Exception in thread "main" java.io.FileNotFoundException: Could not locate ring/core/protocols__init.class or ring/core/protocols.clj on classpath., compiling:(protocols.clj:1:1)

Streaming encoder protocols

a new format key & supporting protocols like:

(defprotocol EncodeToJsonStream
  (encode-to-json-stream [output-stream charset]))

(defprotocol EncodeToTransitJsonStream
  (encode-to-transit-json-stream [output-stream charset]))

Faster JSON encoding with jsonista

Currently, because Muuntaja allows customer to define response charset, we are generating intermediate String-presentations of JSON (there doesn't seem to be a way to define charset in Jackson?). This doubles the time to encoding. As JSON is most (99,99%+?) of the time in UTF-8, we could use Jackson to do stream to bytes directly.

(def j (cheshire/parse-string (slurp (str "dev-resources/json1k.json")) true))

; current
; Execution time mean : 7.233861 µs
(cc/quick-bench
  (.getBytes (jsonista.core/write-value-as-string j) "utf-8"))

; Execution time mean : 3.629525 µs
(cc/quick-bench
  (jsonista.core/write-value-as-bytes j))

muuntaja.json decoder-options

Cheshire has :key-fn, :array-coerce-fn and :bigdecimals?, Ring-json has :keywords? (setting the :key-fn actually), Muuntaja has now :keywordize?.

Would be good to check the muuntaja.json and muuntaja.format.json/make-muuntaja-json-encoder options before release.

More control over what to what to read & write

Jsonista has a good and fast api for reading and writing JSON. One can ask directly for JSON byte-array out of Clojure data - Jackson has optimized path just for this. With Muuntaja, one is forced to generate InputStream, which much slower, no benefits from NIO for example.

So, a new unified api, maybe directly from Jsonista:

(def data {:kikka "kukka"})

;; JSON
(m/write-value-as-string m "application/json" data)
(m/write-value-as-bytes m "application/json" data)
(m/write-value m "application/json" (File. "kikka.json") data)
(m/read-value m "application/json" (File. "kikka.json"))

;; EDN
(m/write-value-as-string m "application/edn" data)
(m/write-value-as-bytes m "application/edn" data)
(m/write-value m "application/edn" (File. "kikka.json") data)
(m/read-value m "application/edn" (File. "kikka.json"))

Option to allow empty bodies

Many existing formatters (like ring-middleware-format) allow empty bodies, just reads them as nil. Use case for this is a client, which always sets the content-type header but conditionally sets the body. Currently, Muuntaja fails on these requests (as nil is invalid in the JSON & Transit specs).

=> new option [:http :allow-empty-body?]

Handle different charsets

ring-middleware-format negotiated charsets, muuntaja currently does not. There is a branch charset, which try to do this, but still not sure how this should be done. Someone should figure out:

  • IN: what is and can be done in the adapter/http-server/container -level? e.g. can it do automatic stream conversion from e.g. ISO-8859-1 to UTF-8?
  • OUT: if we return a String to the http-server with a content-type & a charset, will them all produce a correctly encoded byte-stream already?

For now, the charsets are negotiated, but not used.

Seems to be broken with newest CLJS 1.10.339

java.lang.NoSuchMethodError: com.cognitect.transit.TransitFactory.writer(Lcom/cognitect/transit/TransitFactory$Format;Ljava/io/OutputStream;Ljava/util/Map;Lcom/cognitect/transit/WriteHandler;Ljava/util/function/Function;)Lcom/cognitect/transit/Writer;
                          transit.clj:157 cognitect.transit/writer
                          transit.clj:139 cognitect.transit/writer
                           transit.clj:20 muuntaja.format.transit/encoder[fn]
                             core.clj:344 muuntaja.core/create-coder[fn]
                            core.clj:6118 clojure.core/update
                            core.clj:6108 clojure.core/update
                             core.clj:443 muuntaja.core/create[fn]
                             core.clj:483 muuntaja.core/create[fn]
                        middleware.clj:73 muuntaja.middleware/wrap-format[fn]
                             jsonp.clj:55 doublethedonation.jsonp/wrap-json-with-padding[fn]
                          session.clj:108 ring.middleware.session/wrap-session[fn]
                    keyword_params.clj:53 ring.middleware.keyword-params/wrap-keyword-params[fn]
                     nested_params.clj:89 ring.middleware.nested-params/wrap-nested-params[fn]
                 multipart_params.clj:173 ring.middleware.multipart-params/wrap-multipart-params[fn]
                            params.clj:67 ring.middleware.params/wrap-params[fn]
                          cookies.clj:175 ring.middleware.cookies/wrap-cookies[fn]
                absolute_redirects.clj:47 ring.middleware.absolute-redirects/wrap-absolute-redirects[fn]
                          resource.clj:24 ring.middleware.resource/wrap-resource-prefer-resources[fn]
                      content_type.clj:34 ring.middleware.content-type/wrap-content-type[fn]
                   default_charset.clj:31 ring.middleware.default-charset/wrap-default-charset[fn]
                      not_modified.clj:61 ring.middleware.not-modified/wrap-not-modified[fn]
                         x_headers.clj:22 ring.middleware.x-headers/wrap-x-header[fn]
                         x_headers.clj:22 ring.middleware.x-headers/wrap-x-header[fn]
                               ssl.clj:35 ring.middleware.ssl/wrap-forwarded-scheme[fn]
                     proxy_headers.clj:20 ring.middleware.proxy-headers/wrap-forwarded-remote-addr[fn]
                        stacktrace.clj:26 ring.middleware.stacktrace/wrap-stacktrace-log[fn]
                        stacktrace.clj:96 ring.middleware.stacktrace/wrap-stacktrace-web[fn]
                             jetty.clj:26 ring.adapter.jetty/proxy-handler[fn]
                         (Unknown Source) ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle
                  HandlerWrapper.java:132 org.eclipse.jetty.server.handler.HandlerWrapper.handle
                          Server.java:531 org.eclipse.jetty.server.Server.handle
                     HttpChannel.java:352 org.eclipse.jetty.server.HttpChannel.handle
                  HttpConnection.java:260 org.eclipse.jetty.server.HttpConnection.onFillable
              AbstractConnection.java:281 org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
                    FillInterest.java:102 org.eclipse.jetty.io.FillInterest.fillable
                 ChannelEndPoint.java:118 org.eclipse.jetty.io.ChannelEndPoint$2.run
                  EatWhatYouKill.java:333 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask
                  EatWhatYouKill.java:310 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
                  EatWhatYouKill.java:168 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
                  EatWhatYouKill.java:126 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run
          ReservedThreadExecutor.java:366 org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run
                QueuedThreadPool.java:762 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
                QueuedThreadPool.java:680 org.eclipse.jetty.util.thread.QueuedThreadPool$2.run
                          Thread.java:748 java.lang.Thread.run

can't reify muuntaja.format.core/Decode in lein uberjar

I have tried to create a custom encoder/decoder for xml. When I run lein uberjar from the docker image clojure I get a compile error:

Caused by: clojure.lang.ExceptionInfo: Invalid format "application/xml" for type :muuntaja/decode. It should satisfy muuntaja.format.core.Decode {:format "application/xml", :type :muuntaja/decode, :spec [#object[application.middleware$decoder 0x63fe055 "application.middleware$decoder@63fe055"]], :coder #object[application.middleware$decoder$reify__4217 0xab90b8c "application.middleware$decoder$reify__4217@ab90b8c"], :protocol {:on muuntaja.format.core.Decode, :on-interface muuntaja.format.core.Decode, :sigs {:decode {:name decode, :arglists ([this data charset]), :doc nil}},

The decoder looks like this:

(defn decoder [options]
  (reify muuntaja.format.core/Decode
    (decode [this data charset]
      .....)))))

What's weird is that it works on my local machine (sounds familiar ;). This only occurs in the docker image. I can't see what should make a difference.

Essentially this expression turns out to be false inside the container when lein uberjar executes.

(satisfies? muuntaja.format.core/Decode
  (reify muuntaja.format.core/Decode
    (decode [this data charset]
      .....))
=> false

For other protocols it works:

(defprotocol Qux 
  (decode [this data charset]))

(satisfies? Qux
  (reify Qux
    (decode [this data charset]
      .....))
=> true

This occurs when I aot-compile the whole application. What is not clear to me:

  • does muuntaja support AOT compilation or is it not compatible with?
  • what is special about the namespace muuntaja.format.core
  • why does it work on my local machine but not on docker?
  • is this a clojure bug or a muuntaja bug? (I take docker out of the mix, don't think it's related to it. I suppose this bug should be reproducible on linux.)

I use this dockerfile

FROM clojure
COPY . /usr/src/app
WORKDIR /usr/src/app

RUN lein uberjar

Then execute docker build .

  • lein version 2.8.1
  • java 1.8.0_111

IllegalArgumentException for an empty `application/json` body

If you use muuntaja.middleware/wrap-format-request and do a HTTP request with the header 'Content-Type: application/json` and with an empty body, Muuntaja throws an IllegalArgumentException. I think that empty bodies should be just ignored.

A stacktrace from a Kekkonen hello world app:

clojure.lang.ExceptionInfo: Malformed application/json request. {:type :muuntaja/decode, :default-format "application/json", :format "application/json", :charset "utf-8", :request {:remote-addr "0:0:0:0:0:0:0:1", :params {}, :headers {"origin" "http://localhost:5000", "host" "localhost:5000", "content-type" "application/json", "content-length" "0", "referer" "http://localhost:5000/api-docs/index.html", "connection" "keep-alive", "accept" "application/json", "accept-language" "fi-fi", "accept-encoding" "gzip, deflate", "dnt" "1", "kekkonen.mode" "invoke"}, :async-channel #object[org.httpkit.server.AsyncChannel 0x37ba91df "/0:0:0:0:0:0:0:1:5000<->/0:0:0:0:0:0:0:1:49756"], :server-port 5000, :muuntaja/request #FormatAndCharset{:format "application/json", :charset "utf-8"}, :content-length 0, :form-params {}, :websocket? false, :query-params {}, :content-type "application/json", :character-encoding "utf8", :uri "/api/sample/inc!", :server-name "localhost", :query-string nil, :muuntaja/response #FormatAndCharset{:format "application/json", :charset "utf-8"}, :body nil, :scheme :http, :request-method :post}}
	at clojure.core$ex_info.invokeStatic(core.clj:4617)
	at clojure.core$ex_info.invoke(core.clj:4617)
	at muuntaja.core$fail_on_request_decode_exception.invokeStatic(core.clj:236)
	at muuntaja.core$fail_on_request_decode_exception.invoke(core.clj:230)
	at muuntaja.core$decode_request_body.invokeStatic(core.clj:369)
	at muuntaja.core$decode_request_body.invoke(core.clj:363)
	at muuntaja.core$decode_request.invokeStatic(core.clj:379)
	at muuntaja.core$decode_request.invoke(core.clj:376)
	at muuntaja.core$format_request.invokeStatic(core.clj:387)
	at muuntaja.core$format_request.invoke(core.clj:384)
	at muuntaja.middleware$wrap_format_request$fn__31336.invoke(middleware.clj:113)
	at kekkonen.middleware$wrap_exceptions$fn__32593.invoke(middleware.clj:74)
	at muuntaja.middleware$wrap_format_response$fn__31340.invoke(middleware.clj:131)
	at muuntaja.middleware$wrap_format_negotiate$fn__31333.invoke(middleware.clj:95)
	at kekkonen.middleware$wrap_keyword_keys$fn__32602.invoke(middleware.clj:106)
	at ring.middleware.keyword_params$wrap_keyword_params$fn__30330.invoke(keyword_params.clj:35)
	at ring.middleware.nested_params$wrap_nested_params$fn__30502.invoke(nested_params.clj:86)
	at ring.middleware.params$wrap_params$fn__30592.invoke(params.clj:64)
	at clojure.lang.Var.invoke(Var.java:379)
	at org.httpkit.server.HttpHandler.run(RingHandler.java:91)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: clojure.lang.ExceptionInfo: Malformed application/json in :muuntaja/decode {:type :muuntaja/decode, :format "application/json"}
	at clojure.core$ex_info.invokeStatic(core.clj:4617)
	at clojure.core$ex_info.invoke(core.clj:4617)
	at muuntaja.core$on_exception.invokeStatic(core.clj:66)
	at muuntaja.core$on_exception.invoke(core.clj:64)
	at muuntaja.core$create_coder$f__31187.invoke(core.clj:106)
	at muuntaja.core$decode_request_body.invokeStatic(core.clj:367)
	... 20 more
Caused by: java.lang.IllegalArgumentException: No implementation of method: :as-input-stream of protocol: #'muuntaja.protocols/AsInputStream found for class: nil
	at clojure.core$_cache_protocol_fn.invokeStatic(core_deftype.clj:568)
	at clojure.core$_cache_protocol_fn.invoke(core_deftype.clj:560)
	at muuntaja.protocols$eval30740$fn__30741$G__30731__30746.invoke(protocols.clj:45)
	at muuntaja.core$create_coder$f__31187.invoke(core.clj:104)
	... 21 more

Clojars binary doesn't supports OpenJDK 1.7

Not sure this is intended behavior because of a software dependency from Java 1.8. If not then it may be simply an issue created by the platform you use to build the binary distributed on clojars. The effect is that any project depending from muuntaja loses backward support for openjdk 1.7.
I hope you contemplate this as a bug, it would be nice to grant backward compatibility.

encode-protocol doesn't work

According to the docs, encode-protocol should allow for custom serialization.
But it doesn't seem it is working; besides I couldn't find any reference to encode-protocol on the source code.

Example:

;; custom protocol to encode JSON
(defprotocol EncodeJson
  ;; the 2-arity callback
  (encode-json [this charset]))

;; mount it to a formatter
(def m
  (muuntaja/create
    (-> muuntaja/default-options
        (assoc-in
          [:formats "application/json" :encode-protocol]
          [EncodeJson encode-json]))))

;; custom record with hand-crafted JSON:
(defrecord Hello []
  EncodeJson
  (encode-json [_ _]
    (ByteArrayInputStream.
      (.getBytes "{\"hello\":\"world\"}"))))


(->> (->Hello)
     (muuntaja/encode m "application/json")
     (muuntaja/decode m "application/json"))
;;; returns an empty map instead of a {"hello":"world"} map

Problem with headers

Hi!
I'm using compojure-api, and making a GET request from my frontend code using cljs-ajax.
The request seems to automatically add some headers:

Accept: application/transit+json, application/transit+transit, application/json, text/plain, text/html, */*

This fails on the API and gives me the following error:

clojure.lang.ExceptionInfo: Malformed application/transit+json in :muuntaja/encode

But works when I change the header to:

Accept: application/json

Adding :body-params to :params

What's the best way to do this? Switched from r-m-f and it seems that muuntaja uses :body-params instead of :form-params, so there should be a way for the :body-params to be merged into :params.

Full content negotiation

Currently, Muuntaja only publishes content-negotiation results if it is responsible the formats. e.g. If a client accepts only application/xml, Muuntaja ignores that.

Better way would be to always publish the negotiation results and mark them somehow as :managed if Muuntaja knows how to encode / dedode those.

e.g.

;; request
{:headers {"accept" "application/xml", "content-type" "application/json"}}

;; after negotiation
{:headers {"accept" "application/xml", "content-type" "application/json"}
 :muuntaja/request #FormatAndCharset{:format "application/json", :charset "utf-8", :manged true}
 :muuntaja/response #FormatAndCharset{:format "application/xml", :charset "utf-8", :manged false}}

Pick default charset when there are multiple equally-preferred options in the Accept header

Some HTTP clients send a huge list of charsets they can accept without specifying their preferred option with q=. E.g. apparently Spring sends all the charsets JVM supports in alphabetical order. Now we pick the first one, but we could do a favor to the users and pick :default-charset if it's in the list – or maybe this should be a separate option. I think people would prefer getting UTF-8 instead of, say, Big5.

A multi-module project layout

Muuntaja could be a multi-module project (like Martian, producing the following artefacts:

  • metosin/muuntaja (the suite having all)
  • metosin/muuntaja-core (just the core abstractions - no dependencies!)
  • metosin/muuntaja-json
  • metosin/muuntaja-edn
  • metosin/muuntaja-transit
  • metosin/muuntaja-msgpack
  • metosin/muuntaja-yaml

Re-implement Muuntaja as a Protocol

Currently, it's too easy to modify created Muuntaja instance which has no effect. e.g.

(-> (m/create) (assoc :default-format "application/edn"))

instead of:

(-> (assoc m/default-options :default-format "application/edn") (m/create))

Keep `:http`-options as separate record

Currently, me merge the options under :http into top-level in Muuntaja. For perf reasons. There should a Record HttpOptions or similar, not big penalty and would make configuring Muuntaja much simpler.

Now, compojure-api has to do ask wether it has an instance or just options and set it's override config accordingly to correct level. BadBadbad.

(defn create-muuntaja
  ([] (create-muuntaja m/default-options))
  ([muuntaja-or-options]
   (cond

     (= ::default muuntaja-or-options)
     (m/create
       (assoc-in m/default-options [:http :encode-response-body?] encode?))

     (instance? Muuntaja muuntaja-or-options)
     (assoc muuntaja-or-options :encode-response-body? encode?)

     (map? muuntaja-or-options)
     (m/create
       (assoc-in muuntaja-or-options [:http :encode-response-body?] encode?))

     :else
     (throw (ex-info (str "Invalid :formats - " muuntaja-or-options) {:options muuntaja-or-options})))))

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.