Error "java.nio.charset.MalformedInputException: Input length = 1" about giter8

[n8han]

likely relates to the fact that dispatch apparently going against the RFC and assuming UTF-8

Did you check the content headers?

nathan@x201 ~/Programming/giter8[master] $ curl -Is https://github.com/api/v2/json/blob/show/n8han/conscript.g8/c2ede99b688c84da2aa8179ad9f0ab3e9404635b|grep Content-Type
Content-Type: text/plain; charset=utf-8

Dispatch doesn't use its woefully incompliant default in this case; it's github that has assumed utf-8. Although, there's another header in there that looks interesting:

X-Content-Type-Options: nosniff

If that sniffing includes making an intelligent guess at the encoding, I say we use it. Better them guessing that us. If it's wrong it's wrong. What kind of a source file are you using a non-utf8 charset for, anyway? :)

[eed3si9n]

It's actually just a fork of ngocdaothanh/xitrum.g8 to preserve its current state. It's some kind of keystore file that's causing this.

Dispatch doesn't use its woefully incompliant default in this case; it's github that has assumed utf-8.

Sorry. I stand corrected.

The problem starts when blob/full thinks it's a text/plain when the file is actually binary:

Then it declares utf-8, so that's the second problem.

$ curl https://github.com/api/v2/json/blob/show/eed3si9n/xitrum.g8/5a59c367d97af0fe3eddb4a59173b790a5164ace -I | grep Content-Type
...
Content-Type: text/plain; charset=utf-8

Should we just assume binary if we encounter MalformedInputException?

[eed3si9n]

This seems to work...

  def write(repo: String, templates: Iterable[(String, String, String, String)], parameters: Map[String,String], base: File) = {
    templates foreach { case (name, hash, mime, mode) =>
      import org.clapper.scalasti.StringTemplate
      import java.nio.charset.MalformedInputException
      val f = new File(base, new StringTemplate(name).setAttributes(parameters).toString)
      if (f.exists)
        println("Skipping existing file: %s" format f.toString)
      else {
        f.getParentFile.mkdirs()
        (mime match {
          case Text(_) =>
            try {
              http(show(repo, hash) >- { in =>
                use (new FileWriter(f)) { fw =>
                  fw.write(new StringTemplate(in).setAttributes(parameters).toString)
                  Some(())
                }
              })
            }
            catch {
              case e: MalformedInputException => None
            }
          case binary => None
        }) getOrElse {
          http(show(repo, hash) >> { is =>
            use(is) { in =>
              use(new FileOutputStream(f)) { out =>
                def consume(buf: Array[Byte]): Unit =
                  in.read(buf) match {
                    case -1 => ()
                    case n =>
                       out.write(buf, 0, n)
                       consume(buf)
                  }
                consume(new Array[Byte](1024))
              }
            }
          })
        }
        setFileMode(f, mode)
      }
    }
    Right("Applied %s in %s" format (repo, base.toString))
  }

[eed3si9n]

With the lastest v3 of Github API, the user is supposed to either request for json or raw in the Accept header. WIth raw, it no longer assumes UTF-8.

$ curl https://api.github.com/repos/eed3si9n/xitrum.g8/git/blobs/5a59c367d97af0fe3eddb4a59173b790a5164ace -H "Accept: application/vnd.github-blob.raw" -I | grep Content-Type
...
Content-Type: application/vnd.github-blob.raw