gbif / gbif-httputils

gbif-httputils

The gbif-httputils shared library provides:

• Common HTTP utilities built on top of Apache HttpClient
  • downloadIfChanged
  • downloadIfModifiedSince
  • verifyHost

To build the project

mvn clean install

gbif-httputils's Issues

Unable to download large archive with TLS from one IPT behind Nginx.

The large DWCA from https://ipt.artsdatabanken.no/resource?r=speciesobservationsservice2 cannot be downloaded using gbif-httputils.

org.apache.http.TruncatedChunkException: Truncated chunk (expected size: 16,368; actual size: 11,826)
	at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:197)
	at org.apache.http.impl.io.ChunkedInputStream.read(ChunkedInputStream.java:210)
	at org.apache.http.entity.BasicHttpEntity.writeTo(BasicHttpEntity.java:116)
	at org.apache.http.entity.HttpEntityWrapper.writeTo(HttpEntityWrapper.java:96)
	at org.apache.http.conn.BasicManagedEntity.writeTo(BasicManagedEntity.java:117)
	at org.gbif.utils.HttpUtil.saveToFile(HttpUtil.java:459)
	at org.gbif.utils.HttpUtil.download(HttpUtil.java:351)
	at org.gbif.utils.HttpUtilTest.testConditionalGet(HttpUtilTest.java:62)

• If the archive is copied to a resource on ipt.gbif.org, it downloads correctly
• If using sslsplit as a MITM proxy, the archive downloads correctly

The problem occurs in Java 8, but not in Java 11.