o2r-project / o2r-contentbutler Goto Github PK

As seen in the o2r-muncher file listings, the MIME types of files can easily be reported with the mime npm module. We should probably use this also in contentbutler, as it is a more consistent (i.e. filter by image/*, text/*) and easier to maintain (add extension to types definition) solution.

write a doc explaining how to integrate this service with e.g. nginx. location with regular expressions seem to be the easiest way.

@JanKoppe was not really happy with the name... in line with "muncher" I suggest to make it at least "a person"/use active voice, so rename to o2r-dataserver. And then it's not far to butlers

@JanKoppe Do you imagine this microservice also being the one serving access to complete compendia? Or would you do that in a different one? In case of a different one, a more focussed name would be o2r-contentbutler, the one for complete compendia could be o2r-compendiumbutler

Just saw that the parseInteger check has no else clause: https://github.com/o2r-project/o2r-dataserve/blob/master/lib/resize.js#L48

Should this not be communicated to the client as a 400 error?

errors are directly pushed to the client. there needs to be a abstraction layer and better logging.

We have a Piwik instance running, so we could use it's reporting API to observe direkt calls to the API (which might not come from our website).

• https://github.com/fvdm/nodejs-piwik
• https://github.com/fhemberger/piwik-tracker

Alternatively (or preferably, we can simply analyse the nginx and webapp logs. See discussion below.

Tasks

• extend app logs to reflect all calls and actions that take some time
• evaluate piwik log analytics using the nginx logs, see https://piwik.org/docs/log-analytics-tool-how-to/ and https://piwik.org/log-analytics/
• if piwik does not suffice, then evaluate and install log collection solution (ELK, Prometheus, ..) on the server and collect the microservice's logs, see e.g. http://logz.io/learn/docker-monitoring-elk-stack/

mitigations for directory traversal are missing.

Can we easily add tiff to https://github.com/o2r-project/o2r-dataserve/blob/master/lib/resize.js ?

• The supported formats should be listed in the web API
• What happens if there is no extension but only mime-type?

Maybe we can extract a list of supported formats from the underlying system library?

Add a check running at the startup of the service to see if the required library is installed, and log the version, probably both using convert --version

Provide previews (e.g. by overloading the "size" parameter even more) of PDF documents.

• PDF to jpg can be done with ImageMagick (http://superuser.com/questions/633698/convert-pdf-to-jpg-images-with-imagemagick-how-to-0-pad-file-names)
• a size of 1 or any higher integer would return a JPG of the first page. PDF subsetting (i.e. returning a shorter PDF document) does not make sense for a simple preview.