jtanadi / conveyor Goto Github PK

Queue API is a little awkward. Consider using an event-based approach.

• Send error messages to client instead of throwing internally and only sending 500: Server Error
• How to tell client about pingback error?
  • Can conveyor ensure pingback address is valid before processing files?

For pingbacks and posting to S3

Based on our current task queue, we are only processing one PDF at a time, so files queued later seem like they're taking a long time.

Can we run spawn() concurrently?

pdftocairo is having some trouble parsing CAD files. Look into using ghostscript

For some reason, it seems like "Queuing task" is the last message sent before files are processed and uploaded (ie. scotty shows that message the longest). Check when these messages are being sent & that they're being sent in the right order / at the right time.

The problem

Currently, we calculate gs resolution based on the largest page in the document. This means that given a document with a small enough page, that page will be output at a very small size. Is there a way to vary the output resolution based on page size?

GhostScript documentation

Client should somehow send S3 information (via request header) or we somehow use S3 signed URLs to upload

Look into optipng

Possibly:

/api/pdf2img/?out=png
/api/pdf2img/?out=jpg
etc.

This way, we can add other endpoints in the future, if we need to convert other file types:

/api/doc2pdf/
/api/xls2pdf/
etc.

Send proper error messages back to client.

We're currently not setting an optimization level for optipng, and it may be testing different levels to find a best fit. Perhaps it'll work faster when we set a (low) level?

Look into turning conveyor into an AWS Lambda function.

• Currently only sending pingback when process is completed, but it might be good to send updates to pingback address at major milestones
• Milestones: start, progress: converting file, progress: uploading, end, error

Ideas for pingback message structure, somewhat following node's events (req.on("end"), req.on("error"), etc.):

// For anything in progress, like sending logs
{
  status: "progress",
  message: "Converting PDF with GhostScript"
}

// When process is finally completed, `message`
// will contain relevant data
{
  status: "end",
  message: {
    // contents here (roomID, s3Dir, etc.)
  }
}

// On error
{
  status: "error",
  message: "Network error"
}

Currently gs is outputting at 150dpi resolution:

• Probably excessive for our use case
• Ends up outputting full-scale panels very large (eg. files that are 10–20+ MB)

Is there a way to find out PDF size before converting and resizing as necessary?

Use restana instead of express, since we don't really need all the features of express.

We're currently extending restana's Request interface in src/types.ts. It would be better to use declaration merging and extend the Request interface in types/restana/index.d.ts.