jtanadi / conveyor Goto Github PK
View Code? Open in Web Editor NEWMicroservice to convert PDFs to image files and upload them to AWS S3
Home Page: http://raa-conveyor.herokuapp.com/
Microservice to convert PDFs to image files and upload them to AWS S3
Home Page: http://raa-conveyor.herokuapp.com/
Queue API is a little awkward. Consider using an event-based approach.
500: Server Error
pingback
error?
conveyor
ensure pingback
address is valid before processing files?For pingbacks and posting to S3
Based on our current task queue, we are only processing one PDF at a time, so files queued later seem like they're taking a long time.
Can we run spawn()
concurrently?
pdftocairo
is having some trouble parsing CAD files. Look into using ghostscript
For some reason, it seems like "Queuing task" is the last message sent before files are processed and uploaded (ie. scotty
shows that message the longest). Check when these messages are being sent & that they're being sent in the right order / at the right time.
Currently, we calculate gs
resolution based on the largest page in the document. This means that given a document with a small enough page, that page will be output at a very small size. Is there a way to vary the output resolution based on page size?
Client should somehow send S3 information (via request header) or we somehow use S3 signed URLs to upload
Look into optipng
Possibly:
/api/pdf2img/?out=png
/api/pdf2img/?out=jpg
etc.
This way, we can add other endpoints in the future, if we need to convert other file types:
/api/doc2pdf/
/api/xls2pdf/
etc.
Send proper error messages back to client.
We're currently not setting an optimization level for optipng
, and it may be testing different levels to find a best fit. Perhaps it'll work faster when we set a (low) level?
Look into turning conveyor
into an AWS Lambda function.
pingback
when process is completed, but it might be good to send updates to pingback address at major milestonesstart
, progress: converting file
, progress: uploading
, end
, error
Ideas for pingback message structure, somewhat following node's events (req.on("end")
, req.on("error")
, etc.):
// For anything in progress, like sending logs
{
status: "progress",
message: "Converting PDF with GhostScript"
}
// When process is finally completed, `message`
// will contain relevant data
{
status: "end",
message: {
// contents here (roomID, s3Dir, etc.)
}
}
// On error
{
status: "error",
message: "Network error"
}
Use restana instead of express, since we don't really need all the features of express.
We're currently extending restana's Request
interface in src/types.ts
. It would be better to use declaration merging and extend the Request
interface in types/restana/index.d.ts
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.