michaelpb / omnic Goto Github PK

Stateless microservice for on-the-fly thumbs and conversions of a wide variety of file types, utilizing conversion graph.

Home Page: http://omnic.michaelb.org

License: Other

Makefile 0.88% Python 88.76% HTML 5.67% JavaScript 2.19% CSS 2.18% Dockerfile 0.31%

conversion manipulation python microservice microservice-framework thumbnailer web media media-server

omnic's Introduction

Omni Converter

Mostly stateless microservice for generating on-the-fly thumbs and previews of a wide variety of file types. Comes battery-included, but designed like a framework to be extended into any arbitrary conversion pipelines.

Omni Converter (which can be shortened to OmniC or omnic) is free software, licensed under the GPL 3.0.

WIP WARNING: OmniC is in development hiatus

Docker

This repo provides a (very bulky) Dockerfile for working with OmniC. The advantage is you don't have to worry about tracking down system dependencies to take advantage of the built-in conversion graph.

Install and configure docker on your machine
Build the image: docker build .
Run the image: docker run -it -p 127.0.0.1:8080:8080 <IMAGE HASH>
Go to http://127.0.0.1:8080/admin/ to see the admin interface demo

Run the test suite: docker run -it <IMAGE HASH> py.test

Admin

From here you can paste in an URL to a resource, that OmniC will attempt to display as a thumbnail. In this example an OBJ file (3D model format) of a trumpet was pasted in, and a 200x200 thumbnail was generated:

To the right of the thumbnail it has an HTML snippet (the source-code of the thumbnail to the left), and a button that will take you to the conversion graph for that type:

Installing with pip

If you want to run it outside of Docker, you can simply install it directly on your machine, provided you have at least Python 3.5 installed. The first step is installing the Python package:

pip install omnic

If you intend to run the webserver, you will need to install a few extra dependencies:

pip install sanic jinja2 uvloop

Documentation

What is OmniC?

OmniC can do a lot of things. Most likely you will want it for making visualizations and thumbnails without (any other) backend code. It is inspired in part by White Noise -- notably, reducing complexity by serving media with Python.

On-the-fly media processing

OmniC is a web server that listens to requests like /media/thumb.png:200x200/?url=mysite.com/myimage.jpg, and then downloads the myimage.jpg file, generates a 200x200 thumbnail of it, and responds with that thumbnail.
It can also do filetype conversions like /media/PDF/?url=mysite.com/mydoc.doc for a PDF representation of a .doc file.
OmniC doesn't reinvent any wheels, instead it consists of a framework to stitch together existing CLI converters and expose them all as a microservice

Extensible conversion graph

Central to OmniC is the "Conversion Graph": you give the URL to a file, and the desired type, and it finds the shortest path even if it takes multiple conversions
OmniC comes "batteries included", and comes with converters for 3D files, documents, images, and more -- but if that's not enough, it only takes a few lines to add your own converter

Caching

Since conversion is slow, every step is cached so it is only done once, and in production it should sit behind an upstream cache or CDN
OmniC thus potentially can replace worker/queue systems with a much simpler solution, making dev environments far simpler while resembling production, and potentially reducing worker/queue scaling problems to load balancing problems

JavaScript framework

OmniC comes with some JS to smooth over the experience: For uncached media, it will initially serve a placeholder to avoid timeouts, but with the included JS snippet it will reload the relevant assets when the conversion is finished
NOT COMPLETE: OmniC also provides an optional JavaScript viewer system, hooked right into its conversion system: For example, a Word document might initially show as a JPG thumbnail, then on click show a PDF-based viewer in a modal

Replacing the build step

OmniC's concept of conversion is extremely broad and versatile: For example, it can build minified JS bundles from ES6 sources
NOT COMPLETE: Ideally, OmniC could replace most of the build-step during production deployments, making launches simply deploying new code to app servers, and everything else gets done as-needed on the first request (such as by a tester on staging)

omnic's People

Contributors

Stargazers

Watchers

Forkers

vixtir

omnic's Issues

Docker: Lock down node, compile from src

need to compile nodejs 6.x or 8.x from source (debian only supplies 4.x)

BUG: Can't use security with media

File "/home/michaelb/projects/omnic/omnic/types/resourceurl.py", line 23, in _get_basename_based_on_url
path_basename = resource_url.path_split[-2]
IndexError: list index out of range

BUG: Fix running unoconv within venv

BUG: Fix running unoconv within venv
- Check if in virtualenv and ensure system Python is used when doing
  subprocess calls

Code highlighter converter

Lots of options, the highlight package looks promising

Time-based cache grouping

Time-based grouper: In addition to MD5 structure, have
/var/tmp/omnic/date-cache/2017/08/01/ae08/. This will allow cron to
aggressively purge older cache, e.g. only maintain a couple days in prod.
This is good because IRL it won't be serving media anyway, the cache is only
good for conversion.
'clearcache' command - delete a single foreign resource from cache,
removing all generated media from that foreign resource.
- Also support date-range (requires grouping)

Refactor worker manager, worker, and task system

Refactor worker manager, worker, and task system (right now has a lot
of repetitive code)
- Make tasks be definable independent of ENUM / custom methods

Subprocess runner singleton

Wrap all subprocess calls in a singleton that manages them

Allow a queue that (async blocks) until finished, and thus have a configurable max number of subprocesses (e.g. 1 per core)
Requires some experimentation in how asyncio waits for subprocesses, etc, along with some thinking about how to properly use cores
Maybe merge this with subprocess asyncio stuff?

Locking + sanity check process

Locking + sanity check process:
- [ ] 1. Before, optionally (based on conf) do sanity Detector check
- [ ] 2. After, optionally (based on conf) do sanity Detector check
- [ ] 3. After, (required) make output all exclusively readonly,
recursively in the case of a directory
- [ ] 4. Resource base class should treat writable caches as
non-existent, thus making "in progress" files unusable to front-end
services. This step is essential for data integrity.
- [ ] Obsolete Detector Converter types, I believe

Clean up testing to use use_settings decorator etc

Older tests are a mess, clean up to be much more mocked and not as integrate-y, also use decorators

 37     def __init__(self, use_settings=None):
 38         if use_settings:
 39             # Used by testing, TODO remove
 40             self.placeholders = use_settings.PLACEHOLDERS

Generalized error and failure system

More thoughts on error

Keep a stdout.txt, stderr.txt, and error.log conversion log for each foreign resource, of all stdout/stderr generated... something like aef/a94a/e4/2b/_logs/stdout.txt
Make CLI commands that can show browse log for certain files

Need to have a generalized failure system that permits retries, conversion failures, etc, and escalation after X retries
For both ConversionGrid and ResolverGrid
Possibly handle "invalid" stuff too

(from git resolver)
44 # NOTE: Should do validation on GIT resource urls, notably disallowing
45 # HEAD, and branch names (should only be tags or commit hashes)

110 # TODO: Have failure handling here, if this command fails and can't find
111 # given hash, it should attempt to update, if that fails, then we know tha
112 # the specified "tree object" does not exist, 404

EPIC: Brand new omnic-viewer.js

JS / viewer infrastructure should be its own project

Within omnic (python pkg) there should be only refresh-thumbs.js, and a VERY LIGHTWEIGHT non-built viewer infrastructure hook that can hook into very non-opinionated global omnic-viewer callbacks if they exist
Using normal JS build system we build out a omnic-viewer npm package
Omnic could have a built-in converter that installs and serves this, but thats optional

Rethink infrastructure, make less rigid / assuming. Decouple JS dependencies from omnic. (very few people will want to trust it as a JS build system).

omnic-viewer.js - new NPM package that is used in the normal way, build using modern React etc

MISC STUFF THAT SHOULD BE BROKEN OFF INTO SEPARATE TICKETS LATER

 74 async def all_js(request):
 75     # TODO:
 76     # 1. Check if built cache is ready, if it is, include cached JS in the mix
 77     # 2. Include reload-viewers if not
 78     # 3. Include reload-thumb
 79     # 4. Send back cache headers such that only if everything is working it
 80     # gets a "perma" cache header set
 81     return templates.render(request, 'reload-viewers.js', {})

JS / viewer infrastructure

- [ ] Further improve dev environ
    - [ ] Add in 'freshen' concept
    - [ ] Make conversion path 'freshen installed_node_package' -> 'min.js'
    - Likely related to Mutable concept (?)

- [ ] Create omnic base viewer
    - [X] Move most reload-viewers.js into base viewer.js
    - [ ] Inline (thumbnail gets replaced)
    - [X] Modal (goes into "gallery" lightbox mode)
    - [ ] StaticHTML (Viewer contains a static 'prepareHTML' function,
      that outputs HTML --- possibly with injected styles, etc)

JS viewer system

-[ ] First API call looks for all img[omnic-viewer] and
img[omnic-thumb] tags and sends 1 AJAX call to check if they are all
loaded.
- [ ] If not, it will add a spinner to all thumbnails of them, and
try again in X seconds (V2: could keep running average of every
conversion path, and try again in avg * 1.5 or something)
- [ ] If loaded, it will check if omnic-viewer, then add a
hoverable (>) button in the center, and an onclick event which
will activate the appropriate viewer
- [X] If possible / cheap computationally, avoid API calls by
checking if the image is a 1x1 placeholder image (?)

Use preferred paths or aliases to force ES6 compilation

Adjust default conversion graph for JS, but use preferred paths or aliases to force ES6 compilation

- [ ] For JS, should have full input graph, e.g. can uglify plain JS files,
  but rely on "preferred paths" to ensure that it gets es5ified first (can
  configure in default settings this way)

PillowThumbConverter fails on thumb sizes the same or larger as source size

Creates empty image file

Fix URL generation in admin interface

Add hmac digest generation
Actually use host + port (presently localhost is hardcoded)
GENERALLY, just use the not-yet-written utility function

Switch to `MULTICONVERT` and async enqueues

Switch to MULTICONVERT and async enqueues

- [X] Create `MULTICONVERT` task type
- [ ] Remove sync `FUNC` task type (requires lots of test changes)
- [ ] Remove all synchronous enqueues (requires lots of test changes)

Resolver graph

Fix currently hacked together resolvers.py system with a ConversionGrid based ResolverGrid singleton, that resolves a foreign URL by possibly changing between several steps
Example: Git -> Cloned-git -> Checked out file .... OR Git -> Cloned-git -> git-tree.json... OR Git -> Cloned-git -> git-log.txt (timestamped)

converters/git.py
 10 class GitLsTreeToJson(converter.Converter):
 11     inputs = [
 12         'GIT',
 13         # TODO: Clean this up, use the detector system instead
 14         # ALSO TODO:
 15         # When finishing Resolver graph system, give option of popping out
 16         # known types in addition to unknown (for example, known type of
 17         # .git-ls-tree)
 18     ]

  6 # TODO:
  7 # * Presently hardcoded, refactor this into a "resolver" conversion system
  8 #    * Can use ConverterGraph except with Resolvers instead, which take a
  9 #    resource_url, and ensure that a ForeignResource is cached from that
 10 # * Replace with async subprocess stuff
 11 # * Possibly create async subprocess base class helpers used by both conversio
 12 # system and resolver system

Set-up demo server

Create demo site that simply embeds zoo repo

Argument typestring graph path second pass

Quality graph path pass: Second pass that ensures quality arguments are maintained throughout as necessary.

Currently, for example, STL-> PNG just makes a 500x500 PNG. But what if we want 1024x1024 thumb?

So, after a path is determined, we go through again and attach all necessary arguments. Arguments have to be carried through in order to ensure the right hashes for each Resource.

This might require an addition to the type system, or at least improvements or added structure to the cache system (e.g. dimensions is always first argument)

Markdown, text file converter

Resolver system: Git fetch to freshen mutable

Need the ability to freshen mutable git repos

URL generation utils functions

To be used by CLI command
easily generate a variety of URLs with proper encoding etc

Git repo preview generator should add in hash

Graph renderer

Presently the graph rendering system is crap

Needs new uncluttered probably custom written graph / tree renderer that is less ugly

Idea:

Each row is a converter class - all types are always horizontal
Render as very static stacked divs without zoom / etc features. Use omnic logo like in homepage

Mesh rendering improvements

Replace jsc3d renderer with something better
DXF

Sophisticate-ify domain checking, integrate into security module

Should also add in URL reversal to utils/urls.py
Have it respect the security thing in use

security.py
 51 async def check(typestring, querydata):
 52     if singletons.settings.SECURITY is not None:
 53         checker_class = singletons.settings.load('SECURITY')
 54     else:
 55         checker_class = DummySecurity
 56     checker = checker_class()
 57     url_string = querydata['url'][0]
 58     foreign_res = ForeignResource(url_string)
 59     foreign_res.validate()  # TODO move domain checks into this module
 60     await checker.check(typestring, querydata)

Clean up warnings in tests

Turn on _FORCE_PREVENT_LOGGING_DISABLE and clean up warnings in tests
(since some indicate some messy stuff being left behind during the tests)

For repo renderer, show README by default below all, like github

Asyncio Improvements

Possibly should do this w.r.t. new SubprocessManager

converter.py
 21     def convert_sync(self, in_resource, out_resource):
 22         # TODO: Make both optional (run_until_complete here)
 23         msg = 'Converter subclass must override at least convert_sync.'
 24         raise NotImplementedError(msg)

AsyncIO improvements
- Replace all file system calls with aiofiles
- Replace all subprocess.run system calls with asyncio equivalent

CLI command to generate digest, URL, etc

Generate a digest given a target media type + a source URL

Lock-down web admin to local host only by default

Documentation and site

CLI and packaging

Document the three uses:
- CLI and library - Include in other projects (e.g. Django, Celery) as a
  useful conversion utility
- Ready-to-go server - Provide docker container to spin up behind nginx,
  using env variables to configure, only providing a settings.py file if
  necessary
- Web media conversion framework - In the manner of Django, Flask, etc have
  a cookiecutter example of setting up an new project, and hooking in your
  own Services and Converters

BUG: JS compilation system has broken hack

nodejs.py
 82         if 'es5' in out:
 83             # TODO XXX obviously broken hack: Must fix by adding the
 84             # "configure" step mechanism, that gets the absolute path to
 85             # whichever programs/libs/whatever are necessary, and also what is
 86             # supported
 87             es2015_path = os.path.expanduser('~/.local/lib/node_modules'
 88                                              '/babel-preset-es2015')
 89             extra_args.extend(['--presets=%s' % es2015_path])

Misc future tasks

Integration with existing cloud infrastructure

Cloud upload feature - add '&cloud=true' option
- This would attempt to upload the final result, and then respond with
  a 301 perm redirect instead
- This produces a special _cloud_link format file, that contains the
  URL to the cloud provider (which is sent back as a perm redirect)

Render system

/render/ Service - A service that takes in ByteResources and outputs
other things.
- Example: /render/osm-geo:79.1239,32.231345,12/thumb.png:200x300/ --- This
  would render Open Street Map into an image, and then use the PNG -> Thumb
  converter to resize to the right size
- Example: /render/text3d:"Hello World!"/webm:200x300/ --- This would
  generate Hello World text as 3D shape (obj file), then use the OBJ ->
  WEBM to make a rotating image

Bundling service

NOTE: Even with the manifest.json media type, this still be useful,
for more convenient manifest format...?
zip, tar.gz, tar.bz2, 7z - all bundle formats
A more convenient manifest format that allows conversion, maybe a bundle
service to go along with it: /bundle/ Service - A service that takes in an
URL to a json manifest file, which contains an array of files and conversion
destinations to be processed.
- Example: [
  {
  "url": "http://host.com/file.png",
  "path": "media/image/file.png",
  "type": "JPG"
  },
  ...
  ]

CI + Code badges

"CI as a microservice"
Code badge generation - allow another service that is 'ci', which can be
given a (whitelisted) git repo. It then downloads that git repo and attempts
to run something (e.g. with docker). Any remaining artifacts then enter the
"media conversion graph".
Artifacts could include code coverage graphs, badges, etc
They could also include built websites, then conversion graph could include
launch
Just in time deploy: Have serving HTML (e.g. not data) actually trigger build
of entire (e.g. static) site from a repo (HTML served before ready just shows
spinner and refreshes page when ready)
Thus, omnic becomes "just in time deployment"
Obvs in real life, this would be triggered with a hook

Packaging build server Converters

Convert "tar.gz" -> rpm, deb, appimage, flatpak etc
Convert "python-project.git" -> rpm, deb, appimage, flatpak etc
Convert "electron-project.git" -> rpm, deb, appimage, flatpak etc

Mutable resources

NOTE: with Download Scheme this complication might not be needed
'mutable' concept - some foreign URLs might be mutable. All media
generated from them should have much more cautious client-side cache headers.
e.g. Git would be a ForeignMutable that is turned into a TypedResource
once 'Freshened'
Git implementation would be bare repo
ForeignMutable - Behaves differently than foreign resource, can be
transformed into a TypedResource after being 'locked'
- freshen() - Mutates cached resource
- Store a log of all freshen instances in cache, like history.txt
- get_history() - Returns an abbreviated list of version-strings,
  one of which can be selected, generally current and all previous
  instances of freshen.
- get_locked(LockString) - Returns a LockedTypedResource
LockedForeignMutable - for git would represent a specific hash, for
non-VCS, would be a timestamp
- get_resource(view_selector) - Returns either a LockedResource resource.
  - For VCS, you would use PathViewSelector
  - For Chat history, you would use TimeRangeViewSelector
- get_typed_resource(view_selector) - Possibly later add this, since for
  chat history this would be used, since the resulting resource is of known
  type
- LockedResource - Source from a ForeignMutable, it would
  simply need to be linked from its true location, can be

Chat stream (and timeseries?) as a mutable data source

Needs more thoughts, but OmniChat would utilize omnic to render HTML
backlogs. It would have ws servers waiting to push to a (postgres?)
database, and trigger reloads for clients as new messages come in. However,
at the core would be a multi-page app (progressive enhancement), meaning
the core would not require JS (JS would just add the sugar of
auto-refresh). This is to make it feel faster, simpler, and more "solid"
than sluggish competitors.

Possibly useful front-end fw: http://intercoolerjs.org/docs.html
More OmniChat ideas:
- "channels" are just "pinned" resources - could be any resource, from an
  image to a document
- There's a "auto-pin creator" that in reality just stores a URL to be added
- Full URL:
  - /media/chatlogs.html/?url=postgres://host.com/saved-query:omnichat_logs<select:WHERE channel=general><select:WHERE date > xyz AND date < xyz>save-as:chatlogs.json

Low prio

Possibly add in web-worker hook system to base viewer (?) so I
can add PDF.js to the demo
- Build 1 JS package, but detect if in web worker or not, and
  execute different code path for each
/ws/ Service - Exposes all other services via a websocket RPC-like
interface. Useful for creating more involved progress-bar UIs etc and
pre-caching longer-running processes
- Still would have no trust, simply exposing the public HTTP interface in
  another way, for more involved front-end applications
- Example:
  - < "media", {url: "foo.com/bar.pdf", typestring="thumb.png"}
  - > "downloaded", {url: "foo.com/bar.pdf"}
  - > "converting", {url: "foo.com/bar.pdf", type="PNG"}
  - > "converting", {url: "foo.com/bar.pdf", type="thumb.png"}
  - > "ready", {url: "foo.com/bar.pdf"}
  - < "bundle", {url: "foo.com/bar.json"}
  - > "in-progress", {url: "foo.com/bar.json"}
  - > "downloaded", {url: "foo.com/bar.json"}
  - > "downloaded", {url: "foo.com/bar.json"}

Likely never useful

Checked in cached built version of viewer
- CACHE_ALIAS = {'viewers.min.js': {'/path/to/checkout/...'}}
Pool should probably be configured, and always run with runserver
Redis-only commands:
- omnic runworker -- runs a worker-only process
- omnic runserverworker -- runs a process that is both server AND worker
- omnic runmulti --worker=1 --server=1 --serverworker=2 -- runs X
  processes of the given types
@localserver context manager (?)
- Allows "mini local server" that has can make fake queries via
  custom routing
- Something like: await f.client.get('/media/?qs=abc')
- General merger of client convert code (should actually spin up
  workers) + server routes

Rethink argument system

TypeString - Relates to #38

ForeignUrl - relates to #23

Current notes

More thoughts: piggy back on profile code!

Perhaps the way to think of this is "automatic conversion profiles". If no conversion profile(s) are specified, we get a default one that is the result of back-propagating all relevant arguments based on which converters can accept which arguments.

New resolver system

Resolver system should just use existing converter system (e.g. ExecConverter would do the same thing). Simply register a new ConversionGraph
- ~~Generates faux type strings something like: git:34fea34fe48db34,/some/path -> git:34fea34fe48db34 -> file:/some/path~~
- Better system: git_cloned -> git_updated -> git_file
- ResolverConverters have a `detect_resolver_type
All interior resources would be MutableResource, while the final would be ForeignResource
Converter system has the following concepts built in:
- "understood arguments"
- If any converter in a conversion path understands an argument, that argument is propagated backward to that point
- For example: thumb.png:500x500 gets normalized to thumb.png:resolution=500x500 and for the STL->PNG conversion we'd have STL -> PNG:resolution=500x500

Original notes

The present TypeString argument system is not sufficient, it needs
Arguments need to have type checking and association with converters
They need to be carried through and be universally defined (for example, resolution would be understood by many converters)
Converters need to supply hints as to what arguments they accept (and when a graph is drawn with types with arguments, it chooses paths that include converters)

Nginx for serving directly from cache

Simple route (preferred): Softlinks + nginx rewrite rules

Rewrite static URL to serve directly from a softlinked nginx static cache

/var/tmp/omnic/nginx_cache/9f/2e/4f/..etc/9f2e4f -> /var/tmp/omnic/resource/.../file.jpg

Add complex rewrite rules that md5 the incoming URL or parts of it and then redirect to the nginx cache, which is populated at the end of every successful conversion step.

Complex route (if necesary) OpenResty

OpenResty script that uses same hashing algo as python to very quickly serve up files with nginx BEFORE they even hit Omnic servers

Don't worry about HMAC digests (although maybe 404 obviously bogus requests), since cache miss will happen anyway
Should be VERY short logic: Just take URL parameter, apply MD5 transformation, check cache location, if exists AND read-only, serve file, else fallback to omnic

Find and remove all TODOs, and TODO.md

Redis based queue

Allow swapping out asyncio's gimmick-y queuing for a custom-made
Redis-based async queue (should be easy) e.g. using: aioredis package

Convert Manifest Downloader into a resolver

Also, use CURL

Cookiecutter script

Cookiecutter for creating a new project
Gets ONLY the converters that you need all set up
When creating a new project (with cookiecutter?), a Dockerimage is
automatically created that only includes the software in use
- The docker image derives from one of the others

Full e2e Docker-based test suite

Build CLI-level tests that include mini-scaffold test apps for full e2e (all tests in tests/ directory should be merely unit and integration tests)

Should do binary-level (if possible) comparisons of testing converting a variety of file types
Run as a daily cronjob on travis so it gets alerted if every the Dockerfile becomes broken

Auto-refresh script tags that are marked with omnic

Come up with a standardized docwrite / window._OMNIC system that allows refresh thumbs to properly refresh script tags also

Constant-based TypeString hierarchy system

It's becoming increasingly clear I need to build a constant-based TypeString hierarchy, such that a bundle.js or es5.js is a type of JS file, and thus gets proper mimetypes, etc, while remaining more specific.

This should somehow get hooked into the resolver ConversionGraph AND the Detection / sanity check system

Reqs:

Modifications should be able to be made at LOW LEVELS while preserving HIGH LEVEL concepts. For example, git-tree.json is a form of json and thus can be converted to, say, yaml, HOWEVER, it will remain a git-tree.yml then. JSON -> YAML should not know about git-tree, and git-tree should not know about yaml
Another example: thumb.png is a PNG, and can be convered into thumb.jpg while preserving its characteristiscs
Also keep track of lossy vs nonlossy conversions
Maybe "format" vs. "context"...?

builtin/converters/git.py

 58 class InlineJsVariable(converter.Converter):
 59     inputs = [
 60         'git-tree.html',
 61         # 'HTML',
 62         # TODO: once there is a type hierarchy, then this should handle any
 63         # sub-type of HTML
 64     ]

Examples for different languages

Have examples for HMAC + SHA1 of different languages

PHP
Node.js
Ruby
Python (obvs)
Golang

Add ffmpeg converters for audio and video

Update Dockerfile to catch up with demo

ffmpeg
ffmpegthumbnailer
pandoc
highlight or something (?)

Allow conversions to self

omnic-clear-cache-cascade

'omnic-clear-cache-cascade' application - special separate (?)
application that would hook into any upstream proxies AND all siblings, fully
wiping the cache for a certain foreign resource
MAYBE expose a API that has allows encrypted, expiring requests to clear caches?

Repo renderer: Use individual item sha to improve caching

Only regen previews when an item is updated with a new sha