gojek / darkroom

Darkroom - Yet Another Image Proxy

Introduction

Darkroom combines the storage backend and the image processor and acts as an Image Proxy on your image source.
You may implement your own Storage and Processor interfaces to gain custom functionality while still keeping other Darkroom Server functionality.
The native implementations focus on speed and resiliency.

Features

Darkroom supports several image operations which are documented here.

Installation

go get -u github.com/gojek/darkroom

Other ways to run can be found here.

Metrics Support

Darkroom supports Prometheus and StatsD for tracking and monitoring metrics. You need to specify the metrics system by adding an environment variable, METRICS_SYSTEM=prometheus/statsd

Prometheus

The application exposes the metrics at "http://<application_url>/metrics" endpoint. Since it's a pull based system, Prometheus server that is set up from docker-compose scrapes metrics from the application endpoint and its configuration can be changed in prometheus.yml.

StatsD

In order to use StatsD as your metrics system, you also need to add the following env variables,

METRICS_STATSD_STATSDADDR=hostname:port
METRICS_STATSD_PREFIX=client-prefix
METRICS_STATSD_SAMPLERATE=sample-rate
METRICS_STATSD_FLUSHBYTES=flushbytes

These are used to set up the StatsD client.

Grafana

Darkroom currently supports grafana provisioning for Prometheus based metrics.

Grafana is preconfigured with dashboards and Prometheus as the default data source:

• Name: Prometheus
• Type: Prometheus
• Url: http://prometheus:9090
• Access: proxy

Visualization of Darkroom metrics(prometheus) on Grafana:

Contributing Guide

Read our contributing guide to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes to Darkroom.

License

Darkroom is MIT licensed.

darkroom's Issues

Image load issue when crop only with width

Steps to reproduce

Open this image and you'll find that image not loaded at all.

.../darkroom/gofresh/v2/images/uploads/1c6561c7-87f8-4fa0-a480-f9386dbbed28_Shuffle-Subscription-In-Final.jpg?w=686&fit=crop&auto=compress

Observed result

Expected result

Old version support this query param

.../uploads/c0c65eee-9c64-49e9-af21-6e2378ec96fa.jpg?w=686&fit=crop&auto=compress

This query param include width, crop and compress and calculate height based on image ratio and width.

Failing Deployment

Which jobs are failing:

• The deploy part in the master branch

Since when has it been failing:

• Since add documentation commit https://github.com/gojek/darkroom/commit/21c61f2d1e3f292ff4750f7e8483e83c5c19cd13

• This is the job
  https://travis-ci.org/gojek/darkroom/builds/577348025?utm_source=github_status&utm_medium=notification

On deploying application stage

Reason for failure:

• Git exit with non-zero error code
• It happens because the runner doesn't have push access to the repository

nothing to commit, working tree clean
remote: Invalid username or password.
fatal: Authentication failed for 'https://[email protected]/gojek/darkroom.git/'
Error: Git push failed
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Anything else we need to know:

• Is it really necessary to push to publish-gh-pages branch?
• If it is, then I would recommend giving the runner push access using a new ssh credentials, we can use similar to this gitlab documentation https://docs.gitlab.com/ee/ci/ssh_keys/

Add Progressive JPEG Support

What would you like to be added

Progressive JPEG support

Why is this needed

Currently Darkroom only supports Baseline JPEG which will load from top to bottom.
There exists Progressive JPEG format that loads from low quality to full quality like this:

Comments

Refer to: https://developers.google.com/web/fundamentals/performance/optimizing-content-efficiency/automating-image-optimization/

Support GIF formatted file whenever darkroom's parameter is given

What would you like to be added

There are few cases where we want to serve GIF to our customers, but it will not work if we enable some of the parameters we support, such as: auto=compress

Why is this needed

Currently, the behaviour is to return non successful response, leading to the client to not be able to show anything. At the very least, darkroom can just return the original file in case non jpeg/png file is being accessed with darkroom's supported parameters.

[Request] Add support for fit=scale

What would you like to be added

fit=scale support
Scales the image to fit the constraining dimensions exactly. The resulting image will fill the dimensions, and will not maintain the aspect ratio of the input image.

Why is this needed

To get resized images with fit type scale.

Comments

Outdated documentation

https://github.com/gojek/darkroom/blob/master/pkg/processor/README.md seems outdated, and describes interface incorrectly. You should update it or maybe just nuke it.

Automate cluster management of darkroom instances

What would you like to be added

Add cluster management and portal to manage Darkroom deployments on Kubernetes.

Why is this needed

Darkroom should be super easy to deploy and manage, this can be done in many ways, but one of the faster ones would be to introduce a Kubernetes Operator that manages the lifecycle of a darkroom app instance. This can be backed by a management UI that can abstract away the underlying details and make cluster operation very efficient.

Comments

I did a PoC for this which can be found here.

Add Orientation Auto-fix Support

Currently, darkroom doesn't read the image exif data and when an image is rotated, darkroom doesn't fix the orientation of it. I am suggesting that darkroom should check the orientation of the image based on the exif data and return as a straight image, given the auto=compress parameter being sent.

Some example on how imgix handles this:

• http://gojek-bike.imgix.net/go_jek/image_uploader_service/image/d3bc45f5-c6cf-4f25-9875-3e918bb2996b_Mochammad_arif_nugraha.jpg?auto=compress
  vs
• http://gojek-bike.imgix.net/go_jek/image_uploader_service/image/d3bc45f5-c6cf-4f25-9875-3e918bb2996b_Mochammad_arif_nugraha.jpg

Add ability to utilise GPU Acceleration

What would you like to be added

It would be really great if Darkroom starts to support GPU acceleration for the image processing whenever possible.

Why is this needed

The performance of the application server can be greatly improved if we utilize GPU for processing images when available.

Attaching some benchmarks performed by @sohamkamani to support this feature request.

• Machine details

Machine Type	CPU Platform	GPUs
n1-standard-1 (1 vCPU, 3.75 GB memory)	Intel Ivy Bridge	1 x NVIDIA Tesla K80

• Code used

#include <opencv2/highgui.hpp>
#include "cuda.hpp"
#include <opencv2/cudawarping.hpp>
#include <opencv2/imgproc.hpp>
#include "core.hpp"
#include <iostream>
#include <ctime>

using namespace std;

int main(int argc, char ** argv) {
  string input_file = "sample.jpg";
  string output_file = "out.jpg";
  //Read input image from the disk
  cv::Mat inputCpu = cv::imread(input_file, 1);
  cv::cuda::GpuMat input(inputCpu);
  if (input.empty()) {
    cout << "Image Not Found: " << input_file << endl;
    return -1;
  }

  //Create output image
  cv::cuda::GpuMat output;
  clock_t start = clock();
  for (int i = 0; i < 20; i++) {
    cv::cuda::resize(input, output, cv::Size(0, 0), .25, 0.25, 3); // downscale 4x on both x and y
  }
  clock_t d1 = clock() - start;
  cout << "OpenCv Gpu code ran. Time:" << d1 << "\n";
  cv::Mat outputCpu;
  output.download(outputCpu);
  cv::imwrite(output_file, outputCpu);

  cv::Mat inputCpu2 = cv::imread(input_file, 1);
  cv::Mat outputCpu2;
  start = clock();
  for (int i = 0; i < 20; i++) {
    cv::resize(inputCpu, outputCpu, cv::Size(0, 0), .25, 0.25, 3);
  }
  clock_t d2 = clock() - start;
  cout << "OpenCv Cpu code ran. Time:" << d2 << "\n";
  input.release();
  output.release();
  return 0;
}

• Results

user@localhost:~$ ./a.out
OpenCv Gpu code ran. Time:226962
OpenCv Cpu code ran. Time:3231722

user@localhost:~$ ./a.out
OpenCv Gpu code ran. Time:226377
OpenCv Cpu code ran. Time:3277886

user@localhost:~$ ./a.out
OpenCv Gpu code ran. Time:217269
OpenCv Cpu code ran. Time:3254524

user@localhost:~$ ./a.out
OpenCv Gpu code ran. Time:184617
OpenCv Cpu code ran. Time:3342405

This shows that resizing an image with the GPU is 14-16 times faster than with the CPU.

Feature Status

• Resize (@anubhavp28 @ajatprabha)
• Crop
• Rotate
• Grayscale
• Blur (@jspeedluk)

Add Documentation to the project

What would you like to be added

I suggest adding separate documentation than what we have in README right now. It shall be generated from the source in /docs folder and hosted on GitHub pages on every CI release job.

Why is this needed

The code has good GoDoc coverage but there should be better documentation for the project showing how to use it. Users might not be familiar with all the features that Darkroom provides.

Comments

If you've other suggestions on how the docs should be built other than GH Pages, feel free to add them. Eg: ReadTheDocs, etc.

Unnecessary `break`s.

darkroom/pkg/metrics/statsd_collector.go

Line 73 in 982cf59

switch updateOption.Type {

darkroom/pkg/processor/native/utils.go

Line 84 in b920443

switch cropPoint {

I guess, those breaks aren't really required.

Support specifying dynamic quality for lossy images through query parameter (q)

What would you like to be added

q query parameters support in Manipulator that enables dynamic quality based on query

Why is this needed

Currently all lossy images (JPEG & WebP) that is served by the same Processor will have the same quality, it would be nice to be able to specify q query parameter

Comments

Add Overlaying Support

What would you like to be added

Adding overlaying a base image using multiple images across multiple positions (9 points)

Why is this needed

We have a case where we need to add more than 1 overlays into a single image

Comments

The logic should be almost similar as the existing watermarking feature with few adjustments

Add ability to do byte range request to storage

What would you like to be added

Darkroom storage HTTP byte-range request support

Why is this needed

When utilizing Google CDN in front of an app (which use darkroom), and the app has a large file (>= 10MB), we can forward Google CDN request to "end storage" (e.g: s3, CloudFront, etc), and forwarding response that came from "end storage" to Google CDN

Comments

PR: #49

Add Support for Prometheus

What would you like to be added

Add support for publishing metrics to Prometheus.

Why is this needed

It will be fantastic when the metrics not tied up to spikes, even better when we have Prometheus support for its better integration in other CNCF projects like Kubernetes.

Comments

-> Need to design a metrics RFC for supporting prometheus

Add Helm Deployments

What would you like to be added

helm charts

Why is this needed

Having helm charts will help in 1 command to startup darkroom, enabling more users to try, onboard, and improve darkroom. The helm chart will enable easy local development, integration, and onboarding for darkroom.

Comments

Just need to use the golang binary in the Dockerfile, and run the server with some pre-configured application, the buckets can be interfacing, or created using some gsutil commands. This will help darkroom to be used by more users.

Cleanup linter error

It will be good to clean up these linter errors before it is causing problems. Also by following some simplification and code practice, it will make the code easier to maintain

What should be cleaned up or changed

from make lint

pkg/config/config.go:86:15: exported func Source returns unexported type *config.source, which can be annoying to use

pkg/processor/native/utils_test.go:162:41: exported func NewMockImage returns unexported type *native.mockImage, which can be annoying to use

pkg/service/manipulator.go:166:52: exported func NewManipulator returns unexported type *service.manipulator, which can be annoying to use

pkg/processor/native/encoder.go:79:1: comment on exported method Encoders.Options should be of the form "Options ..."

from golangci-lint

pkg/processor/native/processor_test.go:21:2: `badImage` is unused (structcheck)
	badImage      image.Image
	^
pkg/processor/native/processor_test.go:181:11: ineffectual assignment to `err` (ineffassign)
		img, _, err := s.processor.Decode(file)
		        ^
internal/handler/ping.go:12:3: S1023: redundant `return` statement (gosimple)
		return
		^
pkg/server/api.go:63:2: S1005: '_ = <-ch' can be simplified to '<-ch' (gosimple)
	_ = <-sig
	^
pkg/service/manipulator_test.go:132:27: S1019: should use make([]byte, 10) instead (gosimple)
		ImageData: make([]byte, 10, 10),

To Cleanup makefile

-  We don't need to show the makefile command, can be achieved by using silent @

Provide any links for context

• https://github.com/golangci/golangci-lint

Do anyone want to pick this up or Can I pick this up?

Add Endpoint support for S3 Buckets

What would you like to be added

Currently, the S3 endpoint is automatically created based on the bucket name/region and it is specific to AWS only.
DigitalOcean also supports S3 based API and allows existing tools to work with their offering called Spaces.

Current behavior:

bucket:
  name: test
  region: ams3

Generated URL: test.s3.ams3.amazonaws.com

Required behaviour:

Generated URL: test.ams3.digitaloceanspaces.com

Why is this needed

This will enable DigitalOcean users to utilize Darkroom over Spaces buckets.

Comments

Minimal change is required, S3's Go SDK supports specifying Endpoint in the configuration.

Add blur functionality

What would you like to be added

Add feature to perform blur operations on images.

Comments

I am working on it

Cut down on Processor interface

What should be cleaned up or changed

Currently, the Processor is getting bulky as we add more operations to it, this creates a problem for implementing new Processor variants as they have to implement all the methods, even if they are out of scope for that processor.

The Processor should have an array of operations that it supports, each Operation can be implemented independently and then the Processor can be composed by selecting these Operations. An RFC document can be prepared to have more clarity on what needs to be changed to achieve this.

Provide any links for context

Ref:

darkroom/pkg/processor/interface.go

Lines 5 to 39 in 9186130

	// Processor interface for performing operations on image bytes
	type Processor interface {
	// Crop takes an image.Image, width, height and a Point and returns the cropped image
	Crop(image image.Image, width, height int, point Point) image.Image
	// Resize takes an image.Image, width and height and returns the re-sized image
	Resize(image image.Image, width, height int) image.Image
	// Scale takes an input image, width and height and returns the re-sized
	// image without maintaining the original aspect ratio
	Scale(image image.Image, width, height int) image.Image
	// GrayScale takes an input byte array and returns the grayscaled byte array or error
	GrayScale(image image.Image) image.Image
	// Blur takes an input byte array and returns the blurred byte array by the specified
	// radius(<=1000) or error radius must be larger than 0
	Blur(image image.Image, radius float64) image.Image
	// Watermark takes an input byte array, overlay byte array and opacity value
	// and returns the watermarked image bytes or error
	Watermark(base []byte, overlay []byte, opacity uint8) ([]byte, error)
	// Flip takes an input image and returns the image flipped. The direction of flip
	// is determined by the specified mode - 'v' for a vertical flip, 'h' for a horizontal flip and
	// 'vh'(or 'hv') for both.
	Flip(image image.Image, mode string) image.Image
	// Rotate takes an input image and returns a image rotated by the specified degrees.
	// The rotation is applied clockwise, and fractional angles are supported.
	Rotate(image image.Image, angle float64) image.Image
	// Decode takes a byte array and returns the image, extension, and error
	Decode(data []byte) (img image.Image, format string, err error)
	// Encode takes an image and extension and return the encoded byte array or error
	Encode(img image.Image, format string) ([]byte, error)
	// FixOrientation takes an image and it's EXIF orientation (if exist)
	// and returns the image with its EXIF orientation fixed
	FixOrientation(img image.Image, orientation int) image.Image
	// Overlay takes an input byte array as the base image and
	// an array of OverlayAttrs to be placed as overlays to the base image
	Overlay(base []byte, overlays []*OverlayAttrs) ([]byte, error)
	}

Broken docker images in release

Environment

Installation method: Docker
Darkroom version: >= v0.0.5
Operating system: Ubuntu x86_64
Go version ('go version' output): go version go1.12.9 linux/amd64

Steps to reproduce

docker pull gojektech/darkroom:v0.0.9
docker run -p 3000:3000 --env-file <path-to-darkroom.env> gojektech/darkroom:v0.0.9

Observed result

standard_init_linux.go:211: exec user process caused "no such file or directory"

Expected result

The server should run without errors.

Comments

This has most probably resulted from CGO_ENABLED build pipeline.