bradlarson / gpuimage3

GPUImage 3 is a BSD-licensed Swift framework for GPU-accelerated video and image processing using Metal.

GPUImage 3

Janie Larson

http://redqueencoder.com

@RedQueenCoder @[email protected]

Brad Larson

http://www.sunsetlakesoftware.com

@[email protected]

[email protected]

Overview

GPUImage 3 is the third generation of the GPUImage framework, an open source project for performing GPU-accelerated image and video processing on Mac and iOS. The original GPUImage framework was written in Objective-C and targeted Mac and iOS, the second iteration rewritten in Swift using OpenGL to target Mac, iOS, and Linux, and now this third generation is redesigned to use Metal in place of OpenGL.

The objective of the framework is to make it as easy as possible to set up and perform realtime video processing or machine vision against image or video sources. Previous iterations of this framework wrapped OpenGL (ES), hiding much of the boilerplate code required to render images on the GPU using custom vertex and fragment shaders. This version of the framework replaces OpenGL (ES) with Metal. Largely driven by Apple's deprecation of OpenGL (ES) on their platforms in favor of Metal, it will allow for exploring performance optimizations over OpenGL and a tighter integration with Metal-based frameworks and operations.

The API is a clone of that used in GPUImage 2, and is intended to be a drop-in replacement for that version of the framework. Swapping between Metal and OpenGL versions of the framework should be as simple as changing which framework your application is linked against. A few low-level interfaces, such as those around texture input and output, will necessarily be Metal- or OpenGL-specific, but everything else is designed to be compatible between the two.

As of this point, we are not approving enhancement requests from outside contributors. We are actively working to port all of the functionality between this version of GPUImage and previous versions. Once this task has been completed we will be happy to take community contributions.

License

BSD-style, with the full license available with the framework in License.txt.

Technical requirements

• Swift 5.5
• Xcode 13.0 or higher on Mac or iOS
• iOS: 10.0 or higher
• OSX: 10.13 or higher

General architecture

The framework relies on the concept of a processing pipeline, where image sources are targeted at image consumers, and so on down the line until images are output to the screen, to image files, to raw data, or to recorded movies. Cameras, movies, still images, and raw data can be inputs into this pipeline. Arbitrarily complex processing operations can be built from a combination of a series of smaller operations.

This is an object-oriented framework, with classes that encapsulate inputs, processing operations, and outputs. The processing operations use Metal vertex and fragment shaders to perform their image manipulations on the GPU.

Examples for usage of the framework in common applications are shown below.

Using GPUImage in a Mac or iOS application

GPUImage is provided as a Swift package. To add it to your Mac or iOS application, go to your project settings, choose Package Dependencies, and click the plus button. Enter this repository's URL in the upper-right and hit enter. GPUImage will appear as a package dependency of your project.

In any of your Swift files that reference GPUImage classes, simply add

import GPUImage

and you should be ready to go.

Note that you may need to build your project once to parse and build the GPUImage framework in order for Xcode to stop warning you about the framework and its classes being missing.

Performing common tasks

Filtering live video

To filter live video from a Mac or iOS camera, you can write code like the following:

do {
    camera = try Camera(sessionPreset:.vga640x480)
    filter = SaturationAdjustment()
    camera --> filter --> renderView
    camera.startCapture()
} catch {
    fatalError("Could not initialize rendering pipeline: \(error)")
}

where renderView is an instance of RenderView that you've placed somewhere in your view hierarchy. The above instantiates a 640x480 camera instance, creates a saturation filter, and directs camera frames to be processed through the saturation filter on their way to the screen. startCapture() initiates the camera capture process.

The --> operator chains an image source to an image consumer, and many of these can be chained in the same line.

Capturing and filtering a still photo

Functionality not completed.

Capturing an image from video

Functionality not completed.

Processing a still image

Functionality not completed.

Filtering and re-encoding a movie

Functionality not completed.

Writing a custom image processing operation

The framework uses a series of protocols to define types that can output images to be processed, take in an image for processing, or do both. These are the ImageSource, ImageConsumer, and ImageProcessingOperation protocols, respectively. Any type can comply to these, but typically classes are used.

Many common filters and other image processing operations can be described as subclasses of the BasicOperation class. BasicOperation provides much of the internal code required for taking in an image frame from one or more inputs, rendering a rectangular image (quad) from those inputs using a specified shader program, and providing that image to all of its targets. Variants on BasicOperation, such as TextureSamplingOperation or TwoStageOperation, provide additional information to the shader program that may be needed for certain kinds of operations.

To build a simple, one-input filter, you may not even need to create a subclass of your own. All you need to do is supply a fragment shader and the number of inputs needed when instantiating a BasicOperation:

let myFilter = BasicOperation(fragmentFunctionName:"myFilterFragmentFunction", numberOfInputs:1)

A shader program is composed of matched vertex and fragment shaders that are compiled and linked together into one program. By default, the framework uses a series of stock vertex shaders based on the number of input images feeding into an operation. Usually, all you'll need to do is provide the custom fragment shader that is used to perform your filtering or other processing.

Fragment shaders used by GPUImage look something like this:

#include <metal_stdlib>
#include "OperationShaderTypes.h"

using namespace metal;

fragment half4 passthroughFragment(SingleInputVertexIO fragmentInput [[stage_in]],
                                   texture2d<half> inputTexture [[texture(0)]])
{
    constexpr sampler quadSampler;
    half4 color = inputTexture.sample(quadSampler, fragmentInput.textureCoordinate);

    return color;
}

and are saved within .metal files that are compiled at the same time as the framework / your project.

Grouping operations

If you wish to group a series of operations into a single unit to pass around, you can create a new instance of OperationGroup. OperationGroup provides a configureGroup property that takes a closure which specifies how the group should be configured:

let boxBlur = BoxBlur()
let contrast = ContrastAdjustment()

let myGroup = OperationGroup()

myGroup.configureGroup{input, output in
    input --> self.boxBlur --> self.contrast --> output
}

Frames coming in to the OperationGroup are represented by the input in the above closure, and frames going out of the entire group by the output. After setup, myGroup in the above will appear like any other operation, even though it is composed of multiple sub-operations. This group can then be passed or worked with like a single operation.

Interacting with Metal

[TODO: Rework for Metal]

Common types

The framework uses several platform-independent types to represent common values. Generally, floating-point inputs are taken in as Floats. Sizes are specified using Size types (constructed by initializing with width and height). Colors are handled via the Color type, where you provide the normalized-to-1.0 color values for red, green, blue, and optionally alpha components.

Positions can be provided in 2-D and 3-D coordinates. If a Position is created by only specifying X and Y values, it will be handled as a 2-D point. If an optional Z coordinate is also provided, it will be dealt with as a 3-D point.

Matrices come in Matrix3x3 and Matrix4x4 varieties. These matrices can be build using a row-major array of Floats, or can be initialized from CATransform3D or CGAffineTransform structs.

Built-in operations

Operations are currently being ported over from GPUImage 2. Here are the ones that are currently functional:

Color adjustments

• BrightnessAdjustment: Adjusts the brightness of the image. Described in detail here.

  • brightness: The adjusted brightness (-1.0 - 1.0, with 0.0 as the default)

• ExposureAdjustment: Adjusts the exposure of the image. Described in detail here.

  • exposure: The adjusted exposure (-10.0 - 10.0, with 0.0 as the default)

• ContrastAdjustment: Adjusts the contrast of the image. Described in detail here.

  • contrast: The adjusted contrast (0.0 - 4.0, with 1.0 as the default)

• SaturationAdjustment: Adjusts the saturation of an image. Described in detail here.

  • saturation: The degree of saturation or desaturation to apply to the image (0.0 - 2.0, with 1.0 as the default)

• GammaAdjustment: Adjusts the gamma of an image. Described in detail here.

  • gamma: The gamma adjustment to apply (0.0 - 3.0, with 1.0 as the default)

• LevelsAdjustment: Photoshop-like levels adjustment. The minimum, middle, maximum, minOutput and maxOutput parameters are floats in the range [0, 1]. If you have parameters from Photoshop in the range [0, 255] you must first convert them to be [0, 1]. The gamma/mid parameter is a float >= 0. This matches the value from Photoshop. If you want to apply levels to RGB as well as individual channels you need to use this filter twice - first for the individual channels and then for all channels.

• ColorMatrixFilter: Transforms the colors of an image by applying a matrix to them

  • colorMatrix: A 4x4 matrix used to transform each color in an image
  • intensity: The degree to which the new transformed color replaces the original color for each pixel

• RGBAdjustment: Adjusts the individual RGB channels of an image. Described in detail here.

  • red: Normalized values by which each color channel is multiplied. The range is from 0.0 up, with 1.0 as the default.
  • green:
  • blue:

• WhiteBalance: Adjusts the white balance of an image.

  • temperature: The temperature to adjust the image by, in ºK. A value of 4000 is very cool and 7000 very warm. The default value is 5000. Note that the scale between 4000 and 5000 is nearly as visually significant as that between 5000 and 7000.
  • tint: The tint to adjust the image by. A value of -200 is very green and 200 is very pink. The default value is 0.

• HighlightsAndShadows: Adjusts the shadows and highlights of an image

  • shadows: Increase to lighten shadows, from 0.0 to 1.0, with 0.0 as the default.
  • highlights: Decrease to darken highlights, from 1.0 to 0.0, with 1.0 as the default.

• HueAdjustment: Adjusts the hue of an image

  • hue: The hue angle, in degrees. 90 degrees by default

• ColorInversion: Inverts the colors of an image. Described in detail here.

• Luminance: Reduces an image to just its luminance (greyscale). Described in detail here.

• MonochromeFilter: Converts the image to a single-color version, based on the luminance of each pixel

  • intensity: The degree to which the specific color replaces the normal image color (0.0 - 1.0, with 1.0 as the default)
  • color: The color to use as the basis for the effect, with (0.6, 0.45, 0.3, 1.0) as the default.

• Haze: Used to add or remove haze (similar to a UV filter)

  • distance: Strength of the color applied. Default 0. Values between -.3 and .3 are best.
  • slope: Amount of color change. Default 0. Values between -.3 and .3 are best.

• SepiaToneFilter: Simple sepia tone filter

  • intensity: The degree to which the sepia tone replaces the normal image color (0.0 - 1.0, with 1.0 as the default)

• OpacityAdjustment: Adjusts the alpha channel of the incoming image

  • opacity: The value to multiply the incoming alpha channel for each pixel by (0.0 - 1.0, with 1.0 as the default)

• LuminanceThreshold: Pixels with a luminance above the threshold will appear white, and those below will be black

  • threshold: The luminance threshold, from 0.0 to 1.0, with a default of 0.5

• Vibrance: Adjusts the vibrance of an image

  • vibrance: The vibrance adjustment to apply, using 0.0 as the default, and a suggested min/max of around -1.2 and 1.2, respectively.

• HighlightAndShadowTint: Allows you to tint the shadows and highlights of an image independently using a color and intensity

  • shadowTintColor: Shadow tint RGB color (GPUVector4). Default: {1.0f, 0.0f, 0.0f, 1.0f} (red).
  • highlightTintColor: Highlight tint RGB color (GPUVector4). Default: {0.0f, 0.0f, 1.0f, 1.0f} (blue).
  • shadowTintIntensity: Shadow tint intensity, from 0.0 to 1.0. Default: 0.0
  • highlightTintIntensity: Highlight tint intensity, from 0.0 to 1.0, with 0.0 as the default.

• LookupFilter: Uses an RGB color lookup image to remap the colors in an image. First, use your favourite photo editing application to apply a filter to lookup.png from framework/Operations/LookupImages. For this to work properly each pixel color must not depend on other pixels (e.g. blur will not work). If you need a more complex filter you can create as many lookup tables as required. Once ready, use your new lookup.png file as the basis of a PictureInput that you provide for the lookupImage property.

  • intensity: The intensity of the applied effect, from 0.0 (stock image) to 1.0 (fully applied effect).
  • lookupImage: The image to use as the lookup reference, in the form of a PictureInput.

• AmatorkaFilter: A photo filter based on a Photoshop action by Amatorka: http://amatorka.deviantart.com/art/Amatorka-Action-2-121069631 . If you want to use this effect you have to add lookup_amatorka.png from the GPUImage framework/Operations/LookupImages folder to your application bundle.

• MissEtikateFilter: A photo filter based on a Photoshop action by Miss Etikate: http://miss-etikate.deviantart.com/art/Photoshop-Action-15-120151961 . If you want to use this effect you have to add lookup_miss_etikate.png from the GPUImage framework/Operations/LookupImages folder to your application bundle.

• SoftElegance: Another lookup-based color remapping filter. If you want to use this effect you have to add lookup_soft_elegance_1.png and lookup_soft_elegance_2.png from the GPUImage framework/Operations/LookupImages folder to your application bundle.

• ColorInversion: Inverts the colors of an image

• Luminance: Reduces an image to just its luminance (greyscale).

• MonochromeFilter: Converts the image to a single-color version, based on the luminance of each pixel

  • intensity: The degree to which the specific color replaces the normal image color (0.0 - 1.0, with 1.0 as the default)
  • color: The color to use as the basis for the effect, with (0.6, 0.45, 0.3, 1.0) as the default.

• FalseColor: Uses the luminance of the image to mix between two user-specified colors

  • firstColor: The first and second colors specify what colors replace the dark and light areas of the image, respectively. The defaults are (0.0, 0.0, 0.5) amd (1.0, 0.0, 0.0).
  • secondColor:

• Haze: Used to add or remove haze (similar to a UV filter)

  • distance: Strength of the color applied. Default 0. Values between -.3 and .3 are best.
  • slope: Amount of color change. Default 0. Values between -.3 and .3 are best.

• SepiaToneFilter: Simple sepia tone filter

  • intensity: The degree to which the sepia tone replaces the normal image color (0.0 - 1.0, with 1.0 as the default)

• LuminanceThreshold: Pixels with a luminance above the threshold will appear white, and those below will be black

  • threshold: The luminance threshold, from 0.0 to 1.0, with a default of 0.5

• AdaptiveThreshold: Determines the local luminance around a pixel, then turns the pixel black if it is below that local luminance and white if above. This can be useful for picking out text under varying lighting conditions.

  • blurRadiusInPixels: A multiplier for the background averaging blur radius in pixels, with a default of 4.

• ChromaKeying: For a given color in the image, sets the alpha channel to 0. This is similar to the ChromaKeyBlend, only instead of blending in a second image for a matching color this doesn't take in a second image and just turns a given color transparent.

  • thresholdSensitivity: How close a color match needs to exist to the target color to be replaced (default of 0.4)
  • smoothing: How smoothly to blend for the color match (default of 0.1)

• Vibrance: Adjusts the vibrance of an image

  • vibrance: The vibrance adjustment to apply, using 0.0 as the default, and a suggested min/max of around -1.2 and 1.2, respectively.

• HighlightShadowTint: Allows you to tint the shadows and highlights of an image independently using a color and intensity

  • shadowTintColor: Shadow tint RGB color (GPUVector4). Default: {1.0f, 0.0f, 0.0f, 1.0f} (red).
  • highlightTintColor: Highlight tint RGB color (GPUVector4). Default: {0.0f, 0.0f, 1.0f, 1.0f} (blue).
  • shadowTintIntensity: Shadow tint intensity, from 0.0 to 1.0. Default: 0.0
  • highlightTintIntensity: Highlight tint intensity, from 0.0 to 1.0, with 0.0 as the default.

Image processing

• Sharpen: Sharpens the image

  • sharpness: The sharpness adjustment to apply (-4.0 - 4.0, with 0.0 as the default)

• GaussianBlur: A hardware-optimized, variable-radius Gaussian blur

  • blurRadiusInPixels: A radius in pixels to use for the blur, with a default of 2.0. This adjusts the sigma variable in the Gaussian distribution function.

• BoxBlur: A hardware-optimized, variable-radius box blur

  • blurRadiusInPixels: A radius in pixels to use for the blur, with a default of 2.0. This adjusts the box radius for the blur function.

• iOSBlur: An attempt to replicate the background blur used on iOS 7 in places like the control center.

  • blurRadiusInPixels: A radius in pixels to use for the blur, with a default of 48.0. This adjusts the sigma variable in the Gaussian distribution function.
  • saturation: Saturation ranges from 0.0 (fully desaturated) to 2.0 (max saturation), with 0.8 as the normal level
  • rangeReductionFactor: The range to reduce the luminance of the image, defaulting to 0.6.

• MedianFilter: Takes the median value of the three color components, over a 3x3 area

• TiltShift: A simulated tilt shift lens effect

  • blurRadiusInPixels: The radius of the underlying blur, in pixels. This is 7.0 by default.
  • topFocusLevel: The normalized location of the top of the in-focus area in the image, this value should be lower than bottomFocusLevel, default 0.4
  • bottomFocusLevel: The normalized location of the bottom of the in-focus area in the image, this value should be higher than topFocusLevel, default 0.6
  • focusFallOffRate: The rate at which the image gets blurry away from the in-focus region, default 0.2

• Convolution3x3: Runs a 3x3 convolution kernel against the image

  • convolutionKernel: The convolution kernel is a 3x3 matrix of values to apply to the pixel and its 8 surrounding pixels. The matrix is specified in row-major order, with the top left pixel being m11 and the bottom right m33. If the values in the matrix don't add up to 1.0, the image could be brightened or darkened.

• SobelEdgeDetection: Sobel edge detection, with edges highlighted in white

  • edgeStrength: Adjusts the dynamic range of the filter. Higher values lead to stronger edges, but can saturate the intensity colorspace. Default is 1.0.

• PrewittEdgeDetection: Prewitt edge detection, with edges highlighted in white

  • edgeStrength: Adjusts the dynamic range of the filter. Higher values lead to stronger edges, but can saturate the intensity colorspace. Default is 1.0.

• ThresholdSobelEdgeDetection: Performs Sobel edge detection, but applies a threshold instead of giving gradual strength values

  • edgeStrength: Adjusts the dynamic range of the filter. Higher values lead to stronger edges, but can saturate the intensity colorspace. Default is 1.0.
  • threshold: Any edge above this threshold will be black, and anything below white. Ranges from 0.0 to 1.0, with 0.8 as the default

• LocalBinaryPattern: This performs a comparison of intensity of the red channel of the 8 surrounding pixels and that of the central one, encoding the comparison results in a bit string that becomes this pixel intensity. The least-significant bit is the top-right comparison, going counterclockwise to end at the right comparison as the most significant bit.

• ColorLocalBinaryPattern: This performs a comparison of intensity of all color channels of the 8 surrounding pixels and that of the central one, encoding the comparison results in a bit string that becomes each color channel's intensity. The least-significant bit is the top-right comparison, going counterclockwise to end at the right comparison as the most significant bit.

• LowPassFilter: This applies a low pass filter to incoming video frames. This basically accumulates a weighted rolling average of previous frames with the current ones as they come in. This can be used to denoise video, add motion blur, or be used to create a high pass filter.

  • strength: This controls the degree by which the previous accumulated frames are blended with the current one. This ranges from 0.0 to 1.0, with a default of 0.5.

• HighPassFilter: This applies a high pass filter to incoming video frames. This is the inverse of the low pass filter, showing the difference between the current frame and the weighted rolling average of previous ones. This is most useful for motion detection.

  • strength: This controls the degree by which the previous accumulated frames are blended and then subtracted from the current one. This ranges from 0.0 to 1.0, with a default of 0.5.

• ZoomBlur: Applies a directional motion blur to an image

  • blurSize: A multiplier for the blur size, ranging from 0.0 on up, with a default of 1.0
  • blurCenter: The normalized center of the blur. (0.5, 0.5) by default

• ColourFASTFeatureDetection: Brings out the ColourFAST feature descriptors for an image

  • blurRadiusInPixels: The underlying blur radius for the box blur. Default is 3.0.

Blending modes

• ChromaKeyBlend: Selectively replaces a color in the first image with the second image

  • thresholdSensitivity: How close a color match needs to exist to the target color to be replaced (default of 0.4)
  • smoothing: How smoothly to blend for the color match (default of 0.1)

• DissolveBlend: Applies a dissolve blend of two images

  • mix: The degree with which the second image overrides the first (0.0 - 1.0, with 0.5 as the default)

• MultiplyBlend: Applies a multiply blend of two images

• AddBlend: Applies an additive blend of two images

• SubtractBlend: Applies a subtractive blend of two images

• DivideBlend: Applies a division blend of two images

• OverlayBlend: Applies an overlay blend of two images

• DarkenBlend: Blends two images by taking the minimum value of each color component between the images

• LightenBlend: Blends two images by taking the maximum value of each color component between the images

• ColorBurnBlend: Applies a color burn blend of two images

• ColorDodgeBlend: Applies a color dodge blend of two images

• ScreenBlend: Applies a screen blend of two images

• ExclusionBlend: Applies an exclusion blend of two images

• DifferenceBlend: Applies a difference blend of two images

• HardLightBlend: Applies a hard light blend of two images

• SoftLightBlend: Applies a soft light blend of two images

• AlphaBlend: Blends the second image over the first, based on the second's alpha channel

  • mix: The degree with which the second image overrides the first (0.0 - 1.0, with 1.0 as the default)

• SourceOverBlend: Applies a source over blend of two images

• ColorBurnBlend: Applies a color burn blend of two images

• ColorDodgeBlend: Applies a color dodge blend of two images

• NormalBlend: Applies a normal blend of two images

• ColorBlend: Applies a color blend of two images

• HueBlend: Applies a hue blend of two images

• SaturationBlend: Applies a saturation blend of two images

• LuminosityBlend: Applies a luminosity blend of two images

• LinearBurnBlend: Applies a linear burn blend of two images

Visual effects

• Pixellate: Applies a pixellation effect on an image or video

  • fractionalWidthOfAPixel: How large the pixels are, as a fraction of the width and height of the image (0.0 - 1.0, default 0.05)

• PolarPixellate: Applies a pixellation effect on an image or video, based on polar coordinates instead of Cartesian ones

  • center: The center about which to apply the pixellation, defaulting to (0.5, 0.5)
  • pixelSize: The fractional pixel size, split into width and height components. The default is (0.05, 0.05)

• PolkaDot: Breaks an image up into colored dots within a regular grid

  • fractionalWidthOfAPixel: How large the dots are, as a fraction of the width and height of the image (0.0 - 1.0, default 0.05)
  • dotScaling: What fraction of each grid space is taken up by a dot, from 0.0 to 1.0 with a default of 0.9.

• Halftone: Applies a halftone effect to an image, like news print

  • fractionalWidthOfAPixel: How large the halftone dots are, as a fraction of the width and height of the image (0.0 - 1.0, default 0.05)

• Crosshatch: This converts an image into a black-and-white crosshatch pattern

  • crossHatchSpacing: The fractional width of the image to use as the spacing for the crosshatch. The default is 0.03.
  • lineWidth: A relative width for the crosshatch lines. The default is 0.003.

• SketchFilter: Converts video to look like a sketch. This is just the Sobel edge detection filter with the colors inverted

  • edgeStrength: Adjusts the dynamic range of the filter. Higher values lead to stronger edges, but can saturate the intensity colorspace. Default is 1.0.

• ThresholdSketchFilter: Same as the sketch filter, only the edges are thresholded instead of being grayscale

  • edgeStrength: Adjusts the dynamic range of the filter. Higher values lead to stronger edges, but can saturate the intensity colorspace. Default is 1.0.
  • threshold: Any edge above this threshold will be black, and anything below white. Ranges from 0.0 to 1.0, with 0.8 as the default

• ToonFilter: This uses Sobel edge detection to place a black border around objects, and then it quantizes the colors present in the image to give a cartoon-like quality to the image.

  • threshold: The sensitivity of the edge detection, with lower values being more sensitive. Ranges from 0.0 to 1.0, with 0.2 as the default
  • quantizationLevels: The number of color levels to represent in the final image. Default is 10.0

• SmoothToonFilter: This uses a similar process as the ToonFilter, only it precedes the toon effect with a Gaussian blur to smooth out noise.

  • blurRadiusInPixels: The radius of the underlying Gaussian blur. The default is 2.0.
  • threshold: The sensitivity of the edge detection, with lower values being more sensitive. Ranges from 0.0 to 1.0, with 0.2 as the default
  • quantizationLevels: The number of color levels to represent in the final image. Default is 10.0

• EmbossFilter: Applies an embossing effect on the image

  • intensity: The strength of the embossing, from 0.0 to 4.0, with 1.0 as the normal level

• SwirlDistortion: Creates a swirl distortion on the image

  • radius: The radius from the center to apply the distortion, with a default of 0.5
  • center: The center of the image (in normalized coordinates from 0 - 1.0) about which to twist, with a default of (0.5, 0.5)
  • angle: The amount of twist to apply to the image, with a default of 1.0

• BulgeDistortion: Creates a bulge distortion on the image

  • radius: The radius from the center to apply the distortion, with a default of 0.25
  • center: The center of the image (in normalized coordinates from 0 - 1.0) about which to distort, with a default of (0.5, 0.5)
  • scale: The amount of distortion to apply, from -1.0 to 1.0, with a default of 0.5

• PinchDistortion: Creates a pinch distortion of the image

  • radius: The radius from the center to apply the distortion, with a default of 1.0
  • center: The center of the image (in normalized coordinates from 0 - 1.0) about which to distort, with a default of (0.5, 0.5)
  • scale: The amount of distortion to apply, from -2.0 to 2.0, with a default of 1.0

• StretchDistortion: Creates a stretch distortion of the image

  • center: The center of the image (in normalized coordinates from 0 - 1.0) about which to distort, with a default of (0.5, 0.5)

• SphereRefraction: Simulates the refraction through a glass sphere

  • center: The center about which to apply the distortion, with a default of (0.5, 0.5)
  • radius: The radius of the distortion, ranging from 0.0 to 1.0, with a default of 0.25
  • refractiveIndex: The index of refraction for the sphere, with a default of 0.71

• GlassSphereRefraction: Same as SphereRefraction, only the image is not inverted and there's a little bit of frosting at the edges of the glass

  • center: The center about which to apply the distortion, with a default of (0.5, 0.5)
  • radius: The radius of the distortion, ranging from 0.0 to 1.0, with a default of 0.25
  • refractiveIndex: The index of refraction for the sphere, with a default of 0.71

• Vignette: Performs a vignetting effect, fading out the image at the edges

  • center: The center for the vignette in tex coords (CGPoint), with a default of 0.5, 0.5
  • color: The color to use for the vignette (GPUVector3), with a default of black
  • start: The normalized distance from the center where the vignette effect starts, with a default of 0.5
  • end: The normalized distance from the center where the vignette effect ends, with a default of 0.75

• KuwaharaRadius3Filter: A modified version of the Kuwahara filter, optimized to work over just a radius of three pixels

• CGAColorspace: Simulates the colorspace of a CGA monitor

• Solarize: Applies a solarization effect

  • threshold: Pixels with a luminance above the threshold will invert their color. Ranges from 0.0 to 1.0, with 0.5 as the default.

gpuimage3's Issues

Write GPUimage texture for save frames and video in camera roll

After applied different filter on camera preview and create texture, need to save it.
Please can any one provide sample code for same.
Thanks

Runtime crash due to incorrect uniform buffer size

validateFunctionArguments:3485: failed assertion `Fragment Function(vignetteFragment): argument uniform[0] from buffer(1) with offset(0) and length(40) has space for 40 bytes, but argument has a length(48).'

GPUImage2 video splash screen exported on ios13 use MoiveInput and MovieOutPut

GPUImage 3 has no MovieOutput?

Hi there, I've been using GPUImage 2 for a while and realized that this GPUImage 3 version contains no MovieOutput yet, will there be an implementation for this soon?

I would love to upgrade my App to GPUImage 3, but MovieOutput is critical for my App.

Thanks!
PD: Awesome work you are doing here, I love it.

OverlayBlend with OpacityAdjustment looks not correct

Hi,

I try to add some noise overlay to an image with the following:

var picture = PictureInput(image:UIImage(named:"IMG_3331.jpg")!)
var blendImage = PictureInput(image:UIImage(named:"Noise.jpg")!)
let opacityFilter = OpacityAdjustment()
let blendFilter = OverlayBlend()

opacityFilter.opacity = mySlider.value

blendImage --> opacityFilter --> blendFilter
picture --> blendFilter --> renderView

blendImage.processImage()

When the opacityFilter.opacity is 0 then I see the Noise.jpg completely (expected that I don't see it at all), and when opacityFilter.opacity is 1 then I see the Noise.jpg only slightly as an overlay.

What is the issue?

Unable to export video to Document Directory

I am exporting video after appending filter to Document Directory, but in document directory the video is not getting exported properly.

Would you please check following source what went wrong?

class ViewController: UIViewController {
    
    @IBOutlet weak var renderView: RenderView!
    
    var movie:MovieInput!
    var writer:MovieOutput!
    var filter:LookupFilter!
    
    override func viewDidLayoutSubviews() {
        super.viewDidLayoutSubviews()
        
        let bundleURL = Bundle.main.resourceURL!
        let movieURL = URL(string:"sample_iPod.m4v", relativeTo:bundleURL)!
        
        do {
            let documentDirectory = try FileManager.default.url(for: .documentDirectory, in: .userDomainMask, appropriateFor:nil, create:true)
            let fileURL = documentDirectory.appendingPathComponent("TestVideo.m4v")
            
            movie = try MovieInput(url:movieURL, playAtActualSpeed:true)
            writer = try MovieOutput(URL: fileURL, size: Size(width: 100.0, height: 100.0))
            
            filter = LookupFilter()
            filter.lookupImage = PictureInput(image: UIImage(named: "Image")!)
            movie --> filter --> renderView
            movie.runBenchmark = true
            
            movie.addTarget(writer)
            movie.start()
            writer.startRecording()
            
            self.writer.finishRecording {
                print("Written")
            }
           
        } catch {
            print("Couldn't process movie with error: \(error)")
        }
    }
}

The front camera is upside down.

When I'm trying to swap camera to the front, the preview is rotated upside down. Any solution?

Simple Example code not build success

I open your simple code . when i try to build it has many error check this error massage

Tone Curve Request

Hi,

I have an iOS project that uses GPUImage to load .acv files to manipulate images. I was just wondering if it would be possible to implement the Tone Curve feature in GPUImage3?

Thanks

Objective C compatibility

I'm trying to maintain an old project written by a contracted developer in Objective C. Can I use this latest version? Can it be dropped into an Objc project with cocoapods?

Swift Package Manager

Using the framework by the Swift Package Manager result in a build error on Xcode 11 (but probably also on Xcode 10.2). The reason is because in the Package.swift, line 21 there is
swiftLanguageVersions: [.v4]
While the used version of swift is 4.2. To support build should be changed in
swiftLanguageVersions: [.v4_2, .v5]

Using GPUImage with GIF-image

I want to ask you about using GPUImage with GIF-image. How can this may be done?

Record Video - Picture & Movie Output

Hey,

first of all , nice work (y)
I just converted my custom filter to metal syntax and implemented them into your FilterShowCase Project. I works like charm (y).

Is is right that you can't record a video(1), or get an image from video(2) yet?!

(1) movieOutput = try MovieOutput(URL: url, size: Size(width: 1080, height: 1920), liveVideo: true)
(2) filterOperations[0].filter.saveNextFrameToURL(url, format: .jpeg)

MovieOutput, Camera: cannot be constructed - no accessible constructors

Hi Brad!

I really like your GPUImage lib.
Because of some instability, I'd like to migrate to GPUImage2 or 3.
But whenever I use MovieOutput or Camera (using GPUImage 2 or 3), I get the strange error:

When I do

      do {
           let videoCamera = try Camera(sessionPreset: AVCaptureSessionPresetHigh, location: .backFacing)
       }
       catch {
       }

I get
'Camera' cannot be constructed because it has no accessible initializers

Has anyone got an idea, what to do?

Many thanks
Regards
Werner

GPUImageMotionDetector Missing

Hi Brad, just wondering if GPUImageMotionDetector is on the cards to be added to this version?

Write to file and AssetLibary

Hello ,I'm new in GPUImage.How can I use GPUImage3 write the buffer into file like GPUImage's writer?

RenderView blocks thread when containing view's 'wantsLayer' property is set to 'true' [macOS]

The following applies to macOS (may not for iOS)

Thanks for the rapid addition of PictureInput to the repo. Found this while pipelining it into a RenderView and wanted to share as it may help fellow experimenters of GPUImage3.

TL;DR

Containing a RenderView in an NSView whose wantsLayer property is set to true causes thread blocking after a few, continuous/rapid pipeline-passes.^ This seems to be default behaviour in InterfaceBuilder that you can fix in affected XIBs' XML content, or by setting-up your views programmatically.
^ via .processImage()

Context

I'd been stumped for a while on a simple personal project with a pipeline as follows:
PictureInput --> Filter --> RenderView
Whereby the filter's value is controlled by a continuous NSSlider. After a few render-passes (via PictureInput.processImage(), the UI completely froze up and would no longer render the filter. The application used all of my CPU, and profiling it in Instruments showed the main Thread was being blocked:

After trying to replicate the SimpleImageFilter example project and still getting the issue, I found a difference between the example project's MainMenu.xib's XML content and mine — whereby the containing view (in my app) was set as follows:
<view key="contentView" wantsLayer="YES" id="xxx-xx-xxx">

I didn't realise this was something you could set in IB, and I don't think you can, so I got rid of the property on that XML line and it now performs as well as the example project.

This was a vanilla project on Xcode 10 (beta 3) — so I'm unsure if it applies to 9 or below. This wasn't something I experienced an issue with on the OpenGL-based RenderView (in GPUImage2).

I know it's not a wise idea to mess around with layers on the render view directly, but I don't know why this behaviour happens as a result of a containing view's layer status?

GPUImage3 podspec not provided.

When I created a Metal.framework based render pod library, the .metal files include other custom .h files. Then I pod lib lint, it reports error: metal_stdlib not found. So I don't know how GPUImage3 handle the problem I encountered.

Here is my sample project: https://github.com/MrEndlesswhy/DeviceOnlyMetalPod

Many thanks!

WOW-WOW-WOW: Kudos for GPUImage3

I have been using GPUImage since 2013. Kudos for GPUImage contributors, always keeping up with the leading edge of innovation. The same feeling that I have when using it in 2013 and now using Metal. Yes, it is not that complex to replace OpenGL to Metal, but the appreciation is more in how GPUImage keep going and active.

运行报错

MTLCommandEncoder Type

I have some questions about the MTLCommandEncoder type.
In the BasicOperation , all the operations like Color processing , Blend use the MTLRenderCommandEncoder to process the image and pass to next ImageConsumer. Why not use the MTLComputeCommandEncoder to do the image process work ? Since not every filter need to render to the view. and MTLComputeCommandEncoder can do data-parallel compute which maybe more effective.
Is there any considerations to use MTLRenderCommandEncoder ?

Not all effect are reachable

Why not all of effects which in README and in framework can not be reached by user's code. What I did wrong? related picture

AVSampleBufferDisplayLayer() is not returning after kill an GPUImage3 application

I have an application which is displaying video stream.
in the application AVSampleBufferDisplayLayer() is using to display separate video
I found AVSampleBufferDisplayLayer() is not return for 10 minutes at least when I kill the application after start capturing with following code

      phoneCamera = try Camera(sessionPreset: .vga640x480)
      phoneCamera.runBenchmark = false
      phoneCamera --> renderView
      phoneCamera.startCapture()

I took the code from the sample code
it happens iOS iPhone 13.x only
the problem is not happened on iOS 12.x or iOS iPad 13.x

Thanks!

Any time frame for the BulgeDistortion Filter?

Integrating a new project with GPUImage3 failed

Hello, I am new to GPUImage3, and I try to integrate GPUImage3 in a completely new Swift project, but it shows some error:

My environment is Xcode 10.1 (10B61).
I only followed the steps in Using GPUImage in a Mac or iOS application in a new created Swift project.

Maybe I should do some extra steps like add "MetalKit" or something?

front and back camera performance is different

xcode 11
ios 13
iphone 6s plus

xcode 11 have a tool, we can set the device‘s thermal state, when I set the thermal state to critical, the performance is very different between front camera and back camera。 use back camera , the fps is 30, but user front camera, the fps is only 15-20。why?

I use the SimpleVideoFilter example and add some op on it

    override func viewDidLoad() {
        super.viewDidLoad()
        do {
            operation0 = BrightnessAdjustment()
            operation0.brightness = 0
            
            operation1 = ExposureAdjustment()
            operation1.exposure = 0
            
            operation2 = OpacityAdjustment()
            operation2.opacity = 0
            
            operation3 = SaturationAdjustment()
            operation3.saturation = 0.5
            
            operation4 = GammaAdjustment()
            operation5 = HueAdjustment()
            operation6 = Haze()
            
            operation7 = GammaAdjustment()
            operation8 = HueAdjustment()
            operation9 = Haze()
            
            //camera = try Camera(sessionPreset: .hd1280x720, location: .backFacing)
            camera = try Camera(sessionPreset: .hd1280x720, location: .frontFacing)
            
            camera.runBenchmark = true
            camera.logFPS = true;
            camera --> operation0 --> operation1 --> operation2 --> operation3 --> operation4 --> operation5 --> operation6 --> operation7 --> operation8 --> operation9 --> renderView
            camera.startCapture()
        } catch {
            fatalError("Could not initialize rendering pipeline: \(error)")
        }
    }

Is gpuimage3 recommended for production use yet

Hi
Thanks for updating gpuimage to metal,
I have a number of apps using the gpuimage2 framework
I’ve experimented with gpuimage3 with some success but am not sure if you recommend using the framework in a released app yet?

How to save camera output?

Hi there,

Thanks for your great job. I'm new to GPUImage, and I'm wondering if there is something like 'output'? Since we use Camera as input, there is reasonable to be a output. However, I cannot find something like this.

Looking forward to your reply~

Will custom kernel support be integrated in the future?

Just wondering if there will be support for custom kernels.

Allow Camera's imageOrientation to be controlled externally

During the upgrade to GPUImage3 from GPUImage2, the ability to modify the orientation of the camera was removed. I need to be able to do this, and have tried by converting the imageOrientation property of Camera from a function to a variable that I set externally. It works 95% of the way. For some reason, the very first time I setup the camera, the orientation is wrong. Any subsequent initialization of the camera, seems to work fine.

@BradLarson Could you either make this change, or if not, would you be willing to pair with me remotely (of course I'd be happy to compensate you for your time), or otherwise help me make this happen? I'm totally stuck and can't do anything without your help.

a tiny bug

Contributing / Timelines

Hi there,

Firstly, thanks to both @BradLarson and @RedQueenCoder for the valiant effort to out-do OpenGL's deprecation with a Metal-oriented GPUImage framework! I know I speak for the rest of the community in expressing how excited I am to see how this progresses...

I just wanted to continue-on a little bit from here (moving the conversation into this repo now that it's active), and ask a couple of things regarding contributing and timelines:

1. What is the current scope for contributions? There is clearly a lot of functionality to come before GPUImage3's goal of becoming a 'drop-in' replacement is realised. Selfishly, I'm most dependent on PictureInput before I can fully play-around with the framework — once that's in place I'd like to contribute Metal shaders if that's something you are both looking for.
2. Relatedly to the above: Do you have a rough estimate on timelines for the various components? And if contributions to these (if wanted) would help speed this up? I don't want to come off as pushy/ungrateful, so no worries if it's all up in the air. I'm more than happy to continue with GPUImage2 for now :)

Thanks again, and I look forward to seeing this unravel!

how to support BilateralBlur?

I haven't found BilateralBlur operation.

使用GPUImage3录制视频之后没法获取视频的帧图片

Capture GPU Frame crashes on macOS Demo

Step and smoothStep

Hi Brad, I noticed a "bug" in Apple's metal, that appears to only effect iPhoneXs (probably iPhoneXr) and possibly iPhoneX, (those newer CPU/GPU chips) (I'm sure Apple will not call it a bug, but when you get two devices with different outputs, then something needs fixed)

if you use smoothstep or step, (and possibly many other math functions) in a metal shader, and you use this with "half" type rather than a "float" type, you will get a far different result than if you are say running the code in an iPhone7+ or iPhone6s.

to fix this you can change all smoothstep/step functions form say this below: which you have in some shaders:

half blendValue = smoothstep(half(uniform.thresholdSensitivity), half(uniform.thresholdSensitivity + uniform.smoothing), distance(half2(Cr, Cb), half2(maskCr, maskCb)));

to this:
float blendValue = smoothstep(float(uniform.thresholdSensitivity), float(uniform.thresholdSensitivity + uniform.smoothing), distance(float2(Cr, Cb), float2(maskCr, maskCb)));

and the results are about the same in the iPhone7+ and the iPhoneXs.

LookupFilter(AmatorkaFilter, MissEtikateFilter) result error

I test the AmatorkaFilter and MissEtikateFilter on Iphone 6 iOS 9.3.1 but the result image was strange as below image. When I tried to test on Iphone 6 iOS 12.0 the result was correct as I expected. Please help me to fix it!

"Warning: tried to add target beyond target\'s input capacity"

Hey BradLarson,
i use UICollectionView, and make config in 'didSelectItemAt' func like code blow

let color = colors[indexPath.item]  
filter.red = Float(color.red!)
filter.green = Float(color.green!)
filter.blue = Float(color.blue!)
picture --> filter --> renderView
picture.processImage()

then it'll show "Warning: tried to add target beyond target's input capacity"
how to solve it?

Request for AverageLuminanceExtractor()

Seems like there is no implementation for AverageLuminanceExtractor() which I was using from GPUImage2. I'm trying to get the overall brightness from the camera.

Any suggestions? Or, is this something that might be implemented soon?

How to initialize renderView by code?

Hi BradLarson, i am trying to init a renderView by code like let device = MTLCreateSystemDefaultDevice() renderView = RenderView.init(frame: self.view.bounds, device: device)but it doesn't work, doesn't show the camera. Can u help me? THX

Missing part of the filters?

Update from GPUImage to GPUImage3, some of filters has been removed.
"AdaptiveThreshold" and "Average luminance threshold" filter will be add in next version?

how to implement GPUImageSkinToneFilter in this Swift version

Can anyone please guide me through implementing GPUImageSkinToneFilter using this GPUImage3 framework. Thanks in advance.

Is the project in a buildable state?

I checked out master and tried buiding the iOS framework and am getting various build issues. Is the project in a buildable state?

GL Transitions

Hello!

Is there anyway I can apply these open source GLSL transitions (https://gl-transitions.com) between videos using GPUImage? If not does anyone know how I could apply transitions between videos on iOS? FFMPEG-GL would be an option for desktop but apperently not for ios https://github.com/transitive-bullshit/ffmpeg-gl-transition.

thanks

No Pressure but is GPUImage 3 going to continue?

Hi Brad and Red Queen
No pressure but just wondering if there will be continuing development of GPUImage 3? I've loved using GPUImage in the past but the shift to Metal looks essential. So this is a question flavoured with gratitude for the vast amount of work and generosity that has made GPUImage so useful in the past.

BulgeDistortion metal shader contains a number of errors, rendering it unusable

The BulgeDistortion operation and Metal shader don't have any Target Membership. When added to the GPUImage_iOS Target, the build fails due to a considerable number of errors (revealed through attempting to compile the file using xcrun metal).

AVCaptureSession

This isn't an issue, but more of a question. Is there a way to attach the GPUImageFilters to a running AVCaptureSession?

CPU & Memory usage exceed than GPUImage2 ?

Hello there!

I try to compare the CPU & Memory usage between GPUImage2 and GPUImage3 on filters group, my device is iPhone XR in HongKong, iOS system is 12.1.4. Here is my code:

    var videoCamera:GPUImageVideoCamera?
    var filter1 = GPUImageLuminanceThresholdFilter()
    let filter2 = GPUImagePixellateFilter()
    let filter3 = GPUImagePolarPixellateFilter()
    let filter4 = GPUImageAmatorkaFilter()
    
    
    override func viewDidLoad() {
        super.viewDidLoad()
        
        videoCamera = GPUImageVideoCamera(sessionPreset: AVCaptureSession.Preset.high.rawValue, cameraPosition: .back)
        videoCamera!.outputImageOrientation = .portrait;
        videoCamera?.addTarget(filter1)
        filter1.addTarget(filter2)
        filter2.addTarget(filter3)
        filter3.addTarget(filter4)
        filter4.addTarget(self.view as! GPUImageView)
        videoCamera?.startCapture()
    }

On GPUImage2 the CPU usage between 16% ~ 20%, Memory usage between 90 - 100 MB.
On GPUImage3 the CPU usage between 46% ~ 52%, Memory usage between 190 - 200 MB.

Why GPUImage3 performance worst than GPUImage2 ?

Are there any PictureOutput and MovieOutput yet?

I am using GPUimage well. Thank you. ^^
I am using GPUimage3.

GPUimage3, Can I still import filters and applied images and images?

This project hasn't been updated for a long time. Will it be released?

It looks like the author has no time to maintain it.