silence-github / bbmetalimage Goto Github PK
View Code? Open in Web Editor NEWA high performance Swift library for GPU-accelerated image/video processing based on Metal.
License: MIT License
A high performance Swift library for GPU-accelerated image/video processing based on Metal.
License: MIT License
Videos which were recorded with an iPhones (tested with: iPhone 6, iOS 12.1 and iPhone 11, iOS 13) and put in the VideoFilterVC.swift controller in the demo project are rotated by -90°.
I loaded a video recorded with the iPhone into the project bundle and replaced the default video "test_video.mov". When I run the App, the video is rotated. However, the default "test_video.mov" file is output correctly. So some videos are rotated, others are displayed correctly.
This bug can be worked around with applying a BBMetalRotateFilter(angle: 90, fitSize: true)
, but is not a good fix because it only adresses some videos.
Does anyone have a solution to this?
Hi,
Custom params for the TiltShiftFilter are not working. Not noticeable changes no matter how much you set and change the values
BBMetalTiltShiftFilter(sigma: blurValue, topFocusLevel: topValue, bottomFocusLevel: bottomValue, focusFallOffRate: fallOffRateValue)
Hi @Silence-GitHub, I just tested your demo with and without filter camera preview but the output looked aliasing. You can concentrate on the edge region to see the different with with the default camera. Hopefully you can fix it soon. Thanks!
I tried to create a filteredImage from an image, but it returns nil.
The strange thing is that this image is under the Assets folder, If I move it in the general projects file, the filter then works.
Have you any idea?
There is any way to play video and apply filter ?
i want to make a player that can show filter on run time .
User change filter on video run time . i just want to show how look if filter apply . output will be generate at last . please help me .
Just a small suggestion: organize the framework into groups. By having a long list of files it can be difficult to sort through the key methods from the filters.
i just copy video writing code from here . but its say "Asset writer can not start writing"
. i change only source video url . its play but no write
func setup() {
// Set up video writer
let filePath = NSTemporaryDirectory() + "test.mp4"
let outputUrl = URL(fileURLWithPath: filePath)
videoWriter = BBMetalVideoWriter(url: outputUrl, frameSize: BBMetalIntSize(width: 1080, height: 1920))
print(outputUrl)
// Set up video source
let sourceURL = Bundle.main.url(forResource: "video", withExtension: "mp4")!
videoSource = BBMetalVideoSource(url: sourceURL)
// Set video source audio consumer to write audio data
videoSource!.audioConsumer = videoWriter
// Set up 3 filters to process image
//let contrastFilter = BBMetalContrastFilter(contrast: 1)
//let sharpenFilter = BBMetalSharpenFilter(sharpeness: 1)
let blur = BBMetalGaussianBlurFilter(sigma: 10)
// Set up filter chain
videoSource!.add(consumer: blur)
//.add(consumer: sharpenFilter)
.add(consumer: metalView!)
videoSource?.add(consumer: videoWriter!)
// Start receiving Metal texture and writing video file
videoWriter!.start()
// Start reading and processing video frame and auido data
videoSource!.start { [weak self] (_) in
// All video data is processed
guard let self = self else { return }
// Finish writing video file
self.videoWriter!.finish {
// Do something after writing the video file
}
}
}
I use this library in my app for filter . debug mood its work fine . today i upload my app for test in testflight but no filter show and white image produce . then i try to build release mood in my device then the result same no image return from my filter function . i use this library one of my app thats live on app store and works fine . same code i run in release mood today its not working . it is version problem ? or something else . please help me .
The following code returns a faded image, not one that has its green removed.
BBMetalChromaKeyFilter(thresholdSensitivity: 0.4, smoothing: 1.0, colorToReplace: .green).filteredImage(with: image)
calling camera.switchCameraPosition()
rotates .back
camera instead of switching to .front
camera
Hey,
I am using BBMetalVideoWriter to record video. I should use default AVAssetWriter but I have to blend images with live video capturing so I am using BBMetalVideoWriter. When i start recording, camera start to lag. I would really appreciate your suggestions.
Hi! Thanks for amazing framework.
I have a few videos with alpha channel and I need to blend them into one. In background if possible.
I'm trying to implement GPUImage2 now, but it's too raw. I was able to export 18 seconds long videos in 5 seconds (without presenting it, just processing in background) with GPUImage2, but I got frame drops and pixelbuffer bugs.
Can I achieve success with BBMetalImage?
Is it possible to use BBMetalView to render images from the image processing pipeline? The docs say nothing about that.
//there is the code
print("--- writer STARTED")
videoWriter!.start()
print("--- source STARTED")
videoSource?.start(progress: { _ in
}, completion: { [ weak self ] completed in
guard let self = self else { return }
if completed {
print("--- source COMPLETED")
self.videoWriter?.finish { [weak self] in
DispatchQueue.main.async { [weak self] in
guard let self = self else { return }
print("--- writer COMPLETED")
let videoItem = AVPlayerItem(url: tempVideoUrl)
self.videoPlayer = AVPlayer(playerItem: videoItem)
self.videoPlayerLayer = AVPlayerLayer(player: self.videoPlayer)
self.videoPlayerLayer!.frame = CGRect(x: 0, y: 0, width: UIScreen.main.bounds.width - 32, height: UIScreen.main.bounds.height - 64)
self.videoPlayerLayer!.videoGravity = .resizeAspect
self.videoView.layer.addSublayer(self.videoPlayerLayer!)
self.videoPlayer?.seek(to: .zero)
self.videoPlayer!.play()
self.videoWriter = nil
self.videoSource = nil
}
}
} else {
print("--- source NOT COMPLETED")
}
})
And this is console output:
--- writer STARTED
--- source STARTED
--- source COMPLETED
I use a metalView to show the camera, and after recording I use the same metalView to play back the video that was recorded to allow editing by the user.
videoWriter.finish {
self.camera.isPaused = true
self.videoSource = BBMetalVideoSource(url: self.outputURL)
self.videoSource.playWithVideoRate = true
self.videoSource.add(consumer: self.metalView!)
self.videoSource.start { [weak self] (_) in
guard let self = self else { return }
}
}
I set playWIthVideoRate to true, tried before and after adding the metalView as consumer. So I don't know why this is happening. Any ideas?
I am unable to record media in 1:1 BBMetalImage
Hi,
I was wondering if there is a way to convert the playing video, with filters and effects applied, to a Data object. Or should I first save the video locally and then get the contents of the file? This would take a bit longer to process as I need to wait for the saving first and it seems a bit unnecessary in my opinion.
Thanks!
Hi, I got a problem when tried to turn the flash on or off. I followed this guide and it seem to work well. But the problem occured when I started to record video, the flash is automatically turn off (if currently is on). How can I fix it?
I used GPUImage earlier to blend camera capture with UNIX time stamps for recording experiments and syncing video with measurement data.
I’d like to blend images with camera stream and it works well, but I’d like to change the image on every frame, as the time stamps change. Which point could I catch the video frames asynchronously and blend it with images frame by frame and save it as one video file? Is the time information of the video (sample time) is available and accurate for using as timestamp?
Hi, I've encountered issues while using your framework. I'm pretty sure I'm missing something.
Let's say that I have several LUT filters which can be selected by the user.
I init both the metalView and the metalCamera in viewDidLoad()
:
override func viewDidLoad() {
super.viewDidLoad()
metalView = BBMetalView(frame: aFrame)
metalCamera = BBMetalCamera()
metalCamera.canTakePhoto = true
metalCamera.photoDelegate = self
metalCamera.add(consumer: metalView)
}
as written above, initially no filter is applied.
Now the user selects a filter from a "menu view" (which actually it is a collectionView):
func collectionView(_ collectionView: UICollectionView, didSelectItemAt indexPath: IndexPath) {
metalCamera.removeAllConsumers()
switch indexPath.item {
// case 0 corresponds to no filter applied = "Normal"
case 0:
bbMetalCamera.add(consumer: metalView)
currentFilter = nil
default:
let selectedFilter = filters[indexPath.item]
currentFilter = BBMetalLookupFilter(lookupTable: UIImage(named: selectedFilter.file!)!.bb_metalTexture!)
metalCamera.add(consumer: currentFilter!).add(consumer: metalView)
}
}
Until now everything works pretty good, the LUT filter is finely rendered in the metalView. Notice that currentFilter is a variable that holds the current filter that has been selected by the user.
Next step is to take a photo and save it in the Photo Library:
// Take the photo
@IBAction func takePhoto(_ sender: Any) {
metalCamera.takePhoto()
}
// Save the output
func camera(_ camera: BBMetalCamera, didOutput texture: MTLTexture) {
// In main thread
let filteredImage = currentFilter?.filteredImage(with: texture.bb_image!) ?? texture.bb_image!
let imageView = UIImageView(frame: metalView.frame)
imageView.contentMode = .scaleAspectFill
imageView.clipsToBounds = true
imageView.image = filteredImage
view.addSubview(imageView)
PHPhotoLibrary.requestAuthorization { (status) in
if status == .authorized {
do {
try PHPhotoLibrary.shared().performChangesAndWait {
PHAssetChangeRequest.creationRequestForAsset(from: filteredImage)
print("[PHPhotoLibrary]: a photo has been saved")
}
} catch let error {
print("[PHPhotoLibrary]: failed to save photo in library, ", error)
}
} else {
print("[PHPhotoLibrary]: something went wrong with permission...")
}
}
DispatchQueue.main.asyncAfter(deadline: .now() + 0.5) {
imageView.removeFromSuperview()
}
}
aaand now something breaks. Indeed, while the photo is filtered correctly and saved with the LUT filter applied, the metalCamera/metalView freezes showing the last photo captured. The only thing to get things start working again is to select another filter from the collectionView.
So, I'm supposing that I'm doing something wrong, maybe I'm adding / removing consumers in a very bad way.
May I ask you some help? :)
Hi, I'm having some issues with the transparency in PNG images. When I apply an effect to an image, the transparency is lost. Below is the code and a before/after.
Edit: How I generated the PNG might be helpful:
imageViewOne is just a UIImageView.
How do I apply filters and keep the transparency of a PNG image?
Apple now support Metal in simulator Please check this
https://developer.apple.com/documentation/metal/developing_metal_apps_that_run_in_simulator
HI team.
I'm facing some problem. I have an iOS app that let user custom their image. I apply all the filter which include the BBMetalGaussianBlurFilter at first open app and user will slide the slider to change the param of filter. Other filters work fine except the BBMetalGaussianBlurFilter. It didn't have any effect. Could you guys check this problem.
When saving images by filter chaining with ImageSource, the proper image is not saved.
Below is the code I wrote.
let beautyFilter = BBMetalBeautyFilter()
let blurFilter = BBMetalBilateralBlurFilter(distanceNormalizationFactor: 0.3, stepOffset: 0.3)
let rgbaFilter = BBMetalRGBAFilter(red: 255/255, green: 236/255, blue: 236/255, alpha: 0.1)
weak var wLastFilter = rgbaFilter
imageSource = BBMetalStaticImageSource(image: texture.bb_image!)
imageSource.add(consumer: beautyFilter)
.add(consumer: blurFilter)
.add(consumer: rgbaFilter)
.addCompletedHandler { [weak self] (_) in
guard let self = self else {return}
if let filterImage = wLastFilter?.outputTexture?.bb_image {
// save image
}
}
imageSource.transmitTexture()
when i remove all consumer and add new consumers then metal view not show video . its the main problem . any one help ?
videoSource?.removeAllConsumers()
videoSource?.add(consumer: box)
.add(consumer: lutFilter!)
.add(consumer: metalView!)
videoSource?.add(consumer: videoWriter!)
this is my code . even if i add new consumer after first time thats not effect on view . here i want to change "box" consumer to any other filter consumer . rest of consumer are same . how can i do that . if i remove first and add to consumer first index then view not show that effect .
Hi all, when compiling for iOS Simulator Xcode give the following warning, do you know how to fix this?
Of course I don't use Metal on iOS Simulator, but this is annoying :(
👎 no rule to process file '/dir/Pods/BBMetalImage/BBMetalImage/BBMetalImage/BBMetalColorDodgeBlendFilter.metal' of type 'sourcecode.metal' for architecture 'x86_64' (in target 'BBMetalImage')
When I use the static image hueFilter, black pixels will appear on the filtered image.
This happens on iPhone 5s - iPhone 10, but not on iPhone 11.
I tested the hue filter in the demo project with this photo and did not make any changes in the code.
On the filtered image you can see the black pixels:
The black pixels also occur sometimes with other filters.
Hi @Silence-GitHub, could you please add a way to track video recording duration to BBMetalVideoWriter?
Hi @Silence-GitHub, sorry for bother you again, I want to preview a video filter using your "Process Video File" example. Here is what I tried:
videoSource.add(consumer: contrastFilter) .add(consumer: lookupFilter) .add(consumer: sharpenFilter) .add(consumer: metalView)
It worked fine but you can only preview one time. So could you please add an option to replay the BBMetalVideoSource when it finished? Further more, I think you should add the playback feature to BBMetalVideoSource to support developers who want to use your library to create a video editor application.
Thank you so much!
Hello,
How can i get captureOutput ( CMSampleBuffer ) frame after effect is applied, i want to process frame for face detection. i tried to expose this method via delegate and it's working but camera feed started lagging after use of delegation
I am trying to record a video by start pressing on a record button, and when releasing I want to play back the video I just recorded in the same metalView. Below is my code, but I am facing a few hiccups:
Any ideas on how to improve this code and how I can achieve the result I want? Don't mind the AudioPlayer stuff. I also want to play the audio of the video I just recorded.
@objc func videoAction(_ sender: UILongPressGestureRecognizer) {
if(sender.state == .began) {
print("began")
startTime = CACurrentMediaTime()
isRecording = true
recordTimer = Timer.scheduledTimer(timeInterval: 0.1, target: self, selector: #selector(updateRecordProgress), userInfo: self, repeats: true)
videoWriter.start { (type) in
}
} else if(sender.state == .ended) {
print("ended")
self.recordTimer?.invalidate()
if(isRecording) {
isRecording = false
let videoURL = self.outputURL
videoWriter.finish {
DispatchQueue.main.async { [weak self] in
guard let self = self else { return }
self.camera.removeAllConsumers()
self.videoSource = BBMetalVideoSource(url: self.outputURL)
self.videoSource.playWithVideoRate = true
self.videoSource.start(progress: { (frameTime) in
// print(frameTime)
}) { [weak self] (_) in
guard let self = self else { return }
self.videoSource.add(consumer: self.metalView!)
self.audioItem = AVPlayerItem(url: self.outputURL)
self.audioPlayer = AVPlayer(playerItem: self.audioItem)
self.audioPlayer.addObserver(self, forKeyPath: "status", options: [.old, .new], context: nil)
self.playerLayer = AVPlayerLayer(player: self.audioPlayer)
self.view.layer.addSublayer(self.playerLayer!)
self.playerLayer?.frame = CGRect(x: 0, y: 0, width: 0, height: 0)
self.playerLayer?.backgroundColor = UIColor.black.cgColor
print("done")
}
}
}
}
}
}
I have added your framework to my test application and when trying to activate a filter, the app crashes on MTLCreateSystemDefaultDevice() while running on a simulator
"Fatal error: Unexpectedly found nil while unwrapping an Optional value"
I have even tried running the demo app in the simulator with the same crash.
This seems to only happen in the simulator
Can you possibly assist?
please add this project to swift package
How to blend video with image ?
It's unclear to me how to combine 'regular' filters with two-image filters.
I've tried a number of things but keep getting errors.
For example, how to asynchronously:
Add an overlay blend filter.
Use a brightness filter
If I try to take a blended image and then add brightness etc to it I get an error.
let blendedImage = BBMetalOverlayBlendFilter().filteredImage(with: imageToFilter, topBlendImage(withAlpha: 0.7))
let imageSource = BBMetalStaticImageSource(image: blendedImage) // Throws error
Hello, I issued a crash on iOS 10 for color matrix filter
This is the stack trace:
0 BBMetalImage 0x11959f6 globalinit_33_E52B5AAF7857E3D0EE385C3EE3E5AC06_func6 + 111 1 libdispatch.dylib 0x1d3fac65 dispatch_once_f + 42 2 libswiftCore.dylib 0x2bc31ef swift_once + 22 3 BBMetalImage 0x118ce89 $s12BBMetalImage0A10BaseFilterC18kernelFunctionName12useMPSKernelACSS_SbtcfcTf4gnn_n + 210 4 BBMetalImage 0x1194f4f $s12BBMetalImage0A17ColorMatrixFilterC05colorD09intensityAcA0A9Matrix4x4V_Sftcfc + 140
I currently have an application which allows users to edit photos and videos. For the videos, I can simply adjust the value I want using a UISlider, for example I can change the intensity of the Lookup Filter by simply settings the intensity variable - this automatically updates the filter and results in really fast processing.
However, for images, I have been struggling. I currently regenerate the image and assign it to the imageView where I want to display the image. It looks like this:
self.imageSource.removeAllConsumers()
self.metalFilter?.intensity = self.filterValue
self.brightnessFilter?.brightness = self.brightnessValue
DispatchQueue.global(qos: .default).async {
self.imageSource.add(consumer: self.metalFilter!)
.add(consumer: self.brightnessFilter!)
.runSynchronously = true
self.imageSource.transmitTexture()
let filteredImage = self.brightnessFilter?.outputTexture?.bb_image
DispatchQueue.main.async {
self.imagePreview.image = filteredImage
}
}
Is this how I should do it? Or could I, as I would wish for, also just update the value - somehow - where the imageView would be updated in real time? I guess this is not possible?
The method I show above used to work with just one Lookup Filter, but now, after adding the brightness filter, and when changing the value using the UISlider, it results in the following error:
-[MTLDebugComputeCommandEncoder setTexture:atIndex:]:396: failed assertion `index(31) must be < 31.'
Any advice on how I can fix/improve this? Tl;dr: I want to change the value of filters (intensity, brightness, contrast) of images and videos using UISlider in real-time.
Thanks!
With the latest update you removed BBMetalMatrix4x4 object, breaking the code on production project.
I suggest for the next version to deprecate gradually these things instead of removing them.
Another issue is that there isn't a CHANGELOG for this project, so I have to see commits to see what changes were made...
I have used this library to add filters to images or videos my users record within my app. Additionally, they can choose images from their gallery to import those and add the same filters.
I create my filters using a lookup table in Photoshop and adding additional filters, changing only colors and not blurring or sharpening, so the lookup files are OK.
It all works fine with real images, but as soon as I import a screenshot, it seems to mess up the conversion to a Data object. It looks good in the UIImage itself, but to upload it to my server, I need the Data of this image.
This is what my app looks like when opening a screenshot (crop and resize automatically).
I append the filters in an array in the most simple way possible, like this:
self.filteredImages.append(BBMetalLookupFilter(lookupTable: UIImage(named: "lookup_01")!.bb_metalTexture!).filteredImage(with: extractedImage!)!)
Also tried the complexer and async methods, but the result is the same. Then, to convert it to Data, I do this:
if let imageData = self.filteredImages[self.currentFilter].jpegData(compressionQuality: 1) {
userMedia.append(UserMedia(mediaType: .photo, mediaData: imageData, mediaThumbnail: imageData))
}
I retrieve the data in another view controller:
if let imageToShow = UIImage(data: userMedia[indexPath.item].mediaThumbnail!) {
cell.thumbnailView.image = imageToShow
}
This works perfectly fine with regular images from the gallery, video from gallery (I take a snapshot of the first frame of the video), images taken within the app and video taken within the app. The problem seems to occur only when it involves a screenshot in the gallery. I noticed a screenshot is a PNG file rather than a JPEG file, so I tried encoding it as pngData() instead of jpegData(compressionQuality: 1), but this doesn't change anything either.
Any idea on how this can be resolved? The main idea is of course to let users use "real" photos, but if they choose a screenshot for any reason whatsoever, I would like it to work as well.
I am currently wondering if it is possible to access the AVCaptureDevice of the BBMetalCamera. I am trying to implement some logic towards enabling/disabling flash, zooming in/out on the camera, ...
Any ideas?
When you filter the video in VideoFilterVC2.swift with iPhone 11 (iOS 13), the video falters during playback and becomes partially pixelated. Everything works fine on an iPhone 6 (iOS 12.1).
I used the current demo version for this test.
It's super easy to test the filters live on SwiftUI. Check this out.
import SwiftUI
import BBMetalImage
struct SwiftUIView: View {
var body: some View {
let image = UIImage(named: "sunflower")!
return VStack {
Image(uiImage: image).resizable().scaledToFit()
Image(uiImage:BBMetalBrightnessFilter(brightness: 0.15).filteredImage(with: image)!).resizable().scaledToFit()
}
}
}
Has anyone tried this framework in a release app? In debug, everything works fine for me but when I change the scheme to release, I see a crash on line return context.makeImage()
(170) in BBMetalStaticImageSource.swift
Debug output: [Unknown process name] copy_read_only: vm_copy failed: status 1.
public extension MTLTexture {
var bb_cgimage: CGImage? {
let bytesPerPixel: Int = 4
let bytesPerRow: Int = width * bytesPerPixel
var data = [UInt8](repeating: 0, count: Int(width * height * bytesPerPixel))
getBytes(&data, bytesPerRow: bytesPerRow, from: MTLRegionMake2D(0, 0, width, height), mipmapLevel: 0)
let bitmapInfo: UInt32 = CGImageAlphaInfo.premultipliedLast.rawValue
if let context = CGContext(data: &data,
width: width,
height: height,
bitsPerComponent: 8,
bytesPerRow: bytesPerRow,
space: BBMetalDevice.sharedColorSpace,
bitmapInfo: bitmapInfo) {
return context.makeImage()
}
return nil
}
var bb_image: UIImage? {
if let sourceImage = bb_cgimage {
return UIImage(cgImage: sourceImage)
}
return nil
}
}
Hi @Silence-GitHub, thanks for your great project, i integrated BBMetalImage to my project and tested build-in filter succesfully. But when I tried to create a custom filter, it failed. Bewlow is the flow i did:
super.init(kernelFunctionName: "contrastKernel")
to super.init(kernelFunctionName: "testKernel")
The program crashed with error Fatal error: Unexpectedly found nil while implicitly unwrapping an Optional value
at line 232: encoder.setComputePipelineState(computePipeline)
of file BBMetalBaseFilter.swift
How could I solve this problem. Thanks in advanced!
How to verify test chart
Hi team.
I'm make an iOS App which can let user edit their image. In my way, I apply the tool chain asynchronously. And then let user change the param like brightness, contrast... . But now I want to change the image in BBMetalStaticImageSource(image: newImage). But the lastTool.outputTexture.bb_image does not produce any image when I change the param of filter. I don't know why? Could you give me some suggestion? Thank you very much,.
Hi @Silence-GitHub,
First of all, thank you for this wonderful library. I've been successfully using it for processing images, but now I want to use it for video. I have set up a BBMetalView successfully, which shows the video I want. However, two concerns:
I use BBMetalView because I want to give my users the ability to edit their video in real-time, when the video plays back to them. This works at the moment, but only within the first playback of the video (the video is usually 30 seconds long, or shorter), and there is no audio to be heard. Any ideas?
Current setup:
self.metalView = BBMetalView(frame: CGRect(x: 0, y: self.view.center.y - ((UIScreen.main.bounds.width * 1.25) / 2), width: UIScreen.main.bounds.width, height: UIScreen.main.bounds.width * 1.25))
self.view.addSubview(self.metalView)
self.videoSource = BBMetalVideoSource(url: outputURL)
self.videoSource.playWithVideoRate = true
self.videoWriter = BBMetalVideoWriter(url: newOutputURL, frameSize: BBMetalIntSize(width: 480, height: 600))
self.videoSource.audioConsumer = self.videoWriter
self.videoSource.add(consumer: self.metalView)
self.videoSource.add(consumer: self.videoWriter)
// self.videoWriter.start()
self.videoSource.start()
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.