theblackplague / dynosharp

Play Chrome's "No Internet" Dino Game using Computer Vision.

Shaheryar Sohail

dynosharp's Issues

Unhandled exception. System.ObjectDisposedException: The object has been closed.

The object has been closed.

   at Windows.Graphics.Capture.Direct3D11CaptureFrame.get_Surface()

DynoSharp/DynoSharp/FramePool.cs

Line 56 in ad11029

direct3DSurfaceDxgiInterfaceAccess = (IDirect3DDxgiInterfaceAccess)frame.Surface;

Direct3D11CaptureFrame not converted properly to Mat.

References

DynoSharp/DynoSharp/FramePool.cs

Lines 181 to 214 in 0272773

	public static (Mat?, int width, int height) GetLatestFrameAsMat()
	{
	Stopwatch watch = new Stopwatch();
	watch.Start();
	(byte[]? frameBytes, int width, int height) = GetLatestFrameAsByteBgra();
	if (frameBytes == null) return (null, 0, 0);
	/*
	Mat frameMat = new Mat(width, height, MatType.CV_8UC4); // 8UC4: 8 unsigned bits * 4 colors (BGRA)
	Mat.Indexer<Vec4b> indexer = frameMat.GetGenericIndexer<Vec4b>();
	for (int y = 0; y < height; y++) {
	for (int x = 0; x < width; x++) {
	int bufferPos = y * width + x * 4;
	// BGRA format
	byte blue = frameBytes[bufferPos];
	byte green = frameBytes[bufferPos + 1];
	byte red = frameBytes[bufferPos + 2];
	byte alpha = frameBytes[bufferPos + 3];
	Vec4b matByteValue = new Vec4b(blue, green, red, alpha);
	indexer[y, x] = matByteValue;
	}
	}
	*/
	// 8UC4: 8 unsigned bits * 4 colors (BGRA)
	Mat frameMat = new Mat(width, height, MatType.CV_8UC4, frameBytes);
	watch.Stop();
	MatTime = watch.ElapsedMilliseconds;
	return (frameMat, width, height);
	}

Above is the function used to convert a Direct3DCaptureFrame to a opencvsharp::Mat. The point of this is so Dyno# can use the frame with the popular Computer Vision library, OpenCV.

The process for conversion is as follows: Direct3DCaptureFrame -> SharpDX::Texture2D -> byte[] which is then referenced by the Mat.

I don't believe anything is going wrong with converting it into a Texture2D, although you're welcome to read the conversion method for that here:

DynoSharp/DynoSharp/FramePool.cs

Lines 42 to 86 in 0272773

	private static Texture2D? GetLatestFrameAsTexture2D()
	{
	Stopwatch watch = new Stopwatch();
	watch.Start();
	Direct3D11CaptureFrame? frame = GetLatestFrame();
	Device device = CaptureHandler.GraphicCaptureDevice();
	if (frame == null) return null;
	// Somehow at certain times, an ObjectDisposedException comes here.
	// It makes zero sense exactly why at the moment, so using the
	// trusty "try-catch" to workaround it for now. Really odd
	// considering all the checks done previous to running this.
	IDirect3DDxgiInterfaceAccess direct3DSurfaceDxgiInterfaceAccess;
	try {
	// ReSharper disable once SuspiciousTypeConversion.Global
	direct3DSurfaceDxgiInterfaceAccess = (IDirect3DDxgiInterfaceAccess)frame.Surface;
	} catch (ObjectDisposedException exception) {
	Console.WriteLine("[EXCEPTION OCCURED]: " + exception.ObjectName + " - " + exception.Message);
	return null;
	}
	IntPtr resourcePointer = direct3DSurfaceDxgiInterfaceAccess.GetInterface(ResourceGuid);
	using Texture2D surfaceTexture = new Texture2D(resourcePointer);
	frame.Dispose();
	Texture2DDescription description = new Texture2DDescription
	{
	ArraySize = 1,
	BindFlags = BindFlags.None,
	CpuAccessFlags = CpuAccessFlags.Read,
	Format = Format.B8G8R8A8_UNorm,
	Height = surfaceTexture.Description.Height,
	Width = surfaceTexture.Description.Width,
	MipLevels = 1,
	SampleDescription = new SampleDescription(1, 0),
	Usage = ResourceUsage.Staging
	};
	Texture2D texture2DFrame = new Texture2D(device, description);
	device.ImmediateContext.CopyResource(surfaceTexture, texture2DFrame);
	watch.Stop();
	Texture2DTime = watch.ElapsedMilliseconds;
	return texture2DFrame;
	}

But I do believe something is going wrong in the conversion to a byte[] from Texture2D. The conversation is done using memcpy, by literally using pointers to copy the memory into the byte[].

I first map the memory then set relevant pointers to both the mapped memory and byte[]. The capture format is B8G8R8A8_UNorm (1 byte per 4 values) so I believe a stride of the width * 4 would work best, correct me if I'm wrong. Afterwards, I loop over the height, copying memory across into a byte[]. To parallelize the process, I use some math here (which is probably where something is going wrong, so if you can recheck it, please do).

Mapping memory:

DynoSharp/DynoSharp/FramePool.cs

Lines 134 to 135 in 0272773

	DataBox mappedMemory =
	device.ImmediateContext.MapSubresource(frame, 0, MapMode.Read, MapFlags.None);

Establishing byte[]:

DynoSharp/DynoSharp/FramePool.cs

Line 144 in 0272773

byte[] frameBytes = new byte[width * height * 4]; // 4 bytes / pixel (High Mem. Allocation)

Copying memory:

DynoSharp/DynoSharp/FramePool.cs

Lines 159 to 168 in 0272773

	CopyMemory(
	true, // Copy memory in parallel.
	false,
	0,
	height,
	sourcePointer,
	destinationPointer,
	sourceStride,
	destinationStride
	);

and

DynoSharp/DynoSharp/FramePool.cs

Lines 88 to 117 in 0272773

	private static void CopyMemory(
	bool parallel,
	bool orEqualTo,
	int from,
	int to,
	IntPtr sourcePointer,
	IntPtr destinationPointer,
	int sourceStride,
	int destinationStride)
	{
	if (!orEqualTo) to--;
	if (!parallel) {
	for (int i = from; i <= to; i++) {
	IntPtr sourceIteratedPointer = IntPtr.Add(sourcePointer, sourceStride * i);
	IntPtr destinationIteratedPointer = IntPtr.Add(destinationPointer, destinationStride * i);
	Utilities.CopyMemory(destinationIteratedPointer, sourceIteratedPointer, destinationStride);
	}
	return;
	}
	Parallel.For(from, to, i =>
	{
	IntPtr sourceIteratedPointer = IntPtr.Add(sourcePointer, sourceStride * i);
	IntPtr destinationIteratedPointer = IntPtr.Add(destinationPointer, destinationStride * i);
	Utilities.CopyMemory(destinationIteratedPointer, sourceIteratedPointer, destinationStride);
	});
	}

Expected Result:

Actual Result: