openstf / ios-minicap

iOS Minicap provides a socket interface for streaming realtime screen capture data out of iOS devices.

Warning

This project along with other ones in OpenSTF organisation is provided as is for community, without active development.

You can check any other forks that may be actively developed and offer new/different features here.

Active development has been moved to DeviceFarmer organisation.

ios-minicap

iOS Minicap provides a socket interface for streaming realtime screen capture data out of iOS devices. It is built with AVFoundation and iOS Screen mirroring feature.

Requirements

• brew install libjpeg-turbo (>=1.5 is required)
• Xcode (for the Frameworks)
• cmake
• OS X Yosemite (10.9) or higher
• iOS 8 or higher
• Lightning cable. See the list of devices.

Building

./build.sh

You should now have the binaries available in ./build.

Usage

The minicap protocol is a simple push-based binary protocol. When you first connect to the socket, you get a global header followed by the first frame. The global header will not appear again. More frames keep getting sent until you stop minicap.

Before run, please, check that:

• Node.js 6+ is used (required to run example app.js)
• the computer is trusted by the phone
• the phone screen is not turned off

You can try it using:

./run.sh

And in another window:

cd example
npm install
node app.js

Then open http://localhost:9002 in browser

When device have big FPS, minicap is sending frames to example app trough the localhost connection too quickly, so it could not draw it quick enough. That could cause frames to delay.

Global header binary format

Appears once.

Bytes	Length	Type	Explanation
0	1	unsigned char	Version (currently 1)
1	1	unsigned char	Size of the header (from byte 0)
2-5	4	uint32 (low endian)	Pid of the process
6-9	4	uint32 (low endian)	Real display width in pixels
10-13	4	uint32 (low endian)	Real display height in pixels
14-17	4	uint32 (low endian)	Virtual display width in pixels
18-21	4	uint32 (low endian)	Virtual display height in pixels
22	1	unsigned char	Display orientation
23	1	unsigned char	Quirk bitflags (see below)

Quirk bitflags

Currently, the following quirks may be reported:

Value	Name	Explanation
1	QUIRK_DUMB	Frames will get sent even if there are no changes from the previous frame. Informative, doesn't require any actions on your part. You can limit the capture rate by reading frame data slower in your own code if you wish.
2	QUIRK_ALWAYS_UPRIGHT	The frame will always be in upright orientation regardless of the device orientation. This needs to be taken into account when rendering the image.
4	QUIRK_TEAR	Frame tear might be visible. Informative, no action required. Neither of our current two methods exhibit this behavior.

Frame binary format

Appears a potentially unlimited number of times.

Bytes	Length	Type	Explanation
0-3	4	uint32 (low endian)	Frame size in bytes (=n)
4-(n+4)	n	unsigned char[]	Frame in JPG format

Generation for Xcode

Optionally you may want to use Xcode for developing or building. But still, you should use cmake as a build system.

./open_xcode.sh

ios-minicap's Issues

Be sure to run `ios-minicap --port 12345`

Run 'node app.js' and then open 'http://localhost:9002/', sometimes got this:

Listening on port 9002
Got a client
{ Error: connect ETIMEDOUT 127.0.0.1:12345
    at Object.exports._errnoException (util.js:953:11)
    at exports._exceptionWithHostPort (util.js:976:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1080:14)
  code: 'ETIMEDOUT',
  errno: 'ETIMEDOUT',
  syscall: 'connect',
  address: '127.0.0.1',
  port: 12345 }
Be sure to run `ios-minicap --port 12345`

anything wrong?

Can ios minicap be used on the ios simulator

Can you show the ios simulator in minicap? No real machine，Thanks

Why WDA don't work with iOS-minicap ?

What can I do to make that work (or what part of the code I need to take a look) ?

One-way interaction for ios-minicap

Steps to reproduce

1. Navigate to ios-minicap folder, ./run.sh
2. Open another terminal windows:
   cd example
   npm install
   node app.js

Expected results

1. What user performs on iPhone should reflect in real-time on localhost:9002
2. User can use web interface by accessing to localhost:9002 to interact to the phone

Actual results

1. OK
2. User couldn't do that

Environment(version of libraries, mac OS, iOS, etc)
iOS: 10.2
MacOS: 10.11.6
XCode: 7.3.1

Do we have a win32 port of this?

I build and use libimobiledevice with windows + msys2. is it possible to use this with msys/mingw as well?

Can't open multiple iphone screens

. I plugged 2 iPhones and start the example.
. On the http://localhost:9002, I just saw 1 iPhone displayed.

[help] why the width & height is not the real size of iphone?

Dear ios-minicap author,

Why the width & height is smaller used by CVPixelBufferRef by ios-minicap than the real screen size?

I'm try to implement a web app which can remote control ios, but the size is not the same.
ios-minicap use CVPixelBufferRef's width & height, e.g. iphone 6 plus is 1920*1080, but ios-minicap got 600 * 400 and, use 600/400 to set the canvas' size.

Why not make it the real screen size?

build fatal error:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1/stdio.h:108:15: fatal error: 'stdio.h' file not found
#include_next <stdio.h>
^~~~~~~~~
1 error generated.
make[2]: *** [CMakeFiles/ios_minicap.dir/src/minicap.cpp.o] Error 1
make[1]: *** [CMakeFiles/ios_minicap.dir/all] Error 2
make: *** [all] Error 2

==========================
xcodebuild -version
Xcode 9.2
Build version 9C40b

iphoneXs Max ios12.0 not work

$ bash run.sh
++ system_profiler SPUSBDataType
++ sed -n -E -e '/(iPhone|iPad)/,/Serial/s/ *Serial Number: *(.+)/\1/p'

• UDID=00008020001A1DD221F8002E
• PORT=12345
• RESOLUTION=400x600
• ./build/ios_minicap --udid 00008020001A1DD221F8002E --port 12345 --resolution 400x600
  EnableDALDevices
  2018-09-28 17:21:55.753 ios_minicap[18995:801524] Available devices:
  2018-09-28 17:21:55.754 ios_minicap[18995:801524] 00008020-001A1DD221F8002E
  2018-09-28 17:21:55.754 ios_minicap[18995:801524] CC253476BLYG1HNBB
  2018-09-28 17:21:55.754 ios_minicap[18995:801524] device with udid '00008020001A1DD221F8002E' not found

mac os 10.13.6

webdriveragent is running connect to ios-minicap will lead to error like "Connection to WDA timed out"

hello,i want to use ios-minicap combine with appium(when case error will user ios-minicap scrennshot),but when i connect to ios-minicap server errors happend like this:
'''
[XCUITest] Connection to WDA timed out
[iProxy] recv failed: Operation not permitted
[XCUITest] Connection to WDA timed out
'''
and i want to know why this happend? and how to avoid it thank you!

                                                                                                 best wish to you!!

frame format

I have one query about frame format.which format we are getting frame ?

Integrate IOS-MINICAP

How to integrate ios-minicap with stf? where I should work in the code to make this?
please help me.

Connection refused

++ system_profiler SPUSBDataType
++ sed -n -E -e '/(iPhone|iPad)/,/Serial/s/ *Serial Number: *(.+)/\1/p'

• UDID=b2095a48b0df41cdd338297e1e11e6a81b7b0173
• PORT=12345
• RESOLUTION=400x600
• ./build/ios_minicap --udid b2095a48b0df41cdd338297e1e11e6a81b7b0173 --port 12345 --resolution 400x600
  EnableDALDevices
  2017-10-02 10:24:10.937 ios_minicap[72393:178530] Available devices:
  2017-10-02 10:24:10.939 ios_minicap[72393:178530] b2095a48b0df41cdd338297e1e11e6a81b7b0173
  2017-10-02 10:24:10.939 ios_minicap[72393:178530] 0x1a11000005ac8509

localhost:example like$ node app.js
Listening on port 9002
Got a client
{ Error: connect ECONNREFUSED 127.0.0.1:12345
at Object.exports._errnoException (util.js:1020:11)
at exports._exceptionWithHostPort (util.js:1043:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1086:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 12345 }
Be sure to run ios-minicap on port 12345

how i can set device fps?

In my project,I need a 60fps,but the real is about 25 fps,can i control fps in code?

The screen capture of iOS devices was not clear enough.

Hi, Thank you for your great work.

I had followed the steps in Readme.md, and I already had got the screen capture of iOS devices. However, the screen capture was not clear enough. I had tried on two devices, iPhone 6s and iPad Air 2. The results was the same.

How can I get a clear screen capture?

Here is the screen capture.

minicap only support lignting cable?

example, iphone4 or iphone4s that is not lignting cable

minicap is not support them?

Getting error with iPad

When i run, i got this error:

vtDecompressionDuctDecodeSingleFrame signalled err=-8969 (err) (VTVideoDecoderDecodeFrame returned error) at /SourceCache/CoreMedia_frameworks/CoreMedia-1562.19/Sources/VideoToolbox/VTDecompressionSession.c

and the execution stops at
== Banner ==
version: 1
size: 24
pid: 31457
real width: 400
real height: 600
desired width: 400
desired height: 600
orientation:
quirks: 1

And iPhoto keeps flashing on screen. What i am doing wrong? Thanks :)

Could not start Example

I have started ios-minicap (I believe) however could not start the example.

Mac $ node app.js
/Users/serhatbolsu/Downloads/ios-minicap-master/example/app.js:8
const { BannerParser, FrameParser } = require('minicap')
^

SyntaxError: Unexpected token {
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:373:25)
at Object.Module._extensions..js (module.js:416:10)
at Module.load (module.js:343:32)
at Function.Module._load (module.js:300:12)
at Function.Module.runMain (module.js:441:10)
at startup (node.js:134:18)
at node.js:962:3

Thank you for this addition, will be great when it has parallel record support

iPhone was unable to display on the example node server.

Steps to reproduce
I'm pretty sure I followed the insturction of ios-minicap, but the server only display a blank page while nothing more.
Expected results
ios real-time screen
Actual results
a red-bordered rectangle with blank in it.
Environment(version of libraries, mac OS, iOS, etc)
iOS 11.3，iPhone 6 Plus

iOS 10.3.1 seems not supprted

when starting iOS minicap, iPhone will disconnect, but not reconnect again, other lower version not has this issue.
MacOS Sierra 10.12.4
XCode 8.3.2

When I start minicap, why does minicap interrupt the USB connection?

run ./run.sh

USB will reconnect

Unable to start ios minicap

Can it connect two or more iOS devices?

title is my question

start app.js failed

const { Parser } = require('minicap')
^

SyntaxError: Unexpected token {
at exports.runInThisContext (vm.js:53:16)
at Module._compile (module.js:387:25)
at Object.Module._extensions..js (module.js:422:10)
at Module.load (module.js:357:32)
at Function.Module._load (module.js:314:12)
at Function.Module.runMain (module.js:447:10)
at startup (node.js:140:18)
at node.js:1001:3

Device's time always be '9:41'?

why?

Is it possible to compile a executable file run on jailbreak iphone( like minicap for android )

Is it possible to compile a executable file run on jailbreak iphone( like minicap for android ), and receiving screen data through wifi or iproxy( like adb forward )? Thanks.

const { Parser } = require('minicap') SyntaxError: Unexpected token

const { Parser } = require('minicap')
^
SyntaxError: Unexpected token {
at exports.runInThisContext (vm.js:73:16)
at Module._compile (module.js:443:25)
at Object.Module._extensions..js (module.js:478:10)
at Module.load (module.js:355:32)
at Function.Module._load (module.js:310:12)
at Function.Module.runMain (module.js:501:10)
at startup (node.js:129:16)
at node.js:814:3

Error: connect ECONNREFUSED 127.0.0.1:12345

Thanks for your answer .
But,when i run node app.js,i got this error:
Listening on port 9002
Got a client
{ Error: connect ECONNREFUSED 127.0.0.1:12345
at Object.exports._errnoException (util.js:1026:11)
at exports._exceptionWithHostPort (util.js:1049:20)
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1136:14)
code: 'ECONNREFUSED',
errno: 'ECONNREFUSED',
syscall: 'connect',
address: '127.0.0.1',
port: 12345 }
Be sure to run ios-minicap on port 12345

Using IOS-minicap Connection problem

$ brew install libjpeg-turbo Updating Homebrew... Warning: jpeg-turbo is a keg-only and another version is linked to opt. Usebrew install --force` if you want to install this version

$ brew install --force libjpeg-turbo
Updating Homebrew...
Warning: jpeg-turbo-1.5.1 already installed, it's just not linked.
`

I can't use it. help

iphone4s run.sh error

iphone4s ios9.3.5
run.sh error,

I used iphoneSE ios 9.3.2 ,it succeed.So I think that the iphone4s is not supported?

++ system_profiler SPUSBDataType

++ sed -n -E -e '/(iPhone|iPad)/,/Serial/s/ *Serial Number: *(.+)/\1/p'

• UDID=8083a640b72db54d7c08c3d45726b492900bcb11

• PORT=12345

• RESOLUTION=400x600

• ./build/ios_minicap --udid 8083a640b72db54d7c08c3d45726b492900bcb11 --port 12345 --resolution 400x600

EnableDALDevices

2018-08-17 09:40:38.775 ios_minicap[2040:116228] Available devices:

2018-08-17 09:40:38.776 ios_minicap[2040:116228] CC27321Q4RPGJJM19

2018-08-17 09:40:38.776 ios_minicap[2040:116228] device with udid '8083a640b72db54d7c08c3d45726b492900bcb11' not found

when I use it,the device will reconnection

Maybe convert to 100% Swift?

I think we'd be able to get more iOS experts to participate by doing that. No reason to keep the small amount of C++.

I can't get the screen

status:
`
$ ./run.sh
++ system_profiler SPUSBDataType
++ sed -n -E -e '/(iPhone|iPad)/,/Serial/s/ *Serial Number: *(.+)/\1/p'
+ UDID=ff..........e7
+ PORT=12345
+ RESOLUTION=400x600
+ ./build/ios_minicap --udid ff..........e7 --port 12345 --resolution 400x600
EnableDALDevices
2017-06-05 18:05:29.306 ios_minicap[1829:1343756] Available devices:
2017-06-05 18:05:29.315 ios_minicap[1829:1343756] ff..........e7

`

the app status:
$ node app.js Listening on port 9002 Got a client { Error: connect ECONNREFUSED 127.0.0.1:12345 at Object.exports._errnoException (util.js:1022:11) at exports._exceptionWithHostPort (util.js:1045:20) at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1087:14) code: 'ECONNREFUSED', errno: 'ECONNREFUSED', syscall: 'connect', address: '127.0.0.1', port: 12345 } Be sure to run ios-minicap on port 12345
the localhost:9002 have not screen.
what wrong with this?

Example-Server crashes when client leaves page

Hey everyone,

everytime a client leaves the example webpage, app.js crashes.

console output:

`
Listening on port 9002
Got a client
Lost a client
/Users/user/Desktop/ios-minicap-master/example/node_modules/ws/lib/WebSocket.js:219
else throw new Error('not opened');
^

Error: not opened
at WebSocket.send (/Users/user/Desktop/ios-minicap-master/example/node_modules/ws/lib/WebSocket.js:219:16)
at Parser.onFrameAvailable (/Users/user/Desktop/ios-minicap-master/example/app.js:43:8)
at Parser.parse (/Users/user/Desktop/ios-minicap-master/example/node_modules/minicap/lib/parser.js:33:34)
at Socket.tryParse (/Users/user/Desktop/ios-minicap-master/example/app.js:56:14)
at emitNone (events.js:86:13)
at Socket.emit (events.js:185:7)
at emitReadable_ (_stream_readable.js:432:10)
at emitReadable (_stream_readable.js:426:7)
at readableAddChunk (_stream_readable.js:187:13)
at Socket.Readable.push (_stream_readable.js:134:10)`

Segfault on stop

Occasionally when the client disconnects, or ios_minicap is stopped with Ctrl+C, it segfaults. @nwlunatic any chance you could look into it?

Capturing stoped after running for about 1 minutes on iPhone6s Plus iOS10.2!

The process of ios-minicap is still running, but the web page can‘t capture the screen anymore.
Only found on iPhone6s Plus iOS10.2.
But iPhone5 ios10.0.1 and iPhone5s 9.3 runs well.

Browsers on other computer can not get Iphone screen

Steps to reproduce
./run.sh
node app.js

Expected results
Browser (Chrome ) on other computer can get iphone screen pictures

Actual results
Broswer on local mac can get iphone screen.
Browsers on other computer can not get iphone screen.
Browsers on other computer can get other http service on the mac which connected Iphone.

Environment(version of libraries, mac OS, iOS, etc)
OS X 10.11.6

./run.sh When it's finished displaying device's udid, it's stuck.

Steps to reproduce
./build.sh
./run.sh

Expected results
display device's udid, and banner

Actual results
./build.sh no problem
./run.sh When it's finished displaying device's udid, it's stuck.
if continue to open another terminal window to execute command: npm install、node app.js
there's no error but when you open http://localhost:9002 the error appeared.

Environment(version of libraries, mac OS, iOS, etc)
macbook air macOS 10.12.6
iPad Air2 iOS 11.2.6

Capture two or more devices at the same time?

@sorccu Thank you for the great works!!
I have successed on one device.
Can I capture two or more devices in two browser tabs at the same time?

What I have done:
./run.sh

to connect one device, and to connect the other one device:

./build/ios_minicap --udid $device_id --port 9001 --resolution 400x600

and it stop at:

EnableDALDevices
2016-12-02 11:01:46.403 ios_minicap[12142:338516] Available devices:
2016-12-02 11:01:46.404 ios_minicap[12142:338516] 88c2450dd52ffeb9bb24da05e8fc566bf7a2914b
2016-12-02 11:01:46.404 ios_minicap[12142:338516] 1fcab6c6785e3894ac64f4f684b603b27daa1228
2016-12-02 11:01:46.404 ios_minicap[12142:338516] CC25121R3T0F6VVD4

What should I do?

I can't use cmake when i install.

When i install, i got this error:
./open_xcode.sh: line 5: cmake: command not found
make: *** No targets specified and no makefile found. Stop.

A Mac connects two iOS devices at the same time， and use mirror function at the same time. Will it work?

Steps to reproduce

Expected results

Actual results

Environment(version of libraries, mac OS, iOS, etc)

ios-minicap on simulators

Hi guys,
First thanks for the great job you are doing with this.
I am able to use minicap with real devices but it is not accepting UDID from ios Simulators, is there any easy way to allow simulators with minicap?

Thanks in advance.

ios-minicap make device disappear

When run ios-minicap at first time, it works fine but when passed about 1 a 3 minutes de stream stopped, with no reason or warning, when i try to restart ios-minicap i lose the device and i need to disconnect and connect the device once again, to work again.
Someone pass for some problem like this one?
I am working on ios-minicap some advise or tip are always wellcome.

compared with replaykit2, which is faster?

Devices's screen is not showing up

I just got a simple frame without device's screen. Could you possibly help me to point out some clues?

Can't get connection and no error when run node app.js

Steps to reproduce

Expected results

Actual results

Environment(version of libraries, mac OS, iOS, etc)

node is v8.1.4
using iphone6

Cannot use iPhone8_ios11.3.1, This app is not authorized to use iPhone8_ios11.3.1

Steps to reproduce
$ bash run.sh
++ system_profiler SPUSBDataType
++ sed -n -E -e '/(iPhone|iPad)/,/Serial/s/ *Serial Number: *(.+)/\1/p'

• UDID=c21889e091ee2ea39dea67fc5a47f4bc7777d5ca
• PORT=12345
• RESOLUTION=400x600
• ./bin/ios_minicap --udid c21889e091ee2ea39dea67fc5a47f4bc7777d5ca --port 12345 --resolution 400x600
  EnableDALDevices
  2018-11-26 17:49:36.214 ios_minicap[1248:23629] Available devices:
  2018-11-26 17:49:36.214 ios_minicap[1248:23629] c21889e091ee2ea39dea67fc5a47f4bc7777d5ca
  2018-11-26 17:49:36.214 ios_minicap[1248:23629] CC253476BLYG1HNBB
  2018-11-26 17:49:36.235 ios_minicap[1248:23629] Error Domain=AVFoundationErrorDomain Code=-11852 "Cannot use iPhone8_ios11.3.1" UserInfo={NSLocalizedFailureReason=This app is not authorized to use iPhone8_ios11.3.1., AVErrorDeviceKey=<AVCaptureDALDevice: 0x7fd590429c20 [iPhone8_ios11.3.1][c21889e091ee2ea39dea67fc5a47f4bc7777d5ca]>, NSLocalizedDescription=Cannot use iPhone8_ios11.3.1}

Expected results

can use ios-minicap

Actual results
Cannot use iPhone8_ios11.3.1
This app is not authorized to use iPhone8_ios11.3.1

Environment(version of libraries, mac OS, iOS, etc)
macOS Mojave 10.14

build fail when I use ./build.sh

when I use the command ./build.sh
I get an error as follow:

/Users/zhangying/Downloads/ios-minicap-master/src/JpegEncoder.hpp:7:10: fatal error:
'turbojpeg.h' file not found
#include <turbojpeg.h>
^~~~~~~~~~~~~
1 error generated.

Using ios-minicap for connecting iDevices like Android

Hi @sorccu
Thank you for your great work to bring to us a free-tool like openSTF!

I see ios-minicap but I don;t know how to use it for connecting to iDevices like openSTF supports for Android devices?

Thank you!
Tu

can ios-minicap support multiple devices now?

I thought update the run.sh ,make UDID、PROT as parameter,then can support multiple devices，I dont know if it can work，I have not debuged successfully yet.

I can't see the screen of ios 11 device（iPhone 8） frequently

Steps to reproduce
1.start the run.sh in example folder
2.run node app.js
3.open browser and input http://127.0.0.1:9002/
Expected results
I can see the screen of ios device
Actual results
I can see the screen of ios 10 device
But I can't see the screen of ios 11 device frequently
Environment(version of libraries, mac OS, iOS, etc)
Mac 10.13.4
iOS 11.3.1
iPhone 8
It's seems that the ios-minicap is not very stable in iPhone 8+ ios 11 environment