octalmage / robotjs

Node.js Desktop Automation.

Node.js Desktop Automation. Control the mouse, keyboard, and read the screen.

RobotJS supports Mac, Windows, and Linux.

This is a work in progress so the exported functions could change at any time before the first stable release (1.0.0). Ideas?

Check out some of the cool things people are making with RobotJS! Have your own rad RobotJS project? Feel free to add it!

Contents

• Installation
• Examples
• API
• Building
• Plans
• Progress
• FAQ
• License

Installation

Install RobotJS using npm:

npm install robotjs

It's that easy! npm will download one of the prebuilt binaries for your OS.

You can get npm here if you don't have it installed.

If you need to build RobotJS, see the building section. Instructions for Electron.

Examples

Mouse

// Move the mouse across the screen as a sine wave.
var robot = require("robotjs");

// Speed up the mouse.
robot.setMouseDelay(2);

var twoPI = Math.PI * 2.0;
var screenSize = robot.getScreenSize();
var height = (screenSize.height / 2) - 10;
var width = screenSize.width;

for (var x = 0; x < width; x++)
{
	y = height * Math.sin((twoPI * x) / width) + height;
	robot.moveMouse(x, y);
}

Keyboard

// Type "Hello World" then press enter.
var robot = require("robotjs");

// Type "Hello World".
robot.typeString("Hello World");

// Press enter.
robot.keyTap("enter");

Screen

// Get pixel color under the mouse.
var robot = require("robotjs");

// Get mouse position.
var mouse = robot.getMousePos();

// Get pixel color in hex format.
var hex = robot.getPixelColor(mouse.x, mouse.y);
console.log("#" + hex + " at x:" + mouse.x + " y:" + mouse.y);

Read the Wiki for more information!

API

The RobotJS API is hosted at https://robotjs.io/docs/syntax.

Building

Please ensure you have the required dependencies before installing:

• Windows
  • windows-build-tools npm package (npm install --global --production windows-build-tools from an elevated PowerShell or CMD.exe)
• Mac
  • Xcode Command Line Tools.
• Linux
  • Python (v2.7 recommended, v3.x.x is not supported).
  • make.
  • A C/C++ compiler like GCC.
  • libxtst-dev and libpng++-dev (sudo apt-get install libxtst-dev libpng++-dev).

Install node-gyp using npm:

npm install -g node-gyp

Then build:

node-gyp rebuild

See the node-gyp readme for more details.

Plans

• √ Control the mouse by changing the mouse position, left/right clicking, and dragging.
• √ Control the keyboard by pressing keys, holding keys down, and typing words.
• √ Read pixel color from the screen and capture the screen.
• Find an image on screen, read pixels from an image.
• Possibly include window management?

Progress

Module	Status	Notes
Mouse	100%	All planned features implemented.
Keyboard	100%	All planned features implemented.
Screen	85%	Image search, pixel search.
Bitmap	0%	Saving/opening, png support.

FAQ

Does RobotJS support global hotkeys?

Not currently, and I don't know if it ever will. I personally use Electron/NW.js for global hotkeys, and this works well. Later on I might add hotkey support or create a separate module. See #55 for details.

Can I take a screenshot with RobotJS?

Soon! This is a bit more complicated than the rest of the features, so I saved it for last. Luckily the code is already there, I just need to write the bindings, and I've already started. Subscribe to #13 for updates.

Why is <insert key> missing from the keyboard functions?

We've been implementing keys as we need them. Feel free to create an issue or submit a pull request!

How about multi-monitor support?

The library doesn't have explicit multi-monitor support, so anything that works is kind of on accident. Subscribe to #88 for updates.

For any other questions please submit an issue.

Story

I'm a huge fan of AutoHotkey, and I've used it for a very long time. AutoHotkey is great for automation and it can do a bunch of things that are very difficult in other languages. For example, it's imagesearch and pixel related functions are hard to reproduce on Mac, especially in scripting languages. These functions are great for automating apps that can't be automated like Netflix. This has never been a big deal since I've always used Windows at work, but for the past few years I've been using Mac exclusively.

I like AutoHotkey, but I like Node.js more. By developing RobotJS I get an AutoHotkey replacement on Mac (finally!), and I get to use my favorite language.

TLDR: There's nothing like AutoHotkey on Mac, so I'm making it.

License

MIT

Based on autopy. Maintained by Jason Stallings.

robotjs's Issues

Add support for right/middle click.

Modify mouseClick so the button can be specified.

Trigger Cursor style changes

I was wondering if it would be possible to add the ability to trigger the different cursors like wait, pointer, cross. I know in C++ for windows it can be done using the Set Cursor function. I'm open to submitting a pull request I just wanted to check and see if that's even doable.

In linux it could possibly be done like the answer to this stackoverflow post.

Create Windows Installer.

It would be awesome to have an easy Windows installer that installs all of the dependices, including Node.js if needed. This would let people that are just interested in the automation use RobotJS.

I want this project to be as accessible as AutoHotkey.

Windows Support.

Edit by octalmage:

Windows is completely supported! Just install from npm.

Known issues:

All known issues resolved!

• getPixelColor returns "000000" no matter what in my VM, not sure if it's broken. Fixed in #43.
• The sqrt functions in rgb.h cause errors, currently commented out in the windows-2 branch. Fixed in 8730e94.
• The mssleep function needs to be cross platform. Replaced with microsleep.
• moveMouse and moveMouseSmooth wouldn't work in my virtual machine. I'm guessing this is because it's a VM, but it would be cool if someone could confirm. This is working!

---

Original comment:
Hey Octa, I'm trying to use your add on for a node webkit project I'm working on but wasn't able to get very far since the package gets stuck on my windows machine while running the "node-gyp configure" command, just sitting there endlessly. I thought possible your NPM might not be up to date with this github repo, but still the same behavior when I cloned the repo and built it from scratch.

I notice in binding.gyp that your only condition is 'OS' == "mac". I'm not intimately familiar with building native node modules, so does this mean the package simply does not support window builds right now?

add MouseWheel

rt MouseWheel

Include examples in repository?

I'm wondering if we should include some examples in the repository. I always like it when projects do this, it makes it easy to test the library and see what it can do. But it also clutters the repository, which I don't want to do.

Thoughts?

Finalize function naming scheme.

Need to decide function names.

bridge to image-processing

It would be cool if image processing is enabled! (a bot for a game, an intelligent auto app QA agent)
Grabbing bitmap of screen would be a great starting point instead of repeating to get single pixel over region.

Keyboard API complete?

Great work getting this working on windows. NPM install is now no problem for me.

I'm wondering if the keyboard api is complete (the readme indicates it is)? In AutoHotkey the send function accepts a minilanguage like: "{End}Sincerely,{enter 4}John Smith" Any plans to implement that in robotjs... or should write my own wrapper?

install fail on ubuntu 14.04

Hi,

Hope you are doing great.You have developed great package.I am using ubuntu 14.04.
I configure your package but when i run command ( node-gyp build ) it is giving error.

make: Entering directory /home/piyush/app/Nodejs_Application/robotjs/build' CC(target) Release/obj.target/robotjs/src/mouse.o ../src/mouse.c:12:35: fatal error: X11/extensions/XTest.h: No such file or directory #include <X11/extensions/XTest.h> ^ compilation terminated. make: *** [Release/obj.target/robotjs/src/mouse.o] Error 1 make: Leaving directory/home/piyush/app/Nodejs_Application/robotjs/build'
gyp ERR! build error
gyp ERR! stack Error: make failed with exit code: 2
gyp ERR! stack at ChildProcess.onExit (/usr/local/lib/node_modules/node-gyp/lib/build.js:269:23)
gyp ERR! stack at ChildProcess.emit (events.js:98:17)
gyp ERR! stack at Process.ChildProcess._handle.onexit (child_process.js:810:12)
gyp ERR! System Linux 3.13.0-54-generic
gyp ERR! command "node" "/usr/local/bin/node-gyp" "build"
gyp ERR! cwd /home/piyush/app/Nodejs_Application/robotjs
gyp ERR! node -v v0.10.33
gyp ERR! node-gyp -v v2.0.2

something like this . can you explain me what did this error come or what have i done wrong in installation?

OSX El Captain compatibility issue

It seems that using OS X El Captain beta the mouse move doesn't work, mouse get is pretty working well

If you need any log, feel free to ask

Redundant Include Guards

I was going through the code and found that all header files have both #ifndef and #pragma once. This seems to be redundant. There are no significant performance improvements as far as I have seen.

According to wikipedia the #pragma once is supported in most compilers. Why not just use that alone ?
Wiki article link : https://en.wikipedia.org/wiki/Pragma_once#Portability

Blocking Delay function

Some of the times I am having issues were I will have to wait to do something. Currently I am setting a SetTimeout function to achieve this delay. Since we are dealing with C modules why not add such a function to it ? It would use the Sleep in the C module. This will provide some level of code complexity reduction.

Though I may be wrong about it but this is a need that just cropped up.

Cancelling running robotjs methods

Is there a way of cancelling running methods such as mouseMoveSmooth() when the node program exits?

At the moment there is no way of stopping the robotjs from running and pressing buttons even if process.exit(1) is called...

Text Expansion

Would there be any way to do text expansion with robotjs like you can with AutoHotkey?

I was able to get robotjs working with electron so I could listen for global keypresses, but electron (and nw.js) only seems to be able to listen for key combinations, not key sequences.

It would be amazing if we could simply turn any key-presses / mouse-movements into events and/or observables so we could deal with them purely in node-land ( and perhaps use something fancy like http://reactivex.io/ so we could just map/filter/etc the keypresses )

Robotjs is/will be great for sending keyboard commands etc., but I think we'd need a reliable way to listen for these events globally to come close to AutoHotkey. Is there perhaps a node module I haven't found that strictly listens for keypresses / mouse movements globally and outputs them as events ? Surely someone must have written something like this already.

Get Screen Dimensions

Having screen dimensions would be great for calculating where on the screen the mouse is or should be.

This could be returned as a new call in screen or returned with the object from getMousePos() so you would have x, y, width, and height parameters.

Another possibility is passing into getMousePos({percent:true}) and have it return 2 percentages rather than a fixed location { x: 50.44, y: 22.34 }
.

[Linux] buffer overflow in getPixelColor.

When I use getPixelColor, buffer overflow occurred.
code:

var hex = robot.getPixelColor(mouse.x, mouse.y);

*** buffer overflow detected ***: node terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f6fc2dc6e37]
/lib/x86_64-linux-gnu/libc.so.6(+0x109d30)[0x7f6fc2dc5d30]
/lib/x86_64-linux-gnu/libc.so.6(+0x10941b)[0x7f6fc2dc541b]
/lib/x86_64-linux-gnu/libc.so.6(__snprintf_chk+0x78)[0x7f6fc2dc52f8]
/home/smomoo/test/robotjs/node_modules/robotjs/build/Release/robotjs.node(_Z13getPixelColorRKN2v820FunctionCallbackInfoINS_5ValueEEE+0xea)[0x7f6fc22b75da]
node(_ZN2v88internal25FunctionCallbackArguments4CallEPFvRKNS_20FunctionCallbackInfoINS_5ValueEEEE+0x9c)[0x82a11c]
node[0x84c1ff]
[0x30cdd3b060a2]

system info:

node v0.12.0
Ubuntu 12.04.1

If you want I can paste memory map here.

Throw error if mouse event is out of bounds.

Key Toggles

It is extremely common for people to want to send commands such as "Ctrl+v" to toggle a paste command. I am in need of doing this now in my project, Quicklip, and i'd love to use robotjs for this, tho it doesnt appear that the current API allows for it.

It would be extremely helpful to allow for an API like the following:

robotjs.hold('ctrl').tap('v');

Can't build on win8

Did I forget to do something or is this a version mismatch?

Shift case not working in Linux

I was planning to do a "Coding ghost" showcase which automagically recreates code in sublime given a directory of a code base.

It seems like

robot.typeString("~!@#$%^&*()_+");

Outputs :

`12345678()-=

You can checkout

https://github.com/jekku/creeper/tree/issue_link/no_shifts_unix

reserved identifier violation

I would like to point out that identifiers like "_MMSize" and "_MMPointArray" do not fit to the expected naming convention of the C++ language standard.
Would you like to adjust your selection for unique names?

install fails on ubuntu

Edit by octalmage:

If RobotJS fails to build, install these packages:

sudo apt-get install libxtst-dev libpng++-dev

---

Original comment:
There's an error, some header .h file missing?

solution is to install X developer tools with:

sudo apt-get install xorg-dev

Perhaps add this to the README file.
never mind, doesn't work on Ubuntu :(

Use nan to be compatible with newer Node.js versions.

I need to use nan to make RobotJS compatible with newer versions of node.js.

Get RobotJS working with nw.js 0.11.*

Currently it's only working with nw.js 0.10.*.

Completion of error handling

Would you like to add more error handling for return values from functions like the following?

• fclose ⇒ newMMBitmapFromPNG
• fseek ⇒ newMMBitmapFromBMP

npm install failed under windows

here is npm-debug.log
http://pastebin.com/pfgVDBrf

negative mouse coordinates returned as 2^64

When using multiple screens, zero is defined as top left of the main screen, which means that everything to it's left/top should have negative x/y coordinates accordingly. However, those are returned as 2^64.

This image shows the output of the script below as I move the mouse to the left-most screen, and then to the top of it. (node index.js >> data.txt and tail -f data.txt)

Could this simply be a problem with integers becoming unsigned in the C code?

'use strict'

var robot = require('robotjs')

function encodeMouse(mouse) {
  return '[' + mouse.x + ',' + mouse.y + ']'
}

var mouse
var prevCode

setInterval(function() {
  var code = encodeMouse(robot.getMousePos())
  if (code !== prevCode) {
    console.log(code)
  }
  prevCode = code
}, 100)

Implement screen capture, save, open, and search.

Just an issue to track progress and ideas.

Multiple monitor support

On OSX, calling getMousePos() when on a monitor above my main screen returns: { x: 293, y: 18446744073709552000 }

Requesting the mouse position through cocoa returns the mouse position relative to the origin of the main screen, for example: { x: 293, y: -291 }

io.js v3 support.

nan has been updated to v2 to support io.js v3, but it requires significant code changes.

Use node-pre-gyp.

Installing and compiling RobotJS requires Python and Visual Studio on Windows and this takes a while to get setup. Instead we can provide pre-built binaries that can be installed using npm. This will greatly improve the end-user experience.

Logo/Mascot.

I'd like a little robot mascot for RobotJS.

I'm a fan of the PhoneGap robots:

and these:

Definitely something vector and friendly.

Printing UTF-8 Characters with accents

Hi,

when you try to typeString a character with accents, the returned string is not correct.

For example, when I try robot.typeString('à') it returns ￃﾠ.

The same happens when I try to print an ñ, it returns ￃﾱ.

Is there a way to make this work?

Electron (Atom-shell) compatibility break

Hi :)

I want to use your library with electron but I can't get it to work. All I get when I launch my program is:

[3412:0627/165819:INFO:CONSOLE(98)] "Uncaught Error: Module version mismatch. Expected 44, got 14.
Error: Module version mismatch. Expected 44, got 14.
    at Error (native)
    at Object.module.(anonymous function) (ATOM_SHELL_ASAR.js:137:20)
    at Object.module.(anonymous function) [as .node] (ATOM_SHELL_ASAR.js:137:20)
    at Module.load (module.js:351:32)
    at Function.Module._load (module.js:306:12)
    at Module.require (module.js:361:17)
    at require (module.js:380:17)
    at Object.<anonymous> (/Users/perso/Sources/js/hugo/soixantesouris/node_modules/robotjs/index.js:1:80)
    at Module._compile (module.js:426:26)
    at Object.Module._extensions..js (module.js:444:10)", source: /usr/local/lib/node_modules/electron-prebuilt/dist/Electron.app/Contents/Resources/atom.asar/renderer/api/lib/remote.js (98)

I'm running this on Mac OS 10.10.3, with node 0.12.4. I tested the lib with a simple script (i.e not with electron) and it worked perfectly. I also tried to build screencat (a program which uses electron along with robotjs) on my computer and I get the same error, so it seems that it is related with electron itself.

Do you have any idea ? @maxogden maybe ?

Implement callbacks

I want to do something like

robot.moveMouseSmoothly(x, y, function(coordinates, oldCoordinates) {
  console.log('Done moving');
});

Would this be possible to do for all methods, so we can chain them properly, and so the methods are not blocking.

How to capture keyboard events?

Hello, It is possible to receive the key that was pressed when typing? something like onkeypress event in the browser.

My use case would be implement a keylogger :)

moveMouse() is broken on Windows 10

robot.moveMouse(10, 10);
console.log(robot.getMousePos());

This code must return { x: 10, y: 10 }
It returns { x: 9, y: 9 }

Not always pos-1, for example if moveMouse(100,100) then getMousePos gives back 98,98

Windows 10 Build 10240 x64
NodeJS 0.12.7 x64
RobotJS 0.2.3 installed via npm

Add toggle function for keyboard.

Allow holding keys down, and releasing them.

On Keyboard Event typeString there is no option for backspace and enter events.

it would be better if you have keyboard enter and backspace events

Mouse button toggling!

Allow holding buttons down so stuff can be dragged.

keyTap doesn't work with capital letters.

This code doesn't work:

var robot = require("robotjs");
robot.keyTap("H");

but this does:

var robot = require("robotjs");
robot.keyTap("h");

This also works:

var robot = require("robotjs");
robot.typeString("H");

[Use case] webbymouse

Hi, FYI I've made a Desktop app (via node webkit) which use robotjs to control desktop mouse.
It bundle with express server so user can open a mobile browser and the mobile device turn into touchpad immediately.

https://github.com/gasolin/webbymouse

Possibly sleep after mouse/key events.

Apparently on some machines a delay is needed after a mouse move or click before the next action will work. This is probably an OS limitation.

Since this is in C I can use usleep, I just need to find what the limitation is. 2 miliseconds works on Mac.

Support for double click

Hi Jason,

Is there any (plan for) support of double click in robotjs?
You'd say executing two sequential clicks should do it, but that doesn't work for me.
Also doing mouseToggle with "down", "up" twice just functions as a click.

Any experience with this? Double click would be quite nice.

Cheers, Thijs

node-gyp rebuild fails

Error while installing robotjs

gyp ERR! build error 
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/local/lib/node_modules/npm/node_modules/node-gyp/lib/build.js:269:23)
gyp ERR! stack     at ChildProcess.emit (events.js:110:17)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (child_process.js:1074:12)
gyp ERR! System Linux 3.2.0-86-generic
gyp ERR! command "node" "/usr/local/lib/node_modules/npm/node_modules/node-gyp/bin/node-gyp.js" "rebuild"
gyp ERR! cwd /home/itsashis4u/projects/robot/node_modules/robotjs
gyp ERR! node -v v0.12.7
gyp ERR! node-gyp -v v2.0.1
gyp ERR! not ok 
npm ERR! Linux 3.2.0-86-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "robotjs@latest" "--save"
npm ERR! node v0.12.7
npm ERR! npm  v2.11.3
npm ERR! code ELIFECYCLE

npm ERR! [email protected] install: `node-gyp rebuild`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] install script 'node-gyp rebuild'.
npm ERR! This is most likely a problem with the robotjs package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node-gyp rebuild
npm ERR! You can get their info via:
npm ERR!     npm owner ls robotjs
npm ERR! There is likely additional logging output above.

installing Robotjs in windows

See the attached image, I am unable to install it on windows. Tell me if you need additional information about the steps.

Strip unneeded methods

Would it be possible to strip unneeded methods? (ie: keep only .getMousePos())

I'd try to use this in Powder Player (NW.js App) in an attempt to fix chromium's mouse hovering issues.

I would only need to know when the mouse is outside the window (something that chromium itself has serious problems with), all other methods seem very intrusive for unrelated apps (ie: not screen sharing, automation)

Thank you in advance. :)

Domain Name.

I'm going to start working on branding and I'd like to find a good domain. This will later host the forums and possibly a blog.

On Twitter I have @RobotJavaScript, but robotjavascript.com is taken. And of course robotjs.com is taken. robotjs.io is open but it's a little pricy and I feel like I want something more creative. That's probably what I'll fallback on if I can't find anything else.

Maybe something like robotjs.rocks? Or AutomationIs.cool?

Hit me with your suggestions!

Failing build Windows 7

..\src\robotjs.cc(2): fatal error C1083: Cannot open include file: 'nan.h': No such file or directory [c:\Users\mj\code\test\node_modules\robotjs\build\robotjs.vcxproj]