Roadmap: Add multi-monitor support about pyautogui

[mabrady]

ctypes.windll.user32.GetSystemMetrics(80) will return an integer corresponding to the number of display monitors present.

[blackskandix]

for windows - get:

cursor = ctypes.wintypes.POINT()
ctypes.windll.user32.GetCursorPos(ctypes.byref(cursor))

set:

ctypes.windll.user32.SetCursorPos(cursor.x+offset, cursor.y+offset)

works perfectly fine for windows (also tested with negative numbers - check)

---

EDIT: I just realisised, this is almost the way it is implemented right now:

class POINT(ctypes.Structure):
    _fields_ = [("x", ctypes.c_ulong),
                ("y", ctypes.c_ulong)]

with the Problem, that the own Point-class is using unsigned long int. I think that is the problem ...

[mabrady]

ctypes.windll.user32.GetSystemMetrics(60) will return the maximum y value of all monitors attached
ctypes.windll.user32.GetSystemMetrics(59) will return the maximum x value of all monitors attached

Note that this includes any offset between the monitors (i.e. if I have one monitor slightly above the primary display and off to the left, the overall y value will include this offset).

[HixRa]

Hi everyone,

I've been tinkering with this issue for a while and I think I have come up with a solution which allows multi-monitor support for Windows. I don't have a Mac to test whether or not it works on a UNIX platform, so I'd appreciate any input from Mac users.

@CTPaHHuK-HEbA is correct, in Windows pyautogui_win.py in def _size(): must be changed to add in virtual screen capabilities, as shown below:

def _size():
"""Returns the width and height of the screen as a two-integer tuple.

Returns:
  (width, height) tuple of the screen size, in pixels.
"""
# return (ctypes.windll.user32.GetSystemMetrics(0), ctypes.windll.user32.GetSystemMetrics(1))
return (ctypes.windll.user32.GetSystemMetrics(78), ctypes.windll.user32.GetSystemMetrics(79))  # virtual screen

However, the check @Queeq pointed out in init.py in def _mouseMoveDrag(): must instead be changed to the following:

# Make sure x and y are within the screen bounds.

# x = max(0, min(x, width - 1))

# y = max(0, min(y, height - 1))

x = min(x, width - 1)

y = min(y, height - 1)

I'm using Windows 10 and my monitors are arranged 2, 3, 1 in my display settings with monitor 3 being selected as the Primary Display. Removing the max check proved important. In Windows 10, because monitor 3 was set as my primary display, monitor 2 returned negative x values in the above check, and as such monitor 2 could not be used by pyautogui. After some testing, Windows starts the screen coordinate system at 0, 0 on the Primary Display's top left corner (for my setup anyway). Removing the max check allowed me to keep monitor 3 as my primary display and negative values would be accepted.

If I changed monitor 2 to being my primary display, then Windows moved the screen coordinate system and all values returned by _mouseMoveDrag were positive (0 - 5760 for the x values, testing across all three monitors in this setup). However, I wanted my center screen, monitor 3, to be my primary screen. That was just a behavior I noticed from Windows, after changing which monitor was the primary display.

As of yet, I've had no issues with a multi-monitor setup using the above modifications, regardless of how my monitors are arranged in Windows Display Settings.

[florom]

Hi @asweigart ,
hi @ community.
I monitor this issue quite a long time cause I (and I think many) would benefit from this a lot.

I understand that not every fancy monizot allignment can be supported but like having the monitors in one row is very common.
I know there is a hack but I think that is not eneugh.

I am still wondering why this feature does not get any support from the mainline.

Thank you for listening.

[Queeq]

On Mac OS X it seems like it doesn't work simply because it doesn't allow negative numbers for coordinates. This is a piece of code from init.py, _mouseMoveDrag() (line 797-799):

# Make sure x and y are within the screen bounds.
x = max(0, min(x, width - 1))
y = max(0, min(y, height - 1))

If I comment out these checks - it works.
My monitor configuration is like this:

Center of coordinates is at the top left corner of the lower monitor (that is laptop). Thus top monitors have negative Y-coordinate values.

Also pyautogui.size() returns incorrect size (1440x900) while it's obviously much more.
The following simple script, which is based on the doc example, helped me to debug this:

import pyautogui
import time

print(pyautogui.size())

prev_x = 0
prev_y = 0

while True:
    x, y = pyautogui.position()
    if x != prev_x or y != prev_y:
        prev_x = x
        prev_y = y
        print("X: {0}; Y: {1}".format(x, y))

    time.sleep(1)

[segalion]

Hello. Have you see https://github.com/rr-/screeninfo ?

Could be great people working together for python multiplatfom standarization, and could be great integrate screeninfo inside this lib.

Could be great to get info about work-area info (like wmrctl)

We have relative good support with tkinter to interact with gui, but not outside the python application.
get info and control gui related outside the app its really very unfragmented in python uni-platform libraries. https://github.com/asweigart/PyGetWindow is a good start to control windows (like wmcrl x11)

Have you see https://github.com/moses-palmer/pynput ? Could be great joint efforts to get a great good multiplatform platform for windows automate.

I like "autohotkey" style of integrate things, but its only for windows and has a very bad script language. Python multiplatform that could make half of what autohotkey do, could be a disruptive thing around python world.

[digwanderlust]

I'd love to help to get this to work. Has any research been done yet? I know this doesn't work on Mac but does it also not work on Linux and Windows?

[asweigart]

It doesn't work on any platform yet.

Thanks!

[digwanderlust]

Do you mind if we split this into 3 issues then? I'm pretty sure windows/linux/mac will require platform specific tweaks. Happy to start looking at mac and how to get it working. If they are still unresolved after that I can move onto other platforms.

[mabrady]

I, too, would be willing to help on this (for windows). For win32, I think this is quite straightforward...

ctypes.windll.user32.SetCursorPos(x, y) is used by pyautogui to move the mouse to a certain position, and inputting negative numbers (if second monitor is on the left) moves the cursor to the corresponding position on the second monitor. So I believe you can get the entire window size of both monitors using ctypes then map that to positive values and treat the whole thing like one giant monitor?

[asweigart]

I'd recommend we at least do some cursor checks into how OS X and X windows handles multiple monitors. I'm generally in favor of treating it like one giant monitor, but I want to make sure that treatment makes sense with the way OS X and X windows does it.

[mabrady]

I agree. The script below will find the window size and plot it for a two-monitor setup. I would be curious how it works with different numbers of monitors. It seems to work fine with different size monitors when I change the display resolution of an individual monitor (i.e. 1 set to native 1080 and one set to 800x600, etc).

import ctypes
import matplotlib.pyplot as plt

# Data structure for POINTs
class POINT(ctypes.Structure):
    _fields_ = [("x", ctypes.c_long), ("y", ctypes.c_long)]

# Left bounding corner
leftBound = POINT()

ctypes.windll.user32.SetCursorPos(-99999, -99999)
ctypes.windll.user32.GetCursorPos(ctypes.byref(leftBound))

print leftBound.x, leftBound.y

# Right bounding corner
rightBound = POINT()

ctypes.windll.user32.SetCursorPos(99999, -99999)
ctypes.windll.user32.GetCursorPos(ctypes.byref(rightBound))

print rightBound.x, rightBound.y

# Left bounding corner, bottom
leftBoundBot = POINT()

ctypes.windll.user32.SetCursorPos(-99999, 99999)
ctypes.windll.user32.GetCursorPos(ctypes.byref(leftBoundBot))

print leftBoundBot.x, leftBoundBot.y

# Right bounding corner, bottom
rightBoundBot = POINT()

ctypes.windll.user32.SetCursorPos(99999, 99999)
ctypes.windll.user32.GetCursorPos(ctypes.byref(rightBoundBot))

print rightBoundBot.x, rightBoundBot.y

# Reset cursor to middle of screen
ctypes.windll.user32.SetCursorPos(0, 0)

# Plot coordinates
a = [leftBound.x, 0, 0, rightBound.x, rightBoundBot.x,
     0, 0, leftBoundBot.x, leftBound.x]
b = [leftBound.y, leftBound.y, 0, 0, rightBoundBot.y,
     rightBoundBot.y, leftBoundBot.y, leftBoundBot.y, leftBound.y]
b = [-each for each in b]  # Correct for inverted axis locations for plotting
plt.plot(a, b, 'r')
plt.axis([-2000, 2000, -1200, 1200])
plt.show()

[glitchassassin]

Hey everyone,

I built something similar as part of my Python implementation of Sikuli script. Check out the "Screen Functions" section starting on line 438. Multi-monitor support includes not just moving the mouse across monitors, but taking screenshots across monitors (or even of the whole virtual screen, arranged according to the OS's settings).

This particular implementation is Windows-specific, but feel free to copy anything that's helpful!

[CTPaHHuK-HEbA]

1) Mouse move to second screen for Windows
This check does not allow you to go beyond the dimensions of the screen:

    width, height = size()

    # Make sure x and y are within the screen bounds.
    x = max(0, min(x, width - 1))
    y = max(0, min(y, height - 1))

Im fix size() in _pyautogui_win.py for virtual screens
SM_CXVIRTUALSCREEN = 78 # Width of virtual screen
SM_CYVIRTUALSCREEN = 79 # Height of virtual screen

def _size():
    """Returns the width and height of the screen as a two-integer tuple.

    Returns:
      (width, height) tuple of the screen size, in pixels.
    """
    # return (ctypes.windll.user32.GetSystemMetrics(0), ctypes.windll.user32.GetSystemMetrics(1))
    return (ctypes.windll.user32.GetSystemMetrics(78), ctypes.windll.user32.GetSystemMetrics(79))  # virtual screen

2) Can be fix also negative values
Del:

# class POINT(ctypes.Structure):
#    _fields_ = [("x", ctypes.c_ulong),
#                ("y", ctypes.c_ulong)]

and fix function _position()

    # cursor = POINT()
    cursor = ctypes.wintypes.POINT()

[dasbuilder]

I use 3 displays at work (2 monitors and my Macbook Pro 13" display) and am looking to automate a little with my workflow, but pyautogui doesn't support multiple monitors - not yet at least.

I also do not know anything about programming on Mac or within the Cocoa/Objective-C and mainly know Python (Bash too, but I digress). @asweigart @aaruff @pgkos @HesselM any help in this area would be greatly appreciated. I will need some help myself to bring multiple monitor support to pyautogui :)

To further my point, I'd like to comment that I've had success on MacOS with the following code in Python. You have to install pyobjc and pybobj-core to install Quartz and AppKit (which you should already have to run pyautogui).

Here I'm getting the mainScreen ID which can then be used to get monitor information (see my loop below):

>>> AppKit.NSScreen.mainScreen()
<NSScreen: 0x112d48670>

Here we get the size of my main monitor after importing AppKit. My examples presume you've already installed and imported AppKit:

>>> AppKit.NSScreen.mainScreen().visibleFrame()
<NSRect origin=<NSPoint x=0.0 y=0.0> size=<NSSize width=1680.0 height=1028.0>>

I have 3 monitors. I use the below script to get the IDs for all 3, and build a list so I can then reference it later. I don't have the data for my multiple monitor setup at this moment, but will update this post once I have that:

monitors = []
i = 0
while i < 3:
    for displays in AppKit.NSScreen.screens():
        i += 1
        monitors.append("{0}:{1}".format(i, {"displayNum":displays.deviceDescription()['NSScreenNumber'],"height":displays.frame().size.height, "width":displays.frame().size.width}))

This may or may not be the most efficient way to retrieve monitor size, but it is accurate. I'm trying to read and understand more how NSViewController might have to be used to build a "view" in the sense we feed the monitors list into this "view" so our application knows what it's working with. This is just an educated guess, of course. I'm happy to be set straight here.

Thanks!

[glitchassassin]

Looks like you're on the right track with NSScreen.screens(). If you're using the built-in screencapture command to get the screen image, keep in mind that images captured from a Retina display will have double the resolution of the reported screen size - you'll need to scale it down using something like Pillow's resize() for matching purposes.

[dasbuilder]

Also, sorry for issues with my post. I edited then accidentally deleted it :( Thank you for the heads up on screencapturing @glitchassassin

[neighthan]

I was a little confused about how to get the position the way that @blackskandix described above; it's stored in cursor, not returned by the function (I don't have any experience with ctypes). Here are a couple of simple wrappers for their approach that work on Windows, at least for my two-monitor setup:

import ctypes
from typing import Tuple


def get_mouse_loc() -> Tuple[int, int]:
    cursor = ctypes.wintypes.POINT()
    ctypes.windll.user32.GetCursorPos(ctypes.byref(cursor))
    return cursor.x, cursor.y


def move_mouse_to(loc: Tuple[int, int]) -> None:
    ctypes.windll.user32.SetCursorPos(*loc)

[jhuels]

Any update on this? Looks most of the code is in this thread to make a pull request.

[trydalch]

I would also really appreciate this getting done... sounds like it's largely been figured out and PRs just need to be made and merged. Any update?

[fp37]

Hi guys, I bypassed this issue by using DesktopMagic library and modifying pyautogui library.

Install these packages:

pip install pipautogui
pip install pywin32
pip install --user Desktopmagic

Edit the file C:\Program Files\Python\Lib\site-packages\pyautogui\screenshotUtil.py
(inside locateOnScreen and locateAllOnScreen functions):

# screenshotIm = screenshot(region=None) # the locateAll() function must handle cropping to return accurate coordinates, so don't pass a region here.
screenshotIm = screenshot()

Same way here, edit the file C:\Program Files\Python\Lib\site-packages\pyscreeze\ init.py
(inside locateOnScreen and locateAllOnScreen functions):

# screenshotIm = screenshot(region=None) # the locateAll() function must handle cropping to return accurate coordinates, so don't pass a region here.
screenshotIm = screenshot()

Inside your script, import packages like this:

from __future__ import print_function
from desktopmagic.screengrab_win32 import getScreenAsImage
import pyautogui
pyautogui.screenshot = getScreenAsImage
pyautogui.pyscreeze.screenshot = getScreenAsImage

Desktopmagic library return a tuple (or a list of tuples), for example you can write your code like this:

locateImg = pyautogui.locateOnScreen('image.png')
if locateImg is not None: pyautogui.click(pyautogui.center(locateImg))

or

locateImg = list(pyautogui.locateAllOnScreen('image.png'))
if locateImg is not None: pyautogui.click(pyautogui.center(locateImg[0]))

Hope this helps, regards.

[Lan-Rus]

This solution worked for moving the mouse #9

But pyautogui.locateOnScreen still doesn't work on a second screen. How to make it work?

[kylefoley76]

Can someone progam this to support multi-monitor for Mac?

[xXPerditorXx]

In Linux (ArchLinux, KDE Plasma) it works really well. I have two monitors both 1920x1080 but if I do pyautogui.size() I get 3840x1080. So I can use it. Is there any fix for windows now?

[PretheshP]

#Try below code for moving cursor to secondary screen
import pyautogui
import time

Get the screen resolution

screen_width, screen_height = pyautogui.size()

Move the mouse to the center of the primary screen

pyautogui.moveTo(screen_width // 2, screen_height // 2)

Wait for a moment (you can adjust the duration)

time.sleep(2)

Move the mouse to the center of the second screen

pyautogui.moveTo(screen_width + (screen_width // 2), screen_height // 2)

[sagregrevc]

guess different libs can be more friendly for multimonitor. tebelorg/RPA-Python#252