tmandry / axswift Goto Github PK

In my app I'm listening to didActivateApplicationNotification notifications and then add AXObserver to the app that was activated.
I've noticed that in some scenarios I got UnknownUIElement errors when I was trying to add an observable right after launching Chrome.
I think this is due to the fact that chrome has isFinishedLaunching = false when activated.

The solution for me is to listen for another additional event (didLaunchApplicationNotification), but you may want to consider adding a check for isFinishedLaunching in Application initializers.

Hey, thanks so much for this library. It works great for observing changes on windows etc.

However I cannot make it work to observe a MenuItem's Title changes in the app's MenuBar.
I can construct UIElement object that locates that MenuItem. It's the correct pointer, because I can perform an action successfully.

menuItem.performAction(kAXPressAction)

Is there some trick to achieve that or the app may be even broadcasting those events?

Snippet of my code:
var menuItem = UIElement(getMenuItem()) //my function that returns the MenuItem as AXUIElement
//try menuItem.performAction(kAXPressAction) //just a test if the pointer is working

try observer.addNotification(.titleChanged, forElement: menuItem)

No events are triggered using this code. I tried also Accessibility Inspector and the MenuItem has exact ID, and just changes it's Title (a toggle).

Thanks for any clue.

When I ran AXSwiftObserverExample target for the first time, the app asked for accessibility permission. I granted the permission from system preference, and ran it again. The app launches, but nothing gets printed in the console.
I also tried implementing the framework into my own app with observers for mainWindowChanged, focusedWindowChanged, and windowMoved with a callback to print some information, but nothing gets printed in the console either.
I'm on Xcode 10.1 with Mojave 10.14.3.
Any help would be appreciated.
Thanks!

Hey,

I'm using the CLI swift tool and the integrated package manager. I'm still discovering Swift so I might be missing something obvious.

// swift-tools-version:4.0
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "sandbox",
    dependencies: [
        // Dependencies declare other packages that this package depends on.
        .package(url: "https://github.com/tmandry/AXSwift.git", from: "0.2.0"),
    ],
    targets: [
        // Targets are the basic building blocks of a package. A target can define a module or a test suite.
        // Targets can depend on other targets in this package, and on products in packages which this package depends on.
        .target(
            name: "sandbox",
            dependencies: [
                "AXSwift"
            ]),
    ]
)

$ swift --version
Apple Swift version 4.1 (swiftlang-902.0.48 clang-902.0.37.1)
Target: x86_64-apple-darwin17.4.0

Result of swift build:

AXSwift: 0.2.3
XCode: 11.4.1
Swift: 4.2

All crash reports so far are on macOS 10.14.6. My app currently runs on 10.12 through 10.15.4.

I am receiving crash reports that point to a line that is attempting to get the .title attribute of a window element. All crash reports point to line 671 in my AppDelegate. Line 671 is:

if let title: String = try! element.attribute(.title) {
    ...
}

The consistent error message is:

closure #1 (AXSwift.Observer, AXSwift.UIElement, AXSwift.AXNotification, [Swift.String : Swift.AnyObject]?) -> () in WindowMizer.AppDelegate.startWatcher(_: AXSwift.Application, owner: Swift.String) throws -> AXSwift.Observer?
AppDelegate.swift, line 671

The crash log is:

Main Thread - Crashed

libswiftCore.dylib
function signature specialization <Arg[0] = Exploded, Arg[1] = Exploded> of Swift._assertionFailure(_: Swift.StaticString, _: Swift.String, file: Swift.StaticString, line: Swift.UInt, flags: Swift.UInt32) -> Swift.Never
libswiftCore.dylib
swift_unexpectedError
WindowMizer
closure #1 (AXSwift.Observer, AXSwift.UIElement, AXSwift.AXNotification, [Swift.String : Swift.AnyObject]?) -> () in WindowMizer.AppDelegate.startWatcher(_: AXSwift.Application, owner: Swift.String) throws -> AXSwift.Observer? AppDelegate.swift:671
WindowMizer
partial apply forwarder for closure #1 (AXSwift.Observer, AXSwift.UIElement, AXSwift.AXNotification, [Swift.String : Swift.AnyObject]?) -> () in WindowMizer.AppDelegate.startWatcher(_: AXSwift.Application, owner: Swift.String) throws -> AXSwift.Observer? <compiler-generated>:0
AXSwift
0x106f14000 + 79382
AXSwift
0x106f14000 + 77944
HIServices
_XXMIGPostNotification
HIServices
_XPostNotification
HIServices
mshMIGPerform
CoreFoundation
__CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__
CoreFoundation
__CFRunLoopDoSource1
CoreFoundation
__CFRunLoopRun
CoreFoundation
CFRunLoopRunSpecific
HIToolbox
RunCurrentEventLoopInMode
HIToolbox
ReceiveNextEventCommon
HIToolbox
_BlockUntilNextEventMatchingListInModeWithFilter
AppKit
_DPSNextEvent
AppKit
-[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:]
AppKit
-[NSApplication run]
AppKit
NSApplicationMain
WindowMizer
main main.swift:14
libdyld.dylib
start

Any idea why the code would crash trying to get the .title attribute?

I can run AXSwiftObserverExample, but I can't incorporate into my project.
The accessibility permission dialog doesn't come up when I run my project, and it just throws this error.
ObserverTest[14188:300022] Error: Could not watch app [<UnknownUIElement "<AXUIElement Application 0x600000c4c240> {pid=366}" (pid=366)>]: AXError.CannotComplete
(lldb)
Do I need to indicate a special permission in info.plist or target settings?
Here are the steps I took.

1. Create a Coaco project on XCode
2. Add cartfile and update carthage
3. Drop AXSwift framework into Linked Frameworks and Libraries
4. Add a new Run Script
   /usr/local/bin/carthage copy-frameworks
   Add AXSwift.framework as an input file.
5. Copy contents from AppDelegate.swift inside AXSwiftObserverExample to AppDelegate.swift for my project.
   If I run, I get the error above.
   Any help would be appreciated.
   Thanks!

I've allowed accessiblity but always getting error AXError.APIDisabled

as you can see here:

Should I be signing or adding some special permissions?

P.S. I'm running in debug from xcode. Does this impact somehow?

I realize the question is a bit vague, but the Apple documentation is horrendous and hoping you could assist.

Basically, imagine a scenario where you are assisting in spell checking (or some other text manipulation of the given keystrokes in a text element).

I want to listen for events on the text element, run some text processing, and present a popup dialogue that the user can interact with.
Once the user selects (or escapes) out of the dialogue I should be able to replace the typed text

I've gone through the "actions", "attributes" and various other documentation I could find but see nothing about how to create elements.

You can see this type of interaction in apps like Grammarly and Fig etc.

I can't seem to find ANY code at all with examples for this 🤔

No Apple documentation seems to exist as well...

Is there any way possible to get the z-index of each window? I can't find an attribute that seems to indicate z-index.

I'm very new to xcode, so I may be barking up a number of the wrong trees. But I can't figure out how to get a project building which uses AXSwift, nor can I even get the included example to build.

I've built AXSwift.framework OK using XCode (both manually and via carthage). But when I try to compile AXSwiftExample (or a new project which I've added the .framework to), I get No such module 'AXSwift' on the import AXSwift line.

I've tried to learn how frameworks interact with swift modules, but to be honest I'm a bit lost. Most of the resources are just "how to do [specific thing] with xcode", rather than describing how modules actually work and how they interact with frameworks.

I've built other swift modules as frameworks though (my own simple example and also Alamofire), and notice that each of them have something in Modules:

$ ls Carthage/Build/Mac/Alamofire.framework/Versions/Current/Modules/
Alamofire.swiftmodule module.modulemap

But my built framework for axswift seems to have very little:

$ ls Carthage/Build/Mac/AXSwift.framework/Versions/Current/
AXSwift   Resources

AXSwift seems to be a shared object, so maybe I could link against it if I could get that far - but it seems to be missing the compile time .modulemap. Is this why XCode can't find the module? Have I built it wrong, or am I using it wrong?

I just make a demo project basing on AXSwift. And found that if I launch the App directly from the Debug build or Archived App, it works as expected. But once I put the App into the /Application folder, the checkIsProcessTrusted always return false even I have enable the App in Accessiblity panel.

I'm trying to build a sort of POC for a software remote controller to Keynote to pass slides and I found that whenever I click on my application's UI to trigger a Keynote change, nothing happens unless Keynote is in foreground, so I have to resort to cmd+clicking on my app so that Keynote remains foreground and then the AXPress event is sent and received by Keynote

I've tried using AXRaise to bring Keynote to front before sending the AXPress event but I find that it's still not received.

Is there something I'm doing wrong? How can I use AXSwift to bring Keynote to front so that my events are delivered?

Thanks!

Add API to list and perform actions. These are usually things like pushing buttons.

First of all I'd like to thank you for all that work you have done! It really simplifies dealing with Accessibility API.

What I'm trying to do is following. I'd like to change foreground and background color of some text inside NSTextView of some app. From my understanding I should use NSAttributedString (at least that is what I'm getting when I try to read parameters of the text by using AXAttributedStringForRangeParameterized). But I can't really get how to construct my object that I then should pass to setAttribute method. I would assume that I need to specify range of text, the text itself and parameters, which in my case are AXForegroundColorText and AXBackgroundColorText.

Could you please help me?

After updating to XCode 11.4.1 I have a couple errors I do not know how to resolve. Any pointers on how to fix these compile errors?

I have rebuilt the framework in XCode 11.4.1 with Swift 5.2.2 without issue, but when using the framework in my project I receive the following errors:

1. Errors:
   a) Cannot infer contextual base in reference to member 'size' AND
   b) Generic parameter 'T' could not be inferred

    let w = myWindowObject

    if let size: CGSize? = try? w.axData.uiWindowElement.attribute(.size) {
        originalSize = size!
    }

2. Error: Cannot convert value of type 'Optional<Optional<_>>' to specified type 'CGSize??'

        if let size: CGSize? = try? uiElement.attribute(.size) {
            ...
        }

3. Error: Cannot convert value of type 'Optional<Optional<_>>' to specified type 'String??'

    if let title: String? = try? element.attribute(.title) {
        ...
    }

Any suggestion would be greatly appreciated. i will be working to resolve the errors on my own and will report any progress here.

Hello Tyler!

This is a very convenient little wrapper lib, very useful for a certain kind of apps! But it is a bit unclear at the moment what is the licensing for it – AFAIK it is something like copyright by default.

So the question is: did you intend it this way or you plan to release it under some permissive license (say, MIT)?

Thanks in advance!
cheers

Please fix your podspec according to this: CocoaPods/CocoaPods#3638 (comment)

Also version numerb need to be updated or removed from the podspec s.source entry.

Do you manually find the debug build and add it in the Security page?

Current API design restricts notifications to only the ones that are provided in the Constants.swift.
And while notification names are generally standardised, applications are allowed to send notifications with custom names (with api like NSAccessibilityPostNotificationWithUserInfo)
This is what applications like Music, Safari and Dock do. It would be nice to change the API to allow for this use case. I was playing with a bunch fo design, to prototype how it would look like with generics or protocol, but C callbacks interoperability in swift with those features is not the best...

In the same App, press tab to switch focus, focusedUIElementChanged will be triggered.

But when switching between different apps, focusedUIElementChanged will not trigger. I think when switching to a different app, the focusedUIElement changes, then focusedUIElementChanged should be triggered.

for example:

1. Open System Preferences
2. Focus on the search box in the upper right corner
3. Open https://www.google.com/ with Chrome browser
4. Focus on the Google search box
5. Switch to System Preferences
6. After switching to Chrome

We will find that when the two applications are switched, focusedUIElementChanged does not trigger

For example, I want to monitor the size and position of the text box of the TextEdit window:

1. Open the TextEdit app.
2. Press Command + N to create a new text editing window.
3. Press Command + Shift + . to enlarge the text box.
4. The size of the text box is enlarged, but the change of the size and position of the text cannot be monitored through .moved and .resized.

my code is invalid:

  var observer: Observer!

   public func applicationDidFinishLaunching(_ aNotification: Notification) {
//    let app = Application.allForBundleID("com.apple.finder").first!
    let app = Application.allForBundleID("com.apple.TextEdit").first!

    do {
      try startWatcher(app)
    } catch let error {
      NSLog("Error: Could not watch app [\(app)]: \(error)")
      abort()
    }
  }

  func startWatcher(_ app: AXSwift.Application) throws {
    var updated = false
    observer = app.createObserver { (observer: Observer, element: UIElement, event: AXNotification, info: [String: AnyObject]?) in
      var elementDesc: String!
      if let role = try? element.role()!, role == .window {
        elementDesc = "\(element) \"\(try! (element.attribute(.title) as String?)!)\""
      } else {
        elementDesc = "\(element)"
      }
      print("\(event) on \(String(describing: elementDesc)); info: \(info ?? [:])")

      // Watch events on new windows
      if event == .windowCreated {
        do {
          try observer.addNotification(.uiElementDestroyed, forElement: element)
          try observer.addNotification(.moved, forElement: element)
          try observer.addNotification(.resized, forElement: element)
          try observer.addNotification(.created, forElement: element)
        } catch let error {
          NSLog("Error: Could not watch [\(element)]: \(error)")
        }
      }
      if event == .created || event == .focusedUIElementChanged {
        do {
          try observer.addNotification(.moved, forElement: element)
          try observer.addNotification(.resized, forElement: element)
        } catch let error {
          NSLog("Error: Could not watch [\(element)]: \(error)")
        }
      }

      // Group simultaneous events together with --- lines
      if !updated {
        updated = true
        // Set this code to run after the current run loop, which is dispatching all notifications.
        DispatchQueue.main.async {
          print("---")
          updated = false
        }
      }
    }

    try observer.addNotification(.windowCreated, forElement: app)
    try observer.addNotification(.mainWindowChanged, forElement: app)
    try observer.addNotification(.created, forElement: app)
    try observer.addNotification(.focusedUIElementChanged, forElement: app)
  }

Application init gives me an UnknownUIElement if the PID is not that of the project application.

In the below sample code, there is a sleep() function. If you were to set any other app to the front, when the program resumes, the Application will be <UnknownUIElement "<AXUIElement Application 0x600000c682d0> {pid=563}" (pid=563)>. when I add notification to observers created with the Application object, I get a cannot complete error.

    func applicationDidFinishLaunching(_ aNotification: Notification) {
        // Check that we have permission
        guard UIElement.isProcessTrusted(withPrompt: true) else {
            NSLog("No accessibility API permission, exiting")
            NSRunningApplication.current.terminate()
            return
        }

        sleep(2)
        
        if let nsApplication = NSWorkspace.shared.frontmostApplication,
            let application = Application.init(nsApplication) {
            
            print(application)
            
            let optionalApplication = application
            let optionalObserver = attachObserverToApplication(application: application)
        }
    }
    
    func attachObserverToApplication(application: Application) -> Observer {
        let optionalObserver = application.createObserver { (observer: Observer, element: UIElement, event: AXNotification) in
            print("event emitted")
        }
        
        let events: [AXNotification] = [.windowMiniaturized, .windowMoved, .windowResized, .focusedWindowChanged]
        for event in events {
            try! optionalObserver?.addNotification(event, forElement: application)
        }
        
        return optionalObserver!
    }
    

Output:

<UnknownUIElement "<AXUIElement Application 0x600000c682d0> {pid=563}" (pid=563)>
Fatal error: 'try!' expression unexpectedly raised an error: AXError.CannotComplete: file /Users/macintosh/code/demo/demoAppDelegate.swift, line 42
2019-09-09 18:30:19.535602+0800 demo[2968:49579] Fatal error: 'try!' expression unexpectedly raised an error: AXError.CannotComplete: file /Users/macintosh/code/demo/demo/AppDelegate.swift, line 42
(lldb) 

UIElement.attribute<T>() takes a generic parameter for its expected return type. For most attributes, the expected type is known, and a failure to convert an existing value to that known type is an error. (Other attributes are allowed to use attribute<Any>().)

I think crashing the program may be a little extreme for an error in this case. There are a couple ways of solving this:

1. Throw a custom error type and let the user handle it if they wish.
2. Another is to remove the casting and generics altogether, and always return Any so the user can decide how to handle the cast.
3. Return nil if the type is unexpected.

I don't want number 3, because that creates two cases in which the return type could be nil: the attribute doesn't exist, or the attribute exists but is of an unexpected type. Assuming one case or the other could lead to undiscovered bugs in a program.

The main reason I didn't do number 2 is that it makes the API much more awkward to use, in what I expect are the majority of calls. These are cases where there is one expected type, an unexpected type should trigger some error, but we don't want to crash the app if the attribute simply isn't there.

Consider:

// What we have today
let title: String? = element.attribute<String>(.title)

vs

// New signature:
// func attribute(_ attribute: Attribute) -> Any?

// Option 1: Okay, but silently ignores a wrong type!
// What if our program has a bug, or an assumption that isn't always true?
let title: String? = (element.attribute(.title) as? String)

// Option 2: Safer, but uglier and more verbose; requires a temporary
let tmp = element.attribute(.title)
let title: String? = (tmp == nil) ? nil : tmp as! String

As it stands now, passing <Any> as the type devolves to the second situation, but in that case you don't know the type, and therefore need a more complicated case control structure anyway.

So, I'm leaning towards throwing an error and keeping the API the same. This allows an astute programmer to log the error and devolve functionality gracefully, if possible. But if there's substantial evidence that a lot of use cases don't know their expected type, I might reconsider.

Add APIs to list and read parametric attributes. I'm not sure what these are used for, really.

Xcode: 13.3
Swift: 5.6

After installing AXSwift via SPM, I receive the following compiler error:

Could not find module 'AXSwift' for target 'arm64-apple-macos'; found: x86_64-apple-macos

My research into the error indicates that building an xcframework for each architecture is the "new" way to distribute a framework that supports building & debugging on multiple architectures.

Should installation of AXSwift via SPM be working on M1?
or does AXSwift need to be re-built as an xcframework?

Currently, I can build the AXSwift.framework locally and include it in project, but when Swift version is updated, I have to rebuild manually. I am hoping to modernize/automate that process via SPM.

How complicated is it to adapt AXSwift for multiple architectures? and is that something that is on the roadmap?

Thanks!

Hello 👋 ,

I was using Swindler. I have now used Xcode version 13 beta 5 and I received error in AXSwift and Swindler packages.
Cannot find AXError type in scope.
I have created Pull request for AXSwift and for Swindler where I just import Cocoa in 2 files.

I'm a little unsure how to use AXSwift to observe changes to NSAccessibilityFocusedUIElementChangedNotification. Basically, I'd like to know when the user hovers over a dock icon.

I'm currently trying this to observe changes using the provided systemWideElement, but I think I may be using this incorrectly:

let app = Application(NSRunningApplication.current)
observer = app?.createObserver({ (observer, element, notification) in
    print("callback", (observer, element, notification))
})

do {
    try observer?.addNotification(.focusedUIElementChanged, forElement: systemWideElement)
} catch {
	// !!! This fails with AXError.InvalidUIElement
    print("notification: ", error)
}

How are we supposed to use the systemWideElement provided?

I've got this crash more than a couple of times in an app, Shifty, using AXSwift

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       EXC_I386_GPFLT
Exception Note:        EXC_CORPSE_NOTIFY

Termination Signal:    Segmentation fault: 11
Termination Reason:    Namespace SIGNAL, Code 0xb
Terminating Process:   exc handler [576]

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   libswiftCore.dylib            	0x00000001093e14e1 swift::RefCounts<swift::RefCountBitsT<(swift::RefCountInlinedness)1> >::incrementSlow(swift::RefCountBitsT<(swift::RefCountInlinedness)1>, unsigned int) + 33
1   libswiftCore.dylib            	0x00000001093c9984 _swift_retain_(swift::HeapObject*) + 68
2   org.cocoapods.AXSwift         	0x0000000108f04078 specialized internalInfoCallback(_:axElement:notification:cfInfo:userData:) + 552
3   org.cocoapods.AXSwift         	0x0000000108f034ee @objc internalInfoCallback(_:axElement:notification:cfInfo:userData:) + 78
4   com.apple.HIServices          	0x00007fff4b190672 _XXMIGPostNotification + 821
5   com.apple.HIServices          	0x00007fff4b19ddac _XPostNotification + 351
6   com.apple.HIServices          	0x00007fff4b172abe mshMIGPerform + 212
7   com.apple.CoreFoundation      	0x00007fff4c9cc6a9 __CFRUNLOOP_IS_CALLING_OUT_TO_A_SOURCE1_PERFORM_FUNCTION__ + 41
8   com.apple.CoreFoundation      	0x00007fff4c9cc607 __CFRunLoopDoSource1 + 527
9   com.apple.CoreFoundation      	0x00007fff4c9b4689 __CFRunLoopRun + 2574
10  com.apple.CoreFoundation      	0x00007fff4c9b3a28 CFRunLoopRunSpecific + 463
11  com.apple.HIToolbox           	0x00007fff4bc4cb35 RunCurrentEventLoopInMode + 293
12  com.apple.HIToolbox           	0x00007fff4bc4c774 ReceiveNextEventCommon + 371
13  com.apple.HIToolbox           	0x00007fff4bc4c5e8 _BlockUntilNextEventMatchingListInModeWithFilter + 64
14  com.apple.AppKit              	0x00007fff49f08eb7 _DPSNextEvent + 997
15  com.apple.AppKit              	0x00007fff49f07c56 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1362
16  com.apple.AppKit              	0x00007fff49f01cb9 -[NSApplication run] + 699
17  com.apple.AppKit              	0x00007fff49ef13f7 NSApplicationMain + 780
18  io.natethompson.Shifty        	0x0000000108e022a9 0x108e00000 + 8873
19  libdyld.dylib                 	0x00007fff79b9608d start + 1

It doesn't always has the same backtrace, but internalInfoCallback seems to be the cause for all of them.
Looking at the disassembly the crash seems to be caused by the line

    observer.callbackWithInfo!(observer, element, notif, info)

The observer, or the callback aren't valid I think.
This is probably because I'm not using it correctly in the app Shifty (I would actually ask you what is the correct way to stop an observer), but we should probably not crash either way, and probably check if the observer or the callback are valid?

The offending code in Shifty is in this file https://github.com/thompsonate/Shifty/blob/5d58e021827b77fb00b850eb522fa6b73c158626/Shifty/BrowserManager.swift in the stopBrowserWatcher. I wonder if I should remove the notifications instead of calling stop.

Swift Package Manager fails to add AXSwift since v0.2.3 does not contain Package.swift which has been added the previous commit.

Could you release v0.2.4 in order to fix this issue?

Hello,

We are getting close to app submission to the App Store. The problem is that Apple requires the app sandbox to be turned on for all apps submitted to the App Store. Turning on the App Sandbox causes AXSwift to not work. It returns UnknownUIElement when trying to read the app under control using Application(forProcessID: ). Then doing anything else to that top-level UIElement returns an error.

Is there anyway around this?