W3C actions sends wrong action id about appium_capybara

[KazuCocoa]

As a note. The message itself was from Appium server, so changing the message should be in appium server side. (Not client side)

Btw, below code worked with our test app. How about using the latest Appium version? (1.20.2 is the latest for now)
I haven't tested on real devices, so potentially the error happens only on real devices by XCTest internal (by Apple)

[7] pry(#<AppiumLibCoreTest::DriverTest>)> e = @@driver.find_element :predicate, 'value == "Placeholder text"'
=> #<Selenium::WebDriver::Element:0x2b57069392a75fd6 id="AA000000-0000-0000-0737-000000000000">
[8] pry(#<AppiumLibCoreTest::DriverTest>)> e.send_keys [:shift, :end]

[HTTP] --> POST /wd/hub/session/2a7c701c-23a5-4ecc-aac5-6aa0d4070f95/element/AA000000-0000-0000-0737-000000000000/value
[HTTP] {"value":["","",""],"text":""}
[debug] [W3C (2a7c701c)] Calling AppiumDriver.setValue() with args: [["","",""],"AA000000-0000-0000-0737-000000000000","2a7c701c-23a5-4ecc-aac5-6aa0d4070f95"]
[debug] [XCUITest] Executing command 'setValue'
[debug] [WD Proxy] Matched '/element/AA000000-0000-0000-0737-000000000000/value' to command name 'setValue'
[debug] [Protocol Converter] Added 'text' property "" to 'setValue' request body
[debug] [WD Proxy] Proxying [POST /element/AA000000-0000-0000-0737-000000000000/value] to [POST http://127.0.0.1:8100/session/C3440797-CADD-4F2F-8611-5D866063A989/element/AA000000-0000-0000-0737-000000000000/value] with body: {"value":["","",""],"text":""}
[debug] [WD Proxy] Got response with status 200: {"value":null,"sessionId":"C3440797-CADD-4F2F-8611-5D866063A989"}
[debug] [W3C (2a7c701c)] Responding to client with driver.setValue() result: null
[HTTP] <-- POST /wd/hub/session/2a7c701c-23a5-4ecc-aac5-6aa0d4070f95/element/AA000000-0000-0000-0737-000000000000/value 200 246 ms - 14

[luke-hill]

Oki so I was using a mix of appium 1.16 and appium 1.17. Reasons are that appium 1.16 is the latest you can use on iOS11 I believe (So I'll just mark this as unfixable for iOS11)
iOS12/13 I haven't re-tested on a later appium. I'll try this later today. But you're suggesting upgrading appium "should" help?

[luke-hill]

I upgraded appium version to 1.20.2 on ios13, ran the same test. Same error occurred. (Didn't test on lower versions).

[KazuCocoa]

Then, I guess the root cause is in XCTest internal against real devices. Only Apple can modify the message/behavior.

Potentially Appium could catch errors by XCTest framework and modify the error message, but it seems the behavior depends on the device under test. So, it is not easy to unify the message in Appium layer with checking the response by XCTest since the XCTest is a black box. No one can guarantee the message by XCTest will not change. We won't modify errors by XCTest too much.

Btw, I'll try the same command https://github.com/appium/ruby_lib_core/issues/317#issuecomment-790004327 on a real device with iOS 12 tomorrow. If it will work, perhaps the error may depend on the target element.

[luke-hill]

Is XCUITest closed source? Or is there a way we can ask them to fix that?

[KazuCocoa]

Is XCUITest closed source?

Yes. https://developer.apple.com/documentation/xctest

is there a way we can ask them to fix that?

It could be in https://developer.apple.com/bug-reporting/ , but it depends on their priority.

[KazuCocoa]

I tried on a real device. e.send_keys [:shift, :end] worked as same.

Btw, I wondered the error message was by w3c actions. @driver.action.send_keys([:shift, :end]).perform sends W3C action keys as https://gist.github.com/KazuCocoa/b03d89711ae60ab82a91cbb657f0b1ea .

Could you share the full appium log and your appium_lib_core version?
The error log message is probably appium/WebDriverAgent layer to convert W3C actions to appium/WebDriverAgent actions if it was W3C actions. But element.send_keys should not send W3C actions...

[luke-hill]

So the way I'm running this is I have a Capybara ruby driver. So if you can tell me how to configure everything you need I'll do that.

I'm running executions for the legit phones through browserstack. I'm guessing this would be a specific setting somewhere in the driver capabilities (But as you well know, these aren't very well documented)

[luke-hill]

Also from your gist. Does that mean sending 2 keys simultaneously on a phone "should" work? If so this is definitely a bug then. Because that fails for me.

I've got a PR in place where we bump the appium versions a little bit (We were using a combination of 1.16 and 1.17)

[KazuCocoa]

I don't think this is capabilities stuff.

I guess https://github.com/appium/appium_capybara has some issues sending W3C actions.
Could you share the full Appium log and Gem dependencies to understand your situation?

[KazuCocoa]

https://github.com/appium/WebDriverAgent/blob/a1824895274c60798b506f31054e4ce1e0400c71/WebDriverAgentLib/Utilities/FBW3CActionsSynthesizer.m#L879-L890 is the error line.
I'm not so familiar with this library but will take a look when i have time after you attach logs as the above comment

[luke-hill]

You'll need to hand-hold a bit more in order to generate said logs. Sorry this stuff is really poorly documented or I am looking in the wrong location.

I don't want to send you my entire stack so I want to create something slimmed down, but need a way of generating the repo case.

[KazuCocoa]

Ok.
Then, can you share the log of begin/end of element.send_keys like #58 (comment) 's attached log?
If I can take a look at what query was sent by your client, it should become a hint.

[luke-hill]

When you say log. I don't have any way of generating logs like yours.

The logging I'm using is

Selenium - DEBUG set on the Webdriver logger. I don't have any way to generate XCUITest / Appium logs.

[KazuCocoa]

Logs you can see when an appium server launches on your console.

e.g.

$ appium
[Appium] Welcome to Appium v1.21.0-beta.0 (REV f40a2070ba320b97b3c3d3fc610ca4aedbe32c2f)
[Appium] Appium REST http interface listener started on 0.0.0.0:4723

Then, the console will get more logs as I attached in a session.

[luke-hill]

Similar to bugs I have raised in the safaridriver webkit tracker, this isn't possible for me. As we don't start the appium client, we use a third party (browserstack).

[luke-hill]

EDIT: I can do this. I just wasn't aware how to. This will be done before the end of March.

@ me if I don't post them by then!

[luke-hill]

Hi @KazuCocoa I think I've found a way to obtain them are the logs a little like this

2021-03-29 08:42:36:386 - [HTTP] --> POST /wd/hub/session
2021-03-29 08:42:36:386 - [HTTP] {"capabilities":{"firstMatch":[{"newCommandTimeout":0,"platformName":"iOS","browserName":"safari","realMobile":"true","deviceName":"iPhone 11","safariIgnoreFraudWarning":true,"orientation":"PORTRAIT","deviceOrientation":"PORTRAIT","noReset":true}]}}
2021-03-29 08:42:36:387 - [debug] [W3C] Calling AppiumDriver.createSession() with args: [null,null,{"firstMatch":[{"newCommandTimeout":0,"platformName":"iOS","browserName":"safari","realMobile":"true","deviceName":"iPhone 11","safariIgnoreFraudWarning":true,"orientation":"PORTRAIT","deviceOrientation":"PORTRAIT","noReset":true}]}]
2021-03-29 08:42:36:388 - [debug] [BaseDriver] Event 'newSessionRequested' logged at 1617007356387 (08:42:36 GMT+0000 (Coordinated Universal Time))
2021-03-29 08:42:36:394 - [BaseDriver] The following capabilities are not standard capabilities and should have an extension prefix:
2021-03-29 08:42:36:394 - [BaseDriver]   newCommandTimeout
2021-03-29 08:42:36:394 - [BaseDriver]   realMobile
2021-03-29 08:42:36:395 - [BaseDriver]   deviceName
2021-03-29 08:42:36:395 - [BaseDriver]   safariIgnoreFraudWarning
2021-03-29 08:42:36:395 - [BaseDriver]   orientation
2021-03-29 08:42:36:395 - [BaseDriver]   deviceOrientation
2021-03-29 08:42:36:395 - [BaseDriver]   noReset
2021-03-29 08:42:36:395 - [BaseDriver]   useXctestrunFile
2021-03-29 08:42:36:395 - [BaseDriver]   bootstrapPath
2021-03-29 08:42:36:395 - [BaseDriver]   browserstack.isTargetBased
2021-03-29 08:42:36:399 - [Appium] DeprecationWarning: 'automationName' capability was not provided. Future versions of Appium will require 'automationName' capability to be set for iOS sessions.
2021-03-29 08:42:36:400 - [Appium] Requested iOS support with version >= 10, using 'XCUITest' driver instead of UIAutomation-based driver, since the latter is unsupported on iOS 10 and up.
2021-03-29 08:42:37:131 - [Appium] Appium v1.20.2 creating new XCUITestDriver (v3.34.0) session
2021-03-29 08:42:37:134 - [debug] [BaseDriver] Creating session with W3C capabilities: {
2021-03-29 08:42:37:134 - [debug] [BaseDriver]   "alwaysMatch": {
2021-03-29 08:42:37:134 - [debug] [BaseDriver]     "appium:newCommandTimeout": 0,

[KazuCocoa]

The log seems right. Could you get the log when your client calls send_keys?

[luke-hill]

Hi there sorry for delay was on a holiday.

I'll update this in the coming week.

[luke-hill]

appium-logs.txt

[KazuCocoa]

Thanks, I confirmed the log.
So far, I guess capybara issued its own send_keys actions, which is not by appium_lib/appium_lib_core.
I need to find more time to debug this.

Do you have time to ensure if this behavior happens well-known websites like Google?

So,:

1. open https://google.com
2. click the search field
3. Get current active element
4. element.send_keys([:shift, :end])

I don't have this capybara environment for now on my local, so it is helpful if you have time to check this for my further investigation.

[luke-hill]

Initial tests show this is a problem everywhere.

[luke-hill]

Bump @KazuCocoa

[KazuCocoa]

I have added test cases with send_keys as https://github.com/appium/appium_capybara/pull/60/files#diff-b2de4210f3178616088103cc95f066ef492939f4900be343d303dfbe6089c75bR26-R33
They worked like the below.

[HTTP] --> POST /wd/hub/session/5f3062e8-663a-4d37-bc7b-0d69c9260c47/element/BA000000-0000-0000-ED6B-000000000000/value
[HTTP] {"value":["","",""],"text":""}
[debug] [W3C (5f3062e8)] Calling AppiumDriver.setValue() with args: [["","",""],"BA000000-0000-0000-ED6B-000000000000","5f3062e8-663a-4d37-bc7b-0d69c9260c47"]
[debug] [XCUITest] Executing command 'setValue'
[debug] [WD Proxy] Matched '/element/BA000000-0000-0000-ED6B-000000000000/value' to command name 'setValue'
[debug] [Protocol Converter] Added 'text' property "" to 'setValue' request body
[debug] [WD Proxy] Proxying [POST /element/BA000000-0000-0000-ED6B-000000000000/value] to [POST http://127.0.0.1:8100/session/6A1D1803-9D60-4F63-BD7F-DDE47712F76C/element/BA000000-0000-0000-ED6B-000000000000/value] with body: {"value":["","",""],"text":""}
[debug] [W3C (5f3062e8)] Responding to client with driver.setValue() result: null
[HTTP] <-- POST /wd/hub/session/5f3062e8-663a-4d37-bc7b-0d69c9260c47/element/BA000000-0000-0000-ED6B-000000000000/value 200 629 ms - 14

[KazuCocoa]

mm, I guess your case was a different pattern.
Could you reproduce your case with this repository's example app..?

[luke-hill]

I'm no longer at this job, and my new job is a long way away from being able to test on firefox / mobile (Where the main issues occurred).

To reproduce simply try send multiple keys using a mobile device. i.e. use capybara and do [:shift, :end]

[KazuCocoa]

Okay, thanks. Then, probably test target specific behavior in capybara x Firefox. (I guess then capybara assumed the firefox was desktop's one)

On the native iOS app, the below command worked with this module without reported issue.

    e = capy_driver.find_custom(:predicate, 'value == "Placeholder text"').first
    e.send_keys [:shift, :end]

    e = Capybara.current_session.find(:xpath, '//XCUIElementTypeTextField[@value=""]')
    e.send_keys [:shift, :end]

Close for now.