microsoft / microsoft-ui-uiautomation Goto Github PK
View Code? Open in Web Editor NEWUtility library for consuming Windows UIAutomation platform APIs
License: MIT License
Utility library for consuming Windows UIAutomation platform APIs
License: MIT License
UiaInt
's copy constructor works differently from its assignment operator, in that it simply creates an "alias" for the same remote object, rather than creating a new "deep" copy. Not only should the copy ctor always work the same way as the assignment operator, but this is definitely unexpected behavior for consumers.
While a workaround exists, we should still fix this.
After cloning the repository, installing missing NuGet packages and executing: msbuild UIAutomation.sln /p:Configuration=Release,Platform=x86
I got:
Microsoft (R) Build Engine 16.11.2+f32259642 dla platformy .NET Framework
Copyright (C) Microsoft Corporation. Wszelkie prawa zastrzeżone.
Projekty w tym rozwiązaniu są kompilowane pojedynczo. Aby umożliwić kompilację r
ównoległą, dodaj przełącznik "-m".
Kompilacja rozpoczęła się 2022-02-16 20:52:06.
Projekt "c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomation\UIAutomatio
n.sln" w węźle 1 (domyślne elementy docelowe).
ValidateSolutionConfiguration:
Tworzenie konfiguracji rozwiązania "Release|x86".
Projekt "c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomation\UIAutomatio
n.sln" (1) kompiluje "c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomatio
n\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj" (2) w węźle 1 (d
omyślne elementy docelowe).
InitializeBuildStatus:
Modyfikowanie "Release\Microsof.7D645239.tlog\unsuccessfulbuild".
Midl:
C:\Program Files (x86)\Windows Kits\10\bin\10.0.22000.0\x64\midl.exe /metadat
a_dir "C:\Program Files (x86)\Windows Kits\10\References\10.0.22000.0\Windows
.Foundation.FoundationContract\4.0.0.0" /winrt /W1 /nologo /char signed /env
win32 /winmd "Release\Microsoft.UI.UIAutomation.winmd" /h "Microsoft.UI.UIAut
omation_h.h" /tlb "Release\Microsoft.UI.UIAutomation.tlb" /target "NT60" /nom
idl /struct_by_ref Microsoft.UI.UIAutomation.idl
MIDLRT Processing .\Microsoft.UI.UIAutomation.idl
Microsoft.UI.UIAutomation.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\winrt\Windows.Foundation.idl
Windows.Foundation.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\winrt\inspectable.idl
inspectable.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\shared\wtypes.idl
wtypes.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\shared\wtypesbase.idl
wtypesbase.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\shared\basetsd.h
basetsd.h
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\shared\guiddef.h
guiddef.h
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\um\unknwn.idl
unknwn.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\winrt\hstring.idl
hstring.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\winrt\AsyncInfo.idl
AsyncInfo.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\winrt\EventToken.idl
EventToken.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\winrt\windowscontracts.idl
windowscontracts.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\winrt\IVectorChangedEventArgs.idl
IVectorChangedEventArgs.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\um\oaidl.idl
oaidl.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\um\objidl.idl
objidl.idl
MIDLRT Processing C:\Program Files (x86)\Windows Kits\10\Include\10.0.22000.0
\winrt\Windows.UI.UIAutomation.idl
Windows.UI.UIAutomation.idl
.\Microsoft.UI.UIAutomation.idl(9): error MIDL2025: syntax error : expecting .
near "runtimeclass" [c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomation
\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
.\Microsoft.UI.UIAutomation.idl(9): error MIDL2009: undefined symbol : unsealed
.AutomationRemoteObject [c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutoma
tion\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
.\Microsoft.UI.UIAutomation.idl(15): error MIDL2025: syntax error : expecting t
he keyword "interface" near "void" [c:\Users\Lukasz\Microsoft-UI-UIAutomation\s
rc\UIAutomation\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
.\Microsoft.UI.UIAutomation.idl(15): error MIDL2026: cannot recover from earlie
r syntax errors; aborting compilation [c:\Users\Lukasz\Microsoft-UI-UIAutomatio
n\src\UIAutomation\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
Kompilowanie projektu "c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomati
on\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj" wykonane (domyś
lne elementy docelowe) - NIEPOWODZENIE.
Kompilowanie projektu "c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomati
on\UIAutomation.sln" wykonane (domyślne elementy docelowe) - NIEPOWODZENIE.
Kompilacja NIE POWIODŁA SIĘ.
"c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomation\UIAutomation.sln" (
domyślny element docelowy) (1)->
"c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomation\Microsoft.UI.UIAuto
mation\Microsoft.UI.UIAutomation.vcxproj" (domyślny element docelowy) (2)->
(element docelowy Midl) ->
.\Microsoft.UI.UIAutomation.idl(9): error MIDL2025: syntax error : expecting
. near "runtimeclass" [c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAutomati
on\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
.\Microsoft.UI.UIAutomation.idl(9): error MIDL2009: undefined symbol : unseal
ed.AutomationRemoteObject [c:\Users\Lukasz\Microsoft-UI-UIAutomation\src\UIAuto
mation\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
.\Microsoft.UI.UIAutomation.idl(15): error MIDL2025: syntax error : expecting
the keyword "interface" near "void" [c:\Users\Lukasz\Microsoft-UI-UIAutomation
\src\UIAutomation\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxproj]
.\Microsoft.UI.UIAutomation.idl(15): error MIDL2026: cannot recover from earl
ier syntax errors; aborting compilation [c:\Users\Lukasz\Microsoft-UI-UIAutomat
ion\src\UIAutomation\Microsoft.UI.UIAutomation\Microsoft.UI.UIAutomation.vcxpro
j]
Ostrzeżenia: 0
Liczba błędów: 4
Czas, który upłynął: 00:00:02.18
I do not really understand why the version of Windows in use affects the result of the midl compiler. If this is impossible to fix or it is not worth fixing given Windows 7 age I would appreciate a clear note in the readme as to what versions of Windows are supported when building.
As the title mentions, the two types cannot be made elements of other collections. This is because they do not support necessary conversions and construction methods such as:
std::shared_ptr<std::vector<T>>
in case of UiaArray
),This means that types like UiaStringMap<UiaArray<UiaElement>>
(that -- for instance -- could list children of parents) cannot be compiled today.
This issue tracks fixing the compilation (and any other) problems with the collections to enable scenarios around more "structural" type nesting.
The Builder API has a comprehensive test suite implemented in closed-source code. There's no reason these tests couldn't be open-sourced and included in the (now-growing) functional test suite in this repo.
The below code does not compile due to multiple operators matching right-hand side values:
auto operationScope = UiaOperationScope::StartNew();
// Give this operation a remote context.
UiaElement displayElement{ focusedElement };
operationScope.BindInput(displayElement);
// Ask for the control type of the imported element (which is represented as a `UiaEnum`).
auto controlType = displayElement.GetControlType(false /* useCachedApi */);
// Compare against different control type representations.
auto equalToAbstractionValue = (controlType == controlType); // Does not compile [1]
auto equalToComEnum = (controlType == UIA_TextControlTypeId); // Does not compile [2]
auto equalToWinRtEnum = (controlType == winrt::Microsoft::UI::UIAutomation::AutomationControlType::Text);
auto notEqualToAbstractionValue = (controlType != controlType); // Does not compile [1]
auto notEqualToComEnum = (controlType != UIA_TextControlTypeId); // Does not compile [2]
auto notEqualToWinRtEnum = (controlType != winrt::Microsoft::UI::UIAutomation::AutomationControlType::Text);
// Return the result.
operationScope.BindResult(equalToAbstractionValue, equalToComEnum, equalToWinRtEnum, notEqualToAbstractionValue, notEqualToComEnum, notEqualToWinRtEnum);
operationScope.Resolve();
// Ensure the correct results have been returned.
Assert::IsTrue(static_cast<bool>(equalToAbstractionValue));
Assert::IsTrue(static_cast<bool>(equalToComEnum));
Assert::IsTrue(static_cast<bool>(equalToWinRtEnum));
Assert::IsFalse(static_cast<bool>(notEqualToAbstractionValue));
Assert::IsFalse(static_cast<bool>(notEqualToComEnum));
Assert::IsFalse(static_cast<bool>(notEqualToWinRtEnum));
Looks like this is a conflict between conversion operators (to local and remote types) and comparison operators (defined for the types).
It sounds like we could reduce our comparison operators to 2 types:
UiaEnum
,Plus, we could remove the comparisons against rvalue COM enums (that is already covered by other operators as far as I can tell).
UIA returns AutomationRemoteOperationStatus::InstructionLimitExceeded
when the number of instructions executed in the remote process exceeds 10000. However, currently the wrapper just transforms this into E_FAIL
in AutomationRemoteOperationResultSet::OperationStatus
. In UiaOperationScope::Resolve
, this E_FAIL
is plumbed through to the client.
We should provide a way to determine whether the remote operation exited with InstructionLimitExceeded
. I'm listing three potential ways here, although more are certainly possible:
E_FAIL
.UiaOperationScope::Resolve
throws on InstructionLimitExceeded
.AutomationRemoteOperationResultSet
to return whether InstructionLimitExceeded
occurred, and provide an alternative to UiaOperationScope::Resolve
which returns an AutomationRemoteOperationResultSet
.IUIAutomationElement
exposes an API that allows retrieving metadata information about element properties:
This issue tracks adding dedicated GetMetadataValue
helpers in the WinRT Builder and the C++ Abstraction library.
Newer versions of UIA allow UIA clients to Stringify
values of some types.
This issue tracks adding support for the instruction to the bytecode and to the UIA operation abstraction library.
This change aims to avoid common situations where UiaRect
or UiaPoint
need to be created before they can be given (often conditionally) values to store.
In such cases, the two types should be initialized to an empty rectangle and a (0, 0) point respectively. However, right now, that has to be done explicitly by providing RECT
and POINT
of the values.
We can improve it by creating default constructors for the two and using the "empty" values there.
This can be done by creating a custom provider that can be loaded by the ModernApp
test utility. That method involves registering appxes on the user's system, though, so we could also go the route of implementing a Win32 app to act as the provider.
In the following snippet...
void Fn(UiaElement original)
{
UiaElement newElement = original;
}
The newElement
will end up aliasing the same remote object. More precisely, it'll refer to the same operand register as original
(on the lowest-level of Remote Ops bytecode).
Meanwhile, in the following...
void Fn(UiaElement original)
{
UiaElement newElement{ nullptr };
newElement = original;
}
A Set
instruction is emitted in the bytecode that assigns the original element to a new operand register in the bytecode.
This means that two things that typically should work the same (the copy ctor and assignment operator) end up with subtly different behavior.
This issue tracks making a decision about this -- whether to try to make any changes to make these two operations work the same way or whether to document this rough edge as expected/by design.
MalformedBytecode
can occur when:
Currently UiaOperationAbstraction throws E_FAIL
from Resolve
when MalformedBytecode
is encountered. It might be good to throw a more informative error, or provide a mechanism to detect when the error is due to MalformedBytecode
.
UiaOperationScope has a useful utility method: ForEach which executes the given lambda on each element in the given array, whether running locally or remotely.
However, this method fetches each element from the array and passes it to the visitor function as const, which means it is impossible to modify the element from within the visitor function if the visitor function accepts the element by reference.
And because none of the Uia types have methods marked as const, it is therefore impossible to do much at all in the visitor function. For example,
scope.ForEach(myArray, [](auto& element){
auto element.GetName();
});
Results in an error like the following:
build\x86\UIARemote\UIARemote.cpp(367): error C2662: 'UiaOperationAbstraction::UiaString UiaOperationAbstraction::UiaElement::GetName(bool)': cannot convert 'this' pointer from 'const ItemWrapperType' to 'UiaOperationAbstraction::UiaElement &'
with
[
ItemWrapperType=UiaOperationAbstraction::UiaElement
]
build\x86\UIARemote\UIARemote.cpp(367): note: Conversion loses qualifiers
build\x86\microsoft-ui-uiautomation\include\UiaOperationAbstraction\UiaTypeAbstraction.g.h(2065): note: see declaration of 'UiaOperationAbstraction::UiaElement::GetName'
build\x86\microsoft-ui-uiautomation\include\UiaOperationAbstraction/UiaOperationAbstraction.h(2379): note: see reference to function template instantiation 'auto _remoteable_getTextContent::<lambda_c4afe82d3e528f79f41517a2c2e0fcb4>::operator ()<const ItemWrapperType>(const ItemWrapperType &) const' being compiled
with
[
ItemWrapperType=UiaOperationAbstraction::UiaElement
]
A workaround is to have the lambda accept the element by value rather than reference:
scope.ForEach(myArray, [](auto element){
auto element.GetName();
});
As I believe that copying the Uia* types is remotely cheap and ends up using the same underlying object anyway.
But personally I do prefer to code my functions taking Uia* types by reference as it just feels more symantically correct, and removes any guessing about whether copying is in deed cheep or not.
I'm also really not sure why scope.forEach deliberately fetches the element from the array as a const variable? I don't think the concept of const really makes any sense for these types, especially when none of the methods on them are marked as const.
Removing const here would be a one line change. I can open a pr to do this, but I first wanted to better understand why it was made const in the first place.
I don't think this would be possible in the wrappers. We'd need a new instruction, or potentially a modification to NewString which accepts a character. I think the former is the way to go, since modifying existing instructions doesn't really fit into our versioning model.
There's definitely also an argument that this isn't necessary, since someone could write their bytecode to deal in single-character strings, rather than actual character types. It's worth some discussion.
Thanks @mavangur for bringing this up.
The UIA platform supports indexing remote strings, but the UiaOperationAbstraction
helper library doesn't currently expose this.
There is no explicit reason to avoid UiaTuple make move assignments.
But UiaTuple cannot just get the inherent function from UiaTypeBase, so if we want to make a move assignment for UiaTuple, we may need to define the move assignment operator and move constructor for UiaTuple explicitly.
It has been suggested that a potentially good quality of life improvement for UiaOperationAbstraction
would be to add a version of BindInput
that accepts multiple arguments at once, so that binding multiple inputs could be done in a single line.
Something along the lines of...:
template<class... Args>
void BindInput(Args&... args)
{
(BindInputSingle(args), ...);
}
// Where BindInputSingle is what BindInput is today
We would like to be able to automatically build/validate the project as part of an Azure Pipelines build.
Since we're currently targeting an Insider Preview SDK version, this might be tricky to set up.
At the very least, we should add this once the project targets a stable version of the SDK.
Just cloned the project, acquired the latest sdk.. built the project with vs2019 Set the console demo to start and..
: 'Could not find Windows Runtime type 'Windows.UI.UIAutomation.Core.CoreAutomationRemoteOperation'.'
Any suggestions?
Some source files in the Remote Operation builder and abstraction libraries were auto-generated using a tool.
There's no reason the tool itself shouldn't be part of the project.
UiaAbstracion wrappers need to be implemented for the new Extensibility features. This includes CallExtension, IsExtensionTarget, IsExtensionTarget, IsExtensionSupported, UiaEmpty, UiaByteArray, new UiaConnectionBoundObject to be implemented which has their corresponding counter winrt wrappers already implemented.
Currently, there is no corresponding function in the wrapper for IUIAutomationElement::GetCurrentMetadataValue
, which means it is not convenient to get metadata through a remote operation. It would be nice to have something like:
UiaVariant UiaElement::GetMetadataValue(UiaPropertyId propertyId, UiaMetadata metadataId)
If takes one more step, to properly use this newly added GetMetadataValue
, we may need to cast the returned UiaVariant
to some other UiaEnum
types. For example: UiaSayAsInterpretAs sayAsInterpretAs = variant.AsType<UiaSayAsInterpretAs>()
However, the current UiaEnum
implementation does not have c_anyCast
, which means we cannot make such a cast. So, it would be nice to have this casting issue resolved.
The Builder's AutomationRemoteOperationResultSet
wraps the platform Result
object and exposes the operation results through its own set of methods. We should also let callers obtain the underlying platform object, which has some additional rich information that can e.g. help debugging.
Somewhat related to #45 in that it's about exposing additional information that the Builder holds internally to callers to ease their debugging scenarios.
The caller doesn't get their own copy, but rather a direct pointer to the internal data. This makes it easy to mis-use the returned pointer, such as assuming ownership or modifying the contents.
I am working with a UI tree element from Infragistics and I've encountered that the nodes within this tree have the IsOffscreen property set to true, which must be incorrect, since the nodes are clearly visible on screen.
The attached screenshot should illustrate the situation quite well:
I've been through Microsoft's API description of the IsOffscreen-property and I checked against all what it says, concerning the scrolling viewport visibility, the partial visibility, the collapsed- and expanded visibility, etc. and none of those things apply to my UI tree.
And since I keep stumbling across such issues within the UIAutomation framework from time to time, I'd guess it's simply a bug within the framework.
Best regards,
Markus
Currently, UiaOperationAbstraction.h
requires header consumers to pre-#include
headers such as winrt/Windows.Foundation.Collections.h
to consuume the UIA operation abstraction library.
Address this inconvenience by self-containing the header for easier consumption.
Because we cannot construct a UiaEnum from an AutomationRemoteAnyObject, UiaEnum currently cannot be included as part of a UiaTuple type.
It would probably be a good "quality of life" improvement to have a default value for this parameter (probably false
-- as in don't use the cached value), as that would simplify code that is primarily executed in a Remote Operation, where this value is irrelevant.
There are a lot of problems with custom patterns in c# (WPF).
HRESULT RegisterPattern(
[in] const UIAutomationPatternInfo *pattern,
[out] PATTERNID *pPatternId,
[out] PROPERTYID *pPatternAvailablePropertyId,
[in] UINT propertyIdCount,
[out] PROPERTYID *pPropertyIds,
[in] UINT eventIdCount,
[out] EVENTID *pEventIds
);
but when imported into c#, the PROPERTYID arrays are wrong:
void RegisterPattern([In] ref UIAutomationPatternInfo pattern,
out int pPatternId, out int pPatternAvailablePropertyId,
[In] uint propertyIdCount, out int pPropertyIds,
[In] uint eventIdCount, out int pEventIds);
I got around (1) by doing the RegisterPattern() in a c++ DLL that I included in my project, but even when I call GetCurrentPattern() on the registered pattern ID (from my c# automated UI test harness), it returns a null, and I've got a bunch of logging in the c# application being tested, and as far as I can tell the call to GetCurrentPattern() failed inside the test harness (no communication made to the application being tested)
Even if I can get GetCurrentPattern() to work in the test harness, the GetPattern() override in the application being tested has a single parameter which is an enum PatternInterface so it's not really able to pass my custom pattern ID.
Should I create these as separate issues? Or is UIAutomation/c# not a supported combination?
UIA provides IUIAutomationElement::BuildUpdatedCache
to request a new UIA element with fresh, current property values and pattern objects. Currently, there is no equivalent of that in the Remote Operation abstraction library.
This issue tracks adding appropriate support for it.
It could be useful for various clients to have an async version of Execute
that doesn't block the calling thread.
One way to do it in this undocked helper libraries is to dispatch the work to a thread pool thread (co_await winrt::resume_background();
) and do the work there.
Depending on feedback/input, the UIA platform might consider adding a "native" way to do async calls (without burning a threadpool thread) -- if it turns out that doing this in the wrapper only is insufficient for some user scenarios.
We should add support for arithmetic operators to wrapper library for types at lease like UiaInt, UiaUint and UiaDouble. Right now, we cannot do 'a + b' when both 'a' and 'b' are types like UiaInt.
Currently, it's not possible to build UiaOperationAbstraction
with the /permissive-
flag.
It seems that most of the errors are related to "two-phase name lookup".
We should fix these issues and then enable /permissive-
for this project.
As in any programming scenario, users often want to write a loop that just iterates over elements of a collection. To do this with the Remote Op abstraction, one can write this:
const auto arraySize = array.Size();
UiaUint elementIndex{ 0 };
operationScope.For(
[]() {} /* initialize */,
[&]() { return (elementIndex < arraySize); } /* condition */,
[&]() { elementIndex += 1; } /* modification */,
[&]() /* body */
{
const auto element = array.GetAt(elementIndex);
/* Now after all this ceremony do something with element */
});
We could add a helper to make this less verbose and far more readable:
operationScope.ForEach(array, [&](auto element)
{
/* Do something with element! */
});
Under the hood, this could do the same thing as the verbose code to set up a for loop. (Or it could even generate some more efficient bytecode in the future...)
From Adam Hodson:
If I put together a remote operation using the winRT builder api, is there a way to get back the generated byte code for the series of calls strung together? The presented slide on diagnostics functionality assumes you're using byte code for the error location so wouldn't be that helpful if you used the builder api?
Yes, exposing the actual bytecode generated by the builder out to the client would certainly be useful. We should add this.
We could perhaps have both a "current bytecode" exposed off of the AutomationRemoteOperation
itself (the bytecode that has been built up thus far), as well as an "executed bytecode" property on the ResultSet
(the bytecode that was ultimately sent for execution).
TextPattern2 GetCaretRange throw exception by receiving the result E_NOTIMPL from the porvider.
I expect to return null text range.
Since we use exceptions for breaking/continuing in loops in the local case, we need to make sure TryCatch
catches and rethrows those exceptions.
We also need to rethrow the success exception from UiaOperationDelegator::AbortOperationWithHresult
.
Currently, the solution only has x86 and x64 targets. We should configure ARM targets, as well.
Types in UiaOperationAbstraction can often be converted to their local and remote variants implicitly. For example, passing a UiaString
to a function that takes a wil::shared_bstr
will perform the conversion automatically due to the UiaString method operator wil::shared_bstr() const;
. These conversions throw if you try to convert a remote abstraction instance into its local variant, or a local abstraction instance into its remote variant. This can result in accidental and often confusing bugs.
It's worth having discussion on whether we want to move towards more explicit conversions to avoid these subtle bugs. This would be a breaking change for UiaOperationAbstraction consumers.
The below helper function implemented in the UiaOperationAbstraction.cpp file aims to convert a win32 RECT type to a Windows::Foundation::Rect, the returned Windows::Foundation::Rect is expected to represent same rectangle area of the RECT passed into this helper function.
winrt::Windows::Foundation::Rect ConvertRect(RECT rect)
But the order of parameters passed into constructor of Windows::Foundation::Rect is wrong, the order are supposed to be (left, top, height, width), but the parameters passed in are (height, width, left, top). This results in the returned rectangle with wrong position, e.g. the top left position is always at (0, 0).
Adjusting the order of input parameters passed into the constructor should be able to fix this issue.
I'm a dev on the MSVC compiler team and discovered a problem with the implementation of UiaBool while attempting to fix an overload resolution bug in the compiler.
Here's a reduced sample based on a subset of UiaBool that illustrates the problem: https://godbolt.org/z/57nMx5
For the test of a UiaBool with an implicit conversion to bool MSVC currently (but incorrectly) calls UiaBool::operator bool(). All other compilers, and MSVC with my fix, instead call UiaTypeBase::operator LocalT*, which will always yield 'true' as it's returning the non-null address of the variant member.
The main problem here is that the operator bool() overload will be a worse conversion than the base LocalT* conversion if the UiaBool object is not const (it involves a qualification conversion for the implicit object parameter, while the LocalT* conversion does not). This is because operator bool() is declared const, but operator LocalT*() is not.
There is a fundamental ambiguity here where there are implicit conversion sequences to both bool and BOOL*. One potential fix for this would be to make UiaBool::operator BOOL*() explicit.
This specific code snippet:
UiaElement element{ ... };
auto parent = element.GetParentElement(cacheRequest);
operationScope.BindResult(parent);
operationScope.Resolve();
should return a parent
with a cached requested data or no element at all. However, today, it fails if the parent could not be obtained due to not existing or existing across process/connection boundary. The failures come from assuming that the parent is always found and therefore that caching happens on a non-null
element.
This is unintended as the caller cannot control whether the parent exists or not and cannot prevent the caching from happening. We should fix that to continue executing the operation under the assumption that if the parent
has not been found, no attempts to cache its data should be made (and the operation should continue on under the assumption that parent
may or may not have been found -- just like it is for non-caching calls to the same interface method).
Original post at Q&A: https://docs.microsoft.com/en-us/answers/questions/929263/uiautomation-treewalker-misbehaving-in-microsoft-o.html
After getting a child element inside Picture Manager, using the TreeWalker to walk the UIA tree is behaving erratically.
I am running Windows 10 Pro 10.0.19044, and Microsoft Office 2010 Picture Manager.
How to reproduce (using inspect.exe tool in UI Automation mode):
Expected behaviour:
I tried to create a reproducible sample with C#, but the code freezes altogether. Open up a new Notepad window, and Microsoft Office 2010 Picture Manager window, and open a random image. Using System, System.Windows.Automation, System.Runtime.InteropServices:
AutomationElement rootElement = AutomationElement.RootElement;
AutomationElement npEl = rootElement.FindFirst(TreeScope.Children, new PropertyCondition(AutomationElement.NameProperty, "Untitled - Notepad"));
npEl.SetFocus();
Console.WriteLine("Found and activated Notepad");
AutomationElement documentEl = npEl.FindFirst(TreeScope.Descendants, new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Document));
Console.WriteLine("Document element name: " + documentEl.Current.Name);
AutomationElement pmEl = rootElement.FindFirst(TreeScope.Children, new PropertyCondition(AutomationElement.NameProperty, "Microsoft Office 2010"));
pmEl.SetFocus();
Console.WriteLine("Found and activated window");
AutomationElement imageEl = pmEl.FindFirst(TreeScope.Descendants, new PropertyCondition(AutomationElement.ControlTypeProperty, ControlType.Image));
Console.WriteLine("Image element name: " + imageEl.Current.Name);
Console.ReadLine();
This stops indefinitely at line 11.
To simplify code built with the UIA operation abstraction library, we should consider simplifying construction of the types provided by the library.
Specifically, types whose default values are implied or types that in other languages are often default-constructible could be good candidates, such as UiaString
that right now has to be constructed as UiaString{ L"" }
while the C++ equivalent of std::wstring
can be constructed without providing arguments.
The UIA platform supports accessing coordinates and dimensions of rectangles, but the UiaOperationAbstraction
helper library doesn't currently expose this.
When writing remote operation loop, consumers will frequently end up with a line such as this in the body of the loop:
operationScope.If(ancestorElement.IsNull(), [&]()
{
operationScope.Break();
});
It could be helpful to add a helper that lets consumers write:
operationScope.BreakIf(ancestorElement.IsNull());
This would be a little bit less verbose.
The same thing could be said for Continue
...
To make development of more complex Remote Operations easier, the abstraction library could benefit from struct
-like constructs.
For instance, we could add a tuple-like type that would be strongly typed (on the level of providing local and wrapper values -- just like in case of other UIA abstraction types) that could serve as a bridge between the current types and a struct
with named fields/values.
Alternatively, we could work with UiaStringMap
that contains UiaVariant
s but frankly that solution sounds slightly less convenient to me...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.