Skip to content

Commit 636aa27

Browse files
author
Rahim Abdi
committed
WebDriver: Add support for GetAccessibilityPropertiesForElement, GetAccessibilityPropertiesForAccessibilityNode commands
https://bugs.webkit.org/show_bug.cgi?id=299508 rdar://161303091 Reviewed by NOBODY (OOPS!). This patch implements two new WebDriver automation commands used to support more robust WPT accessibility testing: - GetAccessibilityPropertiesForElement - GetAccessibilityPropertiesForAccessibilityNode For this initial implementation, test_driver.get_accessibility_properties...() will return an accessibility property bag containing the following computed properties: - accessibilityNodeId - role - label - checked - pressed - children (array of serialized AXIDs) - parent (serialized AXID) As these new WebDriver commands are test-only interfaces, there's been in-depth discussion around how to treat computed AX properties when a property is missing and supported, or present & unsupported; see WICG/aom#203. Some additional resources: - WebDriver PR for the new commands: w3c/webdriver#1960 - ARIA PR for the initial set of computed accessibility properties, and treatment of undefined/missing/unsupported values: w3c/aria#2800. At a high level, the WebDriver client (i.e., Web Platform Tests' testdriver.js) initiates these new commands which is then dispatched to the UI Process (WebAutomationSession). The WebAutomationSession then parses the payload and extracts the protocol-specific element or accessibility node, after which it identifies the target (WebPageProxy) which invokes an asynchronous IPC message to the corresponding Web Process. Note that GetAccessibilityPropertiesForElement uses the same DOM node/frame-based approach as many other WebDriver commands however, GetAccessibilityPropertiesForAXNode does not require frame-based process routing (it retrieves computed accessibility properties using the page's axObjectCache, rather than a specific frame tied to a DOM node); thus, GetAccessibilityPropertiesForAccessibilityNode has a different message signature. Note that to fully support these new WebDriver commands, corresponding WebDriver HTTP endpoints must be implemented in Webdriver for Safari (safaridriver): - session/{session_id}/element/{elId}/accessibilityproperties: Returns accessibility properties for DOM element {elId} - session/{session_id}/accessibility/<accessibility node id>/properties/{accessibility node id}: Returns accessibility properties for accessibility node {accessibility node id} Due to the existence of two AccessibilityProperties protocol object definitions, documentation for each has been updated: - Source/WebKit/UIProcess/Automation/Automation.json — specification conformance and interoperability testing - Source/JavaScriptCore/inspector/protocol/DOM.json — Web Inspector DOM element inspection As an unrelated small fix, the return type for GetComputedLabel (WebAutomationSessionProxy IPC message) incorrectly names the first string parameter as "role" instead of "label" so this patch also corrects the tuple to return (label, errorType). * LayoutTests/resources/testdriver-vendor.js: (window.test_driver_internal.get_accessibility_properties_for_element): (window.test_driver_internal.get_accessibility_properties_for_accessibility_node): * Source/JavaScriptCore/inspector/protocol/DOM.json * Source/WebKit/UIProcess/Automation/Automation.json: * Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp: (Ref<JSON::ArrayOf<String>> buildArrayForAXChildren) (WebKit::WebAutomationSession::getAccessibilityPropertiesForElement): (WebKit::WebAutomationSession::getAccessibilityPropertiesForAccessibilityNode): * Source/WebKit/UIProcess/Automation/WebAutomationSession.h: * Source/WebKit/UIProcess/WebPageProxy.h * Source/WebKit/UIProcess/WebPageProxy.cpp: * Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.cpp: (WebKit::WebAutomationSessionProxy::getAccessibilityPropertiesForElement): (WebKit::WebAutomationSessionProxy::getAccessibilityPropertiesForAccessibilityNode): (WebKit::WebAutomationSessionProxy::getAccessibilityObjectForAXNode) (WebAutomationSessionProxy::ComputedAXProperties getComputedAccessibilityProperties) * Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.h: * Source/WebKit/WebProcess/Automation/WebAutomationSessionProxy.messages.in: * Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in * Source/WebKit/Scripts/webkit/messages.py * Source/WebCore/accessibility/AXObjectCache.h
1 parent 23903f1 commit 636aa27

13 files changed

Lines changed: 273 additions & 2 deletions

File tree

‎LayoutTests/resources/testdriver-vendor.js‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -713,6 +713,32 @@ window.test_driver_internal.get_computed_role = async function (element, context
713713
return context.internals.getComputedRole(element);
714714
}
715715

716+
/**
717+
*
718+
* @param {Element} element
719+
* @returns {Promise<Object>}
720+
*/
721+
window.test_driver_internal.get_accessibility_properties_for_element = async function (element, context=null) {
722+
if (!element)
723+
return {};
724+
context = context ?? window;
725+
726+
return context.internals.getAccessibilityPropertiesForElement(element);
727+
}
728+
729+
/**
730+
*
731+
* @param {string} accessibilityNodeId
732+
* @returns {Promise<Object>}
733+
*/
734+
window.test_driver_internal.get_accessibility_properties_for_accessibility_node = async function (accessibilityNodeId, context=null) {
735+
if (!element)
736+
return {};
737+
context = context ?? window;
738+
739+
return context.internals.getAccessibilityPropertiesForAccessibilityNode(accessibilityNodeId);
740+
}
741+
716742
/**
717743
*
718744
* @param {string} origin

‎Source/JavaScriptCore/inspector/protocol/DOM.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -101,7 +101,7 @@
101101
},
102102
{
103103
"id": "AccessibilityProperties",
104-
"description": "A structure holding accessibility properties.",
104+
"description": "A structure holding accessibility properties, either forwarded to the DOM Element from its backing Accessibility Node, and/or some local properties (e.g., ignored) that may clarify other characteristics, such as why a backing Accessibility Node is not available for this DOM Element.",
105105
"type": "object",
106106
"properties": [
107107
{ "name": "activeDescendantNodeId", "$ref": "NodeId", "optional": true, "description": "<code>DOMNode</code> id of the accessibility object referenced by aria-activedescendant." },

‎Source/WebCore/accessibility/AXObjectCache.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -106,6 +106,16 @@ enum class AXStreamOptions : uint16_t;
106106
enum class AXProperty : uint16_t;
107107
enum class LiveRegionStatus: uint8_t;
108108

109+
// When this is updated, WebCoreArgumentCoders.serialization.in must be updated as well.
110+
struct AccessibilityProperties {
111+
std::optional<String> accessibilityNodeId;
112+
std::optional<String> role;
113+
std::optional<String> label;
114+
std::optional<String> checked;
115+
std::optional<String> pressed;
116+
std::optional<String> parent;
117+
Vector<String> children;
118+
};
109119

110120
struct AXTreeData {
111121
String liveTree;

‎Source/WebKit/Scripts/webkit/messages.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1162,6 +1162,7 @@ def headers_for_type(type, for_implementation_file=False):
11621162
'WallTime': ['<wtf/WallTime.h>'],
11631163
'WebCore::AXDebugInfo': ['<WebCore/AXObjectCache.h>'],
11641164
'WebCore::AccessibilityMode': ['<WebCore/AXObjectCache.h>'],
1165+
'WebCore::AccessibilityProperties': ['<WebCore/AXObjectCache.h>'],
11651166
'WebCore::AccessibilityRemoteToken': ['<WebCore/AXObjectCache.h>'],
11661167
'WebCore::AccessibilitySearchCriteriaIPC': ['<WebCore/AXSearchManager.h>'],
11671168
'WebCore::AriaNotifyData': ['<WebCore/AXObjectCache.h>'],

‎Source/WebKit/Shared/WebCoreArgumentCoders.serialization.in‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4061,6 +4061,17 @@ header: <WebCore/AXObjectCache.h>
40614061
uint64_t webProcessLocalTokenHash;
40624062
};
40634063

4064+
header: <WebCore/AXObjectCache.h>
4065+
[CustomHeader] struct WebCore::AccessibilityProperties {
4066+
std::optional<String> accessibilityNodeId;
4067+
std::optional<String> role;
4068+
std::optional<String> label;
4069+
std::optional<String> checked;
4070+
std::optional<String> pressed;
4071+
std::optional<String> parent;
4072+
Vector<String> children;
4073+
};
4074+
40644075
header: <WebCore/AXObjectCache.h>
40654076
[CustomHeader] struct WebCore::AriaNotifyData {
40664077
String message;

‎Source/WebKit/UIProcess/Automation/Automation.json‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,20 @@
22
"domain": "Automation",
33
"description": "Automation domain exposes commands for automating user interactions with the browser.",
44
"types": [
5+
{
6+
"id": "AccessibilityProperties",
7+
"type": "object",
8+
"description": "Snapshot of a DOM node's computed accessibility properties used in WebDriver contexts.",
9+
"properties": [
10+
{ "name": "accessibilityNodeId", "type": "string" },
11+
{ "name": "role", "type": "string" },
12+
{ "name": "label", "type": "string" },
13+
{ "name": "checked", "type": "string" },
14+
{ "name": "pressed", "type": "string" },
15+
{ "name": "parent", "type": "string" },
16+
{ "name": "children", "type": "array", "items": { "type": "string" } }
17+
]
18+
},
519
{
620
"id": "Point",
721
"type": "object",
@@ -713,6 +727,32 @@
713727
],
714728
"async": true
715729
},
730+
{
731+
"name": "getAccessibilityPropertiesForElement",
732+
"description": "Get computed accessibility properties for a DOM element.",
733+
"parameters": [
734+
{ "name": "browsingContextHandle", "$ref": "BrowsingContextHandle", "description": "The handle for the browsing context." },
735+
{ "name": "frameHandle", "$ref": "FrameHandle", "optional": true, "description": "The handle for the frame that contains the element. The main frame is used if this parameter is empty string or excluded." },
736+
{ "name": "nodeHandle", "$ref": "NodeHandle", "description": "The handle of the element to get the accessibility properties for." }
737+
],
738+
"returns": [
739+
{ "name": "properties", "$ref": "AccessibilityProperties" }
740+
],
741+
"async": true
742+
},
743+
{
744+
"name": "getAccessibilityPropertiesForAccessibilityNode",
745+
"description": "Get computed accessibility properties for an accessibility node from WebKit's accessibility tree.",
746+
"parameters": [
747+
{ "name": "browsingContextHandle", "$ref": "BrowsingContextHandle", "description": "The handle for the browsing context." },
748+
{ "name": "accessibilityNodeHandle", "type": "string", "description": "The handle of the accessibility node to get the accessibility properties for." }
749+
],
750+
"returns": [
751+
{ "name": "properties", "$ref": "AccessibilityProperties" }
752+
],
753+
"async": true
754+
755+
},
716756
{
717757
"name": "selectOptionElement",
718758
"description": "Selects the given option element. In case of container with multiple options enabled, the element selectedness is toggled.",

‎Source/WebKit/UIProcess/Automation/WebAutomationSession.cpp‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -51,6 +51,7 @@
5151
#include <JavaScriptCore/ConsoleTypes.h>
5252
#include <JavaScriptCore/InspectorBackendDispatcher.h>
5353
#include <JavaScriptCore/InspectorFrontendRouter.h>
54+
#include <WebCore/AXObjectCache.h>
5455
#include <WebCore/MIMETypeRegistry.h>
5556
#include <WebCore/PointerEventTypeNames.h>
5657
#include <algorithm>
@@ -1683,6 +1684,61 @@ void WebAutomationSession::getComputedLabel(const Inspector::Protocol::Automatio
16831684
page->sendWithAsyncReplyToProcessContainingFrameWithoutDestinationIdentifier(frameID, Messages::WebAutomationSessionProxy::GetComputedLabel(page->webPageIDInMainFrameProcess(), frameID, nodeHandle), WTF::move(completionHandler));
16841685
}
16851686

1687+
static Ref<JSON::ArrayOf<String>> buildArrayForAXChildren(Vector<String>& axChildren)
1688+
{
1689+
auto childrenArray = JSON::ArrayOf<String>::create();
1690+
1691+
for (const auto& child : axChildren)
1692+
childrenArray->addItem(child);
1693+
1694+
return childrenArray;
1695+
}
1696+
1697+
static WTF::CompletionHandler<void(std::optional<String>, AccessibilityProperties)> makeAccessibilityPropertiesCompletionHandler(Inspector::CommandCallback<Ref<Inspector::Protocol::Automation::AccessibilityProperties>>&& callback)
1698+
{
1699+
return [callback = WTF::move(callback)](std::optional<String>&& optionalError, AccessibilityProperties&& properties) mutable {
1700+
ASYNC_FAIL_WITH_PREDEFINED_ERROR_IF_SET(optionalError);
1701+
auto axProperties = Inspector::Protocol::Automation::AccessibilityProperties::create()
1702+
.setAccessibilityNodeId(*properties.accessibilityNodeId)
1703+
.setRole(*properties.role)
1704+
.setLabel(*properties.label)
1705+
.setChecked(*properties.checked)
1706+
.setPressed(*properties.pressed)
1707+
.setParent(*properties.parent)
1708+
.setChildren(buildArrayForAXChildren(properties.children))
1709+
.release();
1710+
callback({ WTF::move(axProperties) });
1711+
};
1712+
}
1713+
1714+
void WebAutomationSession::getAccessibilityPropertiesForElement(const Inspector::Protocol::Automation::BrowsingContextHandle& browsingContextHandle, const Inspector::Protocol::Automation::FrameHandle& frameHandle, const Inspector::Protocol::Automation::NodeHandle& nodeHandle, Inspector::CommandCallback<Ref<Inspector::Protocol::Automation::AccessibilityProperties>>&& callback)
1715+
{
1716+
auto page = webPageProxyForHandle(browsingContextHandle);
1717+
ASYNC_FAIL_WITH_PREDEFINED_ERROR_IF(!page, WindowNotFound);
1718+
1719+
bool frameNotFound = false;
1720+
auto frameID = webFrameIDForHandle(frameHandle, frameNotFound);
1721+
ASYNC_FAIL_WITH_PREDEFINED_ERROR_IF(frameNotFound, FrameNotFound);
1722+
1723+
page->sendWithAsyncReplyToProcessContainingFrameWithoutDestinationIdentifier(
1724+
frameID,
1725+
Messages::WebAutomationSessionProxy::GetAccessibilityPropertiesForElement(page->webPageIDInMainFrameProcess(), frameID, nodeHandle),
1726+
Messages::WebAutomationSessionProxy::GetAccessibilityPropertiesForElement::Reply {
1727+
makeAccessibilityPropertiesCompletionHandler(WTF::move(callback))
1728+
});
1729+
}
1730+
1731+
void WebAutomationSession::getAccessibilityPropertiesForAccessibilityNode(const Inspector::Protocol::Automation::BrowsingContextHandle& browsingContextHandle, const String& accessibilityNodeHandle, Inspector::CommandCallback<Ref<Inspector::Protocol::Automation::AccessibilityProperties>>&& callback)
1732+
{
1733+
RefPtr page = webPageProxyForHandle(browsingContextHandle);
1734+
ASYNC_FAIL_WITH_PREDEFINED_ERROR_IF(!page, WindowNotFound);
1735+
1736+
1737+
protect(page->legacyMainFrameProcess()).sendWithAsyncReply(
1738+
Messages::WebAutomationSessionProxy::GetAccessibilityPropertiesForAccessibilityNode(page->webPageIDInMainFrameProcess(), accessibilityNodeHandle),
1739+
makeAccessibilityPropertiesCompletionHandler(WTF::move(callback)));
1740+
}
1741+
16861742
void WebAutomationSession::selectOptionElement(const Inspector::Protocol::Automation::BrowsingContextHandle& browsingContextHandle, const Inspector::Protocol::Automation::FrameHandle& frameHandle, const Inspector::Protocol::Automation::NodeHandle& nodeHandle, CommandCallback<void>&& callback)
16871743
{
16881744
auto page = webPageProxyForHandle(browsingContextHandle);

‎Source/WebKit/UIProcess/Automation/WebAutomationSession.h‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -272,6 +272,8 @@ friend class InspectorPassthroughChannel;
272272
void computeElementLayout(const Inspector::Protocol::Automation::BrowsingContextHandle&, const Inspector::Protocol::Automation::FrameHandle&, const Inspector::Protocol::Automation::NodeHandle&, std::optional<bool>&& scrollIntoViewIfNeeded, Inspector::Protocol::Automation::CoordinateSystem, Inspector::CommandCallbackOf<Ref<Inspector::Protocol::Automation::Rect>, RefPtr<Inspector::Protocol::Automation::Point>, bool>&&) override;
273273
void getComputedRole(const Inspector::Protocol::Automation::BrowsingContextHandle&, const Inspector::Protocol::Automation::FrameHandle&, const Inspector::Protocol::Automation::NodeHandle&, Inspector::CommandCallback<String>&&) override;
274274
void getComputedLabel(const Inspector::Protocol::Automation::BrowsingContextHandle&, const Inspector::Protocol::Automation::FrameHandle&, const Inspector::Protocol::Automation::NodeHandle&, Inspector::CommandCallback<String>&&) override;
275+
void getAccessibilityPropertiesForElement(const Inspector::Protocol::Automation::BrowsingContextHandle&, const Inspector::Protocol::Automation::FrameHandle&, const Inspector::Protocol::Automation::NodeHandle&, Inspector::CommandCallback<Ref<Inspector::Protocol::Automation::AccessibilityProperties>>&&) override;
276+
void getAccessibilityPropertiesForAccessibilityNode(const Inspector::Protocol::Automation::BrowsingContextHandle&, const String& accessibilityNodeHandle, Inspector::CommandCallback<Ref<Inspector::Protocol::Automation::AccessibilityProperties>>&&) override;
275277
void selectOptionElement(const Inspector::Protocol::Automation::BrowsingContextHandle&, const Inspector::Protocol::Automation::FrameHandle&, const Inspector::Protocol::Automation::NodeHandle&, Inspector::CommandCallback<void>&&) override;
276278
Inspector::CommandResult<bool> isShowingJavaScriptDialog(const Inspector::Protocol::Automation::BrowsingContextHandle&) override;
277279
Inspector::CommandResult<void> dismissCurrentJavaScriptDialog(const Inspector::Protocol::Automation::BrowsingContextHandle&) override;

‎Source/WebKit/UIProcess/WebPageProxy.cpp‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18665,6 +18665,7 @@ INSTANTIATE_SEND_WITH_ASYNC_REPLY_TO_PROCESS_CONTAINING_FRAME_WITHOUT_DESTINATIO
1866518665
INSTANTIATE_SEND_WITH_ASYNC_REPLY_TO_PROCESS_CONTAINING_FRAME_WITHOUT_DESTINATION_ID(WebAutomationSessionProxy::FocusFrame);
1866618666
INSTANTIATE_SEND_WITH_ASYNC_REPLY_TO_PROCESS_CONTAINING_FRAME_WITHOUT_DESTINATION_ID(WebAutomationSessionProxy::GetComputedRole);
1866718667
INSTANTIATE_SEND_WITH_ASYNC_REPLY_TO_PROCESS_CONTAINING_FRAME_WITHOUT_DESTINATION_ID(WebAutomationSessionProxy::GetComputedLabel);
18668+
INSTANTIATE_SEND_WITH_ASYNC_REPLY_TO_PROCESS_CONTAINING_FRAME_WITHOUT_DESTINATION_ID(WebAutomationSessionProxy::GetAccessibilityPropertiesForElement);
1866818669
INSTANTIATE_SEND_WITH_ASYNC_REPLY_TO_PROCESS_CONTAINING_FRAME_WITHOUT_DESTINATION_ID(WebAutomationSessionProxy::ResolveParentFrame);
1866918670
INSTANTIATE_SEND_WITH_ASYNC_REPLY_TO_PROCESS_CONTAINING_FRAME_WITHOUT_DESTINATION_ID(WebAutomationSessionProxy::SelectOptionElement);
1867018671
INSTANTIATE_SEND_WITH_ASYNC_REPLY_TO_PROCESS_CONTAINING_FRAME_WITHOUT_DESTINATION_ID(WebAutomationSessionProxy::ResolveChildFrameWithName);

‎Source/WebKit/UIProcess/WebPageProxy.h‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -270,6 +270,7 @@ enum class AttachmentAssociatedElementType : uint8_t;
270270
#endif
271271

272272
struct AXDebugInfo;
273+
struct AccessibilityProperties;
273274
struct AccessibilityRemoteToken;
274275
struct AccessibilitySearchCriteriaIPC;
275276
struct AppHighlight;

0 commit comments

Comments
 (0)