1. Now we are asking the user to grant permission to the service to enable
touch exploration only the first time this service is enabled. If the
service was uninstalled and then later installed we ask the user again.
This avoids the scenario in which rebooting the device or upgrading an
accessibility service leaves the device in a state in which the user
cannot interact with.
bug:6582088
Change-Id: I51d24e4892b3b48c9fb11dfb09ec1118502ba526
1. If a UI test automation accessibility service is connected to the
system we pospone state updates in the AccessibilityManagerService
for the moment the UI automations service dies or is disconnected.
bug:6540522
Change-Id: I48ddf603b53d2158a00edcf8ad05cfe2575d4d75
1. We are passing the interrogating process id in the remote
accessibility requests to catch the query from the same
thread. While all other methods were doing this correctly
somehow the perform action is using the incorrect process id.
bug:6534935
Change-Id: Icef50833903c562758d51ef316b60c53c7a336c0
1. The internal service instance created by AccessibilityManagerService
was getting the looper of the current thread when created. This works
for real accessibility services but since UI automation service is
registered via an IPC the binder thread has no looper. Now we explicitly
get the correct looper.
bug:6535435
Change-Id: I63a2ada1b65c4b3c71c3d1e6deb3dfdeb7a3d6d6
1. Now the user have to double tap to activate the last
item. If the last touched window is not active because
it does not take input focus the click on the last
touch explored location. Othewise the click is on the
accessibility focus location.
bug:5932640
Change-Id: Ibb7b97262a7c5f2f94abef429e02790fdc91a8dd
1. Every accessibility services targeting JellyBean or higher has
to request a special permission for the system to bind to it.
Change-Id: I6e579326bdf3597f148d6c67317455701ec8af68
1. Since the API version has been finalized this change
updates the SDk version checks to use the JellyBean
verson number.
bug:5947249
Change-Id: Ie22fa7e18a7ea7b0c7077d80246a26c17f327ceb
1. The initial design was to have some accessibility gestures
being handled by the system if the gesture handling access
service does not consume the gesture. However, we are not
sure what a good default is and once we add a default handler
we cannot remove it since people may rely on it. Thus, we
take the simples approach and let the accessibility service
handle the gestures. If no gestures are handled the system
will work in explore by touch as before.
bug:5932640
Change-Id: I865a83549fa03b0141d27ce9713e9b7bb45a57b4
1. Scrolling actions are crucial for enabling a gesture based
traversal of the UI and specifically scrollable containers
especially lists and anything backed by an adapter. Since
accessibility focus can land only attached views, it cannot
visit views for adapter items not shown on the screen.
Auto scrolling the list as a result of putting access focus
ot a list item does not work well since the user may get
trapped in a long list. Adding an accessibility node provider
to emit virtual views for one view before the first and one
after the last is complex and suffers the limitation of trapping
the user. Accessibility service need an explicit scroll actions
which may be performed upon an explicit user action. Hence,
the user is informed for the start/end of the visible part of
the list and he makes a deliberate choice to scroll. This will
benefit also people developing Braille devices since they can
scroll the content without telling the user to stop using the
Braille controller and take the device out of his pocket to scroll
and go back to the Braille controller.
NOTE: Without these action large portions of the screen will be
hard to access since users will have to touch and explore to
find and scroll the list.
Change-Id: Iafcf54d4967893205872b3649025a4e347a299ed
1. Delegating activation gestures has several issues that we should
decide how to handle if possible before allowing an accessibility
service to take over them:
A) It is needed that every view than can be clicked or long pressed on
reacts to such as a response to calling performClick and performLongPress
which is not necessary true since the view may watch the touch
events and do its own click long click detection. As a result it may
be possible that there are view a user cannot interact with in
touch exploration mode but can if not in that mode.
B) Clicking or long pressing on a different location in a view may yield
different results, for example NumberPicker. Ideally such views have
to implement AccessibilityNodeProvide which provider handles correctly
the request for click long press on virtual nodes. Some apps however
just fire different hover accessibility events when the user is over
a specific semantic portion of the view but do not provide virtual
nodes. Hence, a user will not be able to interact with such semantic
regions but the system can achieve that by sending the click/long click
at the precise location in the view that was last touch explored.
2. Adding a flag on accessibility service info to request explore by touch
mode. There is no need to put the device in this mode if node of the currently
enabled accessibility services supports it. Now the problem is inverted and
the service has to explicitly state its capability.
3. Fixing a bug where includeImportantViews was ignored for automation
services.
Change-Id: I3b29a19f24ab5e26ee29f974bbac2197614c9e2a
This problem was introduced in I74df9c24. The intention of the
change was still let UiTestAutomationBridge see the
non-important views, but there were bugs in the implementation:
1. AccessibilityManagerService was not really updating
mIncludeNotImportantViews when mIsAutomation is true
2. Wrong constant is used to set the flag
Change-Id: Ia0a2e9ed9720bd0ea3a563e0b492e870a6ec1586
1. Since we are using a stateless proxy accessibility service to
perform default accessibility gesture handling it shuld not
operate against not important views.
bug:6422069
Change-Id: I74df9c2415ab3b164d9ac5873f7004c0459e2bfa
1. Changed all references to granularity to movement
granularity. BTW, to be more precise it should be
text movement granularity.
bug:6435232
Change-Id: If6366b002ca3390f74918995b342baff2cbcfd01
1. The event of setting an accessibility focus on a view should not
make the host window the currently active one.
bug:6400648
Change-Id: Ib45c255f441c38489ee9d4ab5f284550ac5f6b01
1. The checks for action arguments are not needed since they
may cause trouble for developers if we add more args to
an action.
bug:6414006
Change-Id: Ia4212b52be183b1ef1cfd2561ce618cef2b015e4
1. The granularities for traversing the text content of an accessibility
node info are now predefined constants and custom ones will not be
supported. This is the simplest solution - we can always add namespaced
user defined ones (unlikely).
2. Added actions for traversing web content. These actions can be used by
an accessibility service to transparently drive the JavaScript based
screen reader that is used for handling web content.
3. Added a new accessibility event type for traversing the content of a
view. This event is needed to announce to the user what is the next
element, i.e. the one next to the cursor, after the view's text was
traversed.
bug:5932640
bug:6389591
Change-Id: I144647da55bc4005c64f89865ef333af8359e145
1. The accessibility focus directions are not needed since an
accessibility service just get the root, first child, next
sibling, previous sibling and call execute the action to
give it accessibility focus. Now the accessibility node
info tree is properly ordered taking into account layout
manager directions for both layout manager that we report
and ones that we have determined as not important for
accessibility. Also the position of a node info are ordered
properly based on their coordinates after all transformations
as opposed to child index.
bug:5932640
Change-Id: I994a8297cb1e57c829ecbac73a937c2bcbe0bac7
1. A view that creates an accessibility node info may add to the info
a list of granularity labels. These are granularities by which the
source view can iterate over its content. For example a text view
may support character, word link while a web view may additionally
support buttons, tables, etc. There are actions on accessibility
node info to go to the next/previous at a given granularity which
is passesed as an argument.
2. Added Bundle argument to the APIs for performing accessibility
actions. This is generic and extensible.
bug:5932640
Change-Id: I328cbbb4cddfdee082ab2a8b7ff1bd7477d8d6f9
1. The gesture dispatcher thread was not waiting in a loop
that check for complete initialization. Therefore is was
susceptible to missed signals and unexpected interrupts.
2. In the gesture processing message handle the interaction id
was reading the wrong message argument.
bug:5932640
Change-Id: Ic65ecc01a7fe7d43929c6c07d0759ae9001cf515
1. An accessibility service has to explicitly opt in to be notified
for gestures by the system. There is only one accessibility service
that handles gestures and in case it does not handle a gesture
the system performs default handling. This default handling ensures
that we have gesture navigation even if no accessibility service
would like to participate/customize the interaction model.
bug:5932640
Change-Id: Id8194293bd94097b455e9388b68134a45dc3b8fa
1. Added more gesture for accessibility. After a meeting
with the access-eng team we have decided that the current
set of gestures may be smaller than needed considering
that we will use four gestures for home, back, recents,
and notifications.
2. Adding actions for going back, home, opening the recents,
and opening the notifications.
3. Added preliminary mapping from some of the new gestures
to the new actions.
4. Fixed a bug in the accessibility interaction controller
which was trying to create a handled on the main looper
thread which may be null if the queried UI is in the
system process. Now the context looper of the root view
is used.
5. Fixed a bug of using an incorrect constant.
6. Added a missing locking in a couple of places.
7. Fixed view comparison for accessibilityt since it was
not anisymmetric.
bug:5932640
bug:5605641
Change-Id: Icc983bf4eafefa42b65920b3782ed8a25518e94f
Usefulness: Keep track of the current user location in the screen when
traversing the it. Enabling structural and directional
navigation over all elements on the screen. This enables
blind users that know the application layout to efficiently
locate desired elements as opposed to try touch exploring the
region where the the element should be - very tedious.
Rationale: There are two ways to implement accessibility focus One is
to let accessibility services keep track of it since they
have access to the screen content, and another to let the view
hierarchy keep track of it. While the first approach would
require almost no work on our part it poses several challenges
which make it a sub-optimal choice. Having the accessibility focus
in the accessibility service would require that service to scrape
the window content every time it changes to sync the view tree
state and the accessibility focus location. Pretty much the service
will have to keep an off screen model of the screen content. This
could be quite challenging to get right and would incur performance
cost for the multiple IPCs to repeatedly fetch the screen content.
Further, keeping virtual accessibility focus (i.e. in the service)
would require sync of the input and accessibility focus. This could
be challenging to implement right as well. Also, having an unlimited
number of accessibility services we cannot guarantee that they will
have a proper implementation, if any, to allow users to perform structural
navigation of the screen content. Assuming two accessibility
services implement structural navigation via accessibility focus,
there is not guarantee that they will behave similarly by default,
i.e. provide some standard way to navigate the screen content.
Also feedback from experienced accessibility researchers, specifically
T.V Raman, provides evidence that having virtual accessibility focus
creates many issues and it is very hard to get right.
Therefore, keeping accessibility focus in the system will avoid
keeping an off-screen model in accessibility services, it will always
be in sync with the state of the view hierarchy and the input focus.
Also this will allow having a default behavior for traversing the
screen via this accessibility focus that is consistent in all
accessibility services. We provide accessibility services with APIs to
override this behavior but all of them will perform screen traversal
in a consistent way by default.
Behavior: If accessibility is enabled the accessibility focus is the leading one
and the input follows it. Putting accessibility focus on a view moves
the input focus there. Clearing the accessibility focus of a view, clears
the input focus of this view. If accessibility focus is on a view that
cannot take input focus, then no other view should have input focus.
In accessibility mode we initially give accessibility focus to the topmost
view and no view has input focus. This ensures consistent behavior accross
all apps. Note that accessibility focus can move hierarchically in the
view tree and having it at the root is better than putting it where the
input focus would be - at the first input focusable which could be at
an arbitrary depth in the view tree. By default not all views are reported
for accessibility, only the important ones. A view may be explicitly labeled
as important or not for accessibility, or the system determines which one
is such - default. Important views for accessibility are all views that are
not dumb layout managers used only to arrange their chidren. Since the same
content arrangement can be obtained via different combintation of layout
managers, such managers cannot be used to reliably determine the application
structure. For example, a user should see a list as a list view with several
list items and each list item as a text view and a button as opposed to seeing
all the layout managers used to arrange the list item's content.
By default only important for accessibility views are regared for accessibility
purposes. View not regarded for accessibility neither fire accessibility events,
nor are reported being on the screen. An accessibility service may request the
system to regard all views. If the target SDK of an accessibility services is
less than JellyBean, then all views are regarded for accessibility.
Note that an accessibility service that requires all view to be ragarded for
accessibility may put accessibility focus on any view. Hence, it may implement
any navigational paradigm if desired. Especially considering the fact that
the system is detecting some standard gestures and delegates their processing
to an accessibility service. The default implementation of an accessibility
services performs the defualt navigation.
bug:5932640
bug:5605641
Change-Id: Ieac461d480579d706a847b9325720cb254736ebe
1. Before there were two caches one in the app process that
kept track only the ids of infos that were given to a
querying client and one in the querying client that
holds the infos. This design requires precise sync
between the caches. Doing that is somehow complicated
since the app has cache for each window and it has
to intercept all accessibility events from that window
to manage the cache. Each app has to have a cache for
each querying client. This approach would guarantee that
no infos are fetched twice but due to its stateful nature
and the two caches is tricky to implement and adds
unnecessary complexity. Now there is only one cache in
the client and the apps are stateless. The client is
passing flags to the app that are a clue what nodes to
prefetch. This approach may occasionally fetch a node
twice but it is considerably simpler and stateless
from the app perspective - there is only one cache.
Fetching a node more than once does not cause much
overhead compared to the IPC.
Change-Id: Ia02f6fe4f82cff9a9c2e21f4a36747de0f414c6f
1. UiTestAutomationBridge was accessing the root node in the
active window by tracking the accessibility event stream
and keeping the last active window changing event. Now
the bridge is stateless and the root node is fetched by
passing special window and view id with the request to
the system.
2. AccessibilityNodeInfos that are cached were not finished,
i.e. not sealed, causing exception when trying to access
their children or rpedecessors.
3. AccessibilityManagerService was not properly restoring its
state after the UI automation bridge disconnects from it.
I particular the devices was still in explore by touch mode
event if no services are enabled and the sutomation bridge
is disconnected.
4. ViewRootImpl for the focused window now fires accessibility
events when accessibility is enabled to allow accessibility
services to determine the current user location.
5. Several missing null checks in ViewRootImpl are fixed since
there were scenraios in which a NPE can occur.
6. Update the internal window content querying tests.
7. ViewRootImpl was firing one extra focus event.
bug:6009813
bug:6026952
Change-Id: Ib2e058d64538ecc268f9ef7a8f36ead047868a05
1. The AccessibilityManagerService used to disable the IU
automation service on package change. This behavior
was incorrect since the automation service has to
survive package installations.
bug:5975207
Change-Id: Idb5e76d02625c333a5842a6b5c5bc90c9b9634c9
1. Now when an interrogating client requires an AccessibilibtyNodeInfo
we aggressively prefetch all the predecessors of that node and its
descendants. The number of fetched nodes in one call is limited to
keep the APIs responsive. The prefetched nodes infos are cached in
the client process. The node info cache is invalidated partially or
completely based on the fired accessibility events. For example,
TYPE_WINDOW_STATE_CHANGED event clears the cache while
TYPE_VIEW_FOCUSED removed the focused node from the cache, etc.
Note that the cache is only for the currently active window.
The ViewRootImple also keeps track of only the ids of the node
infos it has sent to each querying process to avoid duplicating
work. Usually only one process will query the screen content
but we support the general case. Also all the caches are
automatically invalidated so not additional bookkeeping is
required. This simple strategy leads to 10X improving the
speed of the querying APIs.
2. The Monkey and UI test automation framework were registering a
raw event listener for accessibility events and hence perform
connection and cache management in similar way to an AccessibilityService.
This is fragile and requires the implementer to know internal framework
stuff. Now the functionality required by the Monkey and the UI automation
is encapsulated in a new UiTestAutomationBridge class. To enable this
was requited some refactoring of AccessibilityService.
3. Removed the *doSomethiong*InActiveWindow methods from the
AccessibilityInteractionClient and the AccessibilityInteractionConnection.
The function of these methods is implemented by the not *InActiveWindow
version while passing appropriate constants.
4. Updated the internal window Querying tests to use the new
UiTestAutomationBridge.
5. If the ViewRootImple was not initialized the querying APIs of
the IAccessibilityInteractionConnection implementation were
returning immediately without calling the callback with null.
This was causing the client side to wait until it times out. Now
the client is notified as soon as the call fails.
6. Added a check to guarantee that Views with AccessibilityNodeProvider
do not have children.
bug:5879530
Change-Id: I3ee43718748fec6e570992c7073c8f6f1fc269b3
1. AccessibilityInteractionConnections were removed from the
AccessiiblityManagerService but their DeathRecipents were
not unregistered, thus every removed interaction connection
was essentially leaking. Such connection is registered in
the system for every ViewRootImpl when accessiiblity is
enabled and inregistered when disabled.
2. Every AccessibilityEvent and AccessiilbityEventInfo obtained
from a widnow content querying accessibility service had a
handle to a binder proxy over which to make queries. Hoewever,
holding a proxy to a remote binder prevents the latter from
being garbage collected. Therefore, now the events and infos
have a connection id insteand and the hindden singleton
AccessiiblityInteaction client via which queries are made
has a registry with the connections. This class looks up
the connection given its id before making an IPC. Now the
connection is stored in one place and when an accessibility
service is disconnected the system sets the connection to
null so the binder object in the system process can be GCed.
Note that before this change a bad implemented accessibility
service could cache events or infos causing a leak in the
system process. This should never happen.
3. SparseArray was not clearing the reference to the last moved
element while garbage collecting thus causing a leak.
bug:5664337
Change-Id: Id397f614b026d43bd7b57bb7f8186bca5cdfcff9
Added an interface that is the contract for a client to expose a virtual
view hierarchy to accessibility services. Clients impement this interface
and set it in the View that is the root of the virtual sub-tree. Adding
this finctionality via compostion as opposed to inheritance enables apps
to maintain backwards compatibility by setting the accessibility virtual
hierarchy provider on the View only if the API version is high enough.
bug:5382859
Change-Id: I7e3927b71a5517943c6cb071be2e87fba23132bf
1. Due to a previous change that disabled accessibility if not enabled
and installed serivces are present the automation APIs stopped working
since they use fake automation service that is not installed.
2. Added clean up of death recipients when binders die.
bug:5374662
bug:5239044
Change-Id: I1f3c8cd1d1c79753a4a64e2b8b2963025abb2939
The system is didabling accessiblity if no accessibility serivces are enabled
to avoid sending events across processes if no recipients are present. The
check considers enabled services which may not have been installed. Now the
check is made against enabled and installed serivces.
bug:5347273
Change-Id: Iad391a1a5bf0bbca470584bc8392f35821ba768c
1. Added flags to the search method to specify whether to match text or
content description or both.
2. Added test case for the seach by content description.
3. Updated the code in AccessibilityManager service to reflect the latest
changes there so test automation service works - this is the fake
service used for UI automation.
Change-Id: I14a6779a920ff0430e78947ea5aaf876c2e66076
The pakcage monitor in the AccessibilityManagerService was not
watching for packages that are removed. This is needes since
1) we need to remove the package from the enabled accessibility
serivces and clean up after the removed serivice; 2) we need to
disable accessibility if the last access serivices went away.
Change-Id: I06d33b411ce60703e5a2843107323ffc87046c16
Accessibility was kept enabled even if all accessibility services
are disabled (explicitly by the user or removed) which was causing
the system to fire accessibility events that will never be consumed.
Change-Id: Ifb03e786ac0106687252bd1979725ffd724ad1c5
The content retrieval APIs are synchronous from a client's
perspective but internally they are asynchronous. The client thread
calls into the system requesting an action and providing a callback
to receive the result after which it waits up to a timeout for that
result. The system enforces security and then delegates the request
to a given view hierarchy where a message is posted (from a binder
thread) describing what to be performed by the main UI thread the
result of which it delivered via the mentioned callback. However,
the blocked client thread and the main UI thread of the target view
hierarchy can be the same one, for example an accessibility service
and an activity run in the same process, thus they are executed on the
same main thread. In such a case the retrieval will fail since the UI
thread that has to process the message describing the work to be done
is blocked waiting for a result is has to compute! To avoid this scenario
when making a call the client also passes its process and thread ids so
the accessed view hierarchy can detect if the client making the request
is running in its main UI thread. In such a case the view hierarchy,
specifically the binder thread performing the IPC to it, does not post a
message to be run on the UI thread but passes it to the singleton
interaction client through which all interactions occur and the latter is
responsible to execute the message before starting to wait for the
asynchronous result delivered via the callback. In this case the expected
result is already received so no waiting is performed.
bug:5138933
Change-Id: I382e2d8689f5189110226613c2387f553df98bd3
1. Events not generated by the user can change the interrogation allowing window
unpredicatably. For example when a ListView lays out its children it fires an
accessibility events and changes the currently active window while the user
interaction may be happening in another window say a dialog. Now the interrogation
allowing window is changed when a new window is shown or the user has touch
explored it.
bug:5074116
Change-Id: I8dde12bbec807d32445a781eedced9b95312b3e2
1. Upon registration of an accessibility client the latter received only
the accessiiblity state and waiting for the touch exploration state
to be sent by the system in async manner. This led the very first
check of touch exploration state is checked a wrong value to be reported.
Now a state of the accessibility layer is returned to the client
upon registration.
2. Removing the dependency on talking accessibility service to be enabled
for getting into touch exploration mode. What if the user wants to use
an accessibility service that shows a dialog with the text of the touched
view?
bug:5051546
Change-Id: Ib377babb3f560929ee73bd3d8b0d277341ba23f7
1. Seperated touch exploration to be a seperate setting rather being
magically enabled by the system of accessiiblity is on the there
is at leas one accessibility service that speaks enabled. Now
there is a setting for requesting touch exploration but still the
system will enabled it only if that makes sense i.e. accessibility
is on and one accessibility service that speaks is enabled.
2. Added public API for checking of touch exploration is enabled.
3. Added description attribute in accessibility service declaration
which will be shown to the user before enabling the service.
4. Added API for quick cloning of AccessibilityNodeInfo.
5. Added clone functionality to SparseArray, SparseIntArray, and
SparseBooleanArray.
bug:5034010
bug:5033928
Change-Id: Ia442edbe55c20309244061cd9d24e0545c01b54f
1. If an accessibility service does not request access to the window
content and does so, an exception is thrown to point the developer
to the reason.
bug:5038284
Change-Id: Ibf08f4d2c8ad8939c4f4c2e288048a4f8ff1e31b
1. Touch exploration start and end events are generated
by the sytstem to provide additional information for
accessibility services. Since such events do not come
from any particular window they whould not change the
id of the window that currently allows exploring its
content.
2. Touch exploration start and end events were lealing the
touch explorer class wich is private.
bug:5026258
Change-Id: Icaf3e2bd9566716f2afb876cf8e0d50813b0c76e
