Commit Graph

66 Commits

Author SHA1 Message Date
Svetoslav
11adf6dc24 The touch exploration capability is dynamically granted pre-JellyBeanMR2.
Since the enable touch exploration capability is dynamically granted by
the user for apps targeting pre-JellybeanMR2 API level, we have to properly
update the accessibility service info for that service and also avoid
caching copies of the service info.

bug:8633951

Change-Id: I83dd1c852706ec55d40cda7209ad842889fb970a
2013-04-24 14:59:04 -07:00
Svetoslav
db7da0eb8b Fixing bugs exposed when moving accessibility CTS tests to UiAutomation.
1. UiAutomation#executeAndWaitForEvent method was invoking the passed
   runnable while holding the lock which may lead to a deadlock. For
   example: a runnable that calls getActivity() gets us into a state
   like this.

2. UI automation services did not get all capabilities such a
   service can have. Now a UI test service gets all of them.

3. When UiAutomation was exiting for event fired as a result of a
   performed action, it was checking whether the received evnet time
   is strictly before the time of executing the command that should
   fire the event. However, if the execution is fast enough, i.e.
   less than one millisecond, then the event time and the execution
   time are the same. This was leading to a missed signal in rare
   cases.

4. AccessibilityNodeInfoCache was not clearing the relevant state
   for accessibility focus clearing event.

5. Accessibility text traversal in TextView was partially using text
   and partially content description - broken. Now we are using the
   text since for text view and content desc for other views. In other
   words, we are using the most precise text we have.

6. AccessibilityManagerService was not granting capabilities of a
   UiAutomation service - plainly wrong.

CTS change:https://googleplex-android-review.googlesource.com/#/c/300693/

bug:8695422
bug:8657560

Change-Id: I9afc5c3c69eb51f1c01930959232f44681b15e86
2013-04-23 17:44:00 -07:00
Svetoslav
688a6977cf Replacing accessibility service permissions with capability attributes.
Accessibility services can perform special operations such as retrieve
the screen content, enable explore by touch, etc. To ensure the user
is aware that the service will perform special operations we were using
permissions. However, the special operations cannot be performed unless
the service is really enabled by the user and it is at this point that
we want to notify the user about the service capabilities.

This change adds capability attributes to the accessibility service's
meta-data XML file. The service has to declare the capability and when
it is enabled we show the user the capabilities in the warining dialog.

bug:8633951

Change-Id: Id3442dc71dad018e606888afdc40834682fdb037
2013-04-18 17:45:44 -07:00
Svetoslav
c4fccd183f Adding APIs for an accessibility service to intercept key events.
Now that we have gestures which are detected by the system and
interpreted by an accessibility service, there is an inconsistent
behavior between using the gestures and the keyboard. Some devices
have both. Therefore, an accessibility service should be able to
interpret keys in addition to gestures to provide consistent user
experience. Now an accessibility service can expose shortcuts for
each gestural action.

This change adds APIs for an accessibility service to observe and
intercept at will key events before they are dispatched to the
rest of the system. The service can return true or false from its
onKeyEvent to either consume the event or to let it be delivered
to the rest of the system. However, the service will *not* be
able to inject key events or modify the observed ones.

Previous ideas of allowing the service to say it "tracks" the event
so the latter is not delivered to the system until a subsequent
event is either "handled" or "not handled" will not work. If the
service tracks a key but no other key is pressed essentially this
key is not delivered to the app and at potentially much later point
this stashed event will be delivered in maybe a completely different
context.The correct way of implementing shortcuts is a combination
of modifier keys plus some other key/key sequence. Key events already
contain information about which modifier keys are down as well as
the service can track them as well.

bug:8088812

Change-Id: I81ba9a7de9f19ca6662661f27fdc852323e38c00
2013-04-11 16:00:08 -07:00
Svetoslav
bbfa585d70 Adding new APIs to UiAutomation.
1. Accessibility service may set a flag to request a compressed
   view of the node tree representing the screen. The compressed
   state does not contain nodes that do to react to user actions
   and do not draw content, i.e. they are dumb view managers. This
   compressed hierarchy was very beneficial to the test team and
   this change is exposing the APIs. The compression has to be
   configurable since old tests are written against uncompressed
   view tree. Basically we do not hide the fact that UIAutomation
   is simply an accessibility service with some other useful APIs.

bug:8051095

2. Accessibility service can perform global actions such as opening
   notifications, opening recent apps, etc. These are also needed
   for UI testing since there is no other way to do it via the
   existing UIAutomation APIs. Basically we do not hide the fact
   that UIAutomation is simply an accessibility service with some
   other useful APIs.

bug:8117582

Change-Id: I7b6e24b5f7a973fdada0cc199cff9f882b10720b
2013-02-11 19:59:45 -08:00
Svetoslav
57bf88508e Refactoring of the Accessibility.
1. The accessibility manager service updates its internal state
   based on which settings are enabled, what accessibility services
   are installed and what features are requested by the enabled
   services. It was trying to do the minimal amount of work to
   react to contextual changes like these which resulted in missed
   cases and complex code. Now there is a single method that reads
   the contextual information and single method that reacts to
   contextual changes. This makes the code much easier to maintain.

2. The accessibility manager service was not updating its internal
   state when requested features from accessibility services change.
   It was relying on changing system settings and reacting to the
   settings change. This is problematic since the internal state is
   not updated atomically which leads to race condition bugs. For
   example, if touch exploration is enabled and a service requests
   it is disabled, the internal state will not be updated but a
   request for a settings change will be made. Now while the settings
   change is propagating another request form the same service
   comes to enable touch exploration but the system incorrectly
   thinks touch exploration is enabled. At the end the feature is
   disabled even though it was requested.

3. Fixed a potential NPE if the accessibility input filter's event
   handler was nullified between processing two event batches.

4. Fixed a bug where, if magnification is enabled, it does not work
   on the settings screen since the magnified bounds are not pushed
   from the window manager to the accessibility manager.

Change-Id: Idf629a06480e12f0d88372762df6c024fe0d7856
2013-02-11 13:02:42 -08:00
Svetoslav
901309ce59 Updating AccessibilityNodeInfo JavaDoc
Change-Id: I8f80ed127c73dd4d8bcf9fe81575bb438b3d0f57
2013-02-04 18:16:52 -08:00
Svetoslav Ganov
447d94684e "Fixing backwards cmpatibility for enabling explore by touch""
This reverts commit d1ed3cea37

Change-Id: I622ef034526f8f006a5a233e72487b062020e4f6
2013-02-01 12:24:42 -08:00
Mike Cleron
d1ed3cea37 Revert "Fixing backwards cmpatibility for enabling explore by touch"
This reverts commit 0a9c7c1446

Change-Id: I8da530a6d0a57163522d5a63326e787a8d22ecbf
2013-02-01 18:36:41 +00:00
Svetoslav
0a9c7c1446 Fixing backwards cmpatibility for enabling explore by touch
Change-Id: I8cbc4f03223d289547e3f3f6d1f528ad5b4f6f72
2013-02-01 00:47:29 -08:00
Svetoslav
0ec0418c6e Update the JavaDoc for the new accessibility permissions
Change-Id: I1d52e848fec6c56cdb1f8ff66d07414dd2518086
2013-01-31 16:54:40 -08:00
Svetoslav
3822896e22 Remove "enhance web scripts" from settings and make it requested by plug-ins.
Currently we have an "enhance web accessibility" setting that has to be
enabled to make sure web content is accessible. We added the setting to
get user consent because we are injecting JavaScript-based screen-reader
pulled from the Google infrastructure. However, many users do not know
that and (as expected) do not read the user documentation, resulting in
critique for lacking accessibility support in WebViews with JavaScript
enabled (Browser, Gmail, etc).

To smoothen the user experience now "enhance web accessibility" is a
feature an accessibility plug-in can request, similarly to explore by
touch. Now a user does not need to know that she has to explicitly
enable the setting and web accessibility will work out-of-the-box.

Before we were showing a dialog when a plug-in tries to put the device
in a touch exploration mode. However, now that we have one more feature
a plug-in can request, showing two dialogs (assume a plug-in wants both
features) will mean that a user should potentially deal with three
dialogs, one for enabling the service, and one for each feature. We
could merge the dialogs but still the user has to poke two dialogs.

It seems that the permission mechanism is a perfect fit for getting
user permission for an app to do something, in this case to enable
an accessibility feature. We need a separate permission for explore
by touch and enhance web accessibility since the former changes the
interaction model and the latter injects JavaScript in web pages. It
is critical to get user consent for the script injection part so we
need a well-documented permission rather a vague umbrella permission
for poking accessibility features. To allow better grouping of the
accessibility permissions this patch adds a permission group as well.

bug:8089372

Change-Id: Ic125514c34f191aea0416a469e4b3481ab3200b9
2013-01-29 13:56:22 -08:00
Svetoslav
f7e50993ed AccessibilityNodeINfo cache not cleared when accessibility is disabled.
Change-Id: Ie674e8b2b1ece41a603bc4f39b5ce98e18c3484a
2013-01-22 18:15:40 -08:00
Svetoslav Ganov
80943d8daa Adding UI test automation APIs.
This change adds APIs support for implementing UI tests. Such tests do
not rely on internal application structure and can span across application
boundaries. UI automation APIs are encapsulated in the UiAutomation object
that is provided by an Instrumentation object. It is initialized by the
system and can be used for both introspecting the screen and performing
interactions simulating a user. UI test are normal instrumentation tests
and are executed on the device.

UiAutomation uses the accessibility APIs to introspect the screen and
a special delegate object to perform privileged operations such as
injecting input events. Since instrumentation tests are invoked by a shell
command, the shell program launching the tests creates a delegate object and
passes it as an argument to started instrumentation. This delegate
allows the APK that runs the tests to access some privileged operations
protected by a signature level permissions which are explicitly granted
to the shell user.

The UiAutomation object also supports running tests in the legacy way
where the tests are run as a Java shell program. This enables existing
UiAutomator tests to keep working while the new ones should be implemented
using the new APIs. The UiAutomation object exposes lower level APIs which
allow simulation of arbitrary user interactions and writing complete UI test
cases. Clients, such as UiAutomator, are encouraged to implement higher-
level APIs which minimize development effort and can be used as a helper
library by the test developer.

The benefit of this change is decoupling UiAutomator from the system
since the former was calling hidden APIs which required that it is
bundled in the system image. This prevented UiAutomator from being
evolved separately from the system. Also UiAutomator was creating
additional API surface in the system image. Another benefit of the new
design is that now test cases have access to a context and can use
public platform APIs in addition to the UiAutomator ones. Further,
third-parties can develop their own higher level test APIs on top
of the lower level ones exposes by UiAutomation.

bug:8028258

Also this change adds the fully qualified resource name of the view's
id in the emitted AccessibilityNodeInfo if a special flag is set while
configuring the accessibility service. Also added is API for looking
up node infos by this id. The id resource name is relatively more stable
compared to the generaed id number which may change from one build to
another. This API facilitate reuing the already defined ids for UI
automation.

bug:7678973

Change-Id: I589ad14790320dec8a33095953926c2a2dd0228b
2013-01-22 17:56:53 -08:00
Mita Yun
d44280e288 resolved conflicts for merge of ebcbfb3f to master
Change-Id: I0658dd4a1418fbad91d9663e5b52418a979dc683
2012-12-10 16:50:22 -08:00
Mita Yun
ed218c706d Use asynchronous messages for input method events.
Improves the throughput of IME event handling by ensuring that
input events do not get serialized behind UI traversal and
drawing messages such as when the UI is animating.

Added support for creating an asynchronous Handler as part of a
HandlerCaller.  It turns out we should be using an asynchronous
Handler not only in IME dispatch but also in accessibility and
wallpaper events where HandlerCaller is used.  So fixed those
services to also use an asynchronous Handler.

Change-Id: I0b19140c9d5ca6ee300c1a150c48312fd55ed8eb
2012-12-06 18:11:32 -08:00
Svetoslav Ganov
152e9bb81a Refactoring of the screen magnification feature.
1. The screen magnification feature was implemented entirely as a part of the accessibility
   manager. To achieve that the window manager had to implement a bunch of hooks for an
   external client to observe its internal state. This was problematic since it dilutes
   the window manager interface and allows code that is deeply coupled with the window
   manager to reside outside of it. Also the observer callbacks were IPCs which cannot
   be called with the window manager's lock held. To avoid that the window manager had
   to post messages requesting notification of interested parties which makes the code
   consuming the callbacks to run asynchronously of the window manager. This causes timing
   issues and adds unnecessary complexity.

   Now the magnification logic is split in two halves. The first half that is responsible
   to track the magnified portion of the screen and serve as a policy which windows can be
   magnified and it is a part of the window manager. This part exposes higher level APIs
   allowing interested parties with the right permissions to control the magnification
   of a given display. The APIs also allow a client to be registered for callbacks on
   interesting changes such as resize of the magnified region, etc. This part servers
   as a mediator between magnification controllers and the window manager.

   The second half is a controller that is responsible to drive the magnification
   state based on touch interactions. It also presents a highlight when magnified to
   suggest the magnified potion of the screen. The controller is responsible for auto
   zooming out in case the user context changes - rotation, new actitivity. The controller
   also auto pans if a dialog appears and it does not interesect the magnified frame.

bug:7410464

2. By design screen magnification and touch exploration work separately and together. If
   magnification is enabled the user sees a larger version of the widgets and a sub section
   of the screen content. Accessibility services use the introspection APIs to "see" what
   is on the screen so they can speak it, navigate to the next item in response to a
   gesture, etc. Hence, the information returned to accessibility services has to reflect
   what a sighted user would see on the screen. Therefore, if the screen is magnified
   we need to adjust the bounds and position of the infos describing views in a magnified
   window such that the info bounds are equivalent to what the user sees.

   To improve performance we keep accessibility node info caches in the client process.
   However, when magnification state changes we have to clear these caches since the
   bounds of the cached infos no longer reflect the screen content which just got smaller
   or larger.

   This patch propagates not only the window scale as before but also the X/Y pan and the
   bounds of the magnified portion of the screen to the introspected app. This information
   is used to adjust the bounds of the node infos coming from this window such that the
   reported bounds are the same as the user sees not as the app thinks they are. Note that
   if magnification is enabled we zoom the content and pan it along the X and Y axis. Also
   recomputed is the isVisibleToUser property of the reported info since in a magnified
   state the user sees a subset of the window content and the views not in the magnified
   viewport should be reported as not visible to the user.

bug:7344059

Change-Id: I6f7832c7a6a65c5368b390eb1f1518d0c7afd7d2
2012-12-03 10:38:48 -08:00
Svetoslav Ganov
e20a177d3f Adding a global accessibility action to open quick settings.
1. Added APIs for opening the quick settings to the StatusBarManagerService
   and the local StatausBarManager. The new APIs are protected by the old
   EXPAND_STATUS_BAR permission.
   Renamed the expand* and collapse* non-public APIs that are expanding
   the notifications to expandNotifications* collapseNotifications* to
   better convey what they do given that this change adds
   expandQuickSettings* and collapseQuickSettings*.
   Added a global action to the accessibility layer to expand the quick
   settings which is calling into the new status bar manager APIs.

bug:7030487

Change-Id: Ic7b46e1a132f1c0d71355f18e7c5a9a2424171c3
2012-09-25 16:07:59 -07:00
Svetoslav Ganov
eb9862fd4c Adding FEEDBACK_BRAILLE to AccessibilityServiceInfo.
1. The feedback type provided by braille devices differs from all
   already predefined types and this change defines braille
   feedback type. Braille does not fit in the already predefined
   categories.

bug:6998945

Change-Id: Ide6043bed03fdecd6d2ee45a08762f5bd07a2118
2012-09-06 19:40:34 -07:00
Scott Main
53b0fda5fd misc doc bugs.
Fix sdk samples path
Fix method signature in fragment doc code snippet
Fix typo in a11y javadoc  code snippet
Fix markup error in brands page
Fix markup error on content providers doc
Fix typo on about>start page

Change-Id: Ie2a528dd4fbd19e5cb1c0844f6fc6272669c8ab9
2012-08-06 15:43:01 -07:00
Dianne Hackborn
636fd528f7 Docs (mostly): enumerate JELLY_BEAN target SDK changes.
Okay this isn't entirely docs.  I have un-hidden the new permission
for binding to accessibility services; we were going to go out with
it hidden since it didn't go into factory ROM, but now that we are
doing other things we might as well expose it.

Also changed a log from E to W, since it is not an error.

Change-Id: I9226c95e3b63e12218fe41dc3f4290d9824a9e5c
2012-06-06 11:36:49 -07:00
Svetoslav Ganov
1e7456ba74 Fixing the build for real.
Change-Id: I792d029c7fff4d96fbe238ff4b3f268cb8603b95
2012-05-16 19:15:34 -07:00
Svetoslav Ganov
53e184d34e Accessibility service needs to request permission to be bound to.
1. Every accessibility services targeting JellyBean or higher has
   to request a special permission for the system to bind to it.

Change-Id: I6e579326bdf3597f148d6c67317455701ec8af68
2012-05-16 15:57:15 -07:00
Svetoslav Ganov
7b1e0c7046 Removing default accessibility gesture handling.
1. The initial design was to have some accessibility gestures
   being handled by the system if the gesture handling access
   service does not consume the gesture. However, we are not
   sure what a good default is and once we add a default handler
   we cannot remove it since people may rely on it. Thus, we
   take the simples approach and let the accessibility service
   handle the gestures. If no gestures are handled the system
   will work in explore by touch as before.

bug:5932640

Change-Id: I865a83549fa03b0141d27ce9713e9b7bb45a57b4
2012-05-13 12:39:51 -07:00
Svetoslav Ganov
e4abc512aa Remove activation gestures form reported and add a touch explore requesting flag.
1. Delegating activation gestures has several issues that we should
   decide how to handle if possible before allowing an accessibility
   service to take over them:

   A) It is needed that every view than can be clicked or long pressed on
      reacts to such as a response to calling performClick and performLongPress
      which is not necessary true since the view may watch the touch
      events and do its own click long click detection. As a result it may
      be possible that there are view a user cannot interact with in
      touch exploration mode but can if not in that mode.

   B) Clicking or long pressing on a different location in a view may yield
      different results, for example NumberPicker. Ideally such views have
      to implement AccessibilityNodeProvide which provider handles correctly
      the request for click long press on virtual nodes. Some apps however
      just fire different hover accessibility events when the user is over
      a specific semantic portion of the view but do not provide virtual
      nodes. Hence, a user will not be able to interact with such semantic
      regions but the system can achieve that by sending the click/long click
      at the precise location in the view that was last touch explored.

2. Adding a flag on accessibility service info to request explore by touch
   mode. There is no need to put the device in this mode if node of the currently
   enabled accessibility services supports it. Now the problem is inverted and
   the service has to explicitly state its capability.

3. Fixing a bug where includeImportantViews was ignored for automation
   services.

Change-Id: I3b29a19f24ab5e26ee29f974bbac2197614c9e2a
2012-05-09 16:17:20 -07:00
Guang Zhu
df549f8381 Make UiTestAutomationBridge see non-important views again
This problem was introduced in I74df9c24. The intention of the
change was still let UiTestAutomationBridge see the
non-important views, but there were bugs in the implementation:

1. AccessibilityManagerService was not really updating
   mIncludeNotImportantViews when mIsAutomation is true

2. Wrong constant is used to set the flag

Change-Id: Ia0a2e9ed9720bd0ea3a563e0b492e870a6ec1586
2012-05-09 14:32:15 -07:00
Svetoslav Ganov
2aaa6bbd53 Merge "Renaming the constants for activation and long press." into jb-dev 2012-05-08 18:21:02 -07:00
Svetoslav Ganov
87bd0dccf7 Renaming the constants for activation and long press.
1. We will have a double tap to activate an item and
   tap and hold to perform a long press.

Change-Id: Ibd9d8774e5a4693d8868d1128e55a82ff0c3a80b
2012-05-08 14:29:33 -07:00
Svetoslav Ganov
ef5889810c DefaultGestureHandlingHelperService should not include non-important views.
1. Since we are using a stateless proxy accessibility service to
   perform default accessibility gesture handling it shuld not
   operate against not important views.

bug:6422069

Change-Id: I74df9c2415ab3b164d9ac5873f7004c0459e2bfa
2012-05-07 18:05:31 -07:00
Svetoslav Ganov
65d98ad5b3 Remove accessibility gestures for clockwise and counter clock wise circle.
1. The clockwise and counter clockwise circle gestures are difficult to
   perform for a blind user. Some user studies have shown that so nuke them.

bug:6435260

Change-Id: I81d23bd3c450239fe17b4b8c60bafab00daa480a
2012-05-02 17:57:13 -07:00
Svetoslav Ganov
aa780c1109 Adding support for traversing the content of a node info at granularity.
1. A view that creates an accessibility node info may add to the info
   a list of granularity labels. These are granularities by which the
   source view can iterate over its content. For example a text view
   may support character, word link while a web view may additionally
   support buttons, tables, etc. There are actions on accessibility
   node info to go to the next/previous at a given granularity which
   is passesed as an argument.

2. Added Bundle argument to the APIs for performing accessibility
   actions. This is generic and extensible.

bug:5932640

Change-Id: I328cbbb4cddfdee082ab2a8b7ff1bd7477d8d6f9
2012-04-20 15:12:13 -07:00
Svetoslav Ganov
fefd20e927 Adding an opt-in mechanism for gesture detection in AccessibilityService.
1. An accessibility service has to explicitly opt in to be notified
   for gestures by the system. There is only one accessibility service
   that handles gestures and in case it does not handle a gesture
   the system performs default handling. This default handling ensures
   that we have gesture navigation even if no accessibility service
   would like to participate/customize the interaction model.

bug:5932640

Change-Id: Id8194293bd94097b455e9388b68134a45dc3b8fa
2012-04-19 22:08:42 -07:00
Svetoslav Ganov
0846e29d0b Adding API for getting the root node info in the active window.
1. This APIs is useful for accessibility services since very often
   they want to get access to the root node in the current window
   to determine where accessibility focus is and subsequently find
   a next view to take accessibility focus.

Change-Id: Ide5231f832d9ce2e9b14dc411dfd675f753a4edb
2012-04-18 18:50:57 -07:00
Svetoslav Ganov
005b83b0c6 Adding some more gestures and actions for accessibility.
1. Added more gesture for accessibility. After a meeting
   with the access-eng team we have decided that the current
   set of gestures may be smaller than needed considering
   that we will use four gestures for home, back, recents,
   and notifications.

2. Adding actions for going back, home, opening the recents,
   and opening the notifications.

3. Added preliminary mapping from some of the new gestures
   to the new actions.

4. Fixed a bug in the accessibility interaction controller
   which was trying to create a handled on the main looper
   thread which may be null if the queried UI is in the
   system process. Now the context looper of the root view
   is used.

5. Fixed a bug of using an incorrect constant.

6. Added a missing locking in a couple of places.

7. Fixed view comparison for accessibilityt since it was
   not anisymmetric.

bug:5932640
bug:5605641

Change-Id: Icc983bf4eafefa42b65920b3782ed8a25518e94f
2012-04-18 13:43:55 -07:00
Svetoslav Ganov
4213804541 Accessibility focus - framework
Usefulness: Keep track of the current user location in the screen when
            traversing the it. Enabling structural and directional
            navigation over all elements on the screen. This enables
            blind users that know the application layout to efficiently
            locate desired elements as opposed to try touch exploring the
            region where the the element should be - very tedious.

Rationale: There are two ways to implement accessibility focus One is
           to let accessibility services keep track of it since they
           have access to the screen content, and another to let the view
           hierarchy keep track of it. While the first approach would
           require almost no work on our part it poses several challenges
           which make it a sub-optimal choice. Having the accessibility focus
           in the accessibility service would require that service to scrape
           the window content every time it changes to sync the view tree
           state and the accessibility focus location. Pretty much the service
           will have to keep an off screen model of the screen content. This
           could be quite challenging to get right and would incur performance
           cost for the multiple IPCs to repeatedly fetch the screen content.
           Further, keeping virtual accessibility focus (i.e. in the service)
           would require sync of the input and accessibility focus. This could
           be challenging to implement right as well. Also, having an unlimited
           number of accessibility services we cannot guarantee that they will
           have a proper implementation, if any, to allow users to perform structural
           navigation of the screen content. Assuming two accessibility
           services implement structural navigation via accessibility focus,
           there is not guarantee that they will behave similarly by default,
           i.e. provide some standard way to navigate the screen content.
           Also feedback from experienced accessibility researchers, specifically
           T.V Raman, provides evidence that having virtual accessibility focus
           creates many issues and it is very hard to get right.
           Therefore, keeping accessibility focus in the system will avoid
           keeping an off-screen model in accessibility services, it will always
           be in sync with the state of the view hierarchy and the input focus.
           Also this will allow having a default behavior for traversing the
           screen via this accessibility focus that is consistent in all
           accessibility services. We provide accessibility services with APIs to
           override this behavior but all of them will perform screen traversal
           in a consistent way by default.

Behavior:  If accessibility is enabled the accessibility focus is the leading one
           and the input follows it. Putting accessibility focus on a view moves
           the input focus there. Clearing the accessibility focus of a view, clears
           the input focus of this view. If accessibility focus is on a view that
           cannot take input focus, then no other view should have input focus.
           In accessibility mode we initially give accessibility focus to the topmost
           view and no view has input focus. This ensures consistent behavior accross
           all apps. Note that accessibility focus can move hierarchically in the
           view tree and having it at the root is better than putting it where the
           input focus would be - at the first input focusable which could be at
           an arbitrary depth in the view tree. By default not all views are reported
           for accessibility, only the important ones. A view may be explicitly labeled
           as important or not for accessibility, or the system determines which one
           is such - default. Important views for accessibility are all views that are
           not dumb layout managers used only to arrange their chidren. Since the same
           content arrangement can be obtained via different combintation of layout
           managers, such managers cannot be used to reliably determine the application
           structure. For example, a user should see a list as a list view with several
           list items and each list item as a text view and a button as opposed to seeing
           all the layout managers used to arrange the list item's content.
           By default only important for accessibility views are regared for accessibility
           purposes. View not regarded for accessibility neither fire accessibility events,
           nor are reported being on the screen. An accessibility service may request the
           system to regard all views. If the target SDK of an accessibility services is
           less than JellyBean, then all views are regarded for accessibility.
           Note that an accessibility service that requires all view to be ragarded for
           accessibility may put accessibility focus on any view. Hence, it may implement
           any navigational paradigm if desired. Especially considering the fact that
           the system is detecting some standard gestures and delegates their processing
           to an accessibility service. The default implementation of an accessibility
           services performs the defualt navigation.

bug:5932640
bug:5605641

Change-Id: Ieac461d480579d706a847b9325720cb254736ebe
2012-04-13 19:05:24 -07:00
Joe Fernandez
110414928a am c1a0e54d: am b459b619: am 945b7cb3: Merge "docs: Accessibility Dev Guide (subsumes Accessibility Best Practices)" into ics-mr1
* commit 'c1a0e54de12aee41163b84a25ea8dfc8b64304dc':
  docs: Accessibility Dev Guide (subsumes Accessibility Best Practices)
2012-04-03 10:11:59 -07:00
Joe Fernandez
e1302edd40 docs: Accessibility Dev Guide (subsumes Accessibility Best Practices)
Change-Id: Id7e3f647042d2afd390abe851be1c3b561af33ca
2012-03-28 00:57:06 -07:00
Svetoslav Ganov
19a07d9747 Merge "AccessibilityServiceInfo does not convert correctly feedback type to string." 2012-03-09 16:05:10 -08:00
Svetoslav Ganov
c6c25f9e28 AccessibilityServiceInfo does not convert correctly feedback type to string.
1. When the feedback type is set to TYPES_ALL_MASK the method
   feedbackTypeToString was not generating a string with all
   feedback types. Also the generated string sometimes could
   have a comma appended after the last feedback type name
   - this happens if the feedback type contains unknown type
   flag.

Change-Id: I848c520cf7dae16dcd882f68275869c7ab656dd7
2012-03-09 16:01:23 -08:00
Ken Wakasa
f76a50ce8f Fix obvious typos under frameworks/base/core
Change-Id: Ia5fc3db1bb51824e7523885553be926bcc42d736
2012-03-09 22:48:43 +09:00
Svetoslav Ganov
ae7a51f143 UiTestAutomationBridge does not terminate its HandlerThread.
1. Now the thread is terminated in the disconnect() method
   and also it is made demon since it has no pupose outside
   the context of the bridge client.

bug:6053108

Change-Id: Idc25373fddf501eda4f875fea3e944367e4f04bf
2012-02-24 10:47:31 -08:00
Svetoslav Ganov
57c7fd5a43 Fixing issues with the AccessibilityNodeInfo cache.
1. Before there were two caches one in the app process that
   kept track only the ids of infos that were given to a
   querying client and one in the querying client that
   holds the infos. This design requires precise sync
   between the caches. Doing that is somehow complicated
   since the app has cache for each window and it has
   to intercept all accessibility events from that window
   to manage the cache. Each app has to have a cache for
   each querying client. This approach would guarantee that
   no infos are fetched twice but due to its stateful nature
   and the two caches is tricky to implement and adds
   unnecessary complexity. Now there is only one cache in
   the client and the apps are stateless. The client is
   passing flags to the app that are a clue what nodes to
   prefetch. This approach may occasionally fetch a node
   twice but it is considerably simpler and stateless
   from the app perspective - there is only one cache.
   Fetching a node more than once does not cause much
   overhead compared to the IPC.

Change-Id: Ia02f6fe4f82cff9a9c2e21f4a36747de0f414c6f
2012-02-23 18:51:04 -08:00
Svetoslav Ganov
0d04e24553 Improving accessibility APIs used for UI automation.
1. UiTestAutomationBridge was accessing the root node in the
   active window by tracking the accessibility event stream
   and keeping the last active window changing event. Now
   the bridge is stateless and the root node is fetched by
   passing special window and view id with the request to
   the system.

2. AccessibilityNodeInfos that are cached were not finished,
   i.e. not sealed, causing exception when trying to access
   their children or rpedecessors.

3. AccessibilityManagerService was not properly restoring its
   state after the UI automation bridge disconnects from it.
   I particular the devices was still in explore by touch mode
   event if no services are enabled and the sutomation bridge
   is disconnected.

4. ViewRootImpl for the focused window now fires accessibility
   events when accessibility is enabled to allow accessibility
   services to determine the current user location.

5. Several missing null checks in ViewRootImpl are fixed since
   there were scenraios in which a NPE can occur.

6. Update the internal window content querying tests.

7. ViewRootImpl was firing one extra focus event.
bug:6009813
bug:6026952

Change-Id: Ib2e058d64538ecc268f9ef7a8f36ead047868a05
2012-02-21 17:09:09 -08:00
Svetoslav Ganov
b765db590f Adding a method for retching the root node in UiTestAutomationBridge
bug:5974791

Change-Id: I46a8b6a259ba31f27fcd6a71d06dc34efdb4de76
2012-02-07 12:05:16 -08:00
Svetoslav Ganov
79311c4af8 Speedup the accessibility window querying APIs and clean up.
1. Now when an interrogating client requires an AccessibilibtyNodeInfo
   we aggressively prefetch all the predecessors of that node and its
   descendants. The number of fetched nodes in one call is limited to
   keep the APIs responsive. The prefetched nodes infos are cached in
   the client process. The node info cache is invalidated partially or
   completely based on the fired accessibility events. For example,
   TYPE_WINDOW_STATE_CHANGED event clears the cache while
   TYPE_VIEW_FOCUSED removed the focused node from the cache, etc.
   Note that the cache is only for the currently active window.
   The ViewRootImple also keeps track of only the ids of the node
   infos it has sent to each querying process to avoid duplicating
   work. Usually only one process will query the screen content
   but we support the general case. Also all the caches are
   automatically invalidated so not additional bookkeeping is
   required. This simple strategy leads to 10X improving the
   speed of the querying APIs.

2. The Monkey and UI test automation framework  were registering a
   raw event listener for accessibility events and hence perform
   connection and cache management in similar way to an AccessibilityService.
   This is fragile and requires the implementer to know internal framework
   stuff. Now the functionality required by the Monkey and the UI automation
   is encapsulated in a new UiTestAutomationBridge class. To enable this
   was requited some refactoring of AccessibilityService.

3. Removed the *doSomethiong*InActiveWindow methods from the
   AccessibilityInteractionClient and the AccessibilityInteractionConnection.
   The function of these methods is implemented by the not *InActiveWindow
   version while passing appropriate constants.

4. Updated the internal window Querying tests to use the new
   UiTestAutomationBridge.

5. If the ViewRootImple was not initialized the querying APIs of
   the IAccessibilityInteractionConnection implementation were
   returning immediately without calling the callback with null.
   This was causing the client side to wait until it times out. Now
   the client is notified as soon as the call fails.

6. Added a check to guarantee that Views with AccessibilityNodeProvider
   do not have children.

bug:5879530

Change-Id: I3ee43718748fec6e570992c7073c8f6f1fc269b3
2012-01-23 20:13:58 -08:00
Svetoslav Ganov
3d0edd31d8 AccessibilityService description not internationalized.
1. AccessibilityServiceInfo loading the description string on
   creation and using this value. Hence, changing the locale
   results in reporting incorrectly localized string. Added a
   new loadDescription method to return the localized version.

2. Deprecated the old method.

bug:5807616

Change-Id: Id82bda3c6a90fa6681b035d20da0b8688ed68da3
2012-01-03 17:05:38 -08:00
Svetoslav Ganov
f3b4f3163b resolved conflicts for merge of 26f7a81f to master
Change-Id: I6bf5fd7c0de7945cef84602dbe3a7bbed587700f
2011-11-30 18:34:56 -08:00
Svetoslav Ganov
d116d7c78a Fixing memory leaks in the accessiiblity layer.
1. AccessibilityInteractionConnections were removed from the
   AccessiiblityManagerService but their DeathRecipents were
   not unregistered, thus every removed interaction connection
   was essentially leaking. Such connection is registered in
   the system for every ViewRootImpl when accessiiblity is
   enabled and inregistered when disabled.

2. Every AccessibilityEvent and AccessiilbityEventInfo obtained
   from a widnow content querying accessibility service had a
   handle to a binder proxy over which to make queries. Hoewever,
   holding a proxy to a remote binder prevents the latter from
   being garbage collected. Therefore, now the events and infos
   have a connection id insteand and the hindden singleton
   AccessiiblityInteaction client via which queries are made
   has a registry with the connections. This class looks up
   the connection given its id before making an IPC. Now the
   connection is stored in one place and when an accessibility
   service is disconnected the system sets the connection to
   null so the binder object in the system process can be GCed.
   Note that before this change a bad implemented accessibility
   service could cache events or infos causing a leak in the
   system process. This should never happen.

3. SparseArray was not clearing the reference to the last moved
   element while garbage collecting thus causing a leak.

bug:5664337

Change-Id: Id397f614b026d43bd7b57bb7f8186bca5cdfcff9
2011-11-29 18:51:30 -08:00
Svetoslav Ganov
71b4e71c67 Minor documentation fixes for the APIs to expose virtual view tree.
Change-Id: I94b74196483fb55ca67e0a50eebab0412c88831c
2011-10-25 11:33:22 -07:00
Svetoslav Ganov
021078554b Adding APIs to enable reporting virtual view hierarchies to accessibility serivces.
Added an interface that is the contract for a client to expose a virtual
view hierarchy to accessibility services. Clients impement this interface
and set it in the View that is the root of the virtual sub-tree. Adding
this finctionality via compostion as opposed to inheritance enables apps
to maintain backwards compatibility by setting the accessibility virtual
hierarchy provider on the View only if the API version is high enough.

bug:5382859

Change-Id: I7e3927b71a5517943c6cb071be2e87fba23132bf
2011-10-21 11:45:44 -07:00