1. The crash was happening if: two active pointers are performing a drag;
there are some inactive pointers down; the main dragging pointer (we are
merging the dragging pointers into one) goes up; now an inactive pointer
goes up and the explorer tries to inject up for the dragging pointer
which is no longer in the event resulting in a crash. Basically two
problems: inactive pointers were not ignored; 2) having only one
active pointer should not only send the up event but also transition
the explorer in touch exploring state.
bug:6874128
Change-Id: I341fc360ebc074fe3919d5ba3b98ee5cb08dd71e
1. The keyguard force hides some windows when it is shown and as soon
as the keyguard goes away there windows are made visible. However,
the window transition that the keyguard is moving away is reported
before the force hidden windows are shown which makes the screen
magnifier compute the magnified region with an incomplete list of
windows of interest.
bug:7215285
Change-Id: I3abc4d97b7a74de8183ad20477dadf66c82da037
1. Since adb is restarted on user switch it makes no sense to
try to reconnect the ui automation service since it will
be killed on a user switch.
Disabling touch exploration on UI automation service
connect since it can explicitly put the device in this
state if needed.
bug:6967373
Change-Id: I8cfde74f28f3f03d4ccf24746d43b8178ae2b5ef
1. This change converts the accessibility manager service to
maintain a state per user. When the user changes the services
for the user that is going away are disconnected, the local
accessibility managers in the processes for this user are
disabled, the state is swapped with the new user's one, and
the new user state is refreshed.
This change updates all calls into the system to use their
user specific versions when applicable. For example, regisetring
content observers, package monitors, calls into other system
services, etc.
There are some components that are shared across users such
as UI created by the system process and the SystemUI package.
Such components are managed as a global state shared across
all users and are updated accordingly on a user switch. Since
the SystemUI is running in a normal app process this change
adds hidden APIs on the local window manager to allow the
SystemUI to notify the accessibility layer that it will run
accross users.
Calls to AccessibiltyManager's isEnabled(), isTouchExplorationEnabled()
and sendAccessibilityEvent return false or a are a nop for a
background user sice he should not send accessibility events,
and should not perform touch exploration.
Update the internal accessibility tests due to changes in the
AccessibilityManager.
This change also fixes several issues that were encountered
such as calling out the accessibility manager service with a
lock held.
Removed some incorrect debugging code from the TouchExplorer
that was leading to a system crash.
bug:6967373
Change-Id: I2cf32ffdee1d827a8197ae4ce717dc0ff798b259
1. Currently the system fires accessibility events to announce the
start and end of a touch exploration gesture. However, such a
gesture starts after we have decided that the user is not
performing a gesture which is achieved by measuring speed of
movement during a threshold distance. This allows an accessibility
service to provide some feedback to the user so he knows that
he is touch exploring.
This change adds event types for the first and last touches
of the user. Note that the first touch does not conincide with
the start of a touch exploration gesture since we need a time
or distance to pass before we know whether the user explores
or gestures. However, it is very useful for an accessibility
service to know when the user starts to interact with the
touch screen so it can turn the speech off, to name one
compelling use case.
This change also provides event types for the start and end
of gesture detection. If the user has moved over the threshold
with a speed greater than X, then the system detects gestures.
It is useful for an accessibility service to know the begin
and end of gesture detection so it can provide given feedback
type for such a gesture, say it may produce haptic feedback
or sound that differs for the one for touch exploration.
The main benefit of announcing these new events is that an
accessibility service can provide feedback for each touch
state allowing the user to always know what he is doing.
bug:7166935
Change-Id: I26270d774cc059cb921d6a4254bc0aab0530c1dd
1. This change enforces an accessibility service to require the system
defined BIND_ACCESSIBILITY_SERVICE permission.
bug:6507771
Change-Id: If5e16bb4fa97891be0ccbb35e343773712e33b98
1. The way for computing the magnified region was simplistic and
incorrect. It was ignoring window layering resulting in broken
behavior. For example, if the IME is up, then the everything else
is magnifed and the IME not. Now the keyguard appears and covers
the IME but the magnified region does not expand while it would
since the keyguard completely covers the not magnified IME window.
bug:7138937
Change-Id: I21414635aefab700ce75d40f3e913c1472cba202
1. When the screen goes off the user will be in a completely
different context upon turning the screen on. Therefore,
if magnification auto update is enabled magnification
will be disengaged on screen off.
bug:7139088
Change-Id: I790cfa5b3cf31d34e95fc9548e6246a84096c37b
1. If screen magnification is enabled the user has to triple tap
and lift or triple tap and hold to engage magnification. Hence,
we delay the touch events until we are sure that it is no longer
possible for the user to perform a multi-tap to engage
magnification. While such a delay is unavoidable it feels a
bit longer than it should be. This change reduces the delay
between taps to be considered a multi-tap, essentially making
the click delay shorter.
bug:7139918
Change-Id: I2100945171fff99600766193f0effdaef1f1db8f
1. If the user changes the magnification level while moving the
viewport the magnification is locked. The gesture handle has
to put device back into a viewport moving state if this was
the last state.
bug:7139363
Change-Id: I24992b973bb15624580114353b004efdb35c2faa
1. Before in magnified state the user was able to only scale or
pan. Based on user input this change allows performing pan
or scale or both. If the user scales more than a threshold
we are performing a scale and independently of that if the
use pans more than a threshold we are performing a pan.
bug:7138928
Change-Id: Ic1511500ba3369091dcfd070669d3e4f0286fbe5
1. Due to frequent changes of the behavior of ScaleGestureDetector
this patch rolls in a gesture detector used for changing the
screen magnification level. It has an improved algorithm which
uses the diameter of min circle around the points as the span, the
center of this circle as the focal point, and the average slop
of the lines from each pointer to the center to determine the
angle of the diameter used when computing the span x and y.
Change-Id: I5cee8dba84032a0702016b8f9632f78139024bbe
1. If screen magnification is disabled when the screen is in a
magnified state we have to zoom out since otherwise the user
is stuck in a magnified state without ability to pan/zoom/
toggle magnification which renders the device useless.
bug:7131030
Change-Id: I8f3339f31310448ec8742f3101c1fdc61a6a5f83
1. If screen magnification is disabled when the screen is in a
magnified state we have to zoom out since otherwise the user
is stuck in a magnified state without ability to pan/zoom/
toggle magnification which renders the device useless.
bug:7131030
Change-Id: Ia620954fbd594e7cd470e43b89d9ed04c0397c3c
This change is the initial check in of the screen magnification
feature. This feature enables magnification of the screen via
global gestures (assuming it has been enabled from settings)
to allow a low vision user to efficiently use an Android device.
Interaction model:
1. Triple tap toggles permanent screen magnification which is magnifying
the area around the location of the triple tap. One can think of the
location of the triple tap as the center of the magnified viewport.
For example, a triple tap when not magnified would magnify the screen
and leave it in a magnified state. A triple tapping when magnified would
clear magnification and leave the screen in a not magnified state.
2. Triple tap and hold would magnify the screen if not magnified and enable
viewport dragging mode until the finger goes up. One can think of this
mode as a way to move the magnified viewport since the area around the
moving finger will be magnified to fit the screen. For example, if the
screen was not magnified and the user triple taps and holds the screen
would magnify and the viewport will follow the user's finger. When the
finger goes up the screen will clear zoom out. If the same user interaction
is performed when the screen is magnified, the viewport movement will
be the same but when the finger goes up the screen will stay magnified.
In other words, the initial magnified state is sticky.
3. Pinching with any number of additional fingers when viewport dragging
is enabled, i.e. the user triple tapped and holds, would adjust the
magnification scale which will become the current default magnification
scale. The next time the user magnifies the same magnification scale
would be used.
4. When in a permanent magnified state the user can use two or more fingers
to pan the viewport. Note that in this mode the content is panned as
opposed to the viewport dragging mode in which the viewport is moved.
5. When in a permanent magnified state the user can use three or more
fingers to change the magnification scale which will become the current
default magnification scale. The next time the user magnifies the same
magnification scale would be used.
6. The magnification scale will be persisted in settings and in the cloud.
Note: Since two fingers are used to pan the content in a permanently magnified
state no other two finger gestures in touch exploration or applications
will work unless the uses zooms out to normal state where all gestures
works as expected. This is an intentional tradeoff to allow efficient
panning since in a permanently magnified state this would be the dominant
action to be performed.
Design:
1. The window manager exposes APIs for setting accessibility transformation
which is a scale and offsets for X and Y axis. The window manager queries
the window policy for which windows will not be magnified. For example,
the IME windows and the navigation bar are not magnified including windows
that are attached to them.
2. The accessibility features such a screen magnification and touch
exploration are now impemented as a sequence of transformations on the
event stream. The accessibility manager service may request each
of these features or both. The behavior of the features is not changed
based on the fact that another one is enabled.
3. The screen magnifier keeps a viewport of the content that is magnified
which is surrounded by a glow in a magnified state. Interactions outside
of the viewport are delegated directly to the application without
interpretation. For example, a triple tap on the letter 'a' of the IME
would type three letters instead of toggling magnified state. The viewport
is updated on screen rotation and on window transitions. For example,
when the IME pops up the viewport shrinks.
4. The glow around the viewport is implemented as a special type of window
that does not take input focus, cannot be touched, is laid out in the
screen coordiates with width and height matching these of the screen.
When the magnified region changes the root view of the window draws the
hightlight but the size of the window does not change - unless a rotation
happens. All changes in the viewport size or showing or hiding it are
animated.
5. The viewport is encapsulated in a class that knows how to show,
hide, and resize the viewport - potentially animating that.
This class uses the new animation framework for animations.
6. The magnification is handled by a magnification controller that
keeps track of the current trnasformation to be applied to the screen
content and the desired such. If these two are not the same it is
responsibility of the magnification controller to reconcile them by
potentially animating the transition from one to the other.
7. A dipslay content observer wathces for winodw transitions, screen
rotations, and when a rectange on the screen has been reqeusted. This
class is responsible for handling interesting state changes such
as changing the viewport bounds on IME pop up or screen rotation,
panning the content to make a requested rectangle visible on the
screen, etc.
8. To implement viewport updates the window manger was updated with APIs
to watch for window transitions and when a rectangle has been requested
on the screen. These APIs are protected by a signature level permission.
Also a parcelable and poolable window info class has been added with
APIs for getting the window info given the window token. This enables
getting some useful information about a window. There APIs are also
signature protected.
bug:6795382
Change-Id: Iec93da8bf6376beebbd4f5167ab7723dc7d9bd00
1. The window manager was not notifying a window when the latter
has been moved. This was causing incorrect coordinates of the
nodes reported to accessibility services. To workaround that
we have carried the correct window location when making a
call from the accessibility layer into a window. Now the
window manager notifies the window when it is moved and the
workaround is no longer needed. This change takes it out.
2. The left and right in the attach info were not updated properly
after a report that the window has moved.
3. The accessibility manager service was calling directly methods
on the window manager service without going through the interface
of the latter. This leads to unnecessary coupling and in the
long rung increases system complexity and reduces maintability.
bug:6623031
Change-Id: Iacb734b1bf337a47fad02c827ece45bb2f53a79d
1. There was a misspelled duplicate member in the accessibility service
class which was causing inconsistent behavior because one field was
updated and another checked.
2. When the set of services that can put the device in explore by touch
mode changes we were disconnecting and reconnecting all services
and this is not correct. Now only the state of explore by touch is
updated appropriately.
bug:6798860
Change-Id: Ib3c119cef8e71c3458d56e4ce6fbde2c2f750dcd
This fix adjusts the sensitivity of the gesture recognizer by
eliminating gesture rotation in the recognition process.
Bug:6697119
Change-Id: Ic767f513c05210b27e583338c4f0adcaa1c4c625
1. Accessibility allows querying only of the active window.
The active window is the one that has input focus or the
one the user is touching. Hence, if the user is touching
a window that does not have input focus this window is
the active one and as soon as the user stops touching
it the active window becomes the one that has input
focus. Currently the active window is not updated properly
when the user lifts his finger. This leads to a scenario
of traversal actions sent to the wrong window and the user
being stuck.
The reason is that the last touch explored event that is
used to determine where to click is cleared when accessibility
focus moves but this event is also used to determine when to
send the hover exit and touch exploration gesture end events.
The problem is that the last hover event is cleared before
it is used for sending the right exit events, thus the event
stream is inconsistent and the accessibility manager service
relies on this stream to update the active window. Now we
are keeping separate copies of the last touch event - one
for clicking and one for determining the which events to
inject to ensure consistent stream.
bug:6666041
Change-Id: Ie9961e562a42ef8a9463afacfff2246adcb66303
1. We are deciding whether the user is performing a gesture or an exploration based
on the gesture velocity. If we are detecting gesture we do the recognition at the
gesture end which is when the finger goes up. This is better than having a mode
toggle gesture for exploring and gestures detection. However, it is possible that
the user really wanted to perform an exploration but was moving too fast and
unless he lifts his finger the device is in gesture detection mode. This is
frustrating since the user has no feedback and assumes exploration does not
work.
We want to perform gesture detection only for a maximal time frame and if the
user did not lift his finger we transition into touch exploration state.
bug:6663173
Change-Id: I954ff937cca902e31b51325d1e1dfce84d239624
1. The touch explorer was notified for accessibility events from
a binder thread which was poking the internal state of the
latter which by design is not tread safe. Since the touch
explorer is expected to be running only on the main thread
the accessibility manager service delivers the accessibility
events to the explorer on that thread.
bug:6635496
Change-Id: Ifdc5329e4be8e485d7f77f0fb472184494fa0d15
1. AccessibilityInput filter was not checking whether the touch
explorer instance is not null before passing it an accessibility
event. If the accessibility event is dispatched before the input
filter is installed but after it is created we runt into this
case.
2. Added a missing null check in accessibility node info.
bug:6635089
Change-Id: Ia389dc1f427427eb73794f6331ccb870e0b44c55
1. When typing into an auto completion edit field a list of completions pops up and if
the user touch explores the list and tries to double tap to select the touched
completion the latter is not selected.
The auto completion is a popup that does not take input focus and is overlaid on
top of the window that has input focus. The touch explorer was clicking on the
location of the accessibility focus if the last touch explored location is within
the bounds of the active window. In this case this was the window with the edit
text into which the user is typing. The check performed by the touch explorer
was missing the case when the last touch explored location was within the bounds
of the active window but it actually was deloverd to another overlaid window.
Now we are poking on the accessibility focus location if the last explored
location is within the active window and was delivered to it.
bug:6629535
Change-Id: Ie66d5bb81ab021f2bb0414339b7de26d96826191
1. If the last touch explored location is within the active window we
used to click on exact location if it is within the accessibility
focus otherwise in the accessibility focus center. If the last touch
explored location is not within the active window we used to just
click there. This breaks in the case were one has touch explored
at a given place in the current window and now a dialog opens *not*
covering the touch explored location. If one uses swipes to move
accessibility focus i.e. to traverse the dialog without touching
it one cannot activate anything because the touch explorer is using
the last touch explored location that is outside of the active
window e.g the dialog.
The solution is to clear the last touch explored location when a
window opens or accessibility focus moves. If the last touch
explored location is null we are clicking in the accessibility
focus location.
bug:6620911
2. There is a bug in the window manager that does not notify a
window that its location has changed (bug:6623031). This breaks
accessibility interaction with dialogs that have input because
when the IME is up the dialog is moved but not notified. Now
the accessibility layer gets incorrect location for the
accessibility focus and the window bounds.
The soluion is when the accessibility manager service calls
into the remove thress to obtain some accessibility node infos
it passes the window left and top which it gets from the
window manager. These values are used to update the attach info
window left and top so all accessibility node infos emitted
from that window had correct bounds in screen coordinates.
bug:6620796
Change-Id: I18914f2095c55cfc826acf5277bd94b776bda0c8
1. Now after setting the content description on a view we mark is as
important for accessibility of the current important for accessibility
mode of that view is auto.
2. Minor tweak to a touch explorer coefficient to make performing double
tapping easier.
bug:6615353
Change-Id: I3b477f533a3ebde85d425caf32ace5e851240f88
1. The global action to open recent apps shows the old dialog style rent apps
panel. Apparently the key code to open recent apps is not opening the new
UI so the AccessibilityManagerService is calling directly the method on
the IStatusBarSerivce to do so.
bug:6607664
Change-Id: I94c1963b07947776bf1c2448903b26f3603f9a59
1. Touch exploration gestures are demarcated by start and end
events. Due to a bug in the AccessibilityManagerService
the gesture end event was not dispatched. This caused the
AccessibilityNodeInfoCache to be off sync since it relies
on getting such events not to mention that the clients were
not getting the end but only the start event. The issue
was that the notified service types variable was not reset
after every event so when the manager sends the last hover
exit it flags that the service type is already notified
resulting in dropping on the floor the following gesture
end event.
bug:6539306
Change-Id: I2b96bcecea3b2240199d67f01afa6a033afce1de
1. If the runnable for performing a long press is not
removed when all pointers are up and it is executed
the explorer gets into delegating mode with no pointer
down and the next down crashes the explorer. Added
code to remove the long press runnable in a few places
it was missing and also added a safety in the runnable
to avoid executing it in case there are no active pointers.
bug:6557183
Change-Id: I9dab3de88fd08d8e2b38af18249ac551837c0736
1. The long press routine was using the coordintates of the
accessibility focused item in the input focused window.
As a result double tap and hold did not work in a window
that does not take input focus such as the system bar.
Now the routine is using the last touch explored location
if it cannot find accessibility focus in the last touched
window.
bug:6584438
Change-Id: Ifd43adb20a066f389a9d4bd5716dd7ad834dd574
1. Now we are asking the user to grant permission to the service to enable
touch exploration only the first time this service is enabled. If the
service was uninstalled and then later installed we ask the user again.
This avoids the scenario in which rebooting the device or upgrading an
accessibility service leaves the device in a state in which the user
cannot interact with.
bug:6582088
Change-Id: I51d24e4892b3b48c9fb11dfb09ec1118502ba526
1. If a UI test automation accessibility service is connected to the
system we pospone state updates in the AccessibilityManagerService
for the moment the UI automations service dies or is disconnected.
bug:6540522
Change-Id: I48ddf603b53d2158a00edcf8ad05cfe2575d4d75
1. We are passing the interrogating process id in the remote
accessibility requests to catch the query from the same
thread. While all other methods were doing this correctly
somehow the perform action is using the incorrect process id.
bug:6534935
Change-Id: Icef50833903c562758d51ef316b60c53c7a336c0
1. The internal service instance created by AccessibilityManagerService
was getting the looper of the current thread when created. This works
for real accessibility services but since UI automation service is
registered via an IPC the binder thread has no looper. Now we explicitly
get the correct looper.
bug:6535435
Change-Id: I63a2ada1b65c4b3c71c3d1e6deb3dfdeb7a3d6d6
1. Now the user have to double tap to activate the last
item. If the last touched window is not active because
it does not take input focus the click on the last
touch explored location. Othewise the click is on the
accessibility focus location.
bug:5932640
Change-Id: Ibb7b97262a7c5f2f94abef429e02790fdc91a8dd
1. Every accessibility services targeting JellyBean or higher has
to request a special permission for the system to bind to it.
Change-Id: I6e579326bdf3597f148d6c67317455701ec8af68
