This is a follow up CL to our previuos CL [1], which enabled
AccessibilityService to use a subset of InputConnection APIs.
In that CL we have reused existing AIDL interfaces that were designed
and maintained for IMEs for simplicity, where a non trivial amount of
unnecessary IPC endpoints were included.
From the security and maintainability viewpoints, however, exposing
unnecessary IPC endpoints is discouraged in general. To address such
concerns this CL introduces a set of dedicated IPC definitions for
A11yIME so that we do not need to reuse IPCs for IMEs.
This CL also stops passing InputBinding object to A11yIME process as
it contains IInputContext Binder Proxy, which can still be used to
directly invoke fallback InputConnection. This is doable now because
A11yIME no longer relies on fallback InputConnection [2].
This CL is should not have any observable changes in the semantics.
End-to-end CTS tests guarantee that everything is still working as
intended now and in the future.
[1]: Ia651a811093a939d00c081be1961e24ed3ad0356
fb17e5ae7a
[2]: I2af3cd50444d8ddf25aa0f6479238156914e6fff
dc635efb68
Fix: 215633021
Fix: 215636776
Test: atest CtsInputMethodTestCases:AccessibilityInputMethodTest
Test: atest CtsAccessibilityServiceTestCases:AccessibilityInputConnectionTest
Test: atest CtsAccessibilityServiceTestCases:AccessibilityImeTest
Change-Id: I5ff2e804cbcf90828370a0612ff54111130bdff4
This is a follow up CL to our previous CL [1], which copied many
things from
android.inputmethodservice.InputMethodService
to
android.accessibilityservice.InputMethod.
As a result currently those two classes share the same systrace method
names as follows.
* IMS.startInput
* IMS.restartInput
With this CL, those systrace method names will be renamed to
* AccessibilityService.startInput
* AccessibilityService.restartInput
so that readers can understand which implementation is used without
having to check its process name.
There should be no user-observable behavior change in this CL.
[1]: Ia651a811093a939d00c081be1961e24ed3ad0356
fb17e5ae7a
Fix: 229118866
Test: Manually verified as follows
1. Build aosp_coral-userdebug and flash it.
2. make -j EditTextVariations
3. adb install -r $ANDROID_TARGET_OUT_TESTCASES/EditTextVariations/arm64/EditTextVariations.apk
4. make -j SampleInputMethodAccessibilityService
5. adb install -r $OUT/system/app/SampleInputMethodAccessibilityService/SampleInputMethodAccessibilityService.apk
6. adb shell settings put secure enabled_accessibility_services com.example.android.sampleinputmethodaccessibilityservice/com.example.android.sampleinputmethodaccessibilityservice.SampleInputMethodAccessibilityService
7. external/perfetto/tools/record_android_trace wm
8. adb shell am start -n com.android.inputmethod.tools.edittextvariations/.EditTextVariations
9. Tap an input field '<input type="text" />' in the WevView section.
10. Tap an input field '<input type="email" />' in the WevView section.
11. Hit Ctrl-C.
12. Verify trace event names from the sample A11yIME.
Change-Id: If89cfaec108a15aa168cfa32c689ca832b410324
Unlike InputMethodService, A11y IME should never receive fallback
InputConnection, which is subject to be deprecated or demoted in the
future.
Fix: 228848796
Test: atest CtsInputMethodTestCases:AccessibilityInputMethodTest
Change-Id: I2af3cd50444d8ddf25aa0f6479238156914e6fff
With my previous CL [1],
android.accessibilityservice.InputMethod#invalidateInputInternal
should now be executed during CTS.
Fix: 217788708
Test: atest CtsInputMethodTestCases:AccessibilityInputMethodTest
Change-Id: I74b06d4211d771f769f3ced6b8dedd37007257a0
We need to be able to initiate touch exploration even if there are no pointers on the screen in order to reliably cover the case where a user just taps the screen.
The user expects to hear if they have tapped on anything.
Bug: 223673301
Test: manual. Tap on the screen and confirm that talkback reads the item in that position.
Change-Id: Id96fa8bd7f000ea153f45024e06915a43ca80826
If this feature is not supported,
1. Settings should hide window magnification settings.
2. Magnification mode settings should fallback to.
full-screen settings.
Bug: 225086379
Test: none
Change-Id: I3248034d25eb88136bda1874092319d4702be455
Per API review feedback.
Test: it builds fine.
Fix: 219740392
Change-Id: Ibad4ead04f4420d8cd80969a9266a419c6ea93e8
(cherry picked from commit 8c4ce40ec9)
When the flag is set dynmatically, we need to create the InputMethod()
instance and request to bind ime. If the flag is unset, we clear the
InputMethod() instance and unbind ime.
Fix: 218193835
Test: Tested with talkback. When the flag is not set, we can't use
ime. When the flag is set, we can use ime again (dynamically set
flags).
Change-Id: Ied48d452108a593e8e4ee5c67c3208acbc87335e
Change-Id: I2a85c64b089983807cb7627de9b8512e8f716426
(cherry picked from commit 14e78228a1)
- Explicitly say that this is only the class name,to be looked up in the a11y services package, and not a full component name
- Fix typo of Javadoc refers to class names (plurar)
- Rename to getTileServiceName()
Bug: 219873375
Test: atest AccessibilityShortcutInfoTest
Change-Id: Ic7231f6e0b34edabea8b4086102e1d7f6ebe0eae
Make the prefetching strategies public so a service can choose
which strategy works best in a particular spot. This should reduce
unnecessary/redundant prefetching.
For example, the FW currently only allows hybrid descendant
prefetching, but a service may want to do depth-first or
breadth-first traversal of the view hierarchy.
Currently, if there is another user interactive request, we
immediately return prefetched nodes. Also allow services to prevent
this interruption and force prefetching to a max of 50 nodes.
Services could potentially request a certain number of nodes, but
since asynchronous prefetching immediately returns the requested
node, the service can force prefetching of 50 nodes if desired,
and only exposing strategies touches less code, I prefer limiting this.
Also use a LinkedHashMap so ordering is kept when prefetching
descendants.
Test: Manual, talkback builds
atest AccessibilityCacheTest,
AccessibilityInteractionControllerNodeRequestsTest
Bug: 192489177
Change-Id: I3d8358411ece5d2e1380282824cd3cf1835658ac
An accessibility service needs to set FLAG_INPUT_METHOD_EDITOR
accessibilityFlags to use ime apis. Capability is not added
per Phil's suggestion.
Bug: 187453053
Test: CTS test added.
Change-Id: I78703c48e343ba09c2a0bf61c3447eea7337cb49
InputMethodService is the primary and a11y is the secondary.
InputMethodService is not
affected by a11y status. When the session from input method is
established, app can start input (pass input context to input method).
When an a11y session comes back, it will be passed to the app. When
InputMethodManagerService binds to/start input with
InputMethodService, it does the same to a11y services which
requested IME functionalities.
It is possible that input method can edit text before ally sessios are
established. So the EditorInfo passed to a11y could be stale.
So when an a11y session is passed to client, client will send
a notification (input method doesn't have this extra notification) for
the current selection. I think since the time for a11y session establish
shouldn't be long, and we get the current state later, it should be fine
for a11y services.
When input method is disconnected from app (client) (even for input
method switching), we cleared a11y and sessions too. When
input method request sessions, we must rerequest sessions for a11y. This
is mainly because when we unbindCurrentClientLocked(SWITCH_IME), we set
active to false for the current client. Suppose we don't want to
change the current structure of input method, an inactive client
probably should clear accessibility sessions too.
When we switch to a client which already has a session with input
method, there might be some a11y sessions with this client, and
some a11y services might be disabled or enabled while the client was
switched out. We pass unchanged a11y
sessions to client, and request sessions for newly enabled a11y services.
When an a11y service is disabled, it removes its session
from all clients in InputMethodManagerService.
Test: type word through modified "switchToInputMethod". Tested session
notification through logs. Tested client switching, input method
switching, a11y service enabled/disable, multiple a11y services,
a11y service enabled before device reboot.
Also tested work profile.
Bug: 187453053
Change-Id: Ia651a811093a939d00c081be1961e24ed3ad0356
This is necessary to make the drag -> delegate execution path work.
Bug: 159651900
Test: atest TouchInteractionControllerTest
Change-Id: Idfb5d5ea8290353a7f4dac9742826a38fdd17b04
When transitioning between states, TouchInteractionController will queue motion events and not deliver them until the state transition has been confirmed.
This avoids the case where a service will request a state transition twice because it processes a motion event after sending the transition request but before the transition completes.
Bug: 159651900
Test: atest TouchInteractionControllerTest
Change-Id: I2437b635a8053de28212d93e67c7144134c1dc08
This reverts commit 331be9a643.
Reintroducing ag/16366278 since it seems unrelated to b/214053959 (more details on b/214053959#comment55).
Original commit message:
Migrate unsafe parcel APIs in framework-minus-apex
Migrate the following unsafe parcel APIs in framework-minus-apex:
* Parcel.readSerializable()
* Parcel.readArrayList()
* Parcel.readList()
* Parcel.readParcelable()
* Parcel.readParcelableList()
* Parcel.readSparseArray()
This CL was generated by applying lint fixes that infer the expected
type from the caller code and provide that as the type parameter
(ag/16365240).
A few observations:
* In some classes we couldn't migrate because the class also belonged to
another build module whose min SDK wasn't current (as is the case for
framework-minus-apex), hence I suppressed the lint check
(since I'll eventually submit the lint check to the tree).
* In some cases, I needed to do the cast in
https://stackoverflow.com/a/1080525/5765705 to make the compiler happy
since there isn't another way of providing a class of type
Class<MyClassWithGenerics<T>>.
* In the readSerializable() case, the new API also requires the class
loader, that was inferred to by InferredClass.class.getClassLoader().
* Note that automatic formatting and import rely on running hooked up
to the IDE, which wasn't the case here.
Bug: 195622897
Change-Id: I272432e6e082a973f7a50492ec35d79c2b577c93
Test: TH passes
Manually migrate the few cases of:
* readArray()
* readParcelableArray()
To the new parcel APIs that take the expected type as the last
parameter. This enhances security because it prevents unexpected types
*before* running unparcelling code. More details at go/safer-parcel.
Owners, please check that the type of the objects expected is always a
subtype of the type provided as the 3rd parameter. This is usually easy
to verify due to casts that happen shortly after.
These changes often allowed further transformations but I decided to
avoid them to keep this change small and targeted.
This was manual since it's tricky to get lint to infer the type in
those cases and it was only a few.
Bug: 195622897
Test: TH passes
Change-Id: I262ed7cd6d3bc15b32e9296e88a8a67fdb59e880
See go/b200769372 - the legacy public methods
The legacy magnification controller methods of a11y service,
getMagnificationRegion(),
reset(),
will keep the behavior before that control
only full-screen magnification.
To make the a11y service able to control the actiavted
magnifier on the display, the service should use the
new public APIs,
1. getCurrentMagnificationRegion(),
2. resetCurrentMagnification(),
instead of the legacy APIs.
Bug: 210069654
Test: atest AccessibilityMagnificationTest,
atest MagnificationProcessorTest,
atest AbstractAccessibilityServiceConnectionTest,
Change-Id: I76bfee43cbb76a9aeec33574a8db18430903e08c
See Design doc: go/b200769372
To make the behavior consistent on the new platform,
The legcay callback,
onMagnificationChanged(MagnificationController controller,
Region region, float scale, float centerX, float centerY)
keep notifying only full-screen magnification change as
before.
To support listening to the magnification changes of all
magnification modes, the service should overide
The new callback proposed in T,
onMagnificationChanged(MagnificationController controller,
Region region, MagnificationConfig config).
TODO: Notify magnifcation change when window magnifier turns off
Bug: 203013925
Test: atest AccessibilityMagnificationTest,
atest MagnificationControllerTest,
atest WindowMagnificationManagerTest,
atest WindowMagnificationControllerTest,
atest FullScreenMagnificationControllerTest,
Change-Id: Ia73195a71f55bb26ef3f2eafce0dc73d8a65583b
Migrate the following unsafe parcel APIs in framework-minus-apex:
* Parcel.readSerializable()
* Parcel.readArrayList()
* Parcel.readList()
* Parcel.readParcelable()
* Parcel.readParcelableList()
* Parcel.readSparseArray()
This CL was generated by applying lint fixes that infer the expected
type from the caller code and provide that as the type parameter
(ag/16365240).
A few observations:
* In some classes we couldn't migrate because the class also belonged to
another build module whose min SDK wasn't current (as is the case for
framework-minus-apex), hence I suppressed the lint check
(since I'll eventually submit the lint check to the tree).
* In some cases, I needed to do the cast in
https://stackoverflow.com/a/1080525/5765705 to make the compiler happy
since there isn't another way of providing a class of type
Class<MyClassWithGenerics<T>>.
* In the readSerializable() case, the new API also requires the class
loader, that was inferred to by InferredClass.class.getClassLoader().
* Note that automatic formatting and import rely on running hooked up
to the IDE, which wasn't the case here.
Bug: 195622897
Test: TH passes
Change-Id: I11a27b9bdab7959ee86e90aa1e1cbebd7aaf883c
Does not yet include the prefetching API, which will be introduced later.
- Have a CacheController as the source of cache manipulation
- Notify system to update user relevant types
Bug: 192489177
Test: Build, added cts tests, atest CacheControllerTest
Change-Id: I150a01192dcf764ffcfe9848b499034a48aded08
Make accessibility services able to enable/disable a specified
IME in the same package for the current user.
Bug: 195476910
Fix: 191386474
Test: atest AccessibilitySoftKeyboardTest. Also tested with
modified talkback (use the swith input method API for enable
ime), and self created empty a11y service with an empty ime.
Mananged device and profile which set limit on user 0 are also
tested.
Change-Id: I4187468076705ac597d680f2f5dc32d7b166da1f
See go/b200769372 - the legacy public methods
The legacy magnification controller methods,
setScale(), setCenter(),
getScale(), getCenterX(), getCenterY(),
getMagnificationRegion(),
reset(),
will keep the behavior before that control
only full-screen magnification.
To make the service able to control the actiavted magnifier
on the display, the service should use the new public APIs,
1. setMagnificationConfig(),
2. getMagnificationConfig(),
3. getCurrentMagnificationRegion()
instead of the legacy APIs.
This change
1. Updates the doc of the legcacy APIs.
2. The legacy APIs controls only fullscreen magnification
TODO: Add new public methods, getCurrentMagnificationRegion(),
resetCurrentMagnification() for the current controlling magnifier
Bug: 210069654
Test: atest MagnificationProcessorTest,
atest AbstractAccessibilityServiceConnectionTest,
atest AccessibilityMagnificationTest,
Change-Id: Ia157971d2c9716df7aa14024525ee0adfdc0d46f
The split-screen behavior is changed, so systemui doesn't
support this action with a single activity. We don't handle
this action by default from sc-v2. If SystemUI has any behavior
changed, they could register this action to support it.
Bug: 207056257
Test: presubmit pass
Change-Id: If569eff1ac52601e6b88324d7b47630a53e198f2
See go/b200769372
AccessibilityService adds new magnificationController APIs,
setMagnificationConfig and getMagnificationConfig, to control
magnification on demand.
The service can set the magnification mode, scale, center postion
of the magnifier on the display to control magnification.
And the service controls the least recently activated magnifier
on the display if the config mode is not specified.
Bug: 199732498
Test: atest MagnificationProcessorTest,
atest AbstractAccessibilityServiceConnectionTest,
atest AccessibilityMagnificationTest,
Change-Id: I51be2bbf96411c6c54c9da30873556d09533bb65
1. Add a range to the parameter in setScale(), setCenterX(), setCenterY()
2. The IntDef annotation name of MagnificationMode should not be
capitalized. The constants names should also start with the same prefix
set in the IntDef annotation.
3. Adds @hide for the IntDef annotation.
4. The related constants uses consistent prefix
Bug: 208422801
Test: none
Change-Id: Ibd997c3a13385d8884e208564092d829833ae637
