Finish fixed rotation of recents if it is invisible

The recents animation may be finished due to device is going to
sleep, and then the recents activity will be invisible so there
won't be a transition finish callback due to no visibility change
when cleaning up recents animation.

So if recents activity was invisible before finishing animation,
just follow the current display orientation and reset the fixed
rotation states.

Bug: 171264143
Test: atest RecentsAnimationControllerTest# \
      testClearFixedRotationLaunchingAppAfterCleanupAnimation

Change-Id: Ic2028b5e1d960d1dcebbc1a56f44f93326335d57
(cherry picked from commit 9f72932f9d)

services/core/java/com/android/server/wm/DisplayContent.java

@@ -5703,7 +5703,8 @@ class DisplayContent extends WindowContainer<DisplayContent.DisplayChildWindowCo
                 return;
             }
 
-            if (animatingRecents != null && animatingRecents == mFixedRotationLaunchingApp) {
+            if (animatingRecents != null && animatingRecents == mFixedRotationLaunchingApp
+                    && animatingRecents.isVisible()) {
                 // The recents activity should be going to be invisible (switch to another app or
                 // return to original top). Only clear the top launching record without finishing
                 // the transform immediately because it won't affect display orientation. And before

services/tests/wmtests/src/com/android/server/wm/RecentsAnimationControllerTest.java

@@ -395,6 +395,21 @@ public class RecentsAnimationControllerTest extends WindowTestsBase {
         // The rotation transform should be cleared after updating orientation with display.
         assertFalse(activity.hasFixedRotationTransform());
         assertFalse(mDefaultDisplay.hasTopFixedRotationLaunchingApp());
+
+        // Simulate swiping up recents (home) in different rotation.
+        final ActivityRecord home = mDefaultDisplay.getDefaultTaskDisplayArea().getHomeActivity();
+        mDefaultDisplay.setFixedRotationLaunchingApp(home, (mDefaultDisplay.getRotation() + 1) % 4);
+        mController = new RecentsAnimationController(mWm, mMockRunner, mAnimationCallbacks,
+                mDefaultDisplay.getDisplayId());
+        initializeRecentsAnimationController(mController, home);
+        assertTrue(home.hasFixedRotationTransform());
+
+        // Assume recents activity becomes invisible for some reason (e.g. screen off).
+        home.setVisible(false);
+        mController.cleanupAnimation(REORDER_MOVE_TO_ORIGINAL_POSITION);
+        // Although there won't be a transition finish callback, the fixed rotation must be cleared.
+        assertFalse(home.hasFixedRotationTransform());
+        assertFalse(mDefaultDisplay.hasTopFixedRotationLaunchingApp());
     }
 
     @Test