Foreign app navigation with Android Accessibility Services
Android Accessibility Services are intended to assist users with disabilities. Using an accessibility service you may for example change the input focus, scroll a list or press a button. Interestingly this can be done not only for your own app, but for any app on the device. Of course your app must request permission for this.
Scenario
This article describes the usage of accessibility services for the following sample use case:
- Open calculator app
- Press button “1”
- Press button “+”
- Press button “2”
- Press button “=”
- Read the result
- Go back and display result
Although this is not really the intended use of an accessibility service, this simple use case should demonstrate the functionality in an easy way.
Service configuration
- Configure accessibility service
In a separate XML file we need to configure our accessibility service. Here we need to declare that our service wants to retrieve the active window content. The file should be placed in the resource directory of the application (<project_dir>/res/xml/myaccessibilityservice.xml
).<?xml version="1.0" encoding="utf-8"?> <accessibility-service xmlns:android="http://schemas.android.com/apk/res/android" android:canRetrieveWindowContent="true" />
- Edit AndroidManifest.xml
Our accessibility service must be declared in the application element of the manifest. In the meta-data tag we reference the previously created accessibility service configuration.... <service android:name="MyAccessibilityService" android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE"> <intent-filter> <action android:name="android.accessibilityservice.AccessibilityService" /> </intent-filter> <meta-data android:name="android.accessibilityservice" android:resource="@xml/myaccessibilityservice" /> </service> ...
Implementation
MyActivity.java
From the main activity (MyActivity.java) we just start the calculator demo (CalculatorDemo.java)..
... private final CalculatorDemo mCalculatorDemo; public MyActivity() { mCalculatorDemo = new CalculatorDemo(this); } ... public void handleStartButtonClick(View view) { mCalculatorDemo.run(); } ...
MyAccessibilityService.java
To implement our accessibility service (MyAccessibilityService.java) we need to derive from the abstract class AccessibilityService. For the calculator demo we just override the onServiceConnected method in order to get a reference of the accessibility service. For the sake of simplicity we just inject it directly to our CalculatorDemo class.
... public class MyAccessibilityService extends AccessibilityService { ... @Override protected void onServiceConnected() { super.onServiceConnected(); CalculatorDemo.initialize(this); } ... }
CalculatorDemo.java
Finally the main functionality is implemented in CalculatorDemo.java.
Using the method getRootInActiveWindow of the accessibility service we can access the view hierarchy of the current window. The view hierarchy is represented as a hierarchy of AccessibilityNodeInfo instances. We use the method findAccessibilityNodeInfosByViewId to find the relevant child nodes.
private static AccessibilityNodeInfo findNodeInCurrentWindow(String id) { AccessibilityNodeInfo window = mAccessibilityService.getRootInActiveWindow(); return window.findAccessibilityNodeInfosByViewId(id).get(0); }
Relevant nodes (and view ID’s) can be found by traversing the node hierarchy. Once we have the relevant nodes, actions can be performed by calling performAction on the specific node.
public void run() { // Start calculator app mContext.startActivity(new Intent().setClassName(PACKAGE, CLASS)); sleep(2000); // Press button "1" findNodeInCurrentWindow(ID_BUTTON_01).performAction(AccessibilityNodeInfo.ACTION_CLICK); sleep(500); // Press button "+" findNodeInCurrentWindow(ID_BUTTON_ADD).performAction(AccessibilityNodeInfo.ACTION_CLICK); sleep(500); // Press button "2" findNodeInCurrentWindow(ID_BUTTON_02).performAction(AccessibilityNodeInfo.ACTION_CLICK); sleep(500); // Press button "=" findNodeInCurrentWindow(ID_BUTTON_EQUAL).performAction(AccessibilityNodeInfo.ACTION_CLICK); sleep(500); // Read result String result = findNodeInCurrentWindow(ID_TEXT_DISPLAY).getText().toString(); Toast.makeText(mContext, String.format("Result is %s", result), Toast.LENGTH_SHORT).show(); // Go back mAccessibilityService.performGlobalAction(AccessibilityService.GLOBAL_ACTION_BACK); }
Full implementation
The following code shows the full implementation of all relevant parts for this demo.
import android.os.Bundle; import android.support.v7.app.AppCompatActivity; public class MyActivity extends AppCompatActivity { private final CalculatorDemo mCalculatorDemo; public MyActivity() { mCalculatorDemo = new CalculatorDemo(this); } @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); setContentView(R.layout.myactivity); } public void handleStartButtonClick(View view) { mCalculatorDemo.run(); } }
import android.accessibilityservice.AccessibilityService; import android.view.accessibility.AccessibilityEvent; public class MyAccessibilityService extends AccessibilityService { @Override public void onAccessibilityEvent(AccessibilityEvent event) { } @Override public void onInterrupt() { } @Override protected void onServiceConnected() { super.onServiceConnected(); CalculatorDemo.initialize(this); } }
import android.accessibilityservice.AccessibilityService; import android.content.Context; import android.content.Intent; import android.support.annotation.NonNull; import android.view.accessibility.AccessibilityNodeInfo; import android.widget.Toast; public class CalculatorDemo { private static final String PACKAGE = "com.sec.android.app.popupcalculator"; private static final String CLASS = "com.sec.android.app.popupcalculator.Calculator"; private static final String ID_BUTTON_01 = "com.sec.android.app.popupcalculator:id/bt_01"; private static final String ID_BUTTON_02 = "com.sec.android.app.popupcalculator:id/bt_02"; private static final String ID_BUTTON_ADD = "com.sec.android.app.popupcalculator:id/bt_add"; private static final String ID_BUTTON_EQUAL = "com.sec.android.app.popupcalculator:id/bt_equal"; private static final String ID_TEXT_DISPLAY = "com.sec.android.app.popupcalculator:id/txtCalc"; private static AccessibilityService mAccessibilityService; private final Context mContext; public CalculatorDemo(Context context) { mContext = context; } public void run() { if (mAccessibilityService == null) { Toast.makeText(mContext, "Accessibility service unavailable", Toast.LENGTH_SHORT).show(); return; } // Start calculator app mContext.startActivity(new Intent().setClassName(PACKAGE, CLASS)); sleep(2000); // Press button "1" findNodeInCurrentWindow(ID_BUTTON_01).performAction(AccessibilityNodeInfo.ACTION_CLICK); sleep(500); // Press button "+" findNodeInCurrentWindow(ID_BUTTON_ADD).performAction(AccessibilityNodeInfo.ACTION_CLICK); sleep(500); // Press button "2" findNodeInCurrentWindow(ID_BUTTON_02).performAction(AccessibilityNodeInfo.ACTION_CLICK); sleep(500); // Press button "=" findNodeInCurrentWindow(ID_BUTTON_EQUAL).performAction(AccessibilityNodeInfo.ACTION_CLICK); sleep(500); // Read result String result = findNodeInCurrentWindow(ID_TEXT_DISPLAY).getText().toString(); Toast.makeText(mContext, String.format("Result is %s", result), Toast.LENGTH_SHORT).show(); // Go back mAccessibilityService.performGlobalAction(AccessibilityService.GLOBAL_ACTION_BACK); } public static void initialize(@NonNull AccessibilityService accessibilityService) { if (mAccessibilityService == null) { mAccessibilityService = accessibilityService; } } private static AccessibilityNodeInfo findNodeInCurrentWindow(String id) { AccessibilityNodeInfo window = mAccessibilityService.getRootInActiveWindow(); return window.findAccessibilityNodeInfosByViewId(id).get(0); } private static void sleep(int milliseconds) { try { Thread.sleep(milliseconds); } catch (InterruptedException e) { e.printStackTrace(); } } }
Enable accessibility service
After installing the CalculatorDemo APK, the service must be enabled in the Android settings:
- Open Android settings
- Go to “Accessibility”
- In the “Services” section tap the service (e.g. “CalculatorDemo”)
- Enable the service
Now the CalculatorDemo APK is ready to be run.