Android Accessibility Services are intended to assist users with disabilities. Using an accessibility service you may for example change the input focus, scroll a list or press a button. Interestingly this can be done not only for your own app, but for any app on the device. Of course your app must request permission for this.

Scenario

This article describes the usage of accessibility services for the following sample use case:

1. Open calculator app
2. Press button “1”
3. Press button “+”
4. Press button “2”
5. Press button “=”
6. Read the result
7. Go back and display result

 

Although this is not really the intended use of an accessibility service, this simple use case should demonstrate the functionality in an easy way.

 

Service configuration

1. Configure accessibility service
   In a separate XML file we need to configure our accessibility service. Here we need to declare that our service wants to retrieve the active window content. The file should be placed in the resource directory of the application (<project_dir>/res/xml/myaccessibilityservice.xml).

   <?xml version="1.0" encoding="utf-8"?>
   <accessibility-service
       xmlns:android="http://schemas.android.com/apk/res/android"
       android:canRetrieveWindowContent="true" />
   

2. Edit AndroidManifest.xml
   Our accessibility service must be declared in the application element of the manifest. In the meta-data tag we reference the previously created accessibility service configuration.

   ...
   <service android:name="MyAccessibilityService" android:permission="android.permission.BIND_ACCESSIBILITY_SERVICE">
       <intent-filter>
           <action android:name="android.accessibilityservice.AccessibilityService" />
       </intent-filter>
       <meta-data android:name="android.accessibilityservice" android:resource="@xml/myaccessibilityservice" />
   </service>
   ...
   

 

Implementation

MyActivity.java

From the main activity (MyActivity.java) we just start the calculator demo (CalculatorDemo.java)..

...
private final CalculatorDemo mCalculatorDemo;

public MyActivity() {
    mCalculatorDemo = new CalculatorDemo(this);
}
...
public void handleStartButtonClick(View view) {
    mCalculatorDemo.run();
}
...

MyAccessibilityService.java

To implement our accessibility service (MyAccessibilityService.java) we need to derive from the abstract class AccessibilityService. For the calculator demo we just override the onServiceConnected method in order to get a reference of the accessibility service. For the sake of simplicity we just inject it directly to our CalculatorDemo class.

...
public class MyAccessibilityService extends AccessibilityService {
    ...
    @Override
    protected void onServiceConnected() {
        super.onServiceConnected();
        CalculatorDemo.initialize(this);
    }
    ...
}

CalculatorDemo.java

Finally the main functionality is implemented in CalculatorDemo.java.

Using the method getRootInActiveWindow of the accessibility service we can access the view hierarchy of the current window. The view hierarchy is represented as a hierarchy of AccessibilityNodeInfo instances. We use the method findAccessibilityNodeInfosByViewId to find the relevant child nodes.

private static AccessibilityNodeInfo findNodeInCurrentWindow(String id) {
    AccessibilityNodeInfo window = mAccessibilityService.getRootInActiveWindow();
    return window.findAccessibilityNodeInfosByViewId(id).get(0);
}

Relevant nodes (and view ID’s) can be found by traversing the node hierarchy. Once we have the relevant nodes, actions can be performed by calling performAction on the specific node.

public void run() {

    // Start calculator app
    mContext.startActivity(new Intent().setClassName(PACKAGE, CLASS));
    sleep(2000);

    // Press button "1"
    findNodeInCurrentWindow(ID_BUTTON_01).performAction(AccessibilityNodeInfo.ACTION_CLICK);
    sleep(500);

    // Press button "+"
    findNodeInCurrentWindow(ID_BUTTON_ADD).performAction(AccessibilityNodeInfo.ACTION_CLICK);
    sleep(500);

    // Press button "2"
    findNodeInCurrentWindow(ID_BUTTON_02).performAction(AccessibilityNodeInfo.ACTION_CLICK);
    sleep(500);

    // Press button "="
    findNodeInCurrentWindow(ID_BUTTON_EQUAL).performAction(AccessibilityNodeInfo.ACTION_CLICK);
    sleep(500);

    // Read result
    String result = findNodeInCurrentWindow(ID_TEXT_DISPLAY).getText().toString();
    Toast.makeText(mContext, String.format("Result is %s", result), Toast.LENGTH_SHORT).show();

    // Go back
    mAccessibilityService.performGlobalAction(AccessibilityService.GLOBAL_ACTION_BACK);
}

Full implementation

The following code shows the full implementation of all relevant parts for this demo.

import android.os.Bundle;
import android.support.v7.app.AppCompatActivity;
public class MyActivity extends AppCompatActivity {

    private final CalculatorDemo mCalculatorDemo;

    public MyActivity() {
        mCalculatorDemo = new CalculatorDemo(this);
    }

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.myactivity);
    }

    public void handleStartButtonClick(View view) {
        mCalculatorDemo.run();
    }
}

import android.accessibilityservice.AccessibilityService;
import android.view.accessibility.AccessibilityEvent;

public class MyAccessibilityService extends AccessibilityService {

    @Override
    public void onAccessibilityEvent(AccessibilityEvent event) {
    }

    @Override
    public void onInterrupt() {
    }

    @Override
    protected void onServiceConnected() {
        super.onServiceConnected();
        CalculatorDemo.initialize(this);
    }
}

import android.accessibilityservice.AccessibilityService;
import android.content.Context;
import android.content.Intent;
import android.support.annotation.NonNull;
import android.view.accessibility.AccessibilityNodeInfo;
import android.widget.Toast;

public class CalculatorDemo {

    private static final String PACKAGE = "com.sec.android.app.popupcalculator";
    private static final String CLASS = "com.sec.android.app.popupcalculator.Calculator";
    private static final String ID_BUTTON_01 = "com.sec.android.app.popupcalculator:id/bt_01";
    private static final String ID_BUTTON_02 = "com.sec.android.app.popupcalculator:id/bt_02";
    private static final String ID_BUTTON_ADD = "com.sec.android.app.popupcalculator:id/bt_add";
    private static final String ID_BUTTON_EQUAL = "com.sec.android.app.popupcalculator:id/bt_equal";
    private static final String ID_TEXT_DISPLAY = "com.sec.android.app.popupcalculator:id/txtCalc";

    private static AccessibilityService mAccessibilityService;

    private final Context mContext;

    public CalculatorDemo(Context context) {
        mContext = context;
    }

    public void run() {
        if (mAccessibilityService == null) {
            Toast.makeText(mContext, "Accessibility service unavailable", Toast.LENGTH_SHORT).show();
            return;
        }

        // Start calculator app
        mContext.startActivity(new Intent().setClassName(PACKAGE, CLASS));
        sleep(2000);

        // Press button "1"
        findNodeInCurrentWindow(ID_BUTTON_01).performAction(AccessibilityNodeInfo.ACTION_CLICK);
        sleep(500);

        // Press button "+"
        findNodeInCurrentWindow(ID_BUTTON_ADD).performAction(AccessibilityNodeInfo.ACTION_CLICK);
        sleep(500);

        // Press button "2"
        findNodeInCurrentWindow(ID_BUTTON_02).performAction(AccessibilityNodeInfo.ACTION_CLICK);
        sleep(500);

        // Press button "="
        findNodeInCurrentWindow(ID_BUTTON_EQUAL).performAction(AccessibilityNodeInfo.ACTION_CLICK);
        sleep(500);

        // Read result
        String result = findNodeInCurrentWindow(ID_TEXT_DISPLAY).getText().toString();
        Toast.makeText(mContext, String.format("Result is %s", result), Toast.LENGTH_SHORT).show();

        // Go back
        mAccessibilityService.performGlobalAction(AccessibilityService.GLOBAL_ACTION_BACK);
    }

    public static void initialize(@NonNull AccessibilityService accessibilityService) {
        if (mAccessibilityService == null) {
            mAccessibilityService = accessibilityService;
        }
    }

    private static AccessibilityNodeInfo findNodeInCurrentWindow(String id) {
        AccessibilityNodeInfo window = mAccessibilityService.getRootInActiveWindow();
        return window.findAccessibilityNodeInfosByViewId(id).get(0);
    }

    private static void sleep(int milliseconds) {
        try {
            Thread.sleep(milliseconds);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
    }
}

 

Enable accessibility service

After installing the CalculatorDemo APK, the service must be enabled in the Android settings:

1. Open Android settings
2. Go to “Accessibility”
3. In the “Services” section tap the service (e.g. “CalculatorDemo”)
4. Enable the service

 

Now the CalculatorDemo APK is ready to be run.

If you’re using JUnit, Mockito and PowerMock to write unit tests in Android Studio you probably often need to mock calls to android.util.Log. That’s because by default in Android Studio unit tests are executed against a modified version of the android.jar library which contains stubs only. If anywhere in your tested code you have a log statement that you don’t mock you will get a RuntimeException.

For example a call like

Log.d("tag", "message");

will throw a RuntimeException:

RuntimeException: Method d in android.util.Log not mocked. See http://g.co/androidstudio/not-mocked for details.

Wouldn’t it be nice if just all the log message would be redirected automatically to the unit test output?

Fortunately PowerMockito provides the possibility to create MockPolicies. Using a mock policy we can setup the log redirection once and then just use an annotation (on the class containing the test cases) to apply it.

Usage example:

@RunWith(PowerMockRunner.class)
@MockPolicy(LogRedirection.class)
public class MyTests {

    @Test
    public void test1() {
    }
}

To create a log redirection mock policy we need to implement the PowerMockPolicy interface. The two methods on this interface allow us to apply a class loading policy and an interception policy.

public interface PowerMockPolicy {

	void applyClassLoadingPolicy(MockPolicyClassLoadingSettings settings);

	void applyInterceptionPolicy(MockPolicyInterceptionSettings settings);
    
}

In the first method we need to tell PowerMock which classes should be modified by the mock class loader before these classes are loaded. In our case it’s just the android.util.Log class. In the second method we create the mocks for the methods of the Log class which should just redirect the messages to the standard output.

The following code shows a sample implementation for a mock policy that redirects all log calls to the standard output.

import android.util.Log;
import org.powermock.core.spi.PowerMockPolicy;
import org.powermock.mockpolicies.MockPolicyClassLoadingSettings;
import org.powermock.mockpolicies.MockPolicyInterceptionSettings;

import java.io.PrintWriter;
import java.io.StringWriter;
import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;

public class LogRedirection implements PowerMockPolicy {

    @Override
    public void applyClassLoadingPolicy(MockPolicyClassLoadingSettings settings) {
        settings.addFullyQualifiedNamesOfClassesToLoadByMockClassloader(Log.class.getName());
    }

    @Override
    public void applyInterceptionPolicy(MockPolicyInterceptionSettings settings) {
        try {

            // Mock Log.v(String tag, String msg)
            settings.proxyMethod(Log.class.getMethod("v", String.class, String.class), new InvocationHandler() {
                @Override
                public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                    String logLevel = method.getName().toUpperCase();
                    String tag = args[0].toString();
                    String message = args[1].toString();
                    return redirect(logLevel, tag, message, null);
                }
            });

            // Mock Log.v(String tag, String msg, Throwable tr)
            settings.proxyMethod(Log.class.getMethod("v", String.class, String.class, Throwable.class), new InvocationHandler() {
                @Override
                public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                    String logLevel = method.getName().toUpperCase();
                    String tag = args[0].toString();
                    String message = args[1].toString();
                    Throwable throwable = (Throwable) args[2];
                    return redirect(logLevel, tag, message, throwable);
                }
            });

            // Mock Log.d(String tag, String msg)
            settings.proxyMethod(Log.class.getMethod("d", String.class, String.class), new InvocationHandler() {
                @Override
                public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                    String logLevel = method.getName().toUpperCase();
                    String tag = args[0].toString();
                    String message = args[1].toString();
                    return redirect(logLevel, tag, message, null);
                }
            });

            // Mock Log.d(String tag, String msg, Throwable tr)
            settings.proxyMethod(Log.class.getMethod("d", String.class, String.class, Throwable.class), new InvocationHandler() {
                @Override
                public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                    String logLevel = method.getName().toUpperCase();
                    String tag = args[0].toString();
                    String message = args[1].toString();
                    Throwable throwable = (Throwable) args[2];
                    return redirect(logLevel, tag, message, throwable);
                }
            });

            // Mock Log.i(String tag, String msg)
            settings.proxyMethod(Log.class.getMethod("i", String.class, String.class), new InvocationHandler() {
                @Override
                public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                    String logLevel = method.getName().toUpperCase();
                    String tag = args[0].toString();
                    String message = args[1].toString();
                    return redirect(logLevel, tag, message, null);
                }
            });

            // Mock Log.i(String tag, String msg, Throwable tr)
            settings.proxyMethod(Log.class.getMethod("i", String.class, String.class, Throwable.class), new InvocationHandler() {
                @Override
                public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                    String logLevel = method.getName().toUpperCase();
                    String tag = args[0].toString();
                    String message = args[1].toString();
                    Throwable throwable = (Throwable) args[2];
                    return redirect(logLevel, tag, message, throwable);
                }
            });

            // Mock Log.w(String tag, String msg)
            settings.proxyMethod(Log.class.getMethod("w", String.class, String.class), new InvocationHandler() {
                @Override
                public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                    String logLevel = method.getName().toUpperCase();
                    String tag = args[0].toString();
                    String message = args[1].toString();
                    return redirect(logLevel, tag, message, null);
                }
            });

            // Mock Log.w(String tag, String msg, Throwable tr)
            settings.proxyMethod(Log.class.getMethod("w", String.class, String.class, Throwable.class), new InvocationHandler() {
                @Override
                public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                    String logLevel = method.getName().toUpperCase();
                    String tag = args[0].toString();
                    String message = args[1].toString();
                    Throwable throwable = (Throwable) args[2];
                    return redirect(logLevel, tag, message, throwable);
                }
            });

            // Mock Log.w(String tag, Throwable tr)
            settings.proxyMethod(Log.class.getMethod("w", String.class, Throwable.class), new InvocationHandler() {
                @Override
                public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                    String logLevel = method.getName().toUpperCase();
                    String tag = args[0].toString();
                    String message = getStackTraceString((Throwable) args[1]);
                    return redirect(logLevel, tag, message, null);
                }
            });

            // Mock Log.e(String tag, String msg)
            settings.proxyMethod(Log.class.getMethod("e", String.class, String.class), new InvocationHandler() {
                @Override
                public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                    String logLevel = method.getName().toUpperCase();
                    String tag = args[0].toString();
                    String message = args[1].toString();
                    return redirect(logLevel, tag, message, null);
                }
            });

            // Mock Log.e(String tag, String msg, Throwable tr)
            settings.proxyMethod(Log.class.getMethod("e", String.class, String.class, Throwable.class), new InvocationHandler() {
                @Override
                public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
                    String logLevel = method.getName().toUpperCase();
                    String tag = args[0].toString();
                    String message = args[1].toString();
                    Throwable throwable = (Throwable) args[2];
                    return redirect(logLevel, tag, message, throwable);
                }
            });

        } catch (NoSuchMethodException e) {
            e.printStackTrace();
        }
    }

    private static int redirect(String logLevel, String tag, String message, Throwable throwable) {
        if (throwable == null) {
            System.out.println(String.format("%s - %s: %s", logLevel, tag, message));
        } else {
            System.out.println(String.format("%s - %s: %s", logLevel, tag, message + '\n' + getStackTraceString(throwable)));
        }
        return 0;
    }

    private static String getStackTraceString(Throwable tr) {
        StringWriter sw = new StringWriter();
        PrintWriter pw = new PrintWriter(sw);
        tr.printStackTrace(pw);
        pw.flush();
        return sw.toString();
    }
}