B4A Library Accessibility Services (assisting users with disabilities, automation etc)

This is a partial wrap/library for Android's Accessibility Services which you can read about here:
https://developer.android.com/reference/android/accessibilityservice/AccessibilityService

Android's Accessibility Services is meant to assisting users with disabilities in using Android devices and apps but can also be used for automation.
Some automation tools, such as Tasker, are using Accessibility Services. In this post, I am attaching a sample app which permits you to automate WhatsApp so messages can be sent to contacts and groups without any user-interaction.

It should be noted that an app such as the attached WhatsApp-sample might not be permitted on Google Play since it could be used as a bot unless you are able to motivate that it is purely used for users with disabilities.

I started to write this wrapper for a personal app I am using when riding my motorbike. I have seen many requests for a wrap of the Accessibility Services and initially, I meant to write a complete and generic wrapper but I have realized that it is a huge task and I simply don't have enough time to do so. It is much easier to write a customized wrapper which covers your requirements. If you would like a customized library and you want me to help you, you can contact me by PM.

That said, the attached library contains the most important functionality and should be usable for many projects you might have in mind. Open the sample-app to better understand how it works. You also need to add some things in the Manifest and there are also some configuration-files which need to be set up properly. You can find these files in the Resource-library which is attached.

Read also the docs. Unfortunately, there isn't much documentation to be found on the internet except for the official documentation provided by Google (see the link earlier in this post) although you can find some information/examples on StackOverflow.

22/02/2020 - updated library to version 0.11 (bug fix)
22/02/2020 - updated sample app to use targetSDK = 28


Here is a YouTube video to see the WhatsApp sample app in action:


Please remember that creating libraries and maintaining them takes time and so does supporting them. Please consider a donation if you use my free libraries as this will surely help keeping me motivated. Thank you!

Good luck!

 

[Star-Dust]

I do not know if I understand correctly, with this app can I make sure that the messages that you receive on WhatsApp and that I read with ListenNotifier, Can send to on a another number ?

Or anyway can I send messages from my app through whatsapp?


Congratulations.

 

[JordiCP]

Not an easy one, congratulations!

I started with it and took another approach since I thought it was going to be a big monster (not moster, sorry for the bad joke )
Will dive into the source when I can, to learn from it

 

[scsjc]

Awesome

 

[Star-Dust]

This is a partial wrap/library for Android's Accessibility Services which you can read about here:
https://developer.android.com/reference/android/accessibilityservice/AccessibilityService

Android's Accessibility Services is meant to assisting users with disabilities in using Android devices and apps but can also be used for automation.
Some automation tools, such as Tasker, are using Accessibility Services. In this post, I am attaching a sample app which permits you to automate WhatsApp so messages can be sent to contacts and groups without any user-interaction.

It should be noted that an app such as the attached WhatsApp-sample might not be permitted on Google Play since it could be used as a bot unless you are able to motivate that it is purely used for users with disabilities.

I started to write this wrapper for a personal app I am using when riding my motorbike. I have seen many requests for a wrap of the Accessibility Services and initially, I meant to write a complete wrapper but I have realized that it is a huge task and I simply don't have enough time to do so. It is much easier to write a customized wrapper which covers your requirements. I am, therefore, also attaching the java source-code for the wrapper which you can extend and/or customize according to your needs. If you would like a customized library and you want me to help you, you can contact me by PM.

That said, the attached library contains the most important functionality and should be usable for many projects you might have in mind. Open the sample-app to understand how it works. You also need to add some things in the Manifest and there are also some configuration-files which need to be set up properly. You can find these files in the Resource-library which is attached.

There isn't much documentation to be found on the internet except for the official documentation provided by Google (see the link earlier in this post) although you can find some information/examples on StackOverflow.

Here is a YouTube video to see the WhatsApp sample app in action (at the time of writing, the video is still being elaborated by YouTube):


Please remember that creating libraries and maintaining them takes time and so does supporting them. Please consider a donation if you use my free libraries as this will surely help keeping me motivated. Thank you!

Good luck!

On the Samsung S6 and Honor 9 they work properly.
On Huawei mediapd M3 does not send the message but simply open whatsApp and write the message in the contact but does not send. What the Web API already does

 

[moster67]

I started with it and took another approach

You can probably approach it in different ways. My intention was to make a generic and complete wrapper but I simply don't have the time.

 

[moster67]

On Huawei mediapd M3 does not send the message but simply open whatsApp and write the message in the contact but does not send.

The purpose of the library is not exclusively to WhatsApp messages - I just thought it was a good sample app to include.
Anyway, it could be that the mediapad is slower (or WhatsApp was executing slower) and when the send-button was going to be "clicked", the screen content had not yet been loaded and the button was not available. In the sample app, there are some delay methods - try to increase the one set at 100 ms to 500ms and see if it works.
I tried the sample app on numerous devices, also slow ones, and it worked just fine.

 

[Star-Dust]

The purpose of the library is not exclusively to WhatsApp messages - I just thought it was a good sample app to include.
Anyway, it could be that the mediapad is slower (or WhatsApp was executing slower) and when the send-button was going to be "clicked", the screen content had not yet been loaded and the button was not available. In the sample app, there are some delay methods - try to increase the one set at 100 ms to 500ms and see if it works.
I tried the sample app on numerous devices, also slow ones, and it worked just fine.

I know that the library has a wider use.
Reading the difficulties of DonManfred I wanted to try the example. you attached WhatsApp example and I try it.
I have not thought of creating another example myself, also because I do not know the library well enough to be able to develop my example.

Huawey MediaPad M3 is not a slow device, it is a fairly powerful tablet. I suppose the problem is something else, but it was just a curiosity.

 

[Lello1964]

wOOOOOOwwwwww

 

[MarcoRome]

Well done Mike

 

[Inman]

Thank you for the library. This was one of my requests in the Wish section.

I was trying your example and once I enable the accessibility permission, I get the following error. I did put the additionalres folder correctly.

*** Service (accservice) Create ***

service was created
** Service (accservice) Start **
service was started
*** Service (starter) Create ***
** Service (starter) Start **
** Activity (main) Create, isFirst = true **
** Activity (main) Resume **
** Service (accservice) Start **
service was started
** Activity (main) Resume **
java.lang.NullPointerException: Attempt to invoke interface method 'java.lang.String java.lang.CharSequence.toString()' on a null object reference
    at com.tillekesoft.accessibilityservices.AccessibilityEventsListenerWrapper.onAccessibilityEvent(AccessibilityEventsListenerWrapper.java:161)
    at android.accessibilityservice.AccessibilityService$2.onAccessibilityEvent(AccessibilityService.java:1524)
    at android.accessibilityservice.AccessibilityService$IAccessibilityServiceClientWrapper.executeMessage(AccessibilityService.java:1710)
    at com.android.internal.os.HandlerCaller$MyHandler.handleMessage(HandlerCaller.java:37)
    at android.os.Handler.dispatchMessage(Handler.java:109)
    at android.os.Looper.loop(Looper.java:166)
    at android.app.ActivityThread.main(ActivityThread.java:7377)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:469)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:963)

There is no B4A line number mentioned as it looks like the error is raised from library. When I checked the library source code, the line in bold is the one raising error (line 161).

public void onAccessibilityEvent(AccessibilityEvent event) {
if(acs != null){
MyViewNodes = new ArrayList<>();
infoNodes = new ArrayList<>();
// AccessibilityNodeInfo node = event.getSource();
AccessibilityNodeInfo node = getRootInActiveWindow();
findChildViews(node);
infoNodes.add(MyViewNodes);
acs.ba.raiseEvent(this,acs.eventName + "_onaccessibilityevent",new Object[]{event, infoNodes});

ComponentName componentName = new ComponentName(event.getPackageName().toString(), event.getClassName().toString());
ActivityInfo activityInfo = getActivityInfo(componentName);
boolean isActivity = activityInfo != null;

if (isActivity) {
String activityName = componentName.flattenToShortString();
acs.ba.raiseEvent(this, acs.eventName + "_onactivitynameretrieved", new Object[]{activityName});
return;
}

}
}

I could see 3-4 log entries for "my event" and "my node". I then commented out the entire code inside acs_OnAccessibilityEvent but is still getting the above error.

I tried on Galaxy S8+ running Android 8.0 and Huawei P20 Pro running Android 8.1 and got the same error. Any idea?

 

[moster67]

It seems like it is the same error @DonManfred mentioned in this thread/post:
https://www.b4x.com/android/forum/threads/accessibility-services-lib-running-the-sample.95221/

As I replied in this thread, I am unsure why this is happening since I tested it on numerous devices and was not able to reproduce it.
Try to uninstall the app and then run it again.

I'll now be away from the forum for a while but when I'll be back maybe I will look into it (since there are some methods to modify a bit to make them more generic). Otherwise, the java sources are there if anyone else would like to have a look.

 

[Inman]

Thank you replying. I somehow missed Don's thread. We will continue the discussion there.

Uninstall and reinstall did not seem to work in my case. Please check Don's thread for more details.

 

[Scotter]

This is sooo useful!
I'm trying to incorporate it into my simple card game B4A app.
Copied all the libraries, etc.
Now I see the AccService.bas has many references to WhatsApp.
Of course, I want to remove or change them.
But I'm not sure which ones are safe to remove and what to change.

Also, in that same AccService.bas module I see this:

Sub CheckIfServiceIsEnabled
    If acs.IsAccessibilityServiceEnabled("com.mytest/.accservice")  = False Then 'insert package-name and name of your service
        CallSub2(Main,"IsTheAccessibilityServiceEnabled", False)
        CallSub2(Main,"SetCheckbox", False)
    Else
        CallSub2(Main,"IsTheAccessibilityServiceEnabled", True)
        CallSub2(Main,"SetCheckbox", True)
    End If
End Sub

I'm not sure what to replace "com.mytest/.accservice" with since I don't see where "com.mytest" is in your sample project so I would figure out - based on the ?name? of my project, how to replace this string.

Any tips?

Thanks!

 

[moster67]

I don't see where "com.mytest" is in your sample project

You can see it in the Build Configuration -> Package Name (Ctrl+B). Study the WhatsApp sample app, look at the Manifest-file and also in the Additional Resources and you should be able to see how it all fits together.
After getting the user to activate the Service, most of your coding will take place in the OnAccessibilityEvent (see AccService module).

 

[Scotter]

Thanks!
And... just checking... I noticed there already is an accessibility library built into B4A.
For a simple card game where my goal is to add the ability for screenreaders to have efficient input (like speaking "Shuffle" instead of "Button Shuffle", is your component overkill because I don't need any kind of automation?

 

[moster67]

I noticed there already is an accessibility library built into B4A

That is news for me. I must have missed that.....that is good to know.

is your component overkill because I don't need any kind of automation?

Hard to tell. Automation is not the main-reason for writing this lib - it is just some extra functionality you get for free.
If you think ContentDescription is enough, then I guess you are fine. Check out the Screener app I mentioned in the other thread and see if you are fine. Check if TalkBack works fine and decide. Do you need to add gestures or other features to help out people with disabilities? Let a blind user try your app and let you know if you can make any improvements...it is really up to you.

 

[Scotter]

I wonder what Erel's take on this is? How does one tag a person in this forum?

 

[DonManfred]

How does one tag a person in this forum?

Begin the name qith @ followed by the username
@Scotter wrote:......

 

[Scotter]

@Erel - May I hear your thoughts on the "accessibility" library built into B4A?
Mainly I'm wondering if it includes enough functionality to do the basics I need for a simple card game.
Like what I'm imagining is the blind person taps on an object and hears what I want them to hear for that object because I added a bit of code to every object that has some text that will get auto-converted for the user into speech.
Does the default accessibility library enable that? Are there examples I can see specifically for B4A?
Thanks! Sorry if I'm asking dumb questions!