Android Question Unable to Perform Click on Buttons (e.g., Send, Follow) Using Accessibility Service in Third-Party Apps like WhatsApp/Instagram

[ykucuk]

Hi everyone,

I'm currently developing an in-house application that leverages Android's accessibility service to automate interactions with third-party apps, specifically for internal use only (this app will not be published on Google Play Store).

I am able to use AccessibilityService to interact with elements like text entry fields in apps such as WhatsApp, and even global actions like AccServConstants.GLOBAL_ACTION_BACK work correctly. However, I am facing an issue with clicking specific buttons like "Send" in WhatsApp or "Follow" in Facebook.

Although I have tried using the correct view IDs (e.g., com.whatsapp:id/send), the click event does not seem to trigger. I've tried several approaches, including PerformNodeActionOnViewWithArgs, but none seem to work on these third-party apps.

I came across information suggesting that UIAutomator might help retrieve the correct resource IDs of these buttons, but I've been struggling to get UIAutomator running properly, even after trying different Java versions.

Is there a reliable way to programmatically click buttons like "Send" in WhatsApp or "Follow" in Facebook using the accessibility service? Also, any guidance on getting UIAutomator to work properly to inspect these third-party apps would be much appreciated.

I'm willing to provide compensation to anyone who can help me solve this issue.

---

Example Code:​

Below is the relevant part of my code:

#Region  Service Attributes 
 #StartAtBoot: False
#End Region
#Extends: com.tillekesoft.accessibilityservices.AccessibilityEventsListenerWrapper 
Sub Process_Globals
    'These global variables will be declared once when the application starts.
    'These variables can be accessed from all modules.

    Public acs As AccessibilityEventsListener
    Private AccServConstants As AccessibilityConstants
    Private ShallWeSendWhatsApp As Boolean
    Private ShallWeSendWhatsAppGroup As Boolean
    Private success As Boolean = False
    Private lastbackout As Boolean = False
    Private strMessage As String
End Sub
Sub Service_Create
    Log("service was created")
    acs.initialize("acs")
End Sub
Sub Service_Start (StartingIntent As Intent)
    Log("service was started")
    Service.StopAutomaticForeground 'Call this when the background task completes (if there is one)
End Sub
Sub Service_Destroy
End Sub
Sub CheckIfServiceIsEnabled
    If acs.IsAccessibilityServiceEnabled("com.mytest/.accservice")  = False Then 'insert package-name and name of your service
        CallSub2(Main,"IsTheAccessibilityServiceEnabled", False)
        CallSub2(Main,"SetCheckbox", False)
    Else
        CallSub2(Main,"IsTheAccessibilityServiceEnabled", True)
        CallSub2(Main,"SetCheckbox", True)
    End If
End Sub
Sub acs_OnAccessibilityEvent (Event As Object, node As Object) 'ignore
    'just some logging.....
    Log("my event: " & Event)
    Log("my node: " & node)
    Dim mylist As List
    mylist.Initialize2(node)
    Log(mylist.Size)
    If ShallWeSendWhatsApp = True Then
        For Each item As String In mylist
            If success = False Then
                success = acs.PerformNodeActionOnViewWithArgs(True, "com.whatsapp:id/send",AccServConstants.ACTION_CLICK, Null)
            Else if success = True And lastbackout = False Then
                acs.delay(500)
                acs.PerformGlobalAction(AccServConstants.GLOBAL_ACTION_BACK)
                acs.delay(500)
                lastbackout = True
            else if success = True And lastbackout = True Then
                acs.PerformGlobalAction(AccServConstants.GLOBAL_ACTION_BACK)
                ShallWeSendWhatsApp = False
                success = False
                lastbackout = False
            End If
        Next
    End If
    If ShallWeSendWhatsAppGroup = True Then
        For Each item As String In mylist
            If success = False Then
                Dim MsgMap As Map
                MsgMap.Initialize
                MsgMap.Put("msg",strMessage)
                success = acs.PerformNodeActionOnViewWithArgs(True,"com.whatsapp:id/entry",AccServConstants.ACTION_SET_TEXT,MsgMap)
                acs.delay(100)
                success = acs.PerformNodeActionOnViewWithArgs(True, "com.whatsapp:id/send",AccServConstants.ACTION_CLICK, Null)
            else if success = True And lastbackout = False Then
                acs.delay(500)
                acs.PerformGlobalAction(AccServConstants.GLOBAL_ACTION_BACK)
                acs.delay(500)
                lastbackout = True
            else if success = True And lastbackout = True Then
                acs.PerformGlobalAction(AccServConstants.GLOBAL_ACTION_BACK)
                ShallWeSendWhatsAppGroup = False
                success = False
                lastbackout = False
                strMessage = ""
            End If
        Next
    End If
End Sub
Sub SendWhatsAppMsg
    ShallWeSendWhatsApp = True
End Sub
Sub SendWhatsAppMsgToGroup (text As String)
    ShallWeSendWhatsAppGroup = True
    strMessage = text.Trim
End Sub

The full code and more information can be found in the link below, where I initially based my approach: Accessibility Services in B4X Forum

Thanks in advance for your help! Again, I'm happy to offer payment for anyone who can provide a working solution or assistance.