Hi everyone,
I'm currently developing an in-house application that leverages Android's accessibility service to automate interactions with third-party apps, specifically for internal use only (this app will not be published on Google Play Store).
I am able to use AccessibilityService to interact with elements like text entry fields in apps such as WhatsApp, and even global actions like AccServConstants.GLOBAL_ACTION_BACK work correctly. However, I am facing an issue with clicking specific buttons like "Send" in WhatsApp or "Follow" in Facebook.
Although I have tried using the correct view IDs (e.g., com.whatsapp:id/send), the click event does not seem to trigger. I've tried several approaches, including PerformNodeActionOnViewWithArgs, but none seem to work on these third-party apps.
I came across information suggesting that UIAutomator might help retrieve the correct resource IDs of these buttons, but I've been struggling to get UIAutomator running properly, even after trying different Java versions.
Is there a reliable way to programmatically click buttons like "Send" in WhatsApp or "Follow" in Facebook using the accessibility service? Also, any guidance on getting UIAutomator to work properly to inspect these third-party apps would be much appreciated.
I'm willing to provide compensation to anyone who can help me solve this issue.
The full code and more information can be found in the link below, where I initially based my approach: Accessibility Services in B4X Forum
Thanks in advance for your help! Again, I'm happy to offer payment for anyone who can provide a working solution or assistance.
I'm currently developing an in-house application that leverages Android's accessibility service to automate interactions with third-party apps, specifically for internal use only (this app will not be published on Google Play Store).
I am able to use AccessibilityService to interact with elements like text entry fields in apps such as WhatsApp, and even global actions like AccServConstants.GLOBAL_ACTION_BACK work correctly. However, I am facing an issue with clicking specific buttons like "Send" in WhatsApp or "Follow" in Facebook.
Although I have tried using the correct view IDs (e.g., com.whatsapp:id/send), the click event does not seem to trigger. I've tried several approaches, including PerformNodeActionOnViewWithArgs, but none seem to work on these third-party apps.
I came across information suggesting that UIAutomator might help retrieve the correct resource IDs of these buttons, but I've been struggling to get UIAutomator running properly, even after trying different Java versions.
Is there a reliable way to programmatically click buttons like "Send" in WhatsApp or "Follow" in Facebook using the accessibility service? Also, any guidance on getting UIAutomator to work properly to inspect these third-party apps would be much appreciated.
I'm willing to provide compensation to anyone who can help me solve this issue.
Example Code:
Below is the relevant part of my code:
B4X:
#Region Service Attributes
#StartAtBoot: False
#End Region
#Extends: com.tillekesoft.accessibilityservices.AccessibilityEventsListenerWrapper
Sub Process_Globals
'These global variables will be declared once when the application starts.
'These variables can be accessed from all modules.
Public acs As AccessibilityEventsListener
Private AccServConstants As AccessibilityConstants
Private ShallWeSendWhatsApp As Boolean
Private ShallWeSendWhatsAppGroup As Boolean
Private success As Boolean = False
Private lastbackout As Boolean = False
Private strMessage As String
End Sub
Sub Service_Create
Log("service was created")
acs.initialize("acs")
End Sub
Sub Service_Start (StartingIntent As Intent)
Log("service was started")
Service.StopAutomaticForeground 'Call this when the background task completes (if there is one)
End Sub
Sub Service_Destroy
End Sub
Sub CheckIfServiceIsEnabled
If acs.IsAccessibilityServiceEnabled("com.mytest/.accservice") = False Then 'insert package-name and name of your service
CallSub2(Main,"IsTheAccessibilityServiceEnabled", False)
CallSub2(Main,"SetCheckbox", False)
Else
CallSub2(Main,"IsTheAccessibilityServiceEnabled", True)
CallSub2(Main,"SetCheckbox", True)
End If
End Sub
Sub acs_OnAccessibilityEvent (Event As Object, node As Object) 'ignore
'just some logging.....
Log("my event: " & Event)
Log("my node: " & node)
Dim mylist As List
mylist.Initialize2(node)
Log(mylist.Size)
If ShallWeSendWhatsApp = True Then
For Each item As String In mylist
If success = False Then
success = acs.PerformNodeActionOnViewWithArgs(True, "com.whatsapp:id/send",AccServConstants.ACTION_CLICK, Null)
Else if success = True And lastbackout = False Then
acs.delay(500)
acs.PerformGlobalAction(AccServConstants.GLOBAL_ACTION_BACK)
acs.delay(500)
lastbackout = True
else if success = True And lastbackout = True Then
acs.PerformGlobalAction(AccServConstants.GLOBAL_ACTION_BACK)
ShallWeSendWhatsApp = False
success = False
lastbackout = False
End If
Next
End If
If ShallWeSendWhatsAppGroup = True Then
For Each item As String In mylist
If success = False Then
Dim MsgMap As Map
MsgMap.Initialize
MsgMap.Put("msg",strMessage)
success = acs.PerformNodeActionOnViewWithArgs(True,"com.whatsapp:id/entry",AccServConstants.ACTION_SET_TEXT,MsgMap)
acs.delay(100)
success = acs.PerformNodeActionOnViewWithArgs(True, "com.whatsapp:id/send",AccServConstants.ACTION_CLICK, Null)
else if success = True And lastbackout = False Then
acs.delay(500)
acs.PerformGlobalAction(AccServConstants.GLOBAL_ACTION_BACK)
acs.delay(500)
lastbackout = True
else if success = True And lastbackout = True Then
acs.PerformGlobalAction(AccServConstants.GLOBAL_ACTION_BACK)
ShallWeSendWhatsAppGroup = False
success = False
lastbackout = False
strMessage = ""
End If
Next
End If
End Sub
Sub SendWhatsAppMsg
ShallWeSendWhatsApp = True
End Sub
Sub SendWhatsAppMsgToGroup (text As String)
ShallWeSendWhatsAppGroup = True
strMessage = text.Trim
End SubThe full code and more information can be found in the link below, where I initially based my approach: Accessibility Services in B4X Forum
Thanks in advance for your help! Again, I'm happy to offer payment for anyone who can provide a working solution or assistance.