B4J Library jAWTRobot - invoke keyboard and mouse events, etc...

This library started out as a simple wrapper for Oracle's java.awt.Robot package but it evolved out of hand into a library of general utilities. Some of these functions appear to duplicate functionality already present in B4J however this library is designed to be used in both UI apps and non-UI apps. Consequently, B4J functionality that requires the use of a JFX object (fx.showExternalDocument(), for example) is duplicated here but without the use of any objects only found in UI apps.

Things you can do with this library:

• programmatically move the mouse and invoke system MouseButtonPressed events or KeyPressed events and much more
• access the system clipboard
  
• get a variety of general operating system, hardware and JVM info
  
• get a variety of JVM, process and system performance info (memory usage, CPU load, etc...)
  
• take screen shots of arbitrary rectangles on the system's screens
  
• run arbitrary commands on the command line
  
• redirect Standard Error and Standard Out to file
  
• open external files or URLs
  
• improved Exception-related stuff
• improved Thread-related stuff
• restart your app

The library is very heavily commented so you need only read them to see what is possible.

EDIT (8JUL2015): Added hostNameAndIPAddress() method. Library version remains the same, but it is a new file.
EDIT (30JUL2015): Updated to 1.1. See post #12 for details.
EDIT (25OCT2015): Updated to 1.3. See post #20 for details.
EDIT (26OCT2015): Updated to 1.35. See post #27 for details.
EDIT (22NOV2015): Updated to 1.37. See post #30 for details.
EDIT (30NOV2015): Updated to 1.4. See post #31 for details.
EDIT (30JAN2016): Updated to 1.42. See post #34 for details.
EDIT (31JAN2016): Updated to 1.45. See post #35 for details.
EDIT (31JAN2017): Updated to 1.50. See post #79 for details.
EDIT (02FEB2017): Updated to 1.51. See post #85 for details.
EDIT (06FEB2017): Updated to 1.52. Minor bug fix.
EDIT (04APR2017): Updated to 1.54. Bug fix for relaunchSelfX methods.
EDIT (28DEC2017): Updated to 1.55. Added ScreenCapture2. Read comments for more info.
EDIT (09JAN2019): Updated to 1.59. RobotPaste debugging build
EDIT (14JAN2019): Updated to 1.61. Another RobotPaste2 debugging build

 

[hibrid0]

I updated the library to version 1.3 and added a few new methods. There's now RobotPaste2 and RobotPaste3 which use different methods for invoking the paste key-combo. See the comments on those methods for more details. One of those methods should work on Macs. Please report back on which, if any, of them work on Macs.

I've also added a JVMAddShutdownHook method which exposes Java's addShutdownHook() method. This method spawns a new (briefly lived) thread when the JVM is being shutdown under its own control (i.e. not being shut down by the OS). This allows you to execute some clean up tasks (releasing resources, saving settings, etc...) as the JVM is being shut down.

Hi thanks for the fix, I update and the RobotPaster2 Work fine on MAC and the program crash in Windows if I use RobotPaste2, no problem for that.
I'm testing and maybe on this version not work RobotSpecialKeyPress("enter"), I use it after RobotPaste. And this not work in windows or MAC, on the old version Work fine "enter" on Mac or Windows.

 

[Roycefer]

It works for me on Windows. Are you making sure that you call RobotSpecialKeyRelease("enter") right afterwards? You should probably also call RobotDelay(10) in between KeyPress and KeyRelease. Some operating systems might not register the key stroke if the key wasn't held down for a certain number of milliseconds (I go with 10 to be safe).

 

[rboeck]

Hi Roycefer,
first thanks for your work! I tried many hours in the past to solve the clipboard paste problem for mac - now its working!
In detail robotPaste2 works for me on mac (10.7.5), robotpaste3 doesnt work on any system, for pc i can only use robotpaste, when i use robotpaste2 or robotpaste3 i get this RuntimeException:

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid key code
at anywheresoftware.b4a.keywords.Common.CallSub4(Common.java:471)
at anywheresoftware.b4a.keywords.Common.CallSubNew3(Common.java:425)
at b4j.example.asyncstreamsobject._astream_newstream(asyncstreamsobject.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:93)
at anywheresoftware.b4a.BA$3.run(BA.java:178)
at com.sun.javafx.application.PlatformImpl.lambda$null$170(PlatformImpl.java:295)
at com.sun.javafx.application.PlatformImpl$$Lambda$48/367522714.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$171(PlatformImpl.java:294)
at com.sun.javafx.application.PlatformImpl$$Lambda$47/1685538367.run(Unknown Source)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$null$145(WinApplication.java:101)
at com.sun.glass.ui.win.WinApplication$$Lambda$36/2058534881.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid key code
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:114)
at anywheresoftware.b4a.keywords.Common.CallSub4(Common.java:462)
... 18 more
Caused by: java.lang.IllegalArgumentException: Invalid key code
at sun.awt.windows.WRobotPeer.keyPress(Native Method)
at java.awt.Robot.keyPress(Robot.java:354)
at butt.droid.awtRobot.AWTRobot.RobotPaste3(AWTRobot.java:565)
at b4j.example.main._astreamo_newobject(main.java:213)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:93)
... 19 more​

 

[hibrid0]

Hi Roycefer,
first thanks for your work! I tried many hours in the past to solve the clipboard paste problem for mac - now its working!
In detail robotPaste2 works for me on mac (10.7.5), robotpaste3 doesnt work on any system, for pc i can only use robotpaste, when i use robotpaste2 or robotpaste3 i get this RuntimeException:

java.lang.RuntimeException: java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid key code
at anywheresoftware.b4a.keywords.Common.CallSub4(Common.java:471)
at anywheresoftware.b4a.keywords.Common.CallSubNew3(Common.java:425)
at b4j.example.asyncstreamsobject._astream_newstream(asyncstreamsobject.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:93)
at anywheresoftware.b4a.BA$3.run(BA.java:178)
at com.sun.javafx.application.PlatformImpl.lambda$null$170(PlatformImpl.java:295)
at com.sun.javafx.application.PlatformImpl$$Lambda$48/367522714.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl.lambda$runLater$171(PlatformImpl.java:294)
at com.sun.javafx.application.PlatformImpl$$Lambda$47/1685538367.run(Unknown Source)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:95)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.lambda$null$145(WinApplication.java:101)
at com.sun.glass.ui.win.WinApplication$$Lambda$36/2058534881.run(Unknown Source)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid key code
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:114)
at anywheresoftware.b4a.keywords.Common.CallSub4(Common.java:462)
... 18 more
Caused by: java.lang.IllegalArgumentException: Invalid key code
at sun.awt.windows.WRobotPeer.keyPress(Native Method)
at java.awt.Robot.keyPress(Robot.java:354)
at butt.droid.awtRobot.AWTRobot.RobotPaste3(AWTRobot.java:565)
at b4j.example.main._astreamo_newobject(main.java:213)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:93)
... 19 more​

Hi I see the same problem, I solved with this SystemOperatingSystemEtc from AWTrobot, this return the operating system and I select if is Windows Use robotpaste or if is Mac use robotpaste2.

 

[Roycefer]

Have either of you tried this library on Linux? I think RobotPaste should work on Linux but I'd like to be sure.

Based on your feedback, I will probably combine RobotPaste and RobotPaste2 into one function (called RobotPaste) that will try-catch one method and if it doesn't work, use the other method. This will enable you to use RobotPaste on all operating systems without having to check what operating system is hosting the JVM first.

I can see that I should also add support for the meta key (this maps to the command key on Macs) in the special key combos and such.

 

[hibrid0]

..............

 

[Roycefer]

The library is updated to version 1.35. All the RobotPasteX methods are combined into one RobotPaste method that should work across all supported platforms (no need to check the OS before). Some meta key (the command key on Macs) support was added as well as a method called RobotSupportsMetaKey that will tell you if the operating system supports the meta key. Make sure you check that the system supports the meta key before you call RobotSpecialKeyPress("meta").

As usual, be sure to read a method's comments.

 

[rboeck]

Hi, i tried it now and removed the os checking code and now it's working for mac and windows with the same code. In the past i had my one sendkeys routine and there was the same code for linux and windows, only the mac didnt work. Currently i have no working linux with java to try it...
Thanks!

 

[Roycefer]

Thanks for the feedback. Be sure to post to this thread any other bugs you find.

 

[Roycefer]

Version 1.37 sees a minor addition of the method relaunchSelfBatchFile that pretty much does what it says. It behaves like relaunchSelf except instead of launching its own jar file, it launches a batch file. This is obviously Windows-only (though I'll be adding Linux-friendly versions of both soon enough). This is useful for programs that are better launched using a batch file or for which you want a persistent cmd.exe window visible.

 

[Roycefer]

Version 1.4 sees the addition of RobotSpecialKeyPress/Release support for the home, end, page up and page down keys. Also, the addition of the RobotFunctionKeyPress/Release methods for activating the function keys (F1-F12).

 

[Erel]

Nice library. I used it here to create a remote desktop controller: https://www.b4x.com/android/forum/threads/60887/#content

 

[Roycefer]

Yes, I saw that. Very neat. That's why I originally wrote this library. I've used it in a bunch of remote control type applications. There are also some interesting automation applications to consider. I'm currently using a JVM KVM (thanks, I'll be here all week. Be sure to tip your waitresses.) for controlling a RasPi2 that makes use of this library and a few of my other libraries. I'm just polishing it up a little and then I'll release it as an open-source project for the B4X community.

 

[Roycefer]

Version 1.42 sees the addition of two new methods. Robot.JVMCallingThreadName() does what it says. This is pretty useful for hunting down threading-related bugs. There's also a new Robot.JVMThrowException() method that allows you to throw custom Exceptions:

Sub Divide(n As Double, d As Double) As Double
   If d==0 Then Robot.JVMThrowException("You blew up the universe.")
   Return n/d
End Sub

 

[Roycefer]

Version 1.45 changes the Robot.JVMCallingThreadName() method to Robot.JVMCallingThreadNameGet(). It also adds the Robot.JVMCallingThreadNameSet() method which allows you to rename the calling Thread. I find this much more useful for troubleshooting threading related stuff. For example, multi-threaded Handlers and WebSocketHandlers all run in their own Threads in B4J servers. In their Initialize() methods, you can rename those Threads to something more descriptive so you can track their actions across various modules more easily. The Threading library can get and set the names of Threads it creates, but these new methods allow you to do the same for any Threads in the application.

Erel,
Do you have any plans to implement a developer-accessible UncaughtExceptionHandler for B4J like you've done for B4A? I've been playing with some solutions that I might incorporate into this library but if you're coming out with your own solution, I'd rather not waste the effort.

 

[jmon]

implement a developer-accessible UncaughtExceptionHandler for B4J

That would be great indeed.

 

[Cableguy]

Hi Guys

I'm having a hard time getting the relaunchtself method to work... I used:

Robot.relaunchSelf(File.DirApp & "\MyApp.jar","")

it starts the batch and when it should re launch the file, it return an error stating it could not access the jar file.
The message is cut off so I cannot see the full path its trying to access...

 

[Cableguy]

yes, win10...
Eventually I will opt out for a custom batch file, if I can figure out how to do it, in order to have a "silent" batch file, with no print out to the screen...
For now I'm just trying to get it to restart my app...

 

[lemonisdead]

I can not help more because I just tested it with the same argument you do use and it was working as espected. The way you create the path is the good one.
Here is the content of the "relaunch.bat" batch file created by the Robot

color 0a
timeout 5
start javaw -jar C:\Users\ann\b4j\Travaux\CloudKVS\CloudKVS_Client\Objects\CloudKVS_Client.jar
del "relaunch.bat"

If I remember correctly, adding an ECHO OFF should stop printing the lines (but I will have to try it to be sure)


EDIT: it is something like this

@ECHO OFF
timeout 1 > null
start javaw -jar C:\Users\ann\b4j\Travaux\CloudKVS\CloudKVS_Client\Objects\CloudKVS_Client.jar

 

[Cableguy]

where can I find the temp batch file? If only I could check the path it tries to access...