B4J Tutorial [example] [B4X] Control the desktop with your phone

This example shows how simple it is to create network solutions with B4J, B4A and B4i.

In this case the network communication is based on UDP.

It implements a remote desktop mouse, where the Android device or iOS device control the desktop mouse.

The B4J app uses jAWTRobot library to control the mouse.
The commands are serialized with B4XSerializator.

Note that the B4J app doesn't show any UI except of a system tray icon.
It uses ControlsFX library to show notifications. Note that this requires adding a dummy form (with a UTILITY style to hide it from the task bar).

The nice thing about this example is that the code is very short and simple.

The desktop app listens on port 54323. See this post for information related to the firewall: https://www.b4x.com/android/forum/t...-read-the-logs-in-release-mode.51164/#content

An extended version that works over UDP, TCP or MQTT is available in post #6.

 

[wonder]

Hi Erel! Thanks for this example!!
To my knowledge, UDP is faster than TCP/IP, right?

 

[Erel]

UDP is a bit faster. Though you won't be able to see any difference in a solution such as this one.

UDP is stateless. It makes it easier to work with for a task such as this one. The desktop app just listens for incoming data and the clients send data to the correct port. There is no connection or session involved.

The downside of UDP compared to TCP is that packets can be lost.

 

[wonder]

Got it, thanks!
Could this solution be implemented over the internet as well?

 

[Erel]

Yes. In that case it would be simpler to run an external MQTT broker (such as cloudmqtt).

 

[Erel]

An extended example that works with UDP ,TCP and MQTT is attached.

Unlike what I previously wrote, UDP behaves significantly better in this case.

This is a good example of using classes to easily switch between different network solutions.

 

[micro]

Hi Erel
a question, you think it's difficult send mouse movements together a rapid screenshot of the desktop?
Of course with appropriate scaling on the device.
It would be great to have a real remote desktop!

 

[Roycefer]

The jAWTRobot library can take screenshots. The problem is that taking and sending screenshots at a rate that simulates video can use up a bit of processing power and a lot of bandwidth. There are some schemes I've used whereby you only send small rectangles that have changed their appearance. But it's still not a trivial system to implement.

 

[micro]

But it's still not a trivial system to implement.

I know it's hard but I think that with a low resolution (but appropriate) you may be able.

 

[derez]

I have a non-ui server that handles commands and sends screenshots, using the jAWTRobot

Public Sub Handle(req As ServletRequest, resp As ServletResponse)
    robot.ScreenCurrentRectangleSetAsArbitrary(0,0,1280,720)
    img  = robot.ScreenCaptureAsByteArray
    Dim su As StringUtils
    resp.Write("<img width=1280 height=720 src='data:image/png;base64,")
    resp.Write(su.EncodeBase64(img))
    resp.Write("'/img>").write("</br>")
End Sub

But it is working with a messageloop. How can I combine the UI application above with the Non-ui ?
The receiving app (b4a) code is this:

Sub JobDone (Job As HttpJob)
     If Job.Success = True Then
       Select Job.jobname
          ...           
        Case "screen"
            Dim st As String = Job.GetString
            Dim su As StringUtils
            Dim barr() As Byte
            barr = su.DecodeBase64(st)
            Dim InputStream1 As InputStream
            InputStream1.InitializeFromBytesArray(barr, 0, barr.Length)
            Dim bmp As Bitmap
            bmp.Initialize2(InputStream1)
            iv.SetBackgroundImage(bmp)
            iv.Visible = True
           
    End Select

Sending the screenshot every few seconds is enough for my needs.
In another way, how to send the picture (lets say by a timer) and how to get it in the receiver app.

 

[derez]

I managed to do it with TCP because it has a two way connection (while UDP requires the ip of the receiver which I don't have as a fix ip).
The server sends this with a timer :

robot.ScreenCurrentRectangleSetAsArbitrary(0,0,1280,720)
img  = robot.ScreenCaptureAsByteArray
astream.Write(img)

The b4a use this in the TCPconnector:

Sub astream_NewData (Buffer() As Byte)
    Dim InputStream1 As InputStream
    InputStream1.InitializeFromBytesArray(Buffer, 0, Buffer.Length)
    bmp.Initialize2(InputStream1)
End Sub

What is left to do is to send somehow the position of the mouse and display it on the image (or draw it and then send ?) because it is not included in the screenshot.

 

[derez]

I would like to know the position of the mouse in the b4j app.
I believe it should be retrieved somehow by the remotecommand object.

 

[Roycefer]

In the B4J app, call robot.RobotMouseCurrentLocation() to get the mouse pointer's current location.

 

[derez]

The attached files work as a remote control of a PC by a b4a device, sending mouse movements and receiving the screenshot with possibility of zoom in and out.
I left only the TCP for communication.
It works with my nexus5, I have not scaled it yet to work with anything else.
The port and host ip are different from the original because this is what I use at home, so change them for your setup.
It is just a demo !

 

[tufanv]

This example shows how simple it is to create network solutions with B4J, B4A and B4i.

In this case the network communication is based on UDP.

It implements a remote desktop mouse, where the Android device or iOS device control the desktop mouse.

The B4J app uses jAWTRobot library to control the mouse.
The commands are serialized with B4XSerializator.

Note that the B4J app doesn't show any UI except of a system tray icon.
It uses ControlsFX library to show notifications. Note that this requires adding a dummy form (with a UTILITY style to hide it from the task bar).

View attachment 39318 View attachment 39323

The nice thing about this example is that the code is very short and simple.

The desktop app listens on port 54323. See this post for information related to the firewall: https://www.b4x.com/android/forum/t...-read-the-logs-in-release-mode.51164/#content

An extended version that works over UDP, TCP or MQTT is available in post #6.

if we want to use this for mac , how does the firewall port allowing works ? Because in windows , it automaticly requests for authorization with a popup , if we want to use it with mac , how can we allow the firewall inside the app , or does it have to be manually added ?

 

[Erel]

Please start a new thread for this question.

 

[Hypnos]

Anyone know what is this error?

Picked up _JAVA_OPTIONS: -Djavax.net.ssl.trustStore=c:\windows\sun\java\deployment\trusted.cacerts
main._shownotification (java line: 163)
java.lang.NoSuchMethodError: javafx.stage.Window.impl_getWindows()Ljava/util/Iterator;
    at org.controlsfx.tools.Utils.getWindow(Utils.java:50)
    at org.controlsfx.control.Notifications$NotificationPopupHandler.show(Notifications.java:310)
    at org.controlsfx.control.Notifications.show(Notifications.java:269)
    at org.controlsfx.control.Notifications.showInformation(Notifications.java:244)
    at anywheresoftware.b4j.objects.ControlsUtils.ShowNotification3(ControlsUtils.java:138)
    at b4j.example.main._shownotification(main.java:163)
    at b4j.example.tcpconnector._showip(tcpconnector.java:87)
    at b4j.example.tcpconnector._initialize(tcpconnector.java:75)
    at b4j.example.main._appstart(main.java:95)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at anywheresoftware.b4a.BA.raiseEvent2(BA.java:91)
    at anywheresoftware.b4a.BA.raiseEvent(BA.java:78)
    at b4j.example.main.start(main.java:37)
    at javafx.graphics/com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$9(LauncherImpl.java:846)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runAndWait$12(PlatformImpl.java:455)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$10(PlatformImpl.java:428)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$11(PlatformImpl.java:427)
    at javafx.graphics/com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
    at javafx.graphics/com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
    at javafx.graphics/com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
    at java.base/java.lang.Thread.run(Thread.java:834)

 

[Ronce]

Anyone know what is this error?

Picked up _JAVA_OPTIONS: -Djavax.net.ssl.trustStore=c:\windows\sun\java\deployment\trusted.cacerts
main._shownotification (java line: 163)
java.lang.NoSuchMethodError: javafx.stage.Window.impl_getWindows()Ljava/util/Iterator;
    at org.controlsfx.tools.Utils.getWindow(Utils.java:50)
    at org.controlsfx.control.Notifications$NotificationPopupHandler.show(Notifications.java:310)
    at org.controlsfx.control.Notifications.show(Notifications.java:269)
    at org.controlsfx.control.Notifications.showInformation(Notifications.java:244)
    at anywheresoftware.b4j.objects.ControlsUtils.ShowNotification3(ControlsUtils.java:138)
    at b4j.example.main._shownotification(main.java:163)
    at b4j.example.tcpconnector._showip(tcpconnector.java:87)
    at b4j.example.tcpconnector._initialize(tcpconnector.java:75)
    at b4j.example.main._appstart(main.java:95)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at anywheresoftware.b4a.BA.raiseEvent2(BA.java:91)
    at anywheresoftware.b4a.BA.raiseEvent(BA.java:78)
    at b4j.example.main.start(main.java:37)
    at javafx.graphics/com.sun.javafx.application.LauncherImpl.lambda$launchApplication1$9(LauncherImpl.java:846)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runAndWait$12(PlatformImpl.java:455)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$10(PlatformImpl.java:428)
    at java.base/java.security.AccessController.doPrivileged(Native Method)
    at javafx.graphics/com.sun.javafx.application.PlatformImpl.lambda$runLater$11(PlatformImpl.java:427)
    at javafx.graphics/com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:96)
    at javafx.graphics/com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
    at javafx.graphics/com.sun.glass.ui.win.WinApplication.lambda$runLoop$3(WinApplication.java:174)
    at java.base/java.lang.Thread.run(Thread.java:834)

I don't know why that error happens...But to omit that error I remove the "Main.ShowNotification" code under public sub ShowIp and error gone then the program works

Public Sub ShowIp
'    Main.ShowNotification("UDP connector",$"Waiting for UDP data.
'Make sure that the following port is open in the firewall (UDP): ${udpsock.Port}"$, False)
End Sub

 

[zabayin]

I've changed this project to B4X but never success. the packet is well-received by B4J but in the handle command, it seems needs to be a little modification. I'm stuck in this.

(ClassNotFoundException) java.lang.ClassNotFoundException: b4j.example.b4xmainpage$_remotecommand

I know this is an old thread But that thing should be here i think. Sorry Sir.

 

[Erel]

Make sure that all the serialized types are declared in the Main module.