B4J Question Firebase notofication stops to work (2)

marcick · Feb 11, 2019

Again I have problems with Firebase notifications.
I have a JRDC server application on three identical server and the problem appears only on one of them.
The JRDC contains code for managing websockets and also the code for sending notification (copied from the Erel SendTool).
After some times (actually happens every couple of days) the JRDC server seems to works (websocket side) but stop to send notification.
This is the error I grabbed when sending notification:

B4X:

java.net.SocketException: No buffer space available (maximum connections reached?): connect
    at java.net.DualStackPlainSocketImpl.connect0(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:83)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at okhttp3.internal.platform.Platform.connectSocket(Platform.java:124)
    at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.java:187)
    at okhttp3.internal.connection.RealConnection.buildConnection(RealConnection.java:173)
    at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:114)
    at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:196)
    at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:132)
    at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:101)
    at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
    at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
    at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
    at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:120)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
    at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
    at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:179)
    at okhttp3.RealCall.execute(RealCall.java:63)
    at anywheresoftware.b4h.okhttp.OkHttpClientWrapper.executeWithTimeout(OkHttpClientWrapper.java:156)
    at anywheresoftware.b4h.okhttp.OkHttpClientWrapper.access$0(OkHttpClientWrapper.java:153)
    at anywheresoftware.b4h.okhttp.OkHttpClientWrapper$ExecuteHelper.run(OkHttpClientWrapper.java:201)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

When I'm in this situation, if I open Google Chrome it says there is no connection (but it's not true because I'm connected to the server with remote desktop).

If I close the JRDC application and launch it again, it doesn't work correctly, it stops after the first debug lines and hangs. I must restart the computer to solve.
Any idea ?

Erel · Feb 12, 2019

It sounds like an external problem. Once you killed the process it shouldn't have any effect on the OS behavior.

marcick · Feb 12, 2019

Searching on internet about the error

java.net.SocketException: No buffer space available (maximum connections reached?): connect

looks like the cause is

B4X:

This is due to the windows having too many active client connections.

The default number of ephemeral TCP ports is 5000. Sometimes this number may be insufficient if the server has too many active client connections. In that case the ephemeral TCP ports are all used up and no more can be allocated to a new client connection request resulting in the error message (for a Java application)

Is that possible I have so many active connections ? Normalli I have maximum 20 or 30 users connecting to my app, I wonder if I don't close properly something in my program flow.

They say to try this, but I'm not sure what I'm doing

B4X:

The resolution is to open the registry editor and locate the registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters and add a new entry :
Value Name: MaxUserPort
Value Type: DWORD
Value data: 65534

marcick · Feb 13, 2019

I know this is not a Windows forum, but any expert is able to understand something of this ?
Today again notification stopped to work. They works for two days and then stop.

Before restart the machine, I have run "netstat -a" from the powershell and this is the result (doesn't look to me that there are too many active connections)
xx.xx.xxx.xxx is the IP address of the server (masked to post here)

B4X:

Windows PowerShell

Copyright (C) 2016 Microsoft Corporation. All rights reserved.



PS C:\Users\Administrator> netstat -a



Active Connections



  Proto  Local Address          Foreign Address        State

  TCP    0.0.0.0:80             GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:135            GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:445            GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:2548           GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:3306           GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:3307           GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:3389           GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:5985           GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:8031           GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:13000          GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:17179          GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:47001          GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:49664          GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:49665          GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:49666          GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:49667          GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:49668          GIGPSWEB:0             LISTENING

  TCP    0.0.0.0:49672          GIGPSWEB:0             LISTENING

  TCP    xx.xx.xxx.xxx:3389     5.90.6.34:61194        ESTABLISHED

  TCP    xx.xx.xxx.xxx:3389     rrcs-74-62-86-10:55359  CLOSE_WAIT

  TCP    xx.xx.xxx.xxx:17179    2.43.62.74:34599       TIME_WAIT

  TCP    xx.xx.xxx.xxx:17179    2.43.62.74:36519       ESTABLISHED

  TCP    xx.xx.xxx.xxx:17179    2.43.62.74:37348       TIME_WAIT

  TCP    xx.xx.xxx.xxx:17179    GIGPSWEB:49695         ESTABLISHED

  TCP    xx.xx.xxx.xxx:17179    31.157.2.68:54026      ESTABLISHED

  TCP    xx.xx.xxx.xxx:17179    31.157.2.68:54936      TIME_WAIT

  TCP    xx.xx.xxx.xxx:17179    31.157.2.68:54937      TIME_WAIT

  TCP    xx.xx.xxx.xxx:17179    31.157.2.68:54938      TIME_WAIT

  TCP    xx.xx.xxx.xxx:17179    31.157.2.68:54940      ESTABLISHED

  TCP    xx.xx.xxx.xxx:17179    31.157.2.68:54941      FIN_WAIT_2

  TCP    xx.xx.xxx.xxx:17179    31.157.220.201:34837   ESTABLISHED

  TCP    xx.xx.xxx.xxx:17179    31.157.220.201:42640   TIME_WAIT

  TCP    xx.xx.xxx.xxx:17179    31.157.220.201:50278   TIME_WAIT

  TCP    xx.xx.xxx.xxx:17179    31.157.220.201:51783   TIME_WAIT

  TCP    xx.xx.xxx.xxx:17179    31.157.220.201:53228   TIME_WAIT

  TCP    xx.xx.xxx.xxx:17179    31.159.43.224:49684    FIN_WAIT_2

  TCP    xx.xx.xxx.xxx:17179    host164-65-46-89:56332  ESTABLISHED

  TCP    xx.xx.xxx.xxx:17179    109.112.85.31:48304    TIME_WAIT

  TCP    xx.xx.xxx.xxx:17179    109.112.85.31:48487    TIME_WAIT

  TCP    xx.xx.xxx.xxx:17179    109.112.85.31:49759    ESTABLISHED

  TCP    xx.xx.xxx.xxx:17179    109.112.85.31:59085    ESTABLISHED

  TCP    xx.xx.xxx.xxx:17179    host91-171-213-188:53161  ESTABLISHED

  TCP    xx.xx.xxx.xxx:49670    host140-20-37-46:8087  ESTABLISHED

  TCP    xx.xx.xxx.xxx:49673    40.67.254.97:https     ESTABLISHED

  TCP    xx.xx.xxx.xxx:49685    host164-65-46-89:17179  ESTABLISHED

  TCP    xx.xx.xxx.xxx:49694    host164-65-46-89:17179  ESTABLISHED

  TCP    xx.xx.xxx.xxx:49695    GIGPSWEB:17179         ESTABLISHED

  TCP    xx.xx.xxx.xxx:49696    host91-171-213-188:17179  ESTABLISHED

  TCP    xx.xx.xxx.xxx:49747    40.67.251.132:https    ESTABLISHED

  TCP    xx.xx.xxx.xxx:60042    40.67.252.206:https    ESTABLISHED

  TCP    127.0.0.1:3306         GIGPSWEB:58656         ESTABLISHED

  TCP    127.0.0.1:3306         GIGPSWEB:59692         ESTABLISHED

  TCP    127.0.0.1:3306         GIGPSWEB:62056         ESTABLISHED

  TCP    127.0.0.1:8031         GIGPSWEB:0             LISTENING

  TCP    127.0.0.1:49683        GIGPSWEB:49684         ESTABLISHED

  TCP    127.0.0.1:49684        GIGPSWEB:49683         ESTABLISHED

  TCP    127.0.0.1:58656        GIGPSWEB:3306          ESTABLISHED

  TCP    127.0.0.1:59692        GIGPSWEB:3306          ESTABLISHED

  TCP    127.0.0.1:62056        GIGPSWEB:3306          ESTABLISHED

  TCP    [::]:80                GIGPSWEB:0             LISTENING

  TCP    [::]:135               GIGPSWEB:0             LISTENING

  TCP    [::]:445               GIGPSWEB:0             LISTENING

  TCP    [::]:3306              GIGPSWEB:0             LISTENING

  TCP    [::]:3307              GIGPSWEB:0             LISTENING

  TCP    [::]:3389              GIGPSWEB:0             LISTENING

  TCP    [::]:5985              GIGPSWEB:0             LISTENING

  TCP    [::]:13000             GIGPSWEB:0             LISTENING

  TCP    [::]:17179             GIGPSWEB:0             LISTENING

  TCP    [::]:47001             GIGPSWEB:0             LISTENING

  TCP    [::]:49664             GIGPSWEB:0             LISTENING

  TCP    [::]:49665             GIGPSWEB:0             LISTENING

  TCP    [::]:49666             GIGPSWEB:0             LISTENING

  TCP    [::]:49667             GIGPSWEB:0             LISTENING

  TCP    [::]:49668             GIGPSWEB:0             LISTENING

  TCP    [::]:49672             GIGPSWEB:0             LISTENING

  UDP    0.0.0.0:123            *:*

  UDP    0.0.0.0:3389           *:*

  UDP    0.0.0.0:5050           *:*

  UDP    0.0.0.0:5353           *:*

  UDP    0.0.0.0:5355           *:*

  UDP    xx.xx.xxx.xxx:1900     *:*

  UDP    xx.xx.xxx.xxx:59300    *:*

  UDP    127.0.0.1:1900         *:*

  UDP    127.0.0.1:58856        *:*

  UDP    127.0.0.1:59301        *:*

  UDP    [::]:123               *:*

  UDP    [::]:3389              *:*

  UDP    [::]:5353              *:*

  UDP    [::]:5355              *:*

  UDP    [::1]:1900             *:*

  UDP    [::1]:59299            *:*

  UDP    [fe80::5c30:1df3:40b7:763a%6]:1900  *:*

  UDP    [fe80::5c30:1df3:40b7:763a%6]:59298  *:*

PS C:\Users\Administrator>

Erel · Feb 13, 2019

Which version of Windows are you using?

marcick · Feb 13, 2019

Windows Server 2016

Erel · Feb 13, 2019

Are you doing anything with sockets yourself? The Jetty server will not cause such errors even if there are more clients than it can handle.

marcick · Feb 13, 2019

Yes, this is a modified version of JRDC where I added ServerSocket to manage connection from some devices.

Erel · Feb 13, 2019

This is probably the cause of this issue. It is not simple to build a robust server with sockets. In most cases it is a mistake.

marcick · Feb 13, 2019

Yes, I'm pretty sure of this .....
Any hints about how to investigate ? Where can I see (in Windows or adding some monitor code in my B4J app) that I'm consuming resources, leaving opened something, etc ?
Well, I know it's very hard to give an answer ...
Also it's strange that closing the B4J app doesn't solve and a full Windows reboot is required.
Also it's strange that I have this problem only in one server and the other two (absolutely identical) not.

Erel · Feb 13, 2019

Why do you need to create a server yourself? Which type of devices use it?

marcick · Feb 13, 2019

I have a fleet of electronics devices that connect to the server via TCP websocket, send their data and the server app store into MySql.
Also, B4A and B4I apps connect to the server via JRDC and synchronize data.

Erel · Feb 13, 2019

marcick said:
I have a fleet of electronics devices that connect to the server via TCP websocket

I guess that they don't support http or mqtt, right?

marcick · Feb 13, 2019

The would, but TCP works fine and it's easy to manage.
Anyway it's not possible to change the platform with other solutions at this stage.

OliverA · Feb 13, 2019

How are you handling your server socket connections? The FTP example on this forum (https://www.b4x.com/android/forum/t...d-with-socket-and-asyncstreams.74320/#content) is a great example of building a TCP client server. Notice that in the FTPClient component, sockets are closed when either a timeout occurs (Timeout_Tick) or when the quit command is received (HandleClientConnection). For data transfer, the FTPDataConnection class shows sockets closing after a successful data transfer. What protocol do you have in place to close the connections? You could create a counter in you program that increases per connection opening and decreases per connection closing. You may then see if there is an issue with you closing server socket connections.

marcick · Feb 13, 2019

I have "Sub AStream_Terminated" and "Sub AStream_Error" that close sockets like this:

B4X:

Dim astream As AsyncStreams = Sender
    If Main.Bls1ClientMap.ContainsKey(astream) =False Then Return
    Dim cl As Bls1Client=Main.Bls1ClientMap.Get(astream)
    Log(Mysub.mydate(DateTime.Now, "yyyy-MM-dd HH:mm:ss") & ", Astream terminated with " & cl.IP & ", BLS1 " & cl.V_ID)
    astream.SendAllAndClose
    cl.Sock.Close
    Main.Bls1ClientMap.Remove(astream)

I'll have a look at FTP example and yes, that's a good idea to create the counter.
Thanks

@edit: I already have the counter (a map that store the asyncstream) and the map size doesn't increase out f control. Also when I have this trouble, the websockets continues to work and receive data from the devices. But Notifications stops to work with that error

java.net.SocketException: No buffer space available (maximum connections reached?): connect

And Google Chrome or IE no longer connect to internet.

OliverA · Feb 13, 2019

But what terminates the astream? Do you have some logic, such as a timeout or a predefined handshake (such as last packet sent) or terminating it after every connection? If not, your relying on the other side to close the connection and that may be an issue.

marcick · Feb 13, 2019

OliverA said:
But what terminates the astream? Do you have some logic, such as a timeout or a predefined handshake (such as last packet sent) or terminating it after every connection? If not, your relying on the other side to close the connection and that may be an issue.

Yes, I have and handshake to properly close the streams when needed. But also I start a timer for each websocket opened and after 1 minutes of no data exchange the socket is manually closed

OliverA · Feb 13, 2019

Is the server and the ports the devices they use to communicate with open to the internet? If so, you could be experiencing a DOS attack. Or one of the devices is acting up and opening a bunch of sockets in succession. A lot of sockets could be opened in a 1 minute time frame. The counter could detect such a situation. BTW, just to clarify, when you write websocket, you actually mean a TCP socket, not a WebSocket (a totally different beast).

marcick · Feb 13, 2019

code for timeout:

B4X:

For Each a As AsyncStreams In Bls1ClientMap.Keys
        bc=Bls1ClientMap.Get(a)
        If DateTime.Now-bc.Time>300000 Then        ' tiene la connessione con i localizzatori per 5 minuti
            Try
                Log(Mysub.mydate(DateTime.Now,"yyyy-MM-dd HH:mm:ss") & ", Bls1Client " & bc.IP & " purged for timeout")
                a.SendAllAndClose
            Catch
                Log("")
            End Try
         end if
next

B4J Question Firebase notofication stops to work (2)

Well-Known Member

B4X founder

Well-Known Member

Well-Known Member

B4X founder

Well-Known Member

B4X founder

Well-Known Member

B4X founder

Well-Known Member

B4X founder

Well-Known Member

B4X founder

Well-Known Member

Expert

Well-Known Member

Expert

Well-Known Member

Expert

Well-Known Member

Similar Threads