B4J Question [TESTING SOLUTION] ServerSocket - CLOSE_WAIT state

[Jmu5667]

Hello

I have a server socket that accepts connects connected from a B4A. It's a plain raw socket and is used to by the server to signal the client do do various tasks etc. The client will send a ping packet every 30 seconds (configurable on the server), a few bytes. We call this the 'Ping Port'

Today I was working on the client app and suddenly the ping port was not accepting ant connections from any app.

I went on the the server and did a netstat -aon (@OliverA). The port in question is 5011.

Line 149:   TCP    51.178.252.89:5011     87.175.186.94:42303    CLOSE_WAIT      23196
Line 150:   TCP    51.178.252.89:5011     87.175.186.94:42304    CLOSE_WAIT      23196
Line 151:   TCP    51.178.252.89:5011     89.19.67.96:1262       CLOSE_WAIT      23196
Line 152:   TCP    51.178.252.89:5011     89.19.67.96:1863       CLOSE_WAIT      23196
Line 153:   TCP    51.178.252.89:5011     89.19.67.96:2138       CLOSE_WAIT      23196
Line 154:   TCP    51.178.252.89:5011     89.19.67.96:4140       CLOSE_WAIT      23196
Line 155:   TCP    51.178.252.89:5011     89.19.67.96:4307       CLOSE_WAIT      23196
Line 156:   TCP    51.178.252.89:5011     89.19.67.96:5555       CLOSE_WAIT      23196
Line 157:   TCP    51.178.252.89:5011     89.19.67.96:5776       CLOSE_WAIT      23196
Line 158:   TCP    51.178.252.89:5011     89.19.67.96:5791       CLOSE_WAIT      23196
Line 159:   TCP    51.178.252.89:5011     89.19.67.96:5795       CLOSE_WAIT      23196
Line 160:   TCP    51.178.252.89:5011     89.19.67.96:6484       CLOSE_WAIT      23196
Line 161:   TCP    51.178.252.89:5011     89.19.67.96:6778       CLOSE_WAIT      23196
Line 162:   TCP    51.178.252.89:5011     89.19.67.96:6951       CLOSE_WAIT      23196
Line 163:   TCP    51.178.252.89:5011     89.19.67.96:8601       CLOSE_WAIT      23196
Line 164:   TCP    51.178.252.89:5011     89.19.67.96:8939       CLOSE_WAIT      23196
Line 165:   TCP    51.178.252.89:5011     89.19.67.96:9093       CLOSE_WAIT      23196
Line 166:   TCP    51.178.252.89:5011     89.204.155.188:14003   ESTABLISHED     23196
Line 167:   TCP    51.178.252.89:5011     91.46.16.243:33261     ESTABLISHED     23196
Line 168:   TCP    51.178.252.89:5011     93.107.118.197:47434   CLOSE_WAIT      23196
Line 169:   TCP    51.178.252.89:5011     93.107.118.197:47436   CLOSE_WAIT      23196
Line 170:   TCP    51.178.252.89:5011     93.107.118.197:47438   CLOSE_WAIT      23196
Line 171:   TCP    51.178.252.89:5011     93.107.118.197:47440   CLOSE_WAIT      23196
Line 172:   TCP    51.178.252.89:5011     93.107.118.197:47442   CLOSE_WAIT      23196
Line 173:   TCP    51.178.252.89:5011     93.107.118.197:47444   CLOSE_WAIT      23196
Line 174:   TCP    51.178.252.89:5011     93.107.118.197:47446   CLOSE_WAIT      23196
Line 175:   TCP    51.178.252.89:5011     93.107.118.197:47450   CLOSE_WAIT      23196
Line 176:   TCP    51.178.252.89:5011     93.107.118.197:47458   CLOSE_WAIT      23196
Line 177:   TCP    51.178.252.89:5011     93.107.118.197:47460   CLOSE_WAIT      23196
Line 178:   TCP    51.178.252.89:5011     93.107.118.197:47462   CLOSE_WAIT      23196
Line 179:   TCP    51.178.252.89:5011     93.107.118.197:47466   CLOSE_WAIT      23196
Line 180:   TCP    51.178.252.89:5011     93.107.118.197:47472   CLOSE_WAIT      23196
Line 181:   TCP    51.178.252.89:5011     93.107.118.197:47474   CLOSE_WAIT      23196
Line 182:   TCP    51.178.252.89:5011     93.107.118.197:47478   CLOSE_WAIT      23196
Line 183:   TCP    51.178.252.89:5011     93.107.118.197:47480   CLOSE_WAIT      23196
Line 184:   TCP    51.178.252.89:5011     93.107.118.197:47482   CLOSE_WAIT      23196
Line 185:   TCP    51.178.252.89:5011     93.107.118.197:47484   CLOSE_WAIT      23196
Line 186:   TCP    51.178.252.89:5011     93.107.118.197:47486   CLOSE_WAIT      23196
Line 187:   TCP    51.178.252.89:5011     93.107.118.197:47488   CLOSE_WAIT      23196
Line 188:   TCP    51.178.252.89:5011     111.7.96.148:22578     ESTABLISHED     23196

I am not too sure why this happens, any insights would be greatly appreciated.

Regards

John.

 

[drgottjr]

tcp connections are expensive (you are paying the price)
looks to me like you have a backlog of "pings". (there is a limit
to the number of connections that can be established). when
you open a socket, eventually there is a corresponding close.
at some point, the close_wait is satisfied, the port is closed
and made available for connections.

in addition, it looks to me like you're opening a new connection to
carry out your "pings".

it's not completely clear why you have to ping, but
let's assume it's to make sure the current work connection
is alive. when you ping from a different port, even if it's to the
same server on its listening port, you are essentially simply
verifying that the server is alive, not that the connection
the work is being done on is alive. the connection you're doing
the work on could well be dead; the ping is coming
from a different port. the only way to know if the
current connection is still alive is to ping from it.
without opening a new connection to carry out the ping
(if that, in fact, is what's occurring. which is what i think is.)
if the ping doesn't answer, then (and only then) would you
want to open a new work connection. fyi, websockets has
this kind of mechanism already built in.

close_wait is part of a connection shutdown. if the other side
does not acknowledge any part of that shutdown, close_wait
waits. if a server is configured to handle X number of connections,
and they're all tied up waiting for wait_close from your pings
to clear, the new connections are dropped.

 

[aminoacid]

The server can only accept one connection at a time. Looks like 111.7.96.148 is still connected to the server socket (line 188). Until this connection is disconnected, the server will not accept a new connection. With TCP, a client can disconnect from the server but the server can still stay in a connected state until you somehow force a disconnect at the server end. You can reproduce this by connecting the client to the server and then "Pull-the-plug" on the client connection by physically disconnecting it from the network.

 

[Jmu5667]

tcp connections are expensive (you are paying the price)
looks to me like you have a backlog of "pings". (there is a limit
to the number of connections that can be established). when
you open a socket, eventually there is a corresponding close.
at some point, the close_wait is satisfied, the port is closed
and made available for connections.

in addition, it looks to me like you're opening a new connection to
carry out your "pings".

it's not completely clear why you have to ping, but
let's assume it's to make sure the current work connection
is alive. when you ping from a different port, even if it's to the
same server on its listening port, you are essentially simply
verifying that the server is alive, not that the connection
the work is being done on is alive. the connection you're doing
the work on could well be dead; the ping is coming
from a different port. the only way to know if the
current connection is still alive is to ping from it.
without opening a new connection to carry out the ping
(if that, in fact, is what's occurring. which is what i think is.)
if the ping doesn't answer, then (and only then) would you
want to open a new work connection. fyi, websockets has
this kind of mechanism already built in.

close_wait is part of a connection shutdown. if the other side
does not acknowledge any part of that shutdown, close_wait
waits. if a server is configured to handle X number of connections,
and they're all tied up waiting for wait_close from your pings
to clear, the new connections are dropped.

Thanks for the insights

 

[drgottjr]

some documentation:
CLOSE_WAIT - Indicates that the server has received the first FIN signal from the
client and the connection is in the process of being closed. This means the socket
is waiting for the application to execute close(). A socket can be in CLOSE_WAIT
state indefinitely until the application closes it. Faulty scenarios would be like a file
descriptor leak: server not executing close() on sockets leading to pile up of
CLOSE_WAIT sockets.


some questions:
1) is the app's network activity running in a service?
2) is this situation a one-off?
3) do you have timeout's in place? (or are pings blindly sent every 30 seconds?)
4) could the pings originate from the server instead of from the device?
dropped connections are more likely to occur on the device side (eg, android
killing the app). if the server doesn't get a response to its ping, it could take action
to protect itself instead of being hung out by an unresponsive client (eg, it kills the process)

 

[Jmu5667]

some documentation:
CLOSE_WAIT - Indicates that the server has received the first FIN signal from the
client and the connection is in the process of being closed. This means the socket
is waiting for the application to execute close(). A socket can be in CLOSE_WAIT
state indefinitely until the application closes it. Faulty scenarios would be like a file
descriptor leak: server not executing close() on sockets leading to pile up of
CLOSE_WAIT sockets.


some questions:
1) is the app's network activity running in a service?
2) is this situation a one-off?
3) do you have timeout's in place? (or are pings blindly sent every 30 seconds?)
4) could the pings originate from the server instead of from the device?
dropped connections are more likely to occur on the device side (eg, android
killing the app). if the server doesn't get a response to its ping, it could take action
to protect itself instead of being hung out by an unresponsive client (eg, it kills the process)

Hi, the issue is not with B4A. It's on B4J, I believe I have solved the problem. I am running test at the moment, and will mark the thread as solved when I am happy with the results, which I will publish.

John.