Read NMEA from Server

[schimanski]

same problem

"Does your app make the assumption that a full message always arrives as a single read?"

Hello agraham!

Thanks for answer. But I know, that even can arrive small messages. In my application, the GPSStream collects the parts of the NMEA, so that I have every 4 seconds a new protocol, when the server sends every 2 seconds. That operates very fine (the image is only 15x15 pixels, so I think, it doesn’t matter):

Sub Timer2_Tick
ErrorLabel(connectionbreak)
If ComboBox2.SelectedIndex = 1 Then
If client.DataAvailable=true Then
Image4.LoadPicture("IconDaten1.bmp")
GPS_Server.GPSStream(filestream.ReadString)
Else
Image4.LoadPicture("IconDaten2.bmp")
End If
Else If client.DataAvailable Then
fileStream.ReadString
End If
Return

connectionbreak:
..
client.close
End Sub

When the timer is bigger than 500, I will take in backlog. So I only can get older data. I can see that, when I take a look at the UTC-Time. The different between the UTC-Time from the protocol which the server sends and the UTC-Time in the GPS_Server_GPSDecoded-sub runs up.
The connectionbreak is only called, when the GPRS-connection breaks down.

When the timer2 is 1000 and smaller and the server sends no data, the app stops, before there is a reaction on client.dataAvailable<>true.

:sign0148::sign0148:

 

[schimanski]

delete the filestream

Hello again!

I haven't found an answer on my problem, so i have one new idea:

With the following sub, i take the data from the server. When the timer2_tick is 2000, the application runs fine, but I will take in backlog. So I only can get older data. I can see that, when I take a look at the UTC-Time. The different between the UTC-Time from the protocol which the server sends and the UTC-Time in the GPS_Server_GPSDecoded-sub runs up.
But I think, that it is not possible to read from the filestream every half or one second, because the server sends only every two seconds. Now my idea:

Is it possible to reset the filestream every read, so that i will not take in backlog and only take the new data?


Sub Timer2_Tick
ErrorLabel(connectionbreak)
If ComboBox2.SelectedIndex = 1 Then
If client.DataAvailable=true Then
Image4.LoadPicture("IconDaten1.bmp")
GPS_Server.GPSStream(filestream.ReadString)
Else
Image4.LoadPicture("IconDaten2.bmp")
End If
Else If client.DataAvailable Then
fileStream.ReadString
End If
Return

connectionbreak:
..
client.close
End Sub

:sign0085::sign0060:

 

[agraham]

If a backlog is building up then multiple messages must be available. Instead of just doing a single read try clearing the backlog with a Do While .. Loop.

Sub Timer2_Tick
  ErrorLabel(connectionbreak)
  If ComboBox2.SelectedIndex = 1 Then
    If client.DataAvailable Then
      Do While eclient.DataAvailable 
        GPS_Server.GPSStream(filestream.ReadString)
      Loop
      Image4.LoadPicture("IconDaten1.bmp")
    Else
      Image4.LoadPicture("IconDaten2.bmp")
    End If
  Else 
    Do While client.DataAvailable 
      fileStream.ReadString
    Loop
  End If
  Return
  connectionbreak:
  ..
  client.close
End Sub

 

[schimanski]

where is the mistake?

This is my code, the same problem as my last posts. A simple code but it doesn' work:

Sub App_Start
Form1.show
client.New1
client.Connect("217.91.33.???,55355)
filestream.New1(client.getstream,false)
timer1.Enabled=true
End Sub

Sub timer1_Tick
if client.DataAvailable=true then
Label1.Text=filestream.readstring
else
Label1.Text="no data"
end if
End Sub


The server sends only string as NMEA (RMC and GGA) every two seconds. When i block the gate from the server, so that there is no data sending, the application blocks in this line: Label1.Text=filestream.readstring. Than, there is no chance to end the application or even push a button. I have tested it with different timer from client and server, but it always stops in filestream.readstring. When I open the gate, the application will continue without problems.
I have spoken with the author from the server-code. He says, that there is shure no data, when the gate is blocked. He says, that the filestream.readstring is a blocked function. It waits until there is any data. But i can't catch the mistake with client.dataavailable. Is it possible, that there is a different between reading the filestream until it is emty and read from a filetream, that is emty from the beginning? Because, if i read a full filestream until it is emty, then Label1.text shows "no data".
When I connect to a port, where is no data from the beginning, the application stops an locks up until I gi´ve data to the stream.

:sign0085::sign0085::sign0085:

I think, that i will fall into despair....

 

[schimanski]

This is my code, the same problem as my last posts. A simple code but it doesn' work:

Sub App_Start
Form1.show
client.New1
client.Connect("217.91.33.???,55355)
filestream.New1(client.getstream,false)
timer1.Enabled=true
End Sub

Sub timer1_Tick
if client.DataAvailable=true then
Label1.Text=filestream.readstring
else
Label1.Text="no data"
end if
End Sub


The server sends only string as NMEA (RMC and GGA) every two seconds. When i block the gate from the server, so that there is no data sending, the application blocks in this line: Label1.Text=filestream.readstring. Than, there is no chance to end the application or even push a button. I have tested it with different timer from client and server, but it always stops in filestream.readstring. When I open the gate, the application will continue without problems.
I have spoken with the author from the server-code. He says, that there is shure no data, when the gate is blocked. He says, that the filestream.readstring is a blocked function. It waits until there is any data. But i can't catch the mistake with client.dataavailable. Is it possible, that there is a different between reading the filestream until it is emty and read from a filetream, that is emty from the beginning? Because, if i read a full filestream until it is emty, then Label1.text shows "no data".
When I connect to a port, where is no data from the beginning, the application stops an locks up until I gi´ve data to the stream.

:sign0085::sign0085::sign0085:

I think, that i will fall into despair....

I have tested more: when i open and close then gate from the server a lot of times, sometimes the application blocks and sometimes the label1.text shows "no data". I think, the is a different between the amount of data...

 

[agraham]

I don't understand your problems. I have used the Network library here http://www.b4x.com/forum/showthread.php?t=1130 and on another app and it does not experience any blocking when no data is available.

It seems that your client.DataAvailable is returning true when there is no data to read, but I have not seen this behaviour myself

What I do notice is "filestream.New1(client.getstream,false)" is specifying that the stream encoding is UTF8. Is this in fact true? I would have thought it more likely that the server data was ASCII encoded. Could this be why filestream is blocking? It might be that there is data available but it is expecting more data than is actually available dueto the encoding difference.

EDIT :- Re-reading the thread you have said before that the app works OK when there is data so my encoding suggestion is probably wrong.

 

[schimanski]

The same problem with Network Demo

Hello agraham!

First I have tested it with filestream.new1(client.Getstream,true and false), but no different. I also have tested your Network Demo-Application. But This app only shows, that it ist connected with the server, but than, the application locks also up. This application doesn't work when there is data and also when there is no data. I have set the filestream.new1 on true and false, but no different. Is it possible to send you the original IP and port-no. to an email-adress? I can send data on this port for perhaps the next three hours.

 

[agraham]

You can certainly PM it to me but unless the problem is with your server there probably isn't much point. I have tested my Network demo in all the combinations possible of a Dell Axim X30 running WM2003SE, a Compaq latptop running Windows XP and my dekstop running Vista Home Premium. If my Network demo locks up I think there must be some sort of bug or problem with whatever you are running your app on. Have you tried it on both a device and a desktop?

 

[schimanski]

Private message to you

i have send a private message with the IP and port to you. I have tested the application on three different devices, not on the desktop. I don't think, that there is a problem with basic4ppc or your application Network Demo. I think, that i have overlook something by the data format from the server.

:sign0013:

 

[agraham]

Hmmm! I'm connecting to your server and I'm getting the same effects as you have desribed and I don't understand it I'll play a bit more.

 

[agraham]

I am at a loss! I can confirm that when connected to your server when DataAvailable is true it can take a long time for ReadStream to return when polling quickly, but it just shouldn't do this, it should return immediately with whatever data is available.

I have knocked up a server to send a long string every 2 seconds and connected my net demo app as a client from another computer to it over my network. The timer tick is 100mS in the client and it behaved exactly as I would have expected, displaying the new string every 2 seconds. Connect the same unchanged client to your server and it locks up. There really should be no difference as far as the client is concerned, it is just waiting for a long string to arrive on a network port. The only thing I can think of is that there is something weird with the TCP/IP stack in that server.

Sory but I can't think of anything else to do!

 

[schimanski]

Thanks for your efforts, agraham....

:sign0188:

 

[schimanski]

New idea!!!

Hello!

Thanks, agraham, for your last threads. I think, I have to take another way.

So, i have a new idea. Is it possible to run a seperat small application in the backround that takes the data from the server and send this data to the main application? So the small application can stop, when it is waiting for data without lock up the main program.

:sign0060::sign0060:

 

[agraham]

I would try using my Threading library http://www.b4x.com/forum/showthread.php?t=1611 to run what is presently your timer code on a thread passing the data to the GPS. The GPSDecoded event is thread safe and will tell the rest of your code that data has arrived. Your thread can use Sleep() to determine the approximate rate you want to check DataAvailable. You should use RunLock in both main program and thread to access the GPS object.

I can't help but think there is something wrong somewhere - you shouldn't need to go to these lengths to get it to work

 

[TWELVE]

Hello schimanski,

i just flew over your code.

From what i can understand so far, your problem is that you queue up data from the server if you don't read the net buffer for a while.

I don't know what kind of server this is and what protocol it's using, but my first guess is, to just close the connection if you don't want to read data anymore.

If you request data from the server and the server is sending reply data to your request continously, it is obvious that you need to read all reponses before you can get to the newer data.Furthermore, new data will be on the end of your buffer and therefore if you just read the first bytes / characters of your buffer you will always get the last server reponse the client received when it stopped reading from the stream.


Receivebuffer:

NMEA sentence 1, NMEA sentence 2

You read the sentences and therefore the buffer is now empty.
Server sends new data and you read the receivebuffer constantly.The data in the first characters / bytes is always current due to this.

You stop then reading the stream.After 2 minutes you start to read the stream again.

The receivebuffer does now look like this:

NMEA Sentence 1 ( 2 minutes old !) NMEA Sentence 2 ( 2 mins - 1sec) NMEA Sentence 3 ( 2mins - 2 secs.)

If you now just look at the first characters / bytes of your buffer, you will see only the old data, to get newer data sentences, you have to look at the end of your buffer, for example by searching strings or characters.






cheers

TWELVE

 

[schimanski]

I mean the other problem!!!

Hello TWELVE!

Thanks for your fast answer, but that was not the problem, which I mean. Sorry, there are al lot of problems in my thread. :sign0013:

I mean this:

I have written a smal testapp to see, where the problem is. 
When the server sends no data, the application will stop as long,
as the server sends data again. When there is data again, the Application 
runs without any problems. Is there no data, i only can stop the app with the
task-manager or break down the GPRS-connection. Here the code:

Sub App_Start
  Form1.Show
  client.New1
  Client.Connect ("87.139.??.???",11112)
  filestream.New1(client.GetStream,false)
  timer3.Enabled=true 'timer3=500 or 1000 ms'
End Sub

Sub Timer3_Tick
  If client.DataAvailable=true Then
    filestream.ReadString
    label5.Text="data!"
  Else
    label5.Text="no data!"
  End If
End Sub

The server-connection is always available!

I think, that client.dataavailable on this server (sends NMEA every two seconds as string in a TCP-protocol) also doesn't run. When the server stop sending data, the application lock up in the line filestream.readstring.

Is there a way to write another sub with a counter etc.?

The device is connected with the server over an GPRS-connection.

m.f.G.

schimanski

 

[TWELVE]

but that was not the problem, which I mean.

Sorry..sometimes hard to get that sorted out in these longer threads.

I think, that client.dataavailable on this server (sends NMEA every two seconds as string in a TCP-protocol) also doesn't run.

If it wouldn't, you never would get any data back from the server.

When the server stop sending data, the application lock up in the line filestream.readstring.

From my point of view this can never happen, because the filestream.ReadString is only called if the client.DataAvailable condition is true.It can happen, if you don't use the client.DataAvailable thing and just try to read the stream.If there's data you will get it, if not, the .readstring will lock until data arrives the net buffer.

Since the timer3 is only called every half or full second, the main app would
not get locked entirely - if there's no deadlock condition within the timer3 code as described by you.If the receiving of data takes a notable amount of time ( and this is the case without doubt when GPRS is used..) , the main app freezes during this time.

Multi-Threading as suggested by agraham would help out here only, if there's no deadlock condition ( ReadString is just waiting and has a chance to get read new data from the stream ever ).

But certainly the exact behavior of this code depends on the latency/speed of your internet connection.If the connections needs 20secs. to retrieve the data, the Main App / GUI gets blocked for this period of time.

So, my question is: does it really deadlock or just block for a couple of seconds...?

If there's a main app and a code part, that is transfering data from/to internet , it is always a good idea to have that run in parallel ( threading, OS callback etc..) because of the freeze effects.Since Basic4PPC is not able to handle threading, this can only be achieved with agraham's threading lib.

Is there a way to write another sub with a counter etc.?

I don't think a counter can help here somehow.The only thing that is able to prevent the .Readstring from blocking is the client.DataAvailable thing.


I don't think you have to struggle here with the issue i described in the telnet client thread, because in that case your "If client.DataAvailable=true Then" would not become true and therefore the code just would not read from the stream.My code works different, because of the used protocol.I send commands to the server and expect a certain reply to this.



regards

TWELVE

 

[agraham]

From my point of view this can never happen, because the filestream.ReadString is only called if the client.DataAvailable condition is true.

It does happen on his server, I've tried it - see post #31.

 

[schimanski]

So, my question is: does it really deadlock or just block for a couple of seconds...?

It is a really deadlock, not only a couple of seconds.


You're so right. I see, that the problem isn't dataavailable.

I don't have any authority to the server-code, so I think, that I have to life with this problem.

So much thanks to you, TWELVE and AGRAHAM, for your efforts.

:sign0188:

 

[agraham]

I have an idea!

This DataAvailable has been worrying me. I have googled around and found a very few instances of other people suffering similar reliability problems with DataAvailable so I have thought of a possible workaround. This needs .NET 2.0 and the Door library. It relies on setting the ReadTimeout of the Stream to a small number then calling Read which then throws an exception on timeout or returns the number of bytes read. I have tried the following code in a real application and it works for me. However I have not experienced any DataAvailable problems myself so I do not know whether this technique will sidestep any problems or not. Anyway being able to try a read and not block if it fails can't help but be a good thing.

' globals
Count = 0 ' count of bytes received
Dim WebResp(4096) As Byte ' read buffer


Sub DataAvailable
  ErrorLabel(DAerr)
  count = Stream.ReadBytes(WebResp(),4096)
  Return true
DAerr:
  Return false ' if the Read times out it throws an exception
End Sub


Sub NetGetData(req)
  msg ="An error occurred connecting to " & URL
  ErrorLabel("err2")
  Client.New1
  Client.Connect(URL, Port)
  Stream.New1(Client.GetStream,true)
  StreamObj.Value = Client.GetStream 'StreamObj is a Door library Object
  StreamObj.SetProperty("ReadTimeout",1) ' set the Read stream timeout to 1 mSec
  msg = "GET /" & req & " HTTP/1.1" & Crlf 
  msg = msg & "Host: " & URL & Crlf
  msg = msg & "Proxy-Connection: Keep-Alive" & Crlf
  msg = msg & "Pragma: no-cache" & crlf
  WebReq() = Stream.StringToBytes(msg)   
  Stream.WriteBytes2(WebReq(),0,StrLength(msg))
  Timer = 0
  Do
    Sleep(SleepMs)
    Timer = Timer + SleepMs
    If Timer >= Timeout Then
       Return  "A timeout occurred communicating with " & Device
    End If
    Loop Until DataAvailable
    msg = Stream.BytesToString (WebResp(), 0, Count)
  Client.Close
  Return msg
End Sub