Android Question ASyncStreams buffer size

[jtim3032]

Is there a size limit on the buffers used by ASyncStreams and if so, is there a way of increasing it?

My problem is that I am connecting an NTRIP client to a NTRIP Caster. Connecting via WiFi and a router poses no problems. However, when I connect via 3G/4G, there is a delay of around 16-18 seconds before the first data sent from the Caster gets back to the device (I can monitor the caster which resides on my own server and compare timings with the android device), by which time the data has accumulated to around 10,000 bytes which seems to break the ASyncStreams input buffer.

It seems there is something in the NTRIP data stream which makes the network provider delay the data as this doesn't seem to happen when connecting to other types of data source, eg raw RTCM data from a relay server.

Since there does not seem to be any method in the NTRIP Client login process to separate the initiation of data from the initial login, I am stumped...

TIA...

 

[Erel]

I guess that you are not using prefix mode, right?
In that case there is no buffer at all. The events are raised immediately when the data is available.

 

[jtim3032]

Yes, the Caster is 3rd party and therefore I have no control over that end. Typically the data is sent in packets of around 500bytes once a second which is all fine. However, over that initial 16 seconds delay this must accumulate and at the first NewData event I get about 8192 bytes, then typically 2000 or so bytes and then a "socket reset" occurs. At the same time, the Caster also reports a client "disconnect" (initiated by the client).

 

[Erel]

The buffering happens somewhere else.

 

[jtim3032]

Agreed...thanks anyway Erel!

 

[emexes]

Typically the data is sent in packets of around 500bytes once a second

I thought NTRIP was HTTP-like, and so presumably each second's request or data is done in a separate connection. Is this the case? I am sorry to ask this dumb question, but it does kinda fit the symptoms - is it possible that you are sending requests, ~500 bytes each second, but not ending the request and starting anew, and thus everything is jammed up together so that it appears to the server as if it is one big long never-ending request?

Or if you are definitely sending requests in separate connections - perhaps try restricting to a maximum of 3 unfulfilled/outstanding requests in the pipeline at any one time?

at the first NewData event I get about 8192 bytes, then typically 2000 or so bytes and then a "socket reset" occurs

That 2^13 number does indeed sound like a packet size limit being hit somewhere. Or that the server isn't closing the HTTP response until it the request is closed, but is sending subresponses to your subrequests, and eventually a MTU limit is reached, flushing the first part of the response, and then a timeout is hit, flushing the rest.

Maybe.

:-/

 

[jtim3032]

I thought NTRIP was HTTP-like, and so presumably each second's request or data is done in a separate connection

Hi emexes...yes I agree with everything you say. NTRIP is indeed an HTTP stateless protocol and request/response pairs should be in individual connections and indeed the initial headers include the "Connection: close" line. I have copied connection code from a well known developer in the field (Lance Lefebure) and it seems that once login details have been sent, then data is "streamed continuously" (or at least once a second) back to the client and it would seem that this is all part of the original request/response pair. It is interesting that going direct to internet (ie via WiFi and a router) gets a response within milliseconds and data continues to come in as expected. It is only 4G that gives me an issue and it is as if the network provider is doing some checking before passing the first data back. I know there is a lot of protection for devices within the cellular network and hence it is difficult to use mobile devices as servers etc. I also do not get this problem if the data is in raw RTCM format (ie I set up my own relay server to do this but I am trying to migrate to a proper NTRIP solution to be able to manage multiple base station data sources). Maybe something in the NTRIP data packets is making the network provider's checkers suspicious?

Interestingly, occasionally the client gets through the initial data connection without the socket breaking and when it does, data continues to flow in fine fashion

Also annoyingly, RTCM, the body that publishes the NTRIP protocol, want $230 or thereabouts for a copy of the protocol and I can't find a copy anywhere else on the net!

 

[emexes]

It is only 4G that gives me an issue

How embarrassment I forgot that it worked via another internet connection

Interestingly, occasionally the client gets through the initial data connection without the socket breaking and when it does, data continues to flow in fine fashion

That is interestingly-squared. Presumably if you stop and restart the same app, no changes, it then fails. So... what could be different between those two runs? Dumb thought, but... can you try a different mobile service provider? Either by swapping SIMs or by hotspotting to another phone? (or both)

Also annoyingly, RTCM, the body that publishes the NTRIP protocol, want $230 or thereabouts for a copy of the protocol and I can't find a copy anywhere else on the net!

And that is annoyingly-squared. I think I followed your footsteps, got the same result - sporadic glimpses of information, but not the actual full standard. One backchannel that occasionally works is a local university library, if you know somebody that studies there.

 

[jtim3032]

One backchannel that occasionally works is a local university library, if you know somebody that studies there.

I eventually found a copy here. It didn't help however . I have instead found another solution to the problem. As I already knew, mobile devices are effectively behind a NAT router which the network provider maintains and device user have no access to port forwarding etc, hence the difficulty of running a server from a mobile device. Eventually I found someone mentioning VPN or even SSH tunnelling, so I thought I would give it a go. It worked straight off! So the moral is, if you are requesting a data stream via HTTP and 3/4G, do it via a VPN. Then the NTRIP server can effectively find you to return ongoing data.

 

[emexes]

I eventually found a copy here.

Heck, that site is a good find!!!

An old boss of mine was a vulcanologist, he’d bring in his magazines about GNSS and surveying and geological data acquisition, and they were always an interesting read. Included stuff like upcoming satellite launches and what new features they had. And now I know way more about GPS than I ever needed (or wanted) to know...

Also a book about the Mars lander/rover missions - that was so excellent, I bought my own copy.