Android Question parsing text that contain invalid charcters

[Addo]

i am trying to receive bytes and convert it to string and parse it

the bytes that is string list that saved to a memorystream and send to b4a Asyncstream client socket

the code of new data looks like following

Public Sub NewData (data() As Byte)
 
Dim msg As String

msg = BytesToString(data, 0, data.Length, "UTF8")

Dim param As String = msg
Dim paramnum() As String = Regex.Split("\~", param)

Log(paramnum(0))

End Sub

the text that sent from server looks like this

url1~
url2~
Url3~

and so on but after received and convert bytes to string there is some invalid charcters inserted that break the parsing like following

url1~
������������
url2~
Url3~

i dont know from where this characters came after bytetostring conversion

now when i try to capture the parsing data as example paramnum(2)
i got an exception

main_vvvvvvvvv4 (java line: 489)
java.lang.ArrayIndexOutOfBoundsException: length=2; index=2
at app.name.main._vvvvvvvvv4(main.java:489)
at app.name.main._astream_newdata(main.java:381)
at java.lang.reflect.Method.invoke(Native Method)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:186)
at anywheresoftware.b4a.BA$2.run(BA.java:360)
at android.os.Handler.handleCallback(Handler.java:743)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:150)
at android.app.ActivityThread.main(ActivityThread.java:5621)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:794)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:684)
java.lang.ArrayIndexOutOfBoundsException: length=2; index=2

how can i solve this invalid charter after conversion ?

i have used msg = msg.trim with no luck to solve

also i have try to capture if there is any hidden charter that could break that so i used check hidden charcters online i come out with this result

URL1~

URL2~
URL3~
URL4~
URL5~

 

[OliverA]

i will not write stream any more i will start to change the sending side

What Indy version are you using? Looks like UTF8 encoding seems to vary depending on the version. Here is a blurb from this link (https://stackoverflow.com/a/27710809)

Indy 9 does not support encodings, and also does not support Delphi 2009+ for Unicode. Everything in Indy 9 assumes/is based on AnsiString only. Strings are transmitted as-is as if they were raw byte arrays.

So, don't send your AnsiString data over a connection using ANSI. You can send/receive it as UTF-8 instead. You just have to encode/decode the AnsiString data manually, that's all. Indy 9 will send a UTF-8 encoded AnsiString as-is, and read a UTF-8 encoded AnsiString as-is. You can then encode/decode the AnsiString data to/from UTF-8 as needed in your surrounding code.

For example, your Indy 9 client can do this:

IdTCPClient1.WriteLn(UTF8Encode('Some Unicode String Here'));
...
S := UTF8Decode(IdTCPClient1.ReadLn);

Then your Indy 10 server can do this:

AContext.Connection.IOHandler.DefStringEncoding := TIdTextEncoding.UTF8;
...
S := AContext.Connection.IOHandler.ReadLn;
...
AContext.Connection.IOHandler.WriteLn(...);

You say you come from Delphi. Did you ever have this work with non-Delphi programming languages? If so, what were they? What versions? BTW, it would be nice if you can provide some documentation links to for anything you are using on the Delphi side (including versions), since I seem to be incapable to find a nice documentation source for Indy.

 

[Addo]

What Indy version are you using? Looks like UTF8 encoding seems to vary depending on the version. Here is a blurb from this link (https://stackoverflow.com/a/27710809) You say you come from Delphi. Did you ever have this work with non-Delphi programming languages? If so, what were they? What versions? BTW, it would be nice if you can provide some documentation links to for anything you are using on the Delphi side (including versions), since I seem to be incapable to find a nice documentation source for Indy.

i am using indy 10 which i am thinking i should get ride of it all , i am totally new here all i know were delphi and did not have big knowledge in it either

 

[OliverA]

indy 10

Do you have a good online documentation source?