Android Question parsing text that contain invalid charcters

[Addo]

i am trying to receive bytes and convert it to string and parse it

the bytes that is string list that saved to a memorystream and send to b4a Asyncstream client socket

the code of new data looks like following

Public Sub NewData (data() As Byte)
 
Dim msg As String

msg = BytesToString(data, 0, data.Length, "UTF8")

Dim param As String = msg
Dim paramnum() As String = Regex.Split("\~", param)

Log(paramnum(0))

End Sub

the text that sent from server looks like this

url1~
url2~
Url3~

and so on but after received and convert bytes to string there is some invalid charcters inserted that break the parsing like following

url1~
������������
url2~
Url3~

i dont know from where this characters came after bytetostring conversion

now when i try to capture the parsing data as example paramnum(2)
i got an exception

main_vvvvvvvvv4 (java line: 489)
java.lang.ArrayIndexOutOfBoundsException: length=2; index=2
at app.name.main._vvvvvvvvv4(main.java:489)
at app.name.main._astream_newdata(main.java:381)
at java.lang.reflect.Method.invoke(Native Method)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:186)
at anywheresoftware.b4a.BA$2.run(BA.java:360)
at android.os.Handler.handleCallback(Handler.java:743)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:150)
at android.app.ActivityThread.main(ActivityThread.java:5621)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:794)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:684)
java.lang.ArrayIndexOutOfBoundsException: length=2; index=2

how can i solve this invalid charter after conversion ?

i have used msg = msg.trim with no luck to solve

also i have try to capture if there is any hidden charter that could break that so i used check hidden charcters online i come out with this result

URL1~

URL2~
URL3~
URL4~
URL5~

 

[MarkusR]

the example used "UTF-8" i don't know if this will be a difference. you used "UTF8" above

 

[Addo]

I am already used utf8 in both sides

 

[MarkusR]

i meant
i just saw in b4a was it mentioned with a minus char. your code above is without.

 

[Addo]

I don't get your point a code explaination would help where in my code shows minus char ?

 

[Addo]

If you are talking about ~ this is not a minus and its already in the split function .

 

[DonManfred]

No, he is talking about that you use "UTF8" and usually it is named "UTF-8"



It would be helpful if you post the data received as HEX. If it contains the Bytes 0xef,0xbb,0xbf then it is probably a BOM-Header.

Can you upload a textfile with the data you are transfering?

 

[Addo]

Hex data attached as image for better view

 

[DonManfred]

http://www.fileformat.info/info/unicode/char/feff/index.htm

Looks like it is a BOM.... Make sure the text you are sending is UTF8 but NOT "UTF8 with BOM".

Again: Please Upload a TEXT FILE (NOT JPG) of the text you are sending from the sending side.

 

[DonManfred]

I did not asked for a code you are using.
Write the data to a textfile instead of a stream you are sending. Or write the data to a stream which points to a file.
Upload this file.

All i can say is that the data looks like to have a Byte order Mark (BOM) which it should NOT.

 

[DonManfred]

You can try to use the BytesBuilder and remove an BOM from the Stream (byte array in new_data) before converting the bytes to a string.

 

[Addo]

here is the saved data from server side after sent

 

[Addo]

You can try to use the BytesBuilder and remove an BOM from the Stream (byte array in new_data) before converting the bytes to a string.

this will corrupt the bytes data and the results will not be showing as expected

 

[klaus]

This is the hex content of your file:

The first three characters are, as DonManfred suggested, the BOM characters.
Then you have 2SCLIENT~.
And a CR and a LF character after each ~ character.

 

[Addo]

i have tried with no luck

msg = BytesToString(data, 0, data.Length, "UTF-8")
msg = msg.Trim
msg = msg.Replace(CRLF, "")
msg = msg.Replace(Chr(10), "")
msg = msg.Replace(Chr(13), "")
msg = msg.Replace(Chr(127), "")

 

[Erel]

Why are there invalid characters in the data? If the data doesn't represent a string then you shouldn't convert it to a string.

Use BytesBuilder instead.

 

[klaus]

The code below works with you file:

Private txt, split() As String

txt = File.ReadString(File.DirAssets, "mss.txt")
Log (txt)
split = Regex.Split("~" & Chr(13) & Chr(10), txt)
For i = 0 To split.Length - 1
    Log(split(i))
Next

In the file you sent, there are no other 'invalid' characters like in the text you show in your firt post.

 

[Addo]

because it was pure text from server side does not process as byetstostring
iam still trying to figure how bytesbuilder works in this case

The code below works with you file:

Private txt, split() As String

txt = File.ReadString(File.DirAssets, "mss.txt")
Log (txt)
split = Regex.Split("~" & Chr(13) & Chr(10), txt)
For i = 0 To split.Length - 1
    Log(split(i))
Next

In the file you sent, there are no other 'invalid' characters like in the text you show in your firt post.

 

[Addo]

@Erel i am doing following

bb.Append(data)
bb.Remove()

but remove part what i have to remove ? U+FEFF ? i still trying to figure out

 

[klaus]

Well, I am a bit surprised!
Why do you sent something which doesn't show your problem!