Android Question Decoding of UTF-8 email message body from POP3?

[b4auser1]

How to decode UTF-8 email message body taken using POP3? If all the characters in the body ASCII ones, then there is no an issue, but for 2 and more byte coded characters for non-ascii (f.e. non-english characters) it is necessary to convert from UTF-8 to usual unicode characters, which are used in Java (B4A, B4J)

 

[Erel]

The expected encoding is UTF8. Did you encounter any problem?

 

[b4auser1]

Yes. The two byte characters are not decoded correctly.

 

[Erel]

Can you upload a text file with the data you are sending?

 

[b4auser1]

Please find below. It's ansi win 1251 cyrillic text

 

[Erel]

How are you sending it? The problem is different than what you described. POP object expects a UTF8 message.

 

[b4auser1]

I send and receive using jNet classes and parse message using MailParser. Please find log below.
I send text in ansi win 1251 cyrillic using smtp from jNet to gmail account.
I see text body correctly in Gmail web client. I attached original of message, taken from Gmail. It has Content-Type: text/plain; charset="utf-8". Which is why I can suggest, that text in body converts to not readable when POP class from jNet downloads it from Gmail via POP3.

 

[Erel]

SMTP object always sends the body encoded with UTF8.

Can you post the code you use to read the file and send it?

 

[b4auser1]

I don't read the file. I just use this code

Private m_smtp As SMTP
...

    m_smtp.Subject = "Тема"

    m_smtp.Body = "Text message. Текстовое сообщение"

    m_smtp.HtmlBody = False
   
    m_smtp.Send
...

 

[Erel]

I was mistaken. The native library uses a different encoding. You can use this code to convert it to UTF8:

Sub pop_DownloadCompleted (Success As Boolean, MessageId As Int, Message As String)
   Dim b() As Byte = Message.GetBytes("ISO-8859-1")
   Log(BytesToString(b, 0, b.Length, "UTF8"))
End Sub

 

[b4auser1]

Erel,

Thank you!
Conversion works correctly.

 

[Eldritch]

HI

I'm having the same problem with the subject field even though I have used you "fix".

Se image where the subject is no decoded.


Here is my code.

Sub POP_DownloadCompleted (Success AsBoolean, MessageId AsInt, MessageAsString)

Dim From AsString
Dim b() AsByte = Message.GetBytes("ISO-8859-1")

Message=BytesToString(b, 0, b.Length, "UTF-8")

If Success Then

Dim m AsMessage
m = MailParser.ParseMail(Message, File.DirRootExternal)

If sf.InString(m.FromField,">")>0 Then
From=m.Subject
From=m.FromField.SubString(m.FromField.IndexOf("<")+1)
From=From.SubString2(0,From.IndexOf(">"))
Else
From=m.FromField
EndIf
From=From.Trim
EndIf
111111111111111
End Sub

 

[Erel]

How are you sending the message?

 

[Eldritch]

How are you sending the message?

Hi Erel

I'm not sending. Just receiving.

/Allan

 

[Erel]

What is the output of Log(Message)?

 

[Eldritch]

Attached the log(message)

The emails from flysas.com and apoteker.dk both

 

[Erel]

This is a different issue than the one discussed in this thread. The subject is encoded as base 64 string. You will need to modify MailParser in order to handle this string.

The current implementation takes the string as is and puts it in the Subject field:

Msg.Subject = line.SubString(line.IndexOf(":") + 1)

You will need to change it to this (untested) code:

Dim su As StringUtils
Dim bytes() As Byte = su.DecodeBase64(line.SubString(line.IndexOf("B?") + 2))
Msg.Subject = BytesToString(bytes, 0, bytes.Length, "utf8")

 

[Eldritch]

I then get this

** Activity (main) Create, isFirst = true **

(Main, 51) FullScreen or IncludeTitle properties in layout file do not match the activity attributes settings. (warning #1004)
** Activity (main) Resume **

java.io.IOException: Bad Base64 input character decimal 58 in array position 6

at anywheresoftware.b4a.objects.Base64.decode(Base64.java:1204)
at anywheresoftware.b4a.objects.Base64.decode(Base64.java:1259)
at anywheresoftware.b4a.objects.Base64.decode(Base64.java:1227)
at anywheresoftware.b4a.objects.StringUtils.DecodeBase64(StringUtils.java:51)
at com.ar.SpamCleaner.mailparser._parseheaders(mailparser.java:234)
at com.ar.SpamCleaner.mailparser._parsemail(mailparser.java:315)
at com.ar.SpamCleaner.main._pop_downloadcompleted(main.java:771)
at java.lang.reflect.Method.invoke(Native Method)
at java.lang.reflect.Method.invoke(Method.java:372)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:187)
at anywheresoftware.b4a.BA$3.run(BA.java:332)
at android.os.Handler.handleCallback(Handler.java:739)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:145)
at android.app.ActivityThread.main(ActivityThread.java:5834)

at java.lang.reflect.Method.invoke(Native Method)
at java.lang.reflect.Method.invoke(Method.java:372)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1388)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1183)

It is not all emails where the UTF-8 not converted, just some.

 

[Erel]

You will need to check the subject raw string and look for the B? prefix to decide how it is encoded.

 

[Eldritch]

Hi Erel

Got it working by your advices. I ended up with this subject part

Case "subject"
tmpString=line.ToLowerCase
If tmpString.Contains("utf-8?b?") Then
Dim su AsStringUtils
Dim bytes() AsByte = su.DecodeBase64(line.SubString(line.IndexOf("B?") + 2))
Msg.Subject = BytesToString(bytes, 0, bytes.Length, "utf8")
Else
Msg.Subject = line.SubString(line.IndexOf(":") + 1)
EndIf

Thanks Erel
You are always a great help.