apostrophe as square - UTF-8 encoding / sqllite

Y

yonson

Active Member
Licensed User
Longtime User
I'm sure someone else will come across this problem so I thought it a good idea to ask for some help.

Basically I'm loading an XML file into a sqlite database, and rendering the data as listviews. The issue is with character encoding, specifically apostrophes are being rendered as squares within the app.

Now I'm aware of the problem i.e. how the characters have been encoded , but I have checked/ verified the following:-

  • the XML is encoded with utf-8 encoding
  • sqllite is, as I understand it, utf-8 by default
  • I've tried replacing the apostrophe character with html markup etc but this doesnt make any difference

I'm assuming the problem is either with the sqllite database having the wrong character encoding set, or the xml it starts off with having the wrong encoding, but both are UTF-8 as far as I can tell.

Can anyone clarify what I need to do to fix this? Please excuse my ignorance and hope you can help!
 
Sort by date Sort by votes
Rusty

Rusty

Well-Known Member
Licensed User
Longtime User
Erel,
I am loading an HTML file into a webview. The HTML is created by saving a Word Document into .htm format and then loading it into direct assets for use within my app.
All web viewers on the PC view the file perfectly, but the webview in b4a, shows black diamonds with a question mark within throughout the document.
Is there an encoding capability or requirement to handle this?

B4X: 
<p class=MsoNormal><b style='mso-bidi-font-weight:normal'>Title:<span
style='mso-spacerun:yes'>  </span></b>This is a test html sentence, 
this text is being displayed within a webview</p>
Word adds the text <span style='mso-spacerun:yes'> </span></b>
this seems to be the source of the black diamonds. Within the Word doc there are two blank spaces.
In either case, when I view the html document in Firefox for example, it looks fine even with the embedded span. In the Android webview it has the diamonds.
Any advice is greatly appreciated.
Thanks,
Rusty
 
Last edited:
Upvote 0
Rusty

Rusty

Well-Known Member
Licensed User
Longtime User
Erel,
I have created an example html file that has the problems of the black diamond with embedded question mark for you. (see attached screen image)
It will load properly with loadurl, but won't load with loadhtml (after the file is read into memory)
This does not work:
B4X: 
Dim HTML As String = File.GetText(File.DirAssets, "test.htm") 
HtmlImageView.LoadHtml(HTML)
This does work:
B4X: 
WebView1.LoadUrl("file://folder/temp.htm")
does work. (folder is replaced for my actual location on the android, basically File.DirRootExternal and another folder)

Thanks in advance for your help.
Rusty
 

Attachments

  • Screenshot_2012-11-25-09-24-30.jpg
    Screenshot_2012-11-25-09-24-30.jpg
    34.7 KB · Views: 250
Upvote 0
mc73

mc73

Well-Known Member
Licensed User
Longtime User
Tried an example inserting the paragraph you gave, works for me. Can you give it a try?
 

Attachments

  • awebviewtest.zip
    5.8 KB · Views: 227
Upvote 0
Rusty

Rusty

Well-Known Member
Licensed User
Longtime User
Thanks for the response.
However, the HTML file you tested with does not contain the critical items.
Try it with the HTML file I attached to the previous post.
Regards,
Rusty
 
Upvote 0
Rusty

Rusty

Well-Known Member
Licensed User
Longtime User
Maybe you can cut and paste this. Sorry for the inconvenience, but the file fails to attach as invalid.

B4X: 
<html xmlns:v="urn:schemas-microsoft-com:vml"
xmlns:o="urn:schemas-microsoft-com:office:office"
xmlns:w="urn:schemas-microsoft-com:office:word"
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml"
xmlns="http://www.w3.org/TR/REC-html40">

<head>
<meta http-equiv=Content-Type content="text/html; charset=windows-1252">
<meta name=ProgId content=Word.Document>
<meta name=Generator content="Microsoft Word 14">
<meta name=Originator content="Microsoft Word 14">
<link rel=File-List href="APGAR_Consent%202012-2013%20new_files/filelist.xml">
<link rel=Edit-Time-Data
href="APGAR_Consent%202012-2013%20new_files/editdata.mso">
<title>General Information About This Study and the Researchers</title>

</head>

<body lang=EN-US style='tab-interval:.5in'>

<div class=WordSection1>
<p class=MsoNormal align=center style='text-align:center;tab-stops:1.0in'><i
style='mso-bidi-font-style:normal'><span style='font-size:16.0pt;color:blue'><o:p>&nbsp;</o:p></span></i></p>

<p class=MsoNormal><b style='mso-bidi-font-weight:normal'>Study Title:<span
style='mso-spacerun:yes'>  </span></b>test test test, a
randomized controlled trial</p>

<p class=MsoNormal><i style='mso-bidi-font-style:normal'><o:p>&nbsp;</o:p></i></p>

<p class=MsoNormal><b style='mso-bidi-font-weight:normal'>More test stuff<o:p></o:p></b></p>

<p class=MsoNormal><b style='mso-bidi-font-weight:normal'><o:p>&nbsp;</o:p></b></p>

<p class=MsoNormal style='tab-stops:right 5.5in'><b style='mso-bidi-font-weight:
normal'>Name of Principal Investigator on this Study:<span
style='mso-spacerun:yes'>  </span></b>NAME </p>

<p class=MsoNormal style='tab-stops:right 5.5in'><o:p>&nbsp;</o:p></p>

<p class=MsoNormal style='margin-bottom:6.0pt'><b style='mso-bidi-font-weight:
normal'>MORE test stuff: <span style='mso-spacerun:yes'>   </span><a
name=facilityname><span style='mso-spacerun:yes'>  </span></a><span
style='mso-spacerun:yes'>    </span><o:p></o:p></b></p>

<p class=MsoNormal style='margin-bottom:12.0pt'>Please sign and date to show
that you have read all of the above guidelines.<span style='mso-spacerun:yes'> 
</span>Please do not sign unless you have read this entire consent form.<span
style='mso-spacerun:yes'>  </span>If you do not want to sign, you don’t have
to, but if you don’t you cannot participate in this research study. </p>

<p class=MsoNormal style='margin-top:0in;margin-right:-40.3pt;margin-bottom:
12.0pt;margin-left:0in;tab-stops:112.5pt 5.5in'><b style='mso-bidi-font-weight:
normal'><u><span style='font-size:14.0pt'>Signature for study participation<o:p></o:p></span></u></b></p>

<p class=MsoNormal style='margin-right:-40.5pt;line-height:13.0pt;mso-line-height-rule:
exactly;tab-stops:112.5pt 5.5in'><a name=surveydate>______________</a><span
style='mso-spacerun:yes'>  </span><span style='mso-tab-count:1'>                                      </span><a
name=Q027A000>___________________________________________________</a><span
style='mso-spacerun:yes'>    </span></p>

<p class=MsoNormal style='margin-right:-49.5pt;line-height:13.0pt;mso-line-height-rule:
exactly;tab-stops:112.5pt 5.5in 5.75in'>(Date / Time)<span style='mso-tab-count:
1'>               </span>(Printed Name of Participant) – Respondent #<a
name=respondent><span style='mso-tab-count:1'>                     </span></a></p>

</div>

</body>

</html>
 
Upvote 0
mc73

mc73

Well-Known Member
Licensed User
Longtime User
I think I don't have to.

I see a meta, tagging windows-1252, where it should use utf-8. Line 7.
 
Upvote 0
Rusty

Rusty

Well-Known Member
Licensed User
Longtime User
I replaced the windows-1252 with utf-8.
no change.

My line of the html looks like this:
B4X: 
<meta http-equiv=Content-Type content="text/html; charset=utf-8">

I appreciate your help. Any advice on the above?
Rusty
 
Upvote 0
Rusty

Rusty

Well-Known Member
Licensed User
Longtime User
MC73,
Thanks so much for all your observations on the UTF-8 issue.
I have discovered that with Word 2010, you can alter the saving of the DOC to HTML and can specify the encoding (as well as eliminate some of the bloat created by Word)

When using Save As;
  • select save as Web Page, Filtered (*.htm; *.html);
  • click TOOLS (next to the Save button)
  • click Web Options
  • click Encoding
  • click Save this document as: Pick Unicode (UTF-8)
After saving, the encoding appears:
B4X: 
<meta http-equiv=Content-Type content="text/html; charset=utf-8">
This will save the document with the proper encoding.
After doing this, my issue with LoadHTML and LoadURL go away.

Regards,
Rusty
 
Upvote 0
mc73

mc73

Well-Known Member
Licensed User
Longtime User
I've not much experience with msOffice, otherwise I would advice you better and earlier :)
Usually, I create my rather basic php and html with jEdit.
 
Upvote 0
Rusty

Rusty

Well-Known Member
Licensed User
Longtime User
Thanks.
I don't have this kind of experience either.
My clients insist upon Microsoft Office products, so I am learning.
Again, thanks for your input.
Rusty
 
Upvote 0
You must log in or register to reply here.

Similar Threads

Dave O
  • Article
Android Code Snippet minor fix to GoogleDrive API for folder names with apostrophes
Replies
0
Views
1K
Dave O
Dave O
MichalK73
Wish Brackets and apostrophes.
Replies
0
Views
589
MichalK73
MichalK73
M
  • Article
Android Code Snippet How To Escape The Apostrophe (single quote) And % Sign In a SQLite Query
Replies
7
Views
17K
ac9ts
ac9ts
D
  • Question
Android Question apostrophe in sql
Replies
3
Views
2K
Erel
Erel
P
  • Question
SQLite and Apostrophes - help
Replies
4
Views
3K
Erel
Erel
Top