B4J Question How to Parse HTML file

atiaust · Oct 13, 2016

Hi All,

I have a small HTML file that I would like to extract data from, name address etc.

Client Title: Mr 
Client First Name: John 
Client Last Name: Smith 
Client Phone: 0400555000 
Client Email: johnS@yahoo.com.au 
Client Preferred Contact: Phone 
Pickup Unit Number:5 
Pickup Street Number: 10 
Pickup Street Name: Long Street 
Pickup Suburb: Sydney

What is the best way to parse the data so that I can add it to a database?

Any and all ideas gratefully accepted.

Thanks

Erel · Oct 13, 2016

You can use jTidy: https://www.b4x.com/android/forum/threads/jtidy-library-convert-html-pages-to-xml.27038/#content to convert the html to xml and then parse it with the XmlSax library.

atiaust · Oct 13, 2016

Thanks Erel,

After reading lots of posts I figured that was probably best.

I get the following error when I try to parse the .xml file.

B4X:

Dim t As Tidy
         t.Initialize
         t.Parse(File.OpenInput(dirData, filename), dirData, "temp.xml")
         Dim In As InputStream = File.OpenInput(dirData, "temp.xml")
             items.Initialize
               sax.Parse(In, "sax")

line 1 column 1 - Warning: missing <!DOCTYPE> declaration
line 35 column 1 - Warning: inserting missing 'title' element
InputStream: Document content looks like HTML 2.0
2 warnings, no errors were found!
[Fatal Error] :1:50: White spaces are required between publicId and systemId.
Error occurred on line: 158 (Main)
org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 50; White spaces are required between publicId and systemId.
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
at anywheresoftware.b4a.objects.SaxParser.parse(SaxParser.java:80)
at anywheresoftware.b4a.objects.SaxParser.Parse(SaxParser.java:73)
at ati.ccsemail.main._handlemessage(main.java:272)
at ati.ccsemail.main._pop_downloadcompleted(main.java:375)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at anywheresoftware.b4a.shell.Shell.runMethod(Shell.java:612)
at anywheresoftware.b4a.shell.Shell.raiseEventImpl(Shell.java:229)
at anywheresoftware.b4a.shell.Shell.raiseEvent(Shell.java:159)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at anywheresoftware.b4a.BA.raiseEvent2(BA.java:93)
at anywheresoftware.b4a.ShellBA.raiseEvent2(ShellBA.java:90)
at anywheresoftware.b4a.BA$4.run(BA.java:196)
at com.sun.javafx.application.PlatformImpl.lambda$null$173(PlatformImpl.java:295)
at java.security.AccessController.doPrivileged(Native Method)

Any ideas?

Thanks

Erel · Oct 13, 2016

What is the output of:

B4X:

Log(File.ReadString(dirData, "temp.xml"))

atiaust · Oct 13, 2016

Output from log.

line 35 column 1 - Warning: inserting missing 'title' element
InputStream: Document content looks like HTML 2.0
2 warnings, no errors were found!
<!DOCTYPE html PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html>
<head>
<meta name="generator" content=
"HTML Tidy for Java (vers. 2009-12-01), see jtidy.sourceforge.net">
<title></title>
</head>
<body>
Client Title: Mr 
Client First Name: Rob 
Client Last Name: Smith 
Client Phone: 0400000205 
Client Email: rsmith@tpg.com.au 
Client Preferred Contact: Email 
Pickup Unit Number: not required as only drop off
was selected 
Pickup Street Number: not required as only drop
off was selected 
Pickup Street Name: not required as only drop off
was selected 
Pickup Suburb: not required as only drop off was
selected
Dropoff Unit Number: 
Dropoff Street Number: 1 
Dropoff Street Name: Short Ave 
Dropoff Suburb: Blue Bay
Travel Details - One Way 
Pickup from/at/to Airport 
____________________________________________________
Departure Date: // (dd/mm/yyyy) 
Departure Time: : 
Departure Flight/Ship/Hotel Number/Name: 
Departure Flight Type: Domestic 
Number of Departing adults: 0 
Number of Departing Children: 0 
_____________________________________________________
Arrival Date: 23/November/2016 (dd/mm/yyyy) 
Arrival Time: 10:15 AM 
Arrival Flight/Ship/Hotel Number/Name: QF0144 
Arrival Flight Type: International 
Number of Arriving adults: 2 
Number of Arriving Children: 0 
_____________________________________________________
Additional Requirements: 
Special Requests/Comments:
</body>
</html>
[Fatal Error] :1:50: White spaces are required between publicId and systemId.
Error occurred on line: 159 (Main)
org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 50; White spaces are required between publicId and systemId.
at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1239)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
at anywheresoftware.b4a.objects.SaxParser.parse(SaxParser.java:80)
at anywheresoftware.b4a.objects.SaxParser.Parse(SaxParser.java:73)
at ati.ccsemail.main._handlemessage(main.java:275)
at ati.ccsemail.main._pop_downloadcompleted(main.java:378)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

etc......

Erel · Oct 13, 2016

Note that the spoiler buttons only make it more difficult to read your posts.

I've made a fix to JTidy library: https://www.b4x.com/android/forum/threads/jtidy-library-convert-html-pages-to-xml.27038/

Please use it. It will fix this issue.

atiaust · Oct 13, 2016

Thanks Erel,

Couldn't figure out what I was doing wrong.

Sorry about the spoilers...

inakigarm · Oct 13, 2016

For parsing Html documents I use jsoup library (quick and easy); your html document is easy to parse without any external library (there's no html tags on body content beyond and )

atiaust · Oct 13, 2016

Thanks Inakigarm,
I tried to parse the file using the suggestion from Erel but I think the items in the resulting xml file don't have tags to parse.

I read in the posts not to use the jsoup library, but I was a little confused as to why.

Attached is the html & xml files.

Appreciate if you can have a look at them and advise.

Thanks

inakigarm · Oct 13, 2016

One question first, is the html structure always the same ?? (html tags doesn't change?) Because if it is, I think it's easier to extract the page content with string operations (As there're one or more strong tags in html source is not automatic to parse with jsoup)

If it's always the same structure, my option will be parsing the page with string functions:

1. Read 0.html from disk file into a list (each entry of the list will be a line -string)
2. Create a map with a key , value pair moving into each line and using substrings methods
3. Get the map and save to disk, write to DB, etc..

B4X:

Sub AppStart (Args() As String)

    Dim lstsource As List
    lstsource.Initialize: lstsource=File.ReadList(File.DirApp,"0.html")
    
    Dim map1 As Map
   
    map1=Createlistmap(lstsource)
   
    For i=0 To map1.Size-1
        Log(map1.GetKeyAt(i) & " " & map1.GetValueAt(i))
    Next
   
   
End Sub

B4X:

Sub Createlistmap (l As List) As Map   
   
    Dim str As String
    Dim lsttmp,lstresults As List
    Dim m As Map
   
    lsttmp.Initialize
    lstresults.Initialize
    m.Initialize
   
    For i=0 To l.Size-1
        Dim strtmp,strtmp1,strtmp2 As String
        Dim count As Int
       
        strtmp=l.Get(i)
        count=StringCount(strtmp,"<strong>",True)
        Select Case count
            Case 0
                Log("no")
            Case 1
                strtmp1=strtmp.SubString2(strtmp.IndexOf("<strong>")+8,strtmp.IndexOf("</strong>"))
                If strtmp.Contains("br") Then
                    strtmp2=strtmp.SubString2(strtmp.LastindexOf("</strong>")+9,strtmp.LastIndexOf("<br />"))
                Else
                    strtmp2=strtmp.SubString2(strtmp.LastindexOf("</strong>")+9,strtmp.LastIndexOf("</p"))
                End If
            Case 2
               
                strtmp1=strtmp.SubString2(strtmp.IndexOf("<strong>")+8,strtmp.IndexOf("</strong>"))
                strtmp2=strtmp.SubString2(strtmp.IndexOf("</strong>")+9,strtmp.LastIndexOf("<strong>"))
                strtmp1=strtmp1 & " " & strtmp2
                strtmp2=strtmp.SubString2(strtmp.lastindexOf("<strong>")+8,strtmp.LastIndexOf("</strong>"))
                strtmp1=strtmp1 & " " & strtmp2
                If strtmp.Contains("br") Then
                    strtmp2=strtmp.SubString2(strtmp.LastindexOf("</strong>")+9,strtmp.LastIndexOf("<br />"))
                Else
                    strtmp2=strtmp.SubString2(strtmp.LastindexOf("</strong>")+9,strtmp.LastIndexOf("</p"))
                End If
            Case 3
                'Other cases
               
               
        End Select
       
        Log(strtmp1 & " " &strtmp2)
        m.Put(strtmp1,strtmp2)
   
    Next
   
    Return m
   
End Sub

'Sub from user stevel05
Sub StringCount(StringToSearch As String,TargetStr As String,IgnoreCase As Boolean) As Int
    If IgnoreCase Then
        StringToSearch = StringToSearch.ToLowerCase
        TargetStr = TargetStr.ToLowerCase
    End If

    Return (StringToSearch.Length - StringToSearch.Replace(TargetStr,"").Length) / TargetStr.Length

End Sub

Erel · Oct 13, 2016

Regex to the rescue:

B4X:

Sub AppStart (Form1 As Form, Args() As String)
   Dim m As Matcher = Regex.Matcher("<strong>([^<]+)</strong>\s*([^<]+)<br />", File.ReadString(File.DirAssets, "0.html"))
   Do While m.Find
     Dim key As String = m.Group(1)
     Dim value As String = m.Group(2)
     Log(key & "=" & value)
   Loop
End Sub

inakigarm · Oct 13, 2016

Look first for Regex but syntax is too complex for me !!

Erel · Oct 13, 2016

The strange component is [^<]+. This means find any character until you reach <

atiaust · Oct 13, 2016

Thank you both for your advise.

The html file is generated from the body of an email sent from an online web enquiry.
The file format will always be constant.

Regex looks by far the simplest now that I can see how to apply.

rwblinn · Oct 13, 2016

Hi,

thought about converting the content to a table, then parse to xml then build a map with fields.
See attached try using jTidy with output.

Note: Corrected a typo spotted in the source. Attached updated.

atiaust · Oct 13, 2016

Another excellent suggestion.

Thanks Rob.

B4J Question How to Parse HTML file

atiaust

Active Member

Erel

B4X founder

atiaust

Active Member

Erel

B4X founder

atiaust

Active Member

Erel

B4X founder

atiaust

Active Member

inakigarm

Well-Known Member

atiaust

Active Member

Attachments

inakigarm

Well-Known Member

Erel

B4X founder

inakigarm

Well-Known Member

Erel

B4X founder

atiaust

Active Member

rwblinn

Well-Known Member

Attachments

atiaust

Active Member

Similar Threads