Android Question PDF to SQLITE

AlpVir said:

I have a number of PDF files.
Each file has a few pages (from 1 to a few dozen).
Each page contains one, two or four tables (not three or five or more).
Each page is characterized by:
1) a header ("class" field) at the top center;
2) a table with:
a) 1+6 columns ("Monday", "Tuesday", "Wednesday", "Thursday", "Friday" and "Saturday"); are the days of the week ("lunedi", "martedi", "mercoledi","giovedi", "venerdi" e "sabato");
b) from 5 to 9 lines ("hour" field)
I would like to store the contents of each cell in a sqlite database having in following fields :
1) text - class
2) numeric - day (from 1 to 6)
3) numeric - hour (from 1 to 8)
4) text - contents of the cell.
In the attached example (assuming you have a single PDF file with single page and two table) the database would contain 65 records (33 with the class field set to "1A CA" and 32 with the class field set to "1A MME")
rec n. 1 : "1A CA";1;1;"TECNOLOGIA RAPP - GRAFICA, PALUMB"
rec n. 2 : "1A CA";2;1;"TECNOLOGIA RAPP - GRAFICA, PALUMB"
rec n. 3 : "1A CA";3;1;"ITALIANO E STORIA - PISANO"
rec n. 4 : "1A CA";4;1;"GEOGRAFIA- CHISU"
etc
How would you suggest we act given that a traditional OCR would most likely not obtain the desired result ?
Thanks for the attention.

Click to expand...

This is a very complex problem. There are multiple master dissertations about this even.
An easy analogy: I have 5000 puzzle-pieces, all of them are perfectly square and could fit anywhere. Some of them have pieces of lines on them, some of them have snippets of text.
However, that does not mean it can't be done. It'll just take work.
General approach:

1. use iText (specifically IEventListener) to get information on all rendering events for every page
2. select those rendering events that make sense for your application. PathRenderInfo and TextRenderInfo.
3. Events in a pdf do not need to appear in order according to the spec. Solve this problem by implementing a comparator over IEventData. This comparator should sort according to reading order. This implies you might have to implement some basic language detection, since not every language reads left-to-right.
4. Once sorted, you can now start clustering items together according to any of the various heuristics you find in literature. For instance, two characters can be grouped into a snippet of text if they follow each other in the sorted list of events (meaning they appear next to each other in reading order), if the y-position does not differ too much (subscript and superscript might screw with this), and if the x-position does not differ too much (kerning).
5. Continue clustering characters until you have formed words
6. Assuming you have formed words, use similar algorithm to form words into lines. Use PathRenderInfo to withhold merging words if they intersect with a line.
7. Assuming you have managed to create lines, now look for tables. One possible approach is apply a horizontal and vertical projection. And look for those sub-areas in the page that (when projected) show a grid-like structure.

This high-level approach should make it painfully obvious why this is not a widely available thing. It's very hard to implement. It requires domain-knowledge of both PDF, fonts, and machine-learning.
If you are ok with commercial solutions, try out pdf2Data. It's an iText add-on that features this exact functionality.
http://itextpdf.com/itext7/pdf2Data

Click to expand...