Android Question Special characters in Text files .. again ...

[GiovanniPolese]

Hi. I have already posted a question on the subject of special chars read from Text file. The common answer to my question of that time and others on same topic was: write the File with UTF8 codes, not using NotePad (who uses Ansi) etc. etc. This helps, of course, if we can write the file.. BUT .. what about the situation in which we cannot write the file and we have to deal with it "as is"? In particular, I am talking about dxf Files, written by Autocad (so not by the first guy on the road..). Dxf files are text files. If I don't miss something, I can only use variables of string type to process ithem Therefore, when I read a string, i can find special characters (btw one of my most hated features in my professional life, sorry). Then I have to display this text on a ListBox in B4A, for example. Of course special chars are displayed as funny characters. Has this problem a solution? Maybe there is a smart conversion which I don't know...
Thanks in advance

 

[Erel]

1. You need to know the file encoding. You can use Notepad++ to help you find it.
2. Open the file with TextReader. It allows specifying the encoding.

Tip: in B4i there is no TextReader and instead you should use File.ReadString2.

 

[GiovanniPolese]

Hi. Thanks. Dxf files are evidently written with Ansi encoding. I know it. Moreover, I have one of over one Gb size.. I know the process of using Notepad to convert. Even normal Notepad allows saving in UTF8. But I cannot ask to the user to do it. The file is as is, unluckily. A string is, for example:

MODELAÇÃO$0$XREF$0$www.dwgautocad.com

Unluckily things are so.

 

[GiovanniPolese]

By the way I confirm, if necessary, that writing the file with Notepad and UTF8 encoding the problem is resolved. As I said, this could not be possible. The problem is that we don't have something to read a string form a txt file, chosing the encoding (that can be assumed to be known) . Alternatively, I was thinking to read the file using File.ReadBytes.. but it seems that I have no function to read a line of the text file. All functions read the whole file, and if its huge, it could be impossible... I hope to miss something.. Thanks..

 

[Erel]

I hope to miss something

Yes. I will write again:

1. You need to know the file encoding. You can use Notepad++ to help you find it.
2. Open the file with TextReader. It allows specifying the encoding.

Ansi is probably Windows-1252 encoding.

I know the process of using Notepad to convert.

Ok. It has nothing to do with what I wrote.

 

[emexes]

Alternatively, I was thinking to read the file using File.ReadBytes.. but it seems that I have no function to read a line of the text file.

If you can read a set number of bytes, then is no problem to collect those chunks into lines, and I'd be surprised if there isn't already public code to do that.

Lol, having said that: I'm going to look for it now, and I'm mildly worried I'm in for a surprise. ?

 

[GiovanniPolese]

Sorry Erel and thanks for your patience. The speech about Notepad++ (or not) misleaded me. It was a further help, to discover the encoding, but I didn't care about, because in my case, things are simpler. I was simply missing that there is a TextReader initiliazation (Initialize2) allowing to choose the encoding. That's it. Your indication of Windows-1252 works. So, for reading Autocad Dxf it is necessary to use:

Dim tr as TextReader
tr.Initialize2(File.OpenInput(DirName, FIleName ),"Windows-1252")

Thanks

 

[GiovanniPolese]

If you can read a set number of bytes, then is no problem to collect those chunks into lines, and I'd be surprised if there isn't already public code to do that.

Lol, having said that: I'm going to look for it now, and I'm mildly worried I'm in for a surprise. ?

Hi, your message arrived while I was answering to Erel. As I wrote, I was simply missing the Initialize2 of TextReader...

 

[emexes]

Sorry Erel and thanks for your patience.

Dim tr as TextReader

Lol you beat me to it, I was about to point you towards:

Another option:

Dim tr As TextReader
tr.Initialize(File.OpenInput(...))
Dim st As String = tr.ReadLine
Do While st <> Null
   Log(st)
   If (st = "abc") Then Exit
   st = tr.ReadLine
Loop

 

[emexes]

was simply missing the Initialize2 of TextReader...

I was mildly surprised that DXF files use Windows-1252 encoding, given that DXF format predates Windows.

 

[GiovanniPolese]

I was mildly surprised that DXF files use Windows-1252 encoding, given that DXF format predates Windows.

I just reported what I see. Now special chars are correctly treated..

 

[Sandman]

Now that the issue has been solved: I'm curious, what are you doing with the dxf files?

 

[GiovanniPolese]

I am doing an App that manages dxf files drawn over an OrtoPhoto. This latter is a BigTiff file, that I "tiled" with a program of mine. This App may be used "just like GoogleMaps", but having a more precise positioning over the OrtoPhoto, which has (theoretical) precision of one pixel for one centimeter. Gps data are transofrmed in cartographic data, to be shown on the OrtoPhoto. I have just sent a short video to my partners. I attach a link for downloading it here, for your curiosity. The resolution is low. On the tablet it is much nicer..

www.galileo2000.it/download/show2.mp4

 

[Sandman]

I've heard dxf is a somewhat complex format (I believe the phrase used to describe it was "spaghetti format from hell") that also can reference other external dxf files. And I see in your video that you have a button for layers. Do you handle all of dxf or a smaller subset of its features?

 

[GiovanniPolese]

I just show the dxf and allow to See/Hide each Layer and change its color. Dxf actually has no "features", it is just the file written by Autocad, which has many features.. not at the level of a single programmer.. I have a Graphical editor in Windows, for Topography, which has some options similar to Autocad, but, clearly, only those related to Topography and Road/Railway design..

 

[Erel]

HugeImageView might be useful here: https://www.b4x.com/android/forum/threads/b4x-hugeimageview-show-very-large-images.132905/#content

 

[GiovanniPolese]

Thanks Erel, always interesting.. My images are 200.000 x 180.000. BigTif LZW compressed are over 30 Gb, not compressed over 100 Gb. I guess that only tiling resolves. Moreover resolution is one pixel per one centimeter... Anyway, the problem is resolved. I have over than 5000 tiles .. well managed with OpenGl. Unluckily now I am fighting with memory limits because, besides the tiles, I also have huge dxf, over 1.3 gb generating a million of lines etc... The Tablet has 4 Gb ram, but I must find some trick anyway. Ram is not enough. The tiles are stored in the Tablet, which has 64 Gb size. They use over 30 Gb of disk. I am thinking to add an additional sd, but , at this level, I always expect bad surprises (low access speed, Android permissions .. etc..) Thanks for reading.

 

[Brian Dean]

I've heard dxf is a somewhat complex format (I believe the phrase used to describe it was "spaghetti format from hell")

Many years ago DXF was a very simple format, and being in clear text was easy to work with. Then AutoCad added a lot of data related and other information which made things look more complicated, but DXF is frame based so you could still pick out the geometry frames and ignore the others. I have not worked with DXFs for a couple of years now so I don't know if that is still true, but I suspect that it is still possible to create a geometry only DXF that is still valid and is still simple.

 

[GiovanniPolese]

Many years ago DXF was a very simple format, and being in clear text was easy to work with. Then AutoCad added a lot of data related and other information which made things look more complicated, but DXF is frame based so you could still pick out the geometry frames and ignore the others. I have not worked with DXFs for a couple of years now so I don't know if that is still true, but I suspect that it is still possible to create a geometry only DXF that is still valid and is still simple.

Dxf is the only "legal" format for exchanging data, in my opinion. People use dwg even in official documents, but it is proprietary technology, and it should not be so. As a matter of fact we don't know what is written in dwg binaries. There is OpenDwg alliance, whose life is based on "hacking" the dwg. A not so clean situation. I read DXF for my uses and I adapt its data to my use. For example, in this App, i need to pass Lines, Polylines, Arcs, Texts etc. to OpenGl.. which has no Arcs or Splines, for example, and Text rendering is a not straightforward (done by lines.. too). The benefit of Dxf is to be readable. It reports also handles to Autocad drawing database, but I never worked with them and I don't know whether it is possible or not. I just use the data..

 

[GiovanniPolese]

It is rather easy to get data from Dxf, anyway, if this was the question. Just read tag and data, two lines by two lines. The tags say what the data is. I guess you know this.. I wrote this for curious people..