B4J Question How to read file and data elements when file layout is provided with data file at the time of the run?

Kris.D said:

Hi,
I'm looking for idea how to solve issue of reading file and data elements not knowing how this file looks prior to the run.
If I know my data prior to compile I think I could create Custom Type object or something like that but what do I do in this case?

Click to expand...

Type FieldDescriptorType( _
    FNumber As Int, _
    FName As String, _
    FType As String, _
    FLength As String, _
    DecimalPlaces As Int _
)

Dim RecordLength As Int = 0
For Each fd As FieldDescriptorType In FieldDescriptorMap    'doesn't matter which order they're summed in
    RecordLength = RecordLength + fd.FLength
Next

emexes said:

I would have two lists or arrays that mirror your two files. Arrays have the advantage that the index is known, eg if you have an array of five descriptors for five fields, then the first field is index 0, the second field is index 1, ... the last field is index 4 (= the five fields - 1).

Or you could use a map, with the field number as the key, eg:

B4X:

Type FieldDescriptorType( _
    FNumber As Int, _
    FName As String, _
    FType As String, _
    FLength As String, _
    DecimalPlaces As Int _
)

and then read the LAYOUT file line by line (skipping the header line, or using it to confirm that the file has the expected columns), and fill in the descriptor fields accordingly, including implied lengths eg:

- date is 8 or 10 or 11 characters (yyyymmdd, dd/mm/yy, dd-mmm-yyyy ?)
- 3.2 decimal is 6 characters, or maybe 7 characters if the +/- sign is not included in the 3.

Once you've filled out the FieldDescriptor Map, then (i) you know how many fields are in each record (FieldDescriptor.Length) and (ii) you can calculate the fixed record length of the DATA file, by summing the field lengths:

B4X:

Dim RecordLength As Int = 0
For Each fd As FieldDescriptorType In FieldDescriptorMap    'doesn't matter which order they're summed in
    RecordLength = RecordLength + fd.FLength
Next

And then "all you have to do is" stroll through the DATA file reading RecordLength-byte blocks and splitting them up according to the FLengths, interpreting each field according to its FType and adding them to an array, and then adding the completed array to a List of records.

On a related note: old DBF files have a structure similar to having your LAYOUT + DATA files but as a single file, and can be read by most spreadsheet programs.

Click to expand...

emexes said:

stroll through the DATA file reading RecordLength-byte blocks and splitting them up according to the FLengths

Click to expand...

For Each fd As FieldDescriptorType In FieldDescriptorMap    'doesn't matter which order they're summed in

Dim NumFields As Int = FieldDescriptorMap.Length
Dim FieldInfo(NumFields) as FieldDescriptorType
For FieldNumber = 1 to NumFields
    FieldInfo(FieldNumber - 1) = FieldDescriptorMap.Get(FieldNumber)
Next

Dim NumFields As Int = FieldDescriptorMap.Length
Dim FieldInfo(NumFields) as FieldDescriptorType
For Each fd As FieldDescriptorType In FieldDescriptorMap    'doesn't matter which order they're copied in
    FieldInfo(fd.FNumber - 1) = fd    'array index = field number - 1
Next

emexes said:

Something like this will split a fixed-length record up into fields:

lol I started writing just the RecordToFields function, but one thing led to another and before I knew what was happening... :

B4X:

Sub Process_Globals
   
Type FieldDescriptorType( _
    FNumber As Int, _
    FName As String, _
    FType As String, _
    FLength As String, _
    DecimalPlaces As Int _
)
   
End Sub


Sub RecordToFields(RecordString As String, FieldInfo() As FieldDescriptorType) As String()

    Dim NumFields As Int =  FieldInfo.Length
 
    Dim FieldStrings(NumFields) As String
   
    Dim StartPos As Int = 0
    For FieldIndex = 0 To NumFields - 1
        Dim EndPos As Int = StartPos + FieldInfo(FieldIndex).FLength
        If EndPos > RecordString.Length Then    'ideally would have check RecordString length is correct before trying to split into fields
            EndPos = RecordString.Length
        End If
        FieldStrings(FieldIndex) = RecordString.SubString2(StartPos, EndPos)    '.Trim
        StartPos = EndPos
    Next
   
    Return FieldStrings
   
End Sub


Sub StringArrayToSV(S() As String, Delimiter As String, Separator As String) As String
   
    Dim sb As StringBuilder
    sb.Initialize
   
    For I = 0 To S.Length - 1
        If I <> 0 Then
            sb.Append(Separator)
        End If
        sb.Append(Delimiter & S(I) & Delimiter)
    Next
   
    Return sb.ToString
   
End Sub


Sub StringArrayToCSV(S() As String) As String
   
    Return StringArrayToSV(S, """", ", ")
   
End Sub


Sub AppStart (Args() As String)
   
    Log("Hello world!!!")
   
    'test sample data:
   
    Dim FieldInfo(3) As FieldDescriptorType
   
    FieldInfo(0).FLength = 5
    FieldInfo(1).FLength = 10
    FieldInfo(2).FLength = 20
   
    'three different was of doing the same thing:
   
    Dim F() As String = RecordToFields("abcdefghijklmnopqrstuvwxyz123456789", FieldInfo)
    Log(StringArrayToCSV(F))
   
    Dim RS As String = "[F1=][FIELD 2=][FIELD THREE=======]junk"
    Dim F() As String = RecordToFields(RS, FieldInfo)
    Dim CSV As String = StringArrayToCSV(F)
    Log(CSV)
   
    Dim RS As String = "FRED FLINTSTONEBEDROCK             junk"
    Log(StringArrayToCSV( RecordToFields(RS, FieldInfo) ))
   
End Sub

Log output:

Waiting for debugger to connect...
Program started.
Hello world!!!
"abcde", "fghijklmno", "pqrstuvwxyz123456789"
"[F1=]", "[FIELD 2=]", "[FIELD THREE=======]"
"FRED ", "FLINTSTONE", "BEDROCK             "

Click to expand...