B4J Question ETL with B4J

[imbault]

Hi,

Does someone made a kind of ETL tool with B4J?

Thanks

 

[Erel]

What are your data sources?

 

[imbault]

For this project CSV input and Output (or maybe SqlServer)

Usually I use Talend or SSIS, but if there is something with B4J, I'd be very pleased

 

[b4auser1]

For this project CSV input and Output (or maybe SqlServer)

https://www.b4x.com/b4j/help/jstringutils.html#stringutils_loadcsv2 -> Then https://www.b4x.com/b4j/help/jsql.html + JDBC for SQL Server.
Restriction of architcture:

Loads a CSV file and stores it in a list of string arrays.

- RAM can be bottleneck for large csv files.

 

[imbault]

My primary thread was about an ETL, not primitive functions like loadcsv...

I'm sure, there is something to build like ABMaterial...

Same thing for a reporting FW

 

[b4auser1]

My primary thread was about an ETL, not primitive functions like loadcsv...

In computing, Extract, Transform and Load (ETL) refers to a process in database usage and especially in data warehousing that:
Extracts data from homogeneous or heterogeneous data sources
Transforms the data for storing it in the proper format or structure for the purposes of querying and analysis
Loads it into the final target (database, more specifically, operational data store, data mart, or data warehouse)

There are two approaches to imlemement ETL process.
1) Use specialiazed solutions like MS SSIS, IBM InfoSphere DataStage, Informatica PowerCenter, Talend Open Studio for Data Integration and etc.
Using these tools is possible to design/develop the ETL process using GUI Designer without writing code.
2) Use any programming language like C, C++, Java, B4J to write code which will extract data from one data source, transform it and load to another data source.

Do I understand you

I'm sure, there is something to build like ABMaterial...

correctly that you are looking for specialiazed solution with GUI Designer, but developped in B4J ?

 

[imbault]

correctly that you are looking for specialiazed solution with GUI Designer, but developped in B4J ?

Right, something like that, many of my customers will pay a lot for that

A big one spent 2 year/man work rewriting all SSIS in Talend, for a bad result, so yes, right, it is always easy to begin a project with SSIS or Talend, then when you need to modify your datasource structures, or add something, those solutions begin to fall

Lately, another one took 2 experts for building a Talend pro solution, and still, it doesn't work

 

[b4auser1]

A big one spent 2 year/man work rewriting all SSIS in Talend, for a bad result, so yes, right, it is always easy to begin a project with SSIS or Talend, then when you need to modify your datasource structures, or add something, those solutions begin to fall

Lately, another one took 2 experts for building a Talend pro solution, and still, it doesn't work

I see. I worked a lot of with DB and with apps to transfer data from between DB. I developed for a company which I worked for, a couple of utilities in C++ to transfer data from ODBC to ODBC datasource. To avoid need to write code for each new ETL process in C++, these utilities interpreted simple declarative script in text file:
Something like this:
data source from 1, data source to 1, SQL statement to select data 1, SQL statement to insert/delete/update data 1
data source from 2, data source to 2, SQL statement to select data 2, SQL statement to insert/delete/update data 2
...
data source from 3, data source to 3, SQL statement to select data 3, SQL statement to insert/delete/update data 3

There were haundreds of batches to transfer data.

Then ETL was moved to IBM Data Stage.

I thought to rewrite such an utility with B4J and JDBC sources, but writing code in B4J is rather simple and flexible.
Now I prefer to write code in B4J instead of using declarative script as I described above.
I personally don't like GUI designer to generate code. From my point of view it's harder to support such a solution and experience of your customers confirms it.
It's only my own opinion .

 

[imbault]

I do agree, I prefer a script approach or a specific syntax than a GUI designer for that

 

[udg]

Hi, I'm new to the subject so my question will certainly be naive:
are you envisioning a system where natural language statements encompass the rules to apply to a given input file in order to produce a specified format output?