Request: Bulk Library Access for Offline AI Indexing & Tooling

T

Tim Chapman

Active Member
Licensed User
Longtime User
Hi everyone (and Erel),

I am opening this thread per Erel's suggestion to discuss a resource that could be valuable for the community, specifically for those of us building AI coding agents or offline tooling.

The Project: I am currently developing a local, offline AI coding agent (running on NVIDIA Jetson hardware) specialized for B4X. To make the AI effective, it needs to ingest the API signatures (Public Subs, Properties, Events) of the community libraries so it can write accurate code without hallucinating methods that don't exist.

The Problem: While the libraries_mapping.json (from the B4X_Forum_Resources repo) is an excellent map of what libraries exist, the actual API definitions are locked inside the .b4xlib (Source) and .jar/.xml files scattered across thousands of forum threads.

To build a comprehensive index, I need to parse the actual library files. However, automatically downloading 700+ attachments from the forum is not a viable option, as it would likely trigger CloudFlare protections and put unnecessary load on the server.

The Request: Is there a centralized archive or a "Master ZIP" of the standard community libraries available for download?

Alternatively, could the AnywhereSoftware/B4X_Forum_Resources GitHub repository be updated to include the actual .b4xlib and .xml files (rather than just the forum metadata)?

Having a single source to download the current ecosystem would allow me (and others) to build powerful, context-aware AI tools for B4X completely offline, without risking IP bans or degrading forum performance.

Thank you for considering this!
 
Erel

Erel

B4X founder
Staff member
Licensed User
Longtime User
T

Tim Chapman

Active Member
Licensed User
Longtime User
Hi Erel,

Thank you for the link to the XML Generation Tool.

I understand that this tool can parse a .b4xlib and generate the XML documentation I need. That solves the "Translation" part of the problem for b4xlibs perfectly.

However, it does not solve the "Acquisition" part of the problem, which applies to both library types:

1. The .b4xlib Challenge: To use your tool, I first need to possess the .b4xlib files. Currently, they exist only as attachments scattered across hundreds of forum threads.

2. The Standard Library (.jar + .xml) Challenge: While Java libraries do have XML documentation, they are also scattered across the forum. To index them, I currently have to find and download each one individually.

The Catch-22: To build a complete index of the ecosystem, I would need to write a script to download 700+ attachments (both .b4xlib and .jar/.xml) from the forum. Doing so would trigger CloudFlare and ban my IP, which I strictly want to avoid to respect your server infrastructure.

The Request: Is there a way to obtain a single "Master ZIP" that contains the current collection of community libraries (both .b4xlib and .jar types)?

If I had that archive, I could:

  1. Run the XML Generation tool locally on the .b4xlibs.
  2. Ingest the existing .xml files from the Java libraries.
  3. Build the entire AI database offline, without ever touching the live server or risking a ban.
Thank you for considering this!
 
T

Tim Chapman

Active Member
Licensed User
Longtime User
I want to share what I have in mind for this. If I can get the data I have requested, I will spend the time and money to get the dataset into the correct form to fine tune a Qwen 2.5 model then give it back to the community. I have a path forward on this. I just need the data. I need all that I can get for the standard and b4xlib libraries as well as the entire forum if possible. Some of that will be too old, but I think 10 years of forum will cover the standard libraries as well as the newer ones. If I am off in my understanding of any of this, please feel free to say so. You certainly won't offend me. I want to get this right the first time.
 
Erel

Erel

B4X founder
Staff member
Licensed User
Longtime User
Tim Chapman said:
The Request: Is there a way to obtain a single "Master ZIP" that contains the current collection of community libraries (both .b4xlib and .jar types)?
Click to expand...
No such thing (currently) available.

Note that you don't need to download the files from the forum. The files you need are available on github. You can download the complete repository.
 
T

Tim Chapman

Active Member
Licensed User
Longtime User
Hi Erel,
Thank you again for the reply. Is there a complete list of libraries available anywhere? You shared a spreadsheet with me a while back, but it seems to be incomplete now. I have the repository from Github and am getting everything out of it that I can. Of the 3400 libraries I have been able to find in that and all of my resources, the attached spreadsheet shows 294 that I don't have any files for. They were in the spreadsheet but I don't dare to try to download that many (using my code) for fear of getting banned by CloudFlare.
 

Attachments

  • B4X-Libraries-Updated.xlsx
    320.1 KB · Views: 53
T

Tim Chapman

Active Member
Licensed User
Longtime User
I have narrowed it down to 53 libraries that I know of that are not in the github repository.
They are at the top left of the attached spreadsheet and are highlighted in yellow.
Will CloudFlare ban me if I download 53 files?
 

Attachments

  • B4X-Libraries-Updated.xlsx
    326.3 KB · Views: 83
E

emexes

Expert
Licensed User
Longtime User
Tim Chapman said:
Will CloudFlare ban me if I download 53 files?
Click to expand...

I have bulk downloaded hundreds of attachments from the forum

(with a voluntary and arbitrarily chosen 5 second gap between downloads ie throttled to maximum of 12 per minute equivalent to 3.6 MB per minute)

without being banned.
 
T

Tim Chapman

Active Member
Licensed User
Longtime User
emexes said:
I have bulk downloaded hundreds of attachments from the forum

(with a voluntary and arbitrarily chosen 5 second gap between downloads ie throttled to maximum of 12 per minute equivalent to 3.6 MB per minute)

without being banned.
Click to expand...

You wouldn't happen to have the 53 libraries that I am missing would you?
 
T

Tim Chapman

Active Member
Licensed User
Longtime User
I think I have hit upon a better solution than training a model to do this. A database that the AI semantically searches will be able to used with any model and will be able to be automatically updated when the github repository is updated. I have already got this well in hand. The models will improve as time goes on which would require training new ones regularly at great expense. I will post the code for my database system when it is done. I am working on it daily so it will be soon.
 
E

emexes

Expert
Licensed User
Longtime User
Tim Chapman said:
You wouldn't happen to have the 53 libraries that I am missing would you?
Click to expand...

Not the first few that I checked. But I was checking for b4xlibs whereas eg CLVBackwards is apparently a class file (CLVBackwards.bas) inside a .zip attachment:

Erel said:
The class is inside the cross platform example project.
Click to expand...

I could probably relatively easily find all .bas files inside forum .zip attachment root directories, but not today, more like next Tuesday.

Also involves scanning further back through the forum cf b4xlibs only need to scan back to late 2018.

[new feature] b4xlib - a new type of library

As B4X matures, more and more reusable components are written in B4X directly. In most cases it is simpler to implement the logic in B4X directly and the code can be cross platform. Especially with the relatively new XUI library. Standard libraries will never be cross platform as they are...
www.b4x.com www.b4x.com
 
Last edited:
T

Tim Chapman

Active Member
Licensed User
Longtime User
I don't need you to find files in the github repository. I have done that quite well. The 53 I am missing are not in the github forum in any form. I have searched multiple ways for them. They are at the top of the spreadsheet in yellow. Note that I am already also unzipping and extracting the xml from the b4xlibs as well. All of that is going into the database as well as code snippets, example code, documentation booklets, etc. I am trying to get every morsel of data that is relevant into the database. It is going well. I just can't find the 53 missing libraries without scraping.
 
E

emexes

Expert
Licensed User
Longtime User
You must log in or register to reply here.
Top