I may write up a quick RAG example for B4J and submit it to the forums. This is super super interesting.
You can actually send a ton of information to the LLM at one time. As long as you are okay with the big price, no need for fine-tuning:
4o for example, has a context window of 128K tokens (Roughly 500k characters) -- You can send up to 500k characters and get a response.
This level of context means: -- You can send your database structures (CREATE TABLE..., CREATE table...), chunked data describing your project, code, etc.. And still get a response. ----- Where the RAG part comes in -- If you have a large database/context (Or just want to save money and get a better response), you can use embedded chunk models (text-embedding-3-small, ada-02, etc..) -- This allows you to store entire tables in a vector array, and compare against the user's query to get the best, most related, chunks to send to your LLM. (ex. TOP 10 closest chunks)
One of the best use cases I have found is SQL generation/execution AI agent. Being able to connect your large database and ask questions such as:
1) What is my average ROI per project for the year?
2) Generate a table for each customer and yearly revenue.
3) Show me each employee with their total number of hours worked for 2025.
The next step would be creating an Agentic agent that can use multiple sources to create an efficient and accurate context to answer whatever question is being asked.
Your agentic agent has access to your other agents (SQL agent, vector search, etc) and outside tools (Google search, third party APIs)
The context starts out small, describing the baseline project, tools it is allowed to use, and the previous answers.
ChatGPT/Gemini (the websites), are not direct access to the LLMs either. They are essentially agents that access their LLM, with built up tools/context windows.
It would be really neat if B4X had an agentic agent built in. Be able to create "RAG" databases of our classes and modules, and an interface to allow AI to make simple code changes with our approval.