B4J Question Counting Tokens for OpenAI

T

Tim Chapman

Active Member
Licensed User
Longtime User
I need to be able to count tokens in a user's text submission (before it is submitted) to OpenAI API.

I have researched and found that TikToken is the best way to do that.
There are two TikToken libraries available at this time. One is in Python. The other in Node.js. Those are shown here:

OpenAI API: How do I count tokens before(!) I send an API request?

OpenAI's text models have a context length, e.g.: Curie has a context length of 2049 tokens. They provide max_tokens and stop parameters to control the length of the generated sequence. Therefore the
stackoverflow.com

What is the easiest way for me to get TikToken into B4J?

The closest I have seen is to compile a Node.js project into a jar file as shown here:

How to compile nodejs project into jar-Front-end Q&A-php.cn

Node.js is a very popular JavaScript runtime environment. Large web frameworks such as Vue.js and React Native run on Node.js. However, for Java programmers, compiling Node.js projects into Jar packages may be more convenient for integration into Java projects. In this article, we will introduce...
www.php.cn

This looks like the correct JS code to work with:

tiktoken/js at main · dqbd/tiktoken

JS port and JS/WASM bindings for openai/tiktoken. Contribute to dqbd/tiktoken development by creating an account on GitHub.
github.com

Any better answers anyone can see for this? Doing the conversion using the instructions above is a bit out of my league.
 
Solution
T
Thank you Don!
Sort by date Sort by votes

jkhazraji

Active Member
Licensed User
Longtime User
Send the text directly to the OpenAI API although it will consume your quota.
 
Upvote 0

DonManfred

Expert
Licensed User
Longtime User
Upvote 0
T

Tim Chapman

Active Member
Licensed User
Longtime User
Hi Don,

I have delayed replying to you until I try to use the jar file you recommended. I apologize for not thanking you sooner. I have had to take a detour and work on software for my business. It is now giving me trouble which I can't figure out. I am posting in the B4J forum about it. When I get back to my OpenAI project, I will try to use the jar file you suggested and will let you know what I find.

I posted my question here: https://www.b4x.com/android/forum/threads/combobox-not-showing-anything-when-i-click-on-it.163439/
 
Upvote 0

DonManfred

Expert
Licensed User
Longtime User
Tim Chapman said:
until I try to use the jar file you recommended
Click to expand...
you are in luck. i already wrote a small wrapper for it ?

Download the jar from the link in #4 and put it into the additional libs folder

Basically you need to have an
B4X: 
    Private tokkit As EncodingRegistry

and then use it
B4X: 
    tokkit.Initialize("") ' Eventname is not used
    tokkit.SetModelEncoding("GPT_3_5_TURBO")
    
    Dim tokens As List = tokkit.encode("This is a Test!")
    Log(tokens)
    
    Dim decoded As String = tokkit.decode(tokens)
    Log($"Decoded ${decoded}"$)
 

Attachments

  • jtokkitV0.02.zip
    3.3 KB · Views: 53
Upvote 0
T

Tim Chapman

Active Member
Licensed User
Longtime User
Thank you Don!
 
Upvote 0
Solution
You must log in or register to reply here.

Similar Threads

  • Question
B4J Question Get URI fragment portion
Replies
4
Views
843
Blueforcer
  • Question
B4J Question Sumup API again
Replies
5
Views
2K
udg
  • Question
B4J Question How to use httpJob to get rest Api Token
Replies
12
Views
2K
EnriqueGonzalez
  • Article
B4J Code Snippet [web]JSON Web Token (auth0/java-jwt)
Replies
7
Views
3K
Johan Hormaza
  • Solved
B4J Question How to make multiple http calls with same session cookies to pass CSRF token check
Replies
5
Views
1K
Erel
Menu
Log in
Register
Cookies are required to use this site. You must accept them to continue using the site. Learn more…