B4J Question Counting Tokens for OpenAI

Tim Chapman · Sep 30, 2024

I need to be able to count tokens in a user's text submission (before it is submitted) to OpenAI API.

I have researched and found that TikToken is the best way to do that.
There are two TikToken libraries available at this time. One is in Python. The other in Node.js. Those are shown here:

OpenAI API: How do I count tokens before(!) I send an API request?

OpenAI's text models have a context length, e.g.: Curie has a context length of 2049 tokens. They provide max_tokens and stop parameters to control the length of the generated sequence. Therefore the

stackoverflow.com

What is the easiest way for me to get TikToken into B4J?

The closest I have seen is to compile a Node.js project into a jar file as shown here:

How to compile nodejs project into jar-Front-end Q&A-php.cn

Node.js is a very popular JavaScript runtime environment. Large web frameworks such as Vue.js and React Native run on Node.js. However, for Java programmers, compiling Node.js projects into Jar packages may be more convenient for integration into Java projects. In this article, we will introduce...

www.php.cn

This looks like the correct JS code to work with:

tiktoken/js at main · dqbd/tiktoken

JS port and JS/WASM bindings for openai/tiktoken. Contribute to dqbd/tiktoken development by creating an account on GitHub.

github.com

Any better answers anyone can see for this? Doing the conversion using the instructions above is a bit out of my league.

Tim Chapman · Oct 5, 2024

DonManfred said:
you are in luck. i already wrote a small wrapper for it ?

Download the jar from the link in #4 and put it into the additional libs folder

Basically you need to have an

B4X:

Private tokkit As EncodingRegistry

and then use it

B4X:

tokkit.Initialize("") ' Eventname is not used tokkit.SetModelEncoding("GPT_3_5_TURBO") Dim tokens As List = tokkit.encode("This is a Test!") Log(tokens) Dim decoded As String = tokkit.decode(tokens) Log($"Decoded ${decoded}"$)

Thank you Don!

jkhazraji · Sep 30, 2024

Send the text directly to the OpenAI API although it will consume your quota.

Tim Chapman · Sep 30, 2024

jkhazraji said:
Send the text directly to the OpenAI API although it will consume your quota.

I need the token count before sending to OpenAI API. That is why I need TikToken in B4J.

DonManfred · Sep 30, 2024

GitHub - knuddelsgmbh/jtokkit: JTokkit is a Java tokenizer library designed for use with OpenAI models.

JTokkit is a Java tokenizer library designed for use with OpenAI models. - knuddelsgmbh/jtokkit

github.com

Download jar

https://mvnrepository.com/artifact/com.knuddels/jtokkit/1.1.0

You probably can use it with JavaObject

Tim Chapman · Oct 5, 2024

DonManfred said:
GitHub - knuddelsgmbh/jtokkit: JTokkit is a Java tokenizer library designed for use with OpenAI models.

JTokkit is a Java tokenizer library designed for use with OpenAI models. - knuddelsgmbh/jtokkit

github.com

Download jar

https://mvnrepository.com/artifact/com.knuddels/jtokkit/1.1.0

You probably can use it with JavaObject

Hi Don,

I have delayed replying to you until I try to use the jar file you recommended. I apologize for not thanking you sooner. I have had to take a detour and work on software for my business. It is now giving me trouble which I can't figure out. I am posting in the B4J forum about it. When I get back to my OpenAI project, I will try to use the jar file you suggested and will let you know what I find.

I posted my question here: https://www.b4x.com/android/forum/threads/combobox-not-showing-anything-when-i-click-on-it.163439/

DonManfred · Oct 5, 2024

Tim Chapman said:
I posted my question here

See my answer/solution in that that. ;-)

DonManfred · Oct 5, 2024

Tim Chapman said:
until I try to use the jar file you recommended

you are in luck. i already wrote a small wrapper for it ?

Download the jar from the link in #4 and put it into the additional libs folder

Basically you need to have an

B4X:

    Private tokkit As EncodingRegistry

and then use it

B4X:

    tokkit.Initialize("") ' Eventname is not used
    tokkit.SetModelEncoding("GPT_3_5_TURBO")
    
    Dim tokens As List = tokkit.encode("This is a Test!")
    Log(tokens)
    
    Dim decoded As String = tokkit.decode(tokens)
    Log($"Decoded ${decoded}"$)

Tim Chapman · Oct 5, 2024

DonManfred said:
you are in luck. i already wrote a small wrapper for it ?

Download the jar from the link in #4 and put it into the additional libs folder

Basically you need to have an

B4X:

Private tokkit As EncodingRegistry

and then use it

B4X:

tokkit.Initialize("") ' Eventname is not used tokkit.SetModelEncoding("GPT_3_5_TURBO") Dim tokens As List = tokkit.encode("This is a Test!") Log(tokens) Dim decoded As String = tokkit.decode(tokens) Log($"Decoded ${decoded}"$)

Thank you Don!

B4J Question Counting Tokens for OpenAI

Tim Chapman

Active Member

OpenAI API: How do I count tokens before(!) I send an API request?

How to compile nodejs project into jar-Front-end Q&A-php.cn

tiktoken/js at main · dqbd/tiktoken

Tim Chapman

jkhazraji

Active Member

Tim Chapman

Active Member

DonManfred

Expert

GitHub - knuddelsgmbh/jtokkit: JTokkit is a Java tokenizer library designed for use with OpenAI models.

Tim Chapman

Active Member

GitHub - knuddelsgmbh/jtokkit: JTokkit is a Java tokenizer library designed for use with OpenAI models.

DonManfred

Expert

DonManfred

Expert

Attachments

Tim Chapman

Active Member

Similar Threads

B4J Question Counting Tokens for OpenAI

Active Member

Active Member

Active Member

Expert

Active Member

Expert

Expert

Attachments

Active Member

Similar Threads

Privacy & Transparency

Privacy & Transparency