I need to be able to count tokens in a user's text submission (before it is submitted) to OpenAI API.
I have researched and found that TikToken is the best way to do that.
There are two TikToken libraries available at this time. One is in Python. The other in Node.js. Those are shown here:
stackoverflow.com
What is the easiest way for me to get TikToken into B4J?
The closest I have seen is to compile a Node.js project into a jar file as shown here:
www.php.cn
This looks like the correct JS code to work with:
github.com
Any better answers anyone can see for this? Doing the conversion using the instructions above is a bit out of my league.
I have researched and found that TikToken is the best way to do that.
There are two TikToken libraries available at this time. One is in Python. The other in Node.js. Those are shown here:
OpenAI API: How do I count tokens before(!) I send an API request?
OpenAI's text models have a context length, e.g.: Curie has a context length of 2049 tokens. They provide max_tokens and stop parameters to control the length of the generated sequence. Therefore the
What is the easiest way for me to get TikToken into B4J?
The closest I have seen is to compile a Node.js project into a jar file as shown here:
How to compile nodejs project into jar-Front-end Q&A-php.cn
Node.js is a very popular JavaScript runtime environment. Large web frameworks such as Vue.js and React Native run on Node.js. However, for Java programmers, compiling Node.js projects into Jar packages may be more convenient for integration into Java projects. In this article, we will introduce...
This looks like the correct JS code to work with:
tiktoken/js at main · dqbd/tiktoken
JS port and JS/WASM bindings for openai/tiktoken. Contribute to dqbd/tiktoken development by creating an account on GitHub.
github.com
Any better answers anyone can see for this? Doing the conversion using the instructions above is a bit out of my league.
