What is Jev? And why do we need it?

I recently had the opportunity to participate in the AI Next Door podcast. In the episode, Revathy and I discussed about the Jev model that was announced a few weeks ago.

You can find the episode below. While the episode is in Tamil, there are subtitles in English available that you can turn on.

Transcript

Siddharta: So Revathy, right now everyone on social media is talking about this JEV model. What is this JEV model? Is it a new LLM? What kind of model is this?

Revathy: Right. Like you said, JEV is something that is being talked about very virally within the AI community. This was started by a company called TypeSafe AI. It was founded by very old OpenAI researchers.

The moment they released it, we started seeing other open-source variations also. Not only that, now OpenAI has also made an announcement that they are coming up with - they didn't say it's their own version of JEV, but the Decisions API they are going to release is more or less comparable to JEV.

If you ask why this is so special, before Generative AI came along, AI mostly meant predictive analytics, then some classification use cases, and they were using it for some linear use cases. An important component in that is classification. That is, what it does is, we expose it to a large amount of training data, label it, for example saying these kinds of emails are spam, these kinds of emails are normal legitimate emails, and we used to train the model with that labels. And when we give it another unknown email, that AI model will use that training to find out if it is spam or a normal email.

Now, in the absence of training data, a lot of people started using Generative AI models for this. But the thing is, I would say it's over-engineering. If we only need two intents like that, Generative AI models can tend to overdo a lot. So if I show this email and ask, "Is this spam or a normal email?", it will say, "This is spam, because I am seeing this kind of content." After giving reasoning like this, it will say "This is spam".

When you look at this in real life use cases, if you are giving this as a decision to an agent itself, you will have to do a lot of post-processing. Because the one thing that agent needs is if this is spam email or not. Now if we just give it as spam, it will automatically take those decisions and go to the next set of actions. So post-processing is one thing.

Another thing is, when we automate, we need to threshold by confidences. Now Generative AI model gives a response. We don't know how correctly or wrongly it is giving it. Classification models will always give it to us with a confidence estimate. So lower confidences can be ignored, and we can just take those responses given with higher confidence.

Now for these downsides of Generative AI model, JEV is a proper option. Because it does not need training data, it comes with the worldly knowledge that an LLM can have. At the same time, it can limit your output, does not give reasoning that you might not need, and it will pick and give from the choices you provide. Another thing is confidences, so that will help you threshold if this response is really legitimate or not. So that's why I would say it's like throwing waves in the community.

Siddharta: Regarding this JEV, you talked about three models. Classification model, JEV and LLMs like ChatGPT. So, why can't we use a classification model? Why can't we use ChatGPT? Why do we need JEV for this?

Revathy: So if you want to use classification models, for that we will need training data, a massive amount of training data. So before training, we will have to do a lot of pre-processing. If that training data is not there, some people use a lot of... even in many use cases I work on, we use Generative AI models instead of classification models when we do not have the data. As I said, the drawback of those two methods is, Generative AI is not structured, classification models will need training data. So if you - if you need a structured output from your set of options as labels without training data, I think JEV plays an important role there.

Siddharta: I heard JEV is a System 1 model and ChatGPT is a System 2 model. So, what are all these System 1, System 2 models?

Revathy: A System 1 model is something that won't give any reasoning or anything like that. It will just give the decision directly. Just to quote an example, if we have an e-commerce business, I will get some mails in customer support. Some mails, emails will be about billing. Some emails will be about inquiries. Some emails will be about sales. Some emails will be complaints. Out of these four different types, if it just tells me what is the category of my email, and I don't need any reasoning, that's the system, that's what a System 1 model does. Now the second, a System 2 model, if you see, it will give you responses. It will do a good analysis, and tell you why it chose this particular decision like this, what is its backing, and it would give you a response. So, if you look at JEV, it is not a generation model. A generation model means each and every text will come as an output. If it's "This is a cat.", to generate that token "cat" at the end, it will need these three sentences "this is a". But type, I mean System 1 models, if you look at them, it needs a single decision pass. It's not dependent on any tokens that are generated as such. So, in a single forward pass itself, it will directly give us what the categorization is, whether it's billing, or sales, or a complaint.

Siddharta: I saw an interesting example. They say this System 1, System 2 all comes from brain psychology. (Revathy: That's right.) So, what System 1 is, immediately when something happens, the System 1 part of the brain is used. (Revathy: Yes.) So, if we touch something hot, we have to suddenly pull our hand away. We can't do any analysis for that. So, that is System 1. (Revathy: Yeah.) But if we have to decide which college to go to, we will collect all the data, analyze it, (Revathy: Yeah.) and then make a decision. So, that is System 2. (Revathy: Exactly, yeah.) So, have they mapped it similarly here in LLMs too? If we want an immediate decision, System 1 will be fast, but it won't do reasoning. (Revathy: Yes.) System 2 means, it will do good reasoning, analyze it, and give an answer.

Revathy: Absolutely. Yes, that's a great source of analogy.

Siddharta: For JEV, if we want an immediate decision now, we can use System 1. But how is the accuracy and all? Will it hallucinate? Because the one complaint with ChatGPT is that we cannot trust that output. Hallucinations and all can happen. Even if it happens, we don't have the confidence whether it is correct or not. But how is it in JEV? Some people are saying hallucinations cannot happen in JEV. The architecture itself is different. So, how is it for JEV?

Revathy: Right. Now how we should look at this is, accuracy and hallucination are totally two different things. Although, hallucination does affect accuracy. Now if we talk about those email responses, I have four different categories. So, billing, sales, complaints and you know, returns. If I want an option to come from these four, I would go for JEV. JEV will not skip these four options and search and find some other new label or categorization and give it to me.

Siddharta: So it won't make up something suddenly...

Revathy: Yeah, it will never do that. It will only restrict itself to the number of choices that we give. If we give this same thing to an LLM model, because it is a generation model, it will hallucinate. So, looking at a totally out of the box email, it will find a separate label on its own and give that as the response. So, when it is like that, post-processing will be difficult for us. We will have to think about a lot of corner cases and design it accordingly. So, hallucination will not be there in JEV. But accuracy might take a hit with the number of labels. Now if you look at TypeSafe AI's model, it only has a 255 label limit. Like this example I said, if we have to restrict it within just four choices, it will do it. For example, if you take this particular you know FMCG product, there will be more than 255 different categories. If you ask whether it can differentiate that, in some benchmarking tests they say that as your categories or labels keep increasing, its performance is getting lower. So, accuracy might take a hit as you keep increasing your cardinality.

Siddharta: If I summarize what you just said, JEV will be fast. JEV will be less hallucinated. We can't say about accuracy. Still, JEV has the chance of giving a wrong answer. But it won't make up a new option. That is the difference between ChatGPT and JEV. So JEV can still give the wrong answer, but a new option won't come up just like that as a corner case.

Revathy: That's true. But to see how reliable it is, it gives out a confidence score. So, keeping that confidence score, we can estimate if this is okay or not. So if you just take only those high confidence responses, generally your accuracy can be higher.

Siddharta: So that is another advantage. Because getting a confidence score from ChatGPT is very complicated, right? How do we get the confidence score? Can we do it in ChatGPT?

Revathy: No. Now they are engineering confidences using a lot of methods. For example, now sending the same response to three, four different models or in three, four different temperature settings, taking all those responses and seeing what their average correctness is, we can try to bring up a confidence estimate like that. Now if you look at ChatGPT like GPT LLMs, when you use the API, along with your response, a parameter called logprobs will come. It's - it's nothing but your probabilities of the tokens that are getting generated. Now in generation models, the thing is, the token you generate is automatically a token that has an estimate of highest confidence from the model. So, when you take its average, it will only give the confidences of that token, but it cannot give you the full confidences of that response. So for GPT or any other LLMs for that matter, you can never trust a confidence that you are estimating or engineering. We can only say it approximately.

Siddharta: Then what are all the use cases for JEV? The fast cheap model but where can we use it?

Revathy: So, as you said it is very cheap output. Unlike LLMs, it does not get charged for input tokens and output tokens. There is no option called output tokens in this at all. It only charges for input tokens. So, if you look at that cost scale, it's massive savings you can take with JEV for these applications. Like I said, for typical classification problems like if you have to segregate your customer emails, or if we have to look at the categories of some products, or else to quickly estimate what the next step of action will be, we can use JEV. Typical classic machine learning classification use cases.

Siddharta: So, some use cases can be, because payments come in a bank, we have to classify whether it is fraud or not. Earlier we developed specialized models for that same problem. Now using JEV, generally we can do this without separate model training. In healthcare, we have to make a lot of decisions. We can use JEV for all that. Then in software development too, now we will send complicated problems to a Frontier Model. Simple problems can be handled with a cheap model itself. We don't have to send everything to Frontier Models. Now we can use JEV for this model routing too. In fact, just yesterday an email came. There is a company called OpenRouter. They do this model routing. They are using JEV. Earlier they were using some other model. Now they have changed it to JEV. Like this, there are a lot of applications just for decision making. Which is a large part of what we were using LLMs for. We can replace all that with JEV now. It will be faster. It will be cheaper too.

Revathy: That's true. So for chatbot related applications, you can keep retaining your LLMs. For making quick decisions like this, switching to JEV is a much more smarter option. TypeSafe AI, this is costed. It's not open-source, but now you have a lot of open-source models like OpenJEV, AnyJEV, LLaVA, with options like this you can keep experimenting.

Siddharta: So, the summary of this conversation is that JEV is a new model. It is a specialized model just for decision making. It won't do text generation and all. But for that use case, it is a very fast model, a cheap model, and it won't make up hallucinations that much. So with all these advantages, a lot of companies are now shifting to JEV in areas where they used LLMs.