Lessons from an LLM integration, part 2: swapping models

When ending part 1, I said that the next instalment would be about response times. Since a new model was released, I paused that text for a while and will focus on a model swap instead today. In a way it’s actually about response times, but...differently.

Preview of Large 4 (2026-10-06)

Mistral announced a preview of Large 4 a few weeks ago and I thought at the time that it would be fun to run some experiments with it in my application called ”factor10 Solution Designer” (an application for collaboratively creating solution proposals for software initiatives). I ran a small subset of my automatic tests in parallel against Medium 3.5, which I used in the application, and the preview of Large 4.

In the small subset of tests, the two models behaved somewhat similarly regarding input caching (which was the topic part 1 focused on). The overall cost was similar too. (factor 1.6 for Large 4, as the expected cost was described then. It was later changed, which would have made it factor 0.8.) But there was a clear difference in how many reasoning tokens and output tokens Large 4 used compared to Medium 3.5. Large 4 used many more. As a result, the latency was around a factor of 4 for Large 4 in amixed set of tests.

I did see indications of better results with Large 4, but since it was a preview and since the latency difference was that big, I decided to not move further with the experiment. I skipped swapping models for the application.

The experiment wasn’t a waste of time though. I found several weaknesses in my test suite that I fixed. Improvements that will be helpful the next time. And the next and… Again I felt like the gardener nurturing the garden, a little bit every day.

Official release of Large 4 (2026-10-29)

Three weeks after the preview, there was the official release. I’m not going to repeat all of the official information of course, you can easily find that on your own. But let’s have a quick look at the prices. Mistral’s price list says that Large 4 is priced at $0.70 input / M, $0.07 cached input / M, and $2.10 output / M tokens. The prices for Medium 3.5 aren’t listed any longer, but my latest notes say $1.50 input / M, $0.15 cached input / M, and $7.50 output / M tokens. It’s good to keep that in mind when we come to the comparisons.

Three effort levels

What I didn’t try with the preview of Large 4 was to set a specific effort level. You can choose between effort levels low, medium and high. I used the default instead, which for Large 4 (at least now for the release, and I guess for the preview too) is the same as effort high.

My first tests were to see which effort level of Large 4 gave the results most similar to Medium 3.5 (without effort level). It was clear that effort low had the lowest increase in latency, but without degrading the quality of responses. (It rather seemed to be better regarding quality.) I quickly zoomed in on Large 4 with effort low as the candidate for swapping out Medium 3.5. Let’s see how it compares on a couple of important aspects.

Input caching and Time To Live (TTL)

After a few iterations, my conclusion was that the input caching functionality for Large 4 (low) was pretty much on par with Medium 3.5. A cache hit was just slightly less likely, but the difference was so small that it doesn’t matter that much.

But a big difference was the TTL. It now seems to be something like 13 minutes for Large 4 (low), compared to at least 40 minutes for Medium 3.5. Depending on your usage scenario, this can be important and become a big difference in cost (since cached input tokens cost a tenth of cold input tokens).

Note

A bit of trivia: in part 1 I measured a low likelihood for a cache hit after 10-20 minutes for Medium 3.5. I couldn’t replicate that during the preview tests. During recent tests on the other hand it was, as just mentioned, rather at least 40 minutes for Medium 3.5.

The basic rule stays the same: a lot of similar calls are cheap. Few and/or different calls are expensive by a factor of 10! TTL makes the risk of the higher cost slightly greater.

Quality of results

My defence tests showed better results for Large 4 (low) than for Medium 3.5 when it comes to content injection. That’s good, but not actually a big difference since I have other protection in place for that.

For other tests, I got indications of slightly better results here and there. (And a few worse which needed fixes from me.)

I didn’t just stay with automatic tests, but used the application too of course. Those tests (for this article) weren’t very systematic, but I got a good feeling from them.

Latency

The problem that I expected with Large 4 was increased latency. I decided when I started that factor 2 was a limit for me. Effort high and especially medium was just barely above factor 2, while low was just barely below. When running more test suites, the difference to Medium 3.5 increased slightly, and I ended up with something like factor 2 on average overall for low. Definitely worse, but within the limit by the skin of our teeth, I think.

Cost

Surprisingly, the cost difference is very beneficial right now. After all, we manage to benefit from input caching a lot and are careful with the reasoning and output size/cost. And since Large 4 is priced much lower per token, we see the cost go down to a factor of 0.45 on average over the chosen subset of mixed tests.

Summary

Again, it’s a subset of mixed tests, but here’s a summary for Large 4 (low) compared to Medium 3.5:

So, two improvements and two drawbacks.

Side effects of the swap work

Just as with my test executions for the preview, the tests improved further now. I also learned more. Pretty good side effects if you ask me. Probably the most important part.

An example of test improvements and learnings were that during the preview tests I came to the conclusion that Medium 3.5 couldn’t spend reasoning tokens. I came to that understanding since I tested effort=xhigh. The problem was that my call didn’t lead to an error code back then. xhigh is not supported, but high is. Today xhigh leads to error 400, but according to my test back then it was quiet… (That said, I was too quick to judge, looking in the rear-view mirror.)

Quickly comparing Medium 3.5 without setting any effort today against using it with effort high, it’s very clear that Medium 3.5 can spend tokens on reasoning. :) It can also be seen in that latency and cost are affected negatively compared to Medium 3.5 without defining effort. (Latency factor 2.6-3.3 and cost factor 1.6-2.5. I haven’t compared quality for different working effort settings of Medium that much. But one surprise was a degradation from 78 to 72 working tests in one of the test suites when trying high instead of default.)

Comparing Medium 3.5 (high) to Large 4 (low) then, Medium is much slower and very much more expensive.

Another motivation and making the decision

A few days after the release, I got an email from Mistral with information that Medium 3.5 will be retiring at the end of April next year. (On Mistral’s website it says its deprecation date was 2026-10-31.)

Not too stressful receiving that information about retirement half a year in advance, but a sign that Medium 3.5 won’t be the road forward. Even more so when I now saw the deprecation date.

Not that the swap decision in my application is an agonising decision at all. It’s software. We make the best decision we can today. Another day we change easily if we realise there’s a better decision to be made. And there will be, sooner or later.

All in all, I decided to make the swap in the application. So at the time of writing, it uses Large 4 (low).

## Further work I had made a few tweaks (compensations) in the code that were added specifically for Medium 3.5. Those 10 places carried notes to make it clear what to re-check with a new model. I could delete 3 of them, 3 will be measured over a few weeks in the observability solution and 4 had to be moved over to Large 4 (low).

*Note

The code for the deleted notes was actually kept since the compensation avoids model calls. The notes are gone since Large 4 (low) now handels this correctly, but ordinary code is faster and cheaper and therefore stays.

By the way, those notes in the code are ”protected” with a ratchet test. That way, they can’t silently increase, such a decision has to be made intentionally. It’s also the case that the ratchet will be notified to adjust downwards when a note is fixed and deleted, to avoid the risk of the list silently growing back after it has decreased.

A thought from the past, revisited

In the past I have often heard the question ”what percentage of red tests can we allow?” To me it was a strange question, why allow any? If you know you have an error, you should of course fix it. (And I will not tire you with explaining all the other negative things about allowing red tests.)

Now in this new time of incorporating AI models in your application, models with a certain element of stochasticity and simulation of input as the user’s “voice, the situation for certain subsets of your tests becomes different. Not that you should allow red tests of course, but you need to take steps to allow ”intervals” rather than yes or no. It certainly challenges how the tests are designed. At the same time, the tests have (if possible?) become more important than ever. (Much more about this in a later text.)

Would you like to see how Large 4 (low) behaves?

If you want to see how Large 4 (low) behaves in the application, go to https://fsd.factor10.cloud and register an account for free. :)

Next time

After swapping the Mistral model, I thought it would be useful to use both effort low in certain situations and effort high in others. Learnings around that (which to a large degree come back to testing, surprise!) will be the theme of part 3.