After my recent posts on this subject I got some questions, and I thought I would use my responses to write a new post. I’m hoping this makes the situation more clear. I’ll link my other posts on this subject at the bottom, in case you want to look at them. Some of these topics have been touched on in them also.
Why bother with local models?
I covered that here. In summary it’s cost and security.
The model matters
When running LLMs locally you have a massive list of open models to choose from. Hugging Face has a list of many of them here. It’s easy to get fooled into thinking the only models worth using are the frontier models, but that’s not true. Depending on the task you are doing, some smaller models give comparable results at zero cost and reasonable speed on low powered hardware.
So how do you know which model to choose? Every time I hear someone say something positive about a model I try it. It’s very easy to repeat some previous prompts and compare the results. There is no “best” model, even with the frontier models. So much depends on what you are doing. You may find you use several models on a regular basis. The great thing about running LLMs locally is you can do this with no cost implications.
What I’ve learned is that the size of the model is not always a reflection of the quality of output. Sure the bigger models are better all-rounders, but most of the time I’m using the LLM for specific tasks, and some of the smaller models work really well. The recent release of the Google Gemma4 QAT models really demonstrates this. They are designed to run on end user kit. They use a lot less memory, but perform equivalent to much bigger models. For example, gemma4:12b-it-qat is a 12b parameter model that performs nearly as well as the gemma4:26b model, which is three times the size. They are designed for agentic processing, so using them with a coding agent is cool.
Ultimately you can only use the models your kit can cope with. Just play around. I’ll mention what I’m using a little later, but if you look at my previous posts you will see I’ve changed several times. 🙂
The prompt matters
The more specific you are, the better the result! Some people feel like this is magic and they can put very little in the prompt, then they are disappointed with the result.
Just think of it like a program specification. The more specific you make it, the less room there is for a developer to make a mistake, or forget something. LLMs are the same in this respect. Some people go as far as providing pseudo code. For some tasks you will refer to your existing code base to give extra context. The trick is to provide enough information to get the job done, but not too much so you confuse the situation. Sound familiar? As I said, it’s like a program specification.
It takes time to get a feel for this. It also varies a little depending on the model used.
If you are working on some very specific stuff, you can direct the
LLM using skills, to keep it focussed. For example I cloned the Oracle
skills (https://github.com/oracle/skills.git) repo to help with creating an
APEX app. My prompt references a subset of the skills.
Generate a new APEX app.
Skills defined under the /u01/skills/apex directory
Table metadata in the /u01/apex_code/schema.sql file
Application name : Clocking
APEX workspace : CLOCKING_WS
Required pages:
– home page
– a page display and edit staff
– a page to log start and end times for a member of staff
– navigation
– breadcrumbs
– use default security.
Place the resulting code in the /u01/apex_code directory.
In this example I referenced a schema definition in a file. You could allow an agent to connect to the database and get information from there.
Here’s an example prompt from a previous blog post, where used an agent to generate some Python code to populate a vector column in the database.
Create a Python script called “/tmp/add_vector_embeddings.py”.
The script connects to an Oracle database.
Assume a database connection string will be provided as an environment variable.
The script reads the MOVIE_QUOTE column from a table called MOVIE_QUOTES.
It uses the data from the MOVIE_QUOTE column to create a vector
embedding using the EmbeddingGemma model running under Ollama.
The Ollama base URL will be provided by an environment variable.
The MOVIE_QUOTES table is then updated, setting the MOVIE_QUOTE_VECTOR column to the vector embedding value.
This should be done for all rows in the table.
Over time I’ve got better at prompting, the way it took me some time
to learn how to Google. I’m not saying I’m a prompt God, but I’m getting
better over time.
Retrieval-Augmented Generation (RAG)
I’m going to keep this really simple, but hopefully you get the idea. 🙂
Models are trained on data, and they answer your question based on the training data they have. If you want to work on a subject that is not part of their training data, they are going to struggle. If their training data contains a lot of “questionable” stuff, they may follow a path you do not agree with. With Retrieval-Augmented Generation (RAG) you can provide the LLM with information that you believe is relevant, and it will interpret and use that information.
For example you might provide a PDF of some documentation and tell the
model/agent to use that as the basis of the answer. This way it doesn’t search the net for old rubbish. It goes straight for docs you have said are quality. You can also point it at your existing code repos, and tell it to give the answer in the style of your existing code base. That way you are likely to get an answer more pleasing to you. It’s effectively making the prompt better, and we’ve already said the prompt matters.
RAG can also include searching the internet to see if alternative information is available to augment the training data. If you use agentic workflows, searching for additional information is quite a common step.
RAG is a whole subject in its own right, but I think we’ll stop at this point.
Chat vs Agentic
Using a model in a chat is nothing like using a model from a harness
(Pi agent, Hermes Agent, OpenCode, Claude Code, OpenAI Codex etc.).
These agents take your prompt and refine it using search. They come up
with a plan, then validate the plan. Once they’ve decided what they
need to do, they do it, then check the result. Sometimes they will
throw the result away and start again. It’s an iterative process that
may take some time, but the result is generally better than a one-off
question in chat.
This iterative process uses a lot more tokens, but when you are running locally that is fine. It’s all free. That’s very different to using a subscription or API billing, where an agent can end up costing you a fortune.
I didn’t appreciate the difference between chat and the agentic approach until I tried it. Once I did it was a light bulb moment. I now rarely use a chat and instead use an agent to get an answer. This is especially true of writing code, where I actually want it to write the files. An agent can do that, but a chat session can’t.
Agents are too dangerous!
If you give an agent full access to your machine, and link it into all your services (email, chat, git, deployment pipelines) there is a possibility it will make mistakes, but you don’t have to do that. For example I might let my agent read the contents of a git repo, but tell it to write changes to another location. I can then validate them separately.
If you are giving your agent access to your database, simply set up a read-only user. It will be able to look at the metadata, but not change anything.
You can give an agent as much or as little access to your system and services, so you are in control.
What kit should I buy?
This is going to be very vague and simplistic!
I discussed kit here. The problem is it is very nuanced. It depends what you are planning to do. You can get away with a small inference machine like a mini PC or a Mac mini, or you can buy a cluster of machines. I’m mostly focussed on running stuff at home, so I’m purposely not spending a fortune.
Personally I would get a Mac Mini or a MacBook Pro. If you are in a rush I
would get a M5 MBP. If you can wait, I would wait for the M5 Mac Mini
or the M6 MBP, both of which are meant to be released soon. I will
probably get an M5 Mac Mini when it is released.
The later the M series, the better it is for AI, as the number and
quality of the GPU cores increases over the generations, so in this
case later is better. That’s not to say the older M series chips can’t
do it. They just won’t do it as well.
The real kicker for larger models is memory. If you want to run the
big models you need a lot of memory for the GPU. This is where the
Apple Silicon architecture helps…
For a Windows PC using a GPU such as a RTX card, the model runs on GPU
memory, not on the system memory. If the GPU memory is not enough, it
will run on the CPU plus the system memory, but that is going to be
way slower than running on the GPU. As a result, you need a card with
the most VRAM you can get. For home use that is currently the 5090 with 64G. With Apple Silicon, the system memory is unified (shared between the system and the GPU), so if you get a 64G Mac, you can use a lot of that memory for
running models. What a lot of people do is get a Mac Studio with 128G
RAM, which allows them to run the bigger models, or even cluster
several together. Of course, this is really expensive.
I have a mini PC at the moment. I will probably end up getting a M5 mac mini with 32G RAM (if they don’t limit it to 24G), or a low end Mac Studio with 64G.
My experience of using the mini PC is pretty good for coding. I can
ask regular questions and it responds quickly. For a coding agent,
such as Pi Agent, it is surprisingly fast. I was expecting it to be
way worse. 🙂 It all depends on the model selection.
Remember, what I need and what you need may not be the same thing. Don’t take what I do as a recommendation for your requirements.
What models am I using now?
I’m currently using the Gemma4 (Google) models most of the time. The smaller ones are quicker, but the bigger ones give better output.
- gemma4:e2b
- gemma4:e4b
- gemma4:12b
The normal 12b model was a little heavy. It worked, but it was a bit
sluggish. They now have these new versions using Quantization-Aware Training (QAT), which are smaller and give much better results.
- gemma4:e2b-it-qat
- gemma4:e4b-it-qat
- gemma4:12b-it-qat
I now use “gemma4:12b-it-qat” for almost everything and it is really
good. I’m not sure I will use bigger models when I get better kit. I
will probably use the same ones, but get even speedier results.
Check out these posts from Google.
Previous posts
Here are some of the previous posts I’ve written about running LLMs locally.
Thoughts
I’m not a LLM or AI guy. I don’t have all the answers. I’m just a few weeks ahead of some of you, and a few years behind others. 🙂
Cheers
Tim…
