How local AI works
A plain-English guide to the technology behind everything on this site: what an AI model actually is, how it runs on a computer in your building, and how I make one smaller and quicker for a single job.
It writes one word at a time.
A language model has read an enormous amount of text and learned which words tend to follow which. Give it a question and it predicts the most likely next word, adds it, and repeats.
Reading an invoice, summarising a long thread and drafting a reply all come from that one skill, done very quickly. Press play, or step through it yourself.
A model is a very large file of numbers.
What a model learns is stored as billions of numbers, called parameters. More parameters usually means a more capable model, but also a bigger file, and the whole file has to sit in the computer’s memory while it works.
Compression stores each number in fewer bits. Done carefully it costs little quality for everyday business tasks, which is why a Mac on a desk can now run models that once needed a server. Move the slider to see what fits where.
Some models are teams of specialists.
Many of the newest open models are built as a “mixture of experts”: dozens of smaller specialist networks, with a router that picks a few of them for each word. Most experts sit idle most of the time.
A business that uses a model for one job may never call on many of them. Recent research has shown that removing rarely used experts can make a model much smaller while keeping most of its ability on the work that matters. Pick a job, let it run, then trim.
Custom models, built for one job.
I start from an open model and shape it around your work, so it fits a smaller machine, answers faster and uses your categories, wording and formats.
1
Choose the right open model
Picked for the job, its licence and the hardware you have.
2
Trim what the job doesn’t use
Remove experts your examples never call on.
3
Compress it to fit
Store the numbers in fewer bits so it fits your machine with room to spare.
4
Teach it your way of working
A short round of training on your own examples and formats.
5
Test it on your records
Compare it with the full model on real cases before anyone relies on it.
Not every job needs this. Often an open model straight off the shelf does the work well, and I’ll tell you when that’s the case.
Why I use Macs.
Most computers keep the graphics chip’s memory separate and small, which limits the size of model they can run. Apple’s chips share one large pool of memory, up to 512 GB on a Mac Studio, so a quiet machine on a desk can hold a model that would otherwise need specialist hardware.
Apple’s MLX software is built to run models on exactly these chips. I use other hardware when it suits the job better.
Cloud, local or offline.
Some work suits a cloud model. Some information shouldn’t leave the building. Most businesses end up with a mix, decided job by job.
Questions people ask about local AI.
Is my data sent anywhere?
Not for the model to do its work. Files are read and processed on the Mac in your building. Data only leaves if a job uses a cloud tool you already rely on, such as your email, and that’s agreed and written down before I build it.
Is it more secure than ChatGPT or Copilot?
It removes the biggest unknown, because your files aren’t sent to an AI company to be processed or stored. Security still rests on the basics, like who can log in and how your network is set up, so I go through those with you rather than promise anything is unbreakable.
Are you SOC 2 or ISO 27001 certified?
No. Those are formal audits, mostly taken by companies that host and store other people’s data on their own servers. With Asrun Local, your data stays on hardware you own rather than mine. I work within your security and IT policies, and if part of a job needs a certified supplier, I’ll tell you.
How does this fit with UK GDPR?
Processing personal data on your own hardware means fewer outside companies handle it, which can make your records and supplier contracts simpler. You stay responsible for how the data is used, and I set the system up around your data protection policies. I’m not a lawyer, so check anything formal with your adviser.
Is it as good as ChatGPT?
For open-ended conversation, the largest cloud models are ahead. For one specific job on your own files, set up and tested properly, an open model on a Mac is usually all you need. I test it against your real examples before anyone relies on it.
What does the hardware cost?
Usually one Mac mini or Mac Studio, bought once and owned by you. I recommend the smallest machine that does the job well, and you see the price before you commit to anything.
Does it need the internet?
Not to run. The model works offline. It only needs a connection for updates you approve, or to reach cloud tools you choose to connect.
Can it still use our cloud tools?
Yes. Most setups are hybrid: the model runs locally and reads from the tools you already use, such as shared drives, email or your accounts system. Where a job genuinely suits a cloud model, I’ll say so.
Who can use it?
People on your network, with the access you agree. Each person sees only what they’re allowed to, and the system keeps a record of what it did.
What if the machine breaks?
Your files never live only on it. The tool can be set up again on a replacement, and your support agreement sets how quickly.
Will the model go out of date?
Open models improve quickly. As part of support I test newer ones against your work and only switch when one does the job better.
Do I need to understand any of this?
No. You need to know the job that takes too long. The rest is my job.