What on-device AI actually means
On-device AI means the model runs on hardware you own: your laptop, your phone or a small office server. Nothing you type, and no document you attach, ever leaves the machine. This has moved from hobbyist territory to genuinely practical. Recent Windows laptops with neural processing units, any Apple silicon Mac with a reasonable amount of memory, and current flagship phones can all run capable small language models at usable speeds.
Contrast that with cloud AI, where ChatGPT, Claude, Gemini and the rest process everything you send on someone else's servers. The big cloud models remain far more capable at hard reasoning, but for a large share of everyday small-business tasks, summarising, drafting, rewriting, extracting and reformatting, a good small local model is more than enough.
Why local is a genuine privacy win
When data never leaves the device, several compliance headaches simply disappear. There is no third-party processor to vet, no data processing agreement to sign, no international data transfer to assess and no vendor retention policy to police. Under UK GDPR you remain the controller of data sitting on your own laptop, exactly as you are for a Word document, and no new party enters the picture.
- No processor agreements or vendor due diligence for the AI step itself
- No risk of prompts being retained, reviewed by staff or used for model training
- No exposure if an AI vendor suffers a breach, because they never held your data
- Works offline, which also suits site visits, travel and patchy rural broadband
One caveat: local AI is not automatically compliant. Outputs can still be wrong or biased, and device security, disk encryption, screen locks and patching, matters more than ever because the sensitive material stays on the laptop.
Need a hand with this?
Our team delivers AI & Machine Learning for UK businesses — with a free initial consultation, transparent fixed quotes and no lock-in contracts. Tell us what you're working on →
Where cloud AI is simply the wrong call
Drafting HR documents
Disciplinary letters, grievance responses, redundancy consultation notes and occupational health correspondence often contain special category data about named employees. Pasting that into a free consumer chatbot risks breaching your own confidentiality policies and your data protection obligations in one move. A local model can draft, redraft and tone-check the same letter with zero disclosure to anyone.
Client and case data
Accountants, solicitors, consultants and agencies working under NDA face a blunt contractual problem: most confidentiality clauses prohibit sharing client information with third parties, and a cloud AI vendor is a third party. Running the analysis locally sidesteps the question entirely rather than arguing about it.
Unreleased commercial plans
Pricing changes, acquisition conversations, investor decks and unannounced products are exactly the material you do not want sitting in a vendor's logs, however trustworthy the vendor. If a document is too sensitive to email to a stranger, it is too sensitive for a consumer chatbot.
The hardware you probably already own
You may not need to buy anything. Copilot+ PCs, the Windows laptops sold since 2024 with dedicated neural processors, are built for exactly this workload. Any Apple silicon Mac with 16GB of memory runs models in the seven-to-eight-billion-parameter class comfortably, and 32GB opens up noticeably smarter mid-size models. Recent iPhones and Android flagships run smaller models on the phone itself, and Apple Intelligence already handles many of its tasks on-device by design.
Memory is the main constraint, not processor speed. If your laptop is less than three or four years old, try local AI before spending a penny; if you are buying new anyway, prioritise RAM over almost everything else.
Tools to start with this week
- Ollama: free and runs on Windows, Mac and Linux; a single command downloads and runs open models such as Meta's Llama, Google's Gemma or Microsoft's Phi families.
- LM Studio: a friendlier graphical app for the same job, with a built-in chat window and a model browser that flags what will actually fit on your machine.
- GPT4All and AnythingLLM: let you chat privately with your own documents, a local alternative to uploading client files to a cloud chatbot.
Start with a small model, give it a real task from your week, an email to rewrite, minutes to summarise, and judge the output honestly. Most people are surprised in both directions: weaker than the big cloud models at reasoning, better than expected at routine drafting.
Key Takeaway
If a document is too sensitive to email to a stranger, it is too sensitive for a consumer cloud chatbot. Install Ollama or LM Studio on a reasonably modern laptop, download a small open model, and route confidential work (HR letters, client files, unreleased plans) through it. Keep cloud AI for low-risk tasks under a proper business agreement, and write the split into a one-page AI use policy your whole team can follow.
Limits, and a sensible hybrid policy
Local models are the wrong tool for some jobs. They are weaker at complex multi-step reasoning, they know nothing about this morning's news, and very long documents can exceed what a laptop handles gracefully. The answer is not choosing a side; it is a written split.
- Public or low-risk work (blog drafts, generic research, code snippets): cloud AI, ideally on a business tier with training opt-outs and a signed data processing agreement
- Confidential work (HR, client data, financials, unreleased plans): local models only
- Anything regulated or privileged: local only, on an encrypted device, with the rule written into your AI use policy
Put that split on one page, share it with the team, and you have solved most of your AI privacy exposure in an afternoon. If you want help setting up local AI or drafting a workable AI use policy, our team can help.