Local LLM for small business: a 7B model in about 4 GB
In short. A local LLM for small business is a language model that runs entirely on your own hardware, with no call to an outside service. That reduces one risk, but it settles neither the quality of the answers, nor security, nor the GDPR. This guide offers seven questions to ask before installing, and a rough idea of model sizes.
“Local LLM” has become a common phrase, but it covers very different situations: free software on a laptop, a server in a technical room, or simply an online tool that was described as private. This guide helps the owner of a small business decide whether they need one, and on what conditions to try one. It draws on the llama.cpp and Ollama project pages, the Hugging Face documentation, one CNIL page and Article 32 of the GDPR, all read on 29 October 2026.
What exactly is a local LLM?
An LLM (large language model) is a program that produces text from an instruction. It has two parts: an engine that runs it, and a file of “weights” that holds the trained model. Running an LLM locally means downloading those weights and running them on your hardware.
Here, local means the entire chain: no external API, no automatic fallback to the cloud, no telemetry, no folder sync to a third-party service. Check backup destinations too. Software installed on a computer but wired to an online service is not local in the sense of this guide.
A note on vocabulary: models are often called “open source” when many are only open-weight. The licence of the engine and the licence of the model are two different things. The llama.cpp and Ollama projects are published under the MIT licence, but that says nothing about the licence of the model you will download: read it model by model, especially for commercial use.
Which tools run a model?
| llama.cpp | Ollama | |
|---|---|---|
| Role | inference engine | tool that downloads and launches models |
| Systems | a wide range of hardware (x86 processors, Apple silicon, Nvidia, AMD and Intel graphics cards) | macOS, Windows, Linux and Docker |
| Software licence | MIT | MIT |
| Interface | command line and server | command line and local API |
According to its README, llama.cpp aims to enable inference “with minimal setup” on a wide range of hardware, and supports quantization from 1.5 to 8 bits. Ollama’s README says a model can be run locally and the API is served on port 11434 of the machine. These pages establish that it is technically feasible; they say nothing about the quality of answers for your trade, or about the compliance of a given use.
What machine should you plan for?
A model is a large file of numbers. Quantization lowers the precision of each number so the file gets smaller. The Hugging Face documentation on the GGUF format gives the bits per weight for each type: about 4.5 for Q4_K, 6.56 for Q6_K, 16 for F16.
A simple calculation gives a rough idea of the file size (parameters × bits per weight ÷ 8). This is our calculation from those tables, not a measurement:
| Model | In F16 | In Q4_K |
|---|---|---|
| 7 billion parameters | about 14 GB | about 3.9 GB |
| 14 billion parameters | about 28 GB | about 7.9 GB |
The memory actually used is higher than the file size: it also depends on the context length, the engine and the settings. Quality also drops as quantization gets heavier, by an amount that depends on the model and the task. These figures are for ruling out impossible cases, not for sizing a purchase: test first on the machine you already have.
Do you need a local LLM? Seven questions to decide
Ask these for the specific task you have in mind, not for “AI” in general.
| Question | If the answer is… | Then |
|---|---|---|
| 1. What data must the model read? | personal or confidential | local is worth serious study |
| 2. Would an online tool do, on texts with no sensitive data? | yes | local is not necessary |
| 3. What level of quality do you need? | fine writing, long reasoning | test first: small models sometimes disappoint |
| 4. How many people will use it? | several, at the same time | plan a server, not a single computer |
| 5. Who installs, updates and troubleshoots? | nobody | do not start |
| 6. Who proofreads answers before use? | nobody | do not start |
| 7. How will you know it works? | no criterion | write twenty real questions and the expected answer |
If you answer “no” to question 1 and “yes” to question 2, an online tool is probably simpler. If questions 5, 6 and 7 have no answer, the project is not ready, whatever the hardware.
What do the CNIL and the GDPR say?
Two points are enough to decide whether to look into local.
-
A CNIL recommendation, in context. In its page on deploying generative AI, the CNIL recommends favouring “local, secure and specialised systems” (our translation of the French). That page gives first answers on deployment; it does not detail what on-site hosting requires, nor security measures, and it leaves out the design and training of models.
-
Security, wherever the system runs. Article 32 of the GDPR asks the controller to put in place technical and organisational measures appropriate to the risk. It does not require local processing, and local processing does not replace those measures: named access, logs, backups, retention period.
These pages do not cover every case. For health, client or employee data, define the purpose and the legal basis with your data protection officer or a lawyer before launching anything.
What does a reasonable first test look like?
Here is a fictional example, with no real client. A company wants to summarise internal meeting notes. Instead of buying hardware, it proceeds like this:
- Build the test set. Twenty fictional or anonymised reports, each with the summary a person would write.
- Install a small model on the existing machine with one of the tools above, offline if the constraint is strict.
- Compare. Note omissions, errors in figures and names, and response time. Do the same with the online tool currently used.
- Decide only afterwards, with the results in hand: stay with the online tool, use a larger local model, or change nothing.
This illustrates a method, not a measured result. Real differences depend on the model, the language, the machine and the documents.
What are the limits of a local LLM?
-
Quality is not guaranteed. The models you can run on a computer are often smaller than the largest online models. Errors, omissions and wrong answers stated with confidence remain possible: human proofreading is needed.
-
You need hardware and maintenance. Memory, possibly a graphics card, updates of tools and models, and someone to troubleshoot. Local is not always free: hardware, electricity and time all count.
-
Model licences vary. The engine’s licence says nothing about the weights; some restrict commercial use.
-
Local is not a compliance certificate. It reduces one risk (sending data to a third party) but does not replace informing people, security or the advice of your data protection officer.
-
Shared use adds complexity. A model on a personal computer cannot be shared without network setup, access rights and security.
How does Castel Studio work on local AI?
Our Private AI page groups our offers, including transcription without the cloud from 90 € per batch (place of processing, access and retention agreed before any files are sent), an assistant that knows your business from 790 €, and a workstation built for your work by written quote. Hosting, model or API costs, electricity and maintenance are agreed separately; we agree written tests, and we do not guarantee results.
To compare options before choosing, read our guide to private AI for business, which sets out ten questions to put to a provider. If your need is rather to automate a task, see also our article on AI agents for a small business and on PDF invoices to Excel.
Describe the task, the type of documents and the hardware you have in a few lines, without attaching sensitive files through the form. We answer in writing within 48 hours, including to tell you that an online tool is enough in your case.
Questions
What is a local LLM?
A language model whose weights are downloaded and run on your own hardware. For "local" to be accurate, the whole chain must be local: no external API, no cloud fallback, no telemetry and no automatic folder synchronisation. An application installed on a computer does not guarantee that by itself.
Is a local LLM GDPR compliant?
Not automatically. Keeping data on your machines avoids sending it to a provider, but you still need to handle purpose, information to people, access, retention and security. The GDPR does not require local processing: it asks for measures appropriate to the risk.
What machine do you need for a local LLM?
It depends on the model, its quantization and the context length. As a rough guide, a 7-billion-parameter model quantized at about 4.5 bits per weight is roughly a 4 GB file; the memory actually used is higher. Test on your own hardware before buying anything.
Does a local model match ChatGPT or Claude?
There is no general answer. The models a small business can run are often smaller than the largest online models. Test on twenty real questions from your own work, and compare with the tool you use today.
How much does it cost to set up local AI?
At Castel Studio, the Private AI page lists offers from 90 € (transcription without the cloud, per batch) and an assistant on your documents from 790 €. A workstation is priced by written quote. Hardware, electricity, maintenance and proofreading time are extra.