All articles

Local LLM for small business: a 7B model in about 4 GB

In short. A local LLM for small business is a language model that runs entirely on your own hardware, with no call to an outside service. That reduces one risk, but it settles neither the quality of the answers, nor security, nor the GDPR. This guide offers seven questions to ask before installing, and a rough idea of model sizes.

Pencil and watercolour illustration of a business owner using a desktop assistant beside a compact workstation and project documents.
Fictional office setup. Local processing must be checked across the whole workflow; the illustration is not proof of security or compliance.

“Local LLM” has become a common phrase, but it covers very different situations: free software on a laptop, a server in a technical room, or simply an online tool that was described as private. This guide helps the owner of a small business decide whether they need one, and on what conditions to try one. It draws on the llama.cpp and Ollama project pages, the Hugging Face documentation, one CNIL page and Article 32 of the GDPR, all read on 29 October 2026.

What exactly is a local LLM?

An LLM (large language model) is a program that produces text from an instruction. It has two parts: an engine that runs it, and a file of “weights” that holds the trained model. Running an LLM locally means downloading those weights and running them on your hardware.

Here, local means the entire chain: no external API, no automatic fallback to the cloud, no telemetry, no folder sync to a third-party service. Check backup destinations too. Software installed on a computer but wired to an online service is not local in the sense of this guide.

A note on vocabulary: models are often called “open source” when many are only open-weight. The licence of the engine and the licence of the model are two different things. The llama.cpp and Ollama projects are published under the MIT licence, but that says nothing about the licence of the model you will download: read it model by model, especially for commercial use.

Which tools run a model?

llama.cpp Ollama
Role inference engine tool that downloads and launches models
Systems a wide range of hardware (x86 processors, Apple silicon, Nvidia, AMD and Intel graphics cards) macOS, Windows, Linux and Docker
Software licence MIT MIT
Interface command line and server command line and local API

According to its README, llama.cpp aims to enable inference “with minimal setup” on a wide range of hardware, and supports quantization from 1.5 to 8 bits. Ollama’s README says a model can be run locally and the API is served on port 11434 of the machine. These pages establish that it is technically feasible; they say nothing about the quality of answers for your trade, or about the compliance of a given use.

What machine should you plan for?

A model is a large file of numbers. Quantization lowers the precision of each number so the file gets smaller. The Hugging Face documentation on the GGUF format gives the bits per weight for each type: about 4.5 for Q4_K, 6.56 for Q6_K, 16 for F16.

A simple calculation gives a rough idea of the file size (parameters × bits per weight ÷ 8). This is our calculation from those tables, not a measurement:

Model In F16 In Q4_K
7 billion parameters about 14 GB about 3.9 GB
14 billion parameters about 28 GB about 7.9 GB

The memory actually used is higher than the file size: it also depends on the context length, the engine and the settings. Quality also drops as quantization gets heavier, by an amount that depends on the model and the task. These figures are for ruling out impossible cases, not for sizing a purchase: test first on the machine you already have.

Do you need a local LLM? Seven questions to decide

Ask these for the specific task you have in mind, not for “AI” in general.

Question If the answer is… Then
1. What data must the model read? personal or confidential local is worth serious study
2. Would an online tool do, on texts with no sensitive data? yes local is not necessary
3. What level of quality do you need? fine writing, long reasoning test first: small models sometimes disappoint
4. How many people will use it? several, at the same time plan a server, not a single computer
5. Who installs, updates and troubleshoots? nobody do not start
6. Who proofreads answers before use? nobody do not start
7. How will you know it works? no criterion write twenty real questions and the expected answer

If you answer “no” to question 1 and “yes” to question 2, an online tool is probably simpler. If questions 5, 6 and 7 have no answer, the project is not ready, whatever the hardware.

What do the CNIL and the GDPR say?

Two points are enough to decide whether to look into local.

  • A CNIL recommendation, in context. In its page on deploying generative AI, the CNIL recommends favouring “local, secure and specialised systems” (our translation of the French). That page gives first answers on deployment; it does not detail what on-site hosting requires, nor security measures, and it leaves out the design and training of models.

  • Security, wherever the system runs. Article 32 of the GDPR asks the controller to put in place technical and organisational measures appropriate to the risk. It does not require local processing, and local processing does not replace those measures: named access, logs, backups, retention period.

These pages do not cover every case. For health, client or employee data, define the purpose and the legal basis with your data protection officer or a lawyer before launching anything.

What does a reasonable first test look like?

Here is a fictional example, with no real client. A company wants to summarise internal meeting notes. Instead of buying hardware, it proceeds like this:

  1. Build the test set. Twenty fictional or anonymised reports, each with the summary a person would write.
  2. Install a small model on the existing machine with one of the tools above, offline if the constraint is strict.
  3. Compare. Note omissions, errors in figures and names, and response time. Do the same with the online tool currently used.
  4. Decide only afterwards, with the results in hand: stay with the online tool, use a larger local model, or change nothing.

This illustrates a method, not a measured result. Real differences depend on the model, the language, the machine and the documents.

A person checks two candidate answers against an original document, marking a discrepancy with a blue pencil.
Illustration of a proposed comparison: check omissions, names, figures and response time. No benchmark result or winning model is shown.

What are the limits of a local LLM?

  • Quality is not guaranteed. The models you can run on a computer are often smaller than the largest online models. Errors, omissions and wrong answers stated with confidence remain possible: human proofreading is needed.

  • You need hardware and maintenance. Memory, possibly a graphics card, updates of tools and models, and someone to troubleshoot. Local is not always free: hardware, electricity and time all count.

  • Model licences vary. The engine’s licence says nothing about the weights; some restrict commercial use.

  • Local is not a compliance certificate. It reduces one risk (sending data to a third party) but does not replace informing people, security or the advice of your data protection officer.

  • Shared use adds complexity. A model on a personal computer cannot be shared without network setup, access rights and security.

How does Castel Studio work on local AI?

Our Private AI page groups our offers, including transcription without the cloud from 90 € per batch (place of processing, access and retention agreed before any files are sent), an assistant that knows your business from 790 €, and a workstation built for your work by written quote. Hosting, model or API costs, electricity and maintenance are agreed separately; we agree written tests, and we do not guarantee results.

To compare options before choosing, read our guide to private AI for business, which sets out ten questions to put to a provider. If your need is rather to automate a task, see also our article on AI agents for a small business and on PDF invoices to Excel.

Describe the task, the type of documents and the hardware you have in a few lines, without attaching sensitive files through the form. We answer in writing within 48 hours, including to tell you that an online tool is enough in your case.

Questions

What is a local LLM?

A language model whose weights are downloaded and run on your own hardware. For "local" to be accurate, the whole chain must be local: no external API, no cloud fallback, no telemetry and no automatic folder synchronisation. An application installed on a computer does not guarantee that by itself.

Is a local LLM GDPR compliant?

Not automatically. Keeping data on your machines avoids sending it to a provider, but you still need to handle purpose, information to people, access, retention and security. The GDPR does not require local processing: it asks for measures appropriate to the risk.

What machine do you need for a local LLM?

It depends on the model, its quantization and the context length. As a rough guide, a 7-billion-parameter model quantized at about 4.5 bits per weight is roughly a 4 GB file; the memory actually used is higher. Test on your own hardware before buying anything.

Does a local model match ChatGPT or Claude?

There is no general answer. The models a small business can run are often smaller than the largest online models. Test on twenty real questions from your own work, and compare with the tool you use today.

How much does it cost to set up local AI?

At Castel Studio, the Private AI page lists offers from 90 € (transcription without the cloud, per batch) and an assistant on your documents from 790 €. A workstation is priced by written quote. Hardware, electricity, maintenance and proofreading time are extra.

Contact

One person reads this inbox — the same person who builds what you order.

Describe your task

Tell us what you have and what result you need — even a single small task. No brief required: within 48 hours you get a plan and a price in writing, a question, or an honest “not a fit”.

Include the situation, files or links, deadline, language, privacy limits and budget boundary if you have one.

File (optional)
No file chosen

A brief, a photo, a spreadsheet — whatever explains the task faster than words. Up to 8 MB, stored with our hosting provider. Please don't send sensitive personal or medical documents through the site.

Sending creates no order and no payment. We reply within 48 hours. Prefer email? contact@castel.studio

Talk to us

Prefer a call or a messenger? Write or call — the studio's founder answers.

Connect on LinkedIn

Prefer a freelance platform?

Order the same work there, with the platform's secure payment.

A one-person AI studio in Nice. Whoever answers your message is the one who does the work — no account managers in between. Files are processed on our own machine, not in someone else's cloud.

Nice 06100, France · Open in maps →

SIREN 103 115 705

Describe your task