A true story. The director of an accounting and audit firm sat across from us with a draft contract in her hand. She was blunt: her data was client financial records, employment contracts, payroll. There was no way she was pushing any of that to some overseas provider’s API.
Then she asked the question we hear a lot: “So can we just run it on-premise?”
Yes. But running it on-premise isn’t the whole answer. The real question sits somewhere else: which parts should stay inside the building, which parts can go out, and who is accountable when the system gets something wrong.
This is what we took away from a few projects like hers — about running a local small language model (SLM), about multi-agent systems, and about wiring both into CRM and ERP.
Running a Local Small Language Model (SLM): When Is It Worth It?
In plain terms: an SLM is a language model that is much smaller than the cloud names you keep hearing about. Instead of hundreds of billions of parameters sitting on somebody else’s cluster, you have something compact, good enough for a specific set of jobs, running on a machine you control.
A couple of terms, one sentence each: quantization is just a way of compressing the model’s weights so it fits on modest hardware. You’ll often see the result as GGUF files, run with tools like llama.cpp or Ollama — install one, and you have a model answering on your own machine, no internet required.
So when does local win?
- Sensitive data never leaves the firewall. Patient records, legal files, payroll, unsigned contracts. Not because the cloud is bad — but because there are documents you don’t want to have to explain to anyone why they’re sitting somewhere else.
- Costs you can predict. You buy the machine once. A hundred requests a day or a hundred thousand, the machine costs the same. No per-token invoice, no end-of-month shock.
- It keeps working when the network doesn’t. Factories, warehouses, provincial branch offices. Internet going down for a few hours is an ordinary Tuesday for plenty of teams.
- Low, steady latency. Your agent’s thinking speed isn’t decided by someone else’s international routing.
- Nobody else decides for you. A vendor changes pricing, changes terms, retires a model version — you don’t have to chase it.
And here’s what you give up, plainly:
- Weaker reasoning. Multi-step work, long chains of logic, precise arithmetic — bigger models still win. Don’t fool yourself.
- Hardware. A machine with a decent GPU, and someone who knows how to look after it.
- An operator. Updating, monitoring, fixing prompts, evaluating output. A local SLM does not get better by itself.
- Quality depends on the context you feed it. An SLM doesn’t “just know”; you have to hand it the right documents, the right examples, the right output shape.
The self-check question: is this work classification, extraction, summarization, or answering from a known template? If yes, a local SLM is often enough. If it needs multi-step reasoning over messy data, that’s the big model’s field.
Multi-Agent AI Systems: Splitting Work Across a Team
One agent doing everything sounds convenient, until it gets something wrong and you can’t tell where. A multi-agent AI system splits the work the way a real team does: one coordinator agent, plus a handful of specialists.
The coordinator holds the big picture. It takes the request, breaks it down, decides who to call, and knows when to hand things to a human. Each specialist has its own toolbox, its own prompt, and its own permissions.
A concrete picture: an enterprise customer sends a quote request.
- A research agent digs through purchase history, looks up the company, and writes a short brief.
- A customer service agent reads that brief, drafts a reply in the company’s voice, and asks for the two details that are still missing.
- An accounting agent checks the receivables data before anyone quotes a price, because this customer has an unpaid invoice.
- All three report back to the coordinator, which merges the results and sends them to a salesperson for approval.
No single agent has to know everything. That’s the real benefit: smaller toolboxes are easier to test, narrower permissions mean a mistake does less damage, and each call carries only the context it needs — so it’s cheaper.
But multi-agent isn’t always right. If your process is a straight line — take input, process, return — then one agent with a few tools is enough, and far easier to reason about.
Sound familiar? Every team has added headcount to a job one person could have done. With agents, the price is: multiplied cost, context lost at every handoff, and sometimes two agents politely passing work back and forth. A multi-agent setup earns its keep when there are genuinely many kinds of work, not when there’s a lot of work.
Pairing local SLMs with multi-agent systems: cost and privacy
This is our favorite part, because it solves both problems at once.
You don’t have to pick one or the other. In a multi-agent AI system, each agent can run a different model — and that’s exactly where the money gets saved and where the data stays home.
Here’s the pattern we usually land on:
- Use a local SLM at the front door. Classify requests, extract fields, answer simple questions, draft replies from templates. Most of the traffic lives here, and it runs on your machine.
- Send only the hard work to a bigger model. Requests that need deep reasoning, or long-document reading. Strip identifying details before anything leaves.
- Let the coordinator be local too. Routing work and calling the right specialist doesn’t require the smartest model in the room.
- Write down what never leaves. Put it in a document, not in someone’s head. And log every handoff.
The result is a predictable baseline cost, plus cloud spend reserved for the genuinely difficult tail.
AI Agents Integrated With CRM and ERP
If you’ve tried connecting AI to internal systems before, you know the feeling: the demo is great, but everything stops at “suggestion”. Someone still opens the CRM and retypes it.
AI agents integrated with CRM and ERP are different because they write. The call ends, and the agent has already logged notes and three next steps on the deal in HubSpot. Stock drops below threshold, and it raises a purchase request in the ERP. An invoice arrives, and it matches it against the purchase order and posts the entry to the accounting system, such as MISA or Xero.
Reading is nice. Writing back is where the time actually returns, because writing is the part humans are doing by hand.
Writing is also the risky part, so we keep a few rules:
- Controlled writes. Anything touching money, customer records, or deletion goes to a human for approval. Everything else can run on its own.
- A trail. Who wrote what, when, and based on which source — all retrievable.
- No duplicates. ERP systems hate a double-posted invoice.
- Start with reads. Master data in ERP is rarely as clean as people assume. Let the agent read and reconcile first, then graduate it to writing.
Back to that accounting director: she didn’t need a system smarter than she is. She needed one that doesn’t leak data and takes the typing off her desk.
Where to start?
Don’t start with “digitize everything”. Start with one process, one obvious pain point, measured before and after.
Three questions worth answering before you write any code:
- Is the data flowing through this sensitive? If so, which parts have to stay on machines you control?
- Is this a straight line or a set of branches? One branch means one agent is enough.
- Where is this system allowed to write, and who approves it?
At Mon AI we work with teams that want to start exactly there — sometimes it’s just a shared inbox, sometimes it’s invoice reconciliation. If you’re weighing up whether to run things on-premise, talk to us. You can also try one yourself on our demo page.
Can it run on-premise? Yes — if you’re clear about which parts belong inside the building.
