Every question typed into a cloud AI tool travels to someone else's computer, and so does whatever document rode along with it. For most businesses that is a reasonable trade, made with open eyes. For a law office holding case files, a practice holding patient records, or a manufacturer holding process documents worth protecting, it is the first question anyone in the room asks — and the answer decides what can be built.
The stack, in plain words
An on-premise deployment is three pieces sitting on a machine you own. The first is the model itself, running as a local endpoint: a program on a server in your building that takes a question and produces an answer, the same kind of engine behind the cloud tools, except the wire between question and answer never leaves the wall. Open-weight models — models whose files you can download and run yourself — now do genuinely useful work on hardware a small business can buy. The second piece is a vector index: a searchable catalog of your documents, built so that when a question comes in, the system can find the three paragraphs across your contracts, specs, and manuals that actually bear on it and put them in front of the model before it answers. The index lives on the same machine. The third piece is ingest: a pipeline that reads your files from wherever they already live — the office server, the practice management system, the job folders — and keeps the index current on a schedule. Ask a question and the search happens in the building, the answer happens in the building, and the citation points at your own file.
What "your hardware" means concretely
Not a data center. A workstation-class server with a capable graphics card, sitting in your office or in a rack you rent under your own account — either way, behind your keys, on your electricity bill, reachable by walking to it. That physicality sounds trivial and isn't: it is the entire difference between "our vendor's terms say your data is safe" and "the machine is in the room with the breaker panel, and here is the list of people with a key."
The trade, stated as a trade
Cloud AI is a subscription: someone else patches the software, capacity stretches when you need it, and your documents transit their systems under their terms — which are real contracts, and often good ones. On-premise inverts every one of those clauses. You pay capital up front for the box. Updates, backups, and hardware failure become your responsibility, which means they become a named person's responsibility, or they don't happen. In exchange, the data path becomes auditable in the plainest sense: the answer to "where do our documents go when we ask the system a question" is a room, not a policy document. That is the deal — subscription cost exchanged for hardware cost plus maintenance responsibility. Neither side of it is free, and the honest framing is not "which is better" but which cost structure and which risk your business would rather carry.
What it changes about the records conversation
For the practices and firms where records rules loom over every software decision, this architecture changes the shape of the conversation without settling it. Records stay on infrastructure you control — that is a real, describable property of the design. It is also a starting point for a compliance conversation, not a substitute for one. No architecture makes a business compliant with anything by itself, and the decision about what your obligations require belongs with your counsel or compliance officer, not with whoever built the server. What genuinely changes is that the questions get smaller and more answerable: who can reach this machine, what does it log, where do its backups go, who holds the keys. Those are questions about your own building, and you can answer them without taking anyone's word for anything.
What day two looks like
The build is the visible part; the ownership is the long part. New open-weight models release steadily, and upgrading means downloading files and then validating the new model against your own test questions — the download is free, the validation is work, and skipping it is how a quiet regression reaches your staff. Documents change, and while ingest runs on a schedule, someone has to own the fact that the price book on the server is the current price book, because the system will answer confidently from a stale one. Hardware ages like any shop equipment and should be budgeted like it. None of this is exotic. All of it has to belong to someone, in writing, before the box is ordered.
When not to do this
If the documents in question are marketing copy and public price sheets, the confidentiality case is thin, and a cloud tool under a decent vendor agreement is cheaper and simpler — buying a server to protect a brochure is theater. If nobody in the company can own the machine, and there is no maintenance contract to own it either, don't build it: an unpatched server slowly rotting in a closet is a worse risk than a reputable cloud vendor, not a better one. If usage would be occasional — a few questions a week — the hardware idles and the arithmetic favors a subscription without much argument. And on the hardest reasoning tasks, local models still trail the largest cloud models; for finding and summarizing answers out of your own documents the gap narrows considerably, but it exists, and pretending otherwise is how a good architecture gets blamed for a bad expectation.
Where to start
Write down the five most sensitive documents in your business and who is allowed to see them today. If the list is hard to produce, that is a finding in itself. Whether this architecture is worth its cost turns almost entirely on what is on that list — so the conversation should start from the list, not from the hardware.