A public AI assistant on a company website, answered by a local model

Most companies want an assistant on their website, and most are told the only practical way is a cloud API - which means every visitor question, including the sensitive ones, leaves the company. We wanted the opposite as a live demonstration: a public chat on a public site where nothing leaves the network, and where a visitor can test the claim by asking.

Public means adversarial. The assistant had to survive misuse without an operator: per-visitor rate limits, a cap on conversation length, a hard message size, and no way to extract more than the company chooses to say. It had to respect GDPR by design: no cookies, visitor addresses stored only as salted hashes, and a visible notice that conversations are kept. And it had to stay responsive on hardware that also runs other workloads.

The browser talks only to the site's own origin. Behind it, a FastAPI gateway assembles each request: a maintained knowledge file with the company's positioning, delivered work and the rules on what may not be discussed, the last six exchanges of the conversation, and the visitor's message. The model is a 35B mixture-of-experts open-weight model on an Apple-silicon inference node; if it does not answer within five seconds, the gateway fails over to a GPU node running a 27B model, and a failure after streaming has begun is reported to the visitor instead of hanging.

Answers stream token by token with the model's hidden reasoning filtered out. Because the same inference node also runs batch research jobs, every accepted chat request raises a priority flag that batch workers honour before each model call - the visitor is never queued behind a background task. Every conversation is stored for review, so the knowledge file improves from real questions rather than guesses.

Live on annatech.pl since July 2026, at the bottom-right of every page. The same gateway pattern - knowledge injection, limits, failover, audit trail - transfers directly to an internal helpdesk, an intranet assistant or a customer-facing chat where the questions must stay in the building.