A private, GDPR-compliant AI hub for a team of 12.

The team wanted GPT-4. The legal team wanted GDPR. The CFO wanted it affordable. Open WebUI + LiteLLM + Azure OpenAI made all three happy.

Flügger Group A/S - internal 2023–2024

Brief

A way to use frontier AI models inside a corporate environment - without leaking data, and without the per-seat costs that make a roll-out economically painful. Built on Open WebUI, LiteLLM and Azure OpenAI. Significant cost savings, healthier compliance posture, and a happier team.

Overview

When ChatGPT launched, the headlines that followed weren't all about productivity. There was, famously, the Samsung incident: confidential code pasted into a public chatbot, fed into training data.

"Leak" is a generous word - nothing crossed into another company. But for corporate environments, the idea that your conversations could end up training someone else's model is often a deal-breaker.

My colleagues felt the same. Many had tried ChatGPT, agreed it had potential, but were hesitant to use it on anything serious. And the best value came from GPT-4 - $25/month per head. For a team of 100, that's $30k/year before anyone has touched the keyboard.

The problem

1. The hype/risk paradox - those who'd benefit most from frontier AI are also the most exposed if it leaks. GDPR and confidentiality both fail loudly here.
2. Cost - paying per seat for the best models is hard to justify against the value most users will draw out of them.

Solution

Honestly, I had the answer before the question was fully formed. As a developer, I wanted both the best models and the freedom to swap when a new state-of-the-art arrived. Without breaking my employer's bank.

The pieces fit together like Lego. After a weekend of hacking I had something that was secure, cost-effective, and easy to maintain. Three pieces:

Why these pieces

Open WebUI replaced Hugging Face's Chat UI, which I'd started with - felt more tuned to my use case. LiteLLM beat out Portkey Gateway for self-hosting: one config file holds all the API keys, so the request just needs a model name. Portkey is great too - better if you're hosting on the edge (Cloudflare Workers, say) - but requires the client to ship endpoint and key headers per request.

Azure OpenAI got us a custom data handling agreement easily, with Gemini on Vertex AI later joining the same model. Both crucial for a Danish company adhering to European standards.

How it was done

Once the pieces were connected, the chat interface sat on our own servers, behind an internal subdomain. Users could hot-swap between models per message - Claude for writing, Gemini for long context, GPT-4 for everything else. For anything touching confidential data, the Azure-backed models were the default.

Worth noting: in the very early days, GPT-4 access through Azure was more exclusive than direct OpenAI access. That gave us an operational edge - especially for the color pages project.

Implementation challenges

1. Integration complexity - wiring Open WebUI, LiteLLM and Azure OpenAI together cleanly took careful configuration and some troubleshooting.
2. Authentication - making sure only authorized employees could reach the chat interface took its own pass.
3. Model selection - choosing the right mix of models for performance vs cost required real testing, not just vibes.
4. Compliance review - each component's data handling needed legal sign-off.
5. Onboarding - even a beautiful tool fails if people don't know how to use it well, especially with sensitive data. Materials and sessions, not just a Slack post.

Results

As I wrote this in my last month at Flügger, I learned it was perhaps the most-loved little piece of software I set up. The brief from the team was clear: keep it running. It had run without a hitch for six months at that point.

234+ chats, 1.5M+ tokens processed - and that's only what survived undeleted chats. Twelve people using it routinely. Around $555/month saved versus everyone having a paid sub - plus the ability to switch between models like Gemini 1.5 Pro (2M token context) or Claude 3 Opus (exceptional at writing).

DIY approach

If you want to replicate this:

1. Get Open WebUI - grab the latest from the GitHub repo.
2. Set up LiteLLM - follow the Azure provider docs.
3. Configure Azure OpenAI - reach out to Microsoft for a custom data-handling agreement. One-time, through the Azure Portal.
4. Connect the dots - drop your Azure API key and endpoint into LiteLLM. Point Open WebUI at LiteLLM. Done.

Links

Chat UIs:

• github.com/open-webui/open-webui
• github.com/huggingface/chat-ui

Proxies:

• github.com/BerriAI/litellm
• github.com/Portkey-AI/gateway

Azure opt-out from data collection:

• Azure OpenAI data privacy docs