What is Ollama?
Ollama is open-source software for running open language models on your own hardware. It downloads models, starts them and provides a local interface that can be connected to chat front ends, document search and your own applications. Requests are processed on your own infrastructure and are not sent to an external AI service.
Three ways to run Ollama
- <b>On existing hardware:</b> A suitable server or workstation in the company is used. Sensible for tests and small models.
- <b>On newly purchased hardware:</b> A server with a suitable graphics card is bought and sized specifically for AI. Sensible for permanent use and high demands on control.
- <b>On GPU cloud servers at Hetzner:</b> Computing power is rented without buying hardware. Location, availability and terms should be checked before deciding.
Typical use cases
- Internal assistant for texts, summaries and translations without data leakage.
- Search and questions on your own documents (retrieval-augmented generation).
- Support for documentation, tickets and knowledge bases.
- Experiments and proofs of concept in the INZOSO Labs before investing.
Benefits and limits
Benefits include control over data and models, no tie to a single provider and predictable costs. Limits lie in hardware: quality depends on model size and graphics memory, and large models need powerful servers. Operations, updates and security must be handled by you or a service provider. Open models have their own licence terms that should be read before use.
Data protection and security: check points
- Where do servers and data run, and who has access?
- Is the interface reachable only internally and protected by sign-in?
- What licence does the chosen model have, is commercial use allowed?
- Are inputs and answers logged, and for how long?
- For cloud servers: clarify data processing agreement, location and deletion concept.
How INZOSO helps with Ollama
We clarify use case and requirements, size the hardware or GPU server, install Ollama, choose suitable models, connect your documents and secure access. On request we take over operations and updates. To try it out, we first build a proof of concept in the Labs.