Ever since the explosion of ChatGPT over a year ago, almost every company I talk to is setting up RAG architectures. But we always hit the same legal wall: sending confidential company documents to OpenAI's API is an unacceptable privacy risk for many corporate clients.

We needed a way to run artificial intelligence on our own servers. A few years ago, trying to run a model with billions of parameters required a cluster of NVIDIA A100 graphics cards costing hundreds of thousands of euros. This weekend, I ran a state-of-the-art model on my humble laptop thanks to the Open Source ecosystem.

Quantization and the Miracle of Ollama

The technical secret that makes a giant model fit into the RAM of a normal laptop is called Quantization.

When Meta or other companies train models, the neural network's weights are saved in 16-bit floating point (FP16). That takes up a massive amount of space. The open-source community discovered that if you compress those numbers down to 4-bit integers (a format called GGUF), the model loses a minuscule fraction of precision, but drastically reduces its memory footprint. An 8-billion parameter model goes from requiring 16 GB of RAM to fitting comfortably into 5 GB.

The tool that makes this ridiculously easy is called Ollama. You no longer need to compile C++ code; it's as simple as using Docker:

# Instalamos ollama y bajamos el modelo (como las filtraciones o versiones tempranas de Llama 3)
ollama run llama3

# Y si queremos exponerlo como una API local (100% compatible con la sintaxis de OpenAI)
curl -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3",
    "messages": [{"role": "user", "content": "Escribe un script de Python"}]
  }'

Reflection: The Revenge of Open Source

Seeing the cursor blink in my laptop's terminal, generating coherent text without being connected to the internet, got me incredibly hyped.

The history of computing is a pendulum swinging between centralized and decentralized. Giants like Google or Microsoft want us to depend on their closed APIs, charging us for every token. But the open-source community is advancing at an unstoppable speed. Just as Linux ended up crushing proprietary server operating systems, I am convinced that in a couple of years the corporate standard won't be connecting to ChatGPT, but rather having your own farm of small Open Source models, private, quantized, and running on your own local servers.