Earlier this year, an organization called OpenAI (initially founded to democratize AI) announced a language model called GPT-2 with 1.5 billion parameters. They showed examples of machine-generated text—like an article about unicorns discovered in the Andes—that were so coherent they gave you chills.

But what made the most noise was their decision not to publish the full model. They claimed it was "too dangerous" because it could be used to generate fake news on an industrial scale. Since then, they have been drip-feeding smaller versions, and the debate in the technical community is blazing. Is it really an existential danger or is it the most brilliant marketing campaign in Silicon Valley history?

Predicting the next word

Technically, GPT-2 is a massive evolution of the Transformer architecture we talked about a couple of years ago. While the original Transformer was invented to translate (using an Encoder to read and a Decoder to write), GPT-2 uses a Decoder-only architecture.

Its only goal, its only mathematical metric during the massive training with 40 Gigabytes of internet text, has been to statistically predict what the next word (or token) is. Nothing more.

The model doesn't have a database of facts, nor does it "understand" what a unicorn is. It has simply internalized the statistical structure of English so well that, if you give it a starting sentence (prompt), it applies its weights to choose the most probable token.

At the code level, generating text with HuggingFace's transformers library on the trimmed-down versions they did release looks something like this:

from transformers import GPT2LMHeadModel, GPT2Tokenizer

# Cargamos el modelo (versión pequeña liberada) y su tokenizador
tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
model = GPT2LMHeadModel.from_pretrained("gpt2")

# Le damos un inicio (prompt)
texto_inicial = "In a shocking finding, scientist discovered a herd of unicorns"
inputs = tokenizer.encode(texto_inicial, return_tensors="pt")

# Le pedimos que prediga los siguientes 50 tokens
salida = model.generate(inputs, max_length=50, do_sample=True, top_k=50)

print(tokenizer.decode(salida[0], skip_special_tokens=True))

Reflection: The common sense barrier

When you try GPT-2, you notice it writes better than many humans (which is easy even for a monkey), but it is prone to suffering "hallucinations". It can state a false fact with absolute certainty because it sounds statistically correct. It doesn't have a logical model of the world, just a statistical model of our language.

Therefore, I think the fear of it being an "Artificial General Intelligence" (AGI) is pure marketing. However, its ability to spit out thousands of syntactically perfect fake tweets or articles is a real risk for disinformation on social networks. But not because Artificial Intelligence is bad, no, it's because people lack critical thinking or are simply idiots.

What fascinates (and terrifies) me about GPT-2 is that there's no theoretical revolution over the previous model; they've simply thrown more layers, more GPUs, and more data at it. If performance continues to scale linearly just by injecting more hardware... where is the ceiling?