What is an LLM? Large language models explained (2026)

A large language model (LLM) is a neural network trained on enormous amounts of text to predict the next token in a sequence. That single objective is the source of everything you see a chatbot do: answering questions, writing code, summarising documents and translating. Tools such as ChatGPT, Claude and Gemini are built on models of this kind.

This guide covers the building blocks in order: tokens, training, parameters, context windows, hallucination, and the split between open and closed models. Model names, prices and limits change quickly. The figures below therefore carry dates, so check the vendors' own documentation for current numbers.

What is an LLM, and what makes it “large”?

“Large” refers to two things. The first is the volume of training data. The second is the number of adjustable internal values, called parameters. To give a sense of scale, Meta said in April 2024 that Llama 3 was trained on more than 15 trillion tokens.

Nearly every modern LLM is built on the Transformer architecture. Google researchers introduced it in the June 2017 paper “Attention Is All You Need.” Its core idea is self-attention, which lets the model weigh how relevant every other part of the text is when it processes each piece.

What are tokens in an LLM?

Models don't read words. They read tokens, which are subword chunks produced by algorithms such as byte-pair encoding (BPE). A common word might be a single token, while a rare one gets split into several pieces.

OpenAI's rule of thumb for English is that one token is about four characters, or roughly three-quarters of a word. On that basis, 100 tokens comes to about 75 words.

The ratio differs by language. Agglutinative languages such as Turkish, where long words are built from many suffixes, are usually split into more tokens than English text with the same meaning. Research such as Petrov et al. (2023) has documented this kind of tokenizer unfairness between languages, although newer tokenizers have narrowed the gap.

Tokens matter because API usage is typically billed per million input and output tokens, with output priced higher than input. More tokens mean higher cost and a context window that fills up faster. Prices change often, so check the OpenAI, Anthropic or Google pricing pages for current rates.

How are large language models trained?

Training usually happens in stages. The first is pre-training. Here the model learns by predicting the next token across trillions of tokens of web text, books and code, a method known as self-supervised learning. The result understands language but doesn't yet behave like an assistant.

Next comes fine-tuning, or instruction tuning, in which the model is shown examples of good answers. Then comes alignment. The best-known alignment method is reinforcement learning from human feedback (RLHF), popularised by OpenAI's 2022 InstructGPT paper. Anthropic's Constitutional AI, also from 2022, is a variant of this approach.

Since around late 2024, reasoning models have joined the mix. Examples include OpenAI o1 and DeepSeek-R1, released in January 2025. These models are trained with reinforcement learning to work through a problem before answering, and they spend extra compute at answer time to do so.

Training costs are hard to pin down. Frontier runs are reported to cost tens to hundreds of millions of dollars, but those are estimates rather than official figures. The widely quoted figure of about $5.6 million for DeepSeek V3 covers only the GPU hours of the final training run, not total costs, and it is disputed.

What are parameters, and does bigger mean better?

Parameters are the numerical weights a model learns during training. Some reference points: GPT-2 had 1.5 billion in 2019, and GPT-3 had 175 billion in 2020. Meta's Llama 3.1, released in July 2024, came in 8B, 70B and 405B sizes.

Some models use a Mixture-of-Experts (MoE) design, in which only part of the network runs for each token. DeepSeek-V3 has 671 billion parameters in total, but only about 37 billion are active per token.

OpenAI, Anthropic and Google don't disclose parameter counts for their flagship models. Figures such as “GPT-4 has 1.76 trillion parameters” are unconfirmed leaks.

Size also isn't everything, because data quality and training methods matter a great deal. The 2022 “Chinchilla” study found that compute-optimal training uses roughly 20 tokens per parameter. Today's models are deliberately trained well beyond that ratio.

What is a context window?

The context window is the maximum number of tokens a model can consider at once. It includes your prompt, the conversation so far and the model's reply. Think of it as working memory rather than long-term memory.

Windows have grown fast. GPT-3 handled 2,048 tokens, and the original GPT-4 offered 8K and 32K versions. Later releases advertised 200K for Claude (up to 1M in some versions), 1–2 million for Gemini 1.5 and 2.x, and 1 million for GPT-4.1. In April 2025 Meta claimed 10 million for Llama 4 Scout. These numbers shift with every release, so check current documentation.

A big advertised window doesn't guarantee the model uses all of it well. The 2023 “Lost in the Middle” study by Liu et al. found that accuracy can drop for information buried in the middle of long inputs.

Why do LLMs hallucinate?

A hallucination is output that sounds fluent but is false or invented, such as fake citations, wrong facts or non-existent court cases. It happens because the model generates plausible text instead of looking facts up. A September 2025 OpenAI paper argues that training and evaluation reward guessing over admitting uncertainty.

A well-known case is Mata v. Avianca (US, 2023). Lawyers in that case were sanctioned for filing case citations that ChatGPT had made up, and many similar incidents have followed.

Retrieval-augmented generation (RAG), web-search grounding, citations and reasoning models all reduce the problem, but none eliminates it. OpenAI's April 2025 system card for o3 and o4-mini even reported higher hallucination rates for some reasoning models on its PersonQA test. Rates vary by benchmark and are disputed, so treat single-number claims with caution.

Open vs closed LLMs: what's the difference?

With closed, or proprietary, models the weights are not released, so you can only use them through an app or API. OpenAI's GPT series, Anthropic's Claude, Google's Gemini and xAI's Grok fall into this group. xAI has, however, released some older Grok weights.

Open-weight models can be downloaded, run locally and fine-tuned. Examples include Meta Llama, Mistral, DeepSeek, Alibaba Qwen, Google Gemma and OpenAI's gpt-oss, released in August 2025 under Apache 2.0.

Open-weight is not the same as open source. The Open Source Initiative's Open Source AI Definition 1.0, published in October 2024, requires much broader openness, including about training data. The OSI says Llama's licence does not qualify as open source.

In practice, open-weight models offer privacy, local control and customisation. Closed models have usually led on raw capability, though the gap narrowed during 2025. Leaderboards such as LMArena reshuffle month to month.

Regulation is also catching up. Under the EU AI Act, obligations for general-purpose AI models have applied since 2 August 2025. Data protection law, such as Türkiye's KVKK, still covers any personal data you paste into a prompt.