GPT, LLM, Tokens, and Prompts Explained Simply

Language Models: From Basic Principles to Practical Application

In dialogue, a modern language model often appears to be a knowledgeable conversational partner. Technically, it is initially a very large computational model that has learned statistical relationships between text segments from examples. This gives rise to astonishing capabilities: explaining, summarizing, translating, programming, comparing, and structuring.

It is important to distinguish between three levels: The model contains learned patterns. The application adds the interface, search, files, tools, and rules. The human formulates the goal, provides context, and reviews the result. ChatGPT, Claude, Gemini, Qwen, or local models are therefore not simply interchangeable „answer machines,“ but rather different combinations of model and system.

Training, Inference, and Fine-Tuning

During training, a model sees a vast number of examples and adjusts billions of numerical values – the parameters – to better predict missing or subsequent text segments. The text is not stored as a lexicon. The model condenses recurring structures: grammar, semantic relationships, styles, typical arguments, and parts of world knowledge.

Inference refers to the subsequent use: From a prompt, conversation, and provided sources, the finished model calculates an output step by step. This is significantly more cost-effective than training, but can still demand considerable memory and processing power for large models. In fine-tuning, a pre-trained model is specialized in style or behavior using additional examples. For current enterprise knowledge, RAG is usually better because documents remain interchangeable and sources visible.

Tokens and Embeddings: How Text Becomes Numbers

A model does not read words, but tokens. These can be words, word parts, punctuation marks, or spaces. For example, „Künstliche Intelligenz“ (Artificial Intelligence) can be broken down into several tokens. Each token is represented as a numerical vector. Similar uses are closer to each other in this mathematical semantic space.

An illustrative idea: On a huge map, „Hund“ (dog), „Katze“ (cat), and „Tier“ (animal) are in the same region, while „Rechnung“ (invoice) and „Buchhaltung“ (accounting) are in another. An embedding is the coordinate on this map. Semantic search uses precisely this principle: it finds semantically relevant sections, even if the query uses different words.

Transformers and Attention Explained Simply

The Transformer, introduced in 2017, is an architecture that weights relationships between all relevant tokens in a context. Attention here does not mean attention in the human sense. It is a computational operation: For each text segment, it determines which other segments are particularly important for its meaning.

Example: „Die Bank am Fluss war nass.“ (The bank by the river was wet.) For the token „Bank,“ „Fluss“ (river) and „nass“ (wet) receive a high weight. In „Die Bank genehmigte den Kredit“ (The bank approved the loan), „genehmigte“ (approved) and „Kredit“ (loan) become more important. Multiple attention heads simultaneously consider different relationships – such as grammar, reference, temporal order, or meaning.

Many Transformer blocks process these relationships sequentially. Positional information ensures that the order is preserved. Feedforward layers further transform the intermediate representations. GPT therefore stands for Generative Pre-trained Transformer: generative, pre-trained, and built on a Transformer architecture.

Parameters, Context Window, and Temperature

Parameters are learned numerical values. More parameters can represent more patterns, but they don't automatically guarantee better answers. Training data, architecture, fine-tuning, tools, and task alignment are equally important.

The context window includes the prompt, conversation, files, and often tool results that the model can consider when generating a response. More context isn't always better: unimportant or contradictory information can obscure relevant points. Good systems therefore selectively choose appropriate sources.

Temperature influences the selection probability of possible next tokens. Lower values usually lead to more stable, similar outputs; higher values lead to more variation. However, temperature does not make a statement truer or more intelligent.

Why Language Models Hallucinate

A language model primarily optimizes for a plausible continuation, not for truth. If reliable information is missing, it can convincingly add names, sources, functions, or numbers. This is not a single software error but a consequence of the generation principle.

Countermeasures include: providing original sources, using search or RAG, explicitly allowing for uncertainty, breaking down statements into individual steps, and verifying critical data outside the model. For legal, medical, financial, and security matters, expert review remains indispensable.

Reasoning, Tools, and RAG

A "reasoning model" does not mean that a system thinks like a human. It receives more computation time or has been trained to break down complex tasks into intermediate steps and verify results. A model becomes particularly effective through tools: web search, calculators, code execution, databases, or APIs.

RAG adds its own sources. The model then doesn't have to guess from its training but answers the question using relevant document sections. An agent goes further and can select tools itself within defined limits.

From Weak Prompts to Reliable Work Orders

Instead of "Write something about AI," a reliable work order is more specific: "Explain the difference between a language model and a search engine for business executives without technical knowledge. Use two examples, state limitations, and mark statements that require a current source."

For larger tasks, a process is suitable: define the goal, provide materials, state criteria, generate a draft, request a cross-check, and professionally approve the result. Examples of useful roles are editor, critic, and reviewer – not because the model becomes a person, but because different review perspectives are activated.

Local Language Models, Quantization, and Memory

Local models run, for example, via Ollama, LM Studio, or MLX on your own hardware. They offer control over data and running costs. Model size alone says little: quantization reduces the precision of weights so that a model requires less memory. A 4-bit version is significantly smaller than 16-bit but may lose some quality.

On Apple Silicon, the CPU and GPU share unified memory. This enables large models without a separate graphics card. However, memory is also needed for context, intermediate results, and other programs. A smaller, well-suited model can be faster and more economical than a very large universal model.

Where Are Language Models Heading?

The development is moving from pure chats to multimodal systems that process text, images, audio, and video together. Models are becoming more strongly integrated with tools, reliable sources, and persistent workflows. Simultaneously, smaller specialized models, quantization, and local execution are becoming more important.

It is likely not "one model for everything," but rather a system comprising routers, specialized models, search, rules, and human approvals. Key factors will be traceable sources, measurable quality, energy and cost control, and the ability to swap models without rebuilding the entire process.

Further Knowledge and Courses

We explain visual models and diffusion in Vision Models and Generative Media. Our course RAG, APIs, AI Agents, and Local Models shows how embeddings, RAG, and agents are integrated into systems.

Access the Three-Hour AI Fundamentals Course Request Training