OpenAI
APIGPT · o-series
The default general model and the reasoning line; the widest product and developer ecosystem.
Everything else — the conversation, the code, the apology when it gets something wrong — is engineering built on top of that one operation, repeated a few hundred times a second.
This page takes that sentence apart. What a token is, how meaning gets turned into numbers, what attention computes, what training actually changes, and why a system this capable still confidently invents things.
Three of the sections are instruments rather than prose. Type in them, drag them, break them. The concepts are exact; where a demo uses stand-in numbers instead of a live model, it says so.
One view of what's happening right now — headlines, fresh research and what people are arguing about — pulled from public sources on the server and refreshed on its own, so you're never more than an hour behind.
One page, once a day: the headlines that mattered, the research worth a look, and the new models people are actually trying — drawn from the live newsroom and the Hugging Face feed, with a short plain-language note on what it adds up to. Refreshed every morning, Malaysia/Singapore time.
Loading today's digest…
Double opt-in: you'll get one confirmation email first. Every digest has a one-click unsubscribe link. Your address is stored only to send this digest — nothing else, and never shared.
The field moves faster than any one page can. So ask. The server gathers fresh, public sources — Wikipedia, arXiv, the live newsroom, Hacker News — and has Claude tidy them into one plain answer, with links you can check yourself. Choose the depth that fits you.
Answers are AI-generated from the linked public sources and can be wrong — treat them as a fast first pass, then check the sources. Nothing you type is stored.
A chatbot that answers from this site's own knowledge — not the open web. Every page here is stored as a portable Open Knowledge Format wiki; the bot finds the relevant pages by keyword ranking and answers from them, citing the exact sections. No embedding model, no vector database — just the LLM-wiki method.
Answers are AI-generated from this site's pages and can be wrong — the cited sections are the source of truth. Nothing you type is stored.
Who builds the frontier, in one view. These are the families, not version numbers — the labs and their model lines endure; the specific releases churn every few weeks, and those live in the newsroom above. Colour marks how you get access.
GPT · o-series
The default general model and the reasoning line; the widest product and developer ecosystem.
Claude
Opus / Sonnet / Haiku tiers; known for long context, coding, and a safety-first posture.
Gemini
Natively multimodal with very large context windows, wired into Search, Workspace and Android.
Llama
The open-weight model that anchored the ecosystem — fine-tune it, run it on your own hardware.
Mistral · Mixtral
Efficient European models, several open-weight; early champion of mixture-of-experts.
V-series · R-series
Open models that showed frontier-level reasoning can be trained strikingly cheaply.
Qwen
A broad, strongly multilingual open-weight family; a common base for fine-tuned models.
Grok
Built around real-time data from X; positioned as the less-filtered assistant.
A stable map of the players, not a spec sheet — for the newest release from any of them, watch the newsroom. Access can change as labs open or close a line.
The map above is the who; this is what shipped lately. The server pulls the trending models from Hugging Face, sorts them into a few plain categories, and — where it can — has Claude add a one-line take on each. Refreshed a few times a day.
Three numbers tell the story of the last few years better than any headline: how much a model can read at once, how much compute went into training it, and how little it now costs to use. Each has moved by orders of magnitude — so these are drawn on a log scale, where every gridline is ten times the one below. Switch between them.
Figures are rounded to the nearest order of magnitude and gathered from provider documentation and public estimates (Epoch AI for training compute); they mark the trend, not exact values. The point isn't any single dot — it's the slope.
Before anything else happens, your text is chopped into pieces from a fixed vocabulary. Almost every strange behaviour you have noticed starts here.
A model has a vocabulary — typically somewhere between 32,000 and 200,000 entries — and text must be expressed entirely in those entries. Common words get one token each. Rarer words get broken into fragments. The algorithm is usually byte-pair encoding: start from raw bytes, then repeatedly merge whichever adjacent pair occurs most often in the training corpus, until you have filled the vocabulary. Frequency buys wholeness.
Leading spaces belong to the token, which is why the and the are
different entries. As a rough English rule: one token ≈ 4 characters ≈ 0.75 words.
Faithful reproduction of GPT-2/3-style pre-tokenisation (the regex that splits contractions, leading spaces, digit runs and punctuation), followed by a simplified merge step standing in for the real learned merge table. Segment boundaries and counts are close to a production tokeniser, not identical to it.
1234567 may arrive as several digit-group
tokens, so column alignment is something the model reconstructs rather than perceives.Token 3,290 means nothing. The model's first move is to swap that ID for a long list of numbers — and those numbers are where meaning lives.
The embedding matrix is a lookup table with one row per vocabulary entry. Each row is a vector of a few thousand numbers. Nothing about it is hand-designed; the values are learned during training, adjusted by the same gradient descent that shapes everything else.
What emerges is geometry. Tokens used in similar contexts end up pointing in similar directions, and direction can carry relationships. The famous early demonstration came from word2vec in 2013: take the vector for king, subtract man, add woman, and the nearest neighbour is often queen. Relationships like gender, plurality and capital-of behave, loosely, like consistent offsets you can add.
One more thing matters, and it is the upgrade the transformer brought. A raw embedding is static — one vector per token, identical everywhere. As the vector passes up through the model's layers it becomes contextual: the representation of "bank" in river bank genuinely diverges from "bank" in bank transfer. The mechanism that does the diverging is the subject of the next part.
Each position asks a question, every earlier position answers, and the answers get averaged in proportion to how well they match.
From each token's vector the model projects three new vectors: a query (what I am looking for), a key (what I offer), and a value (what I will hand over if chosen). To update a position, take the dot product of its query against every key, turn those scores into percentages with a softmax, and mix the values by those percentages.
Two structural details do a lot of work. A causal mask sets every future position's score to −∞ before the softmax, so a token can only ever look backwards — that is what makes the model a valid next-token predictor. And attention runs in parallel heads, each with its own projections, so one head can track subject–verb agreement while another tracks quotation marks.
Weights here are illustrative, hand-set to show patterns that interpretability research does find in real heads — long-range subject tracking, pronoun and relative-clause binding. A live model's heads are messier and mostly uninterpretable at a glance.
Attention's cost is the reason long context is expensive: comparing every position against every other is quadratic in sequence length. Double the context, quadruple the attention work. Most of the last few years of systems research — FlashAttention, sliding windows, grouped-query attention, KV caching — is about paying that bill more cheaply.
Here is the entire journey from your text to a single next token. Then it happens again.
Two things worth holding onto. First, the residual stream — each block adds its output back into a running total rather than replacing it, so information can skip layers untouched. This is what makes networks hundreds of layers deep trainable at all.
Second, every token costs the same compute. The model cannot think harder about a difficult token; it can only produce more tokens. That single fact explains chain-of-thought prompting: asking for step-by-step working is not a psychological trick, it literally buys more forward passes to compute in. Reasoning models are trained with reinforcement learning to do this well, and to keep doing it until the answer stabilises.
A base model and a chat assistant are the same network at different points in the same process. Knowing which stage does what tells you which problems are fixable.
Feed the model an enormous amount of text and ask it, over and over, to predict the next token. Compare its distribution to the token that actually came next, measure the gap with cross-entropy loss, and nudge every weight to make the truth slightly more likely. No labels are needed — the text labels itself, which is why this scales to trillions of tokens. This stage is typically upwards of 98% of the total compute, and it is where essentially all factual knowledge and language ability is acquired.
The output is a base model: fluent, knowledgeable, and useless as an assistant. Ask it a question and it may well continue with more questions, because that is what a page of questions usually does.
Now train on a much smaller, curated set of written demonstrations: prompt, then an ideal response. Thousands to hundreds of thousands of examples, not trillions. The model learns the format of being helpful — answer the question, stop when done, hold a turn in a conversation. It learns almost no new facts here.
Demonstrations cannot express "this reply is a bit too smug." So instead: show humans two candidate responses, ask which they prefer, and use those comparisons as the signal. In classic RLHF the comparisons train a reward model, and the language model is then optimised against it with reinforcement learning. DPO and its relatives skip the separate reward model and optimise on the preference pairs directly. Anthropic's Constitutional AI variant has the model critique and revise its own outputs against an explicit written set of principles, so a large share of the preference labels are generated by AI rather than by people.
This is the stage that produces tone, refusals, formatting habits, and hedging. It is also where sycophancy comes from: if raters reliably prefer agreeable answers, agreeableness is exactly what gets optimised.
The network's output is not a word. It is a probability for every word. Something then has to choose — and how it chooses is a dial you control.
The final layer emits one raw score, a logit, per vocabulary entry. Softmax turns those into probabilities summing to 1. Temperature divides the logits before the softmax: below 1 it sharpens the distribution toward the favourite, above 1 it flattens it toward the field. Top-p (nucleus sampling) then keeps only the smallest set of candidates whose probabilities add up to p, and discards the long tail entirely.
Prompt type: . The same two dials behave completely differently across these.
Those ten logits are illustrative stand-ins. Run the real distribution on GPT-2's own weights, in your browser →
Notice what the extremes do. Near zero, the model always picks its favourite — repeatable, and prone to loops and flat prose. Above roughly 1.3 the tail gets real probability mass, and text drifts from creative to incoherent. Top-p is the safety rail: it lets you raise temperature for variety while still refusing to ever sample genuine nonsense.
Then switch prompts, because this is the part that gets missed. On the factual prompt the dials barely matter — the evidence for Paris is so overwhelming that you have to push temperature past 1.5 before anything else gets a serious look, and top-p does nothing until it is almost at 1. On the open-ended prompt the same dials transform the output, because ten continuations were already close to tied. Temperature does not add creativity; it decides how much to respect a distribution the model has already committed to. Where that distribution is peaked, turning the dial mostly buys you errors rather than variety.
The decision to build very large models was not a hunch. It came from plotting loss against compute and finding a straight line on a log scale.
In 2020 Kaplan and colleagues showed that test loss falls as a smooth power law in model size, dataset size and compute — remarkably predictable across many orders of magnitude. That predictability is what made spending nine figures on a training run a defensible engineering decision rather than a gamble.
In 2022 the Chinchilla paper corrected the recipe. Given a fixed compute budget, the earlier generation had made models too big and trained them on too little data. Compute-optimal training, they found, scales parameters and tokens together — roughly 20 training tokens per parameter. The proof was direct: Chinchilla at 70B parameters outperformed Gopher at 280B, using the same compute budget spent differently.
| Model | Params | Training tokens | Tokens / param | What it showed |
|---|---|---|---|---|
| GPT-3 (2020) | 175B | ~300B | 1.7 | Scale alone produced few-shot learning |
| Gopher (2021) | 280B | ~300B | 1.1 | Bigger, similarly under-trained on data |
| Chinchilla (2022) | 70B | 1.4T | 20 | Beat Gopher at a quarter the size |
| Llama 3 8B (2024) | 8B | 15T | ~1875 | Deliberately trained far past optimal |
That last row is the part people miss. Chinchilla optimises training cost. If a model will serve billions of requests, inference cost dominates the budget, and it is worth over-training a small model well past the compute-optimal point to get a permanently cheaper one. Modern open models are trained on hundreds of times more data per parameter than the 2022 rule recommends — not because the rule was wrong, but because it answered a different question.
A second lever changes the arithmetic again: mixture of experts. Route each token to a small subset of many parallel feed-forward blocks, and total parameters can be far larger than the parameters actually used per token — capacity without proportional cost.
Each of these follows directly from something described above. That is what makes them predictable — and what tells you which ones tooling can fix.
The training objective rewards plausible, not true. There is no fact table inside the weights to consult and no internal signal that says "this next part is a guess" — a fabricated citation is generated by exactly the same machinery, at similar confidence, as a correct one. Fluency and accuracy are separate axes, and the model is optimised hard on one of them. Worse, saying "I don't know" is only produced if preference training rewarded it, which requires raters who can tell honest uncertainty from unhelpful hedging.
What helps: retrieval, so the answer is grounded in supplied text rather than recalled; tool use for anything computable; asking for sources you can check; and treating any specific number, name or citation as unverified by default.
The weights are frozen after training. The context window is working memory, and it is the only memory — start a new conversation and everything is gone. Products that appear to remember you are re-inserting stored notes into the prompt behind the scenes. Nothing you say is learned in the moment.
Show a model three examples of a format it has never seen and it will follow the pattern — with no weight update whatsoever. The pattern-matching happens inside a single forward pass, in the activations. This is why few-shot prompting works, and why its effects vanish the moment the conversation ends.
Base models are reasonably well calibrated — their probabilities roughly track how often they are right. Preference training tends to degrade that: optimising for answers humans like pushes toward confident phrasing regardless of underlying uncertainty. Assertive tone carries almost no information about reliability.
A model reading a web page, an email or a document cannot cryptographically distinguish your instructions from text that merely looks like instructions. That is the root of prompt injection, and it is a structural consequence of the fact that there is only one stream of tokens. Mitigations reduce it; nothing yet eliminates it.
Each of these is a reasonable inference from watching a model behave. Each is wrong in a way that changes how you should use one.
Short answers to the things that bring most people here. Each one links to the part of the page that works through it properly.
It converts your text into tokens, turns each token into a vector, passes those vectors through dozens of transformer layers that let every position read from earlier positions, and emits a probability for every entry in its vocabulary. One token is sampled from that distribution, appended to the input, and the whole process repeats. See the full trace →
A token is a chunk of text from a fixed vocabulary — usually a common word, a fragment of a rarer word, or punctuation. Models never see individual letters, only these chunks. In English one token averages about four characters, or roughly ¾ of a word. Try the segmenter →
Because the training objective rewards plausible text, not true text. There is no fact database inside to consult, and no internal signal marking a claim as a guess — a fabricated citation is produced by the same machinery, at similar confidence, as a correct one. Retrieval, tool use and checkable sources help; the tendency is structural, not a bug to be patched. Read why →
Temperature divides the model's raw scores before they become probabilities. Below 1 it sharpens the distribution toward the single most likely token; above 1 it flattens it so unlikely tokens get a real chance. It does not add creativity — it decides how strictly to respect a distribution the model already computed. Drag the dial →
A weighted lookup. Each position emits a query, every position offers a key and a value; the dot product of query against key becomes a percentage after softmax, and the values are mixed in those proportions. A causal mask stops any position reading the future. See the pattern →
No. The weights are frozen after training, and the context window is the only memory — it is cleared when the conversation ends. Products that appear to remember you are storing notes separately and re-inserting them into the prompt. More on this →
Use retrieval for knowledge — anything that changes, needs citing, or must be current. Use fine-tuning for behaviour — format, tone, task structure. Fine-tuning is poor at installing facts; it mostly teaches the model to recite a snapshot confidently. Why that follows →
The 2022 Chinchilla result put the compute-optimal ratio at roughly 20 training tokens per parameter — a 70B model on about 1.4 trillion tokens. Models today are deliberately trained far past that point, because over-training a smaller model makes it permanently cheaper to run. The numbers →
Every claim above traces to one of these. They are more readable than their reputation suggests — start with the abstract and the figures.
Reading about models only goes so far — use one. These are live demos hosted on Hugging Face, the field's public home for open models. Pick one and it loads into the page; chat with it, generate an image, transcribe your voice. Nothing loads until you choose.
Demos run on their owners' servers on Hugging Face, so they can be slow, queue, or occasionally go offline — that's the open-source world, live. They're embedded read-only; the primer doesn't send them anything.
The sampler in Part 06 used stand-in numbers. This is the real thing — GPT-2 itself, downloaded once and run entirely inside your browser. No server, no API key, nothing leaves your device. Type a few words and watch it lay odds on the next token.
Everything above described one forward pass turning your text into a probability for every token in the vocabulary. Here that distribution is real, on real weights — and the same temperature and top-p dials act on it live. This is the exact loop inside every chatbot: predict, sample, append, repeat.
Downloads once, then caches in your browser — the first load is the slow part; after that it runs offline. Weights are served from this site itself, so nothing is sent to any third party.
Drag the dials — the ranking re-weights instantly, without re-running the model, because temperature and top-p act on logits the model has already produced. Greyed rows fall outside the top-p nucleus and can't be sampled.
Model: GPT-2 (124M parameters, 2019) via 🤗 Transformers.js, quantised and served from this origin. It's small and old, so it rambles and gets facts wrong — that's the point: you're seeing the raw mechanism, with no instruction-tuning or safety layer in the way. A frontier model is this same loop, vastly larger and heavily post-trained.
Ask in plain language — "what are the latest speech-to-text models?", then "show me one of them". An agent crawls the Hugging Face Hub on the server (keyless); Claude decides what to fetch, then answers with the models it found and links to open each one.
The agent reads the public Hugging Face API on the server, so results are live and change minute to minute. Answers are AI-generated — open the cards to verify. Nothing you type is stored.
Hand-picked companion pieces that open in a reader on this page, so you can follow a thread without losing your place. Each one permits embedding; the rest of the field's writing — the papers in Ref B — doesn't, so those still open in their own tab.
Everything in this primer sits on a short, fast history. The through-line is simple: one architecture from 2017, then a decade of making it bigger, cheaper, and better-behaved.
“Attention Is All You Need” replaces recurrence with self-attention. Every model in this feed is a descendant. Part 03
BERT and GPT-1 show that pre-training on raw text, then adapting, beats training from scratch. The two-stage recipe begins. Part 05
GPT-3 (175B) makes few-shot prompting work; Kaplan’s scaling laws turn “make it bigger” into a predictable curve. Part 07
InstructGPT and RLHF teach models to follow instructions; Chinchilla corrects the scaling recipe; ChatGPT puts it in front of everyone. Part 05
GPT-4 raises the ceiling; Llama and its kin put capable open-weight models in anyone’s hands. Context windows start to stretch.
Models trained to think step-by-step before answering; mixture-of-experts makes huge models cheap to run; tool-use and agents move from demo to product. Part 04
The rest of the story is still being written. See the newsroom →
The same force that made language models work — scale concentrating in a few winners — has been reshaping the stock market. A tiny handful of mega-cap companies now drives most of the S&P 500's return. This tool turns that observation into a hands-on simulation: pick a year, a monthly amount, and watch what a disciplined stake in the ten largest companies would have done — and, on your assumptions, might do next.
Educational simulation only — not financial advice. Past performance does not predict future results. The historical basket is a simplified, equal-weighted model of the largest S&P 500 companies, not a real, buyable fund; future years are a projection from an assumed rate, not a forecast. Do your own research.
Each year, hold the ten largest S&P 500 companies by market value, weighted equally, and add a fixed amount every month (dollar-cost averaging). Once a year the basket is refreshed to the new top ten, so winners that grow into giants are captured and names that fade drop out. It is a plain, mechanical rule — no stock-picking, no timing — and a clean way to see how mega-cap concentration and steady contributions compound over time.
The top 5 are highlighted. Refreshed every day from State Street's SPDR S&P 500 (SPY) holdings.
| # | Company | Price | Top 10 (÷10) | Top 5 (÷5) |
|---|
The basket is re-ranked every month by market cap (computed for 2015+, curated before), acting on the previous month-end ranking so nothing is bought with hindsight, and so newcomers like Nvidia and Tesla are picked up as they grow and faders drop out. The top 5 are highlighted — the tool compares a top-5 basket (more concentrated) against the top 10.
| Strategy | Annualized return | Cumulative return | Max drawdown |
|---|
| Basket | Rebalancing | Annualized | Cumulative | Max drawdown | Changes / mo | Caveat |
|---|
| Stock | Price | Equal share | Shares | If fractions allowed | Cost | Weight it lands on |
|---|
| Year | Your age | Invested | Unguarded | Selected | Top 10 | Top 5 | Top-5 edge | Annual growth (10 / 5) |
|---|
The baskets above choose from 40 companies picked because they are huge today — Nvidia and Tesla are on that list because we know how their stories ended. Here the pool is the S&P 500 as it actually stood each month, the value formulas see only filings that had already been published, and every trade pays 10 bp. These are the combinations that came out on top once those corrections were made.
| Stock pool the momentum top-5 rule may choose from | Annualized |
|---|
| Basket | Rebalancing | Annualized | Cumulative | Max drawdown | Changes / mo | Caveat |
|---|
| Basket | Whole window | Stopped a year early | First half | Second half | Max drawdown |
|---|
| Stock | Price | Equal share | Shares | If fractions allowed | Cost | Weight it lands on |
|---|
| Year | Invested by then | Unguarded | Value | Gain | Gain % | Basket that year |
|---|
Identical engine, identical rules, a different market: rank the universe each month on trailing 12-month return and hold the best N, equal-weighted. Prices are dividend-adjusted and in ringgit, going back to 2001.
| Basket | Rebalancing | Annualized | Cumulative | Max drawdown | Changes / mo | Caveat |
|---|
| Stock | Price | Equal share | Shares | If fractions allowed | Cost | Weight it lands on |
|---|
| Year | Invested by then | Unguarded | Value | Gain | Gain % | Basket that year |
|---|
Not a chat — a design. Tell it what you're trying to do, in your own words, and Claude works through it the way a good solution architect would: what the problem really is, two or three genuinely different ways to tackle it, a recommendation and why, a step-by-step plan, the tools and models that fit, the risks, and the single smallest thing to do today.
Designs are AI-generated and can be wrong or miss something specific to your situation — treat them as a strong first draft to check, not a decision. Nothing you type is stored.
The Invest tab backtests a fixed rule. This lab lets you do what the whole primer is about — train a model — on real market data, entirely in your browser. Pick the features it may look at, choose logistic regression or a small neural network, split the years into train and test, and press Train. Then watch the two numbers that matter: how well it learned the past, and how well that survives contact with data it never saw.
Educational only — not financial advice. This is the classic machine-learning lesson in miniature: a model that fits history beautifully usually predicts the future no better than a coin flip, and the honest result here is that markets are mostly noise. Nothing here trades anything.
| Metric | Train | Test | Baseline (always "up") |
|---|