What are the capabilities and evaluation results of the Llama 3 herd of models?

A dense Transformer family with a 405B flagship

Llama 3 is described as a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. The largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. The paper's architecture table also lists 8B and 70B members, with 32, 80, and 126 layers respectively, 8 key-value heads in each, and a shared vocabulary of 128,000 tokens.

The authors state that the architecture does not deviate significantly from Llama and Llama 2. They attribute the performance gains primarily to improvements in data quality and diversity and to increased training scale, not to a new architecture. That framing matters for reading the evaluation: the headline numbers describe a conventionally structured model whose advantage is claimed to come from data and compute.

Scaling laws set the 405B size, not a fixed recipe

The 405B size was chosen with scaling-law experiments rather than picked in advance. The team pre-trained models between 40M and 16B parameters at compute budgets from 6 × 10^18 to 10^22 FLOPs, measured validation loss, and identified the minimum of each IsoFLOPs curve as the compute-optimal model for that budget. Fitting a power law to those minima gave an exponent of 0.53 and a coefficient of 0.29 for the relation between compute and optimal training tokens.

Extrapolating that fit to the flagship budget of 3.8 × 10^25 FLOPs suggested training a 402B-parameter model on 16.55T tokens. The authors also observed that the IsoFLOPs curves become flatter around the minimum as compute grows, so performance is relatively robust to small changes in the model-size versus token trade-off. They therefore trained a 405B model, close to the predicted 402B. This is a model-based projection used to set a design choice, not a measured outcome of the released model.

Data mix and tokenizer carry much of the gain

The final pre-training mix contains roughly 50% general-knowledge tokens, 25% mathematical and reasoning tokens, 17% code tokens, and 8% multilingual tokens. To choose that mix, the authors ran scaling-law experiments on small models and used them to predict how a larger model would perform on each candidate mix before committing to a full run. They also built a classifier to categorize web data and downsample over-represented categories such as arts and entertainment.

The tokenizer combines 100K tokens from the tiktoken tokenizer with 28K additional tokens aimed at non-English languages. On a sample of English data, the new tokenizer raised compression from 3.17 to 3.94 characters per token compared with the Llama 2 tokenizer, so the same training compute covers more text. The authors report that the added non-English tokens improved compression ratios and downstream performance without affecting English tokenization.

Annealing gains shrink as model size grows

Annealing means finishing pre-training on a small, high-quality mix of code and mathematical data. The authors report that annealing improved a pre-trained Llama 3 8B model by 24.0% on the GSM8k validation set and 6.4% on the MATH validation set. For the 405B model, the improvements were negligible.

The authors interpret that gap as evidence that the flagship model already has strong in-context learning and reasoning and does not need in-domain training samples to perform well. The comparison is between validation-set gains for two model sizes under the same annealing procedure, so it describes where the technique pays off rather than a general claim about data quality. The team also used annealing to judge small domain-specific datasets, assigning 30% weight to a new dataset and 70% to the default mix while decaying the learning rate to zero on 40B tokens.

Long context and tools required targeted training data

Applying the existing supervised fine-tuning recipe with only short-context data caused significant regressions in long-context ability inherited from pre-training. Because human annotation of long examples is impractical, the team generated synthetic data with earlier Llama 3 versions for question answering over documents split into 8K-token chunks, hierarchical summarization, and reasoning over code repositories where a key file was removed. Mixing 0.1% synthetic long-context data with the original short-context data optimized performance across both short- and long-context benchmarks.

Direct preference optimization, the post-training step that shapes responses from preference data, did not hurt long-context performance when only short-context data was used, provided the fine-tuned model was already strong at long context. The authors suspect this is because that step uses fewer optimizer steps than supervised fine-tuning. For tool use, Llama 3 was trained to call a search engine for information beyond its knowledge cutoff and a Python interpreter for computation and file-based tasks.

Comparable quality to GPT-4 across many tasks

The paper's central evaluation claim is that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. The abstract states this as a finding from an extensive empirical evaluation; it does not reduce the comparison to a single aggregate score, and the claim is bounded to the tasks the authors evaluated.

Two design choices shape how much weight that comparison can bear. The pre-training data was procured and processed by a separate team incentivized to prevent contamination with commonly used benchmarks, and human evaluations were run and accessed only by a small set of researchers who did not contribute to model development. The authors present these organizational choices as pivotal to trustworthy results, which is an argument about process rather than an independent audit of the benchmarks.

Safety tooling ships, multimodal models do not

Alongside the 405B pre-trained and post-trained language models, the release includes Llama Guard 3, a model for input and output safety. The paper also reports Prompt Guard performance across jailbreak and injection settings: true positive rates of 99.9% for jailbreaks, 99.5% for injections, 97.5% for out-of-distribution jailbreaks, 91.5% for multilingual jailbreaks, and 71.4% for indirect injections, with false positive rates between 0.4% and 5.3%. The multilingual and indirect-injection rows are visibly weaker than the in-distribution rows, so the guard's protection is uneven across attack types.

The image, video, and speech experiments integrate those capabilities into Llama 3 through a compositional approach, and the authors observe performance competitive with the state of the art on image, video, and speech recognition tasks. These models are explicitly not broadly released because they are still under development. The limitations section adds that no testing can be exhaustive, and that Llama 3 may still generate harmful content, particularly for languages beyond English and under skilled adversarial prompting.

Read the release boundary before the benchmark claim

The most decision-relevant split in this paper is between what is released and what is only demonstrated. The 405B language model, its post-trained version, and Llama Guard 3 are public, and the comparable-to-GPT-4 claim applies to that released language model on the evaluated tasks. The multimodal results are preliminary experiments the authors chose to share early, so they should not be treated as available products or as settled capability claims.

Within the released system, the evidence points to a specific priority: data quality, scale, and simplicity drove the gains, while more complex architectures and training recipes were explored and did not justify their added complexity. For a reader weighing Llama 3, the useful question is therefore not whether a 405B dense model can compete on benchmarks, but which of the reported conditions, namely the evaluated task set, the contamination controls, and the uneven guard performance across attack types, match the deployment being considered.

Sources used in this answer