gpt-oss & GPT-5

In August 2025, the AI field witnessed a period of intensive releases from OpenAI. Following GPT-2 (OpenAI, 2019) in 2019, OpenAI has once again contributed to the open-source community with its first open-weight large language model series, gpt-oss (OpenAI, 2025), available in 120B and 20B sizes. Shortly after, the highly anticipated next-generation flagship model, GPT-5 (OpenAI, 2025), was also officially launched. This series of releases not only marks a new high for open-source models in reasoning and agent capabilities but also reveals OpenAI’s latest advancements in model architecture, training methodologies, and safety alignment.

gpt-oss

gpt-oss (OpenAI, 2025) is OpenAI’s first open-weight language model series released since GPT-2, designed to provide the open-source community with powerful reasoning and tool-use capabilities. The series includes two versions, gpt-oss-120b and gpt-oss-20b, both released under the Apache 2.0 license.

Architecture Overview

gpt-oss is built upon the GPT series architecture and incorporates several mainstream technologies from recent years, including RMSNorm, SwiGLU, GQA, RoPE, YaRN, and MoE.

The table below provides a direct comparison of the differences between the GPT-OSS 20B and GPT-2 XL 1.5B models.

Efficient Attention Mechanisms

To maintain high efficiency while supporting a 128k long context, gpt-oss employs several advanced attention mechanisms.

• Grouped-Query Attention (GQA): gpt-oss has 64 query heads and 8 key/value heads, meaning every 8 query heads share a single K/V pair. This significantly reduces the size of the KV cache and memory bandwidth requirements, thereby substantially increasing inference throughput with almost no loss in model performance.

• Sliding Window Attention: To further reduce computational complexity, gpt-oss draws inspiration from Longformer (Jiang et al., 2020) and Mistral (Jiang et al., 2023) by adopting a sliding window attention. Its Transformer layers alternate between Dense Attention and Locally Banded Sparse Attention. The latter is Sliding Window Attention, which limits the attention scope of each token to a fixed-size local window.

In gpt-oss, this window size is set to 128 tokens. This means that in a local attention layer, a token can only attend to the 128 tokens preceding it, not the entire context. This design reduces the computational complexity of attention from \( O(L^2) \) to \( O(L \cdot W) \), where \( L \) is the sequence length and \( W \) is the window size. By alternating with full attention layers, the model can efficiently process local information while integrating global information through the full attention layers.

• Attention Sinks: The model introduces Attention Sinks (Xiao et al., 2023), which learns a bias \( \mathbf{s}_h \) added to the attention scores. This allows initial tokens to be consistently attended to, helping to stabilize the attention distribution and prevent information loss in long-context scenarios.

The figure above compares the performance and efficiency of StreamingLLM with three common long-text processing methods. Assume a language model is pre-trained on texts of length $L$ and needs to predict the $T$-th token during inference (where $T \gg L$):

1. Dense Attention: Retains the key-values (KV) of all historical tokens and computes full attention. The time complexity is $O(T^2)$, and the cache size grows continuously. Performance drops significantly when the input length exceeds the pre-training length $L$.
2. Window Attention: Caches only the KV of the most recent $L$ tokens. It is efficient for inference, but performance plummets once the information from early tokens is replaced.
3. Sliding Window with Re-computation: Reconstructs the KV state from the most recent $L$ tokens each time a new token is generated. Although it performs well on long texts, the re-computation involves quadratic attention, leading to a high time complexity of $O(TL^2)$ and slow inference speed.

This method combines attention sinks with recent tokens during computation, not only maintaining inference efficiency but also sustaining a stable attention distribution and low perplexity in long-text scenarios.

MXFP4 Quantization

To enable the large model to run on consumer-grade hardware, gpt-oss uses the MXFP4 (Tseng et al., 2025) format to quantize the MoE weights. MXFP4 is a micro-scaling floating-point format that can effectively quantize weights to about 4.25 bits. Since MoE weights account for over 90% of the model’s total parameters, this method drastically compresses the model size, allowing the 120B model to fit into a single 80GB GPU and the 20B model to run on devices with 16GB of VRAM.

Training

• Pre-training: The model is pre-trained on a dataset of several trillion tokens of text, with a focus on STEM, coding, and general knowledge. To enhance safety, the pre-training data reuses the CBRN content filters from GPT-4o.

• Post-training (Reasoning & Tool Use): After pre-training, the model undergoes post-training using CoT RL techniques similar to those for OpenAI’s o3. The goal of this stage is to teach the model to:

  1. Reason: Generate detailed Chain-of-Thought (CoT) to solve complex problems.
  2. Use Tools: Learn to call external tools (like web search, code execution) to enhance its capabilities.

To achieve these advanced agent functionalities, OpenAI designed the Harmony Chat Format. This format introduces the concept of “channels” (e.g., analysis for CoT, commentary for tool calls, final for the final answer) and establishes a strict instruction hierarchy (System > Developer > User > Assistant > Tool) to ensure the model’s behavior is controllable.

Additionally, the model supports Variable Effort Reasoning. Users can set Reasoning: low/medium/high in the system prompt to trade off between latency and performance. A higher effort level generates a longer CoT, which typically leads to higher accuracy but also increased latency.

Evaluation

Benchmark results show that gpt-oss-120b’s accuracy surpasses that of OpenAI’s o3-mini and approaches o4-mini. Meanwhile, gpt-oss-20b, at only one-sixth the size, also demonstrates competitive performance.

GPT-5

GPT-5 (OpenAI, 2025) is not a single model but a unified intelligent system. It is not a monolithic, massive model but a complex system of multiple specialized models and an intelligent routing mechanism working in concert to balance performance, speed, and cost.

System Architecture

The GPT-5 system consists of three core components:

1. gpt-5-main: As the system’s default model, it is fast and efficient, handling the vast majority of user queries. It can be considered the successor to GPT-4o.
2. gpt-5-thinking: Used for more complex problems that require deep thought. This model is activated when the user explicitly requests it (e.g., “think hard about this”) or when the system determines the task requires it. It can be seen as the successor to OpenAI’s o3.
3. Real-time Router: This is a continuously trained decision-making model that quickly determines which model to assign a user request to based on various signals. Its decisions are based on:
   • Conversation Type: Whether it’s a casual chat, Q&A, or a task-oriented conversation.
   • Complexity: The difficulty of the question and the depth of reasoning required.
   • Tool Needs: Whether external tools like web search or a code interpreter need to be called.
   • Explicit Intent: Users can guide the router to select the deep reasoning model with explicit instructions (e.g., “think hard about this”).

The router continuously learns from real user signals (such as user model-switching behavior, response preference rates, and measured answer correctness) to constantly optimize its decision-making capabilities.

Safe Completions

The traditional safety training paradigm is Hard Refusals, where the model decides whether to answer fully or refuse directly based on a binary classification of user intent (safe or unsafe). This approach is effective for clearly malicious prompts but is very brittle when dealing with ambiguous intent or topics involving dual-use (e.g., biology, cybersecurity), often leading to over-refusals.

Safe Completions (Baker et al., 2025) moves away from binary classification of user intent and instead focuses on maximizing the model’s helpfulness while adhering to safety policies.

• For clearly harmful requests: The model will still refuse.
• For dual-use requests (e.g., in biology, chemistry, cybersecurity): The model provides safe, high-level answers without directly executable details, rather than refusing entirely.
• For requests with ambiguous intent: The model attempts to complete the task in a safe manner or offers safe alternatives.

This approach significantly improves the model’s safety and utility in dual-use domains and reduces unnecessary over-refusals.

Chain-of-Thought Monitoring

OpenAI employs Chain-of-Thought Monitoring (CoT Monitoring) (Baker et al., 2025) to ensure the reliability and safety of its reasoning models and to prevent issues like reward hacking. Unlike some approaches that try to optimize CoT through SFT, GPT-5’s CoT training does not impose direct alignment. This allows the CoT to more genuinely reflect the model’s “thinking” process, serving as an effective window for detecting erroneous behavior, deceptive intent, or potential risks.

Through CoT monitoring, OpenAI found that the incidence of deceptive behavior in gpt-5-thinking was reduced to 2.1% from 4.8% in o3. This technology is crucial for understanding and mitigating the risks of advanced AI systems.

Evaluation

GPT-5 excels across multiple benchmarks, setting new standards, especially in reasoning, coding, and multimodal capabilities. Compared to its predecessors, it not only achieves a leap in accuracy but also makes significant strides in efficiency, often achieving or surpassing the performance of o3 with 50-80% less output.

🧠 Intelligence

🖼️ Multimodal

💻 Coding

📋 Instruction Following

🔧 Function Calling

📚 Long Context

🚨 Hallucinations

These results indicate that GPT-5 has made significant improvements in complex tasks requiring deep reasoning (like GPQA, AIME) and agentic tasks that require interaction with external environments (like SWE-bench, τ²-bench). At the same time, the substantial improvement in factual accuracy (hallucination rate reduced by nearly 8x) makes it more reliable for practical applications.

References

[1] Raschka, S. (2025). “From GPT-2 to gpt-oss: Analyzing the Architectural Advances.” Ahead of AI.

[2] Radford, Alec, et al. “Language models are unsupervised multitask learners.” OpenAI blog 1.8 (2019): 9.

[3] OpenAI. (2025). “Introducing gpt-oss.” OpenAI Blog.

[4] OpenAI. (2025). “Introducing GPT-5.” OpenAI Blog.

[5] OpenAI. (2025). “gpt-oss-120b & gpt-oss-20b Model Card.”

[6] Beltagy, Iz, Matthew E. Peters, and Arman Cohan. “Longformer: The long-document transformer.” arXiv preprint arXiv:2004.05150 (2020).

[7] Jiang, Dongsheng, et al. “Mistral 7B.” arXiv preprint arXiv:2310.08825 (2023).

[8] Xiao, G., et al. (2023). “Efficient Streaming Language Models with Attention Sinks.” arXiv preprint arXiv:2309.17453.

[9] Tseng, Albert, Tao Yu, and Youngsuk Park. “Training llms with mxfp4.” arXiv preprint arXiv:2502.20586 (2025).

[10] Yuan, Yuan, et al. “From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training.” arXiv preprint arXiv:2508.09224 (2025).

[11] B. Baker, J. Huizinga, L. Gao, Z. Dou, M. Y. Guan, A. Madry, W. Zaremba, J. Pachocki, and D. Farhi, “Monitoring reasoning models for misbehavior and the risks of promoting obfuscation.” arXiv preprint arXiv:2503.11926, 2025. Submitted on 14 March 2025.

[12] OpenAI. (2025). “GPT-5 System Card.”

[13] OpenAI. (2025). “Introducing GPT-5 for developers.” OpenAI Blog.

Citation

Citation: When reproducing or citing the content of this article, please credit the original author and source.

Cited as:

Yue Shui. (Aug 2025). gpt-oss & GPT-5. https://syhya.github.io/posts/2025-08-24-gpt5

Or

@article{yue_shui_gpt5_2025
  title   = "gpt-oss & GPT-5",
  author  = "Yue Shui",
  journal = "syhya.github.io",
  year    = "2025",
  month   = "Aug",
  url     = "https://syhya.github.io/posts/2025-08-24-gpt5"
}