Tag: PyTorch

4 articles

• LLM Training NaN/Inf Production Guide: Non-finite Gates, GradScaler Evidence, and Layer Bisect to Isolate the First Bad Step

  A practical guide to handling NaN/Inf in large model training. Covers non-finite gates, GradScaler skip evidence, distributed consistent skipping, layer bisect localization, and bad batch isolation for replayable, stoppable, and recoverable numerical anomaly management.

  07/22/2026 LLM Training / Numerical Stability / PyTorch / Mixed Precision / Distributed Training

• LLM Activation Checkpointing in Production: Selective Recompute, Non-Reentrant Mode, and RNG Consistency to Reduce Training Memory

  During large model training, activations often exhaust GPU memory before parameters. This article explores selective recomputation, non-reentrant mode, RNG state consistency, and engineering gating strategies to reduce peak memory while preserving gradient correctness and controlling throughput loss.

  07/14/2026 LLM Training / Activation Checkpointing / PyTorch / GPU Memory / Distributed Training

• LLM CPU–GPU Data Path in Production: Reducing Inference Jitter with NUMA Affinity, Pinned Memory, and Async Copy

  A systematic guide to optimizing the CPU-to-GPU data path for LLM inference, covering NUMA affinity, pinned memory, async copy, topology scheduling, and container alignment to reduce host-side transfer jitter, tail latency, and cross-node resource misconfiguration.

  07/12/2026 LLM Serving / CUDA / NUMA / GPU Performance / PyTorch

• LLM GPU Memory Fragmentation in Production: Taming OOMs with Memory Snapshots, Allocator Tuning, and Admission Control

  LLM inference services often hit OOM due to fragmentation, not total capacity exhaustion. This guide covers Memory Snapshots, dynamic shapes, allocator tuning, and vLLM memory budgeting for observable, verifiable, and rollback-safe production management.

  07/11/2026 LLM Serving / CUDA / GPU Memory / PyTorch / vLLM