Tag: Distributed Training

3 articles

• LLM Training NaN/Inf Production Guide: Non-finite Gates, GradScaler Evidence, and Layer Bisect to Isolate the First Bad Step

  A practical guide to handling NaN/Inf in large model training. Covers non-finite gates, GradScaler skip evidence, distributed consistent skipping, layer bisect localization, and bad batch isolation for replayable, stoppable, and recoverable numerical anomaly management.

  07/22/2026 LLM Training / Numerical Stability / PyTorch / Mixed Precision / Distributed Training

• LLM Activation Checkpointing in Production: Selective Recompute, Non-Reentrant Mode, and RNG Consistency to Reduce Training Memory

  During large model training, activations often exhaust GPU memory before parameters. This article explores selective recomputation, non-reentrant mode, RNG state consistency, and engineering gating strategies to reduce peak memory while preserving gradient correctness and controlling throughput loss.

  07/14/2026 LLM Training / Activation Checkpointing / PyTorch / GPU Memory / Distributed Training

• LLM FP8 Fine-Tuning in Production: Amax Scaling, Layer Whitelists, and BF16 Fallback for Stable Convergence

  FP8 reduces compute and memory for LLM fine-tuning, but poor scaling, outliers, and distributed sync issues can cause silent accuracy degradation. This guide provides a complete engineering solution with Amax monitoring, layer whitelists, BF16 fallback, dual-track evaluation, and release gates.

  07/14/2026 LLM Training / FP8 / Transformer Engine / Fine-tuning / MLOps / Mixed Precision Training / Distributed Training / Amax Scaling