Tag: Mixed Precision

2 articles

• LLM Training NaN/Inf Production Guide: Non-finite Gates, GradScaler Evidence, and Layer Bisect to Isolate the First Bad Step

  A practical guide to handling NaN/Inf in large model training. Covers non-finite gates, GradScaler skip evidence, distributed consistent skipping, layer bisect localization, and bad batch isolation for replayable, stoppable, and recoverable numerical anomaly management.

  07/22/2026 LLM Training / Numerical Stability / PyTorch / Mixed Precision / Distributed Training

• LLM Gradient Accumulation in Production: Token-Normalized Loss, DDP no_sync, and Scheduler Alignment to Avoid Equivalent Batch Distortion

  Gradient accumulation seems like simple batch summing, but variable-length samples, distributed sync, mixed precision, and learning rate schedulers silently alter the equivalent batch. This article dives into production-grade solutions like token normalization, DDP no_sync, AMP update boundaries, and tail window handling to achieve mathematically consistent gradient accumulation in LLM fine-tuning.

  07/14/2026 LLM Training / Gradient Accumulation / PyTorch DDP / Fine-Tuning / Mixed Precision