Tag: LLM training

8 articles

• LLM μP Hyperparameter Cross-Scale Transfer in Production: Reducing Large Model Tuning Costs with Base Shape Fingerprints, Coord Checks, and Proxy Sweeps

  A systematic breakdown of μP hyperparameter cross-scale transfer principles and engineering boundaries, covering how to solidify Base Shapes, run Coord Checks, organize proxy model sweeps with regression gates, and avoid misapplying width transfer to training duration, batch size, or architectural changes.

  07/28/2026 LLM training / μP / hyperparameter transfer / pretraining engineering / scaling

• LLM Training NaN/Inf Production Guide: Non-finite Gates, GradScaler Evidence, and Layer Bisect to Isolate the First Bad Step

  A practical guide to handling NaN/Inf in large model training. Covers non-finite gates, GradScaler skip evidence, distributed consistent skipping, layer bisect localization, and bad batch isolation for replayable, stoppable, and recoverable numerical anomaly management.

  07/22/2026 LLM Training / Numerical Stability / PyTorch / Mixed Precision / Distributed Training

• LLM Context Parallel in Production: Breaking Long-Context Bottlenecks with Sequence Sharding, Ring/Ulysses Communication, and Causal Load Balancing

  An in-depth guide to Context Parallel for long-context LLM training, covering Ring Attention vs. Ulysses communication, causal load balancing strategies, and production deployment checks to scale context from 8K to 128K+ tokens on multi-GPU clusters.

  07/16/2026 LLM Training / Context Parallel / Distributed Systems / Long Context / Ring Attention

• LLM Gradient Accumulation in Production: Token-Normalized Loss, DDP no_sync, and Scheduler Alignment to Avoid Equivalent Batch Distortion

  Gradient accumulation seems like simple batch summing, but variable-length samples, distributed sync, mixed precision, and learning rate schedulers silently alter the equivalent batch. This article dives into production-grade solutions like token normalization, DDP no_sync, AMP update boundaries, and tail window handling to achieve mathematically consistent gradient accumulation in LLM fine-tuning.

  07/14/2026 LLM Training / Gradient Accumulation / PyTorch DDP / Fine-Tuning / Mixed Precision

• LLM Activation Checkpointing in Production: Selective Recompute, Non-Reentrant Mode, and RNG Consistency to Reduce Training Memory

  During large model training, activations often exhaust GPU memory before parameters. This article explores selective recomputation, non-reentrant mode, RNG state consistency, and engineering gating strategies to reduce peak memory while preserving gradient correctness and controlling throughput loss.

  07/14/2026 LLM Training / Activation Checkpointing / PyTorch / GPU Memory / Distributed Training

• LLM FP8 Fine-Tuning in Production: Amax Scaling, Layer Whitelists, and BF16 Fallback for Stable Convergence

  FP8 reduces compute and memory for LLM fine-tuning, but poor scaling, outliers, and distributed sync issues can cause silent accuracy degradation. This guide provides a complete engineering solution with Amax monitoring, layer whitelists, BF16 fallback, dual-track evaluation, and release gates.

  07/14/2026 LLM Training / FP8 / Transformer Engine / Fine-tuning / MLOps / Mixed Precision Training / Distributed Training / Amax Scaling

• LLM SFT Sequence Packing in Production: Boosting Throughput with Sample Boundaries, Loss Mask, and Length Buckets

  A practical guide to implementing safe Sequence Packing for SFT training, covering attention isolation, loss masking, position IDs, length bucketing, and validation gates to eliminate padding waste without cross-sample contamination.

  07/14/2026 LLM Training / SFT / Sequence Packing / FlashAttention / Data Collator

• Distributed LLM Fine-Tuning Checkpoint Resume in Production: Sharded Checkpoints, Atomic Commits, and World Size Changes

  A practical guide to production-grade checkpoint and resume for distributed LLM fine-tuning. Covers sharded checkpoints, atomic commits, optimizer/RNG state recovery, world size changes, and fault injection drills to prevent silent training drift.

  07/13/2026 LLM Training / Distributed Checkpoint / FSDP / Fault Tolerance