fsdp

Train speculative decoding models effortlessly and port them smoothly to SGLang serving.

A scalable, agentic-first, and HuggingFace-native RL framework for research (9k lines).

Best practices & guides on how to write distributed pytorch training code

From teacher to tiles — a from-scratch LLM distillation & serving engine: custom Triton/CUDA kernels, FSDP distillation, paged-KV continuous batching, speculative decoding, a Rust gateway, a JAX oracle, and interpretability tooling.

Fault tolerance for PyTorch (HSDP, LocalSGD, DiLoCo, Streaming DiLoCo)

A PyTorch native library for training speculative decoding models

Repo for Qwen Image Finetune

Find why PyTorch training is slow, compare runs, and catch performance regressions in CI.

A native PyTorch backend for AWS Trainium with support for eager execution, torch.compile, distributed training, and custom NKI kernels

Educational distributed training and inference library for heterogeneous local hardware: Mac minis, Raspberry Pis and GPUs over plain Python sockets. FSDP, GRPO, parameter-server, expert and data parallelism.

Write once, run anywhere; ezpz 🍋

A comprehensive hands-on guide to building production-grade distributed applications with Ray - from distributed training and multimodal data processing to inference and reinforcement learning.

Research platform for model training, evaluation, and experimentation across architectures, benchmarks, and recipes.

META LLAMA3 GENAI Real World UseCases End To End Implementation Guide

AWS Neuron's extension of TorchTitan that adds AWS Trainium support with TorchNeuron for training generative AI models

Interactive exercises for understanding compute, communication, and dependencies in distributed LLM training traces.

Fast and easy distributed model training examples.

Billus 大模型技能库：面向 LLM、VL、多模态与图像生成模型的训练、调参、剪枝、量化与工程化技能库

Forge kernels — fused Triton kernels for faster, leaner LLM fine-tuning. One-call patching into Hugging Face models via forge.patch(model), with FSDP2 multi-GPU support and a growing set of kernels and architectures. Every speedup backed by a committed benchmark. Apache-2.0.

A minimal, hackable pre-training stack for GPT-style language models