Distributed Training skills

D

DeepSpeed

davila7

Optimize distributed training with DeepSpeed guidance.

OpenRLHF Training

davila7

Accelerate RLHF training for large models with Ray and vLLM.

Megatron-Core Training

davila7

Efficiently train large language models with advanced parallelism.

TorchTitan

davila7

Efficient distributed LLM pretraining with PyTorch.

Multi-Node Slurm

nvidia

Easily convert single-node scripts to multi-node Slurm jobs.

H

Hierarchical Context Parallel

nvidia

Optimize Megatron-Bridge for hierarchical context parallelism.

M

Megatron Bridge Resiliency

nvidia

Enhance fault tolerance and performance in distributed training.

PhysicsNeMo ShardTensor

nvidia

Optimize domain parallelism for PyTorch models.

PyTorch Lightning

k-dense-ai

Streamline your deep learning workflows with PyTorch.

Run Megatron-LM on SLURM

nvidia

Efficiently launch Megatron-LM training jobs on SLURM clusters.

NeMo AutoModel Distributed Training

nvidia

Streamline your distributed training setup with NeMo AutoModel.

HuggingFace Accelerate

davila7

Simplify distributed training with minimal code changes.

Volcano Engine RL Training

davila7

Efficient RL training for large language models.

T

TP DP PP Communication Overlap

nvidia

Optimize Megatron-Bridge for efficient communication overlap.

HuggingFace Accelerate

nousresearch

Simplify distributed PyTorch training with minimal code changes.

Megatron FSDP

nvidia

Optimize distributed training with Megatron FSDP.

PyTorch FSDP

davila7

Expert guidance for Fully Sharded Data Parallel training in PyTorch.

Ray Train

orchestra-research

Effortlessly scale your ML training across clusters.

Flow Nexus Neural Networks

ruvnet

Train and deploy neural networks in distributed environments.

PyTorch FSDP2 Integration

orchestra-research

Effortlessly implement PyTorch FSDP2 in your training scripts.