Distributed Training skills

Multi-Node Slurm

nvidia

Easily convert single-node scripts to multi-node Slurm jobs.

M

Megatron Bridge Resiliency

nvidia

Enhance fault tolerance and performance in distributed training.

H

Hierarchical Context Parallel

nvidia

Optimize Megatron-Bridge for hierarchical context parallelism.

HuggingFace Accelerate

nousresearch

Simplify distributed PyTorch training with minimal code changes.

Megatron FSDP

nvidia

Optimize distributed training with Megatron FSDP.

T

TP DP PP Communication Overlap

nvidia

Optimize Megatron-Bridge for efficient communication overlap.

PyTorch FSDP

davila7

Expert guidance for Fully Sharded Data Parallel training in PyTorch.

Flow Nexus Neural Networks

ruvnet

Train and deploy neural networks in distributed environments.

Ray Train

orchestra-research

Effortlessly scale your ML training across clusters.

PyTorch FSDP2 Integration

orchestra-research

Effortlessly implement PyTorch FSDP2 in your training scripts.

HuggingFace Accelerate

davila7

Simplify distributed training with minimal code changes.