성능 & Getting Started

Checkpointless Training의 실제 성능 벤치마크와 단계별 통합 가이드를 제공합니다.

1. 성능 벤치마크

1.1 복구 시간 및 Goodput 비교

1.2 기존 방식 대비 비교

2. Amazon Nova 검증

3. Prerequisites

3.1 지원 Instance Types

3.2 소프트웨어 요구사항

3.3 Container Image

AWS에서 제공하는 Pre-built 컨테이너 이미지를 사용합니다.

# Container Image URI (ap-northeast-1 예시)
356859066553.dkr.ecr.ap-northeast-1.amazonaws.com/hyperpod-checkpointless-training:v1.0.0

# 지원 리전
us-east-1, us-east-2, us-west-1, us-west-2
eu-west-1, eu-west-2, eu-central-1, eu-north-1
ap-northeast-1, ap-south-1, ap-southeast-1, ap-southeast-2
sa-east-1

3.4 Container 버전 정보 (v1.0.0)

4. 4-Tier Integration Guide

Checkpointless Training은 단계별로 통합할 수 있습니다. 각 Tier는 이전 Tier를 포함합니다.

Tier 1: 환경 변수만 설정 (Rootless NCCL)

가장 간단한 통합. Rootless NCCL만 활성화하여 NCCL 초기화 시간을 단축합니다.

# entrypoint.sh
#!/bin/bash
set -ex

# Tier 1: Rootless NCCL 활성화
export HPCT_USE_ROOTLESS=1
export TORCH_SKIP_TCPSTORE=1
sysctl -w net.ipv4.ip_local_port_range="20000 65535"

# 기존 학습 스크립트 실행
python training_script.py

Tier 2: MMAP Data Loading 추가

공유 메모리 기반 데이터 로딩으로 DataLoader 초기화 시간을 제거합니다.

from hyperpod_checkpointless_training.dataloader.mmap_data_module import MMAPDataModule
from hyperpod_checkpointless_training.dataloader.config import CacheResumeMMAPConfig

# Tier 2: MMAP DataLoader 설정
mmap_config = CacheResumeMMAPConfig(
    cache_dir="/dev/shm/pdl_cache",
    prefetch_length=10,
    lookback_length=2,
)

data_module = MMAPDataModule(
    data_module=YourDataModule(...),
    mmap_config=mmap_config,
)

# Trainer에서 사용
trainer.fit(model, data_module)

Tier 3: In-Process Recovery (IPR) 추가

프로세스 레벨 장애 격리로 정상 노드의 재시작을 방지합니다.

from hyperpod_checkpointless_training.inprocess.wrap import HPWrapper
from hyperpod_checkpointless_training.inprocess.health_check import CudaHealthCheck
from hyperpod_checkpointless_training.inprocess.train_utils import wait_rank, HPAgentK8sAPIFactory
from hyperpod_checkpointless_training.inprocess.abort import (
    CheckpointlessAbortManager,
    CheckpointlessFinalizeCleanup
)

# Tier 3: In-Process Recovery 설정
wait_rank()  # Training Operator로부터 Rank 정보 수신 대기

@HPWrapper(
    health_check=CudaHealthCheck(),
    hp_api_factory=HPAgentK8sAPIFactory(),
    abort_timeout=60.0,
    abort=CheckpointlessAbortManager.get_default_checkpointless_abort(),
    finalize=CheckpointlessFinalizeCleanup(),
)
def run_main(cfg, caller=None):
    trainer = Trainer(...)
    trainer.wrapper = caller
    trainer.fit(model, data_module)

if __name__ == "__main__":
    run_main(config)

Tier 4: P2P Checkpointless (완전 통합)

메모리 기반 P2P 상태 복제로 디스크 I/O를 완전히 제거합니다.

from hyperpod_checkpointless_training.nemo_plugins.strategy import CheckpointlessMegatronStrategy
from hyperpod_checkpointless_training.nemo_plugins.checkpoint_manager import CheckpointManager
from hyperpod_checkpointless_training.nemo_plugins.callbacks import CheckpointlessCallback

# Tier 4: P2P State Replication 설정
checkpoint_manager = CheckpointManager(
    enable_checksum=False,
    enable_offload=False,
)

@HPWrapper(
    health_check=CudaHealthCheck(),
    hp_api_factory=HPAgentK8sAPIFactory(),
    checkpoint_manager=checkpoint_manager,  # CheckpointManager 추가
    abort=CheckpointlessAbortManager.get_default_checkpointless_abort(),
    finalize=CheckpointlessFinalizeCleanup(),
)
def run_main(cfg, caller=None):
    trainer = Trainer(
        strategy=CheckpointlessMegatronStrategy(
            # N+1 Redundancy: 최소 2개 인스턴스
            num_distributed_optimizer_instances=2,
            tensor_model_parallel_size=8,
            pipeline_model_parallel_size=4,
        ),
        callbacks=[
            CheckpointlessCallback(checkpoint_manager=checkpoint_manager),
        ],
    )
    trainer.wrapper = caller
    trainer.fit(model, data_module)

5. Pre-configured Recipes

AWS에서 제공하는 사전 구성된 학습 레시피입니다.

6. Complete Code Example

4개 Tier를 모두 통합한 전체 Python 스크립트 예시입니다.

#!/usr/bin/env python3
# training_script.py - Checkpointless Training Complete Example

import os
from omegaconf import OmegaConf
from pytorch_lightning import Trainer

# Tier 2: MMAP Data Loading
from hyperpod_checkpointless_training.dataloader.mmap_data_module import MMAPDataModule
from hyperpod_checkpointless_training.dataloader.config import CacheResumeMMAPConfig

# Tier 3: In-Process Recovery
from hyperpod_checkpointless_training.inprocess.wrap import HPWrapper
from hyperpod_checkpointless_training.inprocess.health_check import CudaHealthCheck
from hyperpod_checkpointless_training.inprocess.train_utils import wait_rank, HPAgentK8sAPIFactory
from hyperpod_checkpointless_training.inprocess.abort import (
    CheckpointlessAbortManager,
    CheckpointlessFinalizeCleanup
)

# Tier 4: P2P State Replication
from hyperpod_checkpointless_training.nemo_plugins.strategy import CheckpointlessMegatronStrategy
from hyperpod_checkpointless_training.nemo_plugins.checkpoint_manager import CheckpointManager
from hyperpod_checkpointless_training.nemo_plugins.callbacks import CheckpointlessCallback

# Your model and data modules
from my_project import MyModel, MyDataModule


def create_data_module(cfg):
    """Tier 2: MMAP DataLoader 생성"""
    base_data_module = MyDataModule(
        data_path=cfg.data.path,
        batch_size=cfg.data.batch_size,
    )

    mmap_config = CacheResumeMMAPConfig(
        cache_dir="/dev/shm/pdl_cache",
        prefetch_length=cfg.data.prefetch_length,
        lookback_length=cfg.data.lookback_length,
    )

    return MMAPDataModule(
        data_module=base_data_module,
        mmap_config=mmap_config,
    )


def create_trainer(cfg, checkpoint_manager):
    """Tier 4: Checkpointless Trainer 생성"""
    strategy = CheckpointlessMegatronStrategy(
        num_distributed_optimizer_instances=cfg.model.num_optimizer_instances,
        tensor_model_parallel_size=cfg.model.tp_size,
        pipeline_model_parallel_size=cfg.model.pp_size,
        context_parallel_size=cfg.model.cp_size,
    )

    callbacks = [
        CheckpointlessCallback(checkpoint_manager=checkpoint_manager),
    ]

    return Trainer(
        strategy=strategy,
        callbacks=callbacks,
        max_steps=cfg.training.max_steps,
        accelerator="gpu",
        devices=cfg.training.devices,
        num_nodes=cfg.training.num_nodes,
    )


# Tier 4: CheckpointManager 설정
checkpoint_manager = CheckpointManager(
    enable_checksum=False,
    enable_offload=False,
)


# Tier 3: In-Process Recovery Wrapper
@HPWrapper(
    health_check=CudaHealthCheck(timeout=30),
    hp_api_factory=HPAgentK8sAPIFactory(),
    abort_timeout=60.0,
    checkpoint_manager=checkpoint_manager,
    abort=CheckpointlessAbortManager.get_default_checkpointless_abort(),
    finalize=CheckpointlessFinalizeCleanup(),
)
def run_main(cfg, caller=None):
    """메인 학습 함수 - Re-executable Code Block (RCB)"""

    # 모델 생성
    model = MyModel(cfg.model)

    # Tier 2: MMAP DataLoader
    data_module = create_data_module(cfg)

    # Tier 4: Checkpointless Trainer
    trainer = create_trainer(cfg, checkpoint_manager)
    trainer.wrapper = caller  # HPWrapper 연결 필수

    # 학습 실행
    trainer.fit(model, data_module)


if __name__ == "__main__":
    # Tier 3: Training Operator로부터 Rank 정보 수신 대기
    wait_rank()

    # 설정 로드
    cfg = OmegaConf.load("config.yaml")

    # 학습 실행
    run_main(cfg)

7. HyperPod Elastic Agent

7.1 entrypoint.sh 예시

#!/bin/bash
set -ex

# Tier 1: Rootless NCCL 환경 설정
export HPCT_USE_ROOTLESS=1
export TORCH_SKIP_TCPSTORE=1
sysctl -w net.ipv4.ip_local_port_range="20000 65535"

# 추가 환경 설정
export CUDA_DEVICE_MAX_CONNECTIONS=1
export NCCL_DEBUG=INFO

# HyperPod Elastic Agent로 학습 실행
hyperpodrun \
    --nproc_per_node=8 \
    --inprocess-restart \
    --inprocess-timeout=120 \
    --shutdown-signal=SIGKILL \
    --shutdown-timeout=15 \
    training_script.py

7.2 hyperpodrun Arguments

8. IAM Policy

Tiered Checkpointing 사용 시 S3 접근을 위한 IAM 정책입니다.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-checkpoint-bucket",
        "arn:aws:s3:::your-checkpoint-bucket/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "fsx:DescribeFileSystems"
      ],
      "Resource": "*"
    }
  ]
}

9. Troubleshooting

# NCCL 타임아웃 증가
export NCCL_TIMEOUT=1800

# 디버그 로그 활성화
export NCCL_DEBUG=INFO
export NCCL_DEBUG_SUBSYS=ALL

# /dev/shm 크기 확인
df -h /dev/shm

# K8s Pod에서 shm 크기 증가
# spec.containers.volumeMounts:
#   - name: dshm
#     mountPath: /dev/shm
# spec.volumes:
#   - name: dshm
#     emptyDir:
#       medium: Memory
#       sizeLimit: 64Gi

# 복제 인스턴스 수 증가
strategy = CheckpointlessMegatronStrategy(
    num_distributed_optimizer_instances=3,  # 2에서 3으로 증가
)

# Fallback용 Checkpoint 저장 주기 설정
# 완전한 장애 대비를 위해 주기적 Checkpoint도 유지

# Health check 타임아웃 증가
@HPWrapper(
    health_check=CudaHealthCheck(timeout=60),  # 30에서 60으로
    ...
)

# Training Operator 상태 확인
kubectl get pods -n sagemaker-hyperpod-system

# Operator 로그 확인
kubectl logs -n sagemaker-hyperpod-system deployment/training-operator