About RL training

[ranhaogong]

Thank you for your contribution!
I'm trying to reproduce the training pipeline. The sft training went well, however the RL training went into OOM.
8 * L40S is used for training. Is it necessary to use 8 * A100 for RL training ? And Is there any alternative to RL train with less GPU occupancy?

[ranhaogong]

2026-02-04 15:46:05: DEBUG: JOBUTILS_ENABLE_CONFIG_PRECHECK=''
2026-02-04 15:46:05: DEBUG: SCRIPT_PATH=/jobutils/scripts, BASE_DIR=/jobutils
2026-02-04 15:46:05: DEBUG: JOBUTILS_ENABLE_CONFIG_PRECHECK is not 'true' (value=''), skipping precheck
W0204 15:46:30.964000 125 site-packages/torch/distributed/run.py:793]
W0204 15:46:30.964000 125 site-packages/torch/distributed/run.py:793] *****************************************
W0204 15:46:30.964000 125 site-packages/torch/distributed/run.py:793] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
W0204 15:46:30.964000 125 site-packages/torch/distributed/run.py:793] *****************************************
[2026-02-04 15:47:32,521] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-04 15:47:32,522] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-04 15:47:32,523] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2026-02-04 15:47:32,534] [INFO] [real_accelerator.py:219:get_accelerator] Setting ds_accelerator to cuda (auto detect)
Warning: The cache directory for DeepSpeed Triton autotune, /home/i-gonghaoran/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /home/i-gonghaoran/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /home/i-gonghaoran/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
Warning: The cache directory for DeepSpeed Triton autotune, /home/i-gonghaoran/.triton/autotune, appears to be on an NFS system. While this is generally acceptable, if you experience slowdowns or hanging when DeepSpeed exits, it is recommended to set the TRITON_CACHE_DIR environment variable to a non-NFS path.
INFO 02-04 15:47:48 init.py:190] Automatically detected platform cuda.
INFO 02-04 15:47:48 init.py:190] Automatically detected platform cuda.
INFO 02-04 15:47:48 init.py:190] Automatically detected platform cuda.
INFO 02-04 15:47:48 init.py:190] Automatically detected platform cuda.
[2026-02-04 15:48:00,866] [INFO] [comm.py:652:init_distributed] cdb=None
[2026-02-04 15:48:00,866] [INFO] [comm.py:652:init_distributed] cdb=None
[2026-02-04 15:48:00,866] [INFO] [comm.py:652:init_distributed] cdb=None
[2026-02-04 15:48:00,867] [INFO] [comm.py:683:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
[2026-02-04 15:48:00,867] [INFO] [comm.py:652:init_distributed] cdb=None
using: using: using: <class 'trainer.grpo_trainer.Qwen2VLGRPOTrainer'>
<class 'trainer.grpo_trainer.Qwen2VLGRPOTrainer'><class 'trainer.grpo_trainer.Qwen2VLGRPOTrainer'>

using: <class 'trainer.grpo_trainer.Qwen2VLGRPOTrainer'>
[2026-02-04 15:48:03,728] [INFO] [config.py:733:init] Config mesh_device None world_size = 4
[2026-02-04 15:48:03,728] [INFO] [config.py:733:init] Config mesh_device None world_size = 4
[2026-02-04 15:48:03,728] [INFO] [config.py:733:init] Config mesh_device None world_size = 4
[2026-02-04 15:48:03,729] [INFO] [config.py:733:init] Config mesh_device None world_size = 4
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
You are attempting to use Flash Attention 2.0 without specifying a torch dtype. This might lead to unexpected behaviour
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with model.to('cuda').
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with model.to('cuda').
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with model.to('cuda').
You are attempting to use Flash Attention 2.0 with a model not initialized on GPU. Make sure to move the model to GPU after initializing it on CPU with model.to('cuda').
Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen2_5_VisionTransformerPretrainedModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the with torch.autocast(device_type='torch_device'): decorator, or load the model with the torch_dtype argument. Example: model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)
Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen2_5_VisionTransformerPretrainedModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the with torch.autocast(device_type='torch_device'): decorator, or load the model with the torch_dtype argument. Example: model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)
Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen2_5_VisionTransformerPretrainedModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the with torch.autocast(device_type='torch_device'): decorator, or load the model with the torch_dtype argument. Example: model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)
Flash Attention 2.0 only supports torch.float16 and torch.bfloat16 dtypes, but the current dype in Qwen2_5_VisionTransformerPretrainedModel is torch.float32. You should run training or inference using Automatic Mixed-Precision via the with torch.autocast(device_type='torch_device'): decorator, or load the model with the torch_dtype argument. Example: model = AutoModel.from_pretrained("openai/whisper-tiny", attn_implementation="flash_attention_2", torch_dtype=torch.float16)
[2026-02-04 15:48:05,884] [INFO] [partition_parameters.py:348:exit] finished initializing model - num_params = 729, num_elems = 8.29B
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [03:31<00:00, 53.00s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [03:31<00:00, 53.00s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [03:32<00:00, 53.00s/it]
[2026-02-04 15:51:37,929] [INFO] [config.py:733:init] Config mesh_device None world_size = 4
[2026-02-04 15:51:37,929] [INFO] [config.py:733:init] Config mesh_device None world_size = 4
[2026-02-04 15:51:37,934] [INFO] [config.py:733:init] Config mesh_device None world_size = 4
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [03:51<00:00, 57.96s/it]
[2026-02-04 15:51:57,821] [INFO] [config.py:733:init] Config mesh_device None world_size = 4
Loading checkpoint shards: 0%| | 0/4 [00:00<?, ?it/s][2026-02-04 15:51:58,037] [INFO] [partition_parameters.py:348:exit] finished initializing model - num_params = 1458, num_elems = 16.58B
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:07<00:00, 1.79s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:07<00:00, 1.80s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:07<00:00, 1.80s/it]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:07<00:00, 1.85s/it]
[2026-02-04 15:52:06,459] [INFO] [config.py:733:init] Config mesh_device None world_size = 4
[2026-02-04 15:52:06,460] [INFO] [config.py:733:init] Config mesh_device None world_size = 4
[2026-02-04 15:52:06,460] [INFO] [config.py:733:init] Config mesh_device None world_size = 4
[2026-02-04 15:52:06,487] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed info: version=0.15.4, git-hash=unknown, git-branch=unknown
[2026-02-04 15:52:06,487] [INFO] [config.py:733:init] Config mesh_device None world_size = 4
[2026-02-04 15:52:06,501] [INFO] [logging.py:128:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2026-02-04 15:52:06,503] [INFO] [logging.py:128:log_dist] [Rank 0] Creating ZeRO Offload
[2026-02-04 15:52:06,787] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [begin]
[2026-02-04 15:52:06,788] [INFO] [utils.py:782:see_memory_usage] MA 7.74 GB Max_MA 10.53 GB CA 11.46 GB Max_CA 11 GB
[2026-02-04 15:52:06,789] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 4.2 GB, percent = 0.6%
Parameter Offload: Total persistent parameters: 848896 in 368 params
[2026-02-04 15:52:07,043] [INFO] [utils.py:781:see_memory_usage] DeepSpeedZeRoOffload initialize [end]
[2026-02-04 15:52:07,044] [INFO] [utils.py:782:see_memory_usage] MA 7.74 GB Max_MA 7.74 GB CA 11.46 GB Max_CA 11 GB
[2026-02-04 15:52:07,045] [INFO] [utils.py:789:see_memory_usage] CPU Virtual Memory: used = 4.21 GB, percent = 0.6%
[2026-02-04 15:52:07,047] [INFO] [config.py:999:print] DeepSpeedEngine configuration:
[2026-02-04 15:52:07,047] [INFO] [config.py:1003:print] activation_checkpointing_config {
"partition_activations": false,
"contiguous_memory_optimization": false,
"cpu_checkpointing": false,
"number_checkpoints": null,
"synchronize_checkpoint_boundary": false,
"profile": false
}
[2026-02-04 15:52:07,047] [INFO] [config.py:1003:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True, 'use_gds': False}
[2026-02-04 15:52:07,047] [INFO] [config.py:1003:print] amp_enabled .................. False
[2026-02-04 15:52:07,047] [INFO] [config.py:1003:print] amp_params ................... False
[2026-02-04 15:52:07,047] [INFO] [config.py:1003:print] autotuning_config ............ {
"enabled": false,
"start_step": null,
"end_step": null,
"metric_path": null,
"arg_mappings": null,
"metric": "throughput",
"model_info": null,
"results_dir": "autotuning_results",
"exps_dir": "autotuning_exps",
"overwrite": true,
"fast": true,
"start_profile_step": 3,
"end_profile_step": 5,
"tuner_type": "gridsearch",
"tuner_early_stopping": 5,
"tuner_num_trials": 50,
"model_info_path": null,
"mp_size": 1,
"max_train_batch_size": null,
"min_train_batch_size": 1,
"max_train_micro_batch_size_per_gpu": 1.024000e+03,
"min_train_micro_batch_size_per_gpu": 1,
"num_tuning_micro_batch_sizes": 3
}
[2026-02-04 15:52:07,047] [INFO] [config.py:1003:print] bfloat16_enabled ............. True
[2026-02-04 15:52:07,047] [INFO] [config.py:1003:print] bfloat16_immediate_grad_update False
[2026-02-04 15:52:07,047] [INFO] [config.py:1003:print] checkpoint_parallel_write_pipeline False
[2026-02-04 15:52:07,047] [INFO] [config.py:1003:print] checkpoint_tag_validation_enabled True
[2026-02-04 15:52:07,047] [INFO] [config.py:1003:print] checkpoint_tag_validation_fail False
[2026-02-04 15:52:07,047] [INFO] [config.py:1003:print] comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x7f374af9c450>
[2026-02-04 15:52:07,047] [INFO] [config.py:1003:print] communication_data_type ...... None
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] curriculum_enabled_legacy .... False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] curriculum_params_legacy ..... False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] data_efficiency_config ....... {'enabled': False, 'seed': 1234, 'data_sampling': {'enabled': False, 'num_epochs': 1000, 'num_workers': 0, 'curriculum_learning': {'enabled': False}}, 'data_routing': {'enabled': False, 'random_ltd': {'enabled': False, 'layer_token_lr_schedule': {'enabled': False}}}}
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] data_efficiency_enabled ...... False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] dataloader_drop_last ......... False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] disable_allgather ............ False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] dump_state ................... False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] dynamic_loss_scale_args ...... None
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] eigenvalue_enabled ........... False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] eigenvalue_gas_boundary_resolution 1
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] eigenvalue_layer_name ........ bert.encoder.layer
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] eigenvalue_layer_num ......... 0
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] eigenvalue_max_iter .......... 100
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] eigenvalue_stability ......... 1e-06
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] eigenvalue_tol ............... 0.01
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] eigenvalue_verbose ........... False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] elasticity_enabled ........... False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] flops_profiler_config ........ {
"enabled": false,
"recompute_fwd_factor": 0.0,
"profile_step": 1,
"module_depth": -1,
"top_modules": 1,
"detailed": true,
"output_file": null
}
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] fp16_auto_cast ............... None
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] fp16_enabled ................. False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] fp16_master_weights_and_gradients False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] global_rank .................. 0
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] grad_accum_dtype ............. None
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] gradient_accumulation_steps .. 8
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] gradient_clipping ............ 5.0
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] gradient_predivide_factor .... 1.0
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] graph_harvesting ............. False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] hybrid_engine ................ enabled=False max_out_tokens=512 inference_tp_size=1 release_inference_cache=False pin_parameters=True tp_gather_partition_size=8
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] initial_dynamic_scale ........ 1
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] load_universal_checkpoint .... False
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] loss_scale ................... 1.0
[2026-02-04 15:52:07,048] [INFO] [config.py:1003:print] memory_breakdown ............. False
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] mics_hierarchial_params_gather False
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] mics_shard_size .............. -1
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] monitor_config ............... tensorboard=TensorBoardConfig(enabled=False, output_path='', job_name='DeepSpeedJobName') comet=CometConfig(enabled=False, samples_log_interval=100, project=None, workspace=None, api_key=None, experiment_name=None, experiment_key=None, online=None, mode=None) wandb=WandbConfig(enabled=False, group=None, team=None, project='deepspeed') csv_monitor=CSVConfig(enabled=False, output_path='', job_name='DeepSpeedJobName')
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] nebula_config ................ {
"enabled": false,
"persistent_storage_path": null,
"persistent_time_interval": 100,
"num_of_version_in_retention": 2,
"enable_nebula_load": true,
"load_path": null
}
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] optimizer_legacy_fusion ...... False
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] optimizer_name ............... None
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] optimizer_params ............. None
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0, 'pipe_partitioned': True, 'grad_partitioned': True}
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] pld_enabled .................. False
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] pld_params ................... False
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] prescale_gradients ........... False
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] scheduler_name ............... None
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] scheduler_params ............. None
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] seq_parallel_communication_data_type torch.float32
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] sparse_attention ............. None
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] sparse_gradients_enabled ..... False
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] steps_per_print .............. inf
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] timers_config ................ enabled=True synchronized=True
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] train_batch_size ............. 32
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] train_micro_batch_size_per_gpu 1
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] use_data_before_expert_parallel_ False
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] use_node_local_storage ....... False
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] wall_clock_breakdown ......... False
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] weight_quantization_config ... None
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] world_size ................... 4
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] zero_allow_untested_optimizer False
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] zero_config .................. stage=3 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 use_multi_rank_bucket_allreduce=True allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=True load_from_fp32_weights=True elastic_checkpoint=False offload_param=DeepSpeedZeroOffloadParamConfig(device='none', nvme_path=None, buffer_count=5, buffer_size=100000000, max_in_cpu=1000000000, pin_memory=True) offload_optimizer=DeepSpeedZeroOffloadOptimizerConfig(device='none', nvme_path=None, buffer_count=4, pin_memory=True, pipeline_read=False, pipeline_write=False, fast_init=False, ratio=1.0) sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=True use_all_reduce_for_fetch_params=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False zero_hpz_partition_size=1 zero_quantized_weights=False zero_quantized_nontrainable_weights=False zero_quantized_gradients=False mics_shard_size=-1 mics_hierarchical_params_gather=False memory_efficient_linear=True pipeline_loading_checkpoint=False override_module_apply=True
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] zero_enabled ................. True
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] zero_force_ds_cpu_optimizer .. True
[2026-02-04 15:52:07,049] [INFO] [config.py:1003:print] zero_optimization_stage ...... 3
[2026-02-04 15:52:07,050] [INFO] [config.py:989:print_user_config] json = {
"fp16": {
"enabled": false,
"loss_scale": 0,
"loss_scale_window": 1000,
"initial_scale_power": 16,
"hysteresis": 2,
"min_loss_scale": 1
},
"bf16": {
"enabled": true
},
"zero_optimization": {
"stage": 3,
"offload_optimizer": {
"device": "none",
"pin_memory": true
},
"offload_param": {
"device": "none",
"pin_memory": true
},
"overlap_comm": true,
"contiguous_gradients": true,
"sub_group_size": 1.000000e+09,
"reduce_bucket_size": "auto",
"stage3_prefetch_bucket_size": "auto",
"stage3_param_persistence_threshold": "auto",
"stage3_max_live_parameters": 1.000000e+09,
"stage3_max_reuse_distance": 1.000000e+09,
"stage3_gather_16bit_weights_on_model_save": true
},
"gradient_accumulation_steps": 8,
"gradient_clipping": 5.0,
"steps_per_print": inf,
"train_batch_size": 32,
"train_micro_batch_size_per_gpu": 1,
"wall_clock_breakdown": false,
"zero_optimization.reduce_bucket_size": 1.284506e+07,
"zero_optimization.stage3_param_persistence_threshold": 3.584000e+04,
"zero_optimization.stage3_prefetch_bucket_size": 1.156055e+07
}
Gradient accumulation steps mismatch: GradientAccumulationPlugin has 1, DeepSpeed config has 8. Using DeepSpeed's value.
Parameter Offload: Total persistent parameters: 848896 in 368 params
qwen-vl-utils using decord to read video.
qwen-vl-utils using decord to read video.
qwen-vl-utils using decord to read video.
wandb: Tracking run with wandb version 0.24.1
wandb: W&B syncing is set to offline in this directory. Run wandb online or set WANDB_MODE=online to enable cloud syncing.
wandb: Run data is saved locally in /data/Video-R1/src/r1-v/wandb/offline-run-20260204_155214-8ahtgd7r
wandb: Detected [huggingface_hub.inference, openai] in use.
wandb: Use W&B Weave for improved LLM call tracing. Install Weave with pip install weave then add import weave to the top of your script.
wandb: For more information, check out the docs at: https://weave-docs.wandb.ai/
0%| | 0/833 [00:00<?, ?it/s]qwen-vl-utils using decord to read video.
/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/utils/checkpoint.py:87: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/utils/checkpoint.py:87: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/utils/checkpoint.py:87: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/utils/checkpoint.py:87: UserWarning: None of the inputs have requires_grad=True. Gradients will be None
warnings.warn(
use_cache=True is incompatible with gradient checkpointing. Setting use_cache=False...
use_cache=True is incompatible with gradient checkpointing. Setting use_cache=False...
use_cache=True is incompatible with gradient checkpointing. Setting use_cache=False...
use_cache=True is incompatible with gradient checkpointing. Setting use_cache=False...
path: /data/LlamaFactory/data/videos/002756_f.mp4
problem_id: 002756_f
prompt_length: 1007
path: /data/LlamaFactory/data/videos/004728_e.mp4
problem_id:path: 004728_e
prompt_length: 1076
/data/LlamaFactory/data/videos/004914_i.mp4
problem_id: 004914_i
prompt_length: 850
path: /data/LlamaFactory/data/videos/004290_a.mp4
problem_id: 004290_a
prompt_length: 743
Invalidate trace cache @ step 0 and module 1522: cache has only 0 modules
gt :3 3 3
out:4 4 3
gt :2 3 3
out:1 2 1
gt :4 2 4
out:3 4 3
gt :4 3 4
out:3 3 3
gt :3 3 3
out:3 3 3
gt :2 3 3
out:1 4 2
gt :4 2 4
out:3 4 3
gt :4 3 4
out:2 3 3
gt :3 3 3
out:3 3 3
gt :4 2 4
out:3 2 2
gt :4 3 4
out:4 4 4
gt :2 3 3
out:2 2 3
gt :3 3 3
out:3 3 3
gt :4 2 4
out:4 4 4
gt :4 3 4
out:3 3 3
gt :2 3 3
out:1 3 1
tensor([0., 0., 0., 0.], device='cuda:1')
tensor([490, 283, 495, 348], device='cuda:1')
tensor([0.0000, 0.1000, 0.7000, 0.0000], device='cuda:2')tensor([0.4000, 0.0000, 0.9000, 0.4000], device='cuda:3')tensor([0.6000, 1.0000, 1.2000, 1.2000], device='cuda:0')

tensor([380, 253, 552, 442], device='cuda:2')tensor([819, 684, 462, 274], device='cuda:3')

tensor([493, 285, 496, 366], device='cuda:0')
path: /data/LlamaFactory/data/videos/003975_x.mp4
problem_id: 003975_x
prompt_length: 746
path: /data/LlamaFactory/data/videos/001031_v.mp4
problem_id: 001031_v
prompt_length: 862
path: /data/LlamaFactory/data/videos/000702_e.mp4
problem_id: 000702_e
prompt_length: 1003
path: /data/LlamaFactory/data/videos/000425_a.mp4
problem_id: 000425_a
prompt_length: 795
Invalidate trace cache @ step 0 and module 2283: cache has only 0 modules
gt :3 4 3
out:3 3 3
gt :4 5 4
out:3 3 4
gt :3 4 3
out:1 3 2
gt :4 5 4
out:4 3 4
gt :3 4 3
out:3 3 4
gt :4 5 4
out:3 4 4
gt :3 4 3
out:2 3 2
gt :4 5 4
out:3 3 4
tensor([0.7000, 0.0000, 0.4000, 0.1000], device='cuda:1')
tensor([519, 544, 660, 609], device='cuda:1')
tensor([0.0000, 0.0000, 0.4000, 0.0000], device='cuda:2')
tensor([649, 743, 906, 461], device='cuda:2')
gt :3 3 3
out:3 3 2
gt :3 3 3
out:3 2 3
gt :3 3 3
out:3 3 3
gt :3 3 3
out:4 2 3
tensor([0.9000, 0.7000, 1.2000, 0.6000], device='cuda:0')
tensor([423, 703, 455, 472], device='cuda:0')
gt :3 2 3
out:3 2 3
gt :3 2 3
out:3 2 5
gt :3 2 3
out:3 2 3
gt :3 2 3
out:3 2 3
tensor([1., 0., 1., 1.], device='cuda:3')
tensor([817, 389, 535, 525], device='cuda:3')
path: /data/LlamaFactory/data/videos/003888_f.mp4
problem_id: 003888_f
prompt_length: 1004
path: /data/LlamaFactory/data/videos/002727_c.mp4
problem_id: 002727_c
prompt_length: 738
path: /data/LlamaFactory/data/videos/001071_z.mp4
problem_id: 001071_z
prompt_length: 1055
path: /data/LlamaFactory/data/videos/003865_x.mp4
problem_id: 003865_x
prompt_length: 739
Invalidate trace cache @ step 0 and module 3044: cache has only 0 modules
gt :3 3 3
out:3 4 3
gt :3 3 3
out:3 3 3
gt :3 3 3
out:3 3 3
gt :3 3 3
out:4 2 4
gt :3 3 3
out:3 4 3
gt :3 3 3
out:3 2 3
tensor([0.7000, 1.2000, 1.0000, 0.9000], device='cuda:2')gt :3 5 3

out:4 5 2
tensor([535, 388, 636, 322], device='cuda:2')
gt :3 3 3
out:2 1 4
gt :3 5 3
out:1 4 3
gt :3 3 3
out:2 2 3
gt :3 5 3
out:3 4 2
tensor([0.1000, 0.7000, 0.0000, 0.4000], device='cuda:0')
tensor([276, 849, 657, 575], device='cuda:0')
gt :3 5 3
out:3 4 2
tensor([0.6000, 0.0000, 0.6000, 0.4000], device='cuda:1')
tensor([476, 780, 431, 782], device='cuda:1')
gt :4 3 3
out:3 3 3
gt :4 3 3
out:4 3 3
gt :4 3 3
out:3 3 4
gt :4 3 3
out:3 3 3
tensor([0.9000, 1.2000, 0.6000, 0.9000], device='cuda:3')
tensor([412, 444, 417, 401], device='cuda:3')
path: /data/LlamaFactory/data/videos/001113_m.mp4
problem_id: 001113_m
prompt_length:path: 744
/data/LlamaFactory/data/videos/004993_e.mp4
problem_id: 004993_e
prompt_length: 1001
path: /data/LlamaFactory/data/videos/000861_m.mp4
problem_id: 000861_m
prompt_length: 805
path: /data/LlamaFactory/data/videos/001141_m.mp4
problem_id: 001141_m
prompt_length: 765
Invalidate trace cache @ step 0 and module 3805: cache has only 0 modules
gt :3 2 3
out:3 3 4
gt :2 3 2
out:3 3 3
gt :3 2 3
out:3 3 4
gt :2 3 2
out:3 3 3
gt :3 2 3
out:2 1 3
gt :2 3 2
out:4 3 4
gt :3 2 3
out:2 1 3
gt :2 3 2
out:3 3 3
gt :2 3 2
out:3 3 3
gt :2 3 2
out:4 3 3
tensor([0.6000, 0.4000, 0.4000, 0.4000], device='cuda:3')
tensor([472, 522, 531, 563], device='cuda:3')
tensor([0.6000, 0.4000, 0.0000, 0.6000], device='cuda:2')
tensor([500, 660, 448, 375], device='cuda:2')
gt :2 3 2
out:2 2 4
gt :2 3 2
out:3 4 4
tensor([0.4000, 0.0000, 0.0000, 0.0000], device='cuda:0')
tensor([605, 937, 734, 500], device='cuda:0')
gt :3 4 1
out:3 4 2
gt :3 4 1
out:3 4 3
gt :3 4 1
out:3 5 3
gt :3 4 1
out:3 4 3
tensor([0.7000, 0.0000, 0.0000, 0.0000], device='cuda:1')
tensor([338, 499, 688, 403], device='cuda:1')
path: /data/LlamaFactory/data/videos/003498_g.mp4
problem_id: 003498_g
prompt_length: 988
path: /data/LlamaFactory/data/videos/001559_t.mp4
problem_id: 001559_t
prompt_length: 1099
path:path: /data/LlamaFactory/data/videos/004022_b.mp4
problem_id: 004022_b
prompt_length: 748
/data/LlamaFactory/data/videos/004975_q.mp4
problem_id: 004975_q
prompt_length: 748
Invalidate trace cache @ step 0 and module 4566: cache has only 0 modules
gt :2 3 3
out:2 4 3
gt :1 3 1
out:3 4 3gt :2 3 3
out:3 3 4

gt :2 3 3
out:3 3 3
gt :1 3 1
out:3 3 3
gt :2 3 3
out:3 3 2
gt :1 3 1
out:3 3 3
tensor([0.7000, 0.6000, 0.7000, 0.6000], device='cuda:2')
tensor([558, 405, 518, 387], device='cuda:2')
gt :1 3 1
out:3 3 3
tensor([0., 0., 0., 0.], device='cuda:0')
tensor([632, 644, 528, 493], device='cuda:0')
gt :4 4 4
out:4 3 3
gt :4 4 4
out:3 4 3
gt :4 4 4
out:4 4 4
gt :4 4 4
out:4 5 4
tensor([0.6000, 0.4000, 1.2000, 0.7000], device='cuda:3')gt :4 5 3

out:4 4 4
tensor([461, 842, 459, 567], device='cuda:3')
gt :4 5 3
out:4 4 5
gt :4 5 3
out:3 4 2
gt :4 5 3
out:4 4 4
tensor([0.6000, 0.0000, 0.1000, 0.6000], device='cuda:1')
tensor([353, 568, 705, 478], device='cuda:1')
path: /data/LlamaFactory/data/videos/001958_c.mp4
problem_id: 001958_c
prompt_length: 809
path: /data/LlamaFactory/data/videos/001086_p.mp4
problem_id: 001086_p
prompt_length: 1107
path: /data/LlamaFactory/data/videos/002625_d.mp4
problem_id: 002625_d
prompt_length: 813
path: /data/LlamaFactory/data/videos/002756_i.mp4
problem_id: 002756_i
prompt_length: 854
Invalidate trace cache @ step 0 and module 5327: cache has only 0 modules
gt :4 5 4
out:3 3 3
gt :1 2 2
out:2 2 2
gt :4 5 4
out:3 4 3
gt :1 2 2
out:1 1 1
gt :4 5 4
out:3 4 4
gt :1 2 2
out:2 2 2
gt :4 5 4
out:4 5 4
gt :1 2 2
out:3 3 2
tensor([0.0000, 0.1000, 0.4000, 1.2000], device='cuda:1')
tensor([544, 463, 513, 439], device='cuda:1')
tensor([0.7000, 0.4000, 0.7000, 0.0000], device='cuda:3')
tensor([665, 619, 648, 397], device='cuda:3')
gt :2 4 2
out:3 5 3
gt :2 4 2
out:3 3 3
gt :2 4 2
out:3 4 3
gt :2 4 2
out:3 4 3
tensor([0.1000, 0.1000, 0.6000, 0.6000], device='cuda:2')
tensor([705, 628, 422, 448], device='cuda:2')
gt :3 3 2
out:2 3 5
gt :3 3 2
out:3 4 4
gt :3 3 2
out:2 3 3
gt :3 3 2
out:4 5 4
tensor([0.0000, 0.0000, 0.4000, 0.0000], device='cuda:0')
tensor([429, 412, 441, 531], device='cuda:0')
path: /data/LlamaFactory/data/videos/003653_w.mp4
problem_id: 003653_w
prompt_length: 858
Invalidate trace cache @ step 0 and module 6088: cache has only 0 modules
path: /data/LlamaFactory/data/videos/000015_r.mp4
problem_id: 000015_r
prompt_length: 1103
path: /data/LlamaFactory/data/videos/001113_b.mp4
problem_id: 001113_b
prompt_length: 746
path: /data/LlamaFactory/data/videos/003931_n.mp4
problem_id: 003931_n
prompt_length: 770
gt :4 5 3
out:4 4 4
gt :1 3 1
out:2 2 2
gt :4 5 3
out:4 3 3
gt :1 3 1
out:3 3 2
gt :2 3 3
gt :4 5 3
out:3 3 3out:4 4 4

gt :1 3 1
out:3 3 3
gt :4 5 3
out:1 4 2
gt :1 3 1
out:3 3 3
gt :2 3 3
out:2 4 3
tensor([0.6000, 0.0000, 0.6000, 0.0000], device='cuda:1')
tensor([393, 559, 511, 497], device='cuda:1')
gt :2 3 3
out:2 4 2
tensor([0.1000, 0.0000, 0.0000, 0.0000], device='cuda:3')
tensor([566, 491, 660, 459], device='cuda:3')
gt :2 3 3
out:2 2 3
tensor([0.9000, 0.9000, 0.4000, 0.9000], device='cuda:0')
tensor([509, 440, 660, 493], device='cuda:0')
gt :5 4 2
out:3 3 2
gt :5 4 2
out:3 4 2
gt :5 4 2
out:3 3 2
gt :5 4 2
out:3 3 3
tensor([0., 0., 0., 0.], device='cuda:2')
tensor([141, 524, 373, 516], device='cuda:2')
path: /data/LlamaFactory/data/videos/001992_z.mp4
problem_id: 001992_z
prompt_length: 1018
path:path: /data/LlamaFactory/data/videos/001050_z.mp4
problem_id: 001050_z
prompt_length: /data/LlamaFactory/data/videos/002711_c.mp41003

problem_id: 002711_c
prompt_length: 818
path: /data/LlamaFactory/data/videos/004037_z.mp4
problem_id: 004037_z
prompt_length: 1063
Invalidate trace cache @ step 0 and module 6849: cache has only 0 modules
gt :4 4 3
out:4 4 4
gt :3 3 3
out:3 3 3
gt :4 4 3
out:3 2 5
gt :3 3 3
out:3 2 2
gt :4 4 3
out:3 5 3
gt :3 3 3
out:4 3 4
gt :4 4 3
out:2 3 2
gt :3 3 3
out:2 3 2
tensor([0.7000, 0.0000, 0.6000, 0.0000], device='cuda:1')
tensor([677, 723, 413, 542], device='cuda:1')
tensor([1.0000, 0.4000, 0.6000, 0.6000], device='cuda:2')
tensor([640, 555, 466, 398], device='cuda:2')
gt :4 3 3
out:3 4 2
gt :4 3 3
out:3 4 4
gt :4 3 3
out:4 3 3
gt :4 3 3
out:3 4 5
tensor([0.1000, 0.1000, 1.0000, 0.0000], device='cuda:0')
tensor([734, 562, 589, 471], device='cuda:0')
gt :4 1 4
out:4 4 3
gt :4 1 4
out:3 3 3
gt :4 1 4
out:4 2 3
gt :4 1 4
out:5 2 3
tensor([0.0000, 0.0000, 0.4000, 0.1000], device='cuda:3')
tensor([304, 762, 391, 642], device='cuda:3')
Traceback (most recent call last):
[rank1]: Traceback (most recent call last):
[rank1]: File "/data/Video-R1/src/r1-v/src/open_r1/grpo_vs2_sft.py", line 359, in
[rank1]: main(script_args, training_args, model_args)
[rank1]: File "/data/Video-R1/src/r1-v/src/open_r1/grpo_vs2_sft.py", line 348, in main
[rank1]: trainer.train()
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 2241, in train
[rank1]: return inner_training_loop(
[rank1]: ^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
[rank1]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 3740, in training_step
[rank1]: self.accelerator.backward(loss, **kwargs)
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/accelerate/accelerator.py", line 2844, in backward
[rank1]: self.deepspeed_engine_wrapped.backward(loss, sync_gradients=self.sync_gradients, **kwargs)
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/accelerate/utils/deepspeed.py", line 281, in backward
[rank1]: self.engine.step()
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2213, in step
[rank1]: self._take_model_step(lr_kwargs)
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2119, in _take_model_step
[rank1]: self.optimizer.step()
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
[rank1]: ret_val = func(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 2095, in step
[rank1]: self._optimizer_step(sub_group_id)
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 979, in _optimizer_step
[rank1]: self.optimizer.step()
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/lr_scheduler.py", line 137, in wrapper
[rank1]: return func.get(opt, opt.class)(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 487, in wrapper
[rank1]: out = func(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 91, in _use_grad
[rank1]: ret = func(self, *args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 220, in step
[rank1]: adamw(
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 154, in maybe_fallback
[rank1]: return func(*args, **kwargs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 782, in adamw
[rank1]: func(
[rank1]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 606, in _multi_tensor_adamw
[rank1]: exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs)
[rank1]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank1]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.98 GiB. GPU 1 has a total capacity of 44.64 GiB of which 926.25 MiB is free. Process 790313 has 43.73 GiB memory in use. Of the allocated memory 38.79 GiB is allocated by PyTorch, and 4.09 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank2]: Traceback (most recent call last):
[rank2]: File "/data/Video-R1/src/r1-v/src/open_r1/grpo_vs2_sft.py", line 359, in
[rank2]: main(script_args, training_args, model_args)
[rank2]: File "/data/Video-R1/src/r1-v/src/open_r1/grpo_vs2_sft.py", line 348, in main
[rank2]: trainer.train()
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 2241, in train
[rank2]: return inner_training_loop(
[rank2]: ^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
[rank2]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 3740, in training_step
[rank2]: self.accelerator.backward(loss, **kwargs)
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/accelerate/accelerator.py", line 2844, in backward
[rank2]: self.deepspeed_engine_wrapped.backward(loss, sync_gradients=self.sync_gradients, **kwargs)
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/accelerate/utils/deepspeed.py", line 281, in backward
[rank2]: self.engine.step()
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2213, in step
[rank2]: self._take_model_step(lr_kwargs)
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2119, in _take_model_step
[rank2]: self.optimizer.step()
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
[rank2]: ret_val = func(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 2095, in step
[rank2]: self._optimizer_step(sub_group_id)
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 979, in _optimizer_step
[rank2]: self.optimizer.step()
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/lr_scheduler.py", line 137, in wrapper
[rank2]: return func.get(opt, opt.class)(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 487, in wrapper
[rank2]: out = func(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 91, in _use_grad
[rank2]: ret = func(self, *args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 220, in step
[rank2]: adamw(
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 154, in maybe_fallback
[rank2]: return func(*args, **kwargs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 782, in adamw
[rank2]: func(
[rank2]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 606, in _multi_tensor_adamw
[rank2]: exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs)
[rank2]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank2]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.98 GiB. GPU 2 has a total capacity of 44.64 GiB of which 818.25 MiB is free. Process 790314 has 43.83 GiB memory in use. Of the allocated memory 38.79 GiB is allocated by PyTorch, and 4.20 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
File "/data/Video-R1/src/r1-v/src/open_r1/grpo_vs2_sft.py", line 359, in
main(script_args, training_args, model_args)
File "/data/Video-R1/src/r1-v/src/open_r1/grpo_vs2_sft.py", line 348, in main
trainer.train()
[rank3]: Traceback (most recent call last):
[rank3]: File "/data/Video-R1/src/r1-v/src/open_r1/grpo_vs2_sft.py", line 359, in
[rank3]: main(script_args, training_args, model_args)
[rank3]: File "/data/Video-R1/src/r1-v/src/open_r1/grpo_vs2_sft.py", line 348, in main
[rank3]: trainer.train()
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 2241, in train
[rank3]: return inner_training_loop(
[rank3]: ^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
[rank3]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 3740, in training_step
[rank3]: self.accelerator.backward(loss, **kwargs)
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/accelerate/accelerator.py", line 2844, in backward
[rank3]: self.deepspeed_engine_wrapped.backward(loss, sync_gradients=self.sync_gradients, **kwargs)
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/accelerate/utils/deepspeed.py", line 281, in backward
[rank3]: self.engine.step()
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2213, in step
[rank3]: self._take_model_step(lr_kwargs)
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2119, in _take_model_step
[rank3]: self.optimizer.step()
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
[rank3]: ret_val = func(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 2095, in step
[rank3]: self._optimizer_step(sub_group_id)
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 979, in _optimizer_step
[rank3]: self.optimizer.step()
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/lr_scheduler.py", line 137, in wrapper
[rank3]: return func.get(opt, opt.class)(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 487, in wrapper
[rank3]: out = func(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 91, in _use_grad
[rank3]: ret = func(self, *args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 220, in step
[rank3]: adamw(
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 154, in maybe_fallback
[rank3]: return func(*args, **kwargs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 782, in adamw
[rank3]: func(
[rank3]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 606, in _multi_tensor_adamw
[rank3]: exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs)
[rank3]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank3]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.98 GiB. GPU 3 has a total capacity of 44.64 GiB of which 996.25 MiB is free. Process 790315 has 43.66 GiB memory in use. Of the allocated memory 38.79 GiB is allocated by PyTorch, and 4.10 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 2241, in train
return inner_training_loop(
^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 3740, in training_step
self.accelerator.backward(loss, **kwargs)
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/accelerate/accelerator.py", line 2844, in backward
self.deepspeed_engine_wrapped.backward(loss, sync_gradients=self.sync_gradients, **kwargs)
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/accelerate/utils/deepspeed.py", line 281, in backward
self.engine.step()
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2213, in step
self._take_model_step(lr_kwargs)
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2119, in _take_model_step
self.optimizer.step()
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
ret_val = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 2095, in step
self._optimizer_step(sub_group_id)
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 979, in _optimizer_step
self.optimizer.step()
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/lr_scheduler.py", line 137, in wrapper
return func.get(opt, opt.class)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 487, in wrapper
out = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 91, in _use_grad
ret = func(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 220, in step
adamw(
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 154, in maybe_fallback
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 782, in adamw
func(
File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 606, in _multi_tensor_adamw
exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.98 GiB. GPU 0 has a total capacity of 44.64 GiB of which 1.99 GiB is free. Process 790312 has 42.64 GiB memory in use. Of the allocated memory 38.79 GiB is allocated by PyTorch, and 3.02 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
[rank0]: Traceback (most recent call last):
[rank0]: File "/data/Video-R1/src/r1-v/src/open_r1/grpo_vs2_sft.py", line 359, in
[rank0]: main(script_args, training_args, model_args)
[rank0]: File "/data/Video-R1/src/r1-v/src/open_r1/grpo_vs2_sft.py", line 348, in main
[rank0]: trainer.train()
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 2241, in train
[rank0]: return inner_training_loop(
[rank0]: ^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 2548, in _inner_training_loop
[rank0]: tr_loss_step = self.training_step(model, inputs, num_items_in_batch)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/transformers/trainer.py", line 3740, in training_step
[rank0]: self.accelerator.backward(loss, **kwargs)
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/accelerate/accelerator.py", line 2844, in backward
[rank0]: self.deepspeed_engine_wrapped.backward(loss, sync_gradients=self.sync_gradients, **kwargs)
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/accelerate/utils/deepspeed.py", line 281, in backward
[rank0]: self.engine.step()
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2213, in step
[rank0]: self._take_model_step(lr_kwargs)
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/engine.py", line 2119, in _take_model_step
[rank0]: self.optimizer.step()
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/utils/nvtx.py", line 18, in wrapped_fn
[rank0]: ret_val = func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 2095, in step
[rank0]: self._optimizer_step(sub_group_id)
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/deepspeed/runtime/zero/stage3.py", line 979, in _optimizer_step
[rank0]: self.optimizer.step()
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/lr_scheduler.py", line 137, in wrapper
[rank0]: return func.get(opt, opt.class)(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 487, in wrapper
[rank0]: out = func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 91, in _use_grad
[rank0]: ret = func(self, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 220, in step
[rank0]: adamw(
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/optimizer.py", line 154, in maybe_fallback
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 782, in adamw
[rank0]: func(
[rank0]: File "/data/miniconda3/envs/video_r1/lib/python3.11/site-packages/torch/optim/adamw.py", line 606, in _multi_tensor_adamw
[rank0]: exp_avg_sq_sqrt = torch._foreach_sqrt(device_exp_avg_sqs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.98 GiB. GPU 0 has a total capacity of 44.64 GiB of which 1.99 GiB is free. Process 790312 has 42.64 GiB memory in use. Of the allocated memory 38.79 GiB is allocated by PyTorch, and 3.02 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)