Hugging Face: v1.6.0 release adds FSDPv2, DeepSpeed TP, and XCCL support in Accelerate | SignalBreak | SignalBreak

Hugging Face: v1.6.0 release adds FSDPv2, DeepSpeed TP, and XCCL support in Accelerate | SignalBreak | SignalBreak

Source text

FSDPv2 support This release introduces the support for FSDPv2 thanks to @S1ro1 . If you are using python code, you need to set fsdp_version=2 in FullyShardedDataParallelPlugin : from accelerate import FullyShardedDataParallelPlugin , Accelerator fsdp_plugin = FullyShardedDataParallelPlugin ( fsdp_version = 2 # other options... ) accelerator = Accelerator ( fsdp_plugin = fsdp_plugin ) If want to convert a YAML config that contains the FSDPv1 config to FSDPv2 one , use our conversion tool: accelerate to-fsdp2 --config_file config.yaml --output_file new_config.yaml` To learn more about the difference between FSDPv1 and FSDPv2, read the following documentation . DeepSpeed TP support We have added initial support for DeepSpeed + TP. Not many changes were required as the DeepSpeed APIs was already compatible. We only needed to make sure that the dataloader was compatible with TP and that we were able to save the TP weights. Thanks @inkcherry for the work ! #3390 . To use TP with deepspeed, you need to update the setting in the deepspeed config file by including tensor_parallel key: .... "tensor_parallel":{ "autotp_size": ${autotp_size} }, ... More details in this deepspeed PR . Support for XCCL distributed backend We've added support for XCCL which is an Intel distributed backend which can be used with XPU devices. More details in this torch PR . Thanks @dvrogozh for the integration ! What's Changed Add log_artifact , log_artifacts and log_figure capabilities to the MLflowTracker. by @luiz0992 in #3419 tensor parallel dataloder for deepspeed accelerator by @inkcherry in #3390 Fix prod issues by @muellerzr in #3441 Fix attribute issue with deepspeed tp by @SunMarc in #3443 Fixed typo in the multi node FSDP slurm example script by @JacobB33 in #3447 feat: Add no_ssh and slurm multinode launcher options for deepspeed by @hsmallbone in #3329 Fixup ao module filter func by @muellerzr in #3450 remove device index workaround on xpu since xpu supports integer device index as cuda

View original source