qi,ddlZddlZddlZddlZddlmZmZmZmZddl m Z m Z m Z m Z mZddlZddlmZddlmcmcmZddlmcmcmZddlmcmcmZddlmZddlm Z ddl!m"Z"ddl#m$Z$ddl%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,ddl-m.Z.m/Z/m0Z0m1Z1dd l2m3Z3dd l4m5Z5m6Z6m7Z7m8Z8m9Z9m:Z:m;Z;mZ>dd l?m@Z@dd lAmBZBddlCmDZDe rddlEmFZFdZG ddlHmIZImJZJdZLdZMeNejejfZPe eejePfZQe:je1je:je1je:je1je:je1je:je1jiZWe:je:jgZXe:je:jfZYe d_de'deQde:de e>de e"de'f dZZe de'deQde"de'fdZ[e de de\fdZ]e de"de\fdZ^e de_dejfd Z`e d!ejde_dejfd"Zad!ejde_deNejejffd#Zbe d_de'd$ejd%e eej<jd&ee eej<je eej<jfde'f d'Zed&efe d(e\ddfd)Zge de'd$ejd*ehejd+e ee_ejfde'f d,Zje de'd$ejde'fd-Zke de'de e:d.e e9d/e e6d0e\d1e\d2e_d3e_de'fd4Zle de'de'fd5Zme de'd6e5d7e\de'fd8Zne d_de'de"de'fd9Zoe de'de'fd:Zpd$ejd;efejddfd<Zqe de'd=ejd+e ee_ejfd>e eejgdfd?e\de'f d@Zre de'd;efejd=ejfdAZsdBejdCe eej<jdehejfdDZt d_dBej<jd%ehej<jdEe eej<jdehej<jfdFZudBej<jd%ehej<jdehevfdGZwdBejdehevfdHZxd$ejd*ehejd+e ee_ejfddfdIZyd+e ee_ejfdJe_dKe&de ejfdLZzd$ejd*ehejd%ehejdeNe\e\ffdMZ{dBejd>eejgdfd%ehejddfdNZ|dBejdOe ejd%ehejdKe&fdPZ}dBejd%ehejdefejfdQZ~d$ejd*ehejdRehejdOe ejddf dSZd;efejdTefejdOe ejddfdUZdVZd$ejd*ehejdOe ejdJe_dKe&dejf dWZd$ejd;efejdejddfdXZdYefejddfdZZd$ejd*ehejdeejfd[Zd*ehejddfd\Zde:fd]Zdejde jfd^Zy#eK$rdZGYwxYw)`N)Callable GeneratorIterableIterator)Any no_type_checkOptional TYPE_CHECKINGUnion) default_hooks) DeviceMesh)_get_default_group)_FSDPDeviceHandle _FSDPState_get_module_fsdp_state_is_fsdp_flattened!_named_parameters_with_duplicatesclean_tensor_name TrainingState)_FSDP_USE_FULL_PREC_IN_EVAL FlatParameterFlatParamHandleHandleShardingStrategy)_FreeEventQueue)BackwardPrefetch CPUOffloadFullOptimStateDictConfigFullStateDictConfigMixedPrecisionShardingStrategyStateDictConfig StateDictType)_Policy)DTensorExtensions)_sync_params_and_buffers)is_traceable_wrapper_subclass)RemovableHandleT) deferred_initfakeFi _fsdp_syncedstate process_groupsharding_strategypolicy device_meshreturnc(| | td|tv}|r#|||td|dt|||}n4|r||_|j d|_n||n t |_|j j|_|j j|_ |j}|r||jjz}tjj||_||jz |_|S)NzcCannot pass both process_group and device_mesh at the same time. Please just pass only one of them.zManual wrapping with zA requires explicit specification of process group or device_mesh.rmesh_dim) ValueErrorHYBRID_SHARDING_STRATEGIES*_init_process_group_state_for_hybrid_shard _device_mesh get_groupr,rranksize world_size_inter_node_pgr DefaultState_get_gradient_predivide_factor_gradient_predivide_factor_gradient_postdivide_factor)r+r,r-r.r/is_hybrid_strategydata_parallel_world_sizes h/home/ubuntu/crypto_trading_bot/.venv/lib/python3.12/site-packages/torch/distributed/fsdp/_init_utils.py_init_process_group_staterDYsI [%< <  +.HH  V^ 8K'(9':;SS  ?}kE !,E "-"7"7"7"CE "/!: @R@T  $$))+EJ**//1E$// E$8$8$=$=$?? ""AA $  $ !5#C#CC % Lc|rYt|r6||_|jd|_|jd|_nt d|j |@t}t||jj\}}||_||_n2t|r|\|_|_nt dt|t|j|_|S)Nrr2z,Expected device_mesh to have ndim=2 but got zmExpected process_group to be passed in as either None or Tuple[dist.ProcessGroup, dist.ProcessGroup] but got r,)"_is_valid_hybrid_shard_device_meshr7r8r<r,r4ndimr!_init_intra_and_inter_node_groups_device_handle device_count_is_valid_hybrid_shard_pg_typetype_get_default_comm_hook_state_inter_node_state)r+r,r/ default_groupintra_node_groupinter_node_groups rCr6r6s   -k :!,E $/#8#8!#8#DE "-"7"7"7"CE >{?O?O>PQ   *, -N 5//<<>. **// *- 89F 5E !5GGKMGZF[]  ;**E LrEcjt|txr"t|dk(xrtd|DS)Nc3PK|]}t|tj ywN) isinstancedist ProcessGroup).0pgs rC z1_is_valid_hybrid_shard_pg_type..sJb 2t001J$&)rYtuplelenallrHs rCrNrNs: =%( K  ! # K JMJ JrEcDt|txr|jdk(S)NrV)rYr rJ)r/s rCrIrIs k: . H;3C3Cq3HHrEnum_devices_per_nodec6tj|\}}|S)aU Return a process group across the current node. For example, given each row is a distinct node: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 This API would return an intra-node subgroup across [0, 1, ..., 7] or [8, 9, ..., 15] depending on the process's rank. For example, rank 3 would get [0, 1, ..., 7]. )rZ new_subgroups)rdintra_node_subgroup_s rC_init_intra_node_process_groupris!"//0DE rEglobal_process_groupc\d}tj|}tj|}||z}tj||z}t |D]?}t |Dcgc] }|||zz } }tj | |} ||k(s>| }A|t |d|Scc}w)a Return an inter-node process group where each contained rank has the same local rank. For example, given each row is a distinct node: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 This API would return inter-node process group [0, 8], [1, 9], [2, 10], and so forth depending on the process's rank. For example, rank 1 would get [1, 9], rank 5 would get [5, 13]. N)ranksbackendz. expected to assign inter-node pg, but did not)rZ get_backendget_world_sizeget_rankrange new_groupAssertionError) rjrd inter_node_pgsharding_backendr; num_nodes my_local_rank local_rankiranks_for_inter_groupgrps rC_init_inter_node_process_groupr|s M''(<=$$%9:J22IMM"67:NNM01 =B9=M! 89J!22 3! ! nn#8BRS  &M oK L   ! s%B)c0t|t||fS)a Initialize intra and inter-node process groups and return the ones corresponding to this process's rank. This function can be used to initialize process groups for ``HYBRID_SHARD`` or ``_HYBRID_SHARD_ZERO2`` in FSDP. This function assumes each node has an equal number of CUDA-enabled devices. Returns: Tuple[dist.ProcessGroup, dist.ProcessGroup]: Intra and inter-node process group. )rir|)rjrds rCrKrKs# '';<&';=QR rEmoduleignored_modulesignored_statesc| | tdd}|du}|rt|}t|dng}t| t|ngdt|dkDr"t |dt j r|}n|}t|||_t||j||_ t||j|_ |S)NzfCannot pass both ignored_modules and ignored_states at the same time. Please just pass ignored_states.TFr) r4list_check_ignored_statesrarYnn Parameter_get_ignored_modules_ignored_modules_get_ignored_params_ignored_params_get_ignored_buffer_names_ignored_buffer_names)r+r~rrignored_parameterspassed_as_ignored_statesignored_states_lists rC_init_ignored_module_statesrs"~'A :  -T9">2148 %4%@D !b%  !# )!,bll ;!4 1O1&/JE/ E #< #E LrErct|dk(ry|r`td|D}td|D}|s9|s6t|Dchc] }t|c}t}t d|yytd|Ds6t|Dchc] }t|c}t}t d|ycc}wcc}w) z Check that the ignored states are uniformly parameters or uniformly modules. We may remove this check in the future if we permit mixing. rNc3PK|]}t|tj ywrX)rYrrr\r+s rCr^z(_check_ignored_states..JsUUE2<<8Ur_c3PK|]}t|tj ywrXrYrModulers rCr^z(_check_ignored_states..KsS5*UBII6Sr_)keyzUignored_states expects all nn.Parameter or all nn.Module list elements but got types c3PK|]}t|tj ywrXrrs rCr^z(_check_ignored_states..TsLE:eRYY/Lr_z>ignored_modules expects nn.Module list elements but got types )rarbsortedrOreprr4)rr all_params all_modulesr+ sorted_typess rCrr?s >aUnUU SNSS +!N"K54;"KQUVL**69 #.zL^LL!N"K54;"KQUVL%( M #L#Ls B; Cignored_params device_idc6d}|1t|tjr|ntj|}|t||D]|}|jjdvr| |j}+|jj|jk7sOt d|jd|jj|xstj j}|jdk(r t dtj||_ |S)a= Determine device handle used for initializing FSDP. If a device is specified by ``device_id``, then returns device handle corresponds to that device type. Otherwise, If the module is already on a non-CPU device, then the device type is that non-CPU device type. If the module is on CPU or meta, then the device type is the current accelerator device. See the :ref:`Accelerators` for details. This method will be called once ignored parameters was determined, as the device handle maybe needed for other initialization. N>cpumetazLFSDP does not support modules with different device types but got params on z and rzOFSDP needs a non-CPU accelerator device, but no accelerator device is detected.) rYtorchdevice_get_orig_paramsrO RuntimeError_C_get_acceleratorr from_devicerL)r+r~rrdetermined_deviceparams rC_init_device_handler\s()U\\2 i(   %fn= E||  O3 ($)LL!<<$$(9(>(>>&-->-C-C,DE%,,J[J[I\^ .L1J1J1L  ! !U *a -889JKE LrEct||_i}|jD]\}}t|}|j||<!||_|SrX)_get_buffer_names _buffer_names named_buffersrdtype_buffer_name_to_orig_dtype)r+r~r buffer_namebuffers rC_init_buffer_staters_ ,F3E :<%335? V' 4 28,,";/?(BE$ LrEmixed_precision cpu_offloadlimit_all_gathersuse_orig_paramsbackward_prefetch_limitforward_prefetch_limitc|jdk(rQ|tjk7r-tjd|xstj ddtj}n/|tjk(rtjdt d|xstj |_|xs t|_ |5tjjdt|jtjj!t"d d k(|_|xs t'|_||_||_t.j0|_d|_t7|_t;j<|_tAjB|j>|||_"d|_#i}||_$d} | |_%g} | |_&|S) NrGz/FSDP is switching to use `NO_SHARD` instead of z since the world size is 1.rV stacklevelzoThe `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.z'torch.distributed.fsdp.mixed_precision.1)'r;r NO_SHARDwarningswarn FULL_SHARD FutureWarningr-rrrr_log_api_usage_oncestrosenvirongetr_use_full_prec_in_evalrrr_use_orig_paramsrIDLEtraining_state_is_rootr_free_event_queuerZget_debug_level _debug_levelexec_order_utils_ExecOrderData_exec_order_data_unshard_event_fully_sharded_module_to_handle_handleparams) r+r-rrrrrrrrrs rC_init_core_staters 1  0 9 9 9 MMA$C(8(C(CDE''   -55 .77 7  <   0N3C3N3NE+?~/?E" $$5c%:O:O6P5Q R  2B73>  $3z|E/E,E(--EEN-/E--/E-<< E  EIK#,KE)*.GEM"$FEL LrEcfg}||_g}||_g}||_d|_d|_d|_|S)NT)_root_pre_forward_handles_pre_forward_handles_post_forward_handles_sync_gradients _comm_hook_comm_hook_state)r+rrrs rC_init_runtime_statersK8:&?E#24!5E35"7E EE!E LrEbackward_prefetchforward_prefetchc"||_||_|SrX)rr)r+rrs rC_init_prefetching_staters 0E-E LrEc||jnd}|r+||jk7rt|j|_|Sd|_|SrX)_get_root_meshr7r$rL_fsdp_extension)r+r/ root_meshs rC_init_extensionrsT 1<0G **,TIyE$6$66 1%2F2F G L!% LrEctj|_t}t |_||_i}||_|SrX)r"FULL_STATE_DICT_state_dict_typerr_optim_state_dict_config_state_dict_config_unshard_params_ctx)r+state_dict_configunshard_params_ctxs rC_init_state_dict_staters?*::E)<)>%=%?E"0E57 2E LrErc|D]X}t|jdk(sd}|jD] \}}||us |}n|s tdt d|dy)z Verify if the parameters are accepted by FSDP. The only restriction now is that the parameter cannot be a scalar tensor (param.shape == []). rrzExpected param_name to be setz/FSDP doesn't support scalar parameters. Change z& to a 1D tensor with numel equal to 1.N)rashapenamed_parametersrsr4)r~rr param_namenameparam_s rC_verify_managed_paramsrs   u{{ q J & 7 7 9  fF?!%J $%DEE$%KM  rEfully_sharded_module param_init_fnsync_module_statescrt|j|t|jj}t |jj \}}|s|r|t||j nA|r#t||j jn|rtj|fdj D chc]}|jD]} | } }} t|j| |t|j|jj_tt!|j} t#|| |r@t%|| j&j(t*vrt%|| j,t/| |Scc} }w)zHInitialize a ``FlatParamHandle`` from a module ``fully_sharded_module``.c>t|duxr|jvSrX)rr) submoduler+s rCz0_init_param_handle_from_module..Ps('=i'HD'P(8!7!77rE)check_fn)_check_single_device_moduler_get_device_from_device_idr9rL_need_to_materialize_moduler_materialize_with_param_init_fn_materialize_meta_moduler(materialize_modulebuffers_move_module_to_device_get_compute_devicecompute_devicerrr_sync_module_params_and_buffersr,r-r5r<_init_param_handle_from_params) r+rrrrdevice_from_device_idis_meta_moduleis_torchdistX_deferred_initignored_modulerignored_buffersmanaged_paramss ` rC_init_param_handle_from_moduler1s 4e6K6KYW65::u333Ne33U5K5K3/N/ 5=;T' -1G1G     " "   %(( 8 $44 $,,.  O    /    E*+?AVAVWXN/@' .%2E2E   " "&@ @ +$ne6J6J #5.:NO L?s F3c t|dk(ryt|||jt|j|j j |jj|jj|jj|j|j|j }|j|jr t!d|j"j%|j&||_||j(|j*<t-j.d}|j j r,|j&j.|k7r|j1|yyy)Nr)fsdp_extensionz!Expected state._handle to be Noner)rarr SHARDING_STRATEGY_MAPr-roffload_paramsr param_dtype reduce_dtypekeep_low_precision_gradsr,rrshardrrsrappend flat_paramr_fully_sharded_modulerr flat_param_to)r+rrhandle cpu_devices rCrrvs+  6{a  e556 (( )) ** 66  ,, F LLN }}@AA LL))*EMJPE))&*F*FGe$J ''F,=,=,D,D ,RZ(-S'rE root_modulerc 2d} | t|n t}|D]V}t|tj j st|dt|zt|sMtd|jD])}tj|r|j|+|Dchc]3}|jD]}t|tjs| 5}}}||vrtj dd|jD]B}t|} | t#| d s t%d |j'| j(D|S#t$r }t|dt|z|d}~wwxYwcc}}w) ah Check that ``_ignored_modules`` is an iterable of ``nn.Module`` s without any FSDP instances. Return the modules contained in their module subtrees as a :class:`set`. Nested FSDP instances are excluded, but their already-computed ignored modules are included. ``_ignored_modules`` represents the argument passed by the user to FSDP. z>`ignored_modules` should be an iterable of `torch.nn.Module`s Nzbut got zbut got an iterable with z1`ignored_modules` should not include FSDP moduleszTrying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: rVrrz?Expected optional_fsdp_state to have _ignored_modules attribute)set TypeErrorrOrYrrrrr4modulestraversal_utils _composableadd fsdp_fileFullyShardedDataParallelrrhasattrrsupdater) r%r msg_prefixignored_root_moduleser~childrroptional_fsdp_states rCrrsRJQ%5%AC !su  'R&%((//2J+DT&\N)SST T !& )PQ Q R%%'-**62 $ $V ,-+ ^^% %!C!CD  O o%  228 ;  !((*I 4Y?  *.0BC$U  " "#6#G#G HI Q Q x5E0F/G%HHIqPQ$sE'78F' F0F  Frct}|Dchc]%}|jD]}t|r|'}}}|j||,|Dchc]}t|r|}}|j||j D]B}t |} | t | ds td|j| jD|Scc}}wcc}w)z Return the parameters of the modules in ``ignored_modules`` and the parameters in ``ignored_parameters``. :class:`FlatParameter` s are excluded from the result. rz>Expected optional_fsdp_state to have _ignored_params attribute) r' parametersrr0r)rr/rsr) r%rrall_ignored_paramsmpparams_in_ignored_modulesparams_in_ignored_parametersrr5s rCrrs36%#!ALLN!'(BTUVBW! !!78%)( 1CA1FA( $(  !!">?!((*K 4Y?  *.0AB$T  % %&9&I&I JK -!( s#CCC%Cc t}|Dchc]}|jD]}|}}}|j|jDchc]\}}||vr t |c}}|j D]B}t |}|t|ds td|j|jD|Scc}}wcc}}w)z6Return the cleaned buffer FQNs in ``ignored_modules``.rzDExpected optional_fsdp_state to have _ignored_buffer_names attribute) r'r r0rrr)rr/rsr) r%rall_ignored_buffer_namesr9rbuffers_in_ignored_modulesrrr5s rCrrs *-("aiik",2"""##(3'@'@'B # V33 k * !((*W 4Y?  *.0GH$Z % + +,?,U,U VW $#-"  s CC cf|jDchc]\}}t|c}}Scc}}w)zrReturn the fully prefixed names of all buffers in the module hierarchy rooted at ``root_module`` as a class:`set`.)rr)r%rrhs rCrrs5>I=V=V=X +9;+&  s-ct||Dchc]}|j}}t|dk(r%tjd|vr| t dyt|dkDrt d|ycc}w)z Raise an error if ``module`` has original parameters on multiple devices, ignoring the parameters in ``ignored_params``. Thus, after this method, the module must be either fully on the CPU or fully on a non-CPU device. rVrNzTTo support a module with both CPU and GPU params, please pass in device_id argument.rGz;FSDP only supports single device modules but got params on )rrrarr)r~rrrdevicess rCrrs*:&.)QRu||RGR 7|qU\\%0G;  5   W I' S   SsA4r9 device_handlec b|yt|tjr|ntj|}|jdk7rk|j_t j d|d|d|jd|jd d tj|j}|S) z Return a ``torch.device`` for the specified ``device_id``. Processes ``device_id`` and returns either the corresponding device or ``None`` if ``device_id`` is ``None``. Nrz"FSDP got the argument `device_id` z on rank zJ, which does not have an explicit index. FSDP will use the current device z6. If this is incorrect, please explicitly call `torch.zk.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.rVr)rYrrrOindexrrcurrent_device)rr9rCrs rCrr=s 5<<8 ell9>U {{e 4 0 9f00=0L0L0N/OPCCI;;-P1 1  m::<= MrEctt||}td|D}|jD]-}||vr|j dD]}||j z}/| xrt xrtd|D}||fS)z Return if ``module`` has parameters on meta device and if ``module`` is using torchdistX deferred initialization. At most of the returned bools can be ``True``. If either is ``True``, then ``module`` needs to be materialized. c34K|]}|jywrX)is_metar\rs rCr^z._need_to_materialize_module..hsC5CsFrecursec3FK|]}tj|ywrX)r)is_fakerJs rCr^z._need_to_materialize_module..ts@ U#@s!)rranyr)r rI_TORCHDISTX_AVAIL)r~rrrrrbufrs rCrr[s*6>BCNCNCCN^^%*  ' $$U$3 *C ckk )N **  A  A @@ @ 6 66rEct|std|dt|t||}|D] }|| y)Nz Expected z to be callable but got )callabler4rO_get_modules_to_materialize)r%rrmodules_to_materializer~s rCrrysV M " &>tM?R>S T  9oV(frErc >|xs#tj|j}t||}d} tj5|D]u}t j |jd|jd}tt|dkD}|sS|j|d|jw dddy#1swYyxYw#t$r7} tjdt!| dt#|dd | d} ~ wwxYw) NFrKr)rrLzIUnable to call `reset_parameters()` for module on meta device with error z(. Please ensure that your module oftype z* implements a `reset_parameters()` method.rVr)rrrFrTno_grad itertoolschainr7r rarto_emptyreset_parameters BaseExceptionrrrrO) r%rrrCmaterialization_devicerUr~module_state_iterhas_module_statesr3s rCrrs$3ell$$&79oV F]]_ .0 .%.OO%%e%4NN5N1%! %(->(?$@1$D!$OO+A5OQ++- . . . .   !!$Q)L>!K M  s<C AC!%CCCCC D%2DDc"g}tj|g}|h}|rq|j}|j||j D]:}||vst |||vs|j ||j|<|rq|SrX) collectionsdequepopleftrchildrenrr,)r%rrUqueuevisited_modulesr~ child_modules rCrTrTs /1   {m ,E'2mO %%f-"OO- +LO3*<8@ 7##L1 \* +  "!rErc tjd | tj}|j |g}g}|r|j }|j fd|jdD|j fd|jdD|jD].}t|tjr|j |0|r|D cgc] } | |vs|  } } |D cgc] } | |vs|  } } t| | |ytt||d} | | j k(r t!yyycc} wcc} w)a Move ``module`` depending on ``device_from_device_id`` and its current device. This includes moving ignored modules' parameters. - If ``device_from_device_id`` is not ``None``, then this moves ``module`` to the device. - If ``device_from_device_id`` is ``None``, then this does not move ``module`` but warns the user if it is on CPU. Precondition: ``_check_single_device_module()``. rNc3@K|]}|jk(r|ywrXr)r\rr$s rCr^z)_move_module_to_device..s%<<:-FrKc3@K|]}|jk(r|ywrXrj)r\rr$s rCr^z)_move_module_to_device..s%==J.rk)rrrarbrrcextendr7r rdrYr-r._move_states_to_devicenextr_warn_cpu_init)r~rrrrerr  curr_modulerr:params_to_move bufs_to_moverr$s @rCr r s\$e$J(/:.?.?.A V%'&(--/K MM(33E3B  NN)11%1@  )113 , !)Y-O-OPLL+ ,!&&,Gq/F!GG#*Gaa.FG G~|=RS !&.94 @E U\\Z78 HGs1 E;E EEr c6t|dk(rt|dk(ryt|dkDr|dj}nt|dkDr|dj}tjd}||D]k}tj5|j ||_|j *|j j ||j _dddm|D]}|j ||_y|k(r tyy#1swYxYw)z Move states to the specified device. Precondition: ``_check_single_device_module()`` and module's parameters and buffers have been materialized if needed. rNr)rarrrWtodatagradrp)rr rrFr$rrs rCrnrns  6{aCLA- 6{Q)) W  **e$J( KE K"XX&;< ::)&+jjmm4I&JEJJO K K K  ;F ))$9:FK ; : % &  K Ks A DD c2tjddy)NamThe passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.rVr)rrrErCrprps MM 1 rEctt||d}|&|jjdk7r |j}n#t j|j }|||k7rt d|d|d||S)a) Determine and return this FSDP instance's compute device. If the module is already on a non-CPU device, then the compute device is that non-CPU device. If the module is on CPU, then the compute device is the current device. Since this method should be called after materializing the module, any non-CPU device should not be meta device. For now, the compute device is always a CUDA or CUDA-like device with its explicit index. Precondition: ``_check_single_device_module()`` and ``_move_module_to_device()``. Nrz4Inconsistent compute device and `device_id` on rank z: z vs )rorrrOrrFr4)r~rrr9rCrr s rCr r %s* !&.94 @E U\\..%7m&B&B&DE(^?T-TB4&d#8"9 ;   rEc~g}|jD]}t|tdrt|td|j }t |r>|j \}}|Dcgc]}t||} }|j| |j||D]l} | j } t | r>| j \}}|Dcgc]}t| |} }|j| \|j| nt|t||tdycc}wcc}w)z Synchronize module states (i.e. parameters ``params`` and all not-yet-synced buffers) by broadcasting from rank 0 to all ranks. Precondition: ``sync_module_states == True`` and ``self.process_group`` has been set. FTr)srcN) r getattr FSDP_SYNCEDsetattrdetachr&__tensor_flatten__rmr+_check_module_states_for_sync_module_statesr%PARAM_BROADCAST_BUCKET_SIZE) r~rr, module_statesrdetached_bufferattrsrhattr inner_buffersrdetached_param inner_paramss rCrrHs*)+M.." 6v{E2 FK .$mmoO,_=+==?qLQ RD$!? R R$$]3$$_5 61 ( 8%88:HE1FKLdGND9LLL   .   010 ># !SMs +D5D:rcD|rtd|Dr tdyy)Nc3`K|]&}|jtjdk((yw)rN)rr)r\tensors rCr^z>_check_module_states_for_sync_module_states..ws'17 e,,s,.zThe module has CPU parameters or buffers when `sync_module_states=True`, which requires them to be on GPU. Please specify the `device_id` argument or move the module to GPU before passing it to FSDP.)rOr4)rs rCrrts7;H C  }rEc#K|j} t|}||vrt|s|#t$rYywxYww)aD Return an iterator over the original parameters in ``module``. The iterator does not return the parameters in ``ignored_params``, any ``FlatParameter`` s (which may be present due to nested FSDP wrapping), or any original parameters already flattened (only relevant when ``use_orig_params=True``). N)r7ror StopIteration)r~r param_genrs rCrrsT!!#I OEN*3Ee3L     sA 4 AAAAc t|D]A\}}||vs t|rtd|d|jd|jy)a5 Check that original parameters in ``fsdp_module`` have been flattened. The flattened parameters are made invisible to ``named_parameters()`` for the module hierarchy rooted at ``fsdp_module``. This should be called as a sanity check after flattening the wrapped module's parameters. z Found an unflattened parameter: z;  N)rrrr: __class__) fsdp_modulerrrs rC_check_orig_params_flattenedrs^?{K E  &/A%/H2:,b::<.%//!24 rEch|tjk(rtjStjSrX)r rr allreduce_hookreduce_scatter_hook)r-s rC_get_default_comm_hookrs3  0 9 9 9 $$ . .rEc.tj|S)NrH)r r=rHs rCrPrPs  % %M BBrErX)rarXrrcollections.abcrrrrtypingrrr r r rtorch.distributed distributedrZ(torch.distributed.fsdp._exec_order_utilsfsdp_exec_order_utilsr'torch.distributed.fsdp._traversal_utils_traversal_utilsr*2torch.distributed.fsdp.fully_sharded_data_parallelfully_sharded_data_parallelr-torch.nnr(torch.distributed.algorithms._comm_hooksr torch.distributed.device_meshr "torch.distributed.distributed_c10dr$torch.distributed.fsdp._common_utilsrrrrrrr"torch.distributed.fsdp._flat_paramrrrr%torch.distributed.fsdp._limiter_utilsrtorch.distributed.fsdp.apirrrrrr r!r"torch.distributed.fsdp.wrapr#&torch.distributed.tensor.parallel.fsdpr$torch.distributed.utilsr%torch.utils._python_dispatchr&torch.utils.hooksr'rP torchdistxr(r) ImportErrorrr~r`r[HybridShardProcessGroupTypeProcessGroupTyperr SHARD_GRAD_OP HYBRID_SHARD_HYBRID_SHARD_ZERO2rr5#NO_RESHARD_AFTER_FORWARD_STRATEGIESrDr6boolrNrIintrir|rKrrrrrr'rrrrrrrrrrrrrrrrrrrrrrTTensorr rnrpr rrrrrr=rPryrErCrs  CCEE CCAAFFB4A B   0D<F1.0 #D$5$5t7H7H$HIE$"3"35P"PQR 5>>!7!B!B""$:$H$H!!#9#F#F((*@*T*T !!(( ""(('# )- 0 0#0(0 W  0 *% 0  00f( (#(( ((V#$IJI4II  ARAR  "++"" ""J++ 4  d// /0& + + II+huxx78+%((,,-.%((//9R0SS +++\I9= :- - II- %-c5<</01 -  --`  II"? ? 01?n-?*% ?  ?  ?!? ???D         '     : J *  *299d2<<6HT(A A))Ac5<</01AHbii[$%678 A  A  AAH) )  )))))>::x89: ^:@BF"")"!%((*<*"    "J$$)$ X$@299S II  % c5<</01   <c5<</01 %ell <7 II7 %7^7 4: 7<  RYYK-. ^  ###ELL1#^#% #L""-0^" "))_",3 II3 %3&3$ELL1 3  3l   %,, $ELL1 @  II %$ELL1   %   \\ F) II)  )$$) )X   %      II  % bll , % (.>C$$CCw#s ^^^