qiSGXddlZddlZddlmZddlmZmZmZddlZddl m cm Z ddl m cmcmZddlmZddlmZddlmZddlmZddlmZdd lmZmZmZmZm Z ejBe"Z#Gd d Z$ d(d ed edeede%de&e&e'dfe&e'dfff dZ(ede'de'de'de ezde'de%de&e'eejRffdZ*edejRde'fdZ+ d(d edede,e'dzdeede%de&e&e'dfe&e'dfff dZ-ej\j^Z0dejRd edeede&e,e'e,e'ffdZ1d ejdd edeedejdfd!Z3d"ejhjjd#ee6defd$Z7d%ed edeede&e'dffd&Z8dejdfd'Z9y))N)Sequence)AnycastOptional) ShapeType)maybe_run_for_local_tensor) DeviceMesh)redistribute_cost) DTensorSpec) _StridedShardPartial Placement ReplicateShardcneZdZdZej Zd dedefdZe de de de fdZ d Z d Zy ) ExplicitRedistributionContextaT Within this context manager, DTensor will refuse to perform implicit redistribution, instead raising an error. Manual calls to ``redistribute()`` are required wherever a redistribution must occur to avoid erroring. This can be used to ensure that the user is aware of all redistribution. Note: it is easier to use this mode on just the forward pass of a typical DTensor program, as the backwards pass may contain implicit redistribution calls that are not visible to the user and difficult to replace with manual calls. Redistribution during backward can be made explicit by writing `autograd.Function`s that are no-op during forward and perform a manual redistribution during backwards. enable (bool) if False, disables the context manager. Can be used nested inside an enabled region. strict (bool) if True, triggers on any redistribution. If False, only triggers on redistributions that perform communication. mode (str) Determines what happens when ExplicitRedistributionContext triggers: "raise": raises an exceptoin, "warn" issues a warning enablestrictcX||_||_|dvrtd||dk(|_y)N)raisewarnz Invalid mode r)_enable_strict RuntimeError_raise_on_redistribution)selfrrmodes e/home/ubuntu/crypto_trading_bot/.venv/lib/python3.12/site-packages/torch/distributed/tensor/_utils.py__init__z&ExplicitRedistributionContext.__init__0s7  ( (tf56 6(,%src_specdst_specmessagect|jddx}r\d}|jr|jrd}nt ||dk}|s-|j r t |tj|yyy)N_activeTFr) getattr_localrrr rrloggerwarning)clsr!r"r#instancealloweds robserve_redistributionz4ExplicitRedistributionContext.observe_redistribution7svszz9d; ;8 ;G###G/(CqHG44&w//NN7+   Suppose you have a full tensor with size global_shape, and you have sharded it according to placements for mesh_shape. This function returns, for a specific coordinate my_coordinate in the device mesh: - The size of your local shard WITHOUT padding (i.e., if you have an uneven split, your size might be smaller than the other entries in your dim), and - Where the data for your shard begins, in the full tensor. This function is fairly simple if your tensor is evenly sharded; the complication is around uneven splits. There is also some complication for handling StridedShard, which changes the order you should apply sharding. Args: global_shape (ShapeType): The global shape of the tensor. mesh_shape (ShapeType): The shape of the device mesh. my_coordinate (Optional[list[int]]): The coordinate of the current rank in the device mesh. placements (Sequence[Placement]): The placements of the DTensor. skip_offset (bool): If True, skip computing the global offsets and return an empty tuple for global_offset. This can improve performance when only the local shape is needed. Defaults to False. Returns: tuple: A tuple containing: - local_shape (tuple[int, ...]): The shape of the local shard on the current rank. - global_offset (tuple[int, ...]): The offsets for each dimension identifying where this shard begins in the global tensor. If skip_offset is True, this will be an empty tuple. r@)r Sharding dim  greater than tensor ndim r) rY enumeraterTrr dimlengetr`tupleitemsrc)rArdrerCrD empty_offset local_shapeshard_dim_to_global_offsetsmesh_dimrN shard_dimrOr[r]r^ global_offsetglobal_offsetss rrGrGscNLl##|$K#%(4?))e]%;< MM ))43{++ I;&@[AQ@R S +7::9E$?  " x ( #    % ! M", I1>#I.'?([!<//C#l++M%@%F%F%HE! >#4^#D i E  u]3 33r global_tensorc t|j}t|j}t|D]\}}|j|}|j rt t |}|jdkrtd||j} | t|ksJd| dt|d|d|| } | |zdk(s Jd| d|| |z|| <tt|D]"} | | k7s || || |zk\s|| |z|| <$t|ttfrtd t|d ||fS) ak Compute the local size and stride of a DTensor from the given global tensor info. For example, if we have a global tensor with size (4, 8, 4) and stride (32, 1, 8). If the DTensor placements are [Shard(2)] and world_size is 2; then the local size is (4, 8, 2) and stride is (16, 1, 8). Args: tensor (:class:`torch.Tensor`): Global tensor which DTensor will distribute mesh (:class:`DeviceMesh`): Object which describes the mesh topology of devices for the DTensor. placements (Sequence[:class:`Placement`]): The attribute of the DTensor that describes its layout on the mesh topology. Returns: local_shape: A List of int which specifies the size of the local tensor. local_stride: A List of int which specifies the stride of the local tensor. rzOShard placements should have negative dims normalized in the user-facing APIs: rgrhz for placement number .z Global dim z not divisible by mesh size zplacement type z not supported!)rYsizestrideriis_shardrrrjAssertionErrorrkrangerTrr rtype) rvrBrCrp local_strideidxrNrLshard_placementrsglobal_dim_sizeis rcompute_local_tensor_infors4}))+,K ,,./L#J/SY #    "5)4O""Q&$--<,=?(++Is;//  {*DSEUDVW((+uA/ / *)4O"]2a7 o..J=/Z 7&5 %EK "3|,- GN$Q< +B]+RR&21o&FLO  GI 7';<i0AQR R=S@  $$r rHctdk7r tdtjk7r%tdtdjdt dt rSt dt rtfd}|}tjDcgc]#}tj||j%}}tj||tfd }|||}t} || dj <tj"| Std t%dd cc}w) a  Compute the global size of a DTensor from the given local tensor shape, the mesh and placements. Different from `compute_global_tensor_info`, which assumes sharding is even, this util allgathers local shards' shapes from all ranks and thus can support uneven sharding. NOTE: Currently this function only supports 1D mesh. Args: shape (:class:`torch.Size`): Shape of the local tensor mesh (:class:`DeviceMesh`): Object which describes the mesh topology of devices for the DTensor. placements (Sequence[:class:`Placement`]]): The attribute of the DTensor that describes its layout on the mesh topology. Return: tensor_shape: Shape of the global DTensor. rSz>compute_global_tensor_shape only supports 1 placement for now.z/Expected one placement per mesh dim, but found z placements and z mesh dims.rcXtjt|jS)Ndevice)rVrZrY device_type)rHrBs r_create_local_shape_tensorz?compute_global_tensor_shape.._create_local_shape_tensorms<<U D4D4DE Er rcd}dj}tt Dcgc] }||k7s | }}|D]A}tj||||s t d|j }|||z }C|Scc}w)Nrz?Non-sharded dimensions should have identical size across ranks.)rjr}rkrVequalrtolist) rpgathered_shaped_tensorssharded_dim_sumrsd other_dims shape_tensorshape_tensor_listrCrHs r"_validate_and_compute_global_shapezGcompute_global_tensor_shape.._validate_and_compute_global_shapexsO"1 ))I%*3u:%6I!y.!IJI 7 @ {{;z#:L|D]}t|tjtfr|jcSt|t t fsHt|dkDsWt|dtjtfs{|djcStd|d)z Find the device mesh object from args. It returns None if no mesh is found. NOTE: we can optimize this search if needed rz+Cannot find device mesh from args for op : rx) rTdtensorDTensorr device_meshrYrmrk ValueError)rrargs rtry_find_mesh_from_argsrs& cGOO[9 :?? " sT5M *C1 3q6GOO[#ABq6%% %& B7)1M NNr global_stridecpdgtzt|D]q\}}|jstt|j }t tD]*}||kDs|xx|j|zcc<,stfdt tDS)z Compute the stride of a local tensor shard, given the global stride of the DTensor. NOTE: Currently this function is assuming the DTensor is evenly shardable. rSc34K|]}||zywr2r@).0rrstride_divisorss r z'compute_local_stride..s%34 aOA..s) rkrir{rrrjr}ryrm)rrBrCmesh_idxprjrs` @rcompute_local_striderscC ..O ,> ! ::<UA""A3}-. > #mA&66#A&$))H*==& > > 8=c->P8Q r ct|tjr|St|tr|g}n;t |dk(r"t|dt rt |d}n t |}tj|S)z Unify variable types of size argument to torch.Size Acceptable types include: int, Sequence[int], Tuple[int], Tuple[Sequence[int]], or torch.Size rSr)rTrVrrXrkrrY)ry torch_sizes rnormalize_to_torch_sizerse$ # $V TaJtAw9$q'] $Z ::j !!r )F):loggingr;collections.abcrtypingrrrrV)torch.distributed._functional_collectives distributed_functional_collectivesrtorch.distributed.tensor._apirZ_apirtorch._prims_commonrtorch.distributed._local_tensorrtorch.distributed.device_meshr *torch.distributed.tensor._collective_utilsr &torch.distributed.tensor._dtensor_specr (torch.distributed.tensor.placement_typesr r rrr getLoggerr7r(rr=rmrXrJTensorr`rcrYrG_C#_DTensor_compute_global_tensor_infocompute_global_tensor_inforrr_ops OpOverloadobjectrrrr@r rrs$&& :://)F4H>   8 $4B4Bv 22 2#2 2  5c?E#s(O +, 2j333 3}$ 3  33 3& &'33Bu|| S4S4S49t#S4# S4  S4  5c?E#s(O +, S4l#XXII=%<<=% =%#=% 49d3i  =%@H ::H 'H 5=i5HH  ZZH VO ZZ " "O*26*:OO*$.