qi6ddlZddlZddlmZddlmZddlZddlmZddlm Z ddl m Z ddl m Z mZmZmZmZd gZej&hZgd ZGd d ej,Zy) N) defaultdict)Any)nn) parametrize)type_before_parametrizations) FakeSparsityget_arg_info_from_tensor_fqnmodule_contains_param module_to_fqn swap_moduleBaseSparsifier)module module_fqn tensor_namec *eZdZdZd!deeefdzffd ZdeeeffdZdeeeeeffddfdZ d Z deeeffd Z d"d eeefd e fd Z efdej deeej&ddfdZdZdZ d#deedfdzdeeeedffdzfdZddefdej deeej eej fdzde deej fdZd"de ddfdZej:dej defd ZxZS)$ra'Base class for all sparsifiers. Abstract methods that need to be implemented: - update_mask: Function to compute a new mask for all keys in the `groups`. Args: - model [nn.Module]: model to configure. The model itself is not saved but used for the state_dict saving / loading. - config [list]: configuration elements should be a dict map that includes `tensor_fqn` of tensors to sparsify - defaults [dict]: default configurations will be attached to the configuration. Only the keys that don't exist in the `config` will be updated. Example:: >>> # xdoctest: +SKIP("Can't instantiate abstract class BaseSparsifier with abstract method update_mask") >>> config = [{'tensor_fqn': 'layer1.weight', 'tensor_fqn': 'linear2.weight2', 'sparsity_level': 0.5}] >>> defaults = {'sparsity_level': 0.7} >>> # model.layer1.weight will have `sparsity_level` = 0.7 (getting default) >>> sparsifier = BaseSparsifier(config, defaults) Ndefaultsc|t||xsi|_tt|_g|_d|_y)NT)super__init__rrdictstategroupsenable_mask_update)selfr __class__s q/home/ubuntu/crypto_trading_bot/.venv/lib/python3.12/site-packages/torch/ao/pruning/sparsifier/base_sparsifier.pyrzBaseSparsifier.__init__7s4 (0B &1$&7 ,. "&returncJ|j|j|jdS)Nrrrr!)rs r __getstate__zBaseSparsifier.__getstate__?s! ZZkk  rrc:|jj|yN)__dict__update)rrs r __setstate__zBaseSparsifier.__setstate__Fs U#rc|jjdz}t|jD]T\}}|d}|dz }|d|dz }|d|dz }t |j D]}|dk(r |d|d||dz }V|dz }|S) Nz (r z Group z module: z z: ))r__name__ enumeratersortedkeys)r format_stringi sparse_argsrkeys r__repr__zBaseSparsifier.__repr__Is//$6 ' 4 FNA{ *F T !M xs"- -M ~fXR8 8Mk..01 F(?6#bS1A0B"!EE  F  F  rc |jDcgc]&}ttd|j(}}|j|dScc}w)aReturns the state of the optimizer as a :class:`dict`. It contains: * state - current state of the sparsification. * groups - a list containing all sparsity configuration groups with the key 'tensor_fqn' specifying the path to the sparsified tensor within a model TODO: Need a clean way of loading the state of the "prepared" module c|dtvS)Nr)KEYS_NOT_IN_STATE_DICT) key_values rz+BaseSparsifier.state_dict..esil:P&Prrr)rrfilteritemsr)rmgrs r state_dictzBaseSparsifier.state_dictWs[$kk(  PHHJ ( ( ZZ  ( s+A r=strictc |tj|d}|d}|jD]\}}t|j|}|d}|d} |r|t d|dd} |j | D]} t| tsd} n| sIttjt|| j} tj|| | |jd d|j!d } |  _|D]} | d |k(s | j%||j'||d y) NrrrrzError loading z into the modelFTmask tensor_fqnr9)copydeepcopyr;r model RuntimeErrorparametrizations isinstancer torchonesgetattrshaperregister_parametrizationgetpopr@r&r')rr=r>rstatesrAsarg_inforrfoundpr@r<s rload_state_dictzBaseSparsifier.load_state_dictqsAz(34G$#\\^ (MJ3DJJ KHh'F"=1K&."^J<#OPPE,,[9 a. E  GFK,H,N,N!OP44V[!LuuVT".uuV} (l#z1IIh' (' (, Ff=>rrDSUPPORTED_MODULESc@g|_|g}|r|j}|jD]k\}}t||vrHt ||}t |t s td|jjd|dzi[|j|m|ryy)Nzmodule_fqn must be a stringrAz.weight) configrNnamed_childrentyper rGstrAssertionErrorappend)rrDrUstackr_namechildrs rmake_config_from_modelz%BaseSparsifier.make_config_from_models  YY[F & 5 5 7 ( u;"33!.ue!??t}}5J   m ,# d;J!$M $@z#R , *$,S1Z_D</ #&:3&? ?:c? R-*3%/no    2 3 KK  z *M& +N rc (|jD]}|d}|d}|jdt}|jdtjt ||}||j |dd<tj||||y)z-Adds mask parametrization to the layer weightrrparametrizationr@rAN) rrMr rH ones_likerJrrrL)rargskwargsrWrrrir@s rrczBaseSparsifier._prepareskk FH%F /K$jj):LIO::feoogfk6R&STD7;DJJvl+ ,V 4  0 0 _T%:   rparams_to_keep.params_to_keep_per_layerc\|jD]}|d}|d}tj||di}|$|D cic]} | ||  } } |j| |;|j |dd} | $| D cic]} | ||  } } |j| |s||_ycc} wcc} w)a3 Squashes the sparse masks into the appropriate tensors. If either the `params_to_keep` or `params_to_keep_per_layer` is set, the module will have a `sparse_params` dict attached to it. Args: params_to_keep: List of keys to save in the module or a dict representing the modules and keys that will have sparsity parameters saved params_to_keep_per_layer: Dict to specify the params that should be saved for specific layers. The keys in the dict should be the module fqn, while the values should be a list of strings with the names of the variables to save in the `sparse_params` Examples: >>> # xdoctest: +SKIP("locals are undefined") >>> # Don't save any sparse params >>> sparsifier.squash_mask() >>> hasattr(model.submodule1, "sparse_params") False >>> # Keep sparse params per layer >>> sparsifier.squash_mask( ... params_to_keep_per_layer={ ... "submodule1.linear1": ("foo", "bar"), ... "submodule2.linear42": ("baz",), ... } ... ) >>> print(model.submodule1.linear1.sparse_params) {'foo': 42, 'bar': 24} >>> print(model.submodule2.linear42.sparse_params) {'baz': 0.1} >>> # Keep sparse params for all layers >>> sparsifier.squash_mask(params_to_keep=("foo", "bar")) >>> print(model.submodule1.linear1.sparse_params) {'foo': 42, 'bar': 24} >>> print(model.submodule2.linear42.sparse_params) {'foo': 42, 'bar': 24} >>> # Keep some sparse params for all layers, and specific ones for >>> # some other layers >>> sparsifier.squash_mask( ... params_to_keep=("foo", "bar"), ... params_to_keep_per_layer={"submodule2.linear42": ("baz",)}, ... ) >>> print(model.submodule1.linear1.sparse_params) {'foo': 42, 'bar': 24} >>> print(model.submodule2.linear42.sparse_params) {'foo': 42, 'bar': 24, 'baz': 0.1} rrT)leave_parametrizedNr)rrremove_parametrizationsr&rM sparse_params) rrmrnrkrlrWrrrrk global_paramsparamsper_layer_paramss r squash_maskzBaseSparsifier.squash_masksvkk 5FH%F /K  / /  M)7E F!F1I F F$$]3'3155f\6JDQ%>D'E6!9 'E$'E!(()9:'4$# 5!G (Fs B$7 B)FrmappinginplaceparameterizationcR| td|stj|}i}|jD]F\}}t ||rt ||vrt ||||</|j||d|||<H|jD]\}} | |j|<|S)aConverts submodules in input module to a different module according to `mapping` by calling `from_dense` method on the target module class Args: module: input module mapping: a dictionary that maps from source module type to target module type, can be overwritten to allow swapping user defined Modules inplace: carry out model transformations in-place, the original module is mutated zNeed to auto generate mapping T)rxryrz) NotImplementedErrorrBrCrXr rr convertr;_modules) rrrxryrzreassignnamemodr2values rr}zBaseSparsifier.convert2s" ?%&FG G]]6*F..0 ID#&c+;<05@!,S'!:"&# %5 "."  #..* )JC#(FOOC  ) ruse_pathc|jsytj5|jD]}|jdi| dddy#1swYyxYw)N)rrHno_gradr update_mask)rrrWs rstepzBaseSparsifier.step^sR&&  ]]_ +++ +   *6* + + + +s $AArc yr$r)rrrrls rrzBaseSparsifier.update_maskes rr$)T)NN) r+ __module__ __qualname____doc__rrZrrr"r'r3r=boolrTrUrModulesetrYLinearr`rgrctuplerwr r}rabcabstractmethodr __classcell__)rs@rrrs2'c3h$!6' d38n $$sDcN':";$$  DcN 4?$sCx.?$?<3D(yy(tBII/(  ($8t 26FJL5c3h$.L5#'sE#s(O';"* * ryy/ *X+T+T+  ")) #  r)rrB collectionsrtypingrrHrtorch.nn.utilsrtorch.nn.utils.parametrizerutilsr r r r r __all__rrUr6ABCrrrrrsU # &C  YYK@J SWWJ r