?U6
P U d Z ddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddl Z ddl Z ddl
Z
ddlZddlZddlZddlZddlZddlmZmZ ddlmZmZmZ ddlm Z m Z ddl m Z ddl m!Z! ddl"m#Z# dd l$m%Z% dd
l&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/ dd l0m1Z1 dd l2m3Z3 dd
l4m5Z5 ddl6m7Z7 ddl8m9Z9m:Z:m;Z; ddlZ> ddl?m@Z@ ddlAmBZBmCZC ddlDmEZF ddlGmHZH e#eI ZJ e jK d ZLdZMdZNdeOfdZPdZQd ZRd ZSd ZTd ZU e jK d ZVd!ZWd"ZXd#ZYd$ZZ e j[ Z\e/e]e^e_eOde)e]e'f e+e' f Z`daae,eC ebd%< dace,eB ebd&< deOfd'ZddYd(ZedeCfd)ZfdeBfd*Zg G d+ d,eh Zi G d- d.eh Zjd/ Zkd0e
jl d1e^de(fd2Zmde`fd3Znd4e]d5eoddfd6Zp e:ejekeY7 d4e]de`fd8 Zqi d9d4e]d:e_fd;Zr e:ejekeY7 d4e]dd?d4e]d:e_fd@ ZtdAe(fdBZud5eodCe^deOfdDZv e:ejekeYevE d>ddFd4e]dGe
jl dHe^de]fdI Zw edJd4dKg ZxdLe'de-ex fdMZydN ZzeVfdOe jK dPe jK ddfdQZ{ G dR dS Z|dYdTZ}e|j~ Ze|j Z dZdVe,e] ddfdWZdX ZdS )[aP Utilities for managing local file storage synchronised with a remote
server.
Files are divided into types: signatures, modsecurity bundles, ip white
lists, etc. Each type is represented by an Index instance.
Index has a local subdirectory and a description that contains its
files' metadata used to decide if the update is necessary.
N) defaultdict
namedtuple) ExitStacksuppresscontextmanager)
formatdateparsedate_to_datetime)GzipFile)chain) getLogger)Version) AnyBinaryIODictIterableListOptionalSetTupleUnion)urlparse)config)
LicenseCLN)PanelException) file_hashretry_onrun_with_umask)
rate_limitHOUR) to_thread) UrlTransport RandomIpChooserWithIPv6Toggle)safe_extractall ) default_hookz /var/imunify360/.ipv6_disabledz(/proc/sys/net/ipv6/conf/all/disable_ipv6z(/sys/module/{mod}/parameters/{parameter}returnc t dd } t | 5 }| dk r ddd dS ddd n # 1 swxY w Y n# t
$ r Y dS w xY w t t 5 }| dk r ddd dS ddd n # 1 swxY w Y n# t
$ r Y nw xY wdS ) zCheck whether IPv6 is disabled at the kernel level.
Reads the module parameter and the runtime sysctl without
depending on the im360 package.
ipv6disable)mod parameter0NT1F)
_MOD_PAR_PATHformatopenreadstripOSError_SYSCTL_DISABLE_IPV6)
param_filefs S/opt/imunify360/venv/lib/python3.11/site-packages/defence360agent/files/__init__.py_is_kernel_ipv6_disabledr8 D s %%&I%FFJ
*
vvxx~~ 3&& & tt
&
'
' 1vvxx~~ 3&& &
5sv A? ,A3 A? ' A? 3A77A? :A7;A? ?
B
B
C7 %,C+ C7 C7 +C//C7 2C/3C7 7
DDeulasigszrealtime-av-confzwp-rulesgeoz/var/imunify360/filesz$https://files.imunify360.com/static/i
g? _IP_CHOOSER
_TRANSPORTc P t pt S )zCheck if IPv6 should be disabled at startup.
True when either the kernel has disabled IPv6 or a previous agent
run persisted the disabled state after a runtime network failure.
)r8 _IPV6_DISABLED_STATEexists r7 _should_disable_ipv6rD ~ s" $ % % F)=)D)D)F)FFrC c t dS # t $ r t dd Y dS w xY w)z:Persist IPv6 disabled state so it survives agent restarts.z%Could not persist IPv6 disabled stateT)exc_infoN)r@ touchr3 loggerdebugrB rC r7 _persist_ipv6_disabledrJ s_ M ""$$$$$ M M M r! rO rB rC r7 _get_transportrR s$ !_->->???
rC c e Zd ZdZdS )IntegrityErrorzERaised when on disk content does not match hashes in description.jsonN__name__
__module__ __qualname____doc__rB rC r7 rT rT s OOOOrC rT c e Zd ZdZdS ) UpdateErrora
Raised on other errors during files update.
Possible reasons are:
* server returns non 200 status;
* hash mismatched between downloaded content and description.json;
* urllib errors;
* JSON decoding errors;
* errors while writing to disk.
NrU rB rC r7 r[ r[ s rC r[ c K t d | | t j t j d|z t z d {V d S )Nz2Files update failed with error: {err}, try: {try_})errtry_r$ )rH warningr/ asynciosleeprandom randrange_TIMEOUT_MULTIPLICATORexcis r7 _log_failed_updaterh sy
NN<CC! D
-(a003II
J
JJJJJJJJJJrC pathmodec t d 5 t j | t j t j z t j z | }ddd n # 1 swxY w Y t j |d S )zbOpen file at `path` using permission `mode` for writing in binary mode
and return file object.r Nwb)r osr0 O_WRONLYO_CREATO_TRUNCfdopen)ri rj fds r7 _open_with_moders s
H H
WT2;3bj@$
G
GH H H H H H H H H H H H H H H
9R s ;AA Ac t | | 5 }t j t j |d |d d cd d d S # 1 swxY w Y d S )Ntimeoutfileheaderszutf-8)encoding)
_fetch_urljsonloadio
TextIOWrapperget_content_charset)urlrv responses r7 _fetch_json_syncr s C ) ) )
Xy !),@@II
s AA&&A*-A*r rf c t }| rg| rUt d| | | | t d S d S d S )Nz>Network error for %s via IPv6 IP %r, disabling IPv6. Error: %r)rO is_ipv6_enabledlast_ip_was_ipv6rH r_ last_ip disable_ipv6rJ )r rf choosers r7 _disable_ipv6_on_network_errorr s G !W%=%=%?%? ! L OO
! ! ! !rC )on_error max_triesc Z K t j } | dt | | d{V S # t t
j f$ r(}t d || d}~wt j
$ r7}t | | t d | d}~wt $ r7}t | | t d | d}~wt $ r}t d| d| d}~wt j j t" j j f$ r8}t | | t d | | d}~wt( $ r*}t | | t d| d | d}~ww xY w)
zDownload and decode JSON from *url*.
Return decoded JSON. Raise UpdateError:
* HTTP response status code is not 200;
* Unicode or JSON decoding fails;
* on time outs during HTTP request;
* on other HTTP errors.
Nz!json decode error [{}] for url {}request to {} timed outrequest to {} reset%eof error while updating files, url: , err: 8urllib/http error while updating files, url: {}, err: {} Can't fetch
, reason: )r` get_event_looprun_in_executorr UnicodeDecodeErrorr{ JSONDecodeErrorr[ r/ socketrv r ConnectionResetErrorEOFErrorhttpclient
HTTPExceptionurlliberrorURLErrorr3 )r rv loopes r7 _fetch_jsonr s ! # #D=))$0@#wOOOOOOOOO 4 5 N N N=DDQLLMMM > A A A&sA...3::3??@@@ = = =&sA.../66s;;<<<
CC C C C C
K %v|'< =
&sA... F M MQ
= = =&sA...;;;;;<<<=sQ ": F*#A33F*2B77
F*2C66
F*D(F* 3E33
F* %F%%F*rx rv c
t j | dt j pdi|d }t || 5 }|j |j fcddd S # 1 swxY w Y dS )z>Perform HEAD http request to *url* with *timeout* & *headers*.Imunify-Server-Id HEAD)rx methodru N) r requestRequestr
get_server_idrR r0 coderx )r rv rx reqrs r7 _perform_http_head_syncr s .
!9!;!;!Ar
! C
sG 4 4 !vqy ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !s A88A<?A<
current_mtimec K |t u rdS t |d } t t | |d|i d{V \ }}|dk rt d|d| t t 5 t |d }||k r$t
d
||d || ddd n # 1 swxY w Y dS # t j $ r7}t | | t d | d}~wt $ r7}t | | t d | d}~wt j j t& j j f$ rY}t- |d
r|j dk rY d}~dS t | | t d | | d}~ww xY w)zCheck if we need to download description.json file:
- perform HEAD request if local file exists and older return True
otherwise return False
T)usegmtzIf-Modified-Sincer N zUnexpected http code z for z
Last-Modifieda Got code %r, but last modification date %s is earlier than or equal to the date provided in the If-Modified-Since header, the origin server SHOULD generate a 304 (Not Modified) response [rfc7232]. Here's curl cmd:
curl -s -I -w '%%{http_code}' -H 'If-Modified-Since: %s' '%s'r r r i0 Fr )_NEVERr r r r[ r Exceptionr timestamprH r_ r rv r r/ r r r r r r r hasattrr )r r rv formatted_mtimer rx
last_mtimer s r7 _need_to_downloadr s t t<< A A A&sA...3::3??@@@ = = =&sA.../66s;;<<< K %v|'< =
1f !&C--55555&sA... F M MQ
sH #C 2ACCC G ,2D
G +2E (G G&3GG T)compressc # K i }|||d< dt j pdi}|r| ddi t j | | } t
j |fi |5 }t 5 }|j
d dk }|rl|sj|j
d d
k rLt d |j
d |j
| |r#| t | n||j d
V ddd n # 1 swxY w Y ddd dS # 1 swxY w Y dS )zs
Fetch *url* as binary file.
If *compress* is true, ungzipping is done automatically
if necessary.
Nrv r r zAccept-Encodinggzipr Content-Encodingz Content-Typezapplication/zipzRequested gzip but got Content-Encoding=%r. Read response as is [identity]. Headers: %s, as curl cmd:
curl -Is -H 'Accept-Encoding: gzip' '%s')fileobj)rw rx )r r updater r r rR r0 r rx getrH rN items
enter_contextr
) r rv r
parameters req_headersr r stackgzippeds r7 rz rz Q s7 J '
9 &
(@(B(B(HbIK 8 -v6777
.
k
:
:C
9;;
"'"&&'9::fD $$^448III KKJ $$%788 &&((
##HX$>$>$>??? '
%
s7 7E(B>E E(E E(E E((E,/E, dest_filec 0 t j }| }t | || 5 }|d t
x}rL| | | | |d t
x}Lddd n # 1 swxY w Y |d d dk s| |z
} |d dd }
|
Nt |
} | | k r9t j d
| | | z
d
| } | | |k rt d | d | d
| | S )z
Fetch *url* to *dest_file* and return its md5sum.
Raise *urllib.error.ContentTooShortError* if the downloaded file
has unexpected length.
)rv r rw Nrx r r zContent-Lengthz&{got} bytes read, {diff} more expected)gotdiff)messagecontentzcontent fetched from z does not match hash: expected=z, got=)hashlibmd5tellrz r1 _BUFSIZEr writer intr r ContentTooShortErrorr/ hexdigestr[ )
r r rv r md5sumr initial_file_offsetr chunk file_lengthcontent_length_headerexpected_file_length
got_md5sums
r7 _fetch_n_md5sum_urlr ~ s +--C#..** C8 < < < # ',,X666e # JJu OOE " " " ',,X666e ## # # # # # # # # # # # # # # I " "#5 6 6& @ @ nn&&)<< !) 3 7 78H$ O O ,#&'<#=#= #{22 l77 D K K ' 1K ? L ! 8 J
jF 2 2
4C
4
4
4
4'1
4
4
s A/B55B9<B9rg c ( K dt | vS )NzHTTP Error 404)strre s r7 $_fetch_and_save_should_retry_handlerr s 3s88 ++rC )r r should_retryr r dest_path dest_modec 0 K t || 5 }t t | |||| d{V cddd S # 1 swxY w Y dS # t j $ r7}t | | t
d | d}~wt $ r7}t | | t
d | d}~wt $ r*}t | | t
d| d| d}~wt j j t j j f$ r8}t | | t
d | | d}~wt $ r-}t | | t
d| d | d
| d}~ww xY w) zFetch bytes from `url`, save them to `dest_path`,
and return md5 checksum of downloaded content.
Raise UpdateError:
* HTTP response status code is not 200;
* on time outs during HTTP request;
* on other HTTP errors.
r Nr r r r r r z to r )rs r r r rv r r[ r/ r r r r r r r r r3 )r r rv r r r r r s r7 _fetch_and_saver sT 0 L
Y
2
2 i"#!
> A A A&sA...3::3??@@@ = = =&sA.../66s;;<<<
&sA... CC C C C C
K %v|'< =
&sA... F M MQ
L L L&sA...JJJ)JJqJJKKKLsi A A A AA A A F 2B
F 2C
F%D (F(3E
F((FF_Itemr datac & d | d D S )z,Return a set of _Item for easy manipulation.c F h | ] }t |d |d S )r r )r ).0items r7 z_items.. s* I I I4E$u+tH~ . . I I IrC r rB )r s r7 _itemsr s I I4= I I IIrC c d } || | t j | D ]d\ }}}|D ],} |t j || | -|D ],} |t j || | -edS )zCheck and change file/dir modes recursively.
Starting at dirname, change all inner directory permissions to dir_perm,
file permissions to file_perm
c t j | j dz }||k rmt j | sPt
d| t | t | t j | | d S d S d S # t $ r t
d| Y d S w xY w)Ni zGFixing wrong permission to file/dir %s [%s] expected [%s] (not symlink)z&Failed to change permission to file %s) rm lstatst_moderi islinkrH r_ octchmodPermissionErrorr )
file_dir_path
permission current_modes r7 _os_chmodz"check_mode_dirs.._os_chmod s 8M22:UBLz))"'.. 3 3) ;! %%
OO
33333 *))) LL8-
s B
B %B>=B>N)rm walkri join) dirnamedir_perm file_permr ri dirsfiles directorynames r7 check_mode_dirsr s & Igx WW - - ; ;dE ? ?I Ibgll433X > > > > ; ;D Ibgll4.. : : : : ;; ;rC description_path
files_pathc || j v sJ | j }|}||k rR| r/| d | dd dS |j }||k PdS dS )aP
Try to fix the structure of /var/imunify360/files/ when
NotADirectoryError happens.
It indicates that some part in the path is a file:
/var/imunify360/files/sigs <- is a file
=> open("/var/imunify360/files/sigs/v1/description.json") will fail.
We try to rectify it by deleting the file but up to FILES_DIR.
T)
missing_ok)parentsexist_okN)r
parentis_fileunlinkmkdir)r r _dir topmost_dirs r7 _fix_directory_structurer s )1 1 1 1 1 "DK
*
<<>> KK4K ( ( ( dT : : : E{ *
rC c e Zd Z eej Z ee Zi Z i Z
e Z e Z i Z
dZ edez ej ZdDdZd Zd Zdej d d
fd Zdej fd Zd
Ze dddde de de de de de d d
fd Z e d e fd Z!e d e"e fd Z#e de d e fd Z$dEd e fdZ%d e fdZ&d e"e fd Z'e d Z(d e)e fd Z*d Z+e,fd e-fd Z.d e fd!Z/d"e-d e fd#Z0dEd$Z1de2j3 d%e"e4 d d
fd&Z5d'e"e4 d e6e"e4 e"e f fd(Z7e de d e fd) Z8e de d e fd* Z9e:dEd+ Z;e:d,e2j3 d e2j3 fd- Zd e?e@e?e e e z f f fd2ZAd eBe fd3ZCdFd4ZDd5e2j3 d,e2j3 d eEe2j3 fd6ZFd e fd7ZGe:d8ej d9ej d:e"e d d
fd; ZHdZJdFd?ZKdEdFd@ZLe dGdAeEe d d
fdB ZMe de d d
fdC ZNd
S )HIndexz/static )periodTc X || j vrt d| d| j || _ d| _ dg i| _ | } t
| 5 }t j | | _ ddd n # 1 swxY w Y n# t $ rE t d| t t j | t Y nTt t" t j f$ r6}|r#t' d | |d| _ Y d}~nd}~ww xY w|rX| }t- | r5t' d d
| | j s| dS dS ) z
:param bool integrity_check: check if last update
did not break anything (by interrupting it in the middle or
another programmatic error)
:raise IntegrityError:
z*Trying to initiate unregistered file type z. Allowed types Fr Nz Path %s has a file in parentsz cannot read description file {}Tz'some files are missing or corrupted: {}z, )_TYPES
ValueErrortype _is_blank_json_descriptionfile_pathr0 r{ r| NotADirectoryErrorr _throttled_log_errorr pathlibPath FILES_DIRFileNotFoundErrorr r rT r/ _corrupted_fileslenr r )selftype_integrity_checkri r6 r bad_filess r7 __init__zIndex.__init__1 s) # # (U ( ( +( (
r]
))++ "d
*q !Yq\\
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*! D D D & &'F M M M $W\$%7%7 C C C C C
" " "
$5<}>)r- rV r r r% r r& s r7 __repr__zIndex.__repr__d s\
'
4::<<((
rC r r& Nc t d| j | | | } || j d dS )zjWhether *files_path* dir may be used for this type's file group.
:raises: IntegrityError
zValidating [%s]: %sTr( N)rH rN r _make_file_groupr& r FileGroups r7 validatezIndex.validatel sT
)49jAAA))
$)T222222rC c 6 G fdd j }|S )zV
Return FileGroup class: Index class with local path == *files_path*.
c 6 e Zd Zededef fd ZdS ))Index._make_file_group..FileGroupr' r& c F |j k sJ t j S z+Return local base path for given file type.)r rm fspath)clsr' r r& s r7 r z4Index._make_file_group..FileGroup.files_path~ s( ))))y,,,rC N)rV rW rX classmethodr r )r r& s r7 r8 r< } sP
-s
-s
-
-
-
-
-
- [
-
-
-rC r8 )r- r7 s `` r7 r6 zIndex._make_file_groupx sG
- - - - - - - - - - - rC c . t j | j }t t j t j t t j | j t j
|d |d d S )Ndirrw ) r _PERMSr r rm ri normpathr r" _PATHSpardir)r& permss r7 r zIndex.check_mode_dirs sq TY' G Y TY(?KK
%L &M
rC Fall_zip essentialr'
relative_pathr r rJ rK c | j | |r| j | || j |< ||d| j |< || j |< dS )a Add a type to known file types.
* relative_path is a relative path to all files for that type.
* dir_perm is permission mask used to create directories.
* file_perm is permission mask used to create files.
* all_zip is a flag which shows whether that type of files can
be downloaded in all.zip archive. all.zip is expected to be on
the server.
* essential is whether the agent can start if there are errors
updating that type.
)rC rw N)r add_ESSENTIAL_TYPESrF rD _ALL_ZIP_SUPPORT)r@ r' rL r r rJ rK s r7 add_typezIndex.add_type sj ,
u , $ $U + + + )
5$,i @ @
5&- U###rC c B K t d | j D S )zuWhether essential files exist.
Note: the files may be corrupted (integrity check is not performed).
c 3 D K | ]}t |d j V dS )Fr5 N)r r )r r' s r7 z.Index.essential_files_exist.. sI
eU333= =
rC )allrO r@ s r7 essential_files_existzIndex.essential_files_exist s9
-
rC c 4 | j S )z&Return a set of all known files types.)r copyrV s r7 typesz Index.types s z rC c b t j t | j | S r> )rm ri r r" rF r@ r' s r7 r zIndex.files_path s! w||Isz%'8999rC c
|rJ| j t j j v r2t j | d S t j | | j d S )z9Return local path for description.json for current index.description.json) r r FilesUpdateDISABLEDrm ri r _descriptionfile_path_latestr )r& latests r7 r zIndex._descriptionfile_path sm di6#5#>>>7<<11335G
w||DOODI 6 68JKKKrC c r d | D d }|S )Nc . g | ]}|d
|d S )rb rC rB )r xs r7
z6Index._descriptionfile_path_latest.. s% JJJAakJqxJJJrC r ) _get_listvalues)r& ltss r7 ra z"Index._descriptionfile_path_latest s5 JJ !1!1!8!8!:!:JJJ1M
rC c R t }t | j D ]}| |j } t |t j t }n%# t $ r |
| Y _w xY w||j k r|
| |S )z9Return a set of file paths that are missing or corrupted.) setr r
localfilepathr r r r r r# rN r )r& r) r ri actuals r7 r$ zIndex._corrupted_files s EE 4:&& $ $D%%dh//D
"4h??$
d###
$$
d###s A!! BBc | j | S )z`
usage example:
>> async with Index.locked(WHITELISTS):
...
)_lockr\ s r7 lockedz Index.locked s y rC c D fdt j D S )z(Return iterable over all files in index.c 3 L K | ] } |j V d S r, rl r r r r& s r7 rT z Index.files.. s3 LL""48,,LLLLLLrC )r r r2 s `r7 r z Index.files s' LLLL
9K9KLLLLrC c | j d S )z+Return 'items' field from JSON description.r )r r2 s r7 r z Index.items s z'""rC c t j | d j S # t $ r |cY S w xY w)zBReturn mtime of description file if it exists, otherwise -math.infTrb )rm statr st_mtimer3 )r& defaults r7 _descriptionfile_mtimez Index._descriptionfile_mtime sO 7455T5BBCCL L NNN s ,/ >>c | }|sdS |t j j z t j k S )z4Return True if last update was too late in the past.T)r{ r r_ PERIODtime)r& _desc_mtimes r7 _is_outdatedzIndex._is_outdated s< 1133 4V / 66DDrC rv c K | j pyt | dk pT| o@t | | j | | d{V S )z>Return True if update from server is needed for current index.r N)r r% r$ r r _descriptionfile_urlr r{ )r& rv s r7 is_update_neededzIndex.is_update_needed s
N
4((**++a/
!!## +--di88//11
rC c t d 5 t j ||| ddd dS # 1 swxY w Y dS # t $ r"}t t | |d}~ww xY w)z)Create local directory for current index.r )rj r N)r rm makedirsr3 r[ r )r& r dir_moder r s r7 _makedirszIndex._makedirs s - ""
G
G G(XFFFF
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G - - - c!ff%%1 , -s2 A 6 A :A :A
A/
A**A/ to_updatec K | | } || j d }|j |j d }|j |j d }|D ]}| |j } t
j | }
t
j |
s| |
|d t |j | |||j d{V dS )zX
Fetch files from *to_update* set, verify hashes, save to *files_path*.
Fr5 rC rw r )r r N) r6 r rD rl r rm ri r isdirr r r ) r& r r rv r8 fgr file_moder filenamer s r7
_update_fileszIndex._update_files s ))
Yty%
8
8
89RW%e,Ibg&v. D''11Hgooh//G7==))
Bw5AAA!#{
rC remote_itemsc t j } fd|D } fd|D }||z
} fd|D }||z
}||fS )zFigure out what should be updated based on current items,
file system state and remote items.
Return tuple of files to fetch and files to delete.
Files to fetch is a set of _Item.
Files to delete is a set of file paths.c D h | ] } |j S rB rs rt s r7 r z+Index._calculate_changes..7 s) LLLt))$(33LLLrC c D h | ] } |j S rB rs rt s r7 r z+Index._calculate_changes..8 s) NNN**4844NNNrC c L h | ] } |j v |!S rB rs )r r r) r& s r7 r z+Index._calculate_changes..< s>
!!$(++9<<
<<J#>???rC c K t j | | j }t | }| |j dz }| j | j d }| j | j d }| | j }t 5 }| ||d | t j
||d t ||||d d {V } t j |d
5 }
t# |
| d d d n # 1 swxY w Y t% j | D ]v\ } } }
| D ]5}t% j t$ j | | | 6|
D ]5}t% j t$ j | | | 6w| | | || }nX# t2 t4 t6 t8 t j t j f$ r"}t? tA | |d }~ww xY w|! d d d n # 1 swxY w Y t
||tE | | j tF j$ j% vS ) a
Update current type of files using all.zip archive. Directory with
current type of files will be cleared and replaced with all.zip
contents. all.zip is expected to be on the server
Return whether updated.
:param timeout:
:raise UpdateError: if OSError or http error or
integrity check error (got wrong data from
the server)
all.ziprw rC Fr T)r )r r Nr )&r r! r r r r r r rD r r r callbackr r zipfileZipFile_safe_extractallrm r r ri r r9 _replace_live_with_new_dirr rT r3 r
BadZipfile LargeZipFiler[ r pop_allboolr r_ r` )r& rv r new_path archive_pathr r all_zip_urlrollback_stack_archiveroot directories filenamesr r old_pathr s r7 _run_update_all_zipzIndex._run_update_all_zipe sT L !;!;<< ++I66 ))(-)*CDD K *62 ;ty)%0'' 22
[[ 1 %N NN8XN > > > # #& !
$
& # A
1 _\377 87$Wh7778 8 8 8 8 8 8 8 8 8 8 8 8 8 8
57GH4E4E J J0D+y%0 J J dI!>!>IIII$- J J dH!=!=yIIIIJ
h''' ::8YOO "$
1
1
1 "#a&&))q0
1
" " $ $ $c1 % 1 % 1 % 1 % 1 % 1 % 1 % 1 % 1 % 1 % 1 % 1 % 1 % 1 % 1 %h lh
y 2 ;;;s\ >AJH *E; H E H E B9H J 3I < II JJJversionforcec K t j | | j }| st d| j d| |dk rY| }d | D d }t d| j | |
d } t |d z
t | k r |st d
| d | j n(# t $ r t d| j d | w xY w|j D ]}| s| rt |d z
t | k rh| |j d
z }| |d | | | d d{V dS t d
|d| j )a Update to the version specified in *version*.
:param version: version to update to
:raise UpdateError: if OSError or http error or
integrity check error (got wrong data from
the server)
zCannot update z , because it is not a symlink: rb c ( g | ]\ }}|d
|S rw rB )r verprops r7 rf z#Index.update_to.. s5 !Th rC r z%Try to update to latest version %s %sFstrictVERSIONzVersion z is already set for z5, because current version doesn't have VERSION file: liveTtarget_is_directory
is_updatedNz not found in )r r! r r
is_symlinkr[ rg r rH rN resolver
read_textr2 r# r iterdiris_dirr r
symlink_torename
_run_hooks)r& r r r versions current_pathri
new_live_paths r7 update_tozIndex.update_to s L !;!;<< ##%% +999ii)
h ~~''H %-^^%5%5 G
KK7G
!(((66
1<<>>DDFFGG7##$ $ $ "GwGGDIGG ! +'+yyy,,@
$,,.. D??$$
D9 ,7799??AABBgG G %)NN49v3E$F$FM!,,Tt,LLL!((333 //T/:::::::::FFk'''499MNNNs
A#D7 7%E c t j | | j }| si S | d }i }t
d }|j D ]}| r| |k rd}nd}|dz x} rqd } |
}t
| } |dt | d|| < | |k r| }# t $ r t d|| Y w xY w|t
d k r d|| d< |S ) NFr r, Tr )currentrb rC z Version file %s is not valid: %srb )r r! r r r r r
r r rA r r r rH r )
r& r r result max_versionri r version_filer _vers
r7 rg zIndex._get_list s L !;!;<< ##%% I (((66 cll $,,.. D
||~~ -- $y 00 88::
* 4 4 6 6G"7++D#*"' "4yy$ $F4L
k))&* ! LL:$
H
& % %,0F; )
s AD&D>=D>c g }t | d D ]4\ }}|d rdn |d rdnd}| | | 5|S )z/Return list of versions available in the index.c | d S )Nr rB )re s r7 z Index.get_list.. s
AaD rC )keyr z
(current)rb z (latest)r )sortedrg r append)r& r r r markers r7 get_listzIndex.get_list s # NN " " $ $..
0
0MGT
? >[[
MMW.f.. / / / /
rC c t j | | j }| sdS | d }|j D ]}t j
t j j t j
| j t j j z
j }| rf| sR||k rL|t$ j j k r7t* d| j | t/ j |d dS )z~Remove old versions of files.
This is done by removing old directories in the path, that older than 30 days.
NFr z Removing old version of %s: %sTr )r r! r r r r r r r r nowtimezoneutc
fromtimestamprx ry daysr r r_ DAYS_TO_KEEPrH rN r r )r& r r ri days_olds r7 _clean_old_versionszIndex._clean_old_versions s6 L !;!;<< ##%% F (((66 $,,.. 8 8D 00+++IIKK("+/
8 ))
8 L(( 2 ??? & &
9 % & ) "))*555 '//
0BINNN-O0 %
j====
" " $ $ $[- % - % - % - % - % - % - % - % - % - % - % - % - % - % - %^ s8 =A/G
-DG
B FG
F/G
GGc K j }t || d{V } t | \ }}|p|}|s6t
d j dS t j
j }| r| d nd}t | } t 5 }
| j j d d |
t& j | d |r| r|n|} | rAt | | | fd
|D d{V | || d{V t3 | d z j j d 5 } | t7 j | ddd n # 1 swxY w Y | | | }n6# t@ tB f$ r"}
tE tG |
|
d}
~
ww xY w|
$ ddd n # 1 swxY w Y |rE| r1t
d
| t' j |d j tJ j& j' vS )z
Run update, return whether updated.
:raise UpdateError: if OSError or http error or
integrity check error (got wrong data from
the server)
ru Nz updating %s: nothing to update.Fr rC r Tr c 3 L K | ] } |j V d S r, rs rt s r7 rT z$Index._run_update.. sD $ $9=**4844$ $ $ $ $ $rC r^ rw z,Removing old path on file by file update: %s)(r r r r r rH rN _touchr r! r r r r r r r rD r r r r _copytreeunionr rs r r{ dumpsencoder9 r rT r3 r[ r r r r_ r` )r& rv r as_jsonr r need_updater r r r from_pathrw r s ` r7 _run_updatezIndex._run_update s '' 22#C999999999 # 6 6vg G G 9 ,9 KK949 E E E KKMMM5 L !;!;<< 09/C/C/E/E OI U + + +4 ++I66 [[ * %N NN$+di07%
# #
xt
$
%I ):):I
!!
oo OO $ $ $ $AJ$ $ $ $$Xy'$JJ J J J J J J J
1$ 11K *62 = JJtz' 2 2 9 9 ; ;<<< = = = = = = = = = = = = = = =
h''' ::8YOO"G,
1
1
1!#a&&))q0
1
" " $ $ $U* % * % * % * % * % * % * % * % * % * % * % * % * % * % * %Z 8)) 8 KK>
M($ 7 7 7 7 y 2 ;;;s[ C K15)J :I$ J$I( (J+I( ,.JK1K, K KK11K58K5from_dirto_dir
ignored_pathsc ` K fd}t t j | |d|d d{V dS )z7Copy *from_dir* to *to_dir* except for *ignored_paths*.c t t j t sJ t fd|D S )z(Return names that should not be copied.c 3 ` K | ](}t j | v $|V )d S r, )rm ri r )r r r ri s r7 rT z8Index._copytree..ignore_names.. sJ 7<<d++}<< <<<< rC )
isinstancerm r? r frozenset)ri namesr s ` r7 ignore_namesz%Index._copytree..ignore_names s_ bioos33 3 33 !
rC T)symlinksignore
dirs_exist_okN)r r copytree)r
r r r s ` r7 r zIndex._copytree su O
rC r c t j t | j | j }| j | j }t j | t j | j v s J d
|| t j || }t j | | j | S )z.Return a local file path corresponding to URL.z$url ({}) does not fit file path ({}))
rm ri relpathr _URL_PATH_PREFIXrF r r r! r
r/ r r )r& r url_relpath type_pathrL s r7 rl zIndex.localfilepath s goo SMM 5
K * L # #w|K'@'@'H H H H 1 8 8i H H
I H H Y??
w||DOODI 6 6
FFFrC c | d }t j | rt j | dS dS # t
$ r2}t t | Y d}~dS d}~ww xY w)z5Update mtime of description.json file so it is fresh.Trw N) r rm ri isfileutimer3 rH r_ r )r& ri r s r7 r z Index._touch s #--T-::Dw~~d##
# # # NN3q66 " " " " " " " " " #s A A
B 'BB c K t | j | j t g D ]}} || | d {V # t t
f$ r&}t d|| Y d }~Hd }~wt $ r&}t d|| Y d }~vd }~ww xY wt
d| j d| z d S )Nzhook %s error: %sz%s files update finished%sz (not updated)) r _HOOKSr r% rT r rH r r exceptionrN )r& r hookr s r7 r zIndex._run_hooks s $+di0<.AA ? ?D
?d4,,,,,,,,,,"N3
;
;
; 0$::::::::
?
?
? !4dA>>>>>>>>
? ( I J /
s! ?B$ A11
B$> B B$c $ K t j j }|sy| | d{V s^t d| j t t j j dz | d d{V dS | j
o| j | j }| }|rd}t d| j | t j
| t j j | d{V }|r!t d| j | nG# t j t" f$ r.}t d | j || d
}Y d}~nd}~ww xY w|rd }t d| j | t j
| t j j | d{V }|r!t d| j | nr# t j t" f$ rY}t d | j || | d d{V | j | j v r|Y d}~dS d}~ww xY w| |p| d{V dS ) a Run update for the current `type` of files.
Normally update is performed when either is true:
* index is never been fetched (description.json missing or broken);
* last update was performed longer than configured period of time ago;
* some local files are missing or have wrong content (md5 hash differs
from description.json).
If force is True then update is performed unconditionally.
Raises asyncio.TimeoutError, UpdateError.
Nz(%s was updated less than %s minutes ago.< Fr r zUpdating %s files via %szUpdated %s using %sz%s update error via %s: %sTzfile by file download)r r_ TIMEOUTr rH rN r r r} r r rP r` wait_forr SOCKET_TIMEOUT TimeoutErrorr[ r r rO )r& r rv rJ file_by_filelog_strupdatedr s r7 r z Index.update s $, 4#8#8#A#A A A A A A A KK: F&-344
//U/33 3 3 3 3 3 3 3 F.ET%:49%E"{ $ G KK2DIw G G G
$ ' 0,, *9 ! ! KKK 5ty'JJJ(+6
$
$
$ 0$)Wa $
$
-G KK2DIw G G G
' 0$$V%7%FGG! ! KKK 5ty'JJJ(+6
0$)Wa ooo7777777779 555GFFFFF
oo)9Eo:::::::::::s3 A D2 2E6$E11E6 A H I/AI**I/ only_typec K |rj | |d }| | 4 d{V | | d{V ddd d{V dS # 1 d{V swxY w Y dS |rt d | j D ]i} | |d }| | 4 d{V | | d{V ddd d{V n# 1 d{V swxY w Y jdS t d | j D ]i} | |d }| | 4 d{V | | d{V ddd d{V n# 1 d{V swxY w Y jdS )zkRun update for all registered `types` of files.
Raises asyncio.TimeoutError, UpdateError.
Fr5 NzUpdating essential fileszUpdating all files)rp r rH rN rO r )r@ r- r only_essentialindexr' s r7
update_allzIndex.update_allO ss .C 5999Ezz),,
*
*
*
*
*
*
*
*ll5)))))))))
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
. KK2 3 3 3 -
.
.E5999 ::e,, . . . . . . . . ,,u---------. . . . . . . . . . . . . . . . . . . . . . . . . . .
.
.
KK, - - -
.
.E5999 ::e,, . . . . . . . . ,,u---------. . . . . . . . . . . . . . . . . . . . . . . . . . .
.
.s5 A
A&)A&< C**
C4 7C4 E77
F F c F | j | | dS )z:Add a hook for type_ to be called after successful update.N)r! rN )r@ r' r# s r7 add_hookzIndex.add_hookh s%
5 d#####rC )T)Fr& N)NFF)OrV rW rX r r` Lockro rk r! rF rD r rO rP r r r rH r r r* r/ r3 rm PathLiker9 r6 r rA r r r rQ rW r rZ r r ra r$ rp r r r r floatr{ r r r r r! r r r r r r staticmethodr r r r r r
rg r r r r r r r rl r r r r1 r3 rB rC r7 r r $ s
K % %E
[
F
F
F
SUUFsuu 6::QX666v|DD)# )# )# )#V
32;
34
3
3
3
3 2;
. . .. . .
. . .
. . . [.8
D
[
!c#h ! ! ! [! :s :s : : : [:L LS L L L Lc #c( [ Mx} M M M M# # # .4 Ed E E E E
e
- - - -!,36u:
2$ J$ s5z3s8# $$ $ $ $. K K K K K [K B B B B B [B : : : \ : @gl @w| @ @ @ \@O >? % % % %N$s) 8 8 8 84C C18 C ', C C C CJN r" r r _MAX_TRIES_FOR_DOWNLOADrd infr r r r7 JSONTyper= __annotations__r> rD rJ rO rR RuntimeErrorrT r[ rh r6 rs r r r r r r rz r r r r r r r r r? r1 r rW rA rC rB rC r7 r^ s
/ / / / / / / / : : : : : : : : : : 9 9 9 9 9 9 9 9 % % % % % %
" ! ! ! ! ! - , , , , , 8 8 8 8 8 8 = = = = = = E E E E E E E E E E 9 9 9 9 9 9 9 9 3 3 3 3 3 3 N M M M M M 8 #w|$DEE A :
$ 4
% GL0 1 1 1 ( c5$d38nd3iG H 8< X3
4 ; ; ;%)
H\ " ) ) )Gd G G G GM M M M 6 P P P P P\ P P P
,
K K K "+ S X
h
!
!)
!
!
!
!
!
,8O "=3 "=H "= "= "= "=L *,
!
!
!
!
!
!
!
!
,8O 7 7 "7-27 7 7 7 7t 59 )
)
)
C )
U )
)
)
)
X& & & & &R,I ,# ,$ , , , ,
%5 0L 0L 0L 0L{0L
0L 0L 0L 0L
0Lf
7UH -..J JU J J J J
; ; ;F @I l07 ,G
$ G
$ G
$ G
$ G
$ G
$ G
$ G
$TO O O O,
3 ,1
}
rC