Files
box/watchers/__pycache__/box-stability-watcher.cpython-314.pyc
T

180 lines
38 KiB
Plaintext
Raw Normal View History

+
,³Æj´cãóÜ€Rt^RIt^RIt^RIt^RIt^RIt^RIt^RIt^RIHtH t ^RI
H t ^RI H
t
HtHtHtHt^RIt] !]4P+4P,t]P,t]R, t]R, R, t]R, R , t]R, R
, t^<tR R ltR
RltR5RRllt R6RRllt!R5RRllt"R5RRllt#R5RRllt$R5RRllt%R7RRllt&RRlt'R8RR llt(R!R"lt)R#R$lt*R5R%R&llt+R'R(lt,R9R)R*llt-R9R+R,llt.R5R-R.llt/R7R/R0llt0R1R2lt1R3t2]3R48Xd]Ph!]2!44R#R# ]dRtEL*i;i):u§box-stability-watcher.py — Proactive host and fleet stability guardrail for NetVM & Box.
Features:
1. System Load & Memory Monitoring (1m/5m/15m load, RAM%, Swap%)
2. Multi-tier Mitigation:
- GREEN: Normal.
- YELLOW: Renice runaway CPU hogs to +15.
- ORANGE: Pause runaway workers via SIGSTOP, record in .state/stability-paused.json,
post alert to sidechat (646 tasks), and give 60s grace period before SIGTERM.
- RED: Emergency shedder for processes >1500MB.
3. Tmux Socket Origin & Isolation:
- Allows user-launched agents on interactive desktop socket (/tmp/tmux-1000/default).
- Detects and logs automated workloads launched through Box on the desktop socket,
recommending migration to /tmp/tmux-muse.sock.
4. Systemd Loop Detection: Identifies crashing services trapped in tight restart loops.
5. State Management: Supports explicit "freeze" and "resume" commands.
N)ÚdatetimeÚtimezone)ÚPath)ÚAnyÚDictÚListÚOptionalÚTuplezbox-stability.jsonÚlogszbox-stability.jsonlz.statezstability-paused.jsonzbox-launched-sessions.jsoncó$€V^8„dQhR\/#©éÚreturn)Ústr)Úformats"Ú</home/super/Projects/NetVM/watchers/box-stability-watcher.pyÚ __annotate__r-s€÷EñE”ñEócóh€\P!\P4P R4#)z%Y-%m-%dT%H:%M:%SZ)rÚnowrÚutcÚstrftime©rrÚnow_isor-s!€Ü �<Š<œŸ ™ Ó %× .Ñ .Ð/CÓ DÐDrcó$€V^8„dQhR\/#r )Úfloat)rs"rrr1s€÷ñ”5ñrcó,€\P!4#©N)ÚtimerrrÚ now_epochr1s€Ü �9Š9‹;Ðrcóh€V^8„dQhR\\,R\\\3,/#)r
Ú config_pathr©rrrrr)rs"rrr5s&€÷'ñ'œX¤d�^ð'´t¼CĸHµ~ñ'rcój€T;'g\pVP4'd9\VRRR7;_uu_4p\P!V4uuRRR4#RRRRRR R
R R R
RRRRRRRRR/ R.R'OR.R(ORRRRRRRRR/RR R!R"R#/R$^
R%R&/# +'giLB;i \
dLRi;i))Úrúutf-8©ÚencodingNÚ
thresholdsÚ load_warningç4@Ú
load_criticalç€A@Úload_emergencyçN@Úram_warning_pctçT@Úram_critical_pctç€V@Úswap_warning_pctçÀR@Úswap_critical_pctç@U@Úprocess_rss_warning_mbéÐÚprocess_rss_critical_mbé¸ Úprotected_commandsÚmonitored_targetsÚactionsÚ
enable_reniceTÚenable_cull_runawayÚenable_service_freezeÚnotify_sidechatÚ notificationÚagentÚ646Útargetú 646 tasksÚ interval_secÚlog_filezlogs/box-stability.jsonl) ÚsshdÚ
tailscaledÚ tailscaleÚsystemdz dbus-brokerÚpipewireÚ wireplumberÚtmuxÚbashÚzshÚghosttyÚ alacritty)úmuse-binÚchromiumÚpython3Úparec)ÚDEFAULT_CONFIGÚexistsÚopenÚjsonÚloadÚ Exception)r!ÚpathÚfs& rÚ load_configr`5sñ€Ø × (Ð (œ.€DØ ‡{�{‡}‚}ð Ü�d˜C¨'×2Õ2°aÜ—y’y “|÷3Ò2ð
Ø ˜DØ ˜TØ ˜dØ ˜tØ  Ø  Ø  Ø $ dØ % tð

ð ò
ð ò
ð Ø ˜TØ ! 4Ø # TØ ˜tð 
ð Ø �UØ �kð
ð ˜ØÐ.ð= ð÷ 3×2ûäô Ù ð ús.¨B$¾BÁ
B$Â B! ÂB$Â!B$Â$ B2Â1B2có0€V^8„dQhR\R\/#)r
Úlog_pathÚ max_bytes)rÚint)rs"rrr_s€÷
ñ
¤4ð
´Cñ
rcó€VP4'dXVP4PV8”d7VPVP R24p\
P !W4R#R#R# \dR#i;i)z.1N)rYÚstatÚst_sizeÚ with_nameÚnameÚosÚreplacer])rbrcÚrotateds&& rÚrotate_log_if_neededrm_sk€ð
Ø �?‰?× Ò  §¡£×!8Ñ!8¸9Ô!DØ×(Ñ(¨H¯M©M¨?¸"Ð)=Ó>ˆGÜ �JŠJ�xÖ )ñ"EÑ øô ô
Ú ð
ús‚A1˜AA1Á1 BÁ?Bcóh€V^8„dQhR\\\3,R\\,/#)r
Úeventrb©rrrrr)rs"rrrhs&€÷
ñ
¤$¤s¬C x¥.ð
¼HÄT½Nñ
rcóZ€T;'g\pVPPRRR7\V4\ VRRR7;_uu_4pVP \ P!V4R,4RRR4R# +'giR#;i \dR#i;i)T©ÚparentsÚexist_okÚar%r&Ú
N) Ú DEFAULT_LOGÚparentÚmkdirrmrZÚwriter[Údumpsr])rorbrEr_s&& rÚappend_stability_eventr|hs~€Ø
×
$Ð
$œ€Fð
�
‰
×Ñ D°4ÐÔ8ܘVÔ$Ü
�&˜#¨×
0Õ
0°AØ
�G‰G”D—J’J˜uÓ%¨Õ,Ô -÷1×
0×
0Ò
0ûä ô
Ú ð
ús/’>BÁ-BÁ=BÂ B ÂBÂBÂ B*Â)B*cóh€V^8„dQhR\\,R\\\3,/#)r
Ú
state_filerr")rs"rrrss&€÷ñ¤(¬4¥.ð¼DÄÄcÀ½Nñrcó€T;'g\pVP4'd9\VRRR7;_uu_4p\P!V4uuRRR4#/# +'gi/#;i \
d/#i;i©r$r%r&N)ÚPAUSED_STATE_FILErYrZr[r\r])r~rEr_s& rÚread_paused_stater‚sss€Ø
×
,Ð
,Ô,€FØ
‡}�}‡‚ð Ü�f˜c¨G×4Õ4¸Ü—y’y “|÷5Ò4ð €I÷ 5Ö4ð €Iûôô Ø Ø
€Ið úó.¨A5¾A!Á
A5Á! A2 Á,A5Á2A5Á5 BÂBcóh€V^8„dQhR\\\3,R\\,/#)r
Ústater~rp)rs"rrr~s&€÷ 
ñ 
œd¤3¬ 8�nð 
¼(Ä4½.ñ 
rcóh€T;'g\pVPPRRR7VPR4p\ VRRR7;_uu_4p\
P !W^R7RRR4\P!W24R# +'giL(;i \dR#i;i)Trrz.tmpÚwr%r&©ÚindentN)
r�rxryÚ with_suffixrZr[Údumprjrkr])r…r~rEÚtmpr_s&& rÚwrite_paused_stater�~sˆ€Ø
×
,Ð
,Ô,€Fð
�
‰
×Ñ D°4ÐÔ8Ø× Ñ  Ó(ˆÜ
�#�s W×
-Õ
-°Ü �IŠI�e qÕ )÷.ä
�
Š
�3Ö÷.×
-ûô ô
Ú ð
ús*’AB"ÁBÁ/B"Â B ÂB"Â" B1Â0B1cóh€V^8„dQhR\\,R\\\3,/#)r
Ú
sessions_filerr")rs"rrrŠs'€÷ñ¬X´d­^ðÄtÌCÔQTÈHÅ~ñrcó€T;'g\pVP4'd9\VRRR7;_uu_4p\P!V4uuRRR4#/# +'gi/#;i \
d/#i;ir€)ÚBOX_LAUNCHED_SESSIONS_FILErYrZr[r\r])r�rEr_s& rÚget_box_launched_sessionsr’Šss€Ø
×
8Ð
8Ô8€FØ
‡}�}‡‚ð Ü�f˜c¨G×4Õ4¸Ü—y’y “|÷5Ò4ð €I÷ 5Ö4ð €Iûôô Ø Ø
€Ið úrƒcó^€V^8„dQhR\R\\\3,R\/#)r
ÚmessageÚconfigÚdry_run)rrrÚbool)rs"rrr•s)€÷ñ¤ð¬d´3¼°8­nðÄtñrc óÔ€V'g)VPR/4PRR4'gR#VPR/4pVPRR4pVPRR4p\R , R
, pVP4'dD\P\ V4R R R
VRVRVRV 2. p\ P!VR^
R7R#R# \dR#i;i)r=rATNrBrCrDrErFÚbinz super-cli.pyÚdmÚsendz--agentz--toz--targetz[STABILITY ALERT] )Úcapture_outputÚtimeout) ÚgetÚ REPO_ROOTrYÚsysÚ
executablerÚ
subprocessÚrunr])r”r•r–ÚnotifrCrEÚbox_cliÚcmds&&& rÚsend_sidechat_alertr§•sÑ€ß�f—j‘j ¨BÓ/×3Ñ3Ð4EÀt×LÒLÙØ �J‰J�~ rÓ *€EØ �I‰I�g˜uÓ %€EØ
�Y‰Y�x Ó
-€Fܘ%Õ .Õ0€G؇~�~×Òä �N‰NœC ›L¨$°Ø �uØ �EØ ˜Ø    Ð *ð 
ˆð Ü �NŠN˜3¨t¸R× @ñøôô Ú ð úsÂ;Cà C'Ã&C'cóF€V^8„dQhR\\\3,/#r ©rrr)rs"rrrªs€÷9ñ9œD¤¤c �Nñ9rcó´€\P!4wrp\P!4;'g^p^p^pRp^p^pRp \'dÉ\P!4p
\ V
P R, 4p\ V
PR, 4p\V
P^4p\P!4p \ V P R, 4p\ V PR, 4p\V P^4p EMi/p \RR4;_uu_4p
V
FwpVPR4p\V4^8XgK&V^,P4pV^,P4P4^,p\ V4V V&Ky RRR4V P!R^4pV P!RV P!R ^44pVR,pVR,p\R
VV, ,
^d,^4pV P!R ^4pV P!R ^4pV^8”d;VR,pVV,
R,p\VV,
V, ^d,^4p R
\V^4R\V^4R\V^4RVRVRVRVRVRVRV /
# +'giEL;i \"dLUi;i)éçiz
/proc/meminfor$Ú:NÚMemTotalÚ MemAvailableÚMemFreegð?Ú SwapTotalÚSwapFreeÚload_1mÚload_5mÚload_15mÚ cpu_countÚ ram_total_mbÚ ram_avail_mbÚ ram_used_pctÚ
swap_total_mbÚ swap_used_mbÚ
swap_used_pcté)rjÚ
getloadavgr¶ÚpsutilÚvirtual_memoryrdÚtotalÚ availableÚroundÚpercentÚ swap_memoryÚusedrZÚsplitÚlenÚstripržr])Úload1Úload5Úload15r¶r·r¸r¹rºr»r¼ÚvmÚsmÚmeminfor_ÚlineÚpartsÚkeyÚvalÚtotal_kbÚavail_kbÚ swap_tot_kbÚ swap_free_kbs rÚget_system_metricsrتst€ÜŸ=š=›?Ñ€E�&Ü— ’ “×#Ð# !€Ià€LØ€LØ€LØ€MØ€LØ€Mç
ƒvÜ
×
"Ò
"Ó
$ˆÜ˜2Ÿ8™8 {Õ3Ó4ˆ ܘ2Ÿ<™<¨;Õ7Ó8ˆ ܘRŸZ™Z¨Ó+ˆ ä
×
Ò
Ó
!ˆÜ˜BŸH™H¨ Õ4Ó5ˆ
ܘ2Ÿ7™7 kÕ2Ó3ˆ ܘbŸj™j¨!Ó,Š
ð ØˆGÜ�o s×+Ô+¨qÛ�DØ ŸJ™J s›O�Eܘ5“z Q–Ø# A�hŸn™nÓ.˜Ø# A�hŸn™nÓ.×4Ñ4Ó6°qÕ9˜Ü'*¨3£x˜ › ñ ÷,ð—{‘{ :¨qÓ1ˆHØ—{‘{ >°7·;±;¸yÈ!Ó3LÓMˆHØ# tÕ+ˆLØ# tÕ+ˆLÜ  #¨°HÕ)<Õ"=ÀÕ!DÀaÓHˆLà!Ÿ+™+ k°1Ó5ˆKØ"Ÿ;™; z°1Ó5ˆLؘQŒØ +¨tÕ 3�
Ø +¨lÕ :¸tÕC� Ü %¨ °lÕ(BÀkÕ'QÐUXÕ&XÐZ[Ó \�
ð
”5˜ “?Ø”5˜ “?Ø”E˜& !Ó$Ø�Yؘ ؘ ؘ ؘؘ ؘð ð ÷-,×+Ð+ûô&ô Ù ð ús2ÄK Ä/$J5ÅAJ5Æ-CK Ê5 K Ë K Ë KËKc ó‚€V^8„dQhR\R\R\R\\\3,R\R\/#)r
ÚpidriÚcmdliner•Úrss_mbr)rdrrrr—)rs"rrræs@€÷ñ”cð¤ð¬sð¼DÄÄcÀ½NðÔTWðÔ`dñrcó„€V\P!4\P!4^39dR#VP4pVP4pVR9d
VR8¼dR#VP R.4F0pVP4pW…8XgW†P 49gK/R# RV9gVR8Xd
RV9dR#R#) r«TiFr;ztmux new-sessionrOÚmain)rPrQÚsh)rjÚgetpidÚgetppidÚlowerržrÇ) rÚrirÛr•rÜÚlow_nameÚlow_cmdÚprotÚp_lows &&&&& rÚ is_protectedrçæs�€Ø
Œr�yŠy‹{œBŸJšJ›L¨!Ð,Ô,ÙØ�z‰z‹|€HØ�m‰m‹o€GðÐ(Ô(¨V°t¬^ÙØ—
‘
Ð/°Ö4ˆØ—
‘
“ ˆØ Ô  ¯©«Ö 8Úñ5ð˜WÔ$¨°VÔ);ÀÈ'Ô@QÙÙ rcó0€V^8„dQhR\R\/#)r
rÚr)rdr)rs"rrrøs€÷ ñ ¤ð ¬ñ rcóº€\RV R2R4;_uu_4pVP4PR4pVFmpVPR4'gKVP RRR7pVPR4pV'gKJV^,P R R
4uuuR R R 4# R R R 4R
# +'giR
#;i \ dR
#i;i) z/proc/z/environÚrbósTMUX=r%Úignore)ÚerrorsÚ,zTMUX=ÚN)rZÚreadrÇÚ
startswithÚdecoderkr])rÚr_Úenv_dataÚitemrÓrÑs& rÚget_tmux_socket_for_pidrõøs»€ð

Ü
�F˜3˜%˜xÐ(¨$×
/Ô
/°1Ø—v‘v“x—~‘~ eÓ,ˆHÛ �Ø—?‘? 8×,Ô,ØŸ+™+ g°h˜+Ó?�CØŸI™I c›N�Eß‘uØ$ Q�x×/Ñ/°¸Ó<Ñ<÷0Ò
/á ÷0ñ ÷0Ö
/ñ ûô ô
Ø Ù
ð
úsF‚C š9B7Á)B7ÂB7Â
C Â+B7Â-C Â7 C ÃC ÃC Ã CÃCcóŠ€V^8„dQhR\\\3,R\\\\3,,/#)r
r•r)rrrr)rs"rrrs/€÷%ñ%œd¤3¬ 8�nð%´´d¼3ĸ8µnÕ1Eñ%rcó¾aa€.p\'gV#VPR.4Uu.uFq"P4NK pp\P!.RO4EFMpVPpVR,pVR,;'gRpVR,;'g.pRP V4p VR,p
V
'd\
V
PR, 4M^p VR,;'gR p VR,;'g^p
V P4oVP4o\;QJd VV3R
lV4F 'gK R M R M!VV3R