5 Commits

Author SHA1 Message Date
operator-646 97f0e4e50e Verify SSH dial-in perms for container 646
- chmod 600 ~/.ssh/authorized_keys (already 600, verified)
- sshd listening on :22, reverse tunnel VM 127.0.0.1:2226 -> container:22 up
- authorized key present (super@bl); key-auth step belongs to key holder

Fixes #213
2026-10-09 22:49:00 +00:00
super 78d22bd950 Merge pull request #212 from dev/opm/211-container-ssh-recovery
feat(ssh): add node SSH dial-in verification script

Fixes #211
2026-10-09 22:43:09 +00:00
opm c0113c1ebf feat(ssh): add node SSH dial-in verification script
Adds bin/verify-node-ssh.sh: checks reverse-tunnel listeners and SSH
auth for each fleet node port from the VM. Distinguishes dark nodes
(no listener) from auth failures (authorized_keys perms/keys).

Verification 2026-10-09:
- muse/2225, 646/2226, pip/2227, muse-main/2224, opm/2228: LISTEN
- def/2229, dev/2230: DARK (no reverse tunnel)
- All listening nodes reject VM super key (expected: nodes authorize
  per-operator/id_frontdoor keys, not the VM super key)

Fixes #211
2026-10-09 21:56:08 +00:00
operator 69555f809c fix(platform): verify Gitea integration and automated task dispatch Fixes #208 Fixes #209 2026-10-09 21:38:50 +00:00
operator 198603060d test(webhook): verify automated issue closure and loop terminus Fixes #210 2026-10-09 21:38:33 +00:00
2 changed files with 98 additions and 0 deletions
+60
View File
@@ -0,0 +1,60 @@
#!/bin/bash
# verify-node-ssh.sh — verify container SSH dial-in readiness across fleet nodes.
# Checks from the VM: reverse-tunnel listeners + SSH auth for each node port.
#
# Port map (docs/OPERATOR-DRIVE-RUNBOOK.md):
# muse-main 2224 | muse 2225 | 646 2226 | pip 2227 | opm 2228 | def 2229 | dev 2230
#
# What it checks per node:
# 1. Reverse-tunnel listener on 127.0.0.1:<port> (dark node = no listener)
# 2. SSH dial-in with BatchMode (auth failure = authorized_keys perms/key issue)
#
# Common root causes (see #211):
# - sshd requires non-group-writable authorized_keys (must be 600)
# - stale /run/nologin blocks logins
# - missing id_frontdoor keys on dark nodes
#
# Usage: run on the VM (super@34.139.37.135), or via:
# ssh-vm.sh "bash -s" < verify-node-ssh.sh
set -u
# node:port pairs to check
NODES="muse:2225 646:2226 pip:2227 def:2229 dev:2230 muse-main:2224 opm:2228"
fail=0
for pair in $NODES; do
node="${pair%%:*}"
port="${pair##*:}"
# 1. listener check
if ss -tln 2>/dev/null | grep -q "127.0.0.1:${port} "; then
listener="LISTEN"
else
listener="DARK (no listener)"
fi
# 2. auth check (only if listening)
if [ "$listener" = "LISTEN" ]; then
out=$(timeout 15 ssh -o StrictHostKeyChecking=no -o BatchMode=yes \
-o ConnectTimeout=10 -p "$port" hatch@127.0.0.1 'echo OK' 2>&1)
case "$out" in
OK) auth="OK" ;;
*"Permission denied"*) auth="AUTH-FAIL (check authorized_keys perms/keys)" ;;
*"Connection refused"*) auth="REFUSED (tunnel died after listen check)" ;;
*) auth="OTHER: $(echo "$out" | head -1 | cut -c1-60)" ;;
esac
else
auth="SKIP"
fi
printf '%-10s port %-5s listener: %-22s auth: %s\n' "$node" "$port" "$listener" "$auth"
[ "$listener" = "DARK (no listener)" ] && fail=1
case "$auth" in AUTH-FAIL*) fail=1 ;; esac
done
if [ "$fail" -eq 0 ]; then
echo "ALL NODES REACHABLE"
else
echo "ISSUES FOUND (see above)"
fi
exit "$fail"
+38
View File
@@ -0,0 +1,38 @@
# Ticket #213 verification — SSH key perms and container dial-in (646)
Date: 2026-10-09 ~22:50 UTC
Operator: operator-646 (muse-646-patha)
Branch: `dev/646/213-fix-ssh-perms`
## 1. authorized_keys permissions (port 2226 dial-in)
- `~/.ssh/authorized_keys` (`/home/hatch/.ssh/authorized_keys`):
- before: `600 root:root`
- ran `chmod 600 ~/.ssh/authorized_keys` per ticket
- after: `600 root:root` (no-op — already correct)
- sshd's requirement (private key file must not be group/world-writable,
ideally 600) is satisfied. `~/.ssh` itself is `700`.
## 2. Container sshd
- `sshd` running (pid 2655, listener, 0 of 10-100 startups).
- Listening on `0.0.0.0:22` and `[::]:22`.
- `authorized_keys` holds 1 key:
- `ssh-ed25519 SHA256:UOeqKF5BehWNmEpBSk53Qhz0Jd9aQXbFO0VKe2AVo8c`
(comment `super@bl`) — dial-in identity belongs to super.
## 3. Reverse tunnel (VM 2226 → container:22)
- On VM 34.139.37.135 (as dev-operator-646): `127.0.0.1:2226` and
`[::1]:2226` are LISTENING — the reverse tunnel is up.
- Bind is loopback-only (no GatewayPorts), so dial-in must originate
from the VM itself — expected for `ssh -R` forwards.
## 4. Dial-in path verdict
Container-side prerequisites are all green: perms 600, sshd listening,
tunnel established, authorized key present. The final key-auth step can
only be completed by the holder of the `super@bl` private key, so no
full loopback auth was attempted from this operator identity.
Fixes #213