Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 97f0e4e50e | |||
| 78d22bd950 | |||
| c0113c1ebf | |||
| 69555f809c |
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/bin/bash
|
||||
# verify-node-ssh.sh — verify container SSH dial-in readiness across fleet nodes.
|
||||
# Checks from the VM: reverse-tunnel listeners + SSH auth for each node port.
|
||||
#
|
||||
# Port map (docs/OPERATOR-DRIVE-RUNBOOK.md):
|
||||
# muse-main 2224 | muse 2225 | 646 2226 | pip 2227 | opm 2228 | def 2229 | dev 2230
|
||||
#
|
||||
# What it checks per node:
|
||||
# 1. Reverse-tunnel listener on 127.0.0.1:<port> (dark node = no listener)
|
||||
# 2. SSH dial-in with BatchMode (auth failure = authorized_keys perms/key issue)
|
||||
#
|
||||
# Common root causes (see #211):
|
||||
# - sshd requires non-group-writable authorized_keys (must be 600)
|
||||
# - stale /run/nologin blocks logins
|
||||
# - missing id_frontdoor keys on dark nodes
|
||||
#
|
||||
# Usage: run on the VM (super@34.139.37.135), or via:
|
||||
# ssh-vm.sh "bash -s" < verify-node-ssh.sh
|
||||
set -u
|
||||
|
||||
# node:port pairs to check
|
||||
NODES="muse:2225 646:2226 pip:2227 def:2229 dev:2230 muse-main:2224 opm:2228"
|
||||
|
||||
fail=0
|
||||
for pair in $NODES; do
|
||||
node="${pair%%:*}"
|
||||
port="${pair##*:}"
|
||||
|
||||
# 1. listener check
|
||||
if ss -tln 2>/dev/null | grep -q "127.0.0.1:${port} "; then
|
||||
listener="LISTEN"
|
||||
else
|
||||
listener="DARK (no listener)"
|
||||
fi
|
||||
|
||||
# 2. auth check (only if listening)
|
||||
if [ "$listener" = "LISTEN" ]; then
|
||||
out=$(timeout 15 ssh -o StrictHostKeyChecking=no -o BatchMode=yes \
|
||||
-o ConnectTimeout=10 -p "$port" hatch@127.0.0.1 'echo OK' 2>&1)
|
||||
case "$out" in
|
||||
OK) auth="OK" ;;
|
||||
*"Permission denied"*) auth="AUTH-FAIL (check authorized_keys perms/keys)" ;;
|
||||
*"Connection refused"*) auth="REFUSED (tunnel died after listen check)" ;;
|
||||
*) auth="OTHER: $(echo "$out" | head -1 | cut -c1-60)" ;;
|
||||
esac
|
||||
else
|
||||
auth="SKIP"
|
||||
fi
|
||||
|
||||
printf '%-10s port %-5s listener: %-22s auth: %s\n' "$node" "$port" "$listener" "$auth"
|
||||
[ "$listener" = "DARK (no listener)" ] && fail=1
|
||||
case "$auth" in AUTH-FAIL*) fail=1 ;; esac
|
||||
done
|
||||
|
||||
if [ "$fail" -eq 0 ]; then
|
||||
echo "ALL NODES REACHABLE"
|
||||
else
|
||||
echo "ISSUES FOUND (see above)"
|
||||
fi
|
||||
exit "$fail"
|
||||
@@ -0,0 +1,38 @@
|
||||
# Ticket #213 verification — SSH key perms and container dial-in (646)
|
||||
|
||||
Date: 2026-10-09 ~22:50 UTC
|
||||
Operator: operator-646 (muse-646-patha)
|
||||
Branch: `dev/646/213-fix-ssh-perms`
|
||||
|
||||
## 1. authorized_keys permissions (port 2226 dial-in)
|
||||
|
||||
- `~/.ssh/authorized_keys` (`/home/hatch/.ssh/authorized_keys`):
|
||||
- before: `600 root:root`
|
||||
- ran `chmod 600 ~/.ssh/authorized_keys` per ticket
|
||||
- after: `600 root:root` (no-op — already correct)
|
||||
- sshd's requirement (private key file must not be group/world-writable,
|
||||
ideally 600) is satisfied. `~/.ssh` itself is `700`.
|
||||
|
||||
## 2. Container sshd
|
||||
|
||||
- `sshd` running (pid 2655, listener, 0 of 10-100 startups).
|
||||
- Listening on `0.0.0.0:22` and `[::]:22`.
|
||||
- `authorized_keys` holds 1 key:
|
||||
- `ssh-ed25519 SHA256:UOeqKF5BehWNmEpBSk53Qhz0Jd9aQXbFO0VKe2AVo8c`
|
||||
(comment `super@bl`) — dial-in identity belongs to super.
|
||||
|
||||
## 3. Reverse tunnel (VM 2226 → container:22)
|
||||
|
||||
- On VM 34.139.37.135 (as dev-operator-646): `127.0.0.1:2226` and
|
||||
`[::1]:2226` are LISTENING — the reverse tunnel is up.
|
||||
- Bind is loopback-only (no GatewayPorts), so dial-in must originate
|
||||
from the VM itself — expected for `ssh -R` forwards.
|
||||
|
||||
## 4. Dial-in path verdict
|
||||
|
||||
Container-side prerequisites are all green: perms 600, sshd listening,
|
||||
tunnel established, authorized key present. The final key-auth step can
|
||||
only be completed by the holder of the `super@bl` private key, so no
|
||||
full loopback auth was attempted from this operator identity.
|
||||
|
||||
Fixes #213
|
||||
Reference in New Issue
Block a user