Explain failed jobs
job_explain runs fixed rules over the accounting record and the redacted log tail: out of memory, timeouts, missing modules, Python import errors, CUDA problems, disk full and more. Each finding carries its evidence and a fix.
The Bifrost between AI and your HPC cluster.
Safe, typed MCP tools that let AI assistants do real work on a Slurm cluster over SSH. One Go binary on your own machine that is both a stdio MCP server and a CLI, reaching the cluster through your own OpenSSH login: Claude, Copilot, Gemini, Hermes, Codex, OpenCode and your own scripts get audited, read-mostly access to job status, failure diagnosis, costs, software modules, batch-script checks, results, and (opt-in, two-step) submit and cancel.
You: Why did job 236 fail?
Claude: [job_explain 236]
It failed after one minute on the gpu partition: the log says
"ModuleNotFoundError: No module named 'pandas'". The script calls python
without an environment. Load the site's PyTorch or SciPy module
before running it, or create a uv/Pixi environment inside the job.
Everything the MCP tools do is also a command, and every command takes --json. Every result is an envelope with the data, the evidence, and the scheduler commands that produced it, so an answer can be checked rather than trusted.
job_explain runs fixed rules over the accounting record and the redacted log tail: out of memory, timeouts, missing modules, Python import errors, CUDA problems, disk full and more. Each finding carries its evidence and a fix.
cluster_status and partitions: nodes powered up or down, queue depth, limits, GPUs, prices per node-hour, and the dollars per hour being spent right now.
script_check statically checks a script against the real cluster: partition, limits, modules and their MPI prerequisite, GPU request, a core request on shared partitions, and worst-case cost. No submission.
modules_search, module_show and recipes: module versions, MPI prerequisites, GPU builds, prebuilt containers, known-good recipes and the shared reference databases.
my_usage and waste_report: node-hours, core-hours, CPU efficiency and estimated cost, plus avoidable spend from idle cores, idle nodes and repeated failures.
Paged, redacted log windows; listings and chunked reads of your own home, scratch and job folders; job folders copied to your machine over the same SSH connection.
interactive_help returns the exact srun/salloc command, its hourly and worst-case cost, and how to reach the login node. It runs no job.
Opt-in tier A1, granted per person: plan, show worst-case cost, then confirm with a single-use token. Caps on nodes, hours and dollars are enforced before anything reaches the scheduler.
Everything the MCP tools do is also a clusterwand command. Two examples from the README:
$ clusterwand check train.sbatch
PROBLEMS FOUND partition cpu worst-case cost $X.XX (1 cores, 6% of a node)
error line 4: module "python3" not found in the cluster catalog; closest: python-sci, python-ml
error cpu shares nodes between jobs: this script asks for no cores, so Slurm gives
it 1 core and about 8 GB on cpu. Ask for the cores it needs ...
warning calls python without loading python-sci/python-ml or an environment
$ clusterwand submit hello.sh
Plan (nothing submitted yet):
partition cpu, 1 node(s), time limit 0 h 05 min
scheduler would start at ... on ... (cpu)
worst case $X.XX (today so far $0.00 of your day cap)
Submit this job? [y/N] y
Submitted job 265 ... folder on the cluster: ~/clusterwand-jobs/<stamp>-hello
Submitting needs tier A1. Each submitted job runs in its own ~/clusterwand-jobs/<stamp>-<name> folder, so its results are exactly that folder. Caps limit nodes, hours and spend per job and per day; a plan over a cap is refused before anything reaches the scheduler.
The assistant sits on one side and the scheduler on the other. clusterwand is the bridge: tier checks, rate limits, a short-TTL cache, redaction and audit, and a backend that can only build allow-listed commands.
squeue sacct sinfo scontrol --json module sbatch
Your own SSH login, no server.
clusterwand mcp over stdio, plus the full CLINothing installed or changed on the cluster.
A caller only sees, and can only call, the tiers granted to them (tiers in your local config). Read tools carry readOnlyHint: true, act tools false.
Everyone
Opt-in per person
Research Computing
ticket_draft writes a reply draft for a "my job failed" ticket. It is never sent.
Also: resources hpc://catalog and hpc://policies; prompts diagnose_job, write_batch_script, monthly_usage_summary, triage_ticket. Full catalog: SPEC section 9.
clusterwand assumes the model on the other end will sometimes be wrong, and will occasionally be manipulated by text it reads.
job_submit runs script_check, enforces caps and sbatch --test-only, and returns the plan, estimated start and worst-case cost. Nothing is submitted.
The exact action is stored under a random token: 10-minute expiry, bound to the plan hash, so a model cannot submit something different from what was shown.
job_submit_confirm accepts nothing but the token. The day cap is checked again; cancel, hold and release re-check ownership and state.
Remote commands can only be built by constructors that validate their arguments; caller values arrive only as positional parameters. A test pins the allow-list, and a mutation check disables each guard in turn and requires a test to fail.
State changes exist only in tier A1, and acting takes two steps. Other users' jobs are denied unless the caller has tier R2.
Reads are limited to your own home, scratch and job folders. Hidden and credential-like files are refused; symlinks are resolved and checked again. Log paths come from the job's accounting record, never from the caller.
Log lines, scripts and submit lines come back only in untrusted blocks with a note telling the model it is data, not instructions.
Tokens, keys, passwords, JWTs, private keys and credentials in URLs are removed from every output and from the audit log.
Every call, CLI or MCP, appends a JSON line (mode 600): caller, client, tool, redacted args, decision, commands, size, duration.
Short-TTL cache, a per-user rate limit, one shared SSH connection, and no background polling.
ClusterWand runs as you over your own SSH connection and never sees your private keys. It can do only what your login can, and less: only the allow-listed commands.
Shared partitions: a script must ask for its cores or script_check and job_submit refuse it, and costs are the share of the node the job holds. Details: SPEC section 7.2.
If ssh mycluster works without a prompt, ClusterWand works too: it uses your own OpenSSH login and keeps one multiplexed connection open. Or try it first with no cluster at all.
The repo ships anonymized recordings of real Slurm output; the fixture backend replays them:
git clone https://github.com/UCR-Research-Computing/clusterwand && cd clusterwand
go build -o clusterwand ./cmd/clusterwand
printf 'backend: fixture\nfixtures_dir: %s/testdata\ntiers: [R1, R2, A1]\naudit_path: /tmp/cw-audit.jsonl\nstate_path: /tmp/cw-a1.json\n' "$PWD" > /tmp/cw-demo.yaml
./clusterwand job explain 236 --config /tmp/cw-demo.yaml
Then try status, jobs, modules lammps, or mcp to serve the same over stdio.
Linux/macOS release binary to ~/.local/bin, checksum-verified, with the short alias wand:
curl -fsSL https://raw.githubusercontent.com/UCR-Research-Computing/clusterwand/main/scripts/get.sh | sh
Or with Go:
go install github.com/UCR-Research-Computing/clusterwand/cmd/clusterwand@latest
Then set up and check:
clusterwand config init # writes ~/.config/clusterwand/config.yaml
# set ssh.host: mycluster (see docs/SSH.md)
clusterwand doctor # checks the config, SSH to the login node, the catalog
clusterwand status
Add it to a client as a stdio server, in any JSON config:
{ "mcpServers": { "cluster": { "command": "clusterwand", "args": ["mcp"] } } }
Every client starts clusterwand mcp as a local stdio server. The server name cluster is your choice.
hermes mcp add cluster --command clusterwand --args mcp
claude mcp add cluster -- clusterwand mcp
Add -s user for every project.
{"mcpServers": {"cluster": {
"command": "clusterwand", "args": ["mcp"]}}}
In claude_desktop_config.json.
codex mcp add cluster -- clusterwand mcp
"cluster": {"type": "local",
"command": ["clusterwand", "mcp"]}
Under mcp in opencode.jsonc.
gemini mcp add -s user cluster clusterwand mcp
{"servers": {"cluster": {"type": "stdio",
"command": "clusterwand", "args": ["mcp"]}}}
In .vscode/mcp.json, then enable the tools in Copilot Chat agent mode.
clusterwand job explain 236 --json
python3 examples/mcp_client.py call job_explain '{"job_id": "236"}'
Every tool is a CLI command with --json; the example is a stdio MCP client in one file.
Config-file versions and troubleshooting for every client: docs/CLIENTS.md.
Nothing in ClusterWand is specific to one site except configuration: the login node, the catalog (partitions, prices, modules, recipes), log roots and caps. Start with clusterwand config init, point ssh.host at your login node, and write a catalog. Site profiles and clusterwand discover are on the roadmap to replace the hand-written catalog.
ClusterWand never needs anything installed or changed on the cluster: a normal researcher login is enough, and the catalog can live in your own home folder. Ursa Major's notes live under sites/ursa-major/.
make check # gofmt, go vet, go test -race, build (before every commit)
scripts/mutation_check.sh # every safety guard must be caught by a test
scripts/mcp_smoke.py # drives `clusterwand mcp` over stdio like a real client
Tests run against anonymized recordings of real cluster output in testdata/ (Slurm 25.11.4). scripts/make_fixtures.py regenerates them from a read-only probe and refuses to write if a real username survives; CI checks again.