Troubleshooting / CLI Commands

Essential command-line tools and scripts for Monad operators

Node Status with monad-status

1. Install monad-status:

curl -sSL https://bucket.monadinfra.com/scripts/monad-status.sh -o /usr/local/bin/monad-status
chmod +x /usr/local/bin/monad-status

2. Run monad-status:

monad-status

Prints a full status report including hostname, version, config, services, peers, consensus status, statesync progress, and RPC info.

Monad-BFT Sync Progress Check

Sync phases occur in this order: StateSync → BlockSync → Proposal Activity, depending on your node's current state.

1. Check if node is making StateSync requests:

journalctl -u monad-bft -f | grep "statesync"

2. Check if node is receiving StateSync responses:

journalctl -u monad-bft -f | grep "applied response"

3. Check if node is progressing prefixes toward/up to 255 during StateSync:

journalctl -u monad-bft -f | grep "prefix"

4. Check if node completed StateSync (after reaching prefix 255):

journalctl -u monad-bft -f | grep "done statesync"

5. Check if node is receiving BlockSync responses:

journalctl -u monad-bft -f | grep "blocksync: received payload response"

6. Check if node received proposal messages (after BlockSync completion):

journalctl -u monad-bft -f | grep "proposal message"

✓ 7. Verify node is synced and working normally

If you confirm you are seeing proposal messages in your logs at step 6, your node is synced and working normally.

Monad-Execution Sync Progress Check

Note: Check Monad-Execution sync progress only after Monad-BFT is fully synchronized

1. Check Sync Status:

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' http://localhost:8080

RPC will only respond after the node completes StateSync. Once sync is done, it returns false

2. Check Latest Block Height:

curl -s -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}' http://localhost:8080 | jq .result

Returns the latest block number in hexadecimal format (e.g., "0x2e15af3")

3. Check current execution height vs reference RPCs:

Note: These reference RPCs are for testnet. Use different reference RPCs for mainnet.

alias monadcheck='L=http://127.0.0.1:8080;R1=https://monad-testnet-rpc.huginn.tech;R2=https://rpc-testnet-2.monadinfra.com;for url in $R1 $R2;do h=$(curl -s -X POST $url -H "Content-Type: application/json" -d "{\"jsonrpc\":\"2.0\",\"method\":\"eth_blockNumber\",\"params\":[],\"id\":1}"|jq -r ".result"|awk "{print strtonum(\$1)}");refs+=($h);done;l=$(curl -s -X POST $L -H "Content-Type: application/json" -d "{\"jsonrpc\":\"2.0\",\"method\":\"eth_blockNumber\",\"params\":[],\"id\":1}"|jq -r ".result"|awk "{print strtonum(\$1)}");ref=$(printf "%s\n" "${refs[@]}"|sort -nr|head -1);diff=$((ref-l));[[ $diff -le 10 ]]&&s=🟢||s=🟡;echo Local:$l\|Ref:$ref\|Diff:$diff\|$s'

4. Run the alias to check Execution sync status:

monadcheck

Compares local execution height with multiple reference RPCs and prints real-time sync status (🟢 = Synced / 🟡 = Syncing).

Check Validator Status via monad-ledger-tail

Monitor validator events in real-time:

journalctl -u monad-ledger-tail -o cat -f | grep -i "0X_YOUR_SECP_KEY"

When you're in the active set, this will show your proposed, finalized, and timeout events in real-time

Live Consensus Tracking

Track Online Validator Stake After Hardfork:

After hardfork updates, validators can track the network's online status with this command. Requires 66.67% stake online.

journalctl -u monad-bft -o cat -f | grep --line-buffered '"current_stake"' | jq -r '[(.fields.current_stake | gsub("Ok\$$Stake\\("; "") | gsub("\$$"; "") | tonumber), (.fields.total_stake | gsub("Stake\$$"; "") | gsub("\$$"; "") | tonumber)] | "\(.[ 0]) / \(.[1]) = \((.[ 0] / .[1] * 100) | floor)% online"'

Troubleshooting

1. state root doesn't match, are peers trusted?

Example logs:

{"timestamp":"2025-10-06T16:38:07.419872Z","level":"ERROR","fields":{"message":"state root doesn't match, are peers trusted?"},"target":"monad_consensus_state"}

This indicates a version mismatch between peers. Check your installed version:

dpkg -l | grep monad

If outdated, update to the latest release.

2. Epoch not found in validator_map

Example logs:

{"timestamp":"2025-10-06T16:38:07.419872Z","level":"WARN","fields":{"message":"Epoch not found in validator_map","msg_epoch":"785","self.validator_map":"{784: Group { start: 0, end: 0, other_peers: 170 }}"},"target":"monad_raptorcast::util"}

Your validators.toml file is outdated. If you're close to the network, try soft-reset. If that doesn't work, try hard reset. Check Snapshot page for detailed instructions.

3. Proposal Dropping / Randao Validation Failed

Your validator is synced, but still dropping proposals → SECP/BLS key mismatch.

Example logs

{"message":"dropping proposal, randao validation failed"}

{"message":"Invalid timeout signature"}

{"message":"Timestamp validation failed"}

Check logs

journalctl -u monad-bft -f | grep 'dropping proposal'

journalctl -u monad-bft -f | grep 'validation failed'

Root Cause (2 possibilities)

Wrong keys used when creating the validator

BLS private key was provided without 0x
Old/deprecated backup files were used (IKM format changed)
Mismatched SECP/BLS keys were registered on-chain

→In this case, you must recover the correct keys and recreate the validator. Please get support from the Monad Dev Discord first!

Incorrect import command

If you used monad-keystore import with --key-type,

→ BLS keystore may be corrupted → proposal drops.

→In this case, simply re-import using the correct command + restart monad-bft.

Correct Key Formats

• SECP private key → 64 hex (NO 0x)
• BLS private key → 0x + 64 hex

Fix A — Wrong backup / wrong key used

→Extract your real private keys from the active keystore (never use old backups):

monad-keystore recover --password "$KEYSTORE_PASSWORD" \
  --keystore-path /home/monad/monad-bft/config/id-secp --key-type secp

monad-keystore recover --password "$KEYSTORE_PASSWORD" \
  --keystore-path /home/monad/monad-bft/config/id-bls --key-type bls

Compare the recovered public keys with the ones registered on-chain.
If they do not match, you must recreate your validator using the recovered keys.

Fix B — Incorrect import usage

Do NOT pass --key-type when importing.
(There is a BLS import bug that can write an invalid IKM.)

Correct import commands:

1. SECP Import

monad-keystore import --ikm <SECP_IKM> \
  --keystore-path /home/monad/monad-bft/config/id-secp \
  --password "$KEYSTORE_PASSWORD"

2. BLS Import

monad-keystore import --ikm <BLS_IKM> \
  --keystore-path /home/monad/monad-bft/config/id-bls \
  --password "$KEYSTORE_PASSWORD"

3. Verify

monad-keystore recover --password "$KEYSTORE_PASSWORD" \
  --keystore-path /home/monad/monad-bft/config/id-secp --key-type secp

monad-keystore recover --password "$KEYSTORE_PASSWORD" \
  --keystore-path /home/monad/monad-bft/config/id-bls --key-type bls

If the recovered output matches the expected public keys →
→ Just restart monad-bft:

sudo systemctl restart monad-bft

4. RaptorCastSecondary rejecting invite with group size exceeds max

Example logs:

{"timestamp":"2025-10-06T16:38:07.419872Z","level":"WARN","fields":{"message":"RaptorCastSecondary rejecting invite with group size 150 that exceeds max group size 50"},"target":"monad_raptorcast::raptorcast_secondary::client"}

Your node is dropping from P2P due to an incorrect group size configuration. Update the value in node.toml to match the log:

[fullnode_raptorcast]
max_group_size = 150

Then perform a hard reset and re-sync.

5. Local timeout / High QC

Example logs:

{"timestamp":"2025-09-25T21:34:00.623430Z","level":"DEBUG","fields":{"message":"local timeout","round":"2099506","leader":"03afd66a0822428268bf9bdf06e4038ca240b29051683bedf61aa6f80ac1a9ba7a","next_leader":"03aa8176c7d2451678ccf8e81f416ce8a41f2ba62a1d5217885585177ac4d1b6cc"},"target":"monad_consensus_state"}

Intermittent timeouts are normal. It happens in the network regularly, usually because of offline validator. As long as you're not falling behind (panic with "high qc too far ahead" or stuck on a round for over a minute), you shouldn't need to do anything.

{"timestamp":"2025-09-25T21:34:00.623430Z","level":"WARN","fields":{"message":"high qc too far ahead of block tree root","highqc":"1357428","block-tree-root":"1355576"},"target":"monad_consensus_state"}

If it persists and you see high qc too far ahead, try a soft-reset or hard-reset to resync.

6. result_ptr.is_null() panic in TrieDB

Example logs:

thread '<unnamed>' panicked at monad-triedb/src/lib.rs:490:9:
assertion failed: !result_ptr.is_null()

If the snapshot file you downloaded and the forkpoint.toml are incompatible, or if the forkpoint is behind (meaning forkpoint.toml contains outdated or incorrect epoch/round/root values), you'll encounter this error. The solution is to replace both files with their latest or compatible versions. Check Snapshot page for detailed instructions.

7. Node runs clean but is invisible to the network

Symptoms

monad-bft starts with no error and no panic, and the logs look normal. But peers never learn your address, the node is not coherent, and execution does not progress. Nothing in the logs points at the cause. This is the classic failure after a server migration or a failover.

Cause

Every node announces itself with a name record carrying a sequence number. Peers keep the highest sequence number they have seen for an identity and silently discard any record with the same or a lower sequence, so a stale record already held by the network keeps pointing at your old address.

When --node-config is passed, monad-sign-name-record can take the sequence from self_record_seq_num inside node.toml instead of the value you passed on the command line. On a freshly downloaded validator template that field is 0, so the tool signs seq 1 no matter what you asked for. The resulting node.toml is internally consistent — seq 1 with a valid signature for seq 1 — so the startup check passes and the node boots cleanly. But if the identity has already published up to seq 5, peers keep the record they already have and your new node is never discovered.

Detect

grep self_record_seq_num /home/monad/monad-bft/config/node.toml

Compare it against the last sequence this identity published. If it is lower, the record was dead on arrival.

Fix

Set self_record_seq_num in node.toml to the target value before signing, or drop --node-config and drive the sequence purely from the argument. Either way, read the sequence the signer actually reports and write that value, with its signature, into node.toml.

If the signer emits a sequence lower than the one you requested, stop. Do not restart the node until the sequence is corrected.

sudo systemctl restart monad-bft