Troubleshooting / CLI Commands

Essential command-line tools and scripts for Monad operators

Monlog (by Category Labs)

A lightweight tool that scrapes BFT logs and provides useful status information and suggestions. Examines logs from the last 60 seconds.

1️⃣ Grant monad user journal access (as root):

usermod -a -G systemd-journal monad

💡 If already logged in as monad, log out and back in or run su - monad for changes to take effect.

2️⃣ Download monlog (as monad user):

cd /home/monad
curl -sSL https://pub-b0d0d7272c994851b4c8af22a766f571.r2.dev/scripts/monlog -O
chmod u+x ./monlog

# Verify checksum
sha256sum ./monlog
# Expected: f8a1066d8c093bbdb5f722f5b3d89904c17a76fa01fa1dd62e950f445e88327f

3️⃣ Run monlog:

./monlog              # Basic status
watch -d "./monlog"   # Live updates
./monlog -r           # Show last 10 lines of grabbed logs
./monlog -n           # Show secp keys mapped to validator names
./monlog --no-color   # No color coding

Node Status with monad-status

1️⃣ Install monad-status:

curl -sSL https://bucket.monadinfra.com/scripts/monad-status.sh -o /usr/local/bin/monad-status
chmod +x /usr/local/bin/monad-status

2️⃣ Run monad-status:

monad-status

💡 Prints a full status report including hostname, version, config, services, peers, consensus status, statesync progress, and RPC info.

Monad-BFT Sync Progress Check

Sync phases occur in this order: StateSync → BlockSync → Proposal Activity, depending on your node's current state.

1️⃣ Check if node is making StateSync requests:

journalctl -u monad-bft -f | grep "statesync"

2️⃣ Check if node is receiving StateSync responses:

journalctl -u monad-bft -f | grep "applied response"

3️⃣ Check if node is progressing prefixes toward/up to 255 during StateSync:

journalctl -u monad-bft -f | grep "prefix"

4️⃣ Check if node completed StateSync (after reaching prefix 255):

journalctl -u monad-bft -f | grep "done statesync"

5️⃣ Check if node is receiving BlockSync responses:

journalctl -u monad-bft -f | grep "blocksync: received payload response"

6️⃣ Check if node received proposal messages (after BlockSync completion):

journalctl -u monad-bft -f | grep "proposal message"

✓ 7️⃣ Verify node is synced and working normally

If you confirm you are seeing proposal messages in your logs at step 6, your node is synced and working normally.

Monad-Execution Sync Progress Check

⚠️Note: Check Monad-Execution sync progress only after Monad-BFT is fully synchronized

1️⃣ Check Sync Status:

curl -X POST -H "Content-Type: application/json" --data '{"jsonrpc":"2.0","method":"eth_syncing","params":[],"id":1}' http://localhost:8080

💡 RPC will only respond after the node completes StateSync. Once sync is done, it returns false

2️⃣ Check Latest Block Height:

curl -s -H "Content-Type: application/json" -d '{"id":1,"jsonrpc":"2.0","method":"eth_blockNumber","params":[]}' http://localhost:8080 | jq .result

Returns the latest block number in hexadecimal format (e.g., "0x2e15af3")

3️⃣ Check current execution height vs reference RPCs:

Note: These reference RPCs are for testnet. Use different reference RPCs for mainnet.

alias monadcheck='L=http://127.0.0.1:8080;R1=https://monad-testnet-rpc.huginn.tech;R2=https://rpc-testnet-2.monadinfra.com;for url in $R1 $R2;do h=$(curl -s -X POST $url -H "Content-Type: application/json" -d "{\"jsonrpc\":\"2.0\",\"method\":\"eth_blockNumber\",\"params\":[],\"id\":1}"|jq -r ".result"|awk "{print strtonum(\$1)}");refs+=($h);done;l=$(curl -s -X POST $L -H "Content-Type: application/json" -d "{\"jsonrpc\":\"2.0\",\"method\":\"eth_blockNumber\",\"params\":[],\"id\":1}"|jq -r ".result"|awk "{print strtonum(\$1)}");ref=$(printf "%s\n" "${refs[@]}"|sort -nr|head -1);diff=$((ref-l));[[ $diff -le 10 ]]&&s=🟢||s=🟡;echo Local:$l\|Ref:$ref\|Diff:$diff\|$s'

4️⃣ Run the alias to check Execution sync status:

monadcheck

💡 Compares local execution height with multiple reference RPCs and prints real-time sync status (🟢 = Synced / 🟡 = Syncing).

Check Validator Status via monad-ledger-tail

Monitor validator events in real-time:

journalctl -u monad-ledger-tail -o cat -f | grep -i "0X_YOUR_SECP_KEY"

💡 When you're in the active set, this will show your proposed, finalized, and timeout events in real-time

Live Consensus Tracking

Track Online Validator Stake After Hardfork:

After hardfork updates, validators can track the network's online status with this command. Requires 66.67% stake online.

journalctl -u monad-bft -o cat -f | grep --line-buffered '"current_stake"' | jq -r '[(.fields.current_stake | gsub("Ok\$$Stake\\("; "") | gsub("\$$"; "") | tonumber), (.fields.total_stake | gsub("Stake\$$"; "") | gsub("\$$"; "") | tonumber)] | "\(.[ 0]) / \(.[1]) = \((.[ 0] / .[1] * 100) | floor)% online"'

Troubleshooting

1️⃣ state root doesn't match, are peers trusted?

Example logs:

{"timestamp":"2025-10-06T16:38:07.419872Z","level":"ERROR","fields":{"message":"state root doesn't match, are peers trusted?"},"target":"monad_consensus_state"}

This indicates a version mismatch between peers. Check your installed version:

dpkg -l | grep monad

If outdated, update to the latest release.

2️⃣ Epoch not found in validator_map

Example logs:

{"timestamp":"2025-10-06T16:38:07.419872Z","level":"WARN","fields":{"message":"Epoch not found in validator_map","msg_epoch":"785","self.validator_map":"{784: Group { start: 0, end: 0, other_peers: 170 }}"},"target":"monad_raptorcast::util"}

Your validators.toml file is outdated. If you're close to the network, try soft-reset. If that doesn't work, try hard reset. Check Snapshot page for detailed instructions.

3️⃣ Proposal Dropping / Randao Validation Failed

Your validator is synced, but still dropping proposals → SECP/BLS key mismatch.

Example logs

{"message":"dropping proposal, randao validation failed"}

{"message":"Invalid timeout signature"}

{"message":"Timestamp validation failed"}

Check logs

journalctl -u monad-bft -f | grep 'dropping proposal'

journalctl -u monad-bft -f | grep 'validation failed'

Root Cause (2 possibilities)

1️⃣

Wrong keys used when creating the validator

BLS private key was provided without 0x
Old/deprecated backup files were used (IKM format changed)
Mismatched SECP/BLS keys were registered on-chain

➡️In this case, you must recover the correct keys and recreate the validator. Please get support from the Monad Dev Discord first!

2️⃣

Incorrect import command

If you used monad-keystore import with --key-type,

→ BLS keystore may be corrupted → proposal drops.

➡️In this case, simply re-import using the correct command + restart monad-bft.

Correct Key Formats

• SECP private key → 64 hex (NO 0x)
• BLS private key → 0x + 64 hex

🔧Fix A — Wrong backup / wrong key used

➡️Extract your real private keys from the active keystore (never use old backups):

monad-keystore recover --password "$KEYSTORE_PASSWORD" \
  --keystore-path /home/monad/monad-bft/config/id-secp --key-type secp

monad-keystore recover --password "$KEYSTORE_PASSWORD" \
  --keystore-path /home/monad/monad-bft/config/id-bls --key-type bls

Compare the recovered public keys with the ones registered on-chain.
If they do not match, you must recreate your validator using the recovered keys.

🔧Fix B — Incorrect import usage

❗ Do NOT pass --key-type when importing.
(There is a BLS import bug that can write an invalid IKM.)

Correct import commands:

1️⃣ SECP Import

monad-keystore import --ikm <SECP_IKM> \
  --keystore-path /home/monad/monad-bft/config/id-secp \
  --password "$KEYSTORE_PASSWORD"

2️⃣ BLS Import

monad-keystore import --ikm <BLS_IKM> \
  --keystore-path /home/monad/monad-bft/config/id-bls \
  --password "$KEYSTORE_PASSWORD"

3️⃣ Verify

monad-keystore recover --password "$KEYSTORE_PASSWORD" \
  --keystore-path /home/monad/monad-bft/config/id-secp --key-type secp

monad-keystore recover --password "$KEYSTORE_PASSWORD" \
  --keystore-path /home/monad/monad-bft/config/id-bls --key-type bls

If the recovered output matches the expected public keys →
➡️ Just restart monad-bft:

sudo systemctl restart monad-bft

4️⃣ RaptorCastSecondary rejecting invite with group size exceeds max

Example logs:

{"timestamp":"2025-10-06T16:38:07.419872Z","level":"WARN","fields":{"message":"RaptorCastSecondary rejecting invite with group size 150 that exceeds max group size 50"},"target":"monad_raptorcast::raptorcast_secondary::client"}

Your node is dropping from P2P due to an incorrect group size configuration. Update the value in node.toml to match the log:

[fullnode_raptorcast]
max_group_size = 150

Then perform a hard reset and re-sync.

5️⃣ Local timeout / High QC

Example logs:

{"timestamp":"2025-09-25T21:34:00.623430Z","level":"DEBUG","fields":{"message":"local timeout","round":"2099506","leader":"03afd66a0822428268bf9bdf06e4038ca240b29051683bedf61aa6f80ac1a9ba7a","next_leader":"03aa8176c7d2451678ccf8e81f416ce8a41f2ba62a1d5217885585177ac4d1b6cc"},"target":"monad_consensus_state"}

Intermittent timeouts are normal. It happens in the network regularly, usually because of offline validator. As long as you're not falling behind (panic with "high qc too far ahead" or stuck on a round for over a minute), you shouldn't need to do anything.

{"timestamp":"2025-09-25T21:34:00.623430Z","level":"WARN","fields":{"message":"high qc too far ahead of block tree root","highqc":"1357428","block-tree-root":"1355576"},"target":"monad_consensus_state"}

If it persists and you see high qc too far ahead, try a soft-reset or hard-reset to resync.

6️⃣ result_ptr.is_null() panic in TrieDB

Example logs:

thread '<unnamed>' panicked at monad-triedb/src/lib.rs:490:9:
assertion failed: !result_ptr.is_null()

If the snapshot file you downloaded and the forkpoint.toml are incompatible, or if the forkpoint is behind (meaning forkpoint.toml contains outdated or incorrect epoch/round/root values), you'll encounter this error. The solution is to replace both files with their latest or compatible versions. Check Snapshot page for detailed instructions.