Nodes wrote the WARN log "Head state missing, repairing" and terminated abnormality

[naoki4]

I ran into the problem as below.
What is the cause of the problem, and how can I solve it?

System information

Geth version: Geth/v1.10.3-stable(quorum-v23.4.0)/linux-amd64/go1.19.13
OS & Version: Linux

Expected behaviour

Blocks are produced constantly with no error.
If there is no transaction, empty blocks are produced.

Actual behaviour

After running for 35 days, one of four validator nodes wrote the log "WARN [05-13|21:33:02.069] Head state missing, repairing number=2,536,276 hash=0e15f3..ae5be1 snaproot=936efa..1465ee", and the pod rebooted (geth process was terminated abnormality). The node wrote the log and rebooted repeatedly, and the problem never fixed.

Then afer another 4 days,　another node wrote the same log message, and rebooted repeatedly.

At this point, there were two valid validator nodes, and blocks can no longer be generated.

Steps to reproduce the behaviour

I ran Quorum varidator/transaction nodes on AKS.
The source code to build the container image of Quorum is the one of v23.4.0 tag, and the fix in the PR #1669 (QBFT: fix chain fork when emptyblockperiodseconds is enabled) is applied so as to prevent a chain fork.

parameters in genesis.json

• consensus: qbft
• blockperiodseconds: 5
• emptyblockperiodseconds: 0
• requesttimeoutseconds: 10

Backtrace

The log of first node which wrote the log "Head state missing, repairing".

INFO [05-13|21:32:46.654] Running with private transaction manager disabled - quorum private transactions will not be supported 
INFO [05-13|21:32:46.656] Maximum peer count                       ETH=50 LES=0 total=50
INFO [05-13|21:32:46.656] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [05-13|21:32:46.767] Found deprecated node list file /data/quorum/geth/static-nodes.json, please use the TOML config file instead. 
INFO [05-13|21:32:46.803] Enabling recording of key preimages since archive mode is used 
INFO [05-13|21:32:46.803] Set global gas cap                       cap=25,000,000
INFO [05-13|21:32:46.806] Allocated cache and file handles         database=/data/quorum/geth/chaindata cache=16.00MiB handles=16
INFO [05-13|21:32:56.974] Persisted trie from memory database      nodes=5 size=779.00B time="21.601µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [05-13|21:32:57.643] Successfully wrote genesis state         database=chaindata                   hash=58c9d0..16067d
INFO [05-13|21:32:57.643] Allocated cache and file handles         database=/data/quorum/geth/lightchaindata cache=16.00MiB handles=16
INFO [05-13|21:32:58.146] Persisted trie from memory database      nodes=5 size=779.00B time="24.201µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [05-13|21:32:58.444] Successfully wrote genesis state         database=lightchaindata                   hash=58c9d0..16067d
INFO [05-13|21:32:58.578] Starting pprof server                    addr=http://0.0.0.0:9545/debug/pprof
INFO [05-13|21:32:58.578] Enabling metrics collection 
INFO [05-13|21:32:58.578] Running with private transaction manager disabled - quorum private transactions will not be supported 
INFO [05-13|21:32:58.581] Maximum peer count                       ETH=50 LES=0 total=50
INFO [05-13|21:32:58.581] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [05-13|21:32:58.581] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable 
WARN [05-13|21:32:58.612] Found deprecated node list file /data/quorum/geth/static-nodes.json, please use the TOML config file instead. 
WARN [05-13|21:32:58.641] WARNING: The flag --istanbul.blockperiod is deprecated and will be removed in the future, please use ibft.blockperiodseconds on genesis file 
INFO [05-13|21:32:58.641] Enabling recording of key preimages since archive mode is used 
INFO [05-13|21:32:58.641] Set global gas cap                       cap=25,000,000
INFO [05-13|21:32:58.641] Running with private transaction manager disabled - quorum private transactions will not be supported 
WARN [05-13|21:32:58.641] Sanitizing invalid miner gas price       provided=0 updated=0
INFO [05-13|21:32:58.641] Allocated trie memory caches             clean=154.00MiB dirty=256.00MiB
INFO [05-13|21:32:58.644] Allocated cache and file handles         database=/data/quorum/geth/chaindata cache=768.00MiB handles=524,288
INFO [05-13|21:32:59.383] Opened ancient database                  database=/data/quorum/geth/chaindata/ancient readonly=false
INFO [05-13|21:33:00.560] Initialised chain configuration          config="{ChainID: 1337 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 IsQuorum: true Constantinople: 0 TransactionSizeLimit: 64 MaxCodeSize: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Berlin: 0  Catalyst: <nil> YOLO v3: <nil> PrivacyEnhancements: <nil> PrivacyPrecompile: <nil> EnableGasPriceBlock: <nil> Engine: unknown}"
INFO [05-13|21:33:01.122] Initialising Ethereum protocol           network=10 dbversion=8
INFO [05-13|21:33:01.122] Initialising Quorum consensus protocol   name=istanbul versions="[100 99 64]" network=10 dbversion=8
INFO [05-13|21:33:01.268] Using SegregatedCacheProvider. 
INFO [05-13|21:33:01.994] Loaded most recent local header          number=2,536,276 hash=0e15f3..ae5be1 td=2,536,277 age=15s
INFO [05-13|21:33:01.994] Loaded most recent local full block      number=2,536,276 hash=0e15f3..ae5be1 td=2,536,277 age=15s
INFO [05-13|21:33:01.994] Loaded most recent local fast block      number=2,536,276 hash=0e15f3..ae5be1 td=2,536,277 age=15s
WARN [05-13|21:33:02.069] Head state missing, repairing            number=2,536,276 hash=0e15f3..ae5be1 snaproot=936efa..1465ee

The log of second node which wrote the log "Head state missing, repairing".

INFO [05-17|19:54:07.842] Running with private transaction manager disabled - quorum private transactions will not be supported 
INFO [05-17|19:54:07.844] Maximum peer count                       ETH=50 LES=0 total=50
INFO [05-17|19:54:07.844] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [05-17|19:54:07.950] Found deprecated node list file /data/quorum/geth/static-nodes.json, please use the TOML config file instead. 
INFO [05-17|19:54:08.022] Enabling recording of key preimages since archive mode is used 
INFO [05-17|19:54:08.022] Set global gas cap                       cap=25,000,000
INFO [05-17|19:54:08.025] Allocated cache and file handles         database=/data/quorum/geth/chaindata cache=16.00MiB handles=16
INFO [05-17|19:54:12.644] Persisted trie from memory database      nodes=5 size=779.00B time="22.8µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [05-17|19:54:13.235] Successfully wrote genesis state         database=chaindata                   hash=58c9d0..16067d
INFO [05-17|19:54:13.235] Allocated cache and file handles         database=/data/quorum/geth/lightchaindata cache=16.00MiB handles=16
INFO [05-17|19:54:13.733] Persisted trie from memory database      nodes=5 size=779.00B time="18.2µs" gcnodes=0 gcsize=0.00B gctime=0s livenodes=1 livesize=0.00B
INFO [05-17|19:54:14.068] Successfully wrote genesis state         database=lightchaindata                   hash=58c9d0..16067d
INFO [05-17|19:54:14.202] Starting pprof server                    addr=http://0.0.0.0:9545/debug/pprof
INFO [05-17|19:54:14.203] Enabling metrics collection 
INFO [05-17|19:54:14.203] Running with private transaction manager disabled - quorum private transactions will not be supported 
INFO [05-17|19:54:14.205] Maximum peer count                       ETH=50 LES=0 total=50
INFO [05-17|19:54:14.205] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [05-17|19:54:14.205] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable 
WARN [05-17|19:54:14.233] Found deprecated node list file /data/quorum/geth/static-nodes.json, please use the TOML config file instead. 
WARN [05-17|19:54:14.257] WARNING: The flag --istanbul.blockperiod is deprecated and will be removed in the future, please use ibft.blockperiodseconds on genesis file 
INFO [05-17|19:54:14.257] Enabling recording of key preimages since archive mode is used 
INFO [05-17|19:54:14.257] Set global gas cap                       cap=25,000,000
INFO [05-17|19:54:14.257] Running with private transaction manager disabled - quorum private transactions will not be supported 
WARN [05-17|19:54:14.257] Sanitizing invalid miner gas price       provided=0 updated=0
INFO [05-17|19:54:14.257] Allocated trie memory caches             clean=154.00MiB dirty=256.00MiB
INFO [05-17|19:54:14.260] Allocated cache and file handles         database=/data/quorum/geth/chaindata cache=768.00MiB handles=524,288
INFO [05-17|19:54:15.141] Opened ancient database                  database=/data/quorum/geth/chaindata/ancient readonly=false
INFO [05-17|19:54:15.524] Initialised chain configuration          config="{ChainID: 1337 Homestead: 0 DAO: <nil> DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 IsQuorum: true Constantinople: 0 TransactionSizeLimit: 64 MaxCodeSize: 0 Petersburg: 0 Istanbul: 0, Muir Glacier: 0, Berlin: 0  Catalyst: <nil> YOLO v3: <nil> PrivacyEnhancements: <nil> PrivacyPrecompile: <nil> EnableGasPriceBlock: <nil> Engine: unknown}"
INFO [05-17|19:54:15.945] Initialising Ethereum protocol           network=10 dbversion=8
INFO [05-17|19:54:15.945] Initialising Quorum consensus protocol   name=istanbul versions="[100 99 64]" network=10 dbversion=8
INFO [05-17|19:54:16.481] Loaded most recent local header          number=2,580,737 hash=f6cd96..606c4e td=2,580,738 age=10s
INFO [05-17|19:54:16.106] Using SegregatedCacheProvider. 
INFO [05-17|19:54:16.481] Loaded most recent local full block      number=2,580,737 hash=f6cd96..606c4e td=2,580,738 age=10s
INFO [05-17|19:54:16.481] Loaded most recent local fast block      number=2,580,737 hash=f6cd96..606c4e td=2,580,738 age=10s
WARN [05-17|19:54:16.599] Head state missing, repairing            number=2,580,737 hash=f6cd96..606c4e snaproot=936efa..1465ee