Not producing block if there is an on going transaction when at least 1 validator is down

[athiwathaven]

System information

Geth version: v1.10.3
GoQuorum version: v24.4.1
OS & Version: Linux

Expected behaviour

The blockchain doesn't stop block production.

Actual behaviour

The blockchain stops block production when they are active transactions getting submitted to the chain.

Steps to reproduce the behaviour

The genesis.json

{
  "nonce": "0x0",
  "timestamp": "0x58ee40ba",
  "extraData": "0xf865a00000000000000000000000000000000000000000000000000000000000000000f83f9439ba6009d56eacfa74ccc7b3fe7faaabd022a0fd94d84c6f2209d5a16816093ce2ba77bc0f2aef049994e80ffb50e692128e3390eada9172de15ec4e8272c080c0",
  "gasLimit": "0xFFFFFF",
  "gasUsed": "0x0",
  "number": "0x0",
  "difficulty": "0x1",
  "coinbase": "0x0000000000000000000000000000000000000000",
  "mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
  "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000",
  "config": {
    "chainId": 50000,
    "homesteadBlock": 0,
    "eip150Block": 0,
    "eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000",
    "eip155Block": 0,
    "eip158Block": 0,
    "byzantiumBlock": 0,
    "constantinopleBlock": 0,
    "petersburgBlock": 0,
    "istanbulBlock": 0,
    "muirglacierblock": 0,
    "berlinBlock": 0,
    "londonBlock": 0,
    "isQuorum": true,
    "maxCodeSizeConfig": [
      {
        "block": 0,
        "size": 64
      }
    ],
    "txnSizeLimit": 64,
    "qbft": {
      "epoch": 30000,
      "ceil2Nby3Block": 0,
      "testQBFTBlock": 0,
      "blockperiodseconds": 5,
      "emptyblockperiodseconds": 5,
      "requesttimeoutseconds": 10
    }
  },
  "alloc": {
    "0xb2c2069f052696424699ab9d76b53a459dd37bf9": {
      "balance": "1000000000000000000000000000"
    },
    "0x5aa01539d152819423982aa288bc4550cfd7a973": {
      "balance": "1000000000000000000000000000"
    },
    "0xa7a03f8c313deb5467540508b4dece42d852c7e7": {
      "balance": "1000000000000000000000000000"
    },
    "0x260502d45c22494b3d24a04660aacc865c58c22f": {
      "balance": "1000000000000000000000000000"
    }
  }
}

Command to start the network, replace env with appropriate variables.

--nat extip:${IP} \
--networkid ${NETWORKID} \
--unlock $(cat /data/keystore/accountAddress) \
--allow-insecure-unlock \
--password /data/keystore/accountPassword \
--keystore /data/keystore \
--permissioned \
--mine \
--emitcheckpoints \
--miner.threads 1 \
--verbosity ${VERBOSITY} \
--datadir /data \
--syncmode full \
--port ${GETH_PORT} \
--nousb \
--identity ${HOSTNAME}-validator \
--nodiscover \
--metrics \
--istanbul.blockperiod 5 \
--istanbul.requesttimeout 10000 \
--http \
--http.addr 0.0.0.0 \
--http.port ${HTTP_PORT} \
--http.corsdomain '*' \
--http.vhosts '*' \
--http.api admin,db,eth,debug,miner,net,shh,txpool,personal,web3,quorum,qbft \
--ws \
--ws.addr 0.0.0.0 \
--ws.port ${WS_PORT} \
--ws.origins '*' \
--ws.rpcprefix '/' \
--ws.api admin,db,eth,debug,miner,net,shh,txpool,personal,web3,quorum,qbft

• Setup permission v2 system.
• Take at least 1 validator down and make sure that the network is still producing block.
• Sent a transaction every one seconds( or any number less than the emptyblockperiodseconds/blockperiodseconds.

This will cause the blockchain to stop producing block until the transactions stop getting sent, then it will produce a block with all the transactions. Note that this problem will reappear if we start to spam the transactions again.