How in quorum multi host network we can implement backup and recovery for some node failure scenario in production environment?

[trinayanbhatt]

We are actually using 2 VMs to host our production grade quorum network using docker where we have these VM configuration:

VM1 : 1 member quorum node, 1 rpc node, 1 teserra node and 1 validator node
VM2: 1 member quorum node, 1 teserra node and 1 validator node

Suppose for some reason one of the VM goes down and we require to spin up a new VM with the docker services that were running on the VM that goes down, so what could be the best backup & recovery approach for this?

We don't want to loose any network data and want to have minimal impact on our network because of the new VM.

There are static nodes file and permissions node file that are required for the Quorum network setup so VM's IP change will surely impact the nodes, so how can we manage that scenario?