-
Notifications
You must be signed in to change notification settings - Fork 9.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Specifying a revision for a range request in a transaction may cause data inconsistency #18667
Comments
The simplest solution is that we never remove range request from TXN, see ahrtr@76ac23f .
|
@ArkaSaha30 Thanks for helping out! |
cc @ivanvc as well as you have some context with performance tooling. |
Sorry if I missed this part during the call. Is there an option-2 we've already excluded here, which is to treat working-as-intended range responses (like |
Rough steps as mentioned in the community meeting,
The other solution as mentioned in the meeting is to keep the behaviour unchanged.
But the problem is that this solution will reduce the readability and complicate the etcdserver a lot. So If the performance impact of first solution is minimum, then we should NOT go for this solution 2 |
@ahrtr @serathius say even with the fix to execute the range request on all members, wdyt about the same issue happening when (for lack of a better word) non-deterministic failures lead to one node failing the txn.Range() while the other doesn't? |
It's a generic "problem", not specific to this issue. If a node somehow fails to apply entries due to environment issue (i.e. OOM), then etctserver crashes. When it gets started again, it will re-apply the entries. |
So far the only vector I found for what I was saying above to happen is if the Tx context was cancelled - etcd/server/storage/mvcc/kvstore_txn.go Lines 103 to 104 in c1976a6
But fortunately (or maybe intentionally) the context that gets piped down to it from Apply is a context.TODO() that should never expire: etcd/server/etcdserver/apply/uber_applier.go Line 118 in c1976a6
We probably need to add a big bold note there saying it's risky (wrt data consistency) to change that context to anything finite (I'll send out a PR). But does the overarching risk and #18667 (comment) feel worth discussing to y'all? |
This part makes sense when the failure happens in transaction validation phase (CI isn't incremented). But cases like what you found would be more dangerous when CI is incremented but applied asymmetrically across nodes. Lemme try digging thru the code to see if there are any potential risks there (hopefully none). |
This is a great point about how context can introduce non-determinism into the apply loop, which is highly undesirable. Since the apply loop forms the core of the replicated state machine, any inconsistency caused by context cancellation at different times could lead to significant issues. I think that using context here seems unnecessary and increases the risk of compromising transactionality. From what I understand, it's primarily used for passing call stack metadata like traces and authorization information. We can definitely find alternative ways to achieve this without relying on context. To mitigate the risk, I strongly recommend removing context entirely from the apply loop code. Since the asynchronous code within the loop should not involve any non-deterministic asynchronous calls, we can safely eliminate context. We can then re-implement trace logging without relying on it. |
Agree to this in principle. Clearly, it isn't a minor change. So let's get the low-hanging fix #18674 approved & merged first. Afterwards, we can revisit removing the context without rushing. |
@serathius @ahrtr the idea of removing context sure seems tempting. Should we discuss few things first:
Also happy to discuss this over our next biweekly call, if you prefer. |
Thanks for the comment, my thoughts:
|
Thanks @serathius for laying out that reasoning.
Wdyt about read-only txns (which can go through the apply layer today iiuc)? Those we might want to timeout if taking too long instead of blocking the subsequent applies
+1 to this. The part I wasn't fully sure about is how costly/complex will it be to add that back. Seems not too bad from the size of your PRs :)
Agreed - and the plan laid out above makes full sense to fix this particular issue. Zooming out a bit though, I'm wondering if we need one more invariant besides "txn execution symmetry" - which is "txn side-effect symmetry". FMU the former doesn't necessarily guarantee the latter - potentially because of non-determinism (your context removal change is def going to improve there, but we might have more risks). Since this is a different issue than the original one, I've opened #18679 to discuss more. |
If I remember correctly, Read only TXN don't go through apply. etcd/server/etcdserver/v3_server.go Lines 161 to 192 in 448fb7e
Yea, know the code and re-read it before.
Yes, if I could I would rewrite apply code to ensure clear control of side effects. However, that would require a large investment into testing and don't give a large payout as the apply code is not frequently changed. Now most of the etcd correctness issues come from versions before v3.4. Rewriting it might resurface new bugs, but also introduces a high risk of introducing new issues. That's why I would think robustness testing is better strategically, we need to build a trust into testing that no matter how new is contributor, they can propose improvements to any part of etcd codebase and we can catch up any concurrency bugs. |
Thanks for the code ref, that makes sense. IIUC there can be a case where txn isn't seen as read-only (say there's a write in the if-branch) but the compare operation renders it as read-only (in else-branch). One good thing here is we should be able to know this before executing the txn itself as we know compare result and traverse the txn tree in the check phase. I can make a change to improve this behavior - but a case to keep in mind for timeouts. Wdyt? |
I think #18749 should resolve the issue, but we are missing an e2e regression test. Anyone interested in implementing one? |
#18749 just fixed a separate but related potential issue caught by @shyamjvs . For this issue, it isn't fixed yet. We still need to apply a patch something like ahrtr@76ac23f Pls refer to #18667 (comment). Also @ArkaSaha30 is working on the e2e test. |
/assign @ArkaSaha30 |
@ArkaSaha30 are you still working on this? I know that you are still working on etcd-operator. If you don't have enough bandwidth, would you mind we assign this to other contributors? |
Sure @ahrtr , it is taking more time for me than expected to get familiar with writing the E2E tests. If any of the contributors are interested and familiar with the E2E tests this issue can be assigned to them. Sorry for the inconvenience of blocking this issue. |
What happened?
Specifying a revision for a range request in a transaction may cause data inconsistency. The client may get different values against the same key from different endpoints.
How can we reproduce it?
for i in {1..20}; do ./etcdctl put k$i v$i; done
./etcdctl compact 21
./etcdctl txn --interactive
The client will get a
**etcdserver: mvcc: required revision has been compacted**
error../etcdctl get k2
against different endpoints, you will get different values,Root cause
The root cause is that etcd server removes range requests from the TXN for endpoints the client isn’t connected to.
etcd/server/etcdserver/server.go
Lines 1967 to 1971 in 2c97110
For example, if the client connects to member 1, then etcdserver removes the range request (
get k1 --rev=10
in above example) from the TXN in member 2 and 3. Accordingly, member 1 applies failed due to checkRange's failures, but member 2 & 3 apply the TXN successfully because the range request was removed. Eventually it leads to the situation that different members have different data.etcd/server/etcdserver/txn/txn.go
Lines 431 to 441 in 2c97110
Solution
The simplest solution is we don't remove range requests from TXN for any member. The side effect is that other endpoints (the client isn't connected to) will execute the unnecessary range operations.
To resolve the side effect above, we don't execute the range requests on other endpoints that the client isn't connected to; instead etcdservers only verify them to ensure all endpoints always execute consistent validation.
Workaround
If only one member is inconsistent, just replace it. See this guide. After removing the member, delete its data.
If all members are inconsistent, things get trickier. You'll need to pick one member as the source of truth, force creating a single-member cluster, and then re-add other members (don’t forget to clear their data first).
Impact
All versions (including 3.4.x, 3.5.x and main) are affected.
The text was updated successfully, but these errors were encountered: