Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

etcd_cluster_version metric is missing #18805

Open
4 tasks done
ugur99 opened this issue Oct 29, 2024 · 8 comments
Open
4 tasks done

etcd_cluster_version metric is missing #18805

ugur99 opened this issue Oct 29, 2024 · 8 comments

Comments

@ugur99
Copy link

ugur99 commented Oct 29, 2024

Bug report criteria

What happened?

new etcd members do not generate etcd_cluster_version metrics; this is the only metrics being missing as of we've noticed. According to the etcd logs instance knows the cluster version; but this metrics is still missing.

What did you expect to happen?

etcd_cluster_version should be available

How can we reproduce it (as minimally and precisely as possible)?

Deploy a cluster using kubespray: deploy etcds as a systemd service on each controlplane node. Then remove the controlplanes (and etcds) one by one from the cluster by reprovisioning the new controlplanes from the ignition template (we have baked all the controlplane and etcd setup/join processes into the ignition template). We have exactly the same configuration, same etcd member names; and followed similar steps for the etcd member join/remove process.

Anything else we need to know?

No response

Etcd version (please run commands below)

$ etcd --version
etcd Version: 3.5.12
Git SHA: e7b3bb6cc
Go Version: go1.20.13
Go OS/Arch: linux/amd64
$ etcdctl version
etcdctl version: 3.5.0
API version: 3.5

Etcd configuration (command line flags or environment variables)

Environment file for etcd v3.5.12

ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://12.60.87.25:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://12.60.87.25:2380
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_METRICS=basic
ETCD_LISTEN_METRICS_URLS=http://12.60.87.25:2381,http://127.0.0.1:2381
ETCD_LISTEN_CLIENT_URLS=https://12.60.87.25:2379,https://127.0.0.1:2379
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd
ETCD_LISTEN_PEER_URLS=https://12.60.87.25:2380
ETCD_NAME=etcd1
ETCD_PROXY=off
ETCD_INITIAL_CLUSTER=etcd1=https://12.60.87.25:2380,etcd2=https://12.60.87.90:2380,etcd3=https://12.60.87.152:2380

ETCD_AUTO_COMPACTION_RETENTION=8
ETCD_SNAPSHOT_COUNT=10000
ETCD_QUOTA_BACKEND_BYTES=6147483648

Flannel need etcd v2 API

ETCD_ENABLE_V2=true

TLS settings

ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-dev-k8s-control1.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-dev-k8s-control1-key.pem
ETCD_CLIENT_CERT_AUTH=true

ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-dev-k8s-control1.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-dev-k8s-control1-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=True

CLI settings

ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
ETCDCTL_CACERT=/etc/ssl/etcd/ssl/ca.pem
ETCDCTL_KEY=/etc/ssl/etcd/ssl/admin-dev-k8s-control1-key.pem
ETCDCTL_CERT=/etc/ssl/etcd/ssl/admin-dev-k8s-control1.pem

ETCD 3.5.x issue

https://groups.google.com/a/kubernetes.io/g/dev/c/B7gJs88XtQc/m/rSgNOzV2BwAJ?utm_medium=email&utm_source=footer

ETCD_EXPERIMENTAL_INITIAL_CORRUPT_CHECK=True

ETCD tracing configuration

ETCD_EXPERIMENTAL_ENABLE_DISTRIBUTED_TRACING=true
ETCD_EXPERIMENTAL_DISTRIBUTED_TRACING_SAMPLING_RATE=100
ETCD_EXPERIMENTAL_DISTRIBUTED_TRACING_ADDRESS=0.0.0.0:4317
ETCD_EXPERIMENTAL_DISTRIBUTED_TRACING_SERVICE_NAME=etcd
ETCD_EXPERIMENTAL_DISTRIBUTED_TRACING_INSTANCE_ID=etcd1

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

$ etcdctl member list -w table
+------------------+---------+-------+--------------------------+--------------------------+------------+
|        ID        | STATUS  | NAME  |        PEER ADDRS        |       CLIENT ADDRS       | IS LEARNER |
+------------------+---------+-------+--------------------------+--------------------------+------------+
| 335c9823932a1698 | started | etcd1 |  https://12.60.87.25:2380 |  https://12.60.87.25:2379 |      false |
| 5e66e8bfa5f5d5fb | started | etcd2 |  https://12.60.87.90:2380 |  https://12.60.87.90:2379 |      false |
| f736583c8dc56a31 | started | etcd3 | https://12.60.87.152:2380 | https://12.60.87.152:2379 |      false |
+------------------+---------+-------+--------------------------+--------------------------+------------+

$ etcdctl --endpoints=<member list> endpoint status -w table
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|         ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|  https://12.60.87.25:2379 | 335c9823932a1698 |  3.5.12 |  1.3 GB |      true |      false |       183 | 1580003433 |         1580003433 |        |
|  https://12.60.87.90:2379 | 5e66e8bfa5f5d5fb |  3.5.12 |  1.3 GB |     false |      false |       183 | 1580003434 |         1580003434 |        |
| https://12.60.87.152:2379 | f736583c8dc56a31 |  3.5.12 |  1.3 GB |     false |      false |       183 | 1580003434 |         1580003434 |        |
+--------------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Relevant log output

{"level":"info","ts":"2024-10-28T13:02:57.044182Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"335c9823932a1698","to":"5e66e8bfa5f5d5fb","stream-type":"stream Message"}	
    {"level":"info","ts":"2024-10-28T13:02:57.044216Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.036097Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"335c9823932a1698","to":"5e66e8bfa5f5d5fb","stream-type":"stream MsgApp v2"}	
    {"level":"info","ts":"2024-10-28T13:02:57.036152Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.027078Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.027352Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.023048Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.022501Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"335c9823932a1698","to":"f736583c8dc56a31","stream-type":"stream Message"}	
    {"level":"info","ts":"2024-10-28T13:02:57.02253Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.021749Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"335c9823932a1698","to":"f736583c8dc56a31","stream-type":"stream MsgApp v2"}	
    {"level":"info","ts":"2024-10-28T13:02:57.021771Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.021782Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.021889Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.019182Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.012975Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.01299Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31","remote-peer-urls":["https://12.60.87.152:2380"]}	
    {"level":"info","ts":"2024-10-28T13:02:57.013073Z","caller":"etcdserver/server.go:1361","msg":"added peers from new cluster configuration"}	
    {"level":"info","ts":"2024-10-28T13:02:57.013089Z","caller":"etcdserver/server.go:1240","msg":"applied snapshot","current-snapshot-index":1573588337,"current-applied-index":1573588337,"incoming-leader-snapshot-index":1573588337,"incoming-leader-snapshot-term":182}	
    {"level":"info","ts":"2024-10-28T13:02:57.012781Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.012958Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.012966Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.012056Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.012082Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb","remote-peer-urls":["https://12.60.87.90:2380"]}	
    {"level":"info","ts":"2024-10-28T13:02:57.012097Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.012078Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.0121Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.012119Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.012311Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.012711Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"warn","ts":"2024-10-28T13:02:57.011045Z","caller":"rafthttp/stream.go:286","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.011084Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.011125Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"warn","ts":"2024-10-28T13:02:57.01124Z","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb","error":"context canceled"}	
    {"level":"warn","ts":"2024-10-28T13:02:57.011264Z","caller":"rafthttp/peer_status.go:66","msg":"peer became inactive (message send to peer failed)","peer-id":"5e66e8bfa5f5d5fb","error":"failed to read 5e66e8bfa5f5d5fb on stream MsgApp v2 (context canceled)"}	
    {"level":"info","ts":"2024-10-28T13:02:57.011285Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"warn","ts":"2024-10-28T13:02:57.011347Z","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb","error":"context canceled"}	
    {"level":"info","ts":"2024-10-28T13:02:57.011361Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.011375Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.011389Z","caller":"rafthttp/transport.go:355","msg":"removed remote peer","local-member-id":"335c9823932a1698","removed-remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.011398Z","caller":"etcdserver/server.go:1351","msg":"removed old peers from network"}	
    {"level":"info","ts":"2024-10-28T13:02:57.011409Z","caller":"etcdserver/server.go:1352","msg":"adding peers from new cluster configuration"}	
    {"level":"info","ts":"2024-10-28T13:02:57.011436Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.011456Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.011615Z","caller":"etcdserver/server.go:1302","msg":"closed old backend file"}	
    {"level":"info","ts":"2024-10-28T13:02:57.011706Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.01178Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010068Z","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
	
    {"level":"info","ts":"2024-10-28T13:02:57.009973Z","caller":"etcdserver/server.go:1341","msg":"restoring cluster configuration"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010031Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010043Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"377f8845ceff7871","local-member-id":"335c9823932a1698","recovered-remote-peer-id":"5e66e8bfa5f5d5fb","recovered-remote-peer-urls":["https://12.60.87.90:2380"]}	
    {"level":"info","ts":"2024-10-28T13:02:57.010054Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"377f8845ceff7871","local-member-id":"335c9823932a1698","recovered-remote-peer-id":"f736583c8dc56a31","recovered-remote-peer-urls":["https://12.60.87.152:2380"]}	
    {"level":"info","ts":"2024-10-28T13:02:57.010061Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"377f8845ceff7871","local-member-id":"335c9823932a1698","recovered-remote-peer-id":"335c9823932a1698","recovered-remote-peer-urls":["https://12.60.87.25:2380"]}	
    {"level":"info","ts":"2024-10-28T13:02:57.010075Z","caller":"etcdserver/server.go:1345","msg":"restored cluster configuration"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010084Z","caller":"etcdserver/server.go:1346","msg":"removing old peers from network"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010092Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"warn","ts":"2024-10-28T13:02:57.010225Z","caller":"rafthttp/stream.go:286","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010248Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"warn","ts":"2024-10-28T13:02:57.010381Z","caller":"rafthttp/stream.go:286","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010403Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream Message","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010449Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"warn","ts":"2024-10-28T13:02:57.01056Z","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31","error":"context canceled"}	
    {"level":"warn","ts":"2024-10-28T13:02:57.010608Z","caller":"rafthttp/peer_status.go:66","msg":"peer became inactive (message send to peer failed)","peer-id":"f736583c8dc56a31","error":"failed to read f736583c8dc56a31 on stream MsgApp v2 (context canceled)"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010625Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"warn","ts":"2024-10-28T13:02:57.010685Z","caller":"rafthttp/stream.go:421","msg":"lost TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31","error":"context canceled"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010706Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010719Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010733Z","caller":"rafthttp/transport.go:355","msg":"removed remote peer","local-member-id":"335c9823932a1698","removed-remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010742Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"warn","ts":"2024-10-28T13:02:57.010918Z","caller":"rafthttp/stream.go:286","msg":"closed TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.010935Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:57.009953Z","caller":"etcdserver/server.go:1337","msg":"restored v2 store"}	
    {"level":"info","ts":"2024-10-28T13:02:57.009041Z","caller":"etcdserver/server.go:1328","msg":"restoring v2 store"}	
    {"level":"info","ts":"2024-10-28T13:02:57.009033Z","caller":"etcdserver/server.go:1325","msg":"restored auth store"}	
    {"level":"info","ts":"2024-10-28T13:02:57.00902Z","caller":"etcdserver/server.go:1321","msg":"restoring auth store"}	
    {"level":"info","ts":"2024-10-28T13:02:57.009002Z","caller":"etcdserver/server.go:1318","msg":"restored alarm store"}	
    {"level":"info","ts":"2024-10-28T13:02:57.00849Z","caller":"etcdserver/server.go:1300","msg":"closing old backend file"}	
    {"level":"info","ts":"2024-10-28T13:02:57.008435Z","caller":"etcdserver/server.go:1312","msg":"restoring alarm store"}	
    {"level":"info","ts":"2024-10-28T13:02:57.008419Z","caller":"etcdserver/server.go:1292","msg":"restored mvcc store","consistent-index":1573588337}	
    {"level":"info","ts":"2024-10-28T13:02:57.008341Z","caller":"mvcc/kvstore.go:407","msg":"kvstore restored","current-rev":1490850186}	
    {"level":"info","ts":"2024-10-28T13:02:56.47553Z","caller":"mvcc/kvstore.go:341","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":1490814913}	
    {"level":"info","ts":"2024-10-28T13:02:56.4755Z","caller":"etcdserver/server.go:1285","msg":"restoring mvcc store"}	
    {"level":"info","ts":"2024-10-28T13:02:56.475488Z","caller":"etcdserver/server.go:1282","msg":"restored lease store"}	
    {"level":"info","ts":"2024-10-28T13:02:56.474916Z","caller":"etcdserver/server.go:1278","msg":"restoring lease store"}	
    {"level":"info","ts":"2024-10-28T13:02:56.474867Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"388.662852ms"}	
    {"level":"info","ts":"2024-10-28T13:02:56.086084Z","caller":"etcdserver/raft.go:277","msg":"applied incoming Raft snapshot","snapshot-index":1573588337}	
    {"level":"info","ts":"2024-10-28T13:02:56.085512Z","caller":"etcdserver/server.go:1232","msg":"applying snapshot","current-snapshot-index":0,"current-applied-index":0,"incoming-leader-snapshot-index":1573588337,"incoming-leader-snapshot-term":182}	
    {"level":"info","ts":"2024-10-28T13:02:56.085425Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"335c9823932a1698 [commit: 1573588337] restored snapshot [index: 1573588337, term: 182]"}	
    {"level":"info","ts":"2024-10-28T13:02:56.085418Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"335c9823932a1698 [commit: 1573588337, lastindex: 1573588337, lastterm: 182] restored snapshot [index: 1573588337, term: 182]"}	
    {"level":"info","ts":"2024-10-28T13:02:56.085399Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"335c9823932a1698 switched to configuration voters=(3701000272351991448 6802380197010462203 17813522393210382897)"}	
    {"level":"info","ts":"2024-10-28T13:02:56.085328Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 1573588337, term: 182]"}	
    {"level":"info","ts":"2024-10-28T13:02:56.085172Z","caller":"rafthttp/http.go:286","msg":"received and saved database snapshot","local-member-id":"335c9823932a1698","remote-snapshot-sender-id":"5e66e8bfa5f5d5fb","incoming-snapshot-index":1573588337,"incoming-snapshot-size-bytes":1925492736,"incoming-snapshot-size":"1.9 GB","download-took":"5.781117634s"}	
    {"level":"info","ts":"2024-10-28T13:02:56.085115Z","caller":"snap/db.go:66","msg":"saved database snapshot to disk","path":"/var/lib/etcd/member/snap/000000005dcb0d71.snap.db","bytes":1925492736,"size":"1.9 GB"}	
    2024/10/28 13:02:55 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "0.0.0.0:4317", ServerName: "0.0.0.0:4317", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 0.0.0.0:4317: connect: connection refused"	
    2024/10/28 13:02:52 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "0.0.0.0:4317", ServerName: "0.0.0.0:4317", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 0.0.0.0:4317: connect: connection refused"	
    2024/10/28 13:02:51 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "0.0.0.0:4317", ServerName: "0.0.0.0:4317", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 0.0.0.0:4317: connect: connection refused"	
    {"level":"info","ts":"2024-10-28T13:02:50.332892Z","caller":"etcdserver/server.go:783","msg":"initialized peer connections; fast-forwarding election ticks","local-member-id":"335c9823932a1698","forward-ticks":18,"forward-duration":"4.5s","election-ticks":20,"election-timeout":"5s","active-remote-members":2}	
    {"level":"info","ts":"2024-10-28T13:02:50.305519Z","caller":"rafthttp/http.go:257","msg":"receiving database snapshot","local-member-id":"335c9823932a1698","remote-snapshot-sender-id":"5e66e8bfa5f5d5fb","incoming-snapshot-index":1573588337,"incoming-snapshot-message-size-bytes":37480,"incoming-snapshot-message-size":"38 kB"}	
    {"level":"info","ts":"2024-10-28T13:02:50.301065Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.301035Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"335c9823932a1698","to":"f736583c8dc56a31","stream-type":"stream Message"}	
    {"level":"info","ts":"2024-10-28T13:02:50.297754Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.297722Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"335c9823932a1698","to":"f736583c8dc56a31","stream-type":"stream MsgApp v2"}	
    {"level":"info","ts":"2024-10-28T13:02:50.297371Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.296737Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.29671Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"335c9823932a1698","to":"5e66e8bfa5f5d5fb","stream-type":"stream MsgApp v2"}	
    {"level":"info","ts":"2024-10-28T13:02:50.295942Z","caller":"rafthttp/stream.go:274","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.295908Z","caller":"rafthttp/stream.go:249","msg":"set message encoder","from":"335c9823932a1698","to":"5e66e8bfa5f5d5fb","stream-type":"stream Message"}	
    {"level":"info","ts":"2024-10-28T13:02:50.295405Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.295373Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.29531Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.292626Z","caller":"rafthttp/stream.go:412","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.292571Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.286034Z","caller":"embed/etcd.go:857","msg":"serving metrics","address":"http://127.0.0.1:2381"}	
    {"level":"info","ts":"2024-10-28T13:02:50.286011Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 335c9823932a1698 elected leader 5e66e8bfa5f5d5fb at term 182"}	
    {"level":"info","ts":"2024-10-28T13:02:50.286008Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"335c9823932a1698","initial-advertise-peer-urls":["https://12.60.87.25:2380"],"listen-peer-urls":["https://12.60.87.25:2380"],"advertise-client-urls":["https://12.60.87.25:2379"],"listen-client-urls":["https://12.60.87.25:2379","https://127.0.0.1:2379"],"listen-metrics-urls":["http://12.60.87.25:2381","http://127.0.0.1:2381"]}	
    {"level":"info","ts":"2024-10-28T13:02:50.285991Z","caller":"embed/etcd.go:857","msg":"serving metrics","address":"http://12.60.87.25:2381"}	
    {"level":"info","ts":"2024-10-28T13:02:50.285995Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"335c9823932a1698 became follower at term 182"}	
    {"level":"info","ts":"2024-10-28T13:02:50.285972Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"335c9823932a1698 [term: 0] received a MsgHeartbeat message with higher term from 5e66e8bfa5f5d5fb [term: 182]"}	
    {"level":"info","ts":"2024-10-28T13:02:50.28574Z","caller":"embed/etcd.go:569","msg":"cmux::serve","address":"12.60.87.25:2380"}	
    {"level":"info","ts":"2024-10-28T13:02:50.285715Z","caller":"embed/etcd.go:597","msg":"serving peer traffic","address":"12.60.87.25:2380"}	
    {"level":"warn","ts":"2024-10-28T13:02:50.285646Z","caller":"embed/etcd.go:739","msg":"Flag `enable-v2` is deprecated and will get removed in etcd 3.6."}	
    {"level":"info","ts":"2024-10-28T13:02:50.285623Z","caller":"embed/etcd.go:726","msg":"starting with client TLS","tls-info":"cert = /etc/ssl/etcd/ssl/member-dev-k8s-control1.pem, key = /etc/ssl/etcd/ssl/member-dev-k8s-control1-key.pem, client-cert=, client-key=, trusted-ca = /etc/ssl/etcd/ssl/ca.pem, client-cert-auth = true, crl-file = ","cipher-suites":[]}	
    {"level":"info","ts":"2024-10-28T13:02:50.282909Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/var/lib/etcd/member/wal","suffix":"wal","max":5,"interval":"30s"}	
    {"level":"info","ts":"2024-10-28T13:02:50.282896Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap","max":5,"interval":"30s"}	
    {"level":"info","ts":"2024-10-28T13:02:50.282815Z","caller":"fileutil/purge.go:50","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap.db","max":5,"interval":"30s"}	
    {"level":"info","ts":"2024-10-28T13:02:50.282825Z","caller":"etcdserver/server.go:760","msg":"starting initial election tick advance","election-ticks":20}	
    {"level":"info","ts":"2024-10-28T13:02:50.282732Z","caller":"etcdserver/server.go:860","msg":"starting etcd server","local-member-id":"335c9823932a1698","local-server-version":"3.5.12","cluster-version":"to_be_decided"}	
    {"level":"info","ts":"2024-10-28T13:02:50.282637Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.282683Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31","remote-peer-urls":["https://12.60.87.152:2380"]}	
    {"level":"info","ts":"2024-10-28T13:02:50.282661Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.282634Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.282473Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.281891Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.281518Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.28158Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.281564Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.281531Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.28154Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb","remote-peer-urls":["https://12.60.87.90:2380"]}	
    {"level":"info","ts":"2024-10-28T13:02:50.2815Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.28144Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.280026Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.279737Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.279726Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.279715Z","caller":"rafthttp/transport.go:286","msg":"added new remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31","remote-peer-urls":["https://12.60.87.152:2380"]}	
    {"level":"info","ts":"2024-10-28T13:02:50.279707Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"f736583c8dc56a31"}	
    {"level":"info","ts":"2024-10-28T13:02:50.279681Z","caller":"rafthttp/transport.go:286","msg":"added new remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb","remote-peer-urls":["https://12.60.87.90:2380"]}	
    {"level":"info","ts":"2024-10-28T13:02:50.279659Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"335c9823932a1698","remote-peer-id":"5e66e8bfa5f5d5fb"}	
    {"level":"info","ts":"2024-10-28T13:02:50.278735Z","caller":"etcdserver/quota.go:117","msg":"enabled backend quota","quota-name":"v3-applier","quota-size-bytes":6147483648,"quota-size":"6.1 GB"}	
    {"level":"info","ts":"2024-10-28T13:02:50.278557Z","caller":"mvcc/kvstore.go:407","msg":"kvstore restored","current-rev":1}	
    {"level":"warn","ts":"2024-10-28T13:02:50.278083Z","caller":"auth/store.go:1241","msg":"simple token is not cryptographically signed"}	
    {"level":"info","ts":"2024-10-28T13:02:50.27781Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 335c9823932a1698 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]"}	
    {"level":"info","ts":"2024-10-28T13:02:50.277797Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"335c9823932a1698 became follower at term 0"}	
    {"level":"info","ts":"2024-10-28T13:02:50.277779Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"335c9823932a1698 switched to configuration voters=()"}	
    {"level":"info","ts":"2024-10-28T13:02:50.277706Z","caller":"etcdserver/raft.go:495","msg":"starting local member","local-member-id":"335c9823932a1698","cluster-id":"377f8845ceff7871"}	
    {"level":"info","ts":"2024-10-28T13:02:50.264092Z","caller":"netutil/netutil.go:112","msg":"resolved URL Host","url":"https://12.60.87.25:2380","host":"12.60.87.25:2380","resolved-addr":"12.60.87.25:2380"}	
    {"level":"info","ts":"2024-10-28T13:02:50.264082Z","caller":"netutil/netutil.go:112","msg":"resolved URL Host","url":"https://12.60.87.152:2380","host":"12.60.87.152:2380","resolved-addr":"12.60.87.152:2380"}	
    {"level":"info","ts":"2024-10-28T13:02:50.264067Z","caller":"netutil/netutil.go:112","msg":"resolved URL Host","url":"https://12.60.87.152:2380","host":"12.60.87.152:2380","resolved-addr":"12.60.87.152:2380"}	
    {"level":"info","ts":"2024-10-28T13:02:50.264057Z","caller":"netutil/netutil.go:112","msg":"resolved URL Host","url":"https://12.60.87.90:2380","host":"12.60.87.90:2380","resolved-addr":"12.60.87.90:2380"}	
    {"level":"info","ts":"2024-10-28T13:02:50.264037Z","caller":"netutil/netutil.go:112","msg":"resolved URL Host","url":"https://12.60.87.25:2380","host":"12.60.87.25:2380","resolved-addr":"12.60.87.25:2380"}	
    {"level":"info","ts":"2024-10-28T13:02:50.263995Z","caller":"netutil/netutil.go:112","msg":"resolved URL Host","url":"https://12.60.87.90:2380","host":"12.60.87.90:2380","resolved-addr":"12.60.87.90:2380"}	
    {"level":"info","ts":"2024-10-28T13:02:50.254535Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"2.800391ms"}	
    2024/10/28 13:02:50 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "0.0.0.0:4317", ServerName: "0.0.0.0:4317", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 0.0.0.0:4317: connect: connection refused"	
    {"level":"info","ts":"2024-10-28T13:02:50.251249Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.12","git-sha":"e7b3bb6cc","go-version":"go1.20.13","go-os":"linux","go-arch":"amd64","max-cpu-set":16,"max-cpu-available":16,"member-initialized":false,"name":"etcd1","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"250ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://12.60.87.25:2380"],"listen-peer-urls":["https://12.60.87.25:2380"],"advertise-client-urls":["https://12.60.87.25:2379"],"listen-client-urls":["https://12.60.87.25:2379","https://127.0.0.1:2379"],"listen-metrics-urls":["http://12.60.87.25:2381","http://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"etcd1=https://12.60.87.25:2380,etcd2=https://12.60.87.90:2380,etcd3=https://12.60.87.152:2380","initial-cluster-state":"existing","initial-cluster-token":"k8s_etcd","quota-backend-bytes":6147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"8h0m0s","auto-compaction-interval":"8h0m0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}	
    {"level":"info","ts":"2024-10-28T13:02:50.251209Z","caller":"embed/etcd.go:240","msg":"distributed tracing setup enabled"}	
    {"level":"info","ts":"2024-10-28T13:02:50.250809Z","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["https://12.60.87.25:2379","https://127.0.0.1:2379"]}	
    {"level":"info","ts":"2024-10-28T13:02:50.250171Z","caller":"embed/etcd.go:494","msg":"starting with peer TLS","tls-info":"cert = /etc/ssl/etcd/ssl/member-dev-k8s-control1.pem, key = /etc/ssl/etcd/ssl/member-dev-k8s-control1-key.pem, client-cert=, client-key=, trusted-ca = /etc/ssl/etcd/ssl/ca.pem, client-cert-auth = true, crl-file = ","cipher-suites":[]}	
    {"level":"info","ts":"2024-10-28T13:02:50.250148Z","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["https://12.60.87.25:2380"]}	
    {"level":"warn","ts":"2024-10-28T13:02:50.250134Z","caller":"embed/config.go:679","msg":"Running http and grpc server on single port. This is not recommended for production."}	
    {"level":"info","ts":"2024-10-28T13:02:50.250118Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"}	
    {"level":"info","ts":"2024-10-28T13:02:50.250048Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["/opt/bin/etcd"]}	
    {"level":"warn","ts":"2024-10-28T13:02:50.250008Z","caller":"embed/config.go:679","msg":"Running http and grpc server on single port. This is not recommended for production."}	
    {"level":"info","ts":"2024-10-28T13:02:50.249701Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CERT_FILE","variable-value":"/etc/ssl/etcd/ssl/member-dev-k8s-control1.pem"}	
    {"level":"info","ts":"2024-10-28T13:02:50.249684Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"8"}	
    {"level":"info","ts":"2024-10-28T13:02:50.249605Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"https://12.60.87.25:2379"}	
    {"level":"fatal","ts":"2024-10-28T13:02:40.032197Z","caller":"etcdmain/etcd.go:204","msg":"discovery failed","error":"error validating peerURLs {ClusterID:377f8845ceff7871 Members:[&{ID:5e66e8bfa5f5d5fb RaftAttributes:{PeerURLs:[https://12.60.87.90:2380] IsLearner:false} Attributes:{Name:etcd2 ClientURLs:[https://12.60.87.90:2379]}} &{ID:f736583c8dc56a31 RaftAttributes:{PeerURLs:[https://12.60.87.152:2380] IsLearner:false} Attributes:{Name:etcd3 ClientURLs:[https://12.60.87.152:2379]}}] RemovedMemberIDs:[]}: member count is unequal","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:31\nruntime.main\n\truntime/proc.go:250"}	
    {"level":"info","ts":"2024-10-28T13:02:40.032177Z","caller":"embed/etcd.go:377","msg":"closed etcd server","name":"etcd1","data-dir":"/var/lib/etcd","advertise-peer-urls":["https://12.60.87.25:2380"],"advertise-client-urls":["https://12.60.87.25:2379"]}	
    {"level":"info","ts":"2024-10-28T13:02:40.031953Z","caller":"embed/etcd.go:375","msg":"closing etcd server","name":"etcd1","data-dir":"/var/lib/etcd","advertise-peer-urls":["https://12.60.87.25:2380"],"advertise-client-urls":["https://12.60.87.25:2379"]}	
    {"level":"info","ts":"2024-10-28T13:02:40.017112Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"8.017932ms"}	
    2024/10/28 13:02:40 WARNING: [core] [Channel #1 SubChannel #2] grpc: addrConn.createTransport failed to connect to {Addr: "0.0.0.0:4317", ServerName: "0.0.0.0:4317", }. Err: connection error: desc = "transport: Error while dialing: dial tcp 0.0.0.0:4317: connect: connection refused"	
    {"level":"info","ts":"2024-10-28T13:02:40.00888Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.12","git-sha":"e7b3bb6cc","go-version":"go1.20.13","go-os":"linux","go-arch":"amd64","max-cpu-set":16,"max-cpu-available":16,"member-initialized":false,"name":"etcd1","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"250ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":5,"max-snapshots":5,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://12.60.87.25:2380"],"listen-peer-urls":["https://12.60.87.25:2380"],"advertise-client-urls":["https://12.60.87.25:2379"],"listen-client-urls":["https://12.60.87.25:2379","https://127.0.0.1:2379"],"listen-metrics-urls":["http://12.60.87.25:2381","http://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"etcd1=https://12.60.87.25:2380,etcd2=https://12.60.87.90:2380,etcd3=https://12.60.87.152:2380","initial-cluster-state":"existing","initial-cluster-token":"k8s_etcd","quota-backend-bytes":6147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"periodic","auto-compaction-retention":"8h0m0s","auto-compaction-interval":"8h0m0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}	
    {"level":"info","ts":"2024-10-28T13:02:40.008828Z","caller":"embed/etcd.go:240","msg":"distributed tracing setup enabled"}	
    {"level":"info","ts":"2024-10-28T13:02:40.008333Z","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["https://12.60.87.25:2379","https://127.0.0.1:2379"]}	
    {"level":"info","ts":"2024-10-28T13:02:40.007637Z","caller":"embed/etcd.go:494","msg":"starting with peer TLS","tls-info":"cert = /etc/ssl/etcd/ssl/member-dev-k8s-control1.pem, key = /etc/ssl/etcd/ssl/member-dev-k8s-control1-key.pem, client-cert=, client-key=, trusted-ca = /etc/ssl/etcd/ssl/ca.pem, client-cert-auth = true, crl-file = ","cipher-suites":[]}	
    {"level":"info","ts":"2024-10-28T13:02:40.007611Z","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["https://12.60.87.25:2380"]}	
    {"level":"warn","ts":"2024-10-28T13:02:40.007604Z","caller":"embed/config.go:679","msg":"Running http and grpc server on single port. This is not recommended for production."}	
    {"level":"info","ts":"2024-10-28T13:02:40.007588Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"}	
    {"level":"info","ts":"2024-10-28T13:02:40.00752Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["/opt/bin/etcd"]}	
    {"level":"warn","ts":"2024-10-28T13:02:40.007489Z","caller":"embed/config.go:679","msg":"Running http and grpc server on single port. This is not recommended for production."}	
    {"level":"info","ts":"2024-10-28T13:02:40.007409Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_TRUSTED_CA_FILE","variable-value":"/etc/ssl/etcd/ssl/ca.pem"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007401Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_SNAPSHOT_COUNT","variable-value":"10000"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007391Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"6147483648"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007383Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PROXY","variable-value":"off"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007376Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_TRUSTED_CA_FILE","variable-value":"/etc/ssl/etcd/ssl/ca.pem"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007368Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_KEY_FILE","variable-value":"/etc/ssl/etcd/ssl/member-dev-k8s-control1-key.pem"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007358Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_CLIENT_CERT_AUTH","variable-value":"True"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007351Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_PEER_CERT_FILE","variable-value":"/etc/ssl/etcd/ssl/member-dev-k8s-control1.pem"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007343Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_NAME","variable-value":"etcd1"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007337Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_METRICS","variable-value":"basic"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007327Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"https://12.60.87.25:2380"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007319Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_METRICS_URLS","variable-value":"http://12.60.87.25:2381,http://127.0.0.1:2381"}	
    {"level":"info","ts":"2024-10-28T13:02:40.00731Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"https://12.60.87.25:2379,https://127.0.0.1:2379"}	
    {"level":"info","ts":"2024-10-28T13:02:40.0073Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_KEY_FILE","variable-value":"/etc/ssl/etcd/ssl/member-dev-k8s-control1-key.pem"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007293Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"k8s_etcd"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007287Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"existing"}	
    {"level":"info","ts":"2024-10-28T13:02:40.00728Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"etcd1=https://12.60.87.25:2380,etcd2=https://12.60.87.90:2380,etcd3=https://12.60.87.152:2380"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007274Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"https://12.60.87.25:2380"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007261Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_HEARTBEAT_INTERVAL","variable-value":"250"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007248Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_EXPERIMENTAL_INITIAL_CORRUPT_CHECK","variable-value":"True"}	
    {"level":"info","ts":"2024-10-28T13:02:40.00724Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_EXPERIMENTAL_ENABLE_DISTRIBUTED_TRACING","variable-value":"true"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007233Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_EXPERIMENTAL_DISTRIBUTED_TRACING_SERVICE_NAME","variable-value":"etcd"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007226Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_EXPERIMENTAL_DISTRIBUTED_TRACING_SAMPLING_RATE","variable-value":"100"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007219Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_EXPERIMENTAL_DISTRIBUTED_TRACING_INSTANCE_ID","variable-value":"etcd1"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007212Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_EXPERIMENTAL_DISTRIBUTED_TRACING_ADDRESS","variable-value":"0.0.0.0:4317"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007202Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ENABLE_V2","variable-value":"true"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007194Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ELECTION_TIMEOUT","variable-value":"5000"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007186Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_DATA_DIR","variable-value":"/var/lib/etcd"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007176Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"true"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007167Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_CERT_FILE","variable-value":"/etc/ssl/etcd/ssl/member-dev-k8s-control1.pem"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007157Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"8"}	
    {"level":"info","ts":"2024-10-28T13:02:40.007056Z","caller":"flags/flag.go:113","msg":"recognized and used environment variable","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"https://12.60.87.25:2379"}
@ugur99
Copy link
Author

ugur99 commented Oct 29, 2024

and here is the all available metrics

# HELP etcd_debugging_auth_revision The current revision of auth store.
# TYPE etcd_debugging_auth_revision gauge
etcd_debugging_auth_revision 1
# HELP etcd_debugging_disk_backend_commit_rebalance_duration_seconds The latency distributions of commit.rebalance called by bboltdb backend.
# TYPE etcd_debugging_disk_backend_commit_rebalance_duration_seconds histogram
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="0.001"} 7.028939e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="0.002"} 7.03076e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="0.004"} 7.030914e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="0.008"} 7.030993e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="0.016"} 7.031001e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="0.032"} 7.031003e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="0.064"} 7.031003e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="0.128"} 7.031003e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="0.256"} 7.031003e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="0.512"} 7.031003e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="1.024"} 7.031003e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="2.048"} 7.031003e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="4.096"} 7.031003e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="8.192"} 7.031003e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket{le="+Inf"} 7.031003e+06
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_sum 42.92015874600005
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_count 7.031003e+06
# HELP etcd_debugging_disk_backend_commit_spill_duration_seconds The latency distributions of commit.spill called by bboltdb backend.
# TYPE etcd_debugging_disk_backend_commit_spill_duration_seconds histogram
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="0.001"} 7.016277e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="0.002"} 7.027236e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="0.004"} 7.029733e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="0.008"} 7.030434e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="0.016"} 7.030858e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="0.032"} 7.030993e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="0.064"} 7.031002e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="0.128"} 7.031003e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="0.256"} 7.031003e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="0.512"} 7.031003e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="1.024"} 7.031003e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="2.048"} 7.031003e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="4.096"} 7.031003e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="8.192"} 7.031003e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket{le="+Inf"} 7.031003e+06
etcd_debugging_disk_backend_commit_spill_duration_seconds_sum 584.4020461420602
etcd_debugging_disk_backend_commit_spill_duration_seconds_count 7.031003e+06
# HELP etcd_debugging_disk_backend_commit_write_duration_seconds The latency distributions of commit.write called by bboltdb backend.
# TYPE etcd_debugging_disk_backend_commit_write_duration_seconds histogram
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="0.001"} 6.769404e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="0.002"} 6.89864e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="0.004"} 6.972441e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="0.008"} 7.014485e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="0.016"} 7.022377e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="0.032"} 7.024067e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="0.064"} 7.026698e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="0.128"} 7.030607e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="0.256"} 7.030998e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="0.512"} 7.031003e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="1.024"} 7.031003e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="2.048"} 7.031003e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="4.096"} 7.031003e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="8.192"} 7.031003e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket{le="+Inf"} 7.031003e+06
etcd_debugging_disk_backend_commit_write_duration_seconds_sum 3523.4960634331537
etcd_debugging_disk_backend_commit_write_duration_seconds_count 7.031003e+06
# HELP etcd_debugging_lease_granted_total The total number of granted leases.
# TYPE etcd_debugging_lease_granted_total counter
etcd_debugging_lease_granted_total 242643
# HELP etcd_debugging_lease_renewed_total The number of renewed leases seen by the leader.
# TYPE etcd_debugging_lease_renewed_total counter
etcd_debugging_lease_renewed_total 0
# HELP etcd_debugging_lease_revoked_total The total number of revoked leases.
# TYPE etcd_debugging_lease_revoked_total counter
etcd_debugging_lease_revoked_total 242591
# HELP etcd_debugging_lease_ttl_total Bucketed histogram of lease TTLs.
# TYPE etcd_debugging_lease_ttl_total histogram
etcd_debugging_lease_ttl_total_bucket{le="1"} 0
etcd_debugging_lease_ttl_total_bucket{le="2"} 0
etcd_debugging_lease_ttl_total_bucket{le="4"} 0
etcd_debugging_lease_ttl_total_bucket{le="8"} 0
etcd_debugging_lease_ttl_total_bucket{le="16"} 200665
etcd_debugging_lease_ttl_total_bucket{le="32"} 200665
etcd_debugging_lease_ttl_total_bucket{le="64"} 200665
etcd_debugging_lease_ttl_total_bucket{le="128"} 200665
etcd_debugging_lease_ttl_total_bucket{le="256"} 200665
etcd_debugging_lease_ttl_total_bucket{le="512"} 200665
etcd_debugging_lease_ttl_total_bucket{le="1024"} 200665
etcd_debugging_lease_ttl_total_bucket{le="2048"} 200665
etcd_debugging_lease_ttl_total_bucket{le="4096"} 242643
etcd_debugging_lease_ttl_total_bucket{le="8192"} 242643
etcd_debugging_lease_ttl_total_bucket{le="16384"} 242643
etcd_debugging_lease_ttl_total_bucket{le="32768"} 242643
etcd_debugging_lease_ttl_total_bucket{le="65536"} 242643
etcd_debugging_lease_ttl_total_bucket{le="131072"} 242643
etcd_debugging_lease_ttl_total_bucket{le="262144"} 242643
etcd_debugging_lease_ttl_total_bucket{le="524288"} 242643
etcd_debugging_lease_ttl_total_bucket{le="1.048576e+06"} 242643
etcd_debugging_lease_ttl_total_bucket{le="2.097152e+06"} 242643
etcd_debugging_lease_ttl_total_bucket{le="4.194304e+06"} 242643
etcd_debugging_lease_ttl_total_bucket{le="8.388608e+06"} 242643
etcd_debugging_lease_ttl_total_bucket{le="+Inf"} 242643
etcd_debugging_lease_ttl_total_sum 1.56649455e+08
etcd_debugging_lease_ttl_total_count 242643
# HELP etcd_debugging_mvcc_compact_revision The revision of the last compaction in store.
# TYPE etcd_debugging_mvcc_compact_revision gauge
etcd_debugging_mvcc_compact_revision 1.496635429e+09
# HELP etcd_debugging_mvcc_current_revision The current revision of store.
# TYPE etcd_debugging_mvcc_current_revision gauge
etcd_debugging_mvcc_current_revision 1.496687049e+09
# HELP etcd_debugging_mvcc_db_compaction_keys_total Total number of db keys compacted.
# TYPE etcd_debugging_mvcc_db_compaction_keys_total counter
etcd_debugging_mvcc_db_compaction_keys_total 5.5730786e+07
# HELP etcd_debugging_mvcc_db_compaction_last The unix time of the last db compaction. Resets to 0 on start.
# TYPE etcd_debugging_mvcc_db_compaction_last gauge
etcd_debugging_mvcc_db_compaction_last 1.730190124e+09
# HELP etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds Bucketed histogram of db compaction pause duration.
# TYPE etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds histogram
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="1"} 56966
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="2"} 129310
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="4"} 171201
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="8"} 215150
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="16"} 237067
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="32"} 240634
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="64"} 240816
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="128"} 240860
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="256"} 240870
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="512"} 240870
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="1024"} 240870
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="2048"} 240870
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="4096"} 240870
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket{le="+Inf"} 240870
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_sum 949740
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_count 240870
# HELP etcd_debugging_mvcc_db_compaction_total_duration_milliseconds Bucketed histogram of db compaction total duration.
# TYPE etcd_debugging_mvcc_db_compaction_total_duration_milliseconds histogram
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="100"} 0
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="200"} 0
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="400"} 0
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="800"} 0
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="1600"} 1453
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="3200"} 2233
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="6400"} 2234
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="12800"} 2234
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="25600"} 2234
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="51200"} 2234
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="102400"} 2234
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="204800"} 2234
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="409600"} 2234
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="819200"} 2234
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket{le="+Inf"} 2234
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_sum 3.581816e+06
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_count 2234
# HELP etcd_debugging_mvcc_events_total Total number of events sent by this member.
# TYPE etcd_debugging_mvcc_events_total counter
etcd_debugging_mvcc_events_total 4.8640858e+07
# HELP etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds Bucketed histogram of index compaction pause duration.
# TYPE etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds histogram
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="0.5"} 0
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="1"} 0
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="2"} 0
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="4"} 0
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="8"} 0
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="16"} 0
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="32"} 102
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="64"} 1946
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="128"} 2226
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="256"} 2234
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="512"} 2234
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="1024"} 2234
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="2048"} 2234
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="4096"} 2234
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket{le="+Inf"} 2234
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_sum 109949
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_count 2234
# HELP etcd_debugging_mvcc_keys_total Total number of keys.
# TYPE etcd_debugging_mvcc_keys_total gauge
etcd_debugging_mvcc_keys_total 92814
# HELP etcd_debugging_mvcc_pending_events_total Total number of pending events to be sent.
# TYPE etcd_debugging_mvcc_pending_events_total gauge
etcd_debugging_mvcc_pending_events_total 0
# HELP etcd_debugging_mvcc_range_total Total number of ranges seen by this member.
# TYPE etcd_debugging_mvcc_range_total counter
etcd_debugging_mvcc_range_total 1.81385635e+08
# HELP etcd_debugging_mvcc_slow_watcher_total Total number of unsynced slow watchers.
# TYPE etcd_debugging_mvcc_slow_watcher_total gauge
etcd_debugging_mvcc_slow_watcher_total 0
# HELP etcd_debugging_mvcc_total_put_size_in_bytes The total size of put kv pairs seen by this member.
# TYPE etcd_debugging_mvcc_total_put_size_in_bytes gauge
etcd_debugging_mvcc_total_put_size_in_bytes 2.25192532612e+11
# HELP etcd_debugging_mvcc_watch_stream_total Total number of watch streams.
# TYPE etcd_debugging_mvcc_watch_stream_total gauge
etcd_debugging_mvcc_watch_stream_total 687
# HELP etcd_debugging_mvcc_watcher_total Total number of watchers.
# TYPE etcd_debugging_mvcc_watcher_total gauge
etcd_debugging_mvcc_watcher_total 687
# HELP etcd_debugging_server_lease_expired_total The total number of expired leases.
# TYPE etcd_debugging_server_lease_expired_total counter
etcd_debugging_server_lease_expired_total 115645
# HELP etcd_debugging_snap_save_marshalling_duration_seconds The marshalling cost distributions of save called by snapshot.
# TYPE etcd_debugging_snap_save_marshalling_duration_seconds histogram
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.001"} 5847
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.002"} 5849
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.004"} 5851
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.008"} 5852
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.016"} 5852
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.032"} 5852
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.064"} 5852
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.128"} 5852
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.256"} 5852
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="0.512"} 5852
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="1.024"} 5852
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="2.048"} 5852
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="4.096"} 5852
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="8.192"} 5852
etcd_debugging_snap_save_marshalling_duration_seconds_bucket{le="+Inf"} 5852
etcd_debugging_snap_save_marshalling_duration_seconds_sum 0.15398643800000014
etcd_debugging_snap_save_marshalling_duration_seconds_count 5852
# HELP etcd_debugging_snap_save_total_duration_seconds The total latency distributions of save called by snapshot.
# TYPE etcd_debugging_snap_save_total_duration_seconds histogram
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.001"} 5445
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.002"} 5732
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.004"} 5810
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.008"} 5843
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.016"} 5850
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.032"} 5852
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.064"} 5852
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.128"} 5852
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.256"} 5852
etcd_debugging_snap_save_total_duration_seconds_bucket{le="0.512"} 5852
etcd_debugging_snap_save_total_duration_seconds_bucket{le="1.024"} 5852
etcd_debugging_snap_save_total_duration_seconds_bucket{le="2.048"} 5852
etcd_debugging_snap_save_total_duration_seconds_bucket{le="4.096"} 5852
etcd_debugging_snap_save_total_duration_seconds_bucket{le="8.192"} 5852
etcd_debugging_snap_save_total_duration_seconds_bucket{le="+Inf"} 5852
etcd_debugging_snap_save_total_duration_seconds_sum 4.221123172000007
etcd_debugging_snap_save_total_duration_seconds_count 5852
# HELP etcd_debugging_store_expires_total Total number of expired keys.
# TYPE etcd_debugging_store_expires_total counter
etcd_debugging_store_expires_total 0
# HELP etcd_debugging_store_reads_total Total number of reads action by (get/getRecursive), local to this member.
# TYPE etcd_debugging_store_reads_total counter
etcd_debugging_store_reads_total{action="get"} 1
etcd_debugging_store_reads_total{action="getRecursive"} 11
# HELP etcd_debugging_store_watch_requests_total Total number of incoming watch requests (new or reestablished).
# TYPE etcd_debugging_store_watch_requests_total counter
etcd_debugging_store_watch_requests_total 0
# HELP etcd_debugging_store_watchers Count of currently active watchers.
# TYPE etcd_debugging_store_watchers gauge
etcd_debugging_store_watchers 0
# HELP etcd_debugging_store_writes_total Total number of writes (e.g. set/compareAndDelete) seen by this member.
# TYPE etcd_debugging_store_writes_total counter
etcd_debugging_store_writes_total{action="create"} 4
etcd_debugging_store_writes_total{action="delete"} 2
etcd_debugging_store_writes_total{action="set"} 5
# HELP etcd_disk_backend_commit_duration_seconds The latency distributions of commit called by backend.
# TYPE etcd_disk_backend_commit_duration_seconds histogram
etcd_disk_backend_commit_duration_seconds_bucket{le="0.001"} 5.980291e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.002"} 6.821835e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.004"} 6.940108e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.008"} 7.005373e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.016"} 7.021425e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.032"} 7.023978e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.064"} 7.02666e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.128"} 7.030602e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.256"} 7.030998e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="0.512"} 7.031003e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="1.024"} 7.031003e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="2.048"} 7.031003e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="4.096"} 7.031003e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="8.192"} 7.031003e+06
etcd_disk_backend_commit_duration_seconds_bucket{le="+Inf"} 7.031003e+06
etcd_disk_backend_commit_duration_seconds_sum 6623.993133897517
etcd_disk_backend_commit_duration_seconds_count 7.031003e+06
# HELP etcd_disk_backend_defrag_duration_seconds The latency distribution of backend defragmentation.
# TYPE etcd_disk_backend_defrag_duration_seconds histogram
etcd_disk_backend_defrag_duration_seconds_bucket{le="0.1"} 0
etcd_disk_backend_defrag_duration_seconds_bucket{le="0.2"} 0
etcd_disk_backend_defrag_duration_seconds_bucket{le="0.4"} 0
etcd_disk_backend_defrag_duration_seconds_bucket{le="0.8"} 0
etcd_disk_backend_defrag_duration_seconds_bucket{le="1.6"} 0
etcd_disk_backend_defrag_duration_seconds_bucket{le="3.2"} 1
etcd_disk_backend_defrag_duration_seconds_bucket{le="6.4"} 5
etcd_disk_backend_defrag_duration_seconds_bucket{le="12.8"} 5
etcd_disk_backend_defrag_duration_seconds_bucket{le="25.6"} 5
etcd_disk_backend_defrag_duration_seconds_bucket{le="51.2"} 5
etcd_disk_backend_defrag_duration_seconds_bucket{le="102.4"} 5
etcd_disk_backend_defrag_duration_seconds_bucket{le="204.8"} 5
etcd_disk_backend_defrag_duration_seconds_bucket{le="409.6"} 5
etcd_disk_backend_defrag_duration_seconds_bucket{le="+Inf"} 5
etcd_disk_backend_defrag_duration_seconds_sum 23.220539312
etcd_disk_backend_defrag_duration_seconds_count 5
# HELP etcd_disk_backend_snapshot_duration_seconds The latency distribution of backend snapshots.
# TYPE etcd_disk_backend_snapshot_duration_seconds histogram
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.01"} 0
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.02"} 0
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.04"} 0
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.08"} 0
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.16"} 0
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.32"} 0
etcd_disk_backend_snapshot_duration_seconds_bucket{le="0.64"} 0
etcd_disk_backend_snapshot_duration_seconds_bucket{le="1.28"} 0
etcd_disk_backend_snapshot_duration_seconds_bucket{le="2.56"} 1
etcd_disk_backend_snapshot_duration_seconds_bucket{le="5.12"} 2
etcd_disk_backend_snapshot_duration_seconds_bucket{le="10.24"} 2
etcd_disk_backend_snapshot_duration_seconds_bucket{le="20.48"} 2
etcd_disk_backend_snapshot_duration_seconds_bucket{le="40.96"} 2
etcd_disk_backend_snapshot_duration_seconds_bucket{le="81.92"} 2
etcd_disk_backend_snapshot_duration_seconds_bucket{le="163.84"} 5
etcd_disk_backend_snapshot_duration_seconds_bucket{le="327.68"} 7
etcd_disk_backend_snapshot_duration_seconds_bucket{le="655.36"} 7
etcd_disk_backend_snapshot_duration_seconds_bucket{le="+Inf"} 7
etcd_disk_backend_snapshot_duration_seconds_sum 750.9003464509999
etcd_disk_backend_snapshot_duration_seconds_count 7
# HELP etcd_disk_defrag_inflight Whether or not defrag is active on the member. 1 means active, 0 means not.
# TYPE etcd_disk_defrag_inflight gauge
etcd_disk_defrag_inflight 0
# HELP etcd_disk_wal_fsync_duration_seconds The latency distributions of fsync called by WAL.
# TYPE etcd_disk_wal_fsync_duration_seconds histogram
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.001"} 5.5912789e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.002"} 5.6249583e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.004"} 5.6508846e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.008"} 5.6652122e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.016"} 5.6688257e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.032"} 5.6695139e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.064"} 5.6701936e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.128"} 5.6707865e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.256"} 5.6708525e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="0.512"} 5.6708535e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="1.024"} 5.6708538e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="2.048"} 5.6708538e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="4.096"} 5.6708538e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="8.192"} 5.6708538e+07
etcd_disk_wal_fsync_duration_seconds_bucket{le="+Inf"} 5.6708538e+07
etcd_disk_wal_fsync_duration_seconds_sum 13064.413789368748
etcd_disk_wal_fsync_duration_seconds_count 5.6708538e+07
# HELP etcd_disk_wal_write_bytes_total Total number of bytes written in WAL.
# TYPE etcd_disk_wal_write_bytes_total gauge
etcd_disk_wal_write_bytes_total 4.75544373656e+11
# HELP etcd_grpc_proxy_cache_hits_total Total number of cache hits
# TYPE etcd_grpc_proxy_cache_hits_total gauge
etcd_grpc_proxy_cache_hits_total 0
# HELP etcd_grpc_proxy_cache_keys_total Total number of keys/ranges cached
# TYPE etcd_grpc_proxy_cache_keys_total gauge
etcd_grpc_proxy_cache_keys_total 0
# HELP etcd_grpc_proxy_cache_misses_total Total number of cache misses
# TYPE etcd_grpc_proxy_cache_misses_total gauge
etcd_grpc_proxy_cache_misses_total 0
# HELP etcd_grpc_proxy_events_coalescing_total Total number of events coalescing
# TYPE etcd_grpc_proxy_events_coalescing_total counter
etcd_grpc_proxy_events_coalescing_total 0
# HELP etcd_grpc_proxy_watchers_coalescing_total Total number of current watchers coalescing
# TYPE etcd_grpc_proxy_watchers_coalescing_total gauge
etcd_grpc_proxy_watchers_coalescing_total 0
# HELP etcd_mvcc_db_open_read_transactions The number of currently open read transactions
# TYPE etcd_mvcc_db_open_read_transactions gauge
etcd_mvcc_db_open_read_transactions 1
# HELP etcd_mvcc_db_total_size_in_bytes Total size of the underlying database physically allocated in bytes.
# TYPE etcd_mvcc_db_total_size_in_bytes gauge
etcd_mvcc_db_total_size_in_bytes 1.259360256e+09
# HELP etcd_mvcc_db_total_size_in_use_in_bytes Total size of the underlying database logically in use in bytes.
# TYPE etcd_mvcc_db_total_size_in_use_in_bytes gauge
etcd_mvcc_db_total_size_in_use_in_bytes 1.248944128e+09
# HELP etcd_mvcc_delete_total Total number of deletes seen by this member.
# TYPE etcd_mvcc_delete_total counter
etcd_mvcc_delete_total 3.057641e+06
# HELP etcd_mvcc_hash_duration_seconds The latency distribution of storage hash operation.
# TYPE etcd_mvcc_hash_duration_seconds histogram
etcd_mvcc_hash_duration_seconds_bucket{le="0.01"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="0.02"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="0.04"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="0.08"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="0.16"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="0.32"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="0.64"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="1.28"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="2.56"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="5.12"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="10.24"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="20.48"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="40.96"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="81.92"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="163.84"} 0
etcd_mvcc_hash_duration_seconds_bucket{le="+Inf"} 0
etcd_mvcc_hash_duration_seconds_sum 0
etcd_mvcc_hash_duration_seconds_count 0
# HELP etcd_mvcc_hash_rev_duration_seconds The latency distribution of storage hash by revision operation.
# TYPE etcd_mvcc_hash_rev_duration_seconds histogram
etcd_mvcc_hash_rev_duration_seconds_bucket{le="0.01"} 0
etcd_mvcc_hash_rev_duration_seconds_bucket{le="0.02"} 0
etcd_mvcc_hash_rev_duration_seconds_bucket{le="0.04"} 0
etcd_mvcc_hash_rev_duration_seconds_bucket{le="0.08"} 0
etcd_mvcc_hash_rev_duration_seconds_bucket{le="0.16"} 0
etcd_mvcc_hash_rev_duration_seconds_bucket{le="0.32"} 29
etcd_mvcc_hash_rev_duration_seconds_bucket{le="0.64"} 31
etcd_mvcc_hash_rev_duration_seconds_bucket{le="1.28"} 31
etcd_mvcc_hash_rev_duration_seconds_bucket{le="2.56"} 31
etcd_mvcc_hash_rev_duration_seconds_bucket{le="5.12"} 31
etcd_mvcc_hash_rev_duration_seconds_bucket{le="10.24"} 31
etcd_mvcc_hash_rev_duration_seconds_bucket{le="20.48"} 31
etcd_mvcc_hash_rev_duration_seconds_bucket{le="40.96"} 31
etcd_mvcc_hash_rev_duration_seconds_bucket{le="81.92"} 31
etcd_mvcc_hash_rev_duration_seconds_bucket{le="163.84"} 31
etcd_mvcc_hash_rev_duration_seconds_bucket{le="+Inf"} 31
etcd_mvcc_hash_rev_duration_seconds_sum 8.344977114000002
etcd_mvcc_hash_rev_duration_seconds_count 31
# HELP etcd_mvcc_put_total Total number of puts seen by this member.
# TYPE etcd_mvcc_put_total counter
etcd_mvcc_put_total 5.26738e+07
# HELP etcd_mvcc_range_total Total number of ranges seen by this member.
# TYPE etcd_mvcc_range_total counter
etcd_mvcc_range_total 1.81385635e+08
# HELP etcd_mvcc_txn_total Total number of txns seen by this member.
# TYPE etcd_mvcc_txn_total counter
etcd_mvcc_txn_total 38227
# HELP etcd_network_active_peers The current number of active peer connections.
# TYPE etcd_network_active_peers gauge
etcd_network_active_peers{Local="5e66e8bfa5f5d5fb",Remote="335c9823932a1698"} 1
etcd_network_active_peers{Local="5e66e8bfa5f5d5fb",Remote="ba62798d4a0aef93"} 0
etcd_network_active_peers{Local="5e66e8bfa5f5d5fb",Remote="c71bd928e7a9866e"} 0
etcd_network_active_peers{Local="5e66e8bfa5f5d5fb",Remote="f736583c8dc56a31"} 1
# HELP etcd_network_client_grpc_received_bytes_total The total number of bytes received from grpc clients.
# TYPE etcd_network_client_grpc_received_bytes_total counter
etcd_network_client_grpc_received_bytes_total 1.66213388009e+11
# HELP etcd_network_client_grpc_sent_bytes_total The total number of bytes sent to grpc clients.
# TYPE etcd_network_client_grpc_sent_bytes_total counter
etcd_network_client_grpc_sent_bytes_total 3.81410795278e+12
# HELP etcd_network_disconnected_peers_total The total number of disconnected peers.
# TYPE etcd_network_disconnected_peers_total counter
etcd_network_disconnected_peers_total{Local="5e66e8bfa5f5d5fb",Remote="335c9823932a1698"} 1
etcd_network_disconnected_peers_total{Local="5e66e8bfa5f5d5fb",Remote="ba62798d4a0aef93"} 2
etcd_network_disconnected_peers_total{Local="5e66e8bfa5f5d5fb",Remote="c71bd928e7a9866e"} 2
etcd_network_disconnected_peers_total{Local="5e66e8bfa5f5d5fb",Remote="f736583c8dc56a31"} 1
# HELP etcd_network_peer_received_bytes_total The total number of bytes received from peers.
# TYPE etcd_network_peer_received_bytes_total counter
etcd_network_peer_received_bytes_total{From="0"} 4.813218e+07
etcd_network_peer_received_bytes_total{From="335c9823932a1698"} 1.4651030177e+10
etcd_network_peer_received_bytes_total{From="ba62798d4a0aef93"} 1.71673764291e+11
etcd_network_peer_received_bytes_total{From="c71bd928e7a9866e"} 4.317738241e+09
etcd_network_peer_received_bytes_total{From="f736583c8dc56a31"} 4.920797441e+10
# HELP etcd_network_peer_round_trip_time_seconds Round-Trip-Time histogram between peers
# TYPE etcd_network_peer_round_trip_time_seconds histogram
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.0001"} 3
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.0002"} 3
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.0004"} 1357
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.0008"} 2156
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.0016"} 2326
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.0032"} 2329
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.0064"} 2330
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.0128"} 4651
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.0256"} 4654
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.0512"} 4654
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.1024"} 4654
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.2048"} 4654
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.4096"} 4654
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="0.8192"} 4654
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="1.6384"} 4654
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="3.2768"} 4654
etcd_network_peer_round_trip_time_seconds_bucket{To="335c9823932a1698",le="+Inf"} 4654
etcd_network_peer_round_trip_time_seconds_sum{To="335c9823932a1698"} 23.334861109000073
etcd_network_peer_round_trip_time_seconds_count{To="335c9823932a1698"} 4654
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.0001"} 2
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.0002"} 2
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.0004"} 17793
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.0008"} 19881
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.0016"} 19964
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.0032"} 19977
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.0064"} 19977
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.0128"} 39942
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.0256"} 39957
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.0512"} 39958
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.1024"} 39959
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.2048"} 39959
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.4096"} 39959
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="0.8192"} 39959
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="1.6384"} 39959
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="3.2768"} 39959
etcd_network_peer_round_trip_time_seconds_bucket{To="ba62798d4a0aef93",le="+Inf"} 39959
etcd_network_peer_round_trip_time_seconds_sum{To="ba62798d4a0aef93"} 196.12174685999705
etcd_network_peer_round_trip_time_seconds_count{To="ba62798d4a0aef93"} 39959
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.0001"} 1
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.0002"} 1
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.0004"} 2504
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.0008"} 2597
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.0016"} 2614
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.0032"} 2615
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.0064"} 2615
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.0128"} 5220
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.0256"} 5226
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.0512"} 5226
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.1024"} 5226
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.2048"} 5226
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.4096"} 5226
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="0.8192"} 5226
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="1.6384"} 5226
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="3.2768"} 5226
etcd_network_peer_round_trip_time_seconds_bucket{To="c71bd928e7a9866e",le="+Inf"} 5226
etcd_network_peer_round_trip_time_seconds_sum{To="c71bd928e7a9866e"} 30.11182564599996
etcd_network_peer_round_trip_time_seconds_count{To="c71bd928e7a9866e"} 5226
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.0001"} 5
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.0002"} 5
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.0004"} 16143
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.0008"} 19198
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.0016"} 19626
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.0032"} 19662
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.0064"} 19663
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.0128"} 39311
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.0256"} 39318
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.0512"} 39318
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.1024"} 39318
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.2048"} 39318
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.4096"} 39318
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="0.8192"} 39318
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="1.6384"} 39318
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="3.2768"} 39318
etcd_network_peer_round_trip_time_seconds_bucket{To="f736583c8dc56a31",le="+Inf"} 39318
etcd_network_peer_round_trip_time_seconds_sum{To="f736583c8dc56a31"} 189.49696528099952
etcd_network_peer_round_trip_time_seconds_count{To="f736583c8dc56a31"} 39318
# HELP etcd_network_peer_sent_bytes_total The total number of bytes sent to peers.
# TYPE etcd_network_peer_sent_bytes_total counter
etcd_network_peer_sent_bytes_total{To="335c9823932a1698"} 3.0351528818e+10
etcd_network_peer_sent_bytes_total{To="ba62798d4a0aef93"} 1.60528799081e+11
etcd_network_peer_sent_bytes_total{To="c71bd928e7a9866e"} 1.1159781714e+10
etcd_network_peer_sent_bytes_total{To="f736583c8dc56a31"} 1.26479639717e+11
# HELP etcd_network_peer_sent_failures_total The total number of send failures from peers.
# TYPE etcd_network_peer_sent_failures_total counter
etcd_network_peer_sent_failures_total{To="335c9823932a1698"} 2355
etcd_network_peer_sent_failures_total{To="ba62798d4a0aef93"} 9372
etcd_network_peer_sent_failures_total{To="f736583c8dc56a31"} 4337
# HELP etcd_network_snapshot_receive_inflights_total Total number of inflight snapshot receives
# TYPE etcd_network_snapshot_receive_inflights_total gauge
etcd_network_snapshot_receive_inflights_total{From="ba62798d4a0aef93"} 0
# HELP etcd_network_snapshot_receive_success Total number of successful snapshot receives
# TYPE etcd_network_snapshot_receive_success counter
etcd_network_snapshot_receive_success{From="ba62798d4a0aef93"} 1
# HELP etcd_network_snapshot_receive_total_duration_seconds Total latency distributions of v3 snapshot receives
# TYPE etcd_network_snapshot_receive_total_duration_seconds histogram
etcd_network_snapshot_receive_total_duration_seconds_bucket{From="ba62798d4a0aef93",le="0.1"} 0
etcd_network_snapshot_receive_total_duration_seconds_bucket{From="ba62798d4a0aef93",le="0.2"} 0
etcd_network_snapshot_receive_total_duration_seconds_bucket{From="ba62798d4a0aef93",le="0.4"} 0
etcd_network_snapshot_receive_total_duration_seconds_bucket{From="ba62798d4a0aef93",le="0.8"} 0
etcd_network_snapshot_receive_total_duration_seconds_bucket{From="ba62798d4a0aef93",le="1.6"} 0
etcd_network_snapshot_receive_total_duration_seconds_bucket{From="ba62798d4a0aef93",le="3.2"} 0
etcd_network_snapshot_receive_total_duration_seconds_bucket{From="ba62798d4a0aef93",le="6.4"} 0
etcd_network_snapshot_receive_total_duration_seconds_bucket{From="ba62798d4a0aef93",le="12.8"} 1
etcd_network_snapshot_receive_total_duration_seconds_bucket{From="ba62798d4a0aef93",le="25.6"} 1
etcd_network_snapshot_receive_total_duration_seconds_bucket{From="ba62798d4a0aef93",le="51.2"} 1
etcd_network_snapshot_receive_total_duration_seconds_bucket{From="ba62798d4a0aef93",le="+Inf"} 1
etcd_network_snapshot_receive_total_duration_seconds_sum{From="ba62798d4a0aef93"} 6.8592539949999995
etcd_network_snapshot_receive_total_duration_seconds_count{From="ba62798d4a0aef93"} 1
# HELP etcd_network_snapshot_send_inflights_total Total number of inflight snapshot sends
# TYPE etcd_network_snapshot_send_inflights_total gauge
etcd_network_snapshot_send_inflights_total{To="335c9823932a1698"} 0
etcd_network_snapshot_send_inflights_total{To="f736583c8dc56a31"} 0
# HELP etcd_network_snapshot_send_success Total number of successful snapshot sends
# TYPE etcd_network_snapshot_send_success counter
etcd_network_snapshot_send_success{To="335c9823932a1698"} 1
etcd_network_snapshot_send_success{To="f736583c8dc56a31"} 1
# HELP etcd_network_snapshot_send_total_duration_seconds Total latency distributions of v3 snapshot sends
# TYPE etcd_network_snapshot_send_total_duration_seconds histogram
etcd_network_snapshot_send_total_duration_seconds_bucket{To="335c9823932a1698",le="0.1"} 0
etcd_network_snapshot_send_total_duration_seconds_bucket{To="335c9823932a1698",le="0.2"} 0
etcd_network_snapshot_send_total_duration_seconds_bucket{To="335c9823932a1698",le="0.4"} 0
etcd_network_snapshot_send_total_duration_seconds_bucket{To="335c9823932a1698",le="0.8"} 0
etcd_network_snapshot_send_total_duration_seconds_bucket{To="335c9823932a1698",le="1.6"} 0
etcd_network_snapshot_send_total_duration_seconds_bucket{To="335c9823932a1698",le="3.2"} 0
etcd_network_snapshot_send_total_duration_seconds_bucket{To="335c9823932a1698",le="6.4"} 1
etcd_network_snapshot_send_total_duration_seconds_bucket{To="335c9823932a1698",le="12.8"} 1
etcd_network_snapshot_send_total_duration_seconds_bucket{To="335c9823932a1698",le="25.6"} 1
etcd_network_snapshot_send_total_duration_seconds_bucket{To="335c9823932a1698",le="51.2"} 1
etcd_network_snapshot_send_total_duration_seconds_bucket{To="335c9823932a1698",le="+Inf"} 1
etcd_network_snapshot_send_total_duration_seconds_sum{To="335c9823932a1698"} 5.7818398680000005
etcd_network_snapshot_send_total_duration_seconds_count{To="335c9823932a1698"} 1
etcd_network_snapshot_send_total_duration_seconds_bucket{To="f736583c8dc56a31",le="0.1"} 0
etcd_network_snapshot_send_total_duration_seconds_bucket{To="f736583c8dc56a31",le="0.2"} 0
etcd_network_snapshot_send_total_duration_seconds_bucket{To="f736583c8dc56a31",le="0.4"} 0
etcd_network_snapshot_send_total_duration_seconds_bucket{To="f736583c8dc56a31",le="0.8"} 0
etcd_network_snapshot_send_total_duration_seconds_bucket{To="f736583c8dc56a31",le="1.6"} 0
etcd_network_snapshot_send_total_duration_seconds_bucket{To="f736583c8dc56a31",le="3.2"} 0
etcd_network_snapshot_send_total_duration_seconds_bucket{To="f736583c8dc56a31",le="6.4"} 1
etcd_network_snapshot_send_total_duration_seconds_bucket{To="f736583c8dc56a31",le="12.8"} 1
etcd_network_snapshot_send_total_duration_seconds_bucket{To="f736583c8dc56a31",le="25.6"} 1
etcd_network_snapshot_send_total_duration_seconds_bucket{To="f736583c8dc56a31",le="51.2"} 1
etcd_network_snapshot_send_total_duration_seconds_bucket{To="f736583c8dc56a31",le="+Inf"} 1
etcd_network_snapshot_send_total_duration_seconds_sum{To="f736583c8dc56a31"} 3.300911036
etcd_network_snapshot_send_total_duration_seconds_count{To="f736583c8dc56a31"} 1
# HELP etcd_server_apply_duration_seconds The latency distributions of v2 apply called by backend.
# TYPE etcd_server_apply_duration_seconds histogram
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.0001"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.0002"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.0004"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.0008"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.0016"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.0032"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.0064"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.0128"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.0256"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.0512"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.1024"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.2048"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.4096"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="0.8192"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="1.6384"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="3.2768"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="6.5536"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="13.1072"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="26.2144"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="52.4288"} 54
etcd_server_apply_duration_seconds_bucket{op="Alarm",success="true",version="v3",le="+Inf"} 54
etcd_server_apply_duration_seconds_sum{op="Alarm",success="true",version="v3"} 0.00024294699999999995
etcd_server_apply_duration_seconds_count{op="Alarm",success="true",version="v3"} 54
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.0001"} 178
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.0002"} 185
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.0004"} 219
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.0008"} 1533
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.0016"} 2376
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.0032"} 2402
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.0064"} 2414
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.0128"} 2418
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.0256"} 2420
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.0512"} 2420
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.1024"} 2420
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.2048"} 2421
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.4096"} 2421
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="0.8192"} 2421
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="1.6384"} 2421
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="3.2768"} 2421
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="6.5536"} 2421
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="13.1072"} 2421
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="26.2144"} 2421
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="52.4288"} 2421
etcd_server_apply_duration_seconds_bucket{op="Compaction",success="true",version="v3",le="+Inf"} 2421
etcd_server_apply_duration_seconds_sum{op="Compaction",success="true",version="v3"} 1.9618274850000033
etcd_server_apply_duration_seconds_count{op="Compaction",success="true",version="v3"} 2421
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.0001"} 218723
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.0002"} 239077
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.0004"} 240412
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.0008"} 241654
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.0016"} 242232
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.0032"} 242437
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.0064"} 242553
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.0128"} 242609
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.0256"} 242633
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.0512"} 242639
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.1024"} 242643
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.2048"} 242643
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.4096"} 242643
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="0.8192"} 242643
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="1.6384"} 242643
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="3.2768"} 242643
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="6.5536"} 242643
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="13.1072"} 242643
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="26.2144"} 242643
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="52.4288"} 242643
etcd_server_apply_duration_seconds_bucket{op="LeaseGrant",success="true",version="v3",le="+Inf"} 242643
etcd_server_apply_duration_seconds_sum{op="LeaseGrant",success="true",version="v3"} 21.005283269999723
etcd_server_apply_duration_seconds_count{op="LeaseGrant",success="true",version="v3"} 242643
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.0001"} 0
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.0002"} 31
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.0004"} 10695
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.0008"} 116696
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.0016"} 228582
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.0032"} 238827
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.0064"} 241953
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.0128"} 242440
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.0256"} 242542
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.0512"} 242563
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.1024"} 242583
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.2048"} 242590
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.4096"} 242591
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="0.8192"} 242591
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="1.6384"} 242591
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="3.2768"} 242591
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="6.5536"} 242591
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="13.1072"} 242591
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="26.2144"} 242591
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="52.4288"} 242591
etcd_server_apply_duration_seconds_bucket{op="LeaseRevoke",success="true",version="v3",le="+Inf"} 242591
etcd_server_apply_duration_seconds_sum{op="LeaseRevoke",success="true",version="v3"} 229.11237849100294
etcd_server_apply_duration_seconds_count{op="LeaseRevoke",success="true",version="v3"} 242591
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.0001"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.0002"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.0004"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.0008"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.0016"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.0032"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.0064"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.0128"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.0256"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.0512"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.1024"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.2048"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.4096"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="0.8192"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="1.6384"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="3.2768"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="6.5536"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="13.1072"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="26.2144"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="52.4288"} 5
etcd_server_apply_duration_seconds_bucket{op="PUT",success="true",version="v2",le="+Inf"} 5
etcd_server_apply_duration_seconds_sum{op="PUT",success="true",version="v2"} 0.000352487
etcd_server_apply_duration_seconds_count{op="PUT",success="true",version="v2"} 5
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.0001"} 4.7888489e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.0002"} 5.6952535e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.0004"} 5.740872e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.0008"} 5.7714856e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.0016"} 5.789993e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.0032"} 5.7957926e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.0064"} 5.7995894e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.0128"} 5.8019869e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.0256"} 5.8026903e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.0512"} 5.8028046e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.1024"} 5.8028588e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.2048"} 5.8028711e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.4096"} 5.8028736e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="0.8192"} 5.8028737e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="1.6384"} 5.8028737e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="3.2768"} 5.8028738e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="6.5536"} 5.8028742e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="13.1072"} 5.8028742e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="26.2144"} 5.8028742e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="52.4288"} 5.8028742e+07
etcd_server_apply_duration_seconds_bucket{op="Txn",success="true",version="v3",le="+Inf"} 5.8028742e+07
etcd_server_apply_duration_seconds_sum{op="Txn",success="true",version="v3"} 5412.691028836994
etcd_server_apply_duration_seconds_count{op="Txn",success="true",version="v3"} 5.8028742e+07
# HELP etcd_server_client_requests_total The total number of client requests per client version.
# TYPE etcd_server_client_requests_total counter
etcd_server_client_requests_total{client_api_version="3.5",type="stream"} 5
etcd_server_client_requests_total{client_api_version="3.5",type="unary"} 97
etcd_server_client_requests_total{client_api_version="unknown",type="stream"} 1295
etcd_server_client_requests_total{client_api_version="unknown",type="unary"} 9.6732834e+07
# HELP etcd_server_go_version Which Go version server is running with. 1 for 'server_go_version' label with current version.
# TYPE etcd_server_go_version gauge
etcd_server_go_version{server_go_version="go1.20.13"} 1
# HELP etcd_server_has_leader Whether or not a leader exists. 1 is existence, 0 is not.
# TYPE etcd_server_has_leader gauge
etcd_server_has_leader 1
# HELP etcd_server_health_failures The total number of failed health checks
# TYPE etcd_server_health_failures counter
etcd_server_health_failures 0
# HELP etcd_server_health_success The total number of successful health checks
# TYPE etcd_server_health_success counter
etcd_server_health_success 0
# HELP etcd_server_heartbeat_send_failures_total The total number of leader heartbeat send failures (likely overloaded from slow disk).
# TYPE etcd_server_heartbeat_send_failures_total counter
etcd_server_heartbeat_send_failures_total 0
# HELP etcd_server_id Server or member ID in hexadecimal format. 1 for 'server_id' label with current ID.
# TYPE etcd_server_id gauge
etcd_server_id{server_id="5e66e8bfa5f5d5fb"} 1
# HELP etcd_server_is_leader Whether or not this member is a leader. 1 if is, 0 otherwise.
# TYPE etcd_server_is_leader gauge
etcd_server_is_leader 0
# HELP etcd_server_is_learner Whether or not this member is a learner. 1 if is, 0 otherwise.
# TYPE etcd_server_is_learner gauge
etcd_server_is_learner 0
# HELP etcd_server_leader_changes_seen_total The number of leader changes seen.
# TYPE etcd_server_leader_changes_seen_total counter
etcd_server_leader_changes_seen_total 7
# HELP etcd_server_learner_promote_successes The total number of successful learner promotions while this member is leader.
# TYPE etcd_server_learner_promote_successes counter
etcd_server_learner_promote_successes 0
# HELP etcd_server_proposals_applied_total The total number of consensus proposals applied.
# TYPE etcd_server_proposals_applied_total gauge
etcd_server_proposals_applied_total 1.579980098e+09
# HELP etcd_server_proposals_committed_total The total number of consensus proposals committed.
# TYPE etcd_server_proposals_committed_total gauge
etcd_server_proposals_committed_total 1.579980098e+09
# HELP etcd_server_proposals_failed_total The total number of failed proposals seen.
# TYPE etcd_server_proposals_failed_total counter
etcd_server_proposals_failed_total 36
# HELP etcd_server_proposals_pending The current number of pending proposals to commit.
# TYPE etcd_server_proposals_pending gauge
etcd_server_proposals_pending 0
# HELP etcd_server_quota_backend_bytes Current backend storage quota size in bytes.
# TYPE etcd_server_quota_backend_bytes gauge
etcd_server_quota_backend_bytes 6.147483648e+09
# HELP etcd_server_read_indexes_failed_total The total number of failed read indexes seen.
# TYPE etcd_server_read_indexes_failed_total counter
etcd_server_read_indexes_failed_total 2
# HELP etcd_server_slow_apply_total The total number of slow apply requests (likely overloaded from slow disk).
# TYPE etcd_server_slow_apply_total counter
etcd_server_slow_apply_total 9973
# HELP etcd_server_slow_read_indexes_total The total number of pending read indexes not in sync with leader's or timed out read index requests.
# TYPE etcd_server_slow_read_indexes_total counter
etcd_server_slow_read_indexes_total 0
# HELP etcd_server_snapshot_apply_in_progress_total 1 if the server is applying the incoming snapshot. 0 if none.
# TYPE etcd_server_snapshot_apply_in_progress_total gauge
etcd_server_snapshot_apply_in_progress_total 0
# HELP etcd_server_version Which version is running. 1 for 'server_version' label with current version.
# TYPE etcd_server_version gauge
etcd_server_version{server_version="3.5.12"} 1
# HELP etcd_snap_db_fsync_duration_seconds The latency distributions of fsyncing .snap.db file
# TYPE etcd_snap_db_fsync_duration_seconds histogram
etcd_snap_db_fsync_duration_seconds_bucket{le="0.001"} 0
etcd_snap_db_fsync_duration_seconds_bucket{le="0.002"} 0
etcd_snap_db_fsync_duration_seconds_bucket{le="0.004"} 0
etcd_snap_db_fsync_duration_seconds_bucket{le="0.008"} 0
etcd_snap_db_fsync_duration_seconds_bucket{le="0.016"} 0
etcd_snap_db_fsync_duration_seconds_bucket{le="0.032"} 0
etcd_snap_db_fsync_duration_seconds_bucket{le="0.064"} 0
etcd_snap_db_fsync_duration_seconds_bucket{le="0.128"} 0
etcd_snap_db_fsync_duration_seconds_bucket{le="0.256"} 0
etcd_snap_db_fsync_duration_seconds_bucket{le="0.512"} 0
etcd_snap_db_fsync_duration_seconds_bucket{le="1.024"} 0
etcd_snap_db_fsync_duration_seconds_bucket{le="2.048"} 0
etcd_snap_db_fsync_duration_seconds_bucket{le="4.096"} 1
etcd_snap_db_fsync_duration_seconds_bucket{le="8.192"} 1
etcd_snap_db_fsync_duration_seconds_bucket{le="+Inf"} 1
etcd_snap_db_fsync_duration_seconds_sum 2.746538158
etcd_snap_db_fsync_duration_seconds_count 1
# HELP etcd_snap_db_save_total_duration_seconds The total latency distributions of v3 snapshot save
# TYPE etcd_snap_db_save_total_duration_seconds histogram
etcd_snap_db_save_total_duration_seconds_bucket{le="0.1"} 0
etcd_snap_db_save_total_duration_seconds_bucket{le="0.2"} 0
etcd_snap_db_save_total_duration_seconds_bucket{le="0.4"} 0
etcd_snap_db_save_total_duration_seconds_bucket{le="0.8"} 0
etcd_snap_db_save_total_duration_seconds_bucket{le="1.6"} 0
etcd_snap_db_save_total_duration_seconds_bucket{le="3.2"} 0
etcd_snap_db_save_total_duration_seconds_bucket{le="6.4"} 0
etcd_snap_db_save_total_duration_seconds_bucket{le="12.8"} 1
etcd_snap_db_save_total_duration_seconds_bucket{le="25.6"} 1
etcd_snap_db_save_total_duration_seconds_bucket{le="51.2"} 1
etcd_snap_db_save_total_duration_seconds_bucket{le="+Inf"} 1
etcd_snap_db_save_total_duration_seconds_sum 6.857892294
etcd_snap_db_save_total_duration_seconds_count 1
# HELP etcd_snap_fsync_duration_seconds The latency distributions of fsync called by snap.
# TYPE etcd_snap_fsync_duration_seconds histogram
etcd_snap_fsync_duration_seconds_bucket{le="0.001"} 5472
etcd_snap_fsync_duration_seconds_bucket{le="0.002"} 5736
etcd_snap_fsync_duration_seconds_bucket{le="0.004"} 5811
etcd_snap_fsync_duration_seconds_bucket{le="0.008"} 5844
etcd_snap_fsync_duration_seconds_bucket{le="0.016"} 5851
etcd_snap_fsync_duration_seconds_bucket{le="0.032"} 5852
etcd_snap_fsync_duration_seconds_bucket{le="0.064"} 5852
etcd_snap_fsync_duration_seconds_bucket{le="0.128"} 5852
etcd_snap_fsync_duration_seconds_bucket{le="0.256"} 5852
etcd_snap_fsync_duration_seconds_bucket{le="0.512"} 5852
etcd_snap_fsync_duration_seconds_bucket{le="1.024"} 5852
etcd_snap_fsync_duration_seconds_bucket{le="2.048"} 5852
etcd_snap_fsync_duration_seconds_bucket{le="4.096"} 5852
etcd_snap_fsync_duration_seconds_bucket{le="8.192"} 5852
etcd_snap_fsync_duration_seconds_bucket{le="+Inf"} 5852
etcd_snap_fsync_duration_seconds_sum 4.047367438000011
etcd_snap_fsync_duration_seconds_count 5852
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 4.6051e-05
go_gc_duration_seconds{quantile="0.25"} 7.3383e-05
go_gc_duration_seconds{quantile="0.5"} 9.1352e-05
go_gc_duration_seconds{quantile="0.75"} 0.000133954
go_gc_duration_seconds{quantile="1"} 0.003663881
go_gc_duration_seconds_sum 15.637471928
go_gc_duration_seconds_count 105396
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 6616
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.20.13"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 3.46916568e+08
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.8506893685976e+13
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 6.730492e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 7.0920047625e+10
# HELP go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
# TYPE go_memstats_gc_cpu_fraction gauge
go_memstats_gc_cpu_fraction 0.0010358155710217915
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 3.9325832e+07
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 3.46916568e+08
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 1.163747328e+09
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 4.57383936e+08
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 1.040181e+06
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 9.9500032e+08
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 1.621131264e+09
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7301903444564188e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 7.0921087806e+10
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 19200
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 31200
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 5.07552e+06
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 8.20896e+06
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 5.36005128e+08
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 3.878628e+06
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 4.4007424e+07
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 4.4007424e+07
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 1.7233138e+09
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 26
# HELP grpc_server_handled_total Total number of RPCs completed on the server, regardless of success or failure.
# TYPE grpc_server_handled_total counter
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Aborted",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="AlreadyExists",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 594
grpc_server_handled_total{grpc_code="Canceled",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="DataLoss",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="DeadlineExceeded",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="FailedPrecondition",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Internal",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 50144
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="InvalidArgument",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="NotFound",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 18
grpc_server_handled_total{grpc_code="OK",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 745
grpc_server_handled_total{grpc_code="OK",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 5
grpc_server_handled_total{grpc_code="OK",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 81012
grpc_server_handled_total{grpc_code="OK",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 1
grpc_server_handled_total{grpc_code="OK",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 11
grpc_server_handled_total{grpc_code="OK",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 1
grpc_server_handled_total{grpc_code="OK",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 3
grpc_server_handled_total{grpc_code="OK",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 7.7082711e+07
grpc_server_handled_total{grpc_code="OK",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 5
grpc_server_handled_total{grpc_code="OK",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 98013
grpc_server_handled_total{grpc_code="OK",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 1.9420188e+07
grpc_server_handled_total{grpc_code="OK",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="OK",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="OutOfRange",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="PermissionDenied",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="ResourceExhausted",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unauthenticated",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 2
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 14
grpc_server_handled_total{grpc_code="Unavailable",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unimplemented",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 42
grpc_server_handled_total{grpc_code="Unknown",grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 35
grpc_server_handled_total{grpc_code="Unknown",grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 0
grpc_server_handled_total{grpc_code="Unknown",grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
# HELP grpc_server_msg_received_total Total number of RPC stream messages received on the server.
# TYPE grpc_server_msg_received_total counter
grpc_server_msg_received_total{grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 18
grpc_server_msg_received_total{grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 745
grpc_server_msg_received_total{grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 5
grpc_server_msg_received_total{grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 81012
grpc_server_msg_received_total{grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_msg_received_total{grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 1
grpc_server_msg_received_total{grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 11
grpc_server_msg_received_total{grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 1
grpc_server_msg_received_total{grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 3
grpc_server_msg_received_total{grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 7.7082755e+07
grpc_server_msg_received_total{grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 5
grpc_server_msg_received_total{grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 98013
grpc_server_msg_received_total{grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 1.9470367e+07
grpc_server_msg_received_total{grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_received_total{grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 2342
grpc_server_msg_received_total{grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
# HELP grpc_server_msg_sent_total Total number of gRPC stream messages sent by the server.
# TYPE grpc_server_msg_sent_total counter
grpc_server_msg_sent_total{grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 18
grpc_server_msg_sent_total{grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 745
grpc_server_msg_sent_total{grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 5
grpc_server_msg_sent_total{grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 81012
grpc_server_msg_sent_total{grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_msg_sent_total{grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 1
grpc_server_msg_sent_total{grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 11
grpc_server_msg_sent_total{grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 1
grpc_server_msg_sent_total{grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 3
grpc_server_msg_sent_total{grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 7.7082711e+07
grpc_server_msg_sent_total{grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 201361
grpc_server_msg_sent_total{grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 98013
grpc_server_msg_sent_total{grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 1.9420188e+07
grpc_server_msg_sent_total{grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_msg_sent_total{grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 4.6972078e+07
grpc_server_msg_sent_total{grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
# HELP grpc_server_started_total Total number of RPCs started on the server.
# TYPE grpc_server_started_total counter
grpc_server_started_total{grpc_method="Alarm",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 18
grpc_server_started_total{grpc_method="AuthDisable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="AuthEnable",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="AuthStatus",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="Authenticate",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="Check",grpc_service="grpc.health.v1.Health",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="Compact",grpc_service="etcdserverpb.KV",grpc_type="unary"} 745
grpc_server_started_total{grpc_method="Defragment",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 5
grpc_server_started_total{grpc_method="DeleteRange",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="Downgrade",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="Hash",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="HashKV",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="LeaseGrant",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 81012
grpc_server_started_total{grpc_method="LeaseKeepAlive",grpc_service="etcdserverpb.Lease",grpc_type="bidi_stream"} 0
grpc_server_started_total{grpc_method="LeaseLeases",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="LeaseRevoke",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="LeaseTimeToLive",grpc_service="etcdserverpb.Lease",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="MemberAdd",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 1
grpc_server_started_total{grpc_method="MemberList",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 11
grpc_server_started_total{grpc_method="MemberPromote",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="MemberRemove",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 1
grpc_server_started_total{grpc_method="MemberUpdate",grpc_service="etcdserverpb.Cluster",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="MoveLeader",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 3
grpc_server_started_total{grpc_method="Put",grpc_service="etcdserverpb.KV",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="Range",grpc_service="etcdserverpb.KV",grpc_type="unary"} 7.7082755e+07
grpc_server_started_total{grpc_method="RoleAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="RoleDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="RoleGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="RoleGrantPermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="RoleList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="RoleRevokePermission",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="Snapshot",grpc_service="etcdserverpb.Maintenance",grpc_type="server_stream"} 5
grpc_server_started_total{grpc_method="Status",grpc_service="etcdserverpb.Maintenance",grpc_type="unary"} 98013
grpc_server_started_total{grpc_method="Txn",grpc_service="etcdserverpb.KV",grpc_type="unary"} 1.9470367e+07
grpc_server_started_total{grpc_method="UserAdd",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="UserChangePassword",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="UserDelete",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="UserGet",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="UserGrantRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="UserList",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="UserRevokeRole",grpc_service="etcdserverpb.Auth",grpc_type="unary"} 0
grpc_server_started_total{grpc_method="Watch",grpc_service="etcdserverpb.Watch",grpc_type="bidi_stream"} 1295
grpc_server_started_total{grpc_method="Watch",grpc_service="grpc.health.v1.Health",grpc_type="server_stream"} 0
# HELP os_fd_limit The file descriptor limit.
# TYPE os_fd_limit gauge
os_fd_limit 40000
# HELP os_fd_used The number of used file descriptors.
# TYPE os_fd_used gauge
os_fd_used 1224
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 212628.69
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 40000
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 1225
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 2.005987328e+09
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.72952001869e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 9.927274496e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19
# HELP promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
# TYPE promhttp_metric_handler_requests_in_flight gauge
promhttp_metric_handler_requests_in_flight 1
# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.
# TYPE promhttp_metric_handler_requests_total counter
promhttp_metric_handler_requests_total{code="200"} 44635
promhttp_metric_handler_requests_total{code="500"} 0
promhttp_metric_handler_requests_total{code="503"} 0


@ahrtr
Copy link
Member

ahrtr commented Nov 8, 2024

I couldn't reproduce this issue in 3.5.16. Please try the latest version (3.5.16) and provide the exact steps how to reproduce it.

@ugur99
Copy link
Author

ugur99 commented Nov 15, 2024

Hi @ahrtr, we have upgraded our k8s to v1.30; and afaik this version currently supports 3.5.15. But we have the same problem with this version too. I also added exact steps that we have been following.

@jmhbnz
Copy link
Member

jmhbnz commented Nov 21, 2024

Discussed during sig-etcd triage meeting, @ugur99 we'll need more info on how to reproduce this. We could discuss in sig-etcd slack or would you be able to provide a precise step by step here on how to recreate your scenario? At the moment there is too much ambiguity to be able to recreate your setup.

@ugur99
Copy link
Author

ugur99 commented Nov 22, 2024

Yep you are totally right @jmhbnz. I think I can reproduce the issue on my local with the following steps:

  • provision a vm (multipass launch --cpus 2 --disk 30G --memory 4G --name node1)

  • clone kubespray v2.25.0 (git clone https://github.com/kubernetes-sigs/kubespray.git --branch v2.25.0)

  • define etcd_metrics_port: 2381 in the inventory file either in kubespray/inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml or in kubespray/inventory/mycluster/group_vars/etcd.yml

  • run the kubespray playbooks to provision the cluster (ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml)

  • observe that metric is available curl -s http://0.0.0.0:2381/metrics |grep -i etcd_cluster_version

  • create another vm (multipass launch --cpus 2 --disk 30G --memory 4G --name node2) and setup etcd v3.5.12 on it manually

# download etcd binaries
mkdir -p /tmp/etcd-download
curl -L https://storage.googleapis.com/etcd/v3.5.12/etcd-v3.5.12-linux-arm64.tar.gz -o /tmp/etcd-v3.5.12-linux-arm64.tar.gz
tar xzvf /tmp/etcd-v3.5.12-linux-arm64.tar.gz -C /tmp/etcd-download --strip-components=1
cp /tmp/etcd-download/etcd /usr/bin/etcd
cp /tmp/etcd-download/etcdctl /usr/bin/etcdctl
cp /tmp/etcd-download/etcdutl /usr/bin/etcdutl
# cat /usr/lib/systemd/system/etcd.service
[Unit]
Description=etcd
[Service]
Type=notify
User=root
EnvironmentFile=/etc/etcd.env
ExecStart=/usr/bin/etcd
NotifyAccess=all
Restart=always
RestartSec=10s
LimitNOFILE=40000
[Install]
WantedBy=multi-user.target
# cat /etc/etcd.env 
# Environment file for etcd v3.5.12
ETCD_DATA_DIR=/var/lib/etcd
ETCD_ADVERTISE_CLIENT_URLS=https://<node2_host_ip>:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://<node2_host_ip>:2380
ETCD_INITIAL_CLUSTER_STATE=existing
ETCD_METRICS=basic
ETCD_LISTEN_METRICS_URLS=http://<node2_host_ip>:2381,http://127.0.0.1:2381
ETCD_LISTEN_CLIENT_URLS=https://<node2_host_ip>:2379,https://127.0.0.1:2379
ETCD_ELECTION_TIMEOUT=5000
ETCD_HEARTBEAT_INTERVAL=250
ETCD_INITIAL_CLUSTER_TOKEN=k8s_etcd
ETCD_LISTEN_PEER_URLS=https://<node2_host_ip>:2380
ETCD_NAME=etcd2
ETCD_PROXY=off
ETCD_INITIAL_CLUSTER=etcd1=https://<node1_host_ip>:2380,etcd2=https://<node2_host_ip>:2380
ETCD_AUTO_COMPACTION_RETENTION=8
ETCD_SNAPSHOT_COUNT=10000
ETCD_QUOTA_BACKEND_BYTES=6147483648
# Flannel need etcd v2 API
ETCD_ENABLE_V2=true

# TLS settings
ETCD_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_CERT_FILE=/etc/ssl/etcd/ssl/member-node2.pem
ETCD_KEY_FILE=/etc/ssl/etcd/ssl/member-node2-key.pem
ETCD_CLIENT_CERT_AUTH=true

ETCD_PEER_TRUSTED_CA_FILE=/etc/ssl/etcd/ssl/ca.pem
ETCD_PEER_CERT_FILE=/etc/ssl/etcd/ssl/member-node2.pem
ETCD_PEER_KEY_FILE=/etc/ssl/etcd/ssl/member-node2-key.pem
ETCD_PEER_CLIENT_CERT_AUTH=True

# CLI settings
ETCDCTL_ENDPOINTS=https://127.0.0.1:2379
ETCDCTL_CACERT=/etc/ssl/etcd/ssl/ca.pem
ETCDCTL_KEY=/etc/ssl/etcd/ssl/admin-node2-key.pem
ETCDCTL_CERT=/etc/ssl/etcd/ssl/admin-node2.pem

# ETCD 3.5.x issue
# https://groups.google.com/a/kubernetes.io/g/dev/c/B7gJs88XtQc/m/rSgNOzV2BwAJ?utm_medium=email&utm_source=footer
ETCD_EXPERIMENTAL_INITIAL_CORRUPT_CHECK=True
# ETCD tracing configuration
ETCD_EXPERIMENTAL_ENABLE_DISTRIBUTED_TRACING=true
ETCD_EXPERIMENTAL_DISTRIBUTED_TRACING_SAMPLING_RATE=100
ETCD_EXPERIMENTAL_DISTRIBUTED_TRACING_ADDRESS=0.0.0.0:4317
ETCD_EXPERIMENTAL_DISTRIBUTED_TRACING_SERVICE_NAME=etcd
ETCD_EXPERIMENTAL_DISTRIBUTED_TRACING_INSTANCE_ID=etcd2
# create first helper script for certificate generation
# cat make-ssl-etcd.sh
set -o errexit
set -o pipefail
MASTERS=(node1 node2)
HOSTS=(node1 node2)


CONFIG="/root/openssl.conf"
SSLDIR="/etc/ssl/etcd/ssl"

tmpdir=$(mktemp -d letcd_cacert.XXXXXX)
trap 'rm -rf "${tmpdir}"' EXIT
cd "${tmpdir}"


# Root CA
if [ -e "$SSLDIR/ca-key.pem" ]; then
    # Reuse existing CA
    cp $SSLDIR/{ca.pem,ca-key.pem} .
else
    openssl genrsa -out ca-key.pem 2048 > /dev/null 2>&1
    openssl req -x509 -new -nodes -key ca-key.pem -days 36500 -out ca.pem -subj "/CN=etcd-ca" > /dev/null 2>&1
fi

# ETCD member
if [ -n "$MASTERS" ]; then
    echo "Creating certs for masters"
    for host in "${MASTERS[@]}"; do
        echo "Creating certs for $host"
        cn="${host%%.*}"
        # Member key
        openssl genrsa -out member-${host}-key.pem 2048 > /dev/null 2>&1
        openssl req -new -key member-${host}-key.pem -out member-${host}.csr -subj "/CN=etcd-member-${cn}" -config ${CONFIG} > /dev/null 2>&1
        openssl x509 -req -in member-${host}.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out member-${host}.pem -days 36500 -extensions ssl_client -extfile ${CONFIG} > /dev/null 2>&1

        # Admin key
        openssl genrsa -out admin-${host}-key.pem 2048 > /dev/null 2>&1
        openssl req -new -key admin-${host}-key.pem -out admin-${host}.csr -subj "/CN=etcd-admin-${cn}" > /dev/null 2>&1
        openssl x509 -req -in admin-${host}.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out admin-${host}.pem -days 36500 -extensions ssl_client  -extfile ${CONFIG} > /dev/null 2>&1
    done
fi

# Node keys
if [ -n "$HOSTS" ]; then
    echo "Creating certs for hosts"
    for host in "${HOSTS[@]}"; do
        echo "Creating certs for $host"
        cn="${host%%.*}"
        openssl genrsa -out node-${host}-key.pem 2048 > /dev/null 2>&1
        openssl req -new -key node-${host}-key.pem -out node-${host}.csr -subj "/CN=etcd-node-${cn}" > /dev/null 2>&1
        openssl x509 -req -in node-${host}.csr -CA ca.pem -CAkey ca-key.pem -CAcreateserial -out node-${host}.pem -days 36500 -extensions ssl_client  -extfile ${CONFIG} > /dev/null 2>&1
    done
fi


# Install certs
if [ -e "$SSLDIR/ca-key.pem" ]; then
    # No pass existing CA
    rm -f ca.pem ca-key.pem
fi

if [ -n "$(ls -A *.pem)" ]; then
    mv *.pem ${SSLDIR}/
fi
chmod +x make-ssl-etcd.sh
# create openssl configuration file
# cat /root/openssl.conf
req_extensions = v3_req
distinguished_name = req_distinguished_name

[req_distinguished_name]

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ ssl_client ]
extendedKeyUsage = clientAuth, serverAuth
basicConstraints = CA:FALSE
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid,issuer
subjectAltName = @alt_names

[ v3_ca ]
basicConstraints = CA:TRUE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
authorityKeyIdentifier=keyid:always,issuer

[alt_names]
DNS.1 = localhost
DNS.2 = etcd.kube-system.svc.cluster.local
DNS.3 = etcd.kube-system.svc
DNS.4 = etcd.kube-system
DNS.5 = etcd
DNS.6 = node1
DNS.7 = node2
DNS.8 = lb-apiserver.kubernetes.local

IP.1 = 127.0.0.1
IP.2 = <node1_host_ip>
IP.3 = <node2_host_ip>
# create certificates directory on the second node
mkdir -p /etc/ssl/etcd/ssl
# copy etcd root CA and key to the second node (for simplicity I will skip controlplane joining steps; but in our environments we first join the new controlplane to the cluster and it fetches etcd CA from the secret in the cluster and we have another script that fetches etcd CA.key from the vault)
scp /etc/ssl/etcd/ssl/ca.pem /etc/ssl/etcd/ssl/ca-key.pem root@<node2_host_ip>:/etc/ssl/etcd/ssl/
# run the certificate generation script on the second node
./make-ssl-etcd.sh
# create second helper script for etcd member joining
# cat join-etcd-member.sh
#!/bin/bash

export ETCDCTL_API=3
ETCD_CLUSTER=(<node1_host_ip> <node2_host_ip>)
PEER_URL="https://<node2_host_ip>:2380"
KEY="/etc/ssl/etcd/ssl/admin-node2-key.pem"
CERT="/etc/ssl/etcd/ssl/admin-node2.pem"
CACERT="/etc/ssl/etcd/ssl/ca.pem"
LOG_FILE="/var/log/etcd_join_member.log"

# Function to log messages with timestamps
log() {
  echo "$(date '+%Y-%m-%d %H:%M:%S') - $1" | tee -a $LOG_FILE
}

# Function to check etcd instance status
check_etcd_status() {
  local endpoint=$1
  etcdctl --endpoints=https://$endpoint:2379 --key=$KEY --cert=$CERT --cacert=$CACERT endpoint status > /dev/null 2>&1
  return $?
}

# Function to add new etcd member
add_etcd_member() {
  local endpoint=$1
  etcdctl --endpoints=https://$endpoint:2379 --key=$KEY --cert=$CERT --cacert=$CACERT member add etcd2 --peer-urls=$PEER_URL
  return $?
}

# Retry mechanism
RETRIES=2
SLEEP_INTERVAL=2
for endpoint in "${ETCD_CLUSTER[@]}"; do
  for ((i=1; i<=RETRIES; i++)); do
    check_etcd_status $endpoint
    if [ $? -eq 0 ]; then
      log "etcd instance $endpoint is reachable"
      add_etcd_member $endpoint
      if [ $? -eq 0 ]; then
        log "Successfully added new etcd member using instance $endpoint"
        exit 0
      else
        log "Failed to add new etcd member using instance $endpoint"
      fi
      break
    else
      log "etcd instance $endpoint is not reachable (Attempt $i/$RETRIES)"
      sleep $SLEEP_INTERVAL
    fi
  done
done

log "Failed to add new etcd member after checking all instances"
exit 1
# run the etcd member joining script on the second node
./join-etcd-member.sh
# start the etcd service
systemctl start etcd
  • Observe etcd metrics on the second node and see only etcd_cluster_version is missing

In the flatcar ignition template, we're following almost the same steps. After the new etcd/controlplane node is provisioned, these helper scripts act to set up the etcd service; download the binary; retrieve etcd ca.key from the vault (etcd root CA comes with the kubeadm join controlplane step); generate the certificate and finally join the new etcd member to the etcd cluster. So with this setup we are able to easily perform scaling operations for controlplanes running in a bare-metal environment.

@ugur99
Copy link
Author

ugur99 commented Nov 27, 2024

@ahrtr @jmhbnz did you have a chance to reproduce the issue?

@siyuanfoundation
Copy link
Contributor

Hi @ugur99 can you check the status of the servers with

etcdctl --endpoints=<node1_host_ip>:2379,<node2_host_ip>:2379 endpoint status -w table

?

@ugur99
Copy link
Author

ugur99 commented Dec 3, 2024

Sure @siyuanfoundation, here it is:

+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|     ENDPOINT      |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.64.2:2379 | 1c5bce91c45fca0c |  3.5.12 |  4.9 MB |      true |      false |         4 |       2365 |               2365 |        |
| 192.168.64.3:2379 | 82fc98698e1ab9a9 |  3.5.12 |  4.7 MB |     false |      false |         4 |       2365 |               2365 |        |
+-------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

4 participants