# Source: https://docs.vespa.ai/en/reference/operations/metrics/distributor.html.md

# Distributor Metrics

 

| Name | Unit | Description |
| --- | --- | --- |
| 

vds.idealstate.buckets\_rechecking

 | bucket | The number of buckets that we are rechecking for ideal state operations |
| 

vds.idealstate.idealstate\_diff

 | bucket | A number representing the current difference from the ideal state. This is a number that decreases steadily as the system is getting closer to the ideal state |
| 

vds.idealstate.buckets\_toofewcopies

 | bucket | The number of buckets the distributor controls that have less than the desired redundancy |
| 

vds.idealstate.buckets\_toomanycopies

 | bucket | The number of buckets the distributor controls that have more than the desired redundancy |
| 

vds.idealstate.buckets

 | bucket | The number of buckets the distributor controls |
| 

vds.idealstate.buckets\_notrusted

 | bucket | The number of buckets that have no trusted copies. |
| 

vds.idealstate.bucket\_replicas\_moving\_out

 | bucket | Bucket replicas that should be moved out, e.g. retirement case or node added to cluster that has higher ideal state priority. |
| 

vds.idealstate.bucket\_replicas\_copying\_out

 | bucket | Bucket replicas that should be copied out, e.g. node is in ideal state but might have to provide data other nodes in a merge |
| 

vds.idealstate.bucket\_replicas\_copying\_in

 | bucket | Bucket replicas that should be copied in, e.g. node does not have a replica for a bucket that it is in ideal state for |
| 

vds.idealstate.bucket\_replicas\_syncing

 | bucket | Bucket replicas that need syncing due to mismatching metadata |
| 

vds.idealstate.max\_observed\_time\_since\_last\_gc\_sec

 | second | Maximum time (in seconds) since GC was last successfully run for a bucket. Aggregated max value across all buckets on the distributor. |
| 

vds.idealstate.delete\_bucket.done\_ok

 | operation | The number of operations successfully performed |
| 

vds.idealstate.delete\_bucket.done\_failed

 | operation | The number of operations that failed |
| 

vds.idealstate.delete\_bucket.pending

 | operation | The number of operations pending |
| 

vds.idealstate.delete\_bucket.blocked

 | operation | The number of operations blocked by blocking operation starter |
| 

vds.idealstate.delete\_bucket.throttled

 | operation | The number of operations throttled by throttling operation starter |
| 

vds.idealstate.merge\_bucket.done\_ok

 | operation | The number of operations successfully performed |
| 

vds.idealstate.merge\_bucket.done\_failed

 | operation | The number of operations that failed |
| 

vds.idealstate.merge\_bucket.pending

 | operation | The number of operations pending |
| 

vds.idealstate.merge\_bucket.blocked

 | operation | The number of operations blocked by blocking operation starter |
| 

vds.idealstate.merge\_bucket.throttled

 | operation | The number of operations throttled by throttling operation starter |
| 

vds.idealstate.merge\_bucket.source\_only\_copy\_changed

 | operation | The number of merge operations where source-only copy changed |
| 

vds.idealstate.merge\_bucket.source\_only\_copy\_delete\_blocked

 | operation | The number of merge operations where delete of unchanged source-only copies was blocked |
| 

vds.idealstate.merge\_bucket.source\_only\_copy\_delete\_failed

 | operation | The number of merge operations where delete of unchanged source-only copies failed |
| 

vds.idealstate.split\_bucket.done\_ok

 | operation | The number of operations successfully performed |
| 

vds.idealstate.split\_bucket.done\_failed

 | operation | The number of operations that failed |
| 

vds.idealstate.split\_bucket.pending

 | operation | The number of operations pending |
| 

vds.idealstate.split\_bucket.blocked

 | operation | The number of operations blocked by blocking operation starter |
| 

vds.idealstate.split\_bucket.throttled

 | operation | The number of operations throttled by throttling operation starter |
| 

vds.idealstate.join\_bucket.done\_ok

 | operation | The number of operations successfully performed |
| 

vds.idealstate.join\_bucket.done\_failed

 | operation | The number of operations that failed |
| 

vds.idealstate.join\_bucket.pending

 | operation | The number of operations pending |
| 

vds.idealstate.join\_bucket.blocked

 | operation | The number of operations blocked by blocking operation starter |
| 

vds.idealstate.join\_bucket.throttled

 | operation | The number of operations throttled by throttling operation starter |
| 

vds.idealstate.garbage\_collection.done\_ok

 | operation | The number of operations successfully performed |
| 

vds.idealstate.garbage\_collection.done\_failed

 | operation | The number of operations that failed |
| 

vds.idealstate.garbage\_collection.pending

 | operation | The number of operations pending |
| 

vds.idealstate.garbage\_collection.documents\_removed

 | document | Number of documents removed by GC operations |
| 

vds.idealstate.garbage\_collection.blocked

 | operation | The number of operations blocked by blocking operation starter |
| 

vds.idealstate.garbage\_collection.throttled

 | operation | The number of operations throttled by throttling operation starter |
| 

vds.distributor.puts.latency

 | millisecond | The latency of put operations |
| 

vds.distributor.puts.ok

 | operation | The number of successful put operations performed |
| 

vds.distributor.puts.failures.total

 | operation | Sum of all failures |
| 

vds.distributor.puts.failures.notfound

 | operation | The number of operations that failed because the document did not exist |
| 

vds.distributor.puts.failures.test\_and\_set\_failed

 | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document |
| 

vds.distributor.puts.failures.concurrent\_mutations

 | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID |
| 

vds.distributor.puts.failures.notconnected

 | operation | The number of operations discarded because there were no available storage nodes to send to |
| 

vds.distributor.puts.failures.notready

 | operation | The number of operations discarded because distributor was not ready |
| 

vds.distributor.puts.failures.wrongdistributor

 | operation | The number of operations discarded because they were sent to the wrong distributor |
| 

vds.distributor.puts.failures.safe\_time\_not\_reached

 | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed |
| 

vds.distributor.puts.failures.storagefailure

 | operation | The number of operations that failed in storage |
| 

vds.distributor.puts.failures.timeout

 | operation | The number of operations that failed because the operation timed out towards storage |
| 

vds.distributor.puts.failures.busy

 | operation | The number of messages from storage that failed because the storage node was busy |
| 

vds.distributor.puts.failures.inconsistent\_bucket

 | operation | The number of operations failed due to buckets being in an inconsistent state or not found |
| 

vds.distributor.removes.latency

 | millisecond | The latency of remove operations |
| 

vds.distributor.removes.ok

 | operation | The number of successful removes operations performed |
| 

vds.distributor.removes.failures.total

 | operation | Sum of all failures |
| 

vds.distributor.removes.failures.notfound

 | operation | The number of operations that failed because the document did not exist |
| 

vds.distributor.removes.failures.test\_and\_set\_failed

 | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document |
| 

vds.distributor.removes.failures.concurrent\_mutations

 | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID |
| 

vds.distributor.removes.failures.busy

 | operation | The number of messages from storage that failed because the storage node was busy |
| 

vds.distributor.removes.failures.inconsistent\_bucket

 | operation | The number of operations failed due to buckets being in an inconsistent state or not found |
| 

vds.distributor.removes.failures.notconnected

 | operation | The number of operations discarded because there were no available storage nodes to send to |
| 

vds.distributor.removes.failures.notready

 | operation | The number of operations discarded because distributor was not ready |
| 

vds.distributor.removes.failures.safe\_time\_not\_reached

 | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed |
| 

vds.distributor.removes.failures.storagefailure

 | operation | The number of operations that failed in storage |
| 

vds.distributor.removes.failures.timeout

 | operation | The number of operations that failed because the operation timed out towards storage |
| 

vds.distributor.removes.failures.wrongdistributor

 | operation | The number of operations discarded because they were sent to the wrong distributor |
| 

vds.distributor.updates.latency

 | millisecond | The latency of update operations |
| 

vds.distributor.updates.ok

 | operation | The number of successful updates operations performed |
| 

vds.distributor.updates.failures.total

 | operation | Sum of all failures |
| 

vds.distributor.updates.failures.notfound

 | operation | The number of operations that failed because the document did not exist |
| 

vds.distributor.updates.failures.test\_and\_set\_failed

 | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document |
| 

vds.distributor.updates.failures.concurrent\_mutations

 | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID |
| 

vds.distributor.updates.diverging\_timestamp\_updates

 | operation | Number of updates that report they were performed against divergent version timestamps on different replicas |
| 

vds.distributor.updates.failures.busy

 | operation | The number of messages from storage that failed because the storage node was busy |
| 

vds.distributor.updates.failures.inconsistent\_bucket

 | operation | The number of operations failed due to buckets being in an inconsistent state or not found |
| 

vds.distributor.updates.failures.notconnected

 | operation | The number of operations discarded because there were no available storage nodes to send to |
| 

vds.distributor.updates.failures.notready

 | operation | The number of operations discarded because distributor was not ready |
| 

vds.distributor.updates.failures.safe\_time\_not\_reached

 | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed |
| 

vds.distributor.updates.failures.storagefailure

 | operation | The number of operations that failed in storage |
| 

vds.distributor.updates.failures.timeout

 | operation | The number of operations that failed because the operation timed out towards storage |
| 

vds.distributor.updates.failures.wrongdistributor

 | operation | The number of operations discarded because they were sent to the wrong distributor |
| 

vds.distributor.updates.fast\_path\_restarts

 | operation | Number of safe path (write repair) updates that were restarted as fast path updates because all replicas returned documents with the same timestamp in the initial read phase |
| 

vds.distributor.removelocations.ok

 | operation | The number of successful removelocations operations performed |
| 

vds.distributor.removelocations.failures.total

 | operation | Sum of all failures |
| 

vds.distributor.removelocations.failures.busy

 | operation | The number of messages from storage that failed because the storage node was busy |
| 

vds.distributor.removelocations.failures.concurrent\_mutations

 | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID |
| 

vds.distributor.removelocations.failures.inconsistent\_bucket

 | operation | The number of operations failed due to buckets being in an inconsistent state or not found |
| 

vds.distributor.removelocations.failures.notconnected

 | operation | The number of operations discarded because there were no available storage nodes to send to |
| 

vds.distributor.removelocations.failures.notfound

 | operation | The number of operations that failed because the document did not exist |
| 

vds.distributor.removelocations.failures.notready

 | operation | The number of operations discarded because distributor was not ready |
| 

vds.distributor.removelocations.failures.safe\_time\_not\_reached

 | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed |
| 

vds.distributor.removelocations.failures.storagefailure

 | operation | The number of operations that failed in storage |
| 

vds.distributor.removelocations.failures.test\_and\_set\_failed

 | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document |
| 

vds.distributor.removelocations.failures.timeout

 | operation | The number of operations that failed because the operation timed out towards storage |
| 

vds.distributor.removelocations.failures.wrongdistributor

 | operation | The number of operations discarded because they were sent to the wrong distributor |
| 

vds.distributor.removelocations.latency

 | millisecond | The average latency of removelocations operations |
| 

vds.distributor.gets.latency

 | millisecond | The average latency of gets operations |
| 

vds.distributor.gets.ok

 | operation | The number of successful gets operations performed |
| 

vds.distributor.gets.failures.total

 | operation | Sum of all failures |
| 

vds.distributor.gets.failures.notfound

 | operation | The number of operations that failed because the document did not exist |
| 

vds.distributor.gets.failures.busy

 | operation | The number of messages from storage that failed because the storage node was busy |
| 

vds.distributor.gets.failures.concurrent\_mutations

 | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID |
| 

vds.distributor.gets.failures.inconsistent\_bucket

 | operation | The number of operations failed due to buckets being in an inconsistent state or not found |
| 

vds.distributor.gets.failures.notconnected

 | operation | The number of operations discarded because there were no available storage nodes to send to |
| 

vds.distributor.gets.failures.notready

 | operation | The number of operations discarded because distributor was not ready |
| 

vds.distributor.gets.failures.safe\_time\_not\_reached

 | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed |
| 

vds.distributor.gets.failures.storagefailure

 | operation | The number of operations that failed in storage |
| 

vds.distributor.gets.failures.test\_and\_set\_failed

 | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document |
| 

vds.distributor.gets.failures.timeout

 | operation | The number of operations that failed because the operation timed out towards storage |
| 

vds.distributor.gets.failures.wrongdistributor

 | operation | The number of operations discarded because they were sent to the wrong distributor |
| 

vds.distributor.visitor.latency

 | millisecond | The average latency of visitor operations |
| 

vds.distributor.visitor.ok

 | operation | The number of successful visitor operations performed |
| 

vds.distributor.visitor.failures.total

 | operation | Sum of all failures |
| 

vds.distributor.visitor.failures.notready

 | operation | The number of operations discarded because distributor was not ready |
| 

vds.distributor.visitor.failures.notconnected

 | operation | The number of operations discarded because there were no available storage nodes to send to |
| 

vds.distributor.visitor.failures.wrongdistributor

 | operation | The number of operations discarded because they were sent to the wrong distributor |
| 

vds.distributor.visitor.failures.safe\_time\_not\_reached

 | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed |
| 

vds.distributor.visitor.failures.storagefailure

 | operation | The number of operations that failed in storage |
| 

vds.distributor.visitor.failures.timeout

 | operation | The number of operations that failed because the operation timed out towards storage |
| 

vds.distributor.visitor.failures.busy

 | operation | The number of messages from storage that failed because the storage node was busy |
| 

vds.distributor.visitor.failures.inconsistent\_bucket

 | operation | The number of operations failed due to buckets being in an inconsistent state or not found |
| 

vds.distributor.visitor.failures.notfound

 | operation | The number of operations that failed because the document did not exist |
| 

vds.distributor.visitor.bytes\_per\_visitor

 | operation | The number of bytes visited on content nodes as part of a single client visitor command |
| 

vds.distributor.visitor.docs\_per\_visitor

 | operation | The number of documents visited on content nodes as part of a single client visitor command |
| 

vds.distributor.visitor.failures.concurrent\_mutations

 | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID |
| 

vds.distributor.visitor.failures.test\_and\_set\_failed

 | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document |
| 

vds.distributor.docsstored

 | document | Number of documents stored in all buckets controlled by this distributor |
| 

vds.distributor.bytesstored

 | byte | Number of bytes stored in all buckets controlled by this distributor |
| 

metricmanager.periodichooklatency

 | millisecond | Time in ms used to update a single periodic hook |
| 

metricmanager.resetlatency

 | millisecond | Time in ms used to reset all metrics. |
| 

metricmanager.sleeptime

 | millisecond | Time in ms worker thread is sleeping |
| 

metricmanager.snapshothooklatency

 | millisecond | Time in ms used to update a single snapshot hook |
| 

metricmanager.snapshotlatency

 | millisecond | Time in ms used to take a snapshot |
| 

vds.distributor.activate\_cluster\_state\_processing\_time

 | millisecond | Elapsed time where the distributor thread is blocked on merging pending bucket info into its bucket database upon activating a cluster state |
| 

vds.distributor.bucket\_db.memory\_usage.allocated\_bytes

 | byte | The number of allocated bytes |
| 

vds.distributor.bucket\_db.memory\_usage.dead\_bytes

 | byte | The number of dead bytes (\<= used\_bytes) |
| 

vds.distributor.bucket\_db.memory\_usage.onhold\_bytes

 | byte | The number of bytes on hold |
| 

vds.distributor.bucket\_db.memory\_usage.used\_bytes

 | byte | The number of used bytes (\<= allocated\_bytes) |
| 

vds.distributor.getbucketlists.failures.busy

 | operation | The number of messages from storage that failed because the storage node was busy |
| 

vds.distributor.getbucketlists.failures.concurrent\_mutations

 | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID |
| 

vds.distributor.getbucketlists.failures.inconsistent\_bucket

 | operation | The number of operations failed due to buckets being in an inconsistent state or not found |
| 

vds.distributor.getbucketlists.failures.notconnected

 | operation | The number of operations discarded because there were no available storage nodes to send to |
| 

vds.distributor.getbucketlists.failures.notfound

 | operation | The number of operations that failed because the document did not exist |
| 

vds.distributor.getbucketlists.failures.notready

 | operation | The number of operations discarded because distributor was not ready |
| 

vds.distributor.getbucketlists.failures.safe\_time\_not\_reached

 | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed |
| 

vds.distributor.getbucketlists.failures.storagefailure

 | operation | The number of operations that failed in storage |
| 

vds.distributor.getbucketlists.failures.test\_and\_set\_failed

 | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document |
| 

vds.distributor.getbucketlists.failures.timeout

 | operation | The number of operations that failed because the operation timed out towards storage |
| 

vds.distributor.getbucketlists.failures.total

 | operation | Total number of failures |
| 

vds.distributor.getbucketlists.failures.wrongdistributor

 | operation | The number of operations discarded because they were sent to the wrong distributor |
| 

vds.distributor.getbucketlists.latency

 | millisecond | The average latency of getbucketlists operations |
| 

vds.distributor.getbucketlists.ok

 | operation | The number of successful getbucketlists operations performed |
| 

vds.distributor.recoverymodeschedulingtime

 | millisecond | Time spent scheduling operations in recovery mode after receiving new cluster state |
| 

vds.distributor.set\_cluster\_state\_processing\_time

 | millisecond | Elapsed time where the distributor thread is blocked on processing its bucket database upon receiving a new cluster state |
| 

vds.distributor.state\_transition\_time

 | millisecond | Time it takes to complete a cluster state transition. If a state transition is preempted before completing, its elapsed time is counted as part of the total time spent for the final, completed state transition |
| 

vds.distributor.stats.failures.busy

 | operation | The number of messages from storage that failed because the storage node was busy |
| 

vds.distributor.stats.failures.concurrent\_mutations

 | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID |
| 

vds.distributor.stats.failures.inconsistent\_bucket

 | operation | The number of operations failed due to buckets being in an inconsistent state or not found |
| 

vds.distributor.stats.failures.notconnected

 | operation | The number of operations discarded because there were no available storage nodes to send to |
| 

vds.distributor.stats.failures.notfound

 | operation | The number of operations that failed because the document did not exist |
| 

vds.distributor.stats.failures.notready

 | operation | The number of operations discarded because distributor was not ready |
| 

vds.distributor.stats.failures.safe\_time\_not\_reached

 | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed |
| 

vds.distributor.stats.failures.storagefailure

 | operation | The number of operations that failed in storage |
| 

vds.distributor.stats.failures.test\_and\_set\_failed

 | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document |
| 

vds.distributor.stats.failures.timeout

 | operation | The number of operations that failed because the operation timed out towards storage |
| 

vds.distributor.stats.failures.total

 | operation | The total number of failures |
| 

vds.distributor.stats.failures.wrongdistributor

 | operation | The number of operations discarded because they were sent to the wrong distributor |
| 

vds.distributor.stats.latency

 | millisecond | The average latency of stats operations |
| 

vds.distributor.stats.ok

 | operation | The number of successful stats operations performed |
| 

vds.distributor.update\_gets.failures.busy

 | operation | The number of messages from storage that failed because the storage node was busy |
| 

vds.distributor.update\_gets.failures.concurrent\_mutations

 | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID |
| 

vds.distributor.update\_gets.failures.inconsistent\_bucket

 | operation | The number of operations failed due to buckets being in an inconsistent state or not found |
| 

vds.distributor.update\_gets.failures.notconnected

 | operation | The number of operations discarded because there were no available storage nodes to send to |
| 

vds.distributor.update\_gets.failures.notfound

 | operation | The number of operations that failed because the document did not exist |
| 

vds.distributor.update\_gets.failures.notready

 | operation | The number of operations discarded because distributor was not ready |
| 

vds.distributor.update\_gets.failures.safe\_time\_not\_reached

 | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed |
| 

vds.distributor.update\_gets.failures.storagefailure

 | operation | The number of operations that failed in storage |
| 

vds.distributor.update\_gets.failures.test\_and\_set\_failed

 | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document |
| 

vds.distributor.update\_gets.failures.timeout

 | operation | The number of operations that failed because the operation timed out towards storage |
| 

vds.distributor.update\_gets.failures.total

 | operation | The total number of failures |
| 

vds.distributor.update\_gets.failures.wrongdistributor

 | operation | The number of operations discarded because they were sent to the wrong distributor |
| 

vds.distributor.update\_gets.latency

 | millisecond | The average latency of update\_gets operations |
| 

vds.distributor.update\_gets.ok

 | operation | The number of successful update\_gets operations performed |
| 

vds.distributor.update\_metadata\_gets.failures.busy

 | operation | The number of messages from storage that failed because the storage node was busy |
| 

vds.distributor.update\_metadata\_gets.failures.concurrent\_mutations

 | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID |
| 

vds.distributor.update\_metadata\_gets.failures.inconsistent\_bucket

 | operation | The number of operations failed due to buckets being in an inconsistent state or not found |
| 

vds.distributor.update\_metadata\_gets.failures.notconnected

 | operation | The number of operations discarded because there were no available storage nodes to send to |
| 

vds.distributor.update\_metadata\_gets.failures.notfound

 | operation | The number of operations that failed because the document did not exist |
| 

vds.distributor.update\_metadata\_gets.failures.notready

 | operation | The number of operations discarded because distributor was not ready |
| 

vds.distributor.update\_metadata\_gets.failures.safe\_time\_not\_reached

 | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed |
| 

vds.distributor.update\_metadata\_gets.failures.storagefailure

 | operation | The number of operations that failed in storage |
| 

vds.distributor.update\_metadata\_gets.failures.test\_and\_set\_failed

 | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document |
| 

vds.distributor.update\_metadata\_gets.failures.timeout

 | operation | The number of operations that failed because the operation timed out towards storage |
| 

vds.distributor.update\_metadata\_gets.failures.total

 | operation | The total number of failures |
| 

vds.distributor.update\_metadata\_gets.failures.wrongdistributor

 | operation | The number of operations discarded because they were sent to the wrong distributor |
| 

vds.distributor.update\_metadata\_gets.latency

 | millisecond | The average latency of update\_metadata\_gets operations |
| 

vds.distributor.update\_metadata\_gets.ok

 | operation | The number of successful update\_metadata\_gets operations performed |
| 

vds.distributor.update\_puts.failures.busy

 | operation | The number of messages from storage that failed because the storage node was busy |
| 

vds.distributor.update\_puts.failures.concurrent\_mutations

 | operation | The number of operations that were transiently failed due to a mutating operation already being in progress for its document ID |
| 

vds.distributor.update\_puts.failures.inconsistent\_bucket

 | operation | The number of operations failed due to buckets being in an inconsistent state or not found |
| 

vds.distributor.update\_puts.failures.notconnected

 | operation | The number of operations discarded because there were no available storage nodes to send to |
| 

vds.distributor.update\_puts.failures.notfound

 | operation | The number of operations that failed because the document did not exist |
| 

vds.distributor.update\_puts.failures.notready

 | operation | The number of operations discarded because distributor was not ready |
| 

vds.distributor.update\_puts.failures.safe\_time\_not\_reached

 | operation | The number of operations that were transiently failed due to them arriving before the safe time point for bucket ownership handovers has passed |
| 

vds.distributor.update\_puts.failures.storagefailure

 | operation | The number of operations that failed in storage |
| 

vds.distributor.update\_puts.failures.test\_and\_set\_failed

 | operation | The number of mutating operations that failed because they specified a test-and-set condition that did not match the existing document |
| 

vds.distributor.update\_puts.failures.timeout

 | operation | The number of operations that failed because the operation timed out towards storage |
| 

vds.distributor.update\_puts.failures.total

 | operation | The total number of put failures |
| 

vds.distributor.update\_puts.failures.wrongdistributor

 | operation | The number of operations discarded because they were sent to the wrong distributor |
| 

vds.distributor.update\_puts.latency

 | millisecond | The average latency of update\_puts operations |
| 

vds.distributor.update\_puts.ok

 | operation | The number of successful update\_puts operations performed |
| 

vds.distributor.mutating\_op\_memory\_usage

 | byte | Estimated amount of memory used by active mutating operations across all distributor stripes, in bytes |
| 

vds.idealstate.nodes\_per\_merge

 | node | The number of nodes involved in a single merge operation. |
| 

vds.idealstate.set\_bucket\_state.blocked

 | operation | The number of operations blocked by blocking operation starter |
| 

vds.idealstate.set\_bucket\_state.done\_failed

 | operation | The number of operations that failed |
| 

vds.idealstate.set\_bucket\_state.done\_ok

 | operation | The number of operations successfully performed |
| 

vds.idealstate.set\_bucket\_state.pending

 | operation | The number of operations pending |
| 

vds.idealstate.set\_bucket\_state.throttled

 | operation | The number of operations throttled by throttling operation starter |
| 

vds.bouncer.clock\_skew\_aborts

 | operation | Number of client operations that were aborted due to clock skew between sender and receiver exceeding acceptable range |

 Copyright © 2026 - [Cookie Preferences](#)