> ## Documentation Index > Fetch the complete documentation index at: https://cantonfoundation-integrate-release.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Monitoring Setup > Participant node health and ACS commitment monitoring This page covers Canton-side monitoring topics: participant node health endpoints and how to monitor ACS commitments. For Splice / Canton Network metrics specifically — what each component exposes, how to scrape it, and which metrics to watch on validators and Super Validators — see: * [Splice Metrics Overview](/global-synchronizer/production-operations/splice-metrics-overview) * [Metrics Reference](/global-synchronizer/reference/metrics-reference) ## Participant Node Health The participant exposes health status information in several ways, which may be inspected manually when troubleshooting or integrated into larger monitoring and orchestration systems. ### Using gRPC Health Service for Load Balancing and Orchestration The Participant Node provides a `grpc.health.v1.Health` service, implementing the [gRPC Health Checking Protocol](https://github.com/grpc/grpc/blob/master/doc/health-checking.md) protocol. Kubernetes containers can be [configured](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#define-a-grpc-liveness-probe) to use this for readiness or liveness [probes](https://kubernetes.io/docs/concepts/configuration/liveness-readiness-startup-probes/#readiness-probe), e.g. ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} readinessProbe: grpc: port: ``` By default the port is the one used for the `Ledger API`. Likewise, [gRPC clients](https://grpc.io/docs/guides/health-checking/#enabling-client-health-checking) and [NGinx](https://docs.nginx.com/nginx/admin-guide/load-balancer/grpc-health-check/) can be configured to watch the health service for traffic management and load balancing. You can manually check the health of a Participant with a command line tool such as [grpcurl](https://github.com/fullstorydev/grpcurl) e.g. (using the Participant's actual address): ```bash theme={"theme":{"light":"github-light","dark":"github-dark"}} $ grpcurl -plaintext : grpc.health.v1.Health/Check { "status": "SERVING" } ``` Calling [Check](https://github.com/grpc/grpc-proto/blob/6565a1ba38af695ace7c3ce6e6ff837ee87d4c10/grpc/health/v1/health.proto#L55) will respond with `SERVING` if it is currently ready and available to serve requests. Calling [Watch](https://github.com/grpc/grpc-proto/blob/6565a1ba38af695ace7c3ce6e6ff837ee87d4c10/grpc/health/v1/health.proto#L72) will perform a streaming health check. The server will immediately send the current health of the Participant, and then send a new message whenever the health changes. When multiple Participant replicas are configured, passive nodes return `NOT_SERVING`. In practice, the health of the Participant is composed of the health of the components it depends on. You can query these individually by name, by making a request with the [service](https://github.com/grpc/grpc-proto/blob/6565a1ba38af695ace7c3ce6e6ff837ee87d4c10/grpc/health/v1/health.proto#L29) field set to the name of the component. An empty or unset `service` field returns the aggregate health of all components. An unknown name will result in a gRPC `NOT_FOUND` error. ### Checking health via HTTP Health checking can also be done via HTTP, which is useful for frameworks that don't support gRPC Health Checking Protocol. Setting monitoring.http-health-server.port= in the configuration for your node will expose health information at the URL `http://:/health`. Here the important information is reported via the HTTP Reponse status code. * A status of `200` is equivalent to `SERVING` from the gRPC Health Service. * A status of `503` is equivalent to `NOT_SERVING`. * A status of `500` means the check failed for any other reason. Kubernetes can use also use these for readiness probes: ```text theme={"theme":{"light":"github-light","dark":"github-dark"}} readinessProbe: httpGet: port: path: /health ``` ### Inspection of General Health Status General information about the Participant Node, including about unhealthy synchronizers and dependencies, and whether the node is currently Active, can be displayed in the canton console by invoking the `health.status` command on the node. ```none theme={"theme":{"light":"github-light","dark":"github-dark"}} @ participant1.health.status res1: NodeStatus[ParticipantStatus] = Participant id: PAR::participant1::12201ff69b1d24edbf0ee2028a304ea702ee8536790dab1a31e7136e6d90ff6d473c Uptime: 2.006305s Ports: ledger: 30145 admin: 30146 json: 30147 Connected synchronizers: None Unhealthy synchronizers: None Active: true Components: memory_storage : Ok() connected-synchronizer : Not Initialized sync-ephemeral-state : Not Initialized sequencer-client : Not Initialized acs-commitment-processor : Not Initialized Version: 3.5.0-SNAPSHOT Supported protocol version(s): 34 ``` The Admin API of the Participant Node provides programmatic access to this data in a structured form, via [ParticipantStatusService](https://github.com/DACH-NY/canton/blob/release-line-3.5/community/admin-api/src/main/protobuf/com/digitalasset/canton/admin/participant/v30/participant_status_service.proto#L10)'s `ParticipantStatus` call. The canton console can also provide information about *all* connected nodes, including those remotely connected, by invoking the command at the top level. ```none theme={"theme":{"light":"github-light","dark":"github-dark"}} @ health.status res2: CantonStatus = Status for Sequencer 'sequencer1': Sequencer id: da::1220a82692abc55c0367abefc4bdbc23df25688230430ddfeef5759845f26d5cc29c Synchronizer id: da::1220a82692abc55c0367abefc4bdbc23df25688230430ddfeef5759845f26d5cc29c::34-0 Uptime: 6.470662s Ports: public: 30149 admin: 30150 Connected participants: PAR::participant2::1220a4d7463b... PAR::participant1::12201ff69b1d... Connected mediators: MED::mediator1::122009299340... Sequencer: SequencerHealthStatus(active = true) details-extra: None Components: memory_storage : Ok() sequencer : Ok() Accepts admin changes: true Version: 3.5.0-SNAPSHOT Protocol version: 34 Status for Mediator 'mediator1': Node uid: mediator1::12200929934059da3e012af672ee8a5d26a7e4b3e5084920be298f791f7619843c78 Synchronizer id: da::1220a82692abc55c0367abefc4bdbc23df25688230430ddfeef5759845f26d5cc29c::34-0 Uptime: 6.405066s Ports: admin: 30148 Active: true Components: memory_storage : Ok() sequencer-client : Ok() sequencer-connection-pool : Ok() sequencer-subscription-pool : Ok() internal-sequencer-connection-sequencer1-0 : Ok() subscription-sequencer-connection-sequencer1-0 : Ok() Version: 3.5.0-SNAPSHOT Protocol version: 34 Status for Participant 'participant1': Participant id: PAR::participant1::12201ff69b1d24edbf0ee2028a304ea702ee8536790dab1a31e7136e6d90ff6d473c Uptime: 8.386186s Ports: ledger: 30145 admin: 30146 json: 30147 Connected synchronizers: da::1220a82692ab...::34-0 Unhealthy synchronizers: None Active: true Components: memory_storage : Ok() connected-synchronizer : Ok() sync-ephemeral-state : Ok() sequencer-client : Ok() acs-commitment-processor : Ok() sequencer-connection-pool : Ok() sequencer-subscription-pool : Ok() internal-sequencer-connection-sequencer1-0 : Ok() subscription-sequencer-connection-sequencer1-0 : Ok() Version: 3.5.0-SNAPSHOT Supported protocol version(s): 34 Status for Participant 'participant2': Participant id: PAR::participant2::1220a4d7463bd34b2ba3704401b48ab41d8f88cdcbe512fc1ef071aad97fef106161 Uptime: 8.670214s Ports: ledger: 30180 admin: 30181 json: 30182 Connected synchronizers: da::1220a82692ab...::34-0 Unhealthy synchronizers: None Active: true Components: memory_storage : Ok() connected-synchronizer : Ok() sync-ephemeral-state : Ok() sequencer-client : Ok() acs-commitment-processor : Ok() sequencer-connection-pool : Ok() sequencer-subscription-pool : Ok() internal-sequencer-connection-sequencer1-0 : Ok() subscription-sequencer-connection-sequencer1-0 : Ok() Version: 3.5.0-SNAPSHOT Supported protocol version(s): 34 ``` ### Generating a Node Health Dump for Troubleshooting See [Health Dumps](/global-synchronizer/reference/observability-configuration#health-dumps) for how to capture and share a health dump when troubleshooting with support. ### Monitoring for Slow or Stuck Tasks Some operations can report when they are slow, if you enable ```none theme={"theme":{"light":"github-light","dark":"github-dark"}} canton.monitoring.logging.log-slow-futures = yes ``` If a task is taking longer than expected, a log line will be emitted periodically until it completes, such as ` has not completed after `. This feature is disabled by default to reduce the overhead. Canton also provides a facility to periodically test whether we are able to schedule new tasks in a timely manner, enabled via the configuration ```none theme={"theme":{"light":"github-light","dark":"github-dark"}} canton.monitoring.deadlock-detection.enabled = yes ``` If a problem is detected, a log line containing `Task runner is stuck or overloaded for ` will be emitted. This may indicate that resources such as CPU are overloaded, that the Execution Context is too small, or that too many tasks are otherwise stuck. If the issue resolves itself, a subsequent log message: `Task runner is just overloaded, but operating correctly. Task got executed in the meantime` will be emitted. Delay logging will log a warning if a node falls behind with processing messages from the sequencer. Such a warning indicates that the node is overloaded. ```none theme={"theme":{"light":"github-light","dark":"github-dark"}} canton.monitoring.logging.delay-logging-threshold = 20s ``` As a rule of thumb, configure the maximum latency, that is, the maximum time it should take Canton to process a command. The default is `20s`. ### Disabling Restart on Fatal Failures Processes should be run under a process supervisor, such as `systemd` or Kubernetes, which can monitor them and restart them as needed. By default, the Participant Node process will exit in the event of a fatal failure. If you wish to disable this behaviour ```none theme={"theme":{"light":"github-light","dark":"github-dark"}} canton.parameters.exit-on-fatal-failures = no ``` which will cause the Node to stay alive and report unhealthy in such cases. ## Monitor ACS Commitments A participant that fails to send commitments in a timely manner is problematic for its counter-participants: Counter-participants cannot prune their state, because they have no proof that their state is the same as the state of the participant. More information on commitments is available in the Pruning overview section. This page describes the monitoring options for ACS commitments. Commitment monitoring supports participant node operators in several ways. First, monitoring provides insight into commitment generation performance, allowing the participant node operator to troubleshoot and fix potential performance problems. For example, monitoring metrics indicate potential performance bottlenecks, which the operator can use as input for configuring commitment generation. Second, monitoring provides insights into the status of commitments from counter-participants. This is relevant for the participant node operator because a counter-participant that runs behind in commitment generation, either because it is faulty or because the network is slow, prevents pruning on the participant: The participant does not know whether its state and the counter-participant's state diverged, and cannot prune because it might need to investigate a potential fork. The operator can use the monitoring metrics to identify slow counter-participants and potentially blacklist them. ### Monitoring own commitments We provide the following metrics for commitment generation, which are described in detail in the Metrics reference section: * `daml.participant.sync.commitments.compute`: Measures the time that the participant node spends computing commitments. * `daml.participant.sync.commitments.sequencing-time`: Measures the time between the end of a commitment period, and the time when the sequencer observes the corresponding commitment. * `daml.participant.sync.commitments.catchup-mode-enabled`: Measures how many times the catch-up mode has been triggered. ### Monitoring counter-participant commitments The operator can monitor the status of commitments from the counter-participants through latency metrics. These metrics can reveal slow counter-participants, which are behind in sending commitments, and enable operators to configure thresholds defining when a counter-participant is considered slow. The operator can group counter-participants into three categories, which affect metric reporting: * *Default* * *Distinguished* * *Individually monitored* An *Individually monitored* counter-participant always shows that participant's commitment latency. *Distinguished* and *Default* groupings of counter-participants only show the largest latency in the group. Inspection tools and direct monitoring can then be used to identify slow counter-participant(s). All metrics below are described in detail in the Metrics reference section. * *Default*: All counter-participants that are not distinguished or individually monitored belong to this group by default. We publish one aggregated metric for all participants in this group: `daml.participant.sync.commitments.largest-counter-participant-latency` which represents the highest latency in microseconds for commitments from counter-participants outstanding for more than a threshold number of reconciliation intervals. * *Distinguished*: The operator has the option to upgrade some default counter-participants to the distinguished group, for example, counter-participants with whom it has important business relations. We produce one aggregate metric for all distinguished participants, published under `daml.participant.sync.commitments.largest-distinguished-counter-participant-latency` Just as for the Default group, the metric represents the highest latency in microseconds for commitments outstanding for more than a `thresholdDistinguished` number of reconciliation intervals. The following examples show how the operator of `participant1` adds counter-participant `participant4` to the distinguished group on synchronizer `synchronizer2Id`, and removes counter-participant `participant2` from the distinguished group on synchronizer `synchronizer1Id`: ```none theme={"theme":{"light":"github-light","dark":"github-dark"}} participant1.commitments.add_config_distinguished_slow_counter_participants( Seq(participant4Id), Seq(synchronizer2Id), ) ``` ```none theme={"theme":{"light":"github-light","dark":"github-dark"}} participant1.commitments.remove_config_distinguished_slow_counter_participants( Seq(participant2.id), Seq(synchronizer1Id), ) ``` * *Individually monitored*: The operator can optionally select counter-participants whose commitment status it wants to monitor individually, for example because they recently presented intermittent failures and have just recovered, or because the operator observes a slowdown in one of the other groups and wants to locate the cause. Each participant gets its own unique label under `daml.participant.sync.commitments.counter-participant-latency`. Individual alerting can be set based on the business relations. (Note: any participant, whether *Default* or *Distinguished*, can be added to *Individually monitored*. A distinguished participant remains in the *Distinguished* group even if it is *Individually monitored*. In contrast, a *Default* participant that is added to *Individually monitored* is removed from the Default group.) The following examples show how the operator of `participant1` adds/removes counter-participant `participant3` to be *Individually monitored* on the synchronizer `synchronizerId`: ```none theme={"theme":{"light":"github-light","dark":"github-dark"}} participant1.commitments.add_participant_to_individual_metrics( Seq(participant3.id), Seq(synchronizerId), ) ``` ```none theme={"theme":{"light":"github-light","dark":"github-dark"}} participant1.commitments.remove_participant_from_individual_metrics( Seq(participant3.id), Seq(synchronizerId), ) ``` The operator of a participant can set the monitoring configuration at once on multiple synchronizers, including thresholds for the *Default* and *Distinguished* groups, as well as for the *Individually monitored*. The example below shows how the operator of `participant1` can apply a monitoring configuration to synchronizers `synchronizer1Id` and `synchronizer2Id`. ```none theme={"theme":{"light":"github-light","dark":"github-dark"}} val update1Config = new SlowCounterParticipantSynchronizerConfig( synchronizerIds = Seq(synchronizer1Id, synchronizer2Id), distinguishedParticipants = Seq(participant3.id), thresholdDistinguished = 15, thresholdDefault = 15, individuallyMonitored = Seq.empty, ) participant1.commitments.set_config_for_slow_counter_participants(Seq(update1Config)) ```