Telemetry

Hyx provides built-in telemetry support to help you monitor your fault tolerance components in production. All components emit events that can be captured by listeners and forwarded to your observability stack.

Telemetry is built on top of Hyx's event system. For details on creating custom listeners or understanding how events flow, see the Events documentation.

Supported Backends

Backend	Installation	Description
OpenTelemetry	pip install hyx[otel]	Industry-standard observability framework
Prometheus	pip install hyx[prometheus]	Popular metrics and alerting toolkit
StatsD	pip install hyx[statsd]	Simple daemon for aggregating statistics

OpenTelemetry

OpenTelemetry is the industry-standard observability framework for cloud-native software.

Installation

pip install hyx[otel]

Quick Start

Register listeners for all components with a single call:

from hyx.telemetry.otel import register_listeners

# Uses the global meter provider
register_listeners()

Or with a custom meter:

from opentelemetry import metrics
from hyx.telemetry.otel import register_listeners

meter = metrics.get_meter("my-service")
register_listeners(meter=meter)

Individual Listeners

You can also register listeners for specific components:

from hyx.telemetry.otel import RetryListener
from hyx.retry import retry

listener = RetryListener()

@retry(attempts=3, listeners=[listener])
async def my_function():
    ...

Metrics Reference

Metric	Type	Labels	Description
hyx.retry.attempts	Counter	component, exception	Number of retry attempts
hyx.retry.exhausted	Counter	component	Retry attempts exhausted
hyx.retry.success	Counter	component	Successful operations
hyx.circuitbreaker.state_transitions	Counter	component, from_state, to_state	State transitions
hyx.circuitbreaker.success	Counter	component, state	Successful operations
hyx.timeout.exceeded	Counter	component	Timeout exceeded
hyx.bulkhead.rejected	Counter	component	Rejected due to capacity
hyx.fallback.triggered	Counter	component, reason	Fallback triggered

Prometheus

Prometheus is an open-source monitoring and alerting toolkit.

Installation

pip install hyx[prometheus]

Quick Start

Register listeners for all components:

from hyx.telemetry.prometheus import register_listeners

# Uses the default global registry
register_listeners()

Or with a custom registry:

from prometheus_client import CollectorRegistry
from hyx.telemetry.prometheus import register_listeners

registry = CollectorRegistry()
register_listeners(registry=registry)

Individual Listeners

from hyx.telemetry.prometheus import CircuitBreakerListener
from hyx.circuitbreaker import consecutive_breaker

listener = CircuitBreakerListener()

breaker = consecutive_breaker(
    failure_threshold=5,
    recovery_time_secs=30,
    listeners=[listener],
)

Metrics Reference

Metric	Type	Labels	Description
hyx_retry_attempts_total	Counter	component, exception	Number of retry attempts
hyx_retry_exhausted_total	Counter	component	Retry attempts exhausted
hyx_retry_success_total	Counter	component	Successful operations
hyx_circuitbreaker_state_transitions_total	Counter	component, from_state, to_state	State transitions
hyx_circuitbreaker_success_total	Counter	component, state	Successful operations
hyx_timeout_exceeded_total	Counter	component	Timeout exceeded
hyx_bulkhead_rejected_total	Counter	component	Rejected due to capacity
hyx_fallback_triggered_total	Counter	component, reason	Fallback triggered

StatsD

StatsD is a simple daemon for aggregating statistics.

Installation

pip install hyx[statsd]

Quick Start

Register listeners for all components:

from hyx.telemetry.statsd import register_listeners

# Uses default client (localhost:8125, prefix='hyx')
register_listeners()

Or with a custom client:

import statsd
from hyx.telemetry.statsd import register_listeners

client = statsd.StatsClient('statsd.example.com', 8125, prefix='myapp')
register_listeners(client=client)

Individual Listeners

import statsd
from hyx.telemetry.statsd import TimeoutListener
from hyx.timeout import timeout

client = statsd.StatsClient(prefix='myapp')
listener = TimeoutListener(client=client)

@timeout(timeout_secs=5, listeners=[listener])
async def slow_operation():
    ...

Metrics Reference

All metrics are prefixed with the client prefix (default: hyx).

Metric	Type	Description
retry.<name>.attempts	Counter	Retry attempt made
retry.<name>.attempts.<exception>	Counter	Retry attempt by exception type
retry.<name>.exhausted	Counter	Retry attempts exhausted
retry.<name>.success	Counter	Successful operation
circuitbreaker.<name>.state.working	Counter	Transitioned to working state
circuitbreaker.<name>.state.recovering	Counter	Transitioned to recovering state
circuitbreaker.<name>.state.failing	Counter	Transitioned to failing state
circuitbreaker.<name>.success	Counter	Successful operation
timeout.<name>.exceeded	Counter	Timeout exceeded
bulkhead.<name>.rejected	Counter	Rejected due to capacity
fallback.<name>.triggered	Counter	Fallback triggered
fallback.<name>.triggered.<reason>	Counter	Fallback by reason (exception/predicate)

Custom Listeners

For creating custom listeners, see the Events documentation.