Timeouts

Introduction

Not all actions or requests complete immediately. Most actions take some time to finish, and that time may depend on various factors like user input, current load, etc. At the same time, we don't want to wait forever, as waiting consumes resources, and resources are limited.

One way to address this problem is to specify a timeout: the maximum time you're willing to wait for an action to complete.

Some functionality comes with built-in timeouts. For example, if you're using an HTTP or RPC client, chances are there's a timeout parameter available:

• HTTPX: Timeouts
• AIOHTTP: Timeouts
• gRPC: ClientCallDetails
• IOTHRIFT: Client Timeout

For situations where this is not the case, Hyx provides a decorator and a context manager to help with setting up timeouts.

Use Cases

• Ensure that the caller waits no longer than a given delay. Local timeouts work best when limiting actions that don't touch other microservices.
• Limit a microservice request chain by using a distributed timeout

By Locality

Local Timeouts

decoratorcontext manager

import asyncio

from hyx.timeout import timeout


@timeout(1)
async def copy_directory(source_dir: str, destination_dir: str) -> None:
    """
    Copy a huge directory
    """
    ...


asyncio.run(copy_directory(source_dir="~/datasets", destination_dir="~/data"))

import asyncio

from hyx.timeout import timeout


async def copy_directory(source_dir: str, destination_dir: str) -> None:
    """
    Copy a huge directory
    """
    async with timeout(1):
        ...


asyncio.run(copy_directory(source_dir="~/datasets", destination_dir="~/data"))

class hyx.timeout.timeout(timeout_secs, *, name=None, listeners=None, event_manager=None)

Timeout Decontext

Parameters:

• timeout_secs (float) - Max amount of time to wait for the action in seconds
• name (None | str) - A component name or ID (will be passed to listeners and mention in metrics)
• listeners (None | Sequence[TimeoutListener]) - List of listeners of this concreate component state

Warning

For the sake of simplicity, Hyx assumes that you are following AsyncIO best practices and not running CPU-intensive operations in the main thread. Otherwise, the timeout functionality may fire with a delay after the thread is unblocked.

Distributed Timeout

Distributed systems make things more complicated when it comes to timeouts. When you send a request to a downstream service with a timeout, there's a possibility that the request will be queued for a while. As a result, the caller may give up waiting and move on to handling other requests.

But what eventually happens to that queued request? If it doesn't fail at the load balancing level, it will simply be processed when its turn comes. This wastes resources since the result won't be used anyway.

A common solution is to let all microservices in the request chain know how much time they have to respond. If that limit is exceeded, microservices can ignore requests that have already timed out. This is called a distributed timeout (a.k.a. deadlines or timeout budget).

Info

Hyx doesn't yet provide a way to easily add a distributed budget, as it would require integration with the API framework of your choice. We have some plans to support that. Let us know if this is useful for you.