Bulkheads

Introduction

The cloud engineering world loves ocean and ship analogies: Kubernetes, Docker, containers, Helm, pods, Harbor, Spinnaker, Werf, Shipwright, etc. One word from that vocabulary has been adopted by resiliency engineering as well: the bulkhead.

A bulkhead (a.k.a. bulwark) can be viewed as a virtual room of a certain capacity. That capacity represents the resources you allow to be used simultaneously to process a given action. You can define multiple bulkheads for different functionality in your microservice, ensuring that one part of the system won't consume resources at the expense of another.

There are different ways to implement bulkheads:

• In multithreaded applications, it may take the form of a queue with a fixed-size worker pool
• In single-threaded, event-loop-based applications (like Hyx), it takes the form of concurrency limiting

Hence, the bulkhead is essentially a concurrency limiting mechanism. In turn, concurrency limiting can be seen as a form of rate limiting.

Use Cases

• Limit the number of concurrent requests in one part of the microservice, preventing it from consuming resources needed by other parts during load spikes
• Shed excessive load from the microservice

Usage

decoratorcontext manager

from fastapi import FastAPI

from hyx.bulkhead import bulkhead

# app is an instance of framework like Flask or FastAPI
app = FastAPI()


@app.post("/projects/")
@bulkhead(max_concurrency=10, max_capacity=100)
def create_new_project():
    """
    Create a new project
    """
    ...

from fastapi import FastAPI

from hyx.bulkhead import bulkhead

# app is an instance of framework like Flask or FastAPI
app = FastAPI()

limiter = bulkhead(max_concurrency=10, max_capacity=100)


@app.post("/projects/")
def create_new_project():
    """
    Create a new project
    """
    async with limiter:
        ...

class hyx.bulkhead.bulkhead(max_concurrency, max_capacity, *, name=None, listeners=None, event_manager=None)

Bulkhead defines a fix-size action "queue" to constrain number of times the operation is executed at the same time as well as number of requests to postpone/queue. Bulkhead can be seen as a throttling mechanist for parallel executions.

Parameters

• max_concurrency (int) - Max executions at the same time. If the number is exceeded and max_execs allows, remaining executions are going to be queued
• max_capacity (int) - Overall max number of executions (concurrent and queued). If the number is exceeded, new executions are going to be rejected

Adaptive Limiting

Concurrency can be limited adaptively based on latency statistics from completed requests and a latency objective.

Note

Hyx doesn't provide an ARC implementation at this moment. Let us know if this is useful for you.