In Brief: The Circuit Breaker Pattern
- Dhanushka Madushan
- Software Engineer - WSO2
Say you’re running a web service that requires input. In turn, the web service delivers it to another backend service. If the backend service isn’t available for an extended time, do you have a fail-proof system in place? This is where the circuit breaker design pattern comes in.
Consider a situation when a request goes to a middleware application, and you need to call another remote backend service. If everything goes smoothly, the application can forward the request to the backend service and send the response back to the client. But, if the backend service is down, the request will not be completed. Even if the client makes subsequent requests to call the backend service, all of them will fail. Here, we can use the circuit breaker model to manage backend errors. As shown in Figure 1, according to the circuit breaker pattern, the middleware may be in the following states:
Figure 1: Circuit Breaker Pattern
- Closed State: Just like electronic circuits, the closed state is the default state. If users request a flow through the specified route and there are no system failures, the application will be up and running. Here, the client’s application was sent to the backend and its reply was sent back to the client.
- Open State: Here, the client’s request goes to the middleware application, but the backend server is not available. Once the application enters this state, no upcoming requests will be sent to the backend server for a specified time. Instead of attempting to call the backend service, the application responds to the client with an error message.
- Half-Open State: After the application in the open state is given a timeout, it goes into a half-open state. Here, the application attempts to send the request to the backend again, and verify whether the backend service is accessible or not. If it is accessible, the request will move to the closed state. If the backend is still inaccessible, it will go back to the open state.
Why the Circuit Breaker Pattern?
You may have a backend application that executes a database query and sends the outcome back to the client. The number of individuals sending transactions to your application can be as high as 10,000 TPS (transactions per second). However, the backend server may crash and slowly recover. However, the transactions will still keep coming. To manage this situation, the system should be designed in a way that waits for a while until the service is available again. Once it is, requests can be sent.
In the circuit breaker pattern, if there is a failure, it waits for a timeout and attempts to send a request to the backend service after the timeout. The idea behind this pattern is to encapsulate the logic of managing unforeseen mistakes. This pattern is particularly helpful in circumstances such as database migration and software updates.
Circuit Breakers in Microservices
Containers/Virtual machines (VM) are dynamic in a microservices architecture. One container could disappear, and in the meantime, another container could be launched. The concept of microservices highlights mentions there could be a component failure, and the microservices architecture handles these failures. Expecting errors will make your software architecture more resilient.
Here are some practical complex scenarios where load balancing circuit breakers are used. To explain and provide examples, we will use NGINX as the circuit breaker and WSO2 Micro Integrator Circuit Breaker Pattern. NGINX microservices reference architecture has a resizer service that can resize, rotate, and shrink an image when uploaded to the system. This resizer operation is a CPU and memory-intensive job that can cause memory errors. Here, we can place the circuit breaker between the resizer and the image uploader, as shown by Figure 2.
Figure 2: Circuit Breaker in NGINX
Here, the uploader service examines the health of the resizer service. It checks if the resizer has sufficient memory to perform operations. If NGINX observes the service to be defective, the uploader will distribute loads among other instances. Once the resizer becomes ‘healthy’, the NGINX gradually ramps back up to the restarted resizer.
Implementing the Circuit Breaker Pattern
Here’s the practical application of the WSO2 Micro Integrator Circuit Breaker Pattern. This is an open-source middleware platform used to interconnect distinct web services. The integrator is between two endpoints. One endpoint (the client) sends a request to the other endpoint (the backend). The integrator performs the required conversion to the request through a sequence of mediators, before sending it to the backend endpoint. This scenario can be implemented with the following configurations on WSO2 Micro Integrator.
<api name="ServiceAPI" context="/cal">
<resource methods="GET">
<inSequence>
<send>
<endpoint key="CircuitBreakerEP"/>
</send>
</inSequence>
<outSequence>
<send/>
</outSequence>
<faultSequence>
<header name="To" action="remove"></header>
<property name="RESPONSE" value="true"></property>
<property name="NO_ENTITY_BODY" scope="axis2" action="remove"></property>
<log level="full"></log>
<payloadFactory media-type="xml">
<format>
<ns:MyResponse xmlns:ns="https://services.samples">
<ns:Error>We can't response you at this time, we will reponse through E-mail soon</ns:Error>
</ns:MyResponse>
</format>
</payloadFactory>
<send/>
</faultSequence>
</resource>
</api>
Here, we identify the Service API that forwards the incoming application to the backend endpoint. The mediator, <inSequence> processes the incoming request. Inside, the mediator redirects the application to the endpoint, TimeoutEP. The response to this endpoint will be directed to the <outSequence> mediator. Here, the backend reply is sent back to the client. Next, the <faultSequence> section replies to information if failures exist.
The endpoint definition would be as follows.
<endpoint name="CircuitBreakerEP">
<address uri="https://localhost:9764/CalculatorService-war_1.0.0/services/calculator_service/call">
<suspendOnFailure>
<initialDuration>40000</initialDuration>
</suspendOnFailure>
<markForSuspension>
<errorCodes>101507,101508,101505,101506,101509,101500,101510,101001,101000,101503,101504,101501</errorCodes>
<retriesBeforeSuspension>3</retriesBeforeSuspension>
<retryDelay>400</retryDelay>
</markForSuspension>
<timeout>
<duration>200</duration>
<responseAction>fault</responseAction>
</timeout>
</address>
</endpoint>
In the endpoint section, we can define timeout parameters for the circuit breaker pattern. The Address URL is the backend URL to which the request is redirected to. Here’s a list of configurable parameters:
- initialDuration: This defines how long the system should wait until the service retries the backend service, usually in milliseconds.
- retriesBeforeSuspension: This defines how many times the API is called before going into a suspended state, aka the “Open Circuit” state.
- retryDelay: This refers to the delay between two failure calls.
Conclusion
The circuit breaker design pattern is a basic pattern used in monolithic and microservice-based deployments. It helps systems prevent themselves from sending unnecessary loads to a failed backend service, by providing a delay to recover from errors. It plays a key role in the microservices architecture as its containers are dynamic and can stop working at any time. The middleware platform used to manage traffic should be ready to manage the application when the backend server goes down. Circuit breaker patterns provide an elegant way to handle errors.
To learn more about improving your product resilience in WSO2 API Manager, click here.