Offloading the Event Loop: BullMQ Architecture in Node.js (Part 1)

Introduction

In many Node.js applications, there are tasks don’t need to executed in main flow, like video/image processing, sending emails and others. Execute these tasks inside main flow ( main thread ), increase the possibility of causing bottleneck and that will affect negativly the system performance.
So let’s discuss the importance of background jobs processing.

Motivation Problem

Imagine this scenario, your app sends an email after user signup / verify otp / reset password. The flow might be:

• User send a request to application.
• The app prepare and send a email (user hasn’t received it yet).
• The app waits for email to be sent successfully (If it fails? -> retries).

However, while the email itself is important, sending it immediately within the same request is not strictly required. It can be handled a bit later without affecting the user experience.

Over time, this cause:

• increase delay of response.
• Accumulate all of delays from tasks.
• Unneeded overhead/pressure on main thread.

Native Solution

This approach to handle everything manually inside same process, optimize async functions, implement retries logic for failed tasks, handle task status and it’s errors. But the event loop is still pressured, accumulate heavy tasks delay, no scalability.

Enhanced Solution

The target of this blog to present this solution, execute background tasks through message queue.

The message queue simple flow is:

• The main app fire/add a job.
• The job is enqueued to a queue.
• A worker start process the top tasks of queue one but one.

With message queue the main app can continue recieve requests, without waiting task unneeded to executed immediately.

There are several options to use Message queue, likes BullMQ, RabbitMQ, Kafka, others.

But for now, we’ll discuss BullMQ.

What is BullMQ

BullMQ is a Node.js library used to offload tasks from your main app, especially for background tasks, that uses Redis to store jobs information and offers running multiple jobs concurrently, retrying failed jobs, horizontal scaling, and prioritizing jobs.

BullMQ Components

Queue

The job handler, communicate with Redis to store job info, manage job state (waiting, completed, failed), also acts as a bridge between producer (main app) and worker (processor or executer).

Worker

The job executer, listens to queue to pull jobs, notify the queue events component with job status (completed or failed), can run in multiple instances (separate processes) with same Redis server that helps to scale horizontally.

Queue Events

The job observation, known job status in queue and workers, communicate with Redis to store it, used for logging/monitoring.

Understanding the architecture is important, but seeing it in action is where things start to make sense.

In the next part, we’ll implement a simple example using BullMQ and see how these components work together in a real application.