NServiceBus on AWS SQS: Learn How to Quickly Get Started

This article is sponsored by AWS and is part of my AWS Series.

NServiceBus is a messaging framework provided by Particular Software.

It makes it quick and easy to send, process, and publish messages across various on-premise and cloud-based queuing technologies.

With all the low-level serialization, threading, and transaction management handled out of the box, all we need is to add the NuGet package and start building message-driven applications.

In this article, let’s learn how to

• Get started using NServiceBus and core concepts.
• Send and process messages in a .NET application.
• Use AWS SQS as the underlying transport.
• Handling large message sizes in SQS transport

NServiceBus and Important Concepts

Before diving into any code, let's understand some of the core terms used in building message-based applications and NServiceBus.

• Message: Commands and Events are generally classified as Messages. A Command is used to request an action must be taken, whereas an Event indicates an action has happened.
  
• Endpoint: An Endpoint is a logical entity that sends/receives messages with other Endpoints. An endpoint is identified by a name. A deployed endpoint is referred to as an instance.
• Hosting: Hosting is how an endpoint instance is run on a process. Multiple endpoint instances can be hosted in the same process.
• Transport: Transport is the mechanism by which Endpoints communicate with each other. NServiceBus supports different transport mechanisms and can be easily plugged in when configuring the bus.

Install and Setup NServiceBus

To get started with NServiceBus, all we need to do is install the NuGet package.

Once installed, we need to configure NServiceBus to set up the Endpoint and Transport. In this blog post, we will see how to set up an application to send messages and another to receive and process the message using Amazon SQS as the underlying transport.

The configuration is similar when using other Transport mechanisms.

So, let's get started.

Code Organization for Sender & Receiver

When building message-based applications, there's usually a Sender that sends a message which is then processed asynchronously by the Receiver application.

Before creating the Sender/Receiver, let's first define the Message structure and the associated class.

NServiceBus suggests having the  Messages to be self-contained.

Create a separate class-library project and have your Events and Commands defined inside the assembly. Add a reference to the NServiceBus library to define the message types.

In this example, I have the user-messages project, which has a reference to the NServiceBus NuGet package.

I have one Message defined - CreateUserCommand as shown below.

public class CreateUserCommand : ICommand
{
    public Guid Id { get; set; }
    public string Name { get; set; }

    public CreateUserCommand(Guid id, string name)
    {
        Id = id;
        Name = name ?? throw new ArgumentNullException(nameof(name));
    }
}

The CreateUserCommand is used to create a new user in the system.

Setting Up Receiver on Amazon SQS Transport

To decouple the request to create a user and the actual creation of the user, we can create a background process to handle the user creation.

In the solution, the user-service project is a console application that runs independently of the application that requests the creation of new users ( user-api ).

In the Program.cs of the console application, we can set up the initialization code for NServiceBus.

It sets up a new Endpoint by specifying  EndpointConfiguration and configuring the transport. The Endpoint is named "user-service" and uses Amazon SQS as its underlying transport.

The EnableInstallers method on the EndpointConfiguration ensures that all the associated infrastructure is automatically set up if it does not already exist.

using NServiceBus;

Console.WriteLine("Hello, World!");

var endpointConfiguration = new EndpointConfiguration("user-service");
var transport = endpointConfiguration.UseTransport<SqsTransport>();
endpointConfiguration.EnableInstallers();

var endointInstance = await Endpoint.Start(endpointConfiguration);

Console.WriteLine("Press any key to stop");
Console.ReadKey();

await endointInstance.Stop();

The CreateUserCommandHandler is responsible for handling and processing CreateUserCommand when that message arrives in the queue.

The handler class, by convention, must implement the IHanleMessages<> interface, specifying the message it handles. The Handle method has the necessary logic required to process the message.

The receiver logs the message contents in this example, as shown below.

internal class CreateUserCommandHandler : IHandleMessages<CreateUserCommand>
{
    static ILog log = LogManager.GetLogger<CreateUserCommandHandler>();

    public async Task Handle(
    CreateUserCommand message, IMessageHandlerContext context)
    {
        log.Info($"Processed message for user with Id {message.Id}, and {message.Name}");

        // Create the user
        // Save To database/persistent store
    }
}

Underlying Infrastructure with Amazon SQS

When running the Receiver application for the first time, it automatically configures the infrastructure required.

Below you can see two Amazon SQS Queue is automatically created

• user-service  → For messages sent to the Endpoint
• error → For messages that fail to process by the Receiver

You can see these resources in the Amazon Console or from Visual Studio if you have AWS Toolkit installed and set up.

If you start one more instance of the Receiver Endpoint, it listens to the same user-service SQS Queue. There is only one Queue created, and all the instances process the message of the same Queue.

Creating more instances is useful when there are a lot of messages to process, and you can scale out to process the messages faster.

Setting Up Sender to Send Messages To SQS

With the Receiver ready to process the messages coming into the Amazon SQS Queue, let's start sending some messages.

The user-api project is an ASP NET Core Web API application configured to publish messages.

The Program.cs configures the NServiceBus Endpoint ('user-api') and the Transport as before.

Since the API does not process any messages and is responsible for only sending messages, the EndpointConfiguration is marked as, as shown below.

We also need to configure the Routing to configure the endpoint to which the message must be sent. In this case, the message needs to be sent to the user-service responsible for processing the messages.

builder.Host.UseNServiceBus(context =>
{
    var endpointConfiguration = new EndpointConfiguration("user-api");
    var transport = endpointConfiguration.UseTransport<SqsTransport>();
    transport.Routing().RouteToEndpoint(typeof(CreateUserCommand), "user-service");
    endpointConfiguration.SendOnly();

    return endpointConfiguration;
});

The UserController has a POST method, which uses the IMessageSession instance from NServiceBus to send the message. It creates a new CreateUserCommand message instance and sends it to the bus.

public UserController(IMessageSession session, ILogger<UserController> logger)
{
    this.session = session;
    _logger = logger;
}

[HttpPost]
public async Task Post(UserData userData)
{
    await session.Send(new CreateUserCommand(userData.Id, userData.Name));
}

Since we are publishing the same message type and with the Routing information, NServiceBus knows where to send the message. It places the message in the Amazon SQS Queue created by the Receiver application.

As soon as the message is dropped in the queue, the Receiver application will pick it up for processing and hand it over to the message handler. It processes the messages and writes the output to the console.

Large Message Sizes with NServiceBus and SQS Transport

Amazon SQS has a maximum message size of 256KB.

This means if the message sent from NServiceBus is larger than 256KB, it will have an error when sending it to the SQS Queue.

Messages containing content like images or document streams can easily exceed the maximum size limit size.

In this case, the application throws an error when publishing the message to the Amazon SQS Queue.

NServiceBus provides out-of-the-box support for these scenarios by allowing us to send these messages using Amazon S3. The actual message content gets written into S3, and a link to it is shared as part of the message.

As shown below, setting up Amazon S3 for large messages is easily done by single-line configuration.

transport.S3(
    bucketForLargeMessages: "user-service-large-messages",
    keyPrefix: "user");

The receiver uses the Amazon S3 document link to retrieve the message and pass it on to the message handler.

Additional Reading

• NServiceBus Step-by-step
• Amazon SQS Transport
• NServiceBus Pricing
  

Full source code sample available here.