Querying OpenSearch from .NET: Full-Text Search on DynamoDB Data.

Learn how to query data from Amazon OpenSearch using .NET after importing it from DynamoDB with Zero ETL. This integration allows you to perform full-text searches and advanced queries on DynamoDB data, enhancing your application's search capabilities seamlessly.

Rahul Pulikkot Nath
Rahul Pulikkot Nath

Table of Contents

In this post let's learn how to perform a full-text search on DynamoDB from a .NET application.

Since DynamoDB is not best suited for advanced querying and full-text search capabilities, we first need to get the data from our DynamoDB table into AWS OpenSearch and then query the data from there.

This post will explore how to connect to OpenSearch and search the data from a .NET application.

Thanks to AWS for sponsoring this post in my Amazon DynamoDB Series.

DynamoDB To OpenSearch

In a previous post, we covered how to set up the DynamoDB Zero ETL integration with OpenSearch to transfer data from DynamoDB to OpenSearch automatically.

DynamoDB Zero-ETL Integration With Amazon OpenSearch Service
Let’s learn how to set up the DynamoDB Zero ETL plugin for OpenSearch, a fully managed, no-code setup for ingesting data into Amazon OpenSearch Service.

This article expects you to have this pipeline ready and data flowing from DynamoDB to OpenSearch.

I have a Movie's table in DynamoDB which I have already connected to OpenSearch using the ETL pipeline.

DynamoDB Movie table sample record.

The Movie tables contain the Title, Cast, Description, Genre, Year, Rating, IsAvailableForStreaming, and a few other properties.

Let's set up our .NET application to query these data from Amazon OpenSearch.

Amazon OpenSearch and .NET

OpenSearch supports two .NET clients - a low-level OpenSearch.NET client and a high-level OpenSearch.Client client.

Both are available as NuGet packages to add to your application and start using.

For this post, we will use the OpenSearch.Client high-level NuGet package.

Connecting to Amazon OpenSearch from .NET

The OpenSearch.Client package provides the OpenSearchClient to interact with OpenSearch.

It takes in a ConnectionSettings class that you can configure with the connection and authentication details.

var settings = new ConnectionSettings(
        new Uri("<OPEN SEARCH URL>"),
        new AwsSigV4HttpConnection())
    .DefaultIndex("<OPEN SEARCH INDEX>")
    .DefaultFieldNameInferrer(p => p);
    
var client = new OpenSearchClient(settings);
builder.Services.AddSingleton<IOpenSearchClient>(client);

By default, the client uses camelCasing for the property names. Based on how you have your data in OpenSearch you will need to adjust this using the DefaultFieldNameInferrer.

In my case, I have the property names as PascalCased, hence overriding it to use the same names as my C# property names..

Authenticating with Amazon OpenSearch from .NET

Since I am using Amazon OpenSearch, we can use the AWS Credentials to authenticate with OpenSearch.

First, ensure that the appropriate role/user your application is running under has permission to talk to OpenSearch.

This is very similar to how we added the role ARN to OpenSearch under Security → Roles → all_access → Map user.

Learn How To Manage Credentials When Building .NET Application on AWS
Learn different ways to set up and manage credentials and sensitive information when building .NET applications on AWS. We will also touch upon some of the tools and utilities that I have set up on my local development machine to make working with AWS easier.

OpenSearch clients support the ability to sign requests using AWS Signature V4.

For this, use the overloaded constructor of ConnectionSettings and pass in the AwsSigV4HttpConnection instance. This creates a new connection, discovering both the credentials and region from the environment.

Querying Amazon OpenSearch from .NET

To query the index, I have a movie class defined below that reflects the properties available in the OpenSearch index.

public class Movie
{
    public int IsAvailableForStreaming { get; set; }
    public string[] Genre { get; set; }
    public string Description { get; set; }
    public string[] Languages { get; set; }
    public double Rating { get; set; }
    public int Year { get; set; }
    public string[] Cast { get; set; }
    public string Title { get; set; }
}

Below is a sample ElasticSearch query that searches the movies index for documents where the specified term appears in either the Title or Description fields, returning matching results.

GET  movies/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "<QUERY TERM HERE>",
            "fields": ["Title", "Description"]
          }
        }
      ]
    }
  }
}

To convert this to the OpenSearch.Client .NET format we need to use the SearchDescriptor class combined with the QueryContainer class as shown below.

var searchDescriptor = new SearchDescriptor<Movie>()
    .Query(q =>
    {
        var mustQueryContainer = new List<QueryContainer>();
        if (!string.IsNullOrEmpty(request.Query))
        {
            mustQueryContainer.Add(q.MultiMatch(m => m
                .Fields(f => f.Field(ff => ff.Title).Field(ff => ff.Description))
                .Query(request.Query)
            ));
        }
     
        return q.Bool(b => b
                .Must(mustQueryContainer.ToArray())
        );
    });

     var response = await searchClient.SearchAsync<Movie>(searchDescriptor)

     return response.Documents;

We can then use the SearchAsync method on the OpenSearchClient and pass the searchdescriptor instance of the query.

This returns a list of movies matching the searchdescriptor criteria specified.

Advanced OpenSearch Query Example using .NET

Below is a more advanced query on the OpenSearch Index that searches for documents where the term "avengers" appears in the Title or Description fields, and filters the results to include only those that are available for streaming and have a Rating of 8 and above.

GET  movies/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "multi_match": {
            "query": "avengers",
            "fields": ["Title", "Description"]
          }
        }
      ],
      "filter": [
        {
          "term": {
            "IsAvailableForStreaming": 1
          }
        },
        {
          "range": {
            "Rating": {
              "gte": 8
            }
          }
        }
      ]
    }
  }
}

Below is the equivalent of it in the OpenSearch .NET client.

var searchDescriptor = new SearchDescriptor < Movie > ()
  .Query(q => {
    var mustQueryContainer = new List < QueryContainer > ();
    var filterQueryContainer = new List < QueryContainer > ();
    
    // Search Title and Description with the Query
    if (!string.IsNullOrEmpty(request.Query)) {
      mustQueryContainer.Add(q.MultiMatch(m => m
        .Fields(f => f.Field(ff => ff.Title).Field(ff => ff.Description))
        .Query(request.Query)
      ));
    }
    
    // Filter by IsAvailableForStreaming
    if (request.IsAvailableForStreaming.HasValue) {
      filterQueryContainer.Add(q.Term(
        t => t.Field(f => f.IsAvailableForStreaming)
        .Value(request.IsAvailableForStreaming.GetValueOrDefault() ? 1 L : 0 L)));
    }
    
    // Filter by MinRating
    if (request.MinRating.HasValue) {
      filterQueryContainer.Add(
        q.Range(r => r.Field(f => f.Rating).GreaterThanOrEquals(request.MinRating)));
    }
    
    return q.Bool(b => b
      .Must(mustQueryContainer.ToArray()) // Scoring conditions
      .Filter(filterQueryContainer.ToArray()) // Non-scoring conditions
    );
  });
  
var response = await openSearchClient.SearchAsync < Movie > (searchDescriptor);
return response.Documents;
AWSDynamoDB