Elasticsearch Provider

Use Elasticsearch as a vector search backend for retrieval-augmented generation.

Quick Start

builder.Services.AddCoreElasticsearchServices(
    builder.Configuration.GetSection("CrestApps:Elasticsearch"));

Or without configuration binding:

builder.Services.AddCoreElasticsearchServices();

Configuration

`appsettings.json`

{
  "CrestApps": {
    "Search": {
      "Elasticsearch": {
        "Url": "https://localhost:9200",
        "CloudId": "",
        "AuthenticationType": "Basic",
        "Username": "elastic",
        "Password": "your-password",
        "ApiKey": "",
        "Base64ApiKey": "",
        "ApiKeyId": "",
        "CertificateFingerprint": "AA:BB:CC:..."
      }
    }
  }
}

`ElasticsearchConnectionOptions`

Property	Type	Description
`Url`	`string`	Elasticsearch endpoint URL
`CloudId`	`string`	Elastic Cloud deployment identifier (optional alternative to `Url`)
`AuthenticationType`	`string`	`None`, `Basic`, `ApiKey`, `Base64ApiKey`, or `KeyIdAndKey`
`Username`	`string`	Basic auth username (optional)
`Password`	`string`	Basic auth password (optional)
`ApiKey`	`string`	Raw API key value for `ApiKey` auth, or the key portion for `KeyIdAndKey`
`Base64ApiKey`	`string`	Base64-encoded API key value for `Base64ApiKey` auth
`ApiKeyId`	`string`	API key identifier for `KeyIdAndKey` auth
`CertificateFingerprint`	`string`	TLS certificate fingerprint for verification (optional)

Services Registered (Keyed by `"Elasticsearch"`)

Service	Implementation
`IDataSourceContentManager`	`ElasticsearchDataSourceContentManager`
`IDataSourceDocumentReader`	`DataSourceElasticsearchDocumentReader`
`IODataFilterTranslator`	`ElasticsearchODataFilterTranslator`
`ISearchIndexManager`	`ElasticsearchSearchIndexManager`
`ISearchDocumentManager`	`ElasticsearchSearchDocumentManager`

When the Url is provided, an ElasticsearchClient singleton is also registered.

AI-specific Elasticsearch registrations now live in CrestApps.Core.AI.Elasticsearch. Register that package when you need AddAIDocuments(), AddAIDataSources(), AddAIMemory(), or Elasticsearch-backed AI RAG/search flows.

When you call AddAIDataSources(), the feature builder also pulls in the shared asynchronous data-source synchronization stack from AddCoreAIDataSourceRag(), including:

IAIDataSourceIndexingQueue
IAIDataSourceIndexingService
AIDataSourceCatalogIndexingHandler
AIDataSourceSearchDocumentHandler
AIDataSourceIndexingBackgroundService
AIDataSourceAlignmentBackgroundService
DataSourceSearchIndexProfileHandler

Override IAIDataSourceIndexingQueue when you need a durable or distributed queue, override IAIDataSourceIndexingService when you need different synchronization rules, and add your own ISearchDocumentHandler registrations when source-index writes should trigger additional asynchronous work.

External Elasticsearch source mappings

When an AIDataSource uses Source = "Elasticsearch", the mapping reads documents from a remote Elasticsearch index using source-specific settings stored on the AIDataSource itself:

Url
CloudId
EnvironmentType (SelfManaged or CloudHosted)
AuthenticationType (None, Basic, ApiKey, Base64ApiKey, or KeyIdAndKey)
IndexName
Username (when AuthenticationType = "Basic")
Password (protected at rest when AuthenticationType = "Basic")
ApiKey (protected at rest when AuthenticationType = "ApiKey" or KeyIdAndKey)
Base64ApiKey (protected at rest when AuthenticationType = "Base64ApiKey")
ApiKeyId (when AuthenticationType = "KeyIdAndKey")
CertificateFingerprint

Use EnvironmentType = "SelfManaged" with Url for self-managed clusters or endpoint-based hosted deployments. Use EnvironmentType = "CloudHosted" with CloudId for Elastic Cloud hosted deployments. Older records that only store CloudId are still inferred as cloud-hosted. Elastic Cloud connections require one of the authenticated modes.

This is different from the Elasticsearch knowledge-base backend configuration. The backend settings under CrestApps:Search:Elasticsearch define where the embedded knowledge-base chunks are written. The source mapping settings define where the raw source documents are read from.

Because the remote source index is externally managed, document changes must be pushed into the sync pipeline through IAIDataSourceChangeNotifier. See Custom Sources for the notification pattern.

Docker Setup for Local Development

Use Docker Compose to run Elasticsearch locally with vector search support:

docker-compose.yml
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.15.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=true
      - xpack.security.http.ssl.enabled=false
      - ELASTIC_PASSWORD=changeme
    ports:
      - "9200:9200"
    volumes:
      - es-data:/usr/share/elasticsearch/data
    mem_limit: 2g

volumes:
  es-data:
    driver: local

Start it with:

docker compose up -d

tip

Setting xpack.security.http.ssl.enabled=false simplifies local development by disabling HTTPS. For production, always enable TLS and use the CertificateFingerprint option.

Then configure your appsettings.Development.json:

{
  "CrestApps": {
    "Search": {
      "Elasticsearch": {
        "Url": "http://localhost:9200",
        "Username": "elastic",
        "Password": "changeme"
      }
    }
  }
}

Configuration Reference

Full `appsettings.json` Example

{
  "CrestApps": {
    "Search": {
      "Elasticsearch": {
        "Url": "https://my-cluster.es.us-east-1.aws.found.io:9243",
        "CloudId": "",
        "AuthenticationType": "Basic",
        "Username": "elastic",
        "Password": "your-secure-password",
        "ApiKey": "",
        "Base64ApiKey": "",
        "ApiKeyId": "",
        "CertificateFingerprint": "AA:BB:CC:DD:EE:FF:00:11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33:44:55:66:77:88:99"
      }
    }
  }
}

`ElasticsearchConnectionOptions` — All Properties

Property	Type	Required	Default	Description
`Url`	`string`	Conditionally	—	Elasticsearch endpoint URL. Include the port if non-standard (e.g., `https://localhost:9200`). Required when `CloudId` is empty.
`CloudId`	`string`	Conditionally	—	Elastic Cloud deployment identifier. Required when `Url` is empty.
`AuthenticationType`	`string`	No	`None`	Selects `None`, `Basic`, `ApiKey`, `Base64ApiKey`, or `KeyIdAndKey`. Elastic Cloud requires one of the authenticated modes.
`Username`	`string`	No	—	Username for basic authentication. Typically `"elastic"` for the built-in superuser.
`Password`	`string`	No	—	Password for basic authentication.
`ApiKey`	`string`	No	—	Raw API key for `ApiKey` auth, or the key portion for `KeyIdAndKey`.
`Base64ApiKey`	`string`	No	—	Base64-encoded API key value for `Base64ApiKey` auth.
`ApiKeyId`	`string`	No	—	API key identifier for `KeyIdAndKey` auth.
`CertificateFingerprint`	`string`	No	—	SHA-256 fingerprint of the Elasticsearch TLS certificate. Required when using self-signed certificates. Format: `AA:BB:CC:...`

info

When Url is provided, the framework registers an ElasticsearchClient singleton that all keyed services share. If Url is empty or null, no client is registered and the data source is effectively disabled.

Verification

After configuring the connection, verify it is working:

1. Check Elasticsearch Health

curl -u elastic:changeme http://localhost:9200/_cluster/health?pretty

Expected output should show "status": "green" or "yellow".

2. Verify from the Application

Inject ISearchIndexManager (keyed by "Elasticsearch") and check if the connection is live:

public sealed class ElasticsearchHealthCheck
{
    private readonly ISearchIndexManager _indexManager;

    public ElasticsearchHealthCheck(
        [FromKeyedServices("Elasticsearch")] ISearchIndexManager indexManager)
    {
        _indexManager = indexManager;
    }

    public async Task<bool> IsHealthyAsync()
    {
        // Attempt to check if a known index exists
        return await _indexManager.ExistsAsync("_test_ping");
    }
}

3. Check Indexes via Elasticsearch API

# List all indexes
curl -u elastic:changeme http://localhost:9200/_cat/indices?v

# Check a specific index mapping
curl -u elastic:changeme http://localhost:9200/your-index-name/_mapping?pretty

Index Management

Indexes are created automatically when a data source is configured and content is indexed for the first time. The ElasticsearchSearchIndexManager handles index lifecycle:

Creation — CreateAsync() defines the index schema with vector fields (dense_vector type), content fields, and filter fields.
Existence check — ExistsAsync() verifies an index is present before querying.
Deletion — DeleteAsync() removes an index and all its data.

Index names are generated from the data source configuration and include the tenant prefix in multi-tenant environments.

warning

Deleting an index removes all indexed documents permanently. Re-indexing from the data source is required after deletion.

Troubleshooting

Connection Refused

Error: Elasticsearch.Net.ElasticsearchClientException: Connection refused

Cause: Elasticsearch is not running or the URL is incorrect.

Fix:

Verify Elasticsearch is running: docker ps or curl http://localhost:9200
Check the Url in appsettings.json matches the actual endpoint
Ensure the port is correct (default: 9200)

Authentication Failed

Error: Elasticsearch.Net.ElasticsearchClientException: 401 Unauthorized

Cause: Invalid credentials for the selected authentication type.

Fix:

Verify the selected AuthenticationType and its matching credentials in appsettings.json
Reset the elastic user password: docker exec -it elasticsearch bin/elasticsearch-reset-password -u elastic

Certificate Error

Error: The SSL connection could not be established

Cause: TLS certificate mismatch when connecting to an HTTPS endpoint.

Fix:

Provide the correct CertificateFingerprint in configuration
For local development, disable TLS in Elasticsearch or use http:// instead of https://
Get the fingerprint: openssl s_client -connect localhost:9200 | openssl x509 -fingerprint -sha256 -noout

Index Not Found

Error: index_not_found_exception when querying

Cause: The index has not been created yet, or the index name is incorrect.

Fix:

Trigger indexing from the admin UI or via the data source management API
Verify the index name matches what the application expects: curl -u elastic:changeme http://localhost:9200/_cat/indices?v

Quick Start​

Configuration​

appsettings.json​

ElasticsearchConnectionOptions​

Services Registered (Keyed by "Elasticsearch")​

External Elasticsearch source mappings​

Docker Setup for Local Development​

Configuration Reference​

Full appsettings.json Example​

ElasticsearchConnectionOptions — All Properties​

Verification​

1. Check Elasticsearch Health​

2. Verify from the Application​

3. Check Indexes via Elasticsearch API​

Index Management​

Troubleshooting​

Connection Refused​

Authentication Failed​

Certificate Error​

Index Not Found​

Quick Start

Configuration

`appsettings.json`

`ElasticsearchConnectionOptions`

Services Registered (Keyed by `"Elasticsearch"`)

External Elasticsearch source mappings

Docker Setup for Local Development

Configuration Reference

Full `appsettings.json` Example

`ElasticsearchConnectionOptions` — All Properties

Verification

1. Check Elasticsearch Health

2. Verify from the Application

3. Check Indexes via Elasticsearch API

Index Management

Troubleshooting

Connection Refused

Authentication Failed

Certificate Error

Index Not Found