Skip to content

03_Index_Management.md

Latest commit

 

History

History
192 lines (133 loc) · 5.51 KB

03_Index_Management.md

File metadata and controls

192 lines (133 loc) · 5.51 KB
title Index Management
description Manage search indices for the Generic Data Index, including creation, updates, queue processing, and deployment.

Index Management

The Generic Data Index must index all assets, data objects, and documents to power search and listing features in Pimcore.

Console Commands Overview

Command Description
generic-data-index:update:index Update index mappings and queue all elements for reindex from the database
generic-data-index:update:index -r Delete and recreate indices, then queue all elements
generic-data-index:reindex Native search engine reindex (reorganizes data within existing indices, no database read)
generic-data-index:deployment:reindex Update indices only for class definitions changed since the last deployment

Index Prefix

Define an index name prefix to avoid naming collisions in shared search engine clusters. The default prefix is pimcore_.

pimcore_generic_data_index:
    index_service:
        client_params:
            index_prefix: 'my_prefix'

Created Indices

The Generic Data Index creates the following indices:

  • Assets - one alias and one index
  • Data objects - one alias and one index per class definition

Each index uses an alias (e.g. <prefix>_asset) pointing to the current index (e.g. <prefix>_asset-odd). The alias name stays constant; the backing index alternates between -odd and -even suffixes during reindexing (see Updating index structure below).

Keeping Indices Up to Date

Create and update indices with:

bin/console generic-data-index:update:index

This command creates the indices and queues all assets and data objects for indexing. The Symfony Messenger pimcore_generic_data_index_queue transport processes the queue.

Index Refresh

By default, the index refreshes after each bulk operation since items are processed asynchronously via Symfony Messenger.

To force synchronous processing (immediate refresh), inject SynchronousProcessingServiceInterface and call enable():

Method Description
enable() Enable synchronous processing
disable() Disable synchronous processing
isEnabled() Check current mode

Queue Options

Configure the indexing queue batch behavior:

Option Default Description
worker_count 1 Number of parallel messenger:consume workers. Improves batch size calculation.
min_batch_size 5 Minimum items per batch (relevant with multiple workers)
max_batch_size 400 Maximum items per batch

The queue calculates batch sizes dynamically between min_batch_size and max_batch_size based on the number of workers and queue depth.

pimcore_generic_data_index:
    index_service:
        queue_settings:
            worker_count: 1
            min_batch_size: 5
            max_batch_size: 400

Related Elements

Updating an element automatically enqueues its related elements for reindexing. By default, this runs asynchronously through Symfony Messenger.

For immediate processing, use SynchronousProcessingRelatedIdsServiceInterface:

Method Description
enable() Activate synchronous processing
disable() Revert to asynchronous processing
isEnabled() Return current processing mode

:::info

SynchronousProcessingRelatedIdsServiceInterface does not affect delete operations. Deletes always process synchronously.

:::

Repairing Indices

To delete and recreate an index from the Pimcore database (e.g. after an incompatible mapping change), pass the -r option:

bin/console generic-data-index:update:index -r

Without -r, the command only updates the index mapping and queues all items for reindex.

Updating Index Structure

Index mappings update automatically when system languages or class definition fields change. For manual updates, run the reindex command. This performs a native OpenSearch/Elasticsearch reindex within the search indices (no database read):

bin/console generic-data-index:reindex

Handling Failed Messages

The messenger retries failed messages 3 times, then routes them to the pimcore_generic_data_index_failed transport. Retry failed messages with:

bin/console messenger:failed:retry -vv

See the Symfony Messenger documentation for additional commands.

Index Options

Configure search engine-specific index options. Check your engine's documentation for available settings.

Maximum Result Window

The default limit of retrievable items is 10,000. Increase it with:

pimcore_generic_data_index:
    index_service:
        index_settings:
            max_result_window: 20000

Total Fields Limit

The default field limit per index is 1,000. Increase it with:

pimcore_generic_data_index:
    index_service:
        index_settings:
            mapping.total_fields.limit: 20000

:::info

If the index already exists, recreate it after changing this setting:

bin/console generic-data-index:update:index -r

:::

Deployment and Index Management

Class Definition Changes

After updating class definitions during deployment, run:

bin/console generic-data-index:deployment:reindex

This updates the index structure for all class definitions modified since the last deployment and reindexes data objects for affected classes.