Multi-tenant custom field support

[ejsmith]

Patternize multi-tenant custom field handling

• Custom field mapping storage
  • Entity type (repo), tenant (should allow for composite like organization+type), field type, friendly name, extensible to store additional mapping information
  • Shared across multiple entity types and tenants
  • Auto assign index slots to new custom fields
  • Fill in gaps for index slots when fields have been deleted
  • Ability to efficiently lookup field mapping for an entity type and tenant
    • Aggressively cache mapping for the tenant
  • Ability to have a list of aliases for a field that will be used to resolve queries
    • Only the root name will be used for getting data that gets added to the idx slot
    • Aliases count as used names can't be used by other fields
• Extensible field types
  • Packaged up field type behaviors that can be re-used
  • Built in field types for string, date, bool, number, float, etc
  • Consumers can create custom field types to add additional indexing formats like address, geo, etc
  • Ability to transform values when adding them to idx
    • Allows the data to be optimized for searching
    • Allows things to happen like looking up related data to store denormalized
    • Probably needs to allow batching of values for efficiency when doing idx transforms
  • Standard field types automatically configured, but can be overridden with custom behavior
  • Ignore errors for invalid document field values when saving
  • Ability to do advanced things like scripted fields that have access to the rest of the document to calculate values (proven out in tests) https://github.com/FoundatioFx/Foundatio.Repositories/blob/master/tests/Foundatio.Repositories.Elasticsearch.Tests/Repositories/Configuration/Indexes/CalculatedIntegerFieldType.cs
• Easily implemented in repositories
  • Interface for model to implement that opts the repository into supporting custom fields
  • Automatically copies values from friendly names to idx slots
  • Automatically gets mapping and sets QueryFieldResolver on queries
• Field management
  • Ability to change type of an existing field
    • Could start out as an auto field indexed as string and then be changed to a date or some other more specific type
  • Ability to remove a field
    • Cleanup idx values
    • Optionally remove custom field values
  • Ability to rename a field
• Ability to auto create custom fields on the fly when we see a new value that does not have a mapping
  • Default to store as string with keyword subfield
  • Hook when auto creating custom property mappings that lets you decide if a mapping should be created and determine the default index type

[ejsmith]

Changing data type flow:

• Soft delete existing field
  • Name will be freed up to be used
  • Slot will remain unavailable until fully deleted
  • Data in old idx slot will still be there until we reprocess, but it won't be queried because there won't be a mapping pointing to it any more (it's soft deleted)
• Create new field with new index type and same name, will get a new slot for that index type
  • Copy all settings from previous custom field like the extended data and description, etc so it is a full copy
  • Any new documents will get the field saved to the correct slot
  • Any saved documents will get the idx values reconfigured with the new type
• Batch process any documents where the previous idx slot has a value and just re-save the documents
  • What happens if some documents have incompatible custom field values?
• Fully delete old custom field (no need to keep it around longer)

[ejsmith]

Remove field flow:

• Soft delete field
  • Name is available to be used, if we try to undelete someone might have taken the name
  • Field map will be immediately removed and won't be able to search the field
  • Orphaned idx data needs to be cleaned up before we allow the slot to be re-used
• Batch process any documents where the idx slot has a value and just re-save the documents
• Optionally delete the values from the custom field source as well (not idx)
  • Need new method on IHaveCustomFields to delete a custom field value
  • Seems valid to want to leave the data, but have it be unmapped
• Fully delete old custom field

[ejsmith]

Rename field flow:

• Change name on custom field
  • Add old name as an alias
  • Name won't be allowed to be used since it's being used as an alias
  • Auto create won't attempt to create mapping because the name is used as an alias
• Batch process any documents where the idx slot has a value and move data from old name to new name
• Remove old name from alias list which will open up the name to be used again

[ejsmith]

Make sure that we have everything in place that we would need to support calculated field custom properties. Should have access to the field definition and access to the full document. Also, we would need to calculate the display value as well as the idx value.

[niemyjski]

Is this up to date?

[ejsmith]

Yeah, I think that's where I left off.

[niemyjski]

Custom Fields Gap Analysis (as of March 2026)

Did a thorough audit of the current implementation against every bullet in this issue. Here's where things stand.

---

1. Custom Field Mapping Storage

Requirement	Status	Notes
Entity type (repo) scoping	✅ Done	CustomFieldDefinition.EntityType (immutable after creation)
Tenant key (composite support)	✅ Done	CustomFieldDefinition.TenantKey (immutable). Composite keys documented (e.g., $"{CompanyId}-{SubType}")
Field type, friendly name	✅ Done	IndexType and Name properties
Extensible additional mapping info	✅ Done	Data dictionary on CustomFieldDefinition (used by CalculatedIntegerFieldType for expressions)
Shared across entity types and tenants	✅ Done	Single CustomFieldDefinitionIndex stores all definitions
Auto assign index slots	✅ Done	AddAsync acquires distributed lock, assigns lowest available slot starting at 1
Fill in gaps for deleted slots	✅ Done	Hard-delete frees slots for reuse. Tested in CanUseDeletedSlotAndName
Efficient tenant lookup + aggressive caching	✅ Done	GetFieldMappingAsync caches 15 min. Slot/name caches at 5 min.
Field aliases	❌ Not Done	No Aliases property on CustomFieldDefinition. No alias storage, validation, or query resolution. Issue calls for: aliases resolve in queries, only root name populates idx, aliases count as used names.

2. Extensible Field Types

Requirement	Status	Notes
Packaged field type behaviors	✅ Done	ICustomFieldType with Type, ProcessValueAsync, ConfigureMapping
Built-in types (string, date, bool, number, float)	✅ Done	8 standard types: string, keyword, int, long, float, double, bool, date
Consumer-created custom types	✅ Done	AddCustomFieldType() on Index. CalculatedIntegerFieldType proves extensibility
Value transforms for idx	✅ Done	ProcessFieldValueResult supports separate Value (Data) and Idx (index) values
Related data lookup / denormalization	✅ Done	ProcessValueAsync receives full document + definition, supports async lookups
Batch processing for idx transforms	❌ Not Done	ProcessValueAsync called per-document-per-field. No batch API for field types needing external lookups
Standard types auto-configured, overridable	✅ Done	AddStandardCustomFieldTypes() registers all 8. Types are virtual for inheritance
Ignore errors for invalid values	✅ Done	Type mismatches silently fail at ES level. Tested in CanHandleWrongFieldValueType
Scripted fields with document access	✅ Done	CalculatedIntegerFieldType with Jint + AlwaysProcess mode. Tested in CanUseCalculatedFieldType

3. Easily Implemented in Repositories

Requirement	Status	Notes
Interface for model to opt in	✅ Done	IHaveCustomFields and IHaveVirtualCustomFields. Auto-detected via HasCustomFields
Auto copy values from names to idx slots	✅ Done	OnCustomFieldsDocumentsChanging handler
Auto get mapping + set QueryFieldResolver	✅ Done	OnCustomFieldsBeforeQuery handler with ToHierarchicalFieldResolver("idx.")

4. Field Management

Requirement	Status	Notes
Change type of existing field	❌ Not Done	No API, migration path, or documentation. IndexType is notably not in the immutable enforcement list (OnDocumentsChanged only blocks EntityType, TenantKey, IndexSlot), but no strategy exists for handling stale idx data when type changes
Remove a field	✅ Done	Soft delete frees name. Hard delete frees name + slot. Both tested
Cleanup idx values on removal	❌ Not Done	No mechanism to clear orphaned idx.* values from existing documents when a field is deleted
Optionally remove custom field values	❌ Not Done	No API to strip Data dictionary values for a removed field across documents
Rename a field	⚠️ Partial	Name is mutable on save (not in immutable list), but no dedicated API, no cache invalidation for old name in usedNames cache, and no test coverage

5. Auto-Create Custom Fields

Requirement	Status	Notes
Auto-create on new unmapped value	✅ Done	AutoCreateCustomFields = true + HandleUnmappedCustomField. Tested
Default to string with keyword subfield	✅ Done	Uses StringFieldType.IndexType (Text + Keyword)
Hook to decide if mapping should be created / determine default type	⚠️ Partial	HandleUnmappedCustomField is virtual so subclasses can override. But the value param is unused in the default impl and no example of value-based type detection is provided

---

Summary

Section	Items	Done	Partial	Missing
Mapping storage	10	9	0	1
Extensible field types	9	8	0	1
Repository integration	3	3	0	0
Field management	5	1	1	3
Auto-create fields	3	2	1	0
Total	30	23	2	5

~77% complete. The biggest gaps are:

1. Field aliases — Need Aliases property on CustomFieldDefinition, validation that aliases count as used names, and query resolution through aliases
2. Field type change — Need a strategy (soft-delete old + create new with same name? in-place with reindex?) and proper handling of stale idx data
3. Cleanup on field removal — Need a way to clear orphaned idx.* and optionally Data values from existing documents (background job or explicit API)
4. Field rename — Need proper cache invalidation and a dedicated API or documented pattern
5. Batch processing for field types — Optional batch API on ICustomFieldType for field types that need external lookups during idx transforms