Bug: Ingestion workflow redeployment fails due to missing `sourceConfig.config.type` in automated pipeline creation

[ayush-shah]

Description

Following an upgrade to v1.12.8, certain metadata ingestion workflows failed to redeploy through the UI, showing an "ingestion workflow redeployment error".

The root cause was identified as a configuration discrepancy introduced by automated API creation of pipelines. Specifically, pipelines created via external automation were missing the {sourceConfig, config, type} field in their JSON entity payload, while the system expected it to be explicitly defined (e.g., "DatabaseMetadata").

A database update was required to patch the missing JSON fields and restore redeployment capabilities. We need to ensure that the API handles or defaults this value properly, or document the mandatory fields required for external automation to prevent future instances of this issue.

Steps to Reproduce

1. Create a metadata ingestion pipeline via an external API call/automation framework.
2. Omit the sourceConfig.config.type property in the payload, leaving it NULL in the database.
3. Attempt to redeploy the ingestion workflow via the UI in v1.12.8.
4. The UI displays a deployment error due to the missing configuration block.

Investigation & Database Analysis

Running the following diagnostic query helps identify affected/orphaned or improperly configured pipelines:

SELECT
  i.id AS pipeline_id,
  i.name AS pipeline_name,
  i.json #>> '{fullyQualifiedName}' AS fqn,
  i.json #>> '{pipelineType}' AS pipeline_type,
  i.json #>> '{sourceConfig,config,type}' AS source_config_type,
  er.fromentity AS service_type,
  er.fromid AS service_id,
  CASE
    WHEN er.toid IS NULL THEN 'ORPHANED: missing service CONTAINS relationship'
    WHEN i.json #>> '{sourceConfig,config,type}' IS NULL THEN 'BAD_CONFIG: sourceConfig.config.type missing'
    ELSE 'OK'
  END AS finding
FROM ingestion_pipeline_entity i
LEFT JOIN entity_relationship er
  ON er.toid = i.id
  AND er.toentity = 'ingestionPipeline'
  AND er.relation = 0
  AND er.deleted = false
WHERE
  er.toid IS NULL
  OR i.json #>> '{sourceConfig,config,type}' IS NULL
ORDER BY finding, fqn;