Skip to content

Data Exchange Definition export performance improvement#9073

Open
VolodySyn wants to merge 3 commits into
mainfrom
bugs/633941-SlowDataExchExport
Open

Data Exchange Definition export performance improvement#9073
VolodySyn wants to merge 3 commits into
mainfrom
bugs/633941-SlowDataExchExport

Conversation

@VolodySyn

@VolodySyn VolodySyn commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Fixes AB#633941

What changed

Reworks the export mapping flow in Export Mapping (codeunit 1269) and Payment Export Mgt (codeunit 1210) to eliminate per-row database writes and redundant work during export.

1. In-memory mapping + bulk save (ExportMapping.Codeunit.al)

  • ProcessColumnMapping now writes column values into a temporary Data Exch. Field buffer instead of inserting into the persisted table row-by-row.
  • The mapping is split into two phases:
    • Mapping phase – builds all field values in memory.
    • Saving phase – transfers the temp buffer to the persisted Data Exch. Field table (including Value BLOB) in a single pass at the end.
  • This replaces N individual inserts/modifies with one bulk write, which is the main performance win for large exports.

2. FlowField calculation reuse (PaymentExportMgt.Codeunit.al)

  • New ProcessColumnMapping overload accepts the Data Exch. FlowField Gr. Buff. and the temp Data Exch. Field buffer.
  • FlowField values are read from the pre-aggregated grouping buffer when available, falling back to CalcField() only when not present—avoiding repeated FlowField recalculation per line.
  • PrepopulateColumns was updated to prepopulate the temporary buffer.

3. Progress dialog throttling (ExportMapping.Codeunit.al)

  • The progress window is updated every N lines instead of every line, where the divider scales with row count (>1000 → 200, >100 → 10, else 1), reducing UI-update overhead on large datasets.
  • Progress messages split into two labels: Mapping line no. … and Saving line no. ….

Impact

Significantly faster Data Exchange Definition exports on large record sets, driven by bulk persistence, reduced FlowField recalculation, and throttled progress updates. No change to the produced output/format.

@VolodySyn VolodySyn requested a review from a team July 3, 2026 12:52
@github-actions github-actions Bot added this to the Version 29.0 milestone Jul 3, 2026
@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Copilot PR Review

Iteration 2 · Outcome: completed

Knowledge source: https://github.com/microsoft/BCQuality@822cae1b2771ac25f665f73369f69093bd4fd630

Findings by domain

Findings split into Knowledge-backed (cite a BCQuality article) and Agent (the agent's own judgement, no matching BCQuality rule).

Domain Findings Knowledge-backed Agent Inline Fallback
Agent 1 0 1 0 1

Totals: 0 knowledge-backed · 1 agent findings.

Orchestrator pre-filter (13 file(s) excluded)

  • layer-disabled (knowledge) : 13 file(s)

Findings produced by the Copilot CLI agent against BCQuality at 822cae1b2771ac25f665f73369f69093bd4fd630. Reply 👎 on any inline comment to flag false positives.

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

$\textbf{🟡\ Medium\ Severity\ —\ Agent} \quad \color{gray}{\texttt{\small Iteration\ 2}}$

This PR adds a new 7-parameter ProcessColumnMapping overload (with the TempDataExchField buffering parameter) and updates the only caller (Export Mapping.Codeunit.al) to use it.

That leaves the previous 6-parameter overload (var DataExch; RecRef; var DataExchFlowFieldGrBuff; LineNo; DataExchLineDefCode; TableID) - and the private PrepopulateColumns helper it exclusively calls - with no remaining callers anywhere in the repository.

Recommendation:

  • remove the now-orphaned 6-parameter ProcessColumnMapping overload and its private PrepopulateColumns(4-param) helper, or repoint it if it was meant to be reused, so the codeunit does not carry dead logic that silently drifts from the new buffered/batched write path.

Line mapping was unavailable, so this was posted as an issue comment.

👍 useful · ❤️ especially valuable · 👎 wrong - reply with why

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant