UI display page by page: empty data even there are data available

[amc1999]

UI support display parameter, output table and microdata page by page. It is essential for large tables. However user sometime may see empty data even there are data available:

Steps to reproduce:

It is more easy to reproduce on large parameters / output tables / microdata

1. Use FBCumImmig very large parameter

2. Set page Size = 10
3. Drag Year dimension on slicer and select Year = 2020
4. Drag Sex dimension on columns and select all items (Female and Male)
5. Drag Age dimension on rows and select only 40

Results are empty. If you click Next button then at record = 40 you can see only one value = 9, click next to recored = 50 and there are no data again, until record = 140.

The issue, as far as I understand is coming from the fact I am filtering db records by slicer dimensions only Year = 2020 and filtering display view by user selection: Age = 40. There are 10 records selected from database but some of it may not contain Age = 40.

Solution can be is to provide some message to the user about it (about data filtered out by user selection). Or change back-end API to also filter by user selection.

I am not sure if there are any situation when parameter / output table / microdata display less data than page size. It may be user selection is the only problem, we have to test it more.

[deirdrehennessy]

Hi there- I can't seem to assign this issue- it's yours @scottyunho
Thanks,
D

[amc1999]

Answering @scottyunho questions:

Q:
... where the UI actually references/retrieves the data that populates the table, and where the UI filters that data (based on the slicers and pageSize selected) ...
... how the data ends up displayed in the UI. I.e. does it pull and save ALL the data initially then just filter it based on filters selected? Or is the specific data pulled dynamically based on the filters selected?...

A:
Data always retrieved dynamically based on slicer filter(s) and page size and offset.
I can retrieve all data, but for large parameters / output tables / microdata it can be millions of records and this is why we have pages.

Parameter page is more complicated than output table page because it has to support parameter editing, there is a significant amount of code to handle that, which you can ignore.

There is a function at parameter-page.js line 1154 which retrieves portion of the data based on pageSize , pageStart (staring row number) and slicer filters: otherFields ("other" means not rows or columns).

    // get page of parameter data from current model run or workset
    async doRefreshDataPage (isFullPage = false) {
      .......................
      // make parameter read layout and url
      const layout = Puih.makeSelectLayout(this.parameterName, this .otherFields, Puih.SUB_ID_DIM)
      .......................
      if (this.isPages) {
        layout.Offset = this.pageStart || 0
        layout.Size = (!!this.pageSize && typeof this.pageSize === typeof 1) ? (this.pageSize || 0) : 0
      .......................
          // update pivot table view
          this.inpData = Object.freeze(d)

Parameter data from the server assigned to inpData variable and rendered by other Vue component, which I call pv-table (pivot table) component.
Parameter.vue, line 924:

      <pv-table
        ref="omPivotTable"
        :rowFields="rowFields"
        :colFields="colFields"
        :otherFields="otherFields"
        :pv-data="inpData"
        :pv-control="pvc"
        :refreshViewTickle="ctrl.isPvTickle"
        :refreshDimsTickle="ctrl.isPvDimsTickle"
        :pv-edit="edt"
        @pv-key-pos="onPvKeyPos"
        @pv-edit="onPvEdit"
        >
      </pv-table>

In src/components directory you can find PvTable.vue, pv-table.js and multiple pivot-*.js files which are handling data rendering and formatting according to the language settings.

PvTable component used not only for parameter but also for output table and microdata page view, so, there are many pivot-table functions are not related to parameter at all and very large chunk of code dedicated to parameter editing, which you can ignore.

Important difference and source of our problem (at least part of it) is the fact PvTable render only the data which user selected on all dimensions: rowFields, colFields and slicer otherFields.

On screenshots above data from the server are filtered by slicer only, e.g. Year = 2020, but PvTable also applies rows and columns filter selected by user: Age = 40.

Why it is different? Short answer: I want UI table viewer to be fast and when user select / unselect rows and columns items or drag dimensions to render result quickly.

Is it a right approach? I never sure, we can change it but have to test UI performance for large parameters (mostly UI) and calculated output tables / microdata (mostly back-end).

PvTable process data from the server, please see pv-table.js line 688:

    // update pivot table from input data page
    setData (data) {
   .......................

I think we can detect "no-data" situation by counting size of vcells (we cannot use vcells.length, it has to be count++ of some sort). I am not sure is it vcells or not, please verify. Also I don't know where and how to display "no-data" message, pop-up info is a bad idea unless we suppress duplicated pop-ups (you can try it). And also it would be nice explain to the user absence of values too. I mean user selected page size = 10 but seeing only one value: where is other 9?