exrmetrics print disk size of part

[Arcmantis18]

WIP for #2031 which requested that exrmetrics have the capability to print the packed size of each part's data in a multipart EXR image. Initially problematic as this information is not readily available by default, however, @peterhillman suggested I use the writing position of the memory output stream to compute the bytes written per part. These stats can only be printed when no output file has been specified as this causes a memory output stream to be used.

In my work so far, I have created a flag --part-size-disk which enables printing of the part sizes. The write position of the memory output stream is used to compute how many bytes were written to the stream per image part and the results are stored in fileMetrics struct for printing later.

Though the issue specified that exrmetrics should print the packed/compressed data, I have opted to print this data as the 'size on disk' because I believe that is a more accurate and useful metric.

The eventual name and description of the flag is of course open for discussion, I just needed to get the ball rolling. I will also make other syntax style corrections later, after the main issue is worked out.

[Arcmantis18]

I have been trying to determine the accuracy of using memory output stream as a determination of part size. I decided to process the multipart.001.exr example image with no compression and compare the total raw size of the part, to the part disk sizes:

./exrmetrics --part-disk-size ./multipart.0001.exr -z none

output:

[
 {
  "file":"../openexr/src/test/bin/test_images/multipart.0001.exr",
  "input file size": 2894260,
  "pixels": 6798483,
  "channels": 20,
  "total raw size": 29081142,
  "compression": "zips",
  "part type": "scanlineimage",
  "parts":
   [
    {
      "part": 0,
      "pixels": 768252,
      "channels": 4,
      "compression": "zips",
      "part type": "scanlineimage",
      "total raw size": 6146016
    },
    {
      "part": 1,
      "pixels": 768252,
      "channels": 1,
      "compression": "zips",
      "part type": "scanlineimage",
      "total raw size": 1536504
    },
    {
      "part": 2,
      "pixels": 768252,
      "channels": 2,
      "compression": "zips",
      "part type": "scanlineimage",
      "total raw size": 3073008
    },
    {
      "part": 3,
      "pixels": 296065,
      "channels": 1,
      "compression": "zips",
      "part type": "scanlineimage",
      "total raw size": 592130
    },
    {
      "part": 4,
      "pixels": 768252,
      "channels": 4,
      "compression": "zips",
      "part type": "scanlineimage",
      "total raw size": 6146016
    },
    {
      "part": 5,
      "pixels": 768252,
      "channels": 1,
      "compression": "zips",
      "part type": "scanlineimage",
      "total raw size": 1536504
    },
    {
      "part": 6,
      "pixels": 768252,
      "channels": 2,
      "compression": "zips",
      "part type": "scanlineimage",
      "total raw size": 3073008
    },
    {
      "part": 7,
      "pixels": 798036,
      "channels": 2,
      "compression": "zips",
      "part type": "scanlineimage",
      "total raw size": 3192144
    },
    {
      "part": 8,
      "pixels": 798036,
      "channels": 2,
      "compression": "zips",
      "part type": "scanlineimage",
      "total raw size": 3192144
    },
    {
      "part": 9,
      "pixels": 296834,
      "channels": 1,
      "compression": "zips",
      "part type": "scanlineimage",
      "total raw size": 593668
    }
   ],
  "metrics":
   [
    {
      "compression": "none",
      "pixel mode": "original",
      "output size": 29256013,
      "read time": 0.0900866,
      "write time": 0.011013,
      "re-read time": 0.0066799,
      "parts":
       [
        {
          "part": 0,
          "read time": 0.0244836,
          "write time": 0.00222805,
          "re-read time": 0.00147284,
          "size on disk": 6156528,

        },
        {
          "part": 1,
          "read time": 0.00627512,
          "write time": 0.000610501,
          "re-read time": 0.000357544,
          "size on disk": 1547016,

        },
        {
          "part": 2,
          "read time": 0.00858204,
          "write time": 0.00115291,
          "re-read time": 0.00075305,
          "size on disk": 3083520,

        },
        {
          "part": 3,
          "read time": 0.00421019,
          "write time": 0.000269148,
          "re-read time": 0.000191251,
          "size on disk": 601358,

        },
        {
          "part": 4,
          "read time": 0.0112699,
          "write time": 0.00225511,
          "re-read time": 0.00137871,
          "size on disk": 6156528,

        },
        {
          "part": 5,
          "read time": 0.00621179,
          "write time": 0.000595032,
          "re-read time": 0.000390927,
          "size on disk": 1547016,

        },
        {
          "part": 6,
          "read time": 0.00887145,
          "write time": 0.00117117,
          "re-read time": 0.000666667,
          "size on disk": 3083520,

        },
        {
          "part": 7,
          "read time": 0.00807323,
          "write time": 0.00124133,
          "re-read time": 0.000725688,
          "size on disk": 3202656,

        },
        {
          "part": 8,
          "read time": 0.00806853,
          "write time": 0.00122138,
          "re-read time": 0.000582608,
          "size on disk": 3202656,

        },
        {
          "part": 9,
          "read time": 0.00404079,
          "write time": 0.000268336,
          "re-read time": 0.000160613,
          "size on disk": 602896,

        }
       ]
    }
   ]
 }
]

I was expecting a rough similarity between total raw size and metric's 'size size on disk', however what I found was that KBs of more data was consistently being written.

For example, parts 0,1,2,4,5,6,7,8 were all writing 10512 bytes more than 'total raw size' and parts 3, 9 were writing 9228 bytes more than 'total raw size'.

My only guess so far is that these extra bytes are the metadata of the scanline blocks, but I would appreciate any insight.

As I see it, the disk size reports are inaccurate (in the sense that they don't report the raw disk size of a part) unless we can somehow compute the chunk metadata sizes to subtract from the final result.

It does however, accurately report the actual disk size of each part within the EXR file layout.

[peterhillman]

The 'raw size' is supposed to be just the data required for the pixels themselves (how much memory you'd require to store every channel of every pixel in the image in its native type) not the amount of space that takes on disk, even if no compression is used.

Each chunk also contains a header. That's between 8 bytes per chunk (for a single part scanline file) and 40 bytes per chunk (for a deeptile part in a multipart file). The file also contains a preamble, and one header and chunk offset table per part. The bytes taken up by the chunk headers will be factored into your "size on disk" but the chunk table won't be. That will cause a discrepancy between the total file size on disk and the total size of all parts. For scanline parts, the number of chunks is dependent on the compression scheme, so the discrepancy will be different.

I think that's fine as it is, but perhaps a slightly longer description of what --part-disk-size does could help.
The website/bin/exrmetrics.rst document could be a good place to add extra notes about how to interpret the output

[Arcmantis18]

I guess this really is the best that can be done in this case. As long as the user understands that this is not a "pure" part size metric (either packed or unpacked) and more of a part size + chunk headers on disk metric. Despite this, it still gives a mostly accurate look at the impact different compression algorithms have on individual part data.

I'll clean it up then.

Thanks for the feedback!

[peterhillman]

I think that's nearly there!

I suggested an update to the help (and to the webpage too) that might be less open to misinterpretation. (It could be understood that you run exrmetrics input.exr --part-disk-size output.exr instead of exrmetrics inputs.exr -o output.exr)

It would also be useful to update the parse() function to error if -o and --part-disk-size are used together.

Also, src/test/bin/test_exrmetrics.py should be updated to verify that the output is valid JSON when --part-disk-size is used

[peterhillman]

Thanks! That looks good