feat(implicitTiling): Add support for external buffer

[AnthonyGlt]

Fix #844
Fix #1032

SUPPORT correctly the external buffer presents in the subtree.
The support of an external buffer (.bin) is not done yet #608 (comment) .

@gkjohnson it doesn't seem that hard to implement BUT I don't really know how the manage the cache once the buffer has been uploaded.

In Cesium:
The presence of an external buffer is checked here:
https://github.com/CesiumGS/cesium/blob/466f780f3a50d6edd17b1f27e1cde5f86ce10128/packages/engine/Source/Scene/ImplicitSubtree.js#L700

And loaded here
https://github.com/CesiumGS/cesium/blob/466f780f3a50d6edd17b1f27e1cde5f86ce10128/packages/engine/Source/Scene/ImplicitSubtree.js#L719

How could we do something similar to cache the incoming buffer ?

[jsulli]

This seems like a big improvement, the frequency of errors went way down in my testing - but I was still able to trigger the same error in every map I tested, it just took a lot more navigation and more aggressive refinement before it triggered. Same issue as #1032, the parser is trying to fetch subtree files that don't exist.

If it's helpful, here are two examples:
subtrees.zip
Tried to load subtrees/5/4/2/11.subtree, subtrees/5/5/3/10.subtree, and more

subtrees.zip
Tried to load subtrees/5/10/14/4.subtree, subtrees/5/10/13/9.subtree, etc

From the tileset provided in the #1032:

[gkjohnson]

it doesn't seem that hard to implement BUT I don't really know how the manage the cache once the buffer has been uploaded.
...
How could we do something similar to cache the incoming buffer ?

The Plugin.parseTile function is asynchronous so we can await any extra file loads or asynchronous operations in the SUBTREELoader.parse function and save the buffer on the implicit tile root. Then we can delete it when the disposeTile function is called. That should handle the caching I belive.

[AnthonyGlt]

@gkjohnson @jsulli Thank you both for your feedback.
I think it will be better to implement directly the support of it then. I manage to do it locally. BUT there is something I don't really get with the building of the binary url.
We need to request subtrees/0/0/0/0_1.bin but this format doesn't follow the classic subtree format, which is supposed to be:
subtrees/{level}/{x}/{y}/{z}.{extension}, so as you can see, one 0 seems to be missing in the previous url, it should be subtrees/0/0/0/0/0_1.bin
The same thing is happening for the data from the other issue. So it looks like the format subtrees/{level}/{x}/{y}/{z}.{extension} isn't the one which is apply for external buffer.
Would you happen to know how this external buffer uri is supposed to be handled? I didn't find clear doc about this.
In my local implementation, I replace everything after the last /, which seems to be what is done in Cesium, but it looks arbitrary.

I'll edit this PR, to add the support of external buffer instead of ignoring it.

@jsulli I still can't try with your data right now, so it could be nice if you could test it yourself once I've made the necessary changes (I'll ping you), thank you 👍

[gkjohnson]

We need to request subtrees/0/0/0/0_1.bin but this format doesn't follow the classic subtree format, which is supposed to be:

The bin reference is specified in a json chunk in the .subtree file. See the section in the spec here:

{
  "buffers": [
    {
      "byteLength": 586
    },
    {
      "uri": "0_1.bin",
      "byteLength": 4096
    }
  ],
  "bufferViews": [
    {
      "buffer": 0,
      "byteOffset": 0,
      "byteLength": 586
    },
    {
      "buffer": 1,
      "byteOffset": 0,
      "byteLength": 4096
    }
  ],
  "tileAvailability": {
    "bitstream": 0,
    "availableCount": 157
  },
  "contentAvailability": [
    {
      "bitstream": 0,
      "availableCount": 157
    }
  ],
  "childSubtreeAvailability": {
    "bitstream": 1,
    "availableCount": 429
  }
}

[AnthonyGlt]

@gkjohnson Oh I got confused with the architecture for the folders. Ok, I got it now, thanks 👍

[AnthonyGlt]

@jsulli could you try this updated version please ? It should load the .bin and display it

[jsulli]

Edit: Disregard, some instanceof checks were failing in my application because of a mismatch between its version of three.js and this library's

[jsulli]

@AnthonyGlt tested against 5 octree tiling scheme datasets, I could not repeat the error from #1032 in any of them. Seems to be fixed to me!

[gkjohnson]

Amazing! Thanks for digging into this. And thanks to @jsulli for testing it. I've left a couple comments on the changes.

[gkjohnson]

Can we separate this out into a few lines for readability. Ie a url, fetch promise, and then push the promise onto the array.

[gkjohnson]

Lets include the source of the error and the specific failure so it's more clear what's happened:

throw new Error( `SUBTREELoader: Failed to load external buffer from ${ bufferHeader.uri } with error code ${ response.status }.` );

[gkjohnson]

Let's remove "undefined" here - we can just call Promise.resolve().

[gkjohnson]

Just made a couple changes to remove the __basePath reference. One this I notice is this large, empty bounding box that doesn't disappear when zooming. Is this correct? It seems to have no visible content.

edit
It seems this is correct - it just contains a very small sliver of model highlighted in red here:

[gkjohnson]

@AnthonyGlt I think this looks good to me, now. Once you've taken a look at my final changes and think they're okay we can merge this. Thanks again for putting the time into it!

[AnthonyGlt]

@gkjohnson Thank you for the commits ! All good, can be merged

[gkjohnson]

Thanks @AnthonyGlt - do you mind making sure the remaining list of implicit tiling features from #608 (comment) is up to date and possibly linking to some example data sets for the related features if they're available?