Additional inference measures (LMR)

[pcomw]

Once again, thank you for the wonderful work.

I hope to switch over to Stepmix from MPlus, but a few tests that my group uses to evaluate models with different numbers of classes aren't yet in the package, and I was curious about the roadmap.

In the future, are there plans to add other inference measures to the stepmix class? I am thinking in particular of other IC and LRT-type stats:

1. Sample-size adjusted BIC, e.g., -2 * model.score(X) * X.shape[0] + model.n_parameters * np.log((X.shape[0] + 2) / 24)

2. CAIC, e.g., -2 * model.score(X) * X.shape[0] + model.n_parameters * (np.log(X.shape[0]) + 1)

3. Bootstrap likelihood ratio test (BLRT). E.g., page 543 of https://doi.org/10.1080/10705510701575396

4. Possibly also the Lo–Mendell–Rubin (LMR) and/or Vuong–Lo–Mendell–Rubin (VLMR) tests.

The IC stats are simple enough to implement, but the BLRT would take a little time.

Thanks again

[sachaMorin]

Thanks for the suggestions! Sample-size adjusted BIC and CAIC seem easy enough. I'm not 100% about BLRT but I would like to hear StepMix power user @FelixLaliberte's thoughts on this. Is this similar to what you are working on?

[FelixLaliberte]

Hi @pcomw,

I think all the inference measures you suggest are commonly used and would be much appreciated by users. Thanks for the suggestions!

@sachaMorin: To answer your question, the BLRT would indeed be very useful.

[sachaMorin]

Reviewing this paper, it seems the suggested Sample-Adjusted BIC was missing an X.shape[0] in the log. The current suggested implementation is

        n = X.shape[0]

        return -2 * self.score(X, Y) * n + self.n_parameters * np.log(
            n * ((n + 2) / 24)
        )

[pcomw]

I agree, it seems in the paper you linked that it should be n * ((n + 2) / 24)

However, in the MPlus output, there is a line of text that reads, (n* = (n + 2) / 24), which implies that the asterisk isn't a sign of multiplication, but a signifier of a different value. Similarly, on page 545 of https://doi.org/10.1080/10705510701575396, the text defines SABIC as replacing n in the original BIC calculation with n*, as seen here:

I suppose that the 1987 Sclove paper is the proper resource to check, but I don't have access to it.

I also came across a post with references about the VLMR-LRT in the tidyLPA github that might be relevant, if you choose to implement that in addition the the BLRT: data-edu/tidyLPA#178 (comment)

[sachaMorin]

Linking this paper here for future reference on BLRT. I feel StepMix has all the building blocks to do this. Will try to include BLRT in the next release.

[sachaMorin]

CAIC and SABIC are available as of version 2.0.0.

[sachaMorin]

BLRT is available as of version 2.2.0. See the updated tutorial.

[Acgholic]

We would very appreciate if the LMR is available in the future.