Confidence does a complete 180 when provided with longer training data input

[lukebarton]

https://colab.research.google.com/drive/1llNY6O2kn1GmMroZ352MHO_9XtwM9MyC

swim, expect zing
[('zing', 0.9998135976528079), ('zoob', 0.0001864023471921245)]
[('zoob', 0.8241350650787352), ('zing', 0.17586493492126465)] longer examples
meow, expect zoob
[('zoob', 0.9998439073858366), ('zing', 0.00015609261416340414)]
[('zing', 0.9763082785095902), ('zoob', 0.02369172149040979)] longer examples

classifier goes from being confidently correct, to being confidently incorrect with longer training inputs.

All three of the issues I've raised today make it very difficult to reason about whether the package is working as expected or not -- the behaviour I'm observing makes extremely little sense to my casual understanding.

[codelion]

I have fixed the issues and done a new release v0.0.15. For this particular case, again the underlying model is not giving semantic similarity in the manner you are expecting. I changed the model to:

# Train and save the classifier
classifier                = AdaptiveClassifier("sentence-transformers/all-MiniLM-L6-v2")
classifier_long           = AdaptiveClassifier("sentence-transformers/all-MiniLM-L6-v2")

and it gives the expected results:

swim, expect zing
[('zing', 0.7715373027748803), ('zoob', 0.2284626972251196)]
[('zing', 0.853310729264392), ('zoob', 0.146689270735608)] longer examples
meow, expect zoob
[('zoob', 0.7709077832915469), ('zing', 0.22909221670845314)]
[('zoob', 0.6933931256942506), ('zing', 0.30660687430574934)] longer examples