Description
During training of our retriever, we developed a number of different tools and data elements designed to help us create a high volume of synthetic data that looked like a sample of production data we obtained from APHL. These included things like:
- A unified "Enhancements" dictionary containing all official, LOINC-accepted abbreviations, acronyms, and other names for particular ontology concepts
- Rule-based input formulas that many production samples seemed to follow, e.g. [testing_adjective] [principal_component] [modality] [testing_modifier] [measurement_derived_word]
- Heuristics for directly mapping inputs to combinations of different parts of their related names, varied across the LOINC property axes
- Post-processing heuristics designed to collapse parts of the input with low-consequence, such as bracket or parentheses placement, or the use of dots vs plus signs as a compound signifier
These developments were extremely helpful in creating high-quality training data, but it's possible there's still quite a bit of juice left on the table. Our retriever is very good, but it is somewhat constrained by memory and hardware limits--less than 400M parameters doesn't leave much room to memorize compound rule chains, while larger LLMs might be able to reserve more of their parameter space to memorize different permutations of the input. For example, looking at just a handful of clinically identical concepts (in the eyes of LOINC) shows that our semantic similarity for these could be improved:
- Methicillin-resistant Staphylococcus aureus and MRSA have a cosine similarity of 0.7211
- Immunoglobulin Light Chains and its LOINC-accepted abbreviation “BJ Proteins” only have a similarity of 0.3142
- Hepatitis B Virus and Hep B have a similarity of 0.7548, despite even containing the same prefix structures
Given all this, the broad scope of this ticket is to investigate whether there are ways we could leverage the work we did to create our training data into some form of post-processing. The high-level idea is that we already did a lot of enhancement and re-mapping type work, so we might be able to make more explicit use of it once again. The architectural view is that our retriever model is constrained by fewer parameters, and while this forces it to learn general patterns, it's less good at internalizing specific examples the way a larger model might.
Possible Ideas
- Start by profiling how many production inputs actually look like the form of any of the formulas we found
- Also check how many inputs look like "Axis Builds" that are constructed in some way out of their various property axes
- If these numbers represent a meaningful percentage of the data (say, at least 3-5%), then write a "reverse engineer" formula designed to map a formulaic input back to its root LOINC code (or multiple possible root LOINCs)
- From there, cross-reference this list of candidates with the results returned by the reranker (but be mindful of any small parsing quirks that might exist, such as parens or hyphens--it's possible we'll need to use the retriever to re-cosine-score the candidates against the OpenSearch results to see if there was an improvement)
- Start by profiling how often inputs have all or part of their text as a candidate member in our universal enhancements dictionary (modulo capitalization); we already have functions in data curation that can do this by splitting out into constituent sub-parts and finding maximal length swap options
- Also look at whether there are particular types of data that are abbreviated or acronym'ed more frequently than others (e.g. organism names, particular tests, etc.)
- If there are a reasonable number of these candidates, we can write some functions that create variants of the nonstandard input with one or more enhancements made, and then use the retriever to re-score these against the candidates returned by OpenSearch to see if there's been an improvement
- Varying and enhancing a LOINC code is often iterative: first there'll be an abbreviation swap, then a movement of parentheses, then a replacement of a modality word, and finally a related name insertion
- We have functions in data curation that do most of these things, so one idea is to construct something like a DFS or BFS that applies on the input one change at a time, up to a pre-specified number of levels (probably no more than 5, otherwise it might be too unrelated). We can track the change in cosine similarity with each level of the search and see if any of them are improvements in the cosine similarity. We can also use techniques like Alpha-Beta pruning to cut off branches of the search tree that take us farther away from the OpenSearch candidates, so that the search doesn't spend time tweaking variants that aren't worth it.
Description
During training of our retriever, we developed a number of different tools and data elements designed to help us create a high volume of synthetic data that looked like a sample of production data we obtained from APHL. These included things like:
These developments were extremely helpful in creating high-quality training data, but it's possible there's still quite a bit of juice left on the table. Our retriever is very good, but it is somewhat constrained by memory and hardware limits--less than 400M parameters doesn't leave much room to memorize compound rule chains, while larger LLMs might be able to reserve more of their parameter space to memorize different permutations of the input. For example, looking at just a handful of clinically identical concepts (in the eyes of LOINC) shows that our semantic similarity for these could be improved:
Given all this, the broad scope of this ticket is to investigate whether there are ways we could leverage the work we did to create our training data into some form of post-processing. The high-level idea is that we already did a lot of enhancement and re-mapping type work, so we might be able to make more explicit use of it once again. The architectural view is that our retriever model is constrained by fewer parameters, and while this forces it to learn general patterns, it's less good at internalizing specific examples the way a larger model might.
Possible Ideas