Training Custom Entity Relations

[kibblewhite]

Hello,

Thanks so much for making this possible through .Net, I currently have custom entities working and tested. For example, I've managed to train it to know that 'Micheal Jackson' is an 'artist' and to ignore other names that I'm not interested in. I've also got it to know that when talking about 'Thriller' that it knows to label it as entity type 'album'.

However, I'm now to a point where I would like it to build a custom relation between the custom entities, 'artist' & 'album'. Just as we have an Organization (Org) that is Based In (OrgBased_In) a Location (Loc), I would like to have an 'artist' that is 'involved' in an 'album'.

How would one go about doing this? Is there any documents or C# examples of training some relations and then seeing the results printed out with these custom relations to our custom entities all in action?

Here is some code which I have running at the moment.

            // # TRAINING
            // this is working code that generates relations serialized model files
            // defined as serializedRelationExtractorPath & serializedTrainingSentencesPath
            // artist-album.corp file is based off conll04.corp example file (lines 1-2916 but with 'Artist' and a few custom entries for 'Albums' with custom relations added 'Involved_In')
            // serializedEntityExtractorPath is not generated - tmp/roth_entity_model.ser
            Properties props = new Properties();
            props.setProperty("annotators", "pos, lemma, parse");
            props.setProperty("parse.maxlen", "100");
            props.setProperty("datasetReaderClass", "edu.stanford.nlp.ie.machinereading.domains.roth.RothCONLL04Reader");
            props.setProperty("trainPath", "artist-album.corp");
            props.setProperty("crossValidate", "false");
            props.setProperty("kfold", "10");
            props.setProperty("trainUsePipelineNER", "false");
            props.setProperty("serializedTrainingSentencesPath", "tmp/roth_sentences.ser");
            props.setProperty("serializedEntityExtractorPath", "tmp/roth_entity_model.ser");
            props.setProperty("serializedRelationExtractorPath", "tmp/roth_relation_model_pipeline.ser");
            props.setProperty("relationResultsPrinters", "edu.stanford.nlp.ie.machinereading.RelationExtractorResultsPrinter");
            props.setProperty("entityClassifier", "edu.stanford.nlp.ie.machinereading.domains.roth.RothEntityExtractor");
            props.setProperty("extractRelations", "true");
            props.setProperty("extractEvents", "false");
            props.setProperty("extractEntities", "false"); // setting this to 'true' results in the following error ->
            /* at edu.stanford.nlp.ie.machinereading.GenericDataSetReader.parse(String path)
            at edu.stanford.nlp.ie.machinereading.MachineReading.loadOrMakeSerializedSentences(String sentencesPath, GenericDataSetReader reader, File serializedSentences)
            at edu.stanford.nlp.ie.machinereading.MachineReading.run() */

            props.setProperty("trainOnly", "true");
            props.setProperty("relationFeatures", "arg_words,arg_type,dependency_path_lowlevel,dependency_path_words,surface_path_POS,entities_between_args,full_tree_path");

            var propertyKeys = props.keys();
            var propertyStringArray = new List<string>();
            while (propertyKeys.hasMoreElements())
            {
                var key = propertyKeys.nextElement();
                propertyStringArray.Add($"-{key}");
                propertyStringArray.Add(props.getProperty(key.toString(), string.Empty));
            }

            var machineReader = edu.stanford.nlp.ie.machinereading.MachineReading.makeMachineReading(propertyStringArray.ToArray());
            var utestResultList = machineReader.run();

The next section of code is what I'm using in order to test the entity relations model which was generated by the previous training code.

`            // # TESTING
            Properties props = new Properties();
            props.setProperty("annotators", "tokenize, ssplit, pos, lemma, parse, ner, relation");
            props.setProperty("ner.model", @"edu/stanford/nlp/models/ner/ner-trained-model.ser.gz"); // previously generated/trained entity model with Artists and Albums examples, which is tested and works
            props.setProperty("ner.model.3class", string.Empty);
            props.setProperty("ner.model.7class", string.Empty);
            props.setProperty("ner.model.MISCclass", string.Empty);
            props.setProperty("ner.useSUTime", "0");
            props.setProperty("dcoref.dictlist", "coref.dict1.tsv, coref.dict2.tsv, coref.dict3.tsv, coref.dict4.tsv");
            props.setProperty("sup.relation.model", "tmp/roth_relation_model_pipeline.ser");
            props.setProperty("sup.relation.verbose", "true");

            var text = "..."; // text contains text about Micheal Jackson with references to him and his albums, including Thriller.
            var doc = new Annotation(text);
            StanfordCoreNLP pipeline = new StanfordCoreNLP(props);
            pipeline.annotate(doc);

            var sentences = doc.get(new CoreAnnotations.SentencesAnnotation().getClass()) as ArrayList;
            if (sentences == null) { return; }
            foreach (CoreMap sentc in sentences)
            {
                Debug.WriteLine("For sentence: " + sentc.get(new CoreAnnotations.TextAnnotation().getClass()));
                var rls = sentc.get(new MachineReadingAnnotations.RelationMentionsAnnotation().getClass()) as ArrayList;
                if (rls == null) { return; }
                foreach (RelationMention relationMention in rls)
                {
                    Debug.WriteLine(relationMention.toString());
                }
            }

The results from the testing code, only shows relations between PEOPLE and LOCATIONS, nothing about Artists or Albums.
Is there anything I'm missing? I feel that during the training, I make no references to my custom entity models, it shouldn't know about it anyways, but I can not see how I'm meant to get that working?
Any documentation, or better yet a code example please.

Many thanks

[sergey-tihon]

Would be better to ask it on SO, here I may answer on Java->C# code conversion issues.

Also you may look at TokensRegex it may be somehow useful for your case. JFYI we have TP for TokenRegex - https://github.com/statmuse/TokensRegexProvider

[kibblewhite]

Got it, thanks for that. I'll have a look, and if I'm not able to resolve it, I'll post the same thing up on Stack Overflow and report back. Awesome :)

[kibblewhite]

Going through the Stack Overflow, looks like this may not be possible due to this bit of code found at:
private static String edu.stanford.nlp.ie.machinereading.domains.roth.RothCONLL04Reader.getNormalizedNERTag(String ner)

Found it here - http://stackoverflow.com/questions/34922368/how-to-make-the-custom-relationshipextractor-understand-custom-entities/38814333#38814333

I also agree with 'testTester' quoting: "not sure why these classes are hard-coded!"

TokensRegex may not provide me with what I need either, so I'll keep looking for another solution. Typical hey...