Support XSLT based cleaning of CSDL metadata (#232)

* Add XSLT transformation.
* Added test, doc, logger, indent, annotations.

Author: MIchaelMainer

README.md

@@ -44,14 +44,27 @@ Typewriter is a new solution for generating code files using the GraphODataTempl
 * **-m**, **-metadata**: The local file path or URL to the target input metadata. The default value is `https://graph.microsoft.com/v1.0/$metadata`. This value is required.
 * **-v**, **-verbosity**: The log verbosity level. The values can be: `Minimal`, `Info`, `Debug`, or `Trace`. The default value is `Minimal`.
 * **-o**, **-output**: Specifies the path to the output folder. The default value is the directory that contains typewriter.exe. The structure and contents of the output directory will be different based on the `-generationmode` and `-language` options.
-* **-d**, **-docs**: Specifies the path to the local root of the [microsoft-graph-docs](https://github.com/microsoftgraph/microsoft-graph-docs) repo. The default value is the directory that contains typewriter.exe. The documentation is parsed to provide documentation annotations to the metadata which is then used to add doc comments in the generated code files. This option is required when using `-generationmode` values of `Metadata` or `Full`.
-* **-g**, **-generationmode**: Specifies the generation mode. The values can be: `Full`, `Metadata`, or `Files`. `Full` (default) generation mode produces the output code files by cleaning the input metadata, parsing the documentation, and adding annotations before generating the output files. `Metadata` generation mode produces an output metadata file by cleaning metadata, documentation parsing, and adding documentation annotations. `Files` generation mode produces code files from an input metadata and bypasses the cleaning, documentation parsing, and adding documentation annotations.
+* **-d**, **-docs**: Specifies the path to the local root of the [microsoft-graph-docs](https://github.com/microsoftgraph/microsoft-graph-docs) repo. The default value is the directory that contains typewriter.exe. The documentation is parsed to provide documentation annotations to the metadata which is then used to add doc comments in the generated code files. This option is required when using `-generationmode` values of `Metadata`, `Full`, or `TransformWithDocs`.
+* **-g**, **-generationmode**: Specifies the generation mode. The values can be: `Full`, `Metadata`, `Files`, `Transform` or `TransformWithDocs`. `Full` (default) generation mode produces the output code files by cleaning the input metadata, parsing the documentation, and adding annotations before generating the output files. `Metadata` generation mode produces an output metadata file by cleaning metadata, documentation parsing, and adding documentation annotations. `Files` generation mode produces code files from an input metadata and bypasses the cleaning, documentation parsing, and adding documentation annotations. `Transform` generation mode processes the metadata according to the XSLT provided with the -t option. `TransformWithDocs` generation mode processes the metadata according to the XSLT and adds documentation annotations. `Transform` and `TransformWithDocs` require the -t argument to specify the XSLT file.
 * **-f**, **-outputMetadataFileName**: The base output metadata filename. Only applicable for `-generationmode Metadata`. The default value is `cleanMetadataWithDescriptions` which is used with the value of the `-endpointVersion` to generate a metadata file named `cleanMetadataWithDescriptionsv1.0.xml`.
 * **-e**, **-endpointVersion**: The endpoint version used when naming a metadata file. Expected values are `v1.0` and `beta`. Only applicable for `-generationmode Metadata`.
 * **-p**, **-properties**: Specify properties to support generation logic in the T4 templates. Properties must take the form of *key-string:value-string*. Multiple properties can be specified by setting a space in between property. The only property currently supported is the *php.namespace* property to specify the generated model file namespace. This property is optional.
+* **-t**, **-transform**: Specify the URI to the XSLT that will preprocess the metadata. Only applicable for `-generationmode Transform` or `-generationmode TransformWitDocs`.
 
 ### Example typewriter usage
 
+#### Transform metadata with XSLT.
+
+The output `cleanMetadata.xml` will be located in the same directory as typewriter.exe.
+
+`.\typewriter.exe -v Info -m https://raw.githubusercontent.com/microsoftgraph/msgraph-metadata/mm/xslt/v1.0_metadata.xml -g Transform -t https://raw.githubusercontent.com/microsoftgraph/msgraph-metadata/mm/xslt/transforms/csdl/preprocess_csdl.xsl`
+
+#### Transform metadata with XSLT and add documentation annotations.
+
+The output `cleanMetadataWithDescriptionsv1.0.xml` will be located in the same directory as typewriter.exe.
+
+`.\typewriter.exe -v Info -m https://raw.githubusercontent.com/microsoftgraph/msgraph-metadata/mm/xslt/v1.0_metadata.xml -g TransformWithDocs -t https://raw.githubusercontent.com/microsoftgraph/msgraph-metadata/mm/xslt/transforms/csdl/preprocess_csdl.xsl -d D:\repos\microsoft-graph-docs`
+
 #### Generate TypeScript typings from a CSDL (metadata) file without cleaning or annotating the CSDL.
 
 The output will go in to the `outputTypeScript` directory.
@@ -113,7 +126,7 @@ Example :
 
 It is important to understand that subprocessors are mapped to methods that query the **OdcmModel** and return a set of OData objects. This mapping is maintained in [TemplateProcess.InitializeSubprocessor()](https://github.com/microsoftgraph/MSGraph-SDK-Code-Generator/blob/dev/src/GraphODataTemplateWriter/TemplateProcessor/TemplateProcessor.cs#L54). The language specific mappings exist in the [config directory](https://github.com/microsoftgraph/MSGraph-SDK-Code-Generator/tree/dev/src/GraphODataTemplateWriter/.config). Each OData object returned by the subprocessor is applied to the mapped template which results in a code file output per each OData object.
 
-In the above example, the objects in result set of the NavigationCollectionProperty subprocessor will each be applied to the EntityCollectionPage template. Each result will be a code file for each object returned by the NavigationCollectionProperty subprocessor. 
+In the above example, the objects in result set of the NavigationCollectionProperty subprocessor will each be applied to the EntityCollectionPage template. Each result will be a code file for each object returned by the NavigationCollectionProperty subprocessor.
 
 #### SubProcessors
 

src/Typewriter/Generator.cs

@@ -1,7 +1,13 @@
 using Microsoft.Graph.ODataTemplateWriter.TemplateProcessor;
+using NLog;
 using System;
 using System.Collections.Generic;
+using System.IO;
 using System.Runtime.CompilerServices;
+using System.Text;
+using System.Xml;
+using System.Xml.XPath;
+using System.Xml.Xsl;
 using Vipr.Core;
 using Vipr.Reader.OData.v4;
 
@@ -11,6 +17,8 @@ namespace Typewriter
 {
     internal static class Generator
     {
+        internal static Logger Logger => LogManager.GetLogger("Generator");
+
         /// <summary>
         /// Generate code files from the input metadata and do not preprocess.
         /// </summary>
@@ -59,6 +67,62 @@ static private string CleanMetadata(string csdlContents, Options options)
             return AnnotationHelper.ApplyAnnotationsToCsdl(options, pathToCleanMetadata).Result;
         }
 
+        /// <summary>
+        /// Transform CSDL with XSLT.
+        /// </summary>
+        /// <param name="csdlContents">The CSDL to transform.</param>
+        /// <param name="options">The options bag that must contain the -transform -t parameter.</param>
+        /// <returns>The location of the generated file.</returns>
+        static internal string Transform(string csdlContents, Options options)
+        {
+            var outputDirectoryPath = options.Output;
+
+            if (!string.IsNullOrWhiteSpace(outputDirectoryPath) && !Directory.Exists(outputDirectoryPath))
+                Directory.CreateDirectory(outputDirectoryPath);
+            if (string.IsNullOrWhiteSpace(outputDirectoryPath))
+                outputDirectoryPath = Environment.CurrentDirectory;
+
+            var pathToCleanMetadata = string.Concat(outputDirectoryPath, "\\cleanMetadata.xml");
+
+            using (var reader = new StringReader(csdlContents))
+            using (var doc = XmlReader.Create(reader))
+            using (XmlTextWriter writer = new XmlTextWriter(pathToCleanMetadata, Encoding.ASCII))
+            {
+                writer.Indentation = 2;
+                writer.Formatting = Formatting.Indented;
+
+                XslCompiledTransform transform = new XslCompiledTransform();
+                XsltSettings settings = new XsltSettings();
+                settings.EnableScript = true;
+                transform.Load(options.Transform, settings, null);
+
+                // Execute the transformation, writes the transformed file.
+                transform.Transform(doc, writer);
+            }
+
+            Logger.Info($"Transformed metadata written to {pathToCleanMetadata}");
+
+            return pathToCleanMetadata;
+        }
+
+        /// <summary>
+        /// Transform CSDL with XSLT and add documentation.
+        /// </summary>
+        /// <param name="csdlContents">The CSDL to transform.</param>
+        /// <param name="options">The options bag that must contain the -transform -t parameter,
+        /// and the -docs -d parameter.</param>
+        static internal void TransformWithDocs(string csdlContents, Options options)
+        {
+            var pathToCleanMetadata = Transform(csdlContents, options);
+
+            string csdlWithDocAnnotations = AnnotationHelper.ApplyAnnotationsToCsdl(options, pathToCleanMetadata).Result;
+            string outputMetadataFilename = options.OutputMetadataFileName ?? "cleanMetadataWithDescriptions";
+            string metadataFileName = string.Concat(outputMetadataFilename, options.EndpointVersion, ".xml");
+            FileWriter.WriteMetadata(csdlWithDocAnnotations, metadataFileName, options.Output);
+
+            Logger.Info($"Transformed metadata with documentation written to {metadataFileName}");
+        }
+
         /// <summary>
         /// Generates code files from an edmx file.
         /// </summary>

src/Typewriter/Options.cs

@@ -32,7 +32,15 @@ public enum GenerationMode
         /// <summary>
         /// Uses the input metadata and only generates code files for the target platform. It bypasses the cleaning, doc parsing, and adding doc annotations.
         /// </summary>
-        Files
+        Files,
+        /// <summary>
+        /// Uses the input metadata to transform the CSDL with the specified XSLT.
+        /// </summary>
+        Transform,
+        /// <summary>
+        /// Uses the input metadata to transform the CSDL with the specified XSLT and adds documentation annotations.
+        /// </summary>
+        TransformWithDocs
     }
 
     public class Options
@@ -52,10 +60,11 @@ public class Options
         [Option('d', "docs", Default = ".", HelpText = "Path to the root of the documentation repo folder")]
         public string DocsRoot { get; set; }
 
-        [Option('g', "generationmode", Default = GenerationMode.Full, HelpText = "Specifies the generation mode. The values can be: Full, Metadata, or Files. Full generation mode produces " +
+        [Option('g', "generationmode", Default = GenerationMode.Full, HelpText = "Specifies the generation mode. The values can be: Full, Metadata, Files, Transform, or TransformWithDocs. Full generation mode produces " +
             "the output code files by cleaning the input metadata, parsing the documentation, and adding annotations before generating the output files. Metadata generation mode" +
             "produces an output metadata file by cleaning metadata, documentation parsing, and adding documentation annotations. Files generation mode produces code files from" +
-            "an input metadata and bypasses the cleaning, documentation parsing, and adding documentation annotations.")]
+            "an input metadata and bypasses the cleaning, documentation parsing, and adding documentation annotations. Transform generation mode processes the metadata according to the" +
+            "XSLT provided with the -t option. TransformWithDocs generation mode processes the metadata according to the XSLT and adds documentation annotations.")]
         public GenerationMode GenerationMode { get; set; }
 
         [Option('f', "outputMetadataFileName", Default = "cleanMetadataWithDescriptions", HelpText = "The output metadata filename. Only applicable for GenerationMode.Metadata.")]
@@ -68,5 +77,9 @@ public class Options
             "templates from the TemplateWriterSettings object returned by ConfigurationService.Settings. The suggested convention for specifying a key should be " +
             "the targeted template language name and the property name. For example, php.namespace:Microsoft\\Graph\\Beta\\Model would be a property to be consumed in the PHP templates.")]
         public IEnumerable<string> Properties { get; set; }
+
+        [Option('t', "transform", HelpText = "Specify the URI to the XSLT that will preprocess the metadata. Overrides the" +
+            "cleaning done by embeddeded typewriter.exe rules.")]
+        public string Transform { get; set; }
     }
 }

src/Typewriter/Program.cs

@@ -39,6 +39,12 @@ private static void GenerateSDK(Options options)
                 case GenerationMode.Metadata:
                     Generator.WriteCleanAnnotatedMetadata(csdlContents, options);
                     break;
+                case GenerationMode.Transform:
+                    Generator.Transform(csdlContents, options);
+                    break;
+                case GenerationMode.TransformWithDocs:
+                    Generator.TransformWithDocs(csdlContents, options);
+                    break;
                 case GenerationMode.Full:
                 default:
                     Generator.GenerateFilesFromCleanMetadata(csdlContents, options);

test/Typewriter.Test/Given_a_valid_metadata_file_to_Typewriter.cs

@@ -468,5 +468,57 @@ public void It_creates_disambiguated_IEntityRequestBuilder_parameters()
 
             Assert.IsTrue(hasTestParameter, $"The expected test token string, '{testParameter}', was not set in the generated test file. We didn't properly generate the parameter.");
         }
+
+        [TestMethod]
+        public void It_transforms_metadata()
+        {
+            const string outputDirectory = "output";
+
+            Options options = new Options()
+            {
+                Output = outputDirectory,
+                GenerationMode = GenerationMode.Transform,
+                Transform = "https://raw.githubusercontent.com/microsoftgraph/msgraph-metadata/mm/xslt/transforms/csdl/preprocess_csdl.xsl"
+
+            };
+
+            Generator.Transform(testMetadata, options);
+
+            FileInfo fileInfo = new FileInfo(outputDirectory + @"\cleanMetadata.xml");
+            Assert.IsTrue(fileInfo.Exists, $"Expected: {fileInfo.FullName}. File was not found.");
+
+            IEnumerable<string> lines = File.ReadLines(fileInfo.FullName);
+
+            bool hasThumbnailAnnotationBeenAdded = false; // Expect true
+            bool hasHasStreamAttribute = false; // Expect false
+            bool hasContainsTargetBeenSet = false; // Expect true
+            bool hasCapabilityAnnotations = false; // Expect false
+
+            // Check the document for these values.
+            foreach (var line in lines)
+            {
+                if (line.Contains(@"<Annotation Term=""Org.OData.Core.V1.LongDescription"" String=""navigable"" />"))
+                {
+                    hasThumbnailAnnotationBeenAdded = true;
+                }
+                if (line.Contains("HasStream"))
+                {
+                    hasHasStreamAttribute = true;
+                }
+                if (line.Contains(@"<NavigationProperty Name=""plans"" Type=""Collection(microsoft.graph.plannerPlan)"" ContainsTarget=""true"" />"))
+                {
+                    hasContainsTargetBeenSet = true;
+                }
+                if (line.Contains("Org.OData.Capabilities"))
+                {
+                    hasCapabilityAnnotations = true;
+                }
+            }
+
+            Assert.IsTrue(hasThumbnailAnnotationBeenAdded, $"The expected LongDescription annotation wasn't set in the transformed cleaned metadata.");
+            Assert.IsFalse(hasHasStreamAttribute, $"The HasStream attribute was't removed from the metadata.");
+            Assert.IsTrue(hasContainsTargetBeenSet, $"The expected ContainsTarget attribute wasn't set in the transformed cleaned metadata.");
+            Assert.IsFalse(hasCapabilityAnnotations, $"The expected capability annotations weren't removed in the transformed cleaned metadata.");
+        }
     }
 }