Code Generation

Code Generation

Drill relies heavily on code generation for the data-specific functionality in each operator. For example, the FilterBatch (implements the SQL WHERE clause) generates code for the actual filter condition. Code generation is necessary because Drill works with a large number of data types (over 120 different value vector implementations) and many kinds of expressions. The alternative, an interpreter, would be expensive to maintain (given the large number of value vector types) and slower to execute.

(Code generation seems to follow a pattern established by Hive? Need to research.)

Code Generation Workflow

Code Generation Components

• CodeGenerator<T>: generate the Java source code required to complete the implementation of an abstract template. T is the interface that results from compiling and merging the runtime code that is generated.
• TemplateClassDefinition<T>: Identifies the interface which the code will implement, along with the template code used to (what?). Also provides a serial number for classes generated from this template. The template is created from the interface that the generated code is to implement.
• SignatureHolder: (Seems to hold information about the class to be generated: methods and so on.)
• CodeGeneratorMethod: (Describes one method in the generated code: name, return value, arguments, etc.)
• FunctionImplementationRegistry:
• OptionManager: A Javascript-like, prototype-based "stack" of options set from various sources (system, query, etc.) (Used to customize code?)
• MappingSet:
• GeneratorMapping: Maps the four standard "conceptual" methods (setup, eval, reset, cleanup) to the names of the actual method (if any) in the generated code.
• JCodeModel: "CodeModel is a Java library for code generators; it provides a way to generate Java programs."
• FragmentContext: Provides the getImplementationClass( ) method which uses the DrillbitContext to find the compiler module that generates and compiles the code.
• CodeCompiler: Mechanism to generate and compile code.

Example - ExternalSortBatch, SingleBatchSorter

The SingleBatchSorter interface, template and template definition:

public interface SingleBatchSorter {
  public void setup(FragmentContext context, SelectionVector2 vector2, VectorAccessible incoming) 
              throws SchemaChangeException;
  public void sort(SelectionVector2 vector2);

  public static TemplateClassDefinition<SingleBatchSorter> TEMPLATE_DEFINITION = 
                new TemplateClassDefinition<SingleBatchSorter>(SingleBatchSorter.class, 
                                                               SingleBatchSorterTemplate.class);
}

public abstract class SingleBatchSorterTemplate implements SingleBatchSorter, IndexedSortable{
  ...

  public abstract void doSetup(@Named("context") FragmentContext context,
                               @Named("incoming") VectorAccessible incoming, 
                               @Named("outgoing") RecordBatch outgoing);
  public abstract int doEval(@Named("leftIndex") char leftIndex, 
                             @Named("rightIndex") char rightIndex);
}

The following code in ExternalSortBatch generates the specialized class:

  public SingleBatchSorter createNewSorter(FragmentContext context, VectorAccessible batch)
          throws ClassTransformationException, IOException, SchemaChangeException{
    CodeGenerator<SingleBatchSorter> cg = CodeGenerator.get(SingleBatchSorter.TEMPLATE_DEFINITION, context.getFunctionRegistry(), context.getOptions());
    ClassGenerator<SingleBatchSorter> g = cg.getRoot();

    generateComparisons(g, batch);
    return context.getImplementationClass(cg);
  }