Version 0.60.0 Changes

⚠️ Upcoming release of 0.60.0 has breaking changes due to re-organization, renaming and clean up of implementation classes.

The extent of the changes and optimizations made it impossible to use the slow @Deprecated annotation with eventual removal in all cases.

• Major reorganization and code cleanup of implementation for next version 0.60.0

  • Formatter implementation is now part of core implementation in flexmark module

  • Formatter improved with more options including wrapping text to margins.

    • added ability to track and map source offset(s) to their index in formatted sequence. This feature allows editor caret position preservation across formatting operation.
    • Offset tracking unified using TrackedOffset. Used by MarkdownParagraph for text wrapping and MarkdownTable for table formatting and able to handle caret position during typing and backspace editing operations which are immediately followed by formatting or the edited source.

  • Tests cleaned up to eliminate duplication and hacks

  • flexmark-test-util made reusable for other projects. Having markdown as the source code for tests is too convenient to have it only used for flexmark-java tests.

  • Optimized SegmentedSequence implementation using binary trees for searching segments and byte efficient segment packing. Parser performance is either slightly improved or not affected but allows using SegmentedSequences for collecting Formatter and HtmlRenderer output to track source location of all text with minimal overhead and double the performance of old implementation.

  • new implementation of LineAppendable used for text generation in rendering:

    • can use SequenceBuilder to generate BasedSequence result with original source offsets for those character segments which come from the source. This allows round trip source tracking from Source -> AST -> Formatted Source -> Source throughout the library.

      As an added bonus using the appendable makes formatting to it 40% faster than previous implementation and 160 times (yes times) more efficient in memory use. For the test below, old implementation allocated 6GB worth of segmented sequences, new implementation 37MB. The % overhead is four times greater but that is after a 43 fold reduction in total overhead bytes, old implementation needed 342MB of overhead, new implementation 8MB.

      As a result of increased efficiency, two additional files of about 600kB each can be included in the test run and only add 0.6 sec to the formatter run time.

    Tests run on 1141 markdown files from GitHub projects and some other user samples. Largest was 256k bytes.

    Description	Old SegmentedSequence	New Segmented Sequence	New LineAppendable
    Total wall clock time	13.896 sec	9.672 sec	8.344 sec
    Parse time	2.402 sec	2.335 sec	2.297 sec
    Formatter appendable	0.603 sec	0.602 sec	0.831 sec
    Formatter sequence builder	7.264 sec	3.109 sec	1.772 sec

    The overhead difference is significant. The totals are for all segmented sequences created during the test run of 1141 files. Parser statistics show requirements during parsing and formatter ones are only for formatting of them while accumulating the text as a segmented sequence.

    Description	Old Parser	Old Formatter	New Parser	New Formatter	New LineAppendable
    Bytes for characters of all segmented sequences	917,016	6,029,774,526	917,016	6,029,774,526	37,663,196
    Bytes for overhead of all segmented sequences	1,845,048	12,060,276,408	93,628	342,351,155	8,204,796
    Overhead %	201.2%	200.0%	10.2%	5.7%	21.8%

To Do

• Doc: BasedSequence, SegmentedSequence and BasedSequenceBuilder changes.
• Doc: LineAppendable and using SequenceBuilder as an appendable target for it to accumulate rendered text with source information.

• Break: split out generic AST utilities from flexmark-util module into separate smaller modules. IntelliJ IDEA migration to help with migration from 0.50.40 will be provided where needed if the package or class is changed. com.vladsch.flexmark.util will no longer contain any files but will contain the separate utilities modules with flexmark-utils module being an aggregation of all utilities modules, similar to flexmark-all

  • ast/ classes to flexmark-util-ast
  • builder/ classes to flexmark-util-builder
  • collection/ classes to flexmark-util-collection
  • data/ classes to flexmark-util-data
  • dependency/ classes to flexmark-util-dependency
  • format/ classes to flexmark-util-format
  • html/ classes to flexmark-util-html
  • mappers/ classes to flexmark-util-sequence
  • options/ classes to flexmark-util-options
  • sequence/ classes to flexmark-util-sequence
  • visitor/ classes to flexmark-util-visitor

• Convert anonymous classes to lambda where possible.

• refactor flexmark-util to eliminate dependency cycles between classes in different subdirectories.

• Break: delete deprecated properties, methods and classes

• Add: org.jetbrains:annotations:18.0 dependency to have @Nullable/@NotNull annotations added for all parameters. I use IntelliJ IDEA for development and it helps to have these annotations for analysis of potential problems and use with Kotlin.

• Break: refactor and cleanup tests to eliminate duplicated code and allow easier reuse of test cases with spec example data.

• Break: move formatter tests to flexmark-core-test module to allow sharing of formatter base classes in extensions without causing dependency cycles in formatter module.

• Break: move formatter module into flexmark core. this module is almost always included anyway because most extension have a dependency on formatter for their custom formatting implementations. Having it as part of the core allows relying on its functionality in all modules.

• Break: move com.vladsch.flexmark.spec and com.vladsch.flexmark.util in flexmark-test-util to com.vladsch.flexmark.test.spec and com.vladsch.flexmark.test.util respectively to respect the naming convention between modules and their packages.

• Break: NodeVisitor implementation details have changed. If you were overriding NodeVisitor.visit(Node) in the previous version it is now final to ensure compile time error is generated. You will need to change your implementation. See comment in the class for instructions.

  ℹ️ com.vladsch.flexmark.util.ast.Visitor is only needed for implementation of NodeVisitor and VisitHandler. Converting all anonymous implementations of VisitHandler to lambdas results in removal of all imports for Visitor.

  • Fix: remove old visitor like adapters and implement ones based on generic classes not linked to flexmark AST node.
  • Deprecate old base classes:
    • com.vladsch.flexmark.util.ast.NodeAdaptedVisitor see javadoc for class
    • com.vladsch.flexmark.util.ast.NodeAdaptingVisitHandler
    • com.vladsch.flexmark.util.ast.NodeAdaptingVisitor