diff --git a/WORKFLOW_reference.md b/WORKFLOW_reference.md new file mode 100644 index 000000000..1644f6955 --- /dev/null +++ b/WORKFLOW_reference.md @@ -0,0 +1,239 @@ +# Workflow for generating reference + +There are two generated references: + +1. from openfact +1. from openvox-agent + +## OpenFact + +From OpenFact we collect: + +1. CLI +1. Core Facts + +## OpenVox Agent + +From OpenVox agent we fetch + +1. bundled types and providers +1. bundled functions + +## Changing content + +### OpenFact Content + +#### CLI + +CLI is rendered from man pages to markdown. +Content is in the openfact repository. + +Additional content is added via a preamble: `lib/puppet_references/facter/facter_cli_preamble.md` + +#### Core Facts + +Core Facts are rendered using a script, which is part of the openfact repo (`lib/docs/generate.rb`) + +Additional content is added via a preamble file: `lib/puppet_references/facter/core_facts_preamble.md` + +### OpenVox + +OpenVox autogenerated documentation is rendered differently. + +#### Man + +Rendering takes place in `lib/puppet_references/puppet/man.rb` Line 78 ff + +#### PuppetDoc + +Completely rendered using `puppet doc` command from openvox repo documentation + +#### Type + +Type uses a preamble and a template: + +- `lib/puppet_references/puppet/type_preamble.md` and +- `lib/puppet_references/puppet/type_template.erb` + +#### TypeStrings + +TypeStrings is inherited from Type. + +Content is rendered by running `puppet strings generate`. +Additional content is added in `lib/puppet_references/puppet/type_strings.rb` + +#### Functions + +Functions use a preamble and a template: + +- `lib/puppet_references/puppet/functions_preamble.md` and +- `lib/puppet_references/puppet/functions_template.erb` + +## Workflow + +This is a summarized analysis of the references generation process. + +### Rakefile + +The rakefile has three rake tasks: + +- references - Prints out usage of the other ones +- references:openfact +- references:openvox + +Each reference source is cloned into `vendor/` directory. + +GitHub URL is located in `lib/puppet_references/repo.rb` + +#### OpenFact Workflow + +`PuppetReferences.build_facter_references(ENV.fetch('VERSION', nil))` + +##### build_facter_references + +Building references + +```ruby +references = [ + PuppetReferences::Facter::CoreFacts, + PuppetReferences::Facter::FacterCli +] +``` + +Repo action + +```ruby +repo = PuppetReferences::Repo.new('openfact', FACTER_DIR) +real_commit = repo.checkout(commit) +repo.update_bundle +``` + +1. clone openfact repo into a local dir +1. switches to provided version +1. update all tags (fetch) +1. runs bundle install + +```ruby +build_from_list_of_classes(references, real_commit) +``` + +###### Core Facts Workflow + +1. executes ruby `/lib/docs/generate.rb` +1. adds header data + preamble + data +1. writes core_facts.md file + +###### CLI Workflow + +1. runs bundle update in openfact +1. runs bundle exec facter man +1. uses Pandoc Ruby to convert man to markdown +1. writes cli.md + +#### OpenVox Workflow + +`PuppetReferences.build_puppet_references(ENV.fetch('VERSION', nil))` + +##### build puppet reference + +```ruby +references = [ + PuppetReferences::Puppet::Man, + PuppetReferences::Puppet::PuppetDoc, + PuppetReferences::Puppet::Type, + PuppetReferences::Puppet::TypeStrings, + PuppetReferences::Puppet::Functions, + ] +``` + +1. Reads config - can be removed +1. Clones repo +1. switch to version +1. run bundle install +1. build_from_list_of_classes (build_all) + +##### Man Workflow + +1. reads all files in openvox-agent/lib/puppet/application/*.rb +1. deletes face_base, indirection_base, cert applications +1. build index for all applications + 1. has data with categories: core, occasional and weird +1. select from applications where data match +1. 2 variables: header data and index_text + 1. within index_text, add the markdown files mentioned in data + first core, then occasional, then weird +1. write content to overview.md file + +1. for each command run build_manpage +1. define a header +1. read command from openvox-agent/man/man/command.8 +1. calls PandDoc.ruby to convert from man to markdown, + adding gsub methods on converted content +1. write command file + +##### PuppetDoc Workflow + +1. calls build_reference +1. calls `puppet doc` command +1. calls clean_configuration_reference + 1. replaces fqdn with fixed string +1. sets header data +1. writes each reference.md file + +##### Type Workflow + +typedoc script: `/lib/puppet_references/quarantine/get_typedocs.rb` +typedoc template: `/lib/puppet_references/puppet/type_template.erb` +typedoc preamble: `lib/puppet_references/puppet/type_preamble.md` + +build_all: + +1. get_type_json: execute typedoc script +1. save json file + +1. build_index (type_data.keys.sort): + +- header_data (title & canonical) +- skip component and wiki +- read all md files from type_data +- add header (autogenerated) see lib/puppet_references/util.rb +- write file (puppet/types/overview.md) + +1. build_unified_page: + +- set header (title, canonical, toc_levels, toc) +- sort attributes (namenvar first, ensure second) +- write type individual files + +1. build page: + +- set header (title, canonical) +- sort attributes +- write type file + +##### TypeStrings Workflow + +output dir: puppet/types_strings + +same as type, but different get_type_json + +- PuppetReferences::Puppet::Strings.new +- read all filenames in lib/puppet/**/*.rb +- run bundle exec puppet strings generate + --format json --out string.json filenames.join(' ') +- read strings.json file + +- iterate over content['resource_type'] +- build new structure and content `lib/puppet_references/puppet/type_strings.rb` + THIS IS THE IMPORTANT file +- This has the knowledge on the output and renders the new md + +##### Functions Workflow + +template: functions_template.erb +preamble: functions_preamble.md + +- uses PuppetReferences::Puppet::Strings.new +- sets header +- render template, add header +- write file diff --git a/lib/puppet_references/facter/core_facts.rb b/lib/puppet_references/facter/core_facts.rb index e4ec7f394..78a086622 100644 --- a/lib/puppet_references/facter/core_facts.rb +++ b/lib/puppet_references/facter/core_facts.rb @@ -9,7 +9,7 @@ class CoreFacts < PuppetReferences::Reference PREAMBLE = PREAMBLE_FILE.read def initialize(*) - @latest = '/puppet/latest' + @latest = '/openvox/latest' super end diff --git a/lib/puppet_references/facter/facter_cli.rb b/lib/puppet_references/facter/facter_cli.rb index 11748b634..1626e43e1 100644 --- a/lib/puppet_references/facter/facter_cli.rb +++ b/lib/puppet_references/facter/facter_cli.rb @@ -9,7 +9,7 @@ class FacterCli < PuppetReferences::Reference PREAMBLE = PREAMBLE_FILE.read def initialize(*) - @latest = '/puppet/latest' + @latest = '/openvox/latest' super end diff --git a/lib/puppet_references/puppet/functions.rb b/lib/puppet_references/puppet/functions.rb index ffd0eb284..c9f34e931 100644 --- a/lib/puppet_references/puppet/functions.rb +++ b/lib/puppet_references/puppet/functions.rb @@ -10,12 +10,12 @@ module PuppetReferences module Puppet class Functions < PuppetReferences::Reference TEMPLATE_FILE = Pathname.new(File.expand_path(__FILE__)).dirname + 'functions_template.erb' - OUTPUT_DIR = PuppetReferences::OUTPUT_DIR + 'puppet' + OUTPUT_DIR = PuppetReferences::OUTPUT_DIR + 'openvox' PREAMBLE_FILE = Pathname.new(File.expand_path(__FILE__)).dirname + 'functions_preamble.md' PREAMBLE = PREAMBLE_FILE.read def initialize(*) - @latest = '/puppet/latest' + @latest = '/openvox/latest' super end diff --git a/lib/puppet_references/puppet/http.rb b/lib/puppet_references/puppet/http.rb index 9dbb41fec..6da40b78d 100644 --- a/lib/puppet_references/puppet/http.rb +++ b/lib/puppet_references/puppet/http.rb @@ -5,12 +5,12 @@ module PuppetReferences module Puppet class Http < PuppetReferences::Reference - OUTPUT_DIR = PuppetReferences::OUTPUT_DIR + 'puppet' + OUTPUT_DIR = PuppetReferences::OUTPUT_DIR + 'openvox' DOCS_DIR = OUTPUT_DIR + 'http_api' API_SOURCE = PuppetReferences::PUPPET_DIR + 'api' def initialize(*) - @latest = '/puppet/latest/http_api' + @latest = '/openvox/latest/http_api' super end diff --git a/lib/puppet_references/puppet/man.rb b/lib/puppet_references/puppet/man.rb index 1a1a5e0f8..02dc36ecd 100644 --- a/lib/puppet_references/puppet/man.rb +++ b/lib/puppet_references/puppet/man.rb @@ -5,10 +5,10 @@ module PuppetReferences module Puppet class Man < PuppetReferences::Reference - OUTPUT_DIR = PuppetReferences::OUTPUT_DIR + 'puppet/man' + OUTPUT_DIR = PuppetReferences::OUTPUT_DIR + 'openvox/man' def initialize(*) - @latest = '/puppet/latest/man' + @latest = '/openvox/latest/man' super end diff --git a/lib/puppet_references/puppet/puppet_doc.rb b/lib/puppet_references/puppet/puppet_doc.rb index c2e7175da..78e9ea5db 100644 --- a/lib/puppet_references/puppet/puppet_doc.rb +++ b/lib/puppet_references/puppet/puppet_doc.rb @@ -5,10 +5,10 @@ module PuppetReferences module Puppet class PuppetDoc < PuppetReferences::Reference REFERENCES = %w[configuration].freeze - OUTPUT_DIR = PuppetReferences::OUTPUT_DIR + 'puppet' + OUTPUT_DIR = PuppetReferences::OUTPUT_DIR + 'openvox' def initialize(*) - @latest = '/puppet/latest' + @latest = '/openvox/latest' super end diff --git a/lib/puppet_references/puppet/strings.rb b/lib/puppet_references/puppet/strings.rb index d50a7b1d8..7d30cc01e 100644 --- a/lib/puppet_references/puppet/strings.rb +++ b/lib/puppet_references/puppet/strings.rb @@ -8,7 +8,7 @@ module PuppetReferences module Puppet class Strings < Hash - STRINGS_JSON_FILE = PuppetReferences::OUTPUT_DIR + 'puppet/strings.json' + STRINGS_JSON_FILE = PuppetReferences::OUTPUT_DIR + 'openvox/strings.json' @@strings_data_cached = false def initialize(force_cached: false) diff --git a/lib/puppet_references/puppet/type.rb b/lib/puppet_references/puppet/type.rb index 3cecd837e..c56a86c3d 100644 --- a/lib/puppet_references/puppet/type.rb +++ b/lib/puppet_references/puppet/type.rb @@ -15,9 +15,9 @@ class Type < PuppetReferences::Reference PREAMBLE = PREAMBLE_FILE.read def initialize(*) - @latest = '/puppet/latest' - @output_dir_unified = PuppetReferences::OUTPUT_DIR + 'puppet' - @output_dir_individual = PuppetReferences::OUTPUT_DIR + 'puppet/types' + @latest = '/openvox/latest' + @output_dir_unified = PuppetReferences::OUTPUT_DIR + 'openvox' + @output_dir_individual = PuppetReferences::OUTPUT_DIR + 'openvox/types' @base_filename = 'type' super end diff --git a/lib/puppet_references/puppet/type_preamble.md b/lib/puppet_references/puppet/type_preamble.md index 591717276..dfa1e38ec 100644 --- a/lib/puppet_references/puppet/type_preamble.md +++ b/lib/puppet_references/puppet/type_preamble.md @@ -6,7 +6,7 @@ This is the documentation for Puppet's built-in resource types and providers. Ad You can find and install modules by browsing the [Puppet Forge](http://forge.puppet.com). See each module's documentation for -information on how to use its custom resource types. For more information about creating custom types, see [Custom resources](/docs/puppet/latest/custom_resources.html). +information on how to use its custom resource types. For more information about creating custom types, see [Custom resources](/openvox/latest/custom_resources.html). > As of Puppet 6.0, some resource types were removed from Puppet and repackaged as individual modules. These supported type modules are still included in the `puppet-agent` package, so you don't have to download them from the Forge. See the complete list of affected types in the [supported type modules](#supported-type-modules-in-puppet-agent) section. @@ -14,7 +14,7 @@ information on how to use its custom resource types. For more information about To manage resources on a target system, declare them in Puppet manifests. For more details, see -[the resources page of the Puppet language reference.](/docs/puppet/latest/lang_resources.html) +[the resources page of the Puppet language reference.](/openvox/latest/lang_resources.html) You can also browse and manage resources interactively using the `puppet resource` subcommand; run `puppet resource --help` for more information. diff --git a/lib/puppet_references/puppet/type_strings.rb b/lib/puppet_references/puppet/type_strings.rb index 229f2f281..04c8a0167 100644 --- a/lib/puppet_references/puppet/type_strings.rb +++ b/lib/puppet_references/puppet/type_strings.rb @@ -7,7 +7,7 @@ module Puppet class TypeStrings < PuppetReferences::Puppet::Type def initialize(*) super - @output_dir_individual = PuppetReferences::OUTPUT_DIR + 'puppet/types_strings' + @output_dir_individual = PuppetReferences::OUTPUT_DIR + 'openvox/types_strings' @base_filename = 'type_strings' end @@ -16,7 +16,7 @@ def get_type_json # 2. Munge it to match the old format I threw together, which the template uses. # 3. Dump result to JSON. strings_data = PuppetReferences::Puppet::Strings.new - File.write('references_output/puppet/raw_strings_data_output.json', strings_data) + File.write('references_output/openvox/raw_strings_data_output.json', strings_data) type_hash = strings_data['resource_types'].each_with_object({}) do |type, memo| memo[ type['name'] ] = { 'description' => type['docstring']['text'], diff --git a/lib/puppet_references/puppet/yard.rb b/lib/puppet_references/puppet/yard.rb index 2c260493d..3abc65874 100644 --- a/lib/puppet_references/puppet/yard.rb +++ b/lib/puppet_references/puppet/yard.rb @@ -4,10 +4,10 @@ module PuppetReferences module Puppet class Yard < PuppetReferences::Reference - OUTPUT_DIR = PuppetReferences::OUTPUT_DIR + 'puppet/yard' + OUTPUT_DIR = PuppetReferences::OUTPUT_DIR + 'openvox/yard' def initialize(*) - @latest = '/puppet/latest/yard' + @latest = '/openvox/latest/yard' super end diff --git a/lib/puppet_references/repo.rb b/lib/puppet_references/repo.rb index a4093e0a0..12edac013 100644 --- a/lib/puppet_references/repo.rb +++ b/lib/puppet_references/repo.rb @@ -14,7 +14,7 @@ def initialize(name, directory, sources = nil, config = nil) @sources = if sources [sources].flatten else - ["git@github.com:openvoxproject/#{@name}.git"] + ["https://github.com/openvoxproject/#{@name}.git"] end @main_source = @sources[0] unless Dir.exist?(@directory + '.git') || @config['skip_download']