REFACTOR: Adopt codeberg.org/miekg/dns instead of bespoke fields in RecordConfig

[TomOnTime]

Issue

Problem: RecordConfig has individual fields for each rtype: MxPreference, SrvPriority, SrvWeight, SrvPort, CaaTag, CaaFlag, DsKeyTag, DsAlgorithm, DnskeyFlags, DnskeyProtocol, etc. etc. etc. This wastes memory, requires bespoke functions for each type, and so on.

Solution: Add a field called .RDATA that is an interface reference to dns.RDATA. This stores a reference to a different struct for each rtype. Thus, saving memory (once the old fields are removed). In the meanwhile, bidirectional conversion from the old and new ways is automatic. Old providers still work. New providers can use the new features.

Resolution

Release Notes

DNSControl v5 is refactoring some of the biggest internal parts of the system. There should be zero user-facing
changes. However, I'm releasing a few "test balloons" to get feedback early.

• MAJOR REFACTOR: RecordConfig now uses (a reference to) codeberg.org/miekg/dns "RDATA" struct instead of storing individual fields.
  • This enhances RFC compliance, automatically supports all RFC record types.
  • Currently the RDATA fields are stored in the old and new location, with automatic, bi-directional syncing.
  • Eventually this will save memory in a future release when the old fields are removed.
• Standardized factory for creating DomainConfig and RecordConfig structs. The old "manual" way still works, but is being deprecated.
• Replaced github.com/miekg/dns (v1) with codeberg.org/miekg/dns (v2) in various places.

Extra testing needed!

• BIND: SOA handling has been rewritten to be easier to debug and more reliable. Shouldn't have any user-visible changes but please be on the lookout for problems.
• CLOUDFLAREAPI: CF_WORKER_ROUTES() need extra testing. Internally they were represented sometimes as WORKER_ROUTE and sometimes as CF_WORKER_ROUTE. It's amazing such complex code ever worked. Now we use CF_WORKER_ROUTE exclusively. The changes were core to the worker feature. Pleaes give extra attending and testing.
• IMPORT_TRANSFORM() hasn't changed but related parts have.
• Any provider that implements pseudo-types such as CF_WORKER_ROUTES(), BUNNY_DNS_PZ(), etc.

If you maintain a DNS provider, please give this release extra testing!

I only have automated testing for the following providers. All others are at risk of being broken!

AXFRDDNS_DNSSEC AZURE_DNS AZURE_PRIVATE_DNS BIND CLOUDFLAREAPI CNR
DIGITALOCEAN GANDI_V5 GCLOUD HEDNS MYTHICBEASTS NETNOD NS1 ROUTE53
SAKURACLOUD TRANSIP VERCEL

Provider maintainers! Please run the integration tests on this branch and report back any problems. (This is also an excellent opportunity to add your provider to the automated testing list)

Developer notes

• See documentation/developer-info/cookbook.md for developer notes
• It is now significantly easier to write providers. The translation from native to models.RecordConfig is much easier thanks to new factory functions.
• Supporting new DNS record types, including custom record types, is significantly easier.

Future

• "PTR magic" and "REV()" continue to work, basically unchanged. There is now an opportunity to reimplement them in a cleaner way.

Workarounds required for codeberg.org/miekg/dns

The migration to the new DNS package was very smooth. However there were some things I couldn't figure out how to do:

• To implement pkg/txtutil/miekg.go ZoneifyQuoted(), we needed access to https://codeberg.org/miekg/dns/src/branch/main/internal/ddd/ddd.go. However, that is "internal" and therefore could not be imported. I copied the files needed to pkg/txtutil/{miekg.go miekg_test.go ddd/ddd.go}. It would be nice to not have to copy those.
• dnsutil.Trim() returns "" when z is longer than s. That's rather unintuitive considering that strings.TrimPrefix() returns the original string in that situation. The "" result is ambiguous because it can also mean "s == z". I wrote pkg/txtutil.StripZone() which adopts a more useful behavior (though my implementation is not as performant).
• There's no way to parse the SVCB params ("port=80 ech=1234") into a svcb.Pairs. You can run NewData("1 port=80 ech=1234") and extract the .Value. Not a big deal, but it would be nice to be able to construct the Pairs directly.

Changes

• Add factory for models.DomainConfig: models.NewDomainConfig(zone).
• Add factories for models.RecordConfig, replace old code where we can.
• Down-casing, canonicalization, IDN, Stutter checking, and normalizing fields is now done when making the RecordConfig, not at the validation/normalization step. pkg/normalize/validate.go still exists and is used, but is slowly being deprecated.
• Remove the rtypecontrol module. In the few places it was used, replace with the new RDATA functionality.
• DNS types RP and DS were reimplemented using the new RDATA system.
• Bidirectionally convert old models.RecordConfig fields to new.
• Replace RecordConfig.Comparable with RecordConfig.ComparableV3 (name change to find stragglers).
• Custom types are now described in YAML with code generated automatically (pkg/privatetypes/types_generate.yaml)
• TLSA comparison is now done on ToUpper, not ToLower, strings.
• Add "cookbook" of how to use new factories.
• Integration tests: Test cfworkers and cfredirect by default.
• Integration tests: Improve SVCB/HTTPS tests, especially for ech=IGNORE
• LOC() is now a "builder" that outputs LOC records.
• LOC floating point rounding error fixed
• models.RegisterBuilder() allows you to register a new builder.
• No longer store the "Raw" domain names (the name as the user input them). They were never used.
• D_EXTEND() refactored. It can be much more simple and faster thanks to the other refactoring.
• RecordConfig now stores .TypeNum which is the numeric value for the type. Eventually we'll remove .Type.
• Change github.com/miekg/dns to codeberg.org/miekg/dns (and related packages) where possible, including new helper functions in pkg/dnsrr/dnsrr.go to help migrate away from dnsv1
• pkg/js/helpers.js: Improve rawRecordBuilder() to be feature-compatible with recordBuilder()
• pkg/js/helpers.js: Convert to "the new way" for A AAAA CAA CF_REDIRECT CF_SINGLE_REDIRECT CF_TEMP_REDIRECT CNAME DHCID DNAME DNSKEY DS HTTPS LOC MX NAPTR NS OPENPGPKEY PTR R53_ALIAS RP SMIMEA SOA SRV SSHFP SVCB TLSA
• pkg/js/parse_tests update fixtures due to new JSON fields. .json files no longer have Unicode chars.
• Zonefiles now include "real" data for custom types instead of comments.
• New package: mustbe for converting raw data to the types we need.
• BIND: Zonefile generator produces better files.
• BIND: Refactor SOA serial number handling.
• CLOUDFLAREAPI: Update SINGLE_REDIRECT, CF_REDIRECT, CF_TMP_REDIRECT, CF_WORKER_ROUTE to comply with the new way to do custom record types.
• New functions ZoneifyQuoted, Zoneify, etc. are standard ways to create zonefile-compatible strings.

TODO

These may or may not make it into v5:

• Use codeberg.org/miekg/dns's "Unknown record type" handling for unknown records seen in APIs.
• REV() will switch from RFC2317 to RFC4183 in v5.0. This is a breaking change. Warnings are output if your configuration is affected. No date has been announced for v5.0. See https://docs.dnscontrol.org/language-reference/top-level-functions/revcompat
• Remove the "-" support in "dnscontrol get-zones"
• Bring back TestWriteZoneFileEach
• Move txtutil.StripZone() to a more-appropriate pkg.

[imlonghao]

I tested this with PORKBUN and breaking.

Can we remove PORKBUN_URLFWD since it's an alias to URL and URL301 since #3951 and the new version is v5?

=== RUN   TestDNSProviders
Testing Profile="PORKBUN" (TYPE="PORKBUN")
=== RUN   TestDNSProviders/122323.xyz
=== RUN   TestDNSProviders/122323.xyz/Clean_Slate:Empty
    helpers_integration_test.go:246: 
        - DELETE final.122323.xyz TXT "TestDNSProviders was successful!" ttl=600, porkbun ID: 560927484
=== RUN   TestDNSProviders/122323.xyz/99:PORKBUN_URLFWD_tests:Add_a_urlfwd
WARNING: `PORKBUN_URLFWD` is deprecated. Please use `URL` or `URL301` instead.
    helpers_integration_test.go:246: 
        + CREATE urlfwd1.122323.xyz URL "http://example.com" "" "" "" includePath=no wildcard=yes ttl=0
    helpers_integration_test.go:251: failed create url forwarding record (porkbun): porkbun API error: Invalid value for location. URL:api.porkbun.com/api/json/v3/domain/addUrlForward/122323.xyz 
--- FAIL: TestDNSProviders (10.23s)
    --- FAIL: TestDNSProviders/122323.xyz (10.23s)
        --- PASS: TestDNSProviders/122323.xyz/Clean_Slate:Empty (7.00s)
        --- FAIL: TestDNSProviders/122323.xyz/99:PORKBUN_URLFWD_tests:Add_a_urlfwd (3.21s)
FAIL
exit status 1
FAIL    github.com/DNSControl/dnscontrol/v4/integrationTest     11.085s
failed to wait for command termination: exit status 1