Localized Name Separator Conflict

[KristjanESPERANTO]

While attempting to write a validator for the country prefix (mentioned in #154), I encountered a problem in the data.

Problem

The CSV format uses comma , as separator between localized name entries:

Name: "DE Text,EN Text,FR Text"

However, some holiday names contain commas within the text itself, causing parsing ambiguity:

Examples from Bulgaria (BG)

St. George's Day / Bulgarian Army's Day:

BG Гергьовден, Ден на храбростта и Българската армия,DE St. Georgstag, Tag der bulgarischen Armee,EN St. George's Day, and the Bulgarian Army's Day

Bulgarian Education and Culture / Slavic Script Day:

BG Ден на Българската просвета и култура и на славянската писменост,DE Tag der bulgarischen Aufklärung und Kultur, Tag der slawischen Literatur,EN Bulgarian Education and Culture, and Slavic Script Day

The parser (LocalizedTextListConverter.cs) splits by comma and expects format "XX Text" where XX is a 2-letter language code. When names contain commas, this breaks:

• Tag der bulgarischen Armee is correctly parsed
• , Tag der slawischen Literatur is incorrectly parsed (leading comma, space)

Impact

• 12+ Bulgarian holiday entries affected
• API returns malformed data for these entries
• Similar issues may exist in other languages/countries

Proposed Solution

Use pipe | separator for localized names instead of comma:

Name: "DE Weihnachtsferien|EN Christmas Holidays"

Example with commas in text:

Name: "BG Гергьовден, Ден на храбростта и Българската армия|DE St. Georgstag, Tag der bulgarischen Armee|EN St. George's Day, and the Bulgarian Army's Day"

This requires:

1. Update all CSV files
2. Update LocalizedTextListConverter.cs to split on | instead of ,
3. Update documentation
4. Optional: Add validation check.

---

I'm open to prepare PRs for this. What do you think?

[HolgerJeromin]

The default way to handle this in comma separated CVS files would be quotes.
But as we are already inside a column, this is not really clean here...
https://github.com/openpotato/openholidaysapi.data/blob/develop/src/bg/holidays/holidays.public.csv

c61071e9-d132-495a-be91-d65d22d8a510;BG;2020-05-06;;Public;National;;BG Гергьовден, Ден на храбростта и Българската армия,DE St. Georgstag, Tag der bulgarischen Armee,EN St. George's Day, and the Bulgarian Army’s Day

=>

c61071e9-d132-495a-be91-d65d22d8a510;BG;2020-05-06;;Public;National;;BG Гергьовден, Ден на храбростта и Българската армия,DE St. Georgstag, Tag der bulgarischen Armee,EN "St. George's Day, and the Bulgarian Army’s Day"

Needs less adjustments in parsers (for downstream users of non-gb csv files) as beeing not a breaking change for most languages,
but is more complex and surprising to handle.

[HolgerJeromin]

API returns malformed data for these entries

Proof of broken API data:
https://openholidaysapi.org/PublicHolidays?countryIsoCode=BG&languageIsoCode=EN&validFrom=2022-05-05&validTo=2022-05-7

[fstueber]

I anyway want to refactor the CSV format:

• Comma instead of semicolon as separator > better display in Github
• No embedded format for names/translations, instead each value having its own column. Somehting like
  Id,StartDate,EndDate,Name.de,Name.en,Comment.de,Comment.en

Thoughts?

About current encoding: I use %2C as special encoding for embedded ,. Not ideal, but it works. See examples in https://github.com/openpotato/openholidaysapi.data/blob/develop/src/de/holidays/holidays.school.sh.csv

Inseln Sylt%2C Föhr%2C Amrum%2C Helgoland und die Halligen,EN Islands Sylt%2C Föhr%2C Amrum%2C Helgoland and the Halligen

For BG, its just missing %2C encoding

[HolgerJeromin]

Very nice.
So fixing BG as a quick fix and
having a nicer CSV format in the long run would be cool.

[KristjanESPERANTO]

And with #157, we can ensure that such data with a wrongly formatted name field does not get into the repository.