Contributions by LinkingSST

[leonardehrenfried]

Hi @skinkie, hi @ue71603,

The LinkingSST project, which aims to build a travel planner for Switzerland and South Tyrol, needs to transform a lot of NeTEx feeds in order to import them into OTP.

We have identified this repo as a tool that already gives us a lot of what we need. First of all, we have successfully converted the current Swiss profile into EPIP and can use it in OTP. Thanks!

We will need many more conversion steps. Some examples:

• Converting the South Tyrolian NeTEx from Mentz' line versions into service dates
• Transferring "Global IDs" from <KeyValue> elements into the correct id property
• Combining several NeTEx feeds and removing duplicate stops and rewriting PassgengerStopAssignments
• Adding Quays to feeds that only use StopPlace

As you are aware, there is wide variety of NeTEx feeds and not even the expert agree on some core features, so we therefore think that the above list will become longer.

We would like to collaborate and contribute all of the above, because we have seen that many NeTEx feeds share the same problems.

Our question: are you interested in these contributions and able to review them? Will you be okay if they become quite specific to an individual feed?

Looking forward to hearing your opinions.

Thanks for this repo!

cc @rcavaliere @flaktack @clezag

[skinkie]

@leonardehrenfried the binary serializer branch would be the starting point. The current state of play is that there are much worse feeds from France that can now be successfully transformed into something valid, but due to the nature of those checks it still takes (too) long to complete.

What I would like to avoid is that fixes are are extremely specific, but define the building blocks how to do a fix, and chain them as a specific example. We must mutually agree that the ultimate goal would be to have better input.

I would say lets plan something interactive so we can agree (or disagree) on the vision of this project architecture.

[leonardehrenfried]

We must mutually agree that the ultimate goal would be to have better input.

I cannot overstate how much I agree with this statement. However, getting data producers to even acknowledge spec violations or data errors is an uphill struggle. I'm sure you are familiar with that.

As much as I would like Badger to not need to exist, I'm not holding my breath.

What I would like to avoid is that fixes are are extremely specific, but define the building blocks how to do a fix, and chain them as a specific example.

I generally agree with this. But in the end we will need very specific transformations somehow. They can of course be implemented as a complex configuration file or a kind of plug-in, but I wouldn't like something to be rejected because it's only needed for a single feed. Is that acceptable?

I would say lets plan something interactive

Do you mean a video call?

[skinkie]

I generally agree with this. But in the end we will need very specific transformations somehow. They can of course be implemented as a complex configuration file or a kind of plug-in, but I wouldn't like something to be rejected because it's only needed for a single feed. Is that acceptable?

My vision would be that you write this one of repair as a building block. And we create a separate datapipeline folder / script, to include it. @ue71603 work is basically doing pipes by individual processing steps. This works, but the overhead is serialisation / deserialisation if objects are changed multiple times. My long term goal, preventing serialization / deserialisation, would be that we define which building blocks can run concurrently or directly chained, what steps should be synchronised, etc.

I would say lets plan something interactive

Do you mean a video call?

Yep.

[leonardehrenfried]

Wednesdays (any really) works well for me, as I have no meetings.

Would that work for you @skinkie and @flaktack?

[flaktack]

Wednesdays can work, could it before 1PM (in general)?