[repository schema] Disambiguation of attributes "presence" and "supported"

[kleihan]

Orchestra provides two similar attributes (presence, supported) with partially overlapping values.

• presence: optional, required, forbidden, ignored, constant
• supported: supported, forbidden, ignored

The attribute presence is part of the group, component and field references as well as the field definition (to define a constant value). The attribute supported is part of entityAttribGrp and not described in the specification.

It is proposed to clarify the difference between the two and/or consolidate overlapping values. This is important to identify a potential use of one or the other to exclude specific elements from a scenario that is related to another scenario (see #212, #228).

[JoanPuig]

I would propose:

Presence:

• All non-documented tags are forbidden
• Documented tags can be: Required, Optional, or Ignored
• Enforcing a constant values should be done with a validation rule or by introducing the equivalent of Literal types

Supported:

• Supported, the default
• Deprecated

And if deprecated, we can have some extra metadata informing about the removal and some elaboration text to describe the recommended alternative. Similar to what Java does:
https://www.baeldung.com/java-deprecated

[kleihan]

@JoanPuig, defining all non-documented tags as forbidden sounds like a great simplification for subsetting scenarios, especially in cases where the source scenario is very big and the subset is very small. It does require to fully declare the subset but I think that is in the interest .

In theory and for FIX messages, everybody should be using FIX Latest (or a company-wide spec derived from FIX Latest) as initial source "scenario" (could be seen as base scenario). Using that base as reference, one can define a subset for an element (e.g. Instrument component) with a minimal set of attributes. The reference attribute (e.g. id) is always required, name may be used for legibility, but then you only need to add attributes to override what is pulled in from base (Tablature works this way), e.g. a change of the presence attribute from optionalto required.

[kleihan]

@JoanPuig, I support the proposal to change the supported attribute to being either supported or deprecated as it would remove the disambiguity and solve another problem. Today, deprecation is only implicitly given by the presence of the attributes deprecated and deprecatedEP that contain a version and an EP number. The problem is that you are required to use pedigree information to express deprecation. Making this explicit as supported=deprecated decouples that from the pedigree.

We already have on attribute to point to the alternative, i.e. replacedByField. This may not be enough for all cases (not always a 1:1 replacement) but is a good start (FIX Latest currently does not use it). The elaboration text could be another new purpose (e.g. DEPRECATED) for the documentation of an element (see #229).

[kleihan]

@JoanPuig, I agree that the value constant for the presenceattribute should be discussed and possibly removed. This is how it is today:

<fixr:fieldRef id="22" name="SecurityIDSource" presence="constant" value="1"/>

Would using a literal type require to override the type attribute? We should keep the field datatype to allow validation of the literal defined in Orchestra. The field could also have a code set and the constant must match one of its values.

Using a rule is probably already possible with Orchestra v1.0 with assign (not 100% if this is correct, spec only uses it to assign values from incoming messages to response messages).

<fixr:fieldRef id="22" name="SecurityIDSource">
    <fixr:assign>1</fixr:assign>
</fixr:fieldRef>

How about removing presence=constant and defining the use of presence=ignored for constants (presence of value implies a constant) instead? It would mean that the message may contain the field with a value but that the value will be ignored and overwritten with the value defined in the Orchestra XML file.

<fixr:fieldRef id="22" name="SecurityIDSource" presence="ignored" value="1"/>

It also seems better suited for automated generation of specs out of Orchestra XML files compared to the use of a rule.

[JoanPuig]

My comments were not related to scenario inheritance. That is part of the problem I am trying to highlight, overloading those concepts for inheritance will be problematic down the road. The current definition is already a bit ambiguous.

On a given message scenario (whatever way we have defined the inheritance, not important for this) there will be fields with different types. For each field, I would like to be able to express the following:

1. is the field of type T or of type Optional[T]
2. What are the possible values of the field? Normally that can be solved with a validation rule, but, do we want a short hand notation for the "constant" case which will be common?

There are two other important concepts for a field:

1. Is the field tolerated in the message but it will be ignored by the system. Which can be described as a Optional[T] with no validation rules, but further, we tag it with this extra concept to let implementers know it will not be used.
2. Support / deprecation status of the field

Related to the supported status I would envision that from release to release a field would move through the following states:
supported -> deprecated but still used -> deprecated and not used (tolerated in the message) -> not in the spec, therefore error if field in message

[ltaikit]

@JoanPuig can you help me understand your use of "deprecated" in your last comment. Is this standard FIX fields that have been deprecated by the standard, or fields marked as depreciated in your implementation spec? Just trying to understand your context.

[kleihan]

@JoanPuig agree with the flavors of deprecation as many spec issuers support the old fields prior to actual deprecation. I think we need to take into account that there are incoming and outgoing messages from the perspective of the issuer of a spec. The concept of tolerating a field only applies to incoming messages, only the issuer can "tolerate". The outgoing message may use the term "deprecated", but only in the sense of "still used". When the issuer decides to no longer use a field in an outgoing message, it should simply be removed.

Here is a proposal to cover the state changes:

• supported: may/must be used and will be validated if present (incoming); provided (outgoing)
• marked for deprecation: may be used and will be validated if present (incoming); provided (outgoing)
• deprecated: may be used but ignored if present (incoming); not provided (outgoing)

The presence attribute is relevant except for "deprecated" where it should be "ignored" (incoming) or "optional" (outgoing). The presence attribute ignored does not really apply to outgoing messages, or? presence=required on outgoing messages simply means that it is always present. It is debatable whether fields that are to be deprecated could still be required or would always be optional. Is there a use case for this?

This is a table with the combinations that can occur (rows for supported and columns for presence):

Incoming	Optional	Required	Ignored
Supported	X	X	X
ToBeDeprecated	X	X	X
Deprecated	N/A	N/A	X

Outgoing	Optional	Required	Ignored
Supported	X	X	N/A
ToBeDeprecated	X	X	N/A
Deprecated	X	N/A	N/A

[JoanPuig]

@ltaikit the concept applies both to the FIX standard and specs that a service provider would produce. The idea is to be able to warn consumers that something they are using will change in the future in a machine readable way. If as a consumer my implementation is generating messages with fields tagged as ToBeDeprecated (using @kleihan nomenclature), I would want the FIX engine to issue a warning. Similarly, if an implementation is reading a field from an incoming message that is tagged as ToBeDeprecated, I would want the FIX engine to issue a warning.

[JoanPuig]

@kleihan I agree with the analysis of the different deprecation states and that they depend on direction.

[kleihan]

I have checked the use case for presence=forbidden in RC1 (no change from V1.0). This example is in the spec and uses presence=forbidden as part of a rule for conditional presence and not to disallow a field entirely.

<fixr:fieldRef id="99" name="StopPx" presence="optional">
    <fixr:rule name="StopOrderRequiresStopPx" presence="required">
        <fixr:when>OrdType == ^Stop</fixr:when>
    </fixr:rule>
    <fixr:rule name="LimitOrderForbidsStopPx" presence="forbidden">
        <fixr:when>OrdType \!= ^Stop</fixr:when>
    </fixr:rule>
</fixr:fieldRef>

The Orchestra examples use supported=forbidden to exclude codes from a code set.

<fixr:code value="6" sort="6" added="FIX.2.7" supported="forbidden" id="40006" name="WithOrWithout">
  <fixr:annotation>
    <fixr:documentation purpose="SYNOPSIS">With Or Without/>
  </fixr:annotation>
</fixr:code>

How would we express these use cases if we remove forbidden as value from both attributes? The second example is easier to solve as one would simply not define the code, which implies it is forbidden (see proposal from @JoanPuig above). The first case is harder as conditional rules are an important part of the semantics. Would it suffice to keep forbidden for the presence attribute and use the spec to limit its use to rules? It would then not

It seems that ambiguity can also come from things that are valid as per the schema but not intended to be used that way. The schema may become overly complex if we try to enforce the intended usages. I see this as a question to answer for the discussion #235. How far should the role of the schema go to enforce the validity of an Orchestra XML file? Do we need a compatibility checker that goes beyond the validation against the schema, e.g. to detect the usage of presence=forbidden outside of a rule?

Such a validator sits between the one only looking at the schema and the one that is specific to an encoding such as tag=value where we want to ensure additional things like naming conventions and field ordering. The definition of the term "valid" requires a defined context (schema, Orchestra semantics, encoding,...). These thoughts should feed into the necessary tooling that can be developed as a community.