Issue: had to use same system name for intersect client -> proxy server -> proxy client -> intersect service

[marshallmcdonnell]

This was issue we experienced when we had setup for AWS <-> ORC intersect instances.

Diagram of setup:

Issues:

• [primary]: had to use intersect service name ornl.mdf.roost on AWS intersect client, AWS proxy client, and ORNL proxy client; this did not seem correct in how we want it to behave; Yet, pragmatically trying all the different hierarchy options, this was only way to get messages over the broker proxy setup
  • Setup: AWS intersect client -> AWS proxy service -> ORNL proxy client -> ORNL INTERSECT service
  • The system name was dictated by the ORNL INTERSECT service
  • Example of hierarchy for the AWS intersect client was:

    # Hierarchy for this client
    # NOTE: for proxy hop, MUST MATCH ORNL ONE (weird, I know...)
    hierarchy_client:
      organization: ornl
      facility: mdf
      system: roost
    

  • Example of ORNL proxy client helm chart values.yaml section

  aws-proxy-http-client:
  enabled: true
  image:
  tag: "9284ad6360fd3a1463ad2a37a23642044b952037" # last version before MQTT protocol refactor
  app:
  topic_prefix: "ornl.mdf.roost."
  log_level: "debug"
  broker:
  username: *messageBroker-username
  host: *messageBroker-internalHost
  protocol: "amqp"
  password:
  isSecret: true
  secretName: *messageBroker-password
  secretKey: *messageBroker-password-key
  other_proxy:
  url: "https://intersect.genesismission.click"
  username: "REDACTED"
  password: "" # gets set for real in the secrets file

  - ORNL INTERSECT service is just like normal SDK v0.8 hierarchy definition, can provide if needed.
  
  

• [secondary]: With the current setup above, the big issue was the messages sent from the AWS intersect client were duplicated on the ORNL broker (i.e. ~10-100 intersect messages created in ~1 second; intersect services were able to ignore messages at the application level to allow this to work but only worked approximately 2 / 3 of the time); So this is probably not the main issue! And if we solve the primary one, we can test if this is fixed

[Lance-Drane]

The message duplication on the ORNL broker is probably a result of the duplication strategy you used, so can probably be ignored once we can resolve the primary issue. The fact that you're seeing so many duplicated messages is probably a result of this 'else' block with the basic_reject() being hit, probably because broadcaster.publish_message_to_http() is not succeeding at publishing its message over HTTP to the other proxy-app.

I have considered getting rid of the entire manual acknowledgment mechanism for the AMQP handlers; however, in that case the messages would just silently be dropped in the middle of transmission, which would perhaps be more frustrating to debug. The MQTT handler actually auto-acknowledges messages; unfortunately, while it is normally possible for RabbitMQ to work in such a way that a message published through MQTT/AMQP can be picked up by a queue listening to ANY protocol, the INTERSECT AMQP logic is listening on a specific exchange intersect-messages, while all MQTT logic goes through the amqp.topic exchange by RabbitMQ's design.

had to use intersect service name ornl.mdf.roost on AWS intersect client, AWS proxy client, and ORNL proxy client

Yeah this should be different between AWS and ORNL; note that between the AWS services it actually needs to be consistent, same between the ORNL services. So in total you should have needed two separate system hierarchies.

So to me it seems like the Atlas UI in this diagram is the one which has the unexpected hierarchy, and the proxy-server is the "odd one out" with regards to its hierarchy being the only one different from all the other hierarchies?