Add Alert & Ad-Hoc (email) Queue

[Enovotny]

Enterprise solution to centralize water management emails for ad-hoc and automated processes.

Add the queue needed to allow sending an email via the API.

Needs

• Send email from Batch python script
• Send email from INTERNAL wm webpage
• Send alert if a threshold is met i.e. ELEV is above a threshold (event based)
• Forecast product notifications (Forecast is made, trigger email with rich text)

Considered

• Allow districts to send emails from the scripts directly via the AWS API / SES
• Text to Voice service (Does not exist now, but nice to have)
• Read receipt, proof it was sent out (does not exist yet)
  • Frequency of an alert exists until the receipt is done : Looking at TS and alerting based on delta or threshold Make Sure we have looked at the open source options that could provide. SHOULD this go in the OpenDCS alarm system?
  • Limitation on INBOUND messages to CWBI?
• Consider that the same queue could be used by OpenDCS to trigger alerts
• From @usace.army.mil vs @cwbi.mil
• Template / Config for how/when/what email to send
  • "send my daily email" > queue message knows what this is from a config/template
• Can get email from G6 for Dev/Test/Prod (For @USACE)
  • Must make sure we ask that it can send OUTSIDE corpsnet (not internal)

[adamscarberry]

Adding more notes:

• Sounds like a good job for lambda backed by another SQS queue
• email templates stored in database, S3 or some combo?
  • Jinja2 could be a good option for templates
  • support template versions?
• how does a batch job trigger an email? Direct access to drop message in SQS or via events API call?
  • either way with an async style approach, the batch job is going to get the "trust me bro, I'll send your message" service
    • [From 04/10/2026 chat] Should we have retry, keep track of jobs, see if message is in queue, dead letter queue(?)
• can we verify that a certain batch job/office is allowed to send the predefined email (and not allow another office to accidentally send that to recipients that it shouldn't be able to)
  • User requests a group to send to, but CAN they use that group. Input validation
• if using sqs or api call, how to we limit the payload so that giant reports aren't in the body? The payload should be standardized with the template having the variability

Structured message. Simple example, not a recommendation, but just conceptual.

{
  "template": "job_failure_v1",
  "office": "LRH",
  "severity": "HIGH",
  "recipients": {
    "to": [],
    "cc": []
  },
  "data": {
    "jobName": "hourly-water-check",
    "runId": "abc123",
    "errorMessage": "Timeout calling API",
    "timestamp": "2026-03-24T10:00:00Z"
  }
}

recipients probably belong in the template and not in the above payload.

[adamscarberry]

Even without all this complication mentioned above, I think the batch-events-api service should probably have a notification option in order to alert the proper data-admins when a job has failed.

There are no metrics exposed to cloudwatch that would allow Grafana to see that. The only option would be to have Grafana look for keywords in the logs and even then, it might get complicated to send that alert to the correct office/group.

This is where a national team approach to Batch would have a single alert email and be able to respond.

Charles Edit:
Hook into DB for failed jobs and decided to send email based on predefined POC list

[TroyWeber-SAJ]

Charles G requested I put the prototype code I put together to use Amazon text to voice and calling services. I did not do the calling as there is a cost associated. Next would be to add hooks into CWMS-Python to check some key data elements. Would set some location levels or something like that for min/max values that would be used to check against current time series data. Attached is the python script in text file format and the alert audio file.

alert.mp3

[krowvin]

Meetings

04/10/2026

• Mike said "Alarms" in DCS may not work in Dev even if it was connected to emails
• USACE domained email (have this now)
  • No knowledge of a CWBI domain email to send from
• Identify what needs to be sent (Adam: Where possible, a (Grafana?) dashboard can meet legacy email needs)
  • Daniel: Manual Email - Driven by a push button to generate an alert (not automated)
    • Able to acquire emails from in database tables of users
  • Troy: Locks, Sutron Device, phones not working. Provides a means to (on a Alert condition) send an email
• Centralize (somehow) the ability to send emails.
  • Built-in rate limiting of emails
  • Roles to say WHO can send an email (and to what groups/users)
  • ONE location for credentials/access to send emails (No SES role for each office)
  • Adam: How can BATCH jobs send emails? Does IT communicate with API or the events service itself?
• Scope of the cwms-batch-events module
  • Mike: Whatever the alerts/ad-hoc becomes would be in the same module as cwms-batch-events
• Where does [Ability to send notifications] live
  • If in cwms-batch-events runtime -> Allow User Specified Arguments via the API #156
  • Brandon: Whatever comes out of the API when you submit a request is an SQS message. No input atm (156)
  • Adam: SNS + Templates
  • Ability to trigger a job, with extra args to say "Oh and send this as an attachment and this message to this email group or email"
  • Is this API public?
    • No, something public facing (Blob?) that a backend process would look at to trigger email alerts

User logs in with CDA / cwms-batch-events 
v
user from Batch or HTTP call to cwms-batch-events with payload specifying notification type/email body adam mentions above
v
API puts this in the SQS `notifications` queue
v 
lambda for the given notification runs
v
parties get email/notification

Backlog

• PUBLIC alerts
  • What about Water Alert
  • Public members can pub/sub to "topics(?)" of the alerts channel(?)
  • We have RSS feed from CDA already

[krowvin]

Update schema (postgres) in batch for what to email, different queue, elasticmq

Something must pull messages from SQS (lambda in prod) - dispatcher lambda deployed in cwbi (repo has a dispatcher container along side the api to mock lambda)

System should handle sending alerts not up to users in the scripts (test later on if they can via requests with api key in header but they should not have to define these)

[krowvin]

FOOD FOR THOUGHT:

USER GROUPS, how do we define these and where are they stored? making a table in the Batch postgres DB is detached and subject to sync issues> What if someone is removed from CDA (retire) what is keeping our local postgres DB up-todate then?

Possible do we create 'user-groups' in CDA? Should CDA handle this or should we try to keep track of user groups in the auth services right next to the roles?