New morpheus stackstorm integration pack by xod442 · Pull Request #176 · StackStorm-Exchange/exchange-incubator

xod442 · 2022-02-19T18:59:58Z

Pack for Morpheus

CLAassistant · 2022-05-11T12:36:50Z

All committers have signed the CLA.

cognifloyd

OK. I read through the pack's code. I have several questions and suggestions.

Overall, it looks like this pack gets logs from morpheus, saves them in a new mongo db, then pushes them to splunk. Why this architecture? What goals does this accomplish?

cognifloyd · 2022-05-23T15:39:02Z

+
+Actions are defined in two groups:
+
+### Individual actions: GET, POST, PUT with under bar will precede each individual action


I don't think all of this is a heading.

Maybe call the heading "Action naming convention"?

And then describe how it's the http method underscore action?

Thank you for the questions and suggestions. I cleaned up the Action heading. Just a quick note on the architecture. When I read logs from Morpheus, I'm doing it on an interval timer. If the first time I pull 100 log entries and send them directly to Splunk, and then 5 minutes later I do it again, chances are I will get duplicate entries. I don't want to send anything to Splunk that I already sent, so I store them in a mongodb and mark them as unprocessed. Then when the splunk sender runs, it sets the flag to those same records as processed. When the splunk sender runs again, it will only pull the 'unprocessed' entries to Splunk avoiding any duplicate entries. At the time it was the simplest answer.

cognifloyd · 2022-05-23T15:41:12Z

+* ``get_networks``
+* ``get_alerts``
+
+### Orquestra Workflows: will not


This section seems to be about mongo, not Orquesta.

Did you mean to have a separate note about how this pack won't have Orquesta Workflows?

Fixed as requested

cognifloyd · 2022-05-23T15:43:08Z

+This application uses the mongo db installed by StackStorm. Since the DB is secured
+you will need to log into the StackStorm mongo DB as a StackStorm admin and create a separate DB
+
+# To get this pack to work with A SINGLE HOST DEPLOYMENT StackStorm mongo DB


Hmm. This should probably be a subheading, so #### instead of #.
Same for the other headings after this.

Fixed as requested

cognifloyd · 2022-05-23T15:45:22Z

+You can ignore this section when using StackStorm in docker containers. There is
+no username and password associated with the database running in the mongo container.
+Use at your own discretion.


Maybe say something like:

If your mongo instance does not have auth enabled, then you don't need to provide a dbuser and dbpass.

There are so many ways to install StackStorm, I think it would be best to avoid discussing install methods here.

Agree very confusing! Fixed as requested.

cognifloyd · 2022-05-23T15:46:59Z

@@ -0,0 +1 @@
+#


Suggested change

#

__init__.py can be an empty file.

Removed the # symbol from file.

cognifloyd · 2022-05-23T15:49:29Z

@@ -0,0 +1 @@
+#


I think you can just drop this file. Do you plan to add a sensor?

File dropped.

cognifloyd · 2022-05-23T15:51:43Z

+parameters:
+  logs:
+    required: true
+    type: array


Can you add an items schema?

Not sure how to add the schema, do I add it to the load-morpheus-logs.yaml file?

cognifloyd · 2022-05-23T15:52:21Z

+parameters:
+  logs:
+    required: true
+    type: array


items schema.

Not sure how to add the schema, do I add it to the process_logs.yaml file?

i figured this out

cognifloyd · 2022-05-23T15:52:44Z

@@ -0,0 +1 @@
+#


Suggested change

#

removed comment

cognifloyd · 2022-05-23T15:54:42Z

+        # Uncomment dbuser & dbpass if using password protected mongo database
+        # dbuser = self.config['dbuser']
+        # dbpass = self.config['dbpass']
+
+        # If running stackstorm in a singlehost deployment use this command
+        # dbclient =
+        # MongoClient('mongodb://%s:%s@localhost:27017/' % (dbuser,dbpass))


People should not need to edit the pack to use a pack's features. This should be refactored so that the db connection settings defautl to no user/pass, but allow configuring the db user, pass, host, and port.

This one is tricky and my st2 skills lack in how to allow this to be configured later. I am thinking on dropping the support of dbuser and dbpass. I used to run the st2 all-in-one installation. The mongodb always had a password, so I figured out how to login the the secure mongo client and add another user. Then I bought a macbook M1 and found I could run docker desktop and use the st2 in containers (way cool) but the mongo did not require passwords.

A possible solution could be I add them to the schema file as not required which should allow the user to configure them in the GUI once the pack is installed? Maybe? Can you give me some guidance? Thank YOU!

So, I figured it out. I add the dbuser and dbpass to the config.schema.yaml and not to the morpheus.yaml.example.
When I load the pack I can use the GUI to make sure the dbuser and dbpass are blank. I can add and remove them and st2 always updates the configs yaml file. WOW. I learned something new!

cognifloyd · 2022-05-26T15:33:46Z

I'm not sure about including the morpheus=>splunk workflow in this pack because I expect packs to focus on only one service. In any case, here's some feedback on that workflow + actions:

Passing around all of the logs like that could be resource intensive (eg the workflow context gets copied multiple times and it includes input parameters).

Maybe you could use the action_service to store an ID in the datastore: https://docs.stackstorm.com/actions.html#action-service
So, in an action you

retrieve the last ID that you sent to splunk from the datastore,
loop through the new logs until you find the ID after the last ID sent,
send the remaining log entries
save a new last ID in the datastore

Also, to avoid passing large amounts of data via action input params, I would probably try to combine your actions. So, you could have one python action that:

gets logs from morpheus (yes, in addition to the get_logs action - it can reuse the morpheus API bits)
gets the last ID from the datastore
ships the new log entries to splunk
save the new last ID in the datastore

That also has the benefit of simplifying the configuration so you don't need to know how to connect to mongo.

Another thought: You can extend the get_logs action with something like an after_id input parameter. Then inside the action it would filter the logs and only return log entries after that ID.

xod442 added 3 commits February 19, 2022 11:55

new stackstorm pack for morpheus

c373f7d

cleaned up README.md file

265cc43

fixed some flake8 issues

9bc6864

cognifloyd requested changes May 23, 2022

View reviewed changes

xod442 added 4 commits May 25, 2022 11:05

made chages to readme

803f881

made changed schema to add dbuser-dbpass as not required

549ff96

fixed lib/actions.py to manage db user

dc268e3

fixed dbuser language in the README.md

e810771


		Actions are defined in two groups:

		### Individual actions: GET, POST, PUT with under bar will precede each individual action

Conversation

xod442 commented Feb 19, 2022

Uh oh!

CLAassistant commented May 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cognifloyd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xod442 May 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xod442 May 25, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cognifloyd commented May 26, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

CLAassistant commented May 11, 2022 •

edited

Loading

xod442 May 25, 2022 •

edited

Loading

xod442 May 25, 2022 •

edited

Loading