Skip to content
This repository was archived by the owner on Jun 2, 2025. It is now read-only.

feat: Added multi root queries to the model#84

Open
carlosvdr wants to merge 11 commits into
masterfrom
multi-root-q
Open

feat: Added multi root queries to the model#84
carlosvdr wants to merge 11 commits into
masterfrom
multi-root-q

Conversation

@carlosvdr
Copy link
Copy Markdown
Contributor

Issue: #43

@carlosvdr carlosvdr self-assigned this Jul 19, 2023
Copy link
Copy Markdown
Contributor

@aasseman aasseman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a first pass, I'll review again tomorrow.

Comment thread autoagora/logs_db.py Outdated
Comment thread autoagora/logs_db.py Outdated
Comment thread autoagora/config.py Outdated
Comment thread autoagora/config.py Outdated
Comment thread autoagora/logs_db.py Outdated
hash: bytes
query: str
query_time_ms: int = 0
timestamp: datetime = datetime.now()
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The timestamp will end up being a constant set at the moment the program starts (when the class is created), not when a new instance of this class is created.

Comment thread autoagora/logs_db.py Outdated
Comment thread autoagora/model_builder.py Outdated
Comment thread autoagora/model_builder.py Outdated
Comment thread autoagora/model_builder.py Outdated
Comment thread autoagora/model_builder.py Outdated
Comment thread autoagora/main.py Outdated
model = await mrq_model_builder(subgraph, pgpool)
await set_cost_model(subgraph, model)
# Should this also be this amount of time?
await aio.sleep(args.relative_query_costs_refresh_interval)
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a different time, otherwise it would be too slow.
Also consider using random.lognormvariate: https://www.desmos.com/calculator/k3en6o5ikf

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would be a good "default" value and apply lognormvariate to it, to give it some "randomness"?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The values in the link should be a good start

Comment thread autoagora/logs_db.py Outdated
Stddev(query_time_ms) as stddev_time
FROM
query_logs
{query_logs_table}
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aasseman So this part, didn't let me to either leave it as %s and insert values or using $1, since the syntax doesn't work if the variable is alone like that.
Had to add format to it and use it this way, however this is vulnerable and SQL injections this way are a possibility, although this variable is not accesible by the user, what do you think?
Should I then create a separate query so that we don't have that there?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it's not great. There is a SQL solution mentionned here. It also mentions that psycopg2 has a solution for this, but it's not async.
However, it seems that psycopg3 came out recently, and it supports async! Not sure if it's worth/hard porting to it though.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So looked further into this, the easiest quickest approach is to just separate the function into two or to have in the constanst file the query already defined and select the query depending the type of query its looking for , which should be the one that leaves the code the "cleanest"

@aasseman aasseman linked an issue Nov 6, 2023 that may be closed by this pull request
Signed-off-by: carlos.vdr <carlos.vdr@semiotic.ai>
Signed-off-by: carlos.vdr <carlos.vdr@semiotic.ai>
Signed-off-by: carlos.vdr <carlos.vdr@semiotic.ai>
Signed-off-by: carlos.vdr <carlos.vdr@semiotic.ai>
Signed-off-by: carlos.vdr <carlos.vdr@semiotic.ai>
Signed-off-by: carlos.vdr <carlos.vdr@semiotic.ai>
Signed-off-by: carlos.vdr <carlos.vdr@semiotic.ai>
Signed-off-by: carlos.vdr <carlos.vdr@semiotic.ai>
Signed-off-by: carlos.vdr <carlos.vdr@semiotic.ai>
Signed-off-by: carlos.vdr <carlos.vdr@semiotic.ai>
Signed-off-by: carlos.vdr <carlos.vdr@semiotic.ai>
@carlosvdr carlosvdr marked this pull request as ready for review November 24, 2023 22:17
@coveralls
Copy link
Copy Markdown

Pull Request Test Coverage Report for Build 5686688707

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 79 of 88 (89.77%) changed or added relevant lines in 6 files are covered.
  • 7 unchanged lines in 3 files lost coverage.
  • Overall coverage decreased (-0.2%) to 84.872%

Changes Missing Coverage Covered Lines Changed/Added Lines %
autoagora/logs_db.py 33 34 97.06%
autoagora/main.py 6 7 85.71%
autoagora/graph_node_utils.py 9 12 75.0%
autoagora/model_builder.py 29 33 87.88%
Files with Coverage Reduction New Missed Lines %
autoagora/main.py 1 73.85%
autoagora/model_builder.py 2 88.73%
autoagora/logs_db.py 4 90.74%
Totals Coverage Status
Change from base Build 5459144595: -0.2%
Covered Lines: 432
Relevant Lines: 509

💛 - Coveralls

1 similar comment
@coveralls
Copy link
Copy Markdown

Pull Request Test Coverage Report for Build 5686688707

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 79 of 88 (89.77%) changed or added relevant lines in 6 files are covered.
  • 7 unchanged lines in 3 files lost coverage.
  • Overall coverage decreased (-0.2%) to 84.872%

Changes Missing Coverage Covered Lines Changed/Added Lines %
autoagora/logs_db.py 33 34 97.06%
autoagora/main.py 6 7 85.71%
autoagora/graph_node_utils.py 9 12 75.0%
autoagora/model_builder.py 29 33 87.88%
Files with Coverage Reduction New Missed Lines %
autoagora/main.py 1 73.85%
autoagora/model_builder.py 2 88.73%
autoagora/logs_db.py 4 90.74%
Totals Coverage Status
Change from base Build 5459144595: -0.2%
Covered Lines: 432
Relevant Lines: 509

💛 - Coveralls

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add handling of multi-root queries latency measurement

3 participants