When debugging other things, I took a dump of the docker logs and discovered there to be a single line that, when saved to its own file, was 32.6 MiB in size.
This is insane.
this line came from postgres logging a query line (unsure whether it was an error or not because it was truncated)
This query was attempting to insert (at least from what i could see) at least 85422 records into the repo_labor table.
This logging is probably also made worse by the recursive way we process data in bulk_insert_dicts (causing failures to show up many times).
the affected line
|
bulk_insert_dicts(logger, to_insert, RepoLabor, ["repo_id", "rl_analysis_date", "file_path", "file_name" ]) |
When debugging other things, I took a dump of the docker logs and discovered there to be a single line that, when saved to its own file, was 32.6 MiB in size.
This is insane.
this line came from postgres logging a query line (unsure whether it was an error or not because it was truncated)
This query was attempting to insert (at least from what i could see) at least 85422 records into the repo_labor table.
This logging is probably also made worse by the recursive way we process data in bulk_insert_dicts (causing failures to show up many times).
the affected line
CollectOSS/collectoss/tasks/git/scc_value_tasks/core.py
Line 53 in d289de1