Feedback collector agent

[zytegalaxy]

I am new to this project but I have done some reading on related literature and the code.

Currently moatless-tools is setup to provide code for a given prompt in a single shot manner. However, I believe there would be a lot of room for improvement on code performance if we introduce a feedback collector agent that runs the code (require a run-time environment set for this purpose) and create constructive feedback from execution logs. This feedback can be injected into the context in the next try.

I was wondering if you have any thoughts on this and how the feedback can be best collected and injected.

For example, we can monitor several run-time environment logs for error massages and try to create feedback based on that. In some cases, warning or info level logs might be helpful.

Since logs and run-time setups are program dependent, it would be helpful to separate the concerns as much as possible. This can be done by setting forward a set of data formats how the feedback/logs/etc can be provided or maybe recommend (custom) logging library for python where programmers can inject logs where they desire.

This is a rather open ended problem and I wanted to open a discussion here and see if anyone has tried some of these approaches and what community feedback is on this topic.