The library directly calls exec() directly on model-generated code. The warning notes this should be commented out by default, but it isn't.
|
# WARNING |
|
# This program exists to execute untrusted model-generated code. Although |
|
# it is highly unlikely that model-generated code will do something overtly |
|
# malicious in response to this test suite, model-generated code may act |
|
# destructively due to a lack of model capability or alignment. |
|
# Users are strongly encouraged to sandbox this evaluation suite so that it |
|
# does not perform destructive actions on their host or network. For more |
|
# information on how OpenAI sandboxes its code, see the accompanying paper. |
|
# Once you have read this disclaimer and taken appropriate precautions, |
|
# uncomment the following line and proceed at your own risk: |
|
exec(check_program, exec_globals) |
The best solution would be to provide at least one sandbox integration as a reasonable default. I'm open to contributing this.
@mpokrass
The library directly calls
exec()directly on model-generated code. The warning notes this should be commented out by default, but it isn't.human-eval/human_eval/execution.py
Lines 40 to 50 in 6d43fb9
The best solution would be to provide at least one sandbox integration as a reasonable default. I'm open to contributing this.
@mpokrass