Add REST plugin for user-defined policies

[Nadine-H]

Resolves #2630.

Key notes:

• The REST plugin is a builtin plugin, unlike how other plugins are defined and loaded in the dstack system.
• The REST plugin has to be enabled through the dstack server config if the user wishes to hook up a plugin server.

[Nadine-H]

Realized that setting up this plugin as an external package is unnecessarily complicating the installation process. I'm going to change it so that it's more builtin the server code.

[r4victor]

Realized that setting up this plugin as an external package is unnecessarily complicating the installation process. I'm going to change it so that it's more builtin the server code.

A good place for a built-in server plugins would be to put them in src/dstack/_internal/server/services/plugins/builtin/rest_plugin

May require renaming src/dstack/_internal/server/services/plugins.py to src/dstack/_internal/server/services/plugins/__init__.py

Let me know if you need help with refactoring.

[r4victor]

I'll create an openapi spec doc

@Nadine-H, Do you think of writing the spec manually or generating it automatically? Automatic generation is highly preferable because spec models are regularly updated with new properties and it won't be practical to maintain manually written spec.

[Nadine-H]

I'll create an openapi spec doc

@Nadine-H, Do you think of writing the spec manually or generating it automatically? Automatic generation is highly preferable because spec models are regularly updated with new properties and it won't be practical to maintain manually written spec.

Automatic generation for sure. I set up a local FastAPI server to test the rest plugin and will share the openai.json with you shortly.

[Nadine-H]

Adding documentation to the guides as well

[r4victor]

And regarding the documentation, I think we can generate the OpenAPI schema reference from the example plugin server and put it in the Reference->Plugins->Rest plugin similar to how the server OpenAPI is generated (https://github.com/dstackai/dstack/blob/master/scripts/docs/gen_openapi_reference.py). This could probably be done in a separate PR to limit the scope.

[dakota-shop]

Theoretically error: "" would cause a rejection; is that intended?

[Nadine-H]

It shouldn't, I'll fix it and add additional tests for this.

[Nadine-H]

@r4victor should we not allow empty strings and cause a pydantic ValidationError? or just treat it as a "no error" case?

[Nadine-H]

@r4victor I implemented the first option so that plugin developers are aware if they mistakenly set the error to "", but happy to switch to the other option if you prefer.

[r4victor]

LGTM. I see some unresolved comments so if you're willing to address them, please do. I'm happy to merge once CI is passed.