Reload Configuration

[icb-]

I would like to be able to reload configuration (particularly static bindings) without restarting. This could be as simple as responding to a SIGHUP as is convention, or periodically doing an os.stat on the config files checking for changes in mtime, or even inotify(7)/kqueue(2) type notifications.

[psychomario]

I think I'd rather use SIGHUP than file polling for this. inotify doesn't seem to be available natively and kqueue doesn't seem to be asynchronous. os.stat could work, but could cause crashes if e.g autosaving editor.

I think if #71 is done before this, we don't lose any state when DHCP restarts.

[psychomario]

I think my best idea for this is having a HUP handler in server.py which calls __init__ again for each running service, passing in a newly parsed server_settings object. This should overwrite all the settings in the self object for each service.

This will mean that #110 will need modifying to be called externally by the signal handler in server.py for at least HUP

[mmattioli]

@icb- what would be the use case for what you're proposing?

[icb-]

The biggest thing I'm after is rereading the static bindings without having to restart the daemon.

[psychomario]

What is lost by restarting the whole thing? It's very quick to start up and we don't keep that much important state. Especially when #110 is done.

[icb-]

I don't want the user my webserver is running as (and therefore my provisioning CGIs) to be able to restart processes. If PyPXE is watching for changes in the static bindings, I don't need to do that.

[psychomario]

@mmattioli We didn't come to a resolution on this. Is this a feature that we want? If so I can start to work on it at some point soon.

[mmattioli]

I agree with @icb-'s security concern. What's the cleanest way to implement with SIGHUP?

[psychomario]

This is all I can think of off the top of my head, but I'm very open to suggestions.

1. Rip out the signal handler from dhcp.py but still expose export_leases
2. Implement the same signal handler at the top level
3. On SIGHUP:
   • Call export_leases as normal ? shoud we import or export on HUP? export_leases does --save-leases not --static-config. Users shouldn't really be editing --save-leases so we're safe to export.
   • Re-create args by parsing CLI & cfg identical to in server.py
   • EITHER
     • Create an update method in each service which takes an args and does all of the top block of __init__ (pulling out the relevant variables to self), then the tricky bit; wait for all client interactions to finish (works better for some services than others (NBD is very long lived vs HTTP, TFTP can always be 'ready' as long as we keep the Client list) and aggressively clean up state (per service basis, ie generated leases are safe (dhcp), but files may have changed (anything that uses netboot_dir))
     • Re-start each service with the new args. This is less clean as we could kill some client interactions, but significantly easier to implement

[mmattioli]

@icb-, any opinion?

[icb-]

We should be able to set SO_REUSEPORT before binding to let us run multiple processes accepting incoming connections. When we receive a reload we can then:

1. Spin up new process(es), and wait for those to be ready to accept incoming connections.
2. Signal the old processes to close their incoming socket and terminate when all active transfers are complete.

[psychomario]

How do you propose we handle our generated dhcp leases when the user doesn't choose to save them? Just pull out the objects and somehow pass them in to the new service?

Also nbd. What's the best way to transfer the COW file handles?

Other than that, This seems like a good idea.

[icb-]

It shouldn't be too hard to shuffle the DHCP leases to the new process (multiprocessing.Pipe maybe)?

I'm not sure how to handle NBD. Since the sessions are so long lived, would it make sense to shunt each one off to its own process?

[psychomario]

I think Pipe is a bit overkill and would end up being quite a large implementation for a small item. When calling the close method to signal the old server to shut down it could return a final leases list which we could update into the new server object. There could potentially be a slight race if the new server recieves a re-lease request before we've updated the leases object. This is probably only going to be a few milliseconds though so we should be safe.

I think at the moment they're threads per client. However this doesn't help us, it probably makes it harder to update nbd settings. We need to decide what to do to old clients if the disk image changes.