server/replication: fsync frame data in wallog before writing header (#2200)

Currently in logger.rs we are not issuing any fsyncs at all. We do
`flush()` in `write_header()` but this is a no-op on Unix because there
are no userspace buffers to flush since `write_all_at()` writes directly
to kernel.

Not fsyncing can cause the following bug:

- We write X frames
- We write wallog header (containing frame count X)
- Kernel reorders writes so that header has been written out but not all
of the frame data
- Crash
- Sqld starts up, reads wallog header which claims X frames exist in
wallog, but only Y frames (Y < X) have been written
- short read in read_frame_byte_offset_mut() returns an error and
crashes the server

It's also important to sync after writing frames but before writing the
header so that there is no write reordering scenario where the header is
persistent before the frame data is.

Author: jussisaurio

libsql-server/src/replication/primary/logger.rs

@@ -177,6 +177,10 @@ impl LogFile {
     }
 
     pub fn commit(&mut self) -> anyhow::Result<()> {
+        // Ensure frame data is durable before updating the header.
+        // Without this, a crash could leave the header claiming more frames
+        // than actually exist on disk.
+        self.file.sync_data()?;
         self.header.frame_count += self.uncommitted_frame_count.into();
         self.uncommitted_frame_count = 0;
         self.commited_checksum = self.uncommitted_checksum;
@@ -193,6 +197,7 @@ impl LogFile {
     pub fn write_header(&mut self) -> anyhow::Result<()> {
         self.file.write_all_at(self.header.as_bytes(), 0)?;
         self.file.flush()?;
+        self.file.sync_all()?;
 
         Ok(())
     }
@@ -387,7 +392,7 @@ impl LogFile {
             ..self.header
         };
         new_log_file.header = new_header;
-        new_log_file.write_header().unwrap();
+        new_log_file.write_header()?;
         // swap old and new snapshot
         // FIXME(marin): the dest path never changes, store it somewhere.
         atomic_rename(&to_compact_log_path, path.join("wallog")).unwrap();