|
| 1 | +# [Oneway-Filesync](https://github.com/danlapid/oneway-filesync/) |
| 2 | + |
| 3 | +Sync files between different hosts over a one way network link such as a Data Diode |
| 4 | + |
| 5 | +[](https://github.com/danlapid/oneway-filesync/actions?query=workflow%3ABuild) |
| 6 | +[](https://github.com/danlapid/oneway-filesync/actions?query=workflow%3ATest) |
1 | 7 | [](https://coveralls.io/github/danlapid/oneway-filesync?branch=main) |
| 8 | +[](https://opensource.org/licenses/MIT) |
| 9 | + |
| 10 | +## Usage |
| 11 | + |
| 12 | +### Download binaries from [Releases](https://github.com/danlapid/oneway-filesync/releases) |
| 13 | + |
| 14 | +Create a config.toml file for both the sender and the receiver similarly to the example in the repo |
| 15 | + |
| 16 | +On the source machine: |
| 17 | + |
| 18 | +``` |
| 19 | +./sender |
| 20 | +./watcher |
| 21 | +``` |
| 22 | + |
| 23 | +On the target machine: |
| 24 | + |
| 25 | +``` |
| 26 | +./receiver |
| 27 | +``` |
| 28 | + |
| 29 | +To send a specific file through that is not in the watched folder: |
| 30 | + |
| 31 | +``` ./sendfiles <file/dir path> ``` |
| 32 | + |
| 33 | +## Data flow |
| 34 | + |
| 35 | +### Sender side: |
| 36 | + |
| 37 | +QueueReader (From DB) -> FileReader -> FecEncoder -> BandwidthLimiter -> UdpSender |
| 38 | + |
| 39 | +### -> Data Diode -> |
| 40 | + |
| 41 | +### Receiver side: |
| 42 | + |
| 43 | +UdpReceiver -> ShareAssember -> FecDecoder -> FileWriter -> FileCloser (Updates receiver DB) |
| 44 | + |
| 45 | +## Config |
| 46 | + |
| 47 | +- ReceiverIP : The IP the receiver will listen on and the sender will send to |
| 48 | +- ReceiverPort : The port the receiver will listen on and the sender will send to |
| 49 | +- BandwidthLimit : in Bytes/Second the sender will limit itself to this amount, suggested to be a little under link speed, if you get "buffers are filling up" error code then you might need more compute power on the receiver |
| 50 | +- ChunkSize : Data length sent in each udp datagram should be 42 bytes smaller than link MTU (14 Ethernet, 20 IP, 8 UDP) for compute efficiency it is suggested to increase the link MTU and then increase this value as well |
| 51 | +- EncryptedOutput : If true the files will be encrypted in a zip file with password `filesync` before being sent and saved to the receiver as the encrypted zip |
| 52 | +- ChunkFecRequired : Reed Solomon FEC parameter, the amount of shares that must arrive for the chunk to be reconstructed |
| 53 | +- ChunkFecTotal : Reed Solomon FEC parameter, the total amount of shares that will be sent, it is suggested that this will be a multiple of ChunkFecRequired |
| 54 | +- OutDir : Directory to write output files to on the receiver, the original directory structure will be perserved and appended to this path |
| 55 | +- WatchDir : Directory the watcher will detect file changes on and send every changed files from |
2 | 56 |
|
3 | | -go run oneway-filesync/cmd/receiver \ |
4 | | -go run oneway-filesync/cmd/sender |
5 | | - |
6 | | - |
7 | | - |
8 | | -DB Reader: |
9 | | - output: paths |
10 | | -FileReader |
11 | | - input: paths |
12 | | - output: chunks |
13 | | - -> |
14 | | -FEC |
15 | | - input: chunks |
16 | | - output: chunks |
17 | | - -> |
18 | | -Rate Limiter |
19 | | - input: chunks |
20 | | - output: chunks |
21 | | - -> |
22 | | -Socket Sender |
23 | | - input: chunks |
24 | | - |
25 | | - |
26 | | -Socker reader |
27 | | - output: chunks |
28 | | - -> |
29 | | -DeFEC |
30 | | - input: chunks |
31 | | - output: chunks |
32 | | - -> |
33 | | -File Writer |
34 | | - input: chunks: |
35 | | - output: Finished files |
36 | | - |
37 | | - |
38 | | -psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "SELECT * FROM files" |
39 | | -psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "DELETE FROM files" |
40 | | -TODO: different db for sent and received |
|
0 commit comments