Skip to content

Commit c070e86

Browse files
authored
Update README.md
1 parent c9276a0 commit c070e86

1 file changed

Lines changed: 54 additions & 38 deletions

File tree

README.md

Lines changed: 54 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -1,40 +1,56 @@
1+
# [Oneway-Filesync](https://github.com/danlapid/oneway-filesync/)
2+
3+
Sync files between different hosts over a one way network link such as a Data Diode
4+
5+
[![Build Status](https://github.com/danlapid/oneway-filesync/actions/workflows/build.yml/badge.svg)](https://github.com/danlapid/oneway-filesync/actions?query=workflow%3ABuild)
6+
[![Test Status](https://github.com/danlapid/oneway-filesync/actions/workflows/test.yml/badge.svg)](https://github.com/danlapid/oneway-filesync/actions?query=workflow%3ATest)
17
[![Coverage Status](https://coveralls.io/repos/github/danlapid/oneway-filesync/badge.svg?branch=main)](https://coveralls.io/github/danlapid/oneway-filesync?branch=main)
8+
[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](https://opensource.org/licenses/MIT)
9+
10+
## Usage
11+
12+
### Download binaries from [Releases](https://github.com/danlapid/oneway-filesync/releases)
13+
14+
Create a config.toml file for both the sender and the receiver similarly to the example in the repo
15+
16+
On the source machine:
17+
18+
```
19+
./sender
20+
./watcher
21+
```
22+
23+
On the target machine:
24+
25+
```
26+
./receiver
27+
```
28+
29+
To send a specific file through that is not in the watched folder:
30+
31+
``` ./sendfiles <file/dir path> ```
32+
33+
## Data flow
34+
35+
### Sender side:
36+
37+
QueueReader (From DB) -> FileReader -> FecEncoder -> BandwidthLimiter -> UdpSender
38+
39+
### -> Data Diode ->
40+
41+
### Receiver side:
42+
43+
UdpReceiver -> ShareAssember -> FecDecoder -> FileWriter -> FileCloser (Updates receiver DB)
44+
45+
## Config
46+
47+
- ReceiverIP : The IP the receiver will listen on and the sender will send to
48+
- ReceiverPort : The port the receiver will listen on and the sender will send to
49+
- BandwidthLimit : in Bytes/Second the sender will limit itself to this amount, suggested to be a little under link speed, if you get "buffers are filling up" error code then you might need more compute power on the receiver
50+
- ChunkSize : Data length sent in each udp datagram should be 42 bytes smaller than link MTU (14 Ethernet, 20 IP, 8 UDP) for compute efficiency it is suggested to increase the link MTU and then increase this value as well
51+
- EncryptedOutput : If true the files will be encrypted in a zip file with password `filesync` before being sent and saved to the receiver as the encrypted zip
52+
- ChunkFecRequired : Reed Solomon FEC parameter, the amount of shares that must arrive for the chunk to be reconstructed
53+
- ChunkFecTotal : Reed Solomon FEC parameter, the total amount of shares that will be sent, it is suggested that this will be a multiple of ChunkFecRequired
54+
- OutDir : Directory to write output files to on the receiver, the original directory structure will be perserved and appended to this path
55+
- WatchDir : Directory the watcher will detect file changes on and send every changed files from
256

3-
go run oneway-filesync/cmd/receiver \
4-
go run oneway-filesync/cmd/sender
5-
6-
7-
8-
DB Reader:
9-
output: paths
10-
FileReader
11-
input: paths
12-
output: chunks
13-
->
14-
FEC
15-
input: chunks
16-
output: chunks
17-
->
18-
Rate Limiter
19-
input: chunks
20-
output: chunks
21-
->
22-
Socket Sender
23-
input: chunks
24-
25-
26-
Socker reader
27-
output: chunks
28-
->
29-
DeFEC
30-
input: chunks
31-
output: chunks
32-
->
33-
File Writer
34-
input: chunks:
35-
output: Finished files
36-
37-
38-
psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "SELECT * FROM files"
39-
psql -h 127.0.0.1 -p 5432 -U postgres -d postgres -c "DELETE FROM files"
40-
TODO: different db for sent and received

0 commit comments

Comments
 (0)