error when compiling for x86_64-pc-windows-gnu

[ntzb]

hi and thanks for rperf.
it is mentioned in the readme that rperf should build and work on all major platforms, though its development and usage focus is on Linux-based systems, so that is where it will be most feature-complete.
trying to switch from iperf to rperf, I need some binaries for Windows and Linux aarch64.

compiling for Windows fails with:

❯ cargo build --release --target x86_64-pc-windows-gnu
  Downloaded winapi-build v0.1.1
  Downloaded miow v0.2.2
  Downloaded kernel32-sys v0.2.2
  Downloaded ws2_32-sys v0.2.1
  Downloaded winapi-util v0.1.5
  Downloaded windows-targets v0.48.5
  Downloaded winapi v0.2.8
  Downloaded winapi v0.3.9
  Downloaded windows-sys v0.48.0
  Downloaded winapi-x86_64-pc-windows-gnu v0.4.0
  Downloaded windows_x86_64_gnu v0.48.5
  Downloaded windows v0.48.0
  Downloaded 12 crates (20.0 MB) in 2.60s (largest was `windows` at 11.9 MB)
   Compiling winapi-x86_64-pc-windows-gnu v0.4.0
   Compiling winapi v0.3.9
   Compiling winapi-build v0.1.1
   Compiling windows_x86_64_gnu v0.48.5
   Compiling winapi v0.2.8
   Compiling libc v0.2.147
   Compiling memchr v2.6.3
   Compiling bitflags v1.2.1
   Compiling kernel32-sys v0.2.2
   Compiling ws2_32-sys v0.2.1
   Compiling cfg-if v0.1.10
   Compiling cfg-if v1.0.0
   Compiling regex-syntax v0.7.5
   Compiling windows-targets v0.48.5
   Compiling aho-corasick v1.0.5
   Compiling serde v1.0.188
   Compiling num-traits v0.2.16
   Compiling memoffset v0.6.5
   Compiling slab v0.4.9
   Compiling regex-automata v0.3.8
   Compiling unicode-width v0.1.10
   Compiling log v0.4.20
   Compiling serde_json v1.0.105
   Compiling textwrap v0.11.0
   Compiling windows-sys v0.48.0
   Compiling getrandom v0.2.10
   Compiling num_cpus v1.16.0
   Compiling vec_map v0.8.2
   Compiling ryu v1.0.15
   Compiling regex v1.9.5
   Compiling net2 v0.2.39
   Compiling atty v0.2.14
   Compiling winapi-util v0.1.5
   Compiling termcolor v1.2.0
   Compiling time v0.1.45
   Compiling strsim v0.8.0
   Compiling itoa v1.0.9
   Compiling humantime v2.1.0
   Compiling iovec v0.1.4
   Compiling env_logger v0.8.4
   Compiling clap v2.33.4
   Compiling chrono v0.4.28
   Compiling miow v0.2.2
   Compiling mio v0.6.23
   Compiling core_affinity v0.5.10
   Compiling ctrlc v3.4.1
   Compiling uuid v0.8.2
   Compiling nix v0.20.2
   Compiling simple-error v0.2.3
   Compiling rperf v0.1.8 (/Users/nitzan/Downloads/rperf)
error[E0433]: failed to resolve: could not find `sys` in `nix`
  --> src/stream/tcp.rs:23:10
   |
23 | use nix::sys::socket::{setsockopt, sockopt::RcvBuf, sockopt::SndBuf};
   |          ^^^ could not find `sys` in `nix`

error[E0433]: failed to resolve: could not find `unix` in `os`
  --> src/stream/tcp.rs:70:18
   |
70 |     use std::os::unix::io::AsRawFd;
   |                  ^^^^ could not find `unix` in `os`

error[E0433]: failed to resolve: could not find `unix` in `os`
   --> src/stream/tcp.rs:416:18
    |
416 |     use std::os::unix::io::AsRawFd;
    |                  ^^^^ could not find `unix` in `os`

error[E0433]: failed to resolve: could not find `sys` in `nix`
  --> src/stream/udp.rs:26:10
   |
26 | use nix::sys::socket::{setsockopt, sockopt::RcvBuf, sockopt::SndBuf};
   |          ^^^ could not find `sys` in `nix`

error[E0433]: failed to resolve: could not find `unix` in `os`
  --> src/stream/udp.rs:72:18
   |
72 |     use std::os::unix::io::AsRawFd;
   |                  ^^^^ could not find `unix` in `os`

error[E0433]: failed to resolve: could not find `unix` in `os`
   --> src/stream/udp.rs:439:18
    |
439 |     use std::os::unix::io::AsRawFd;
    |                  ^^^^ could not find `unix` in `os`

error[E0599]: no method named `as_raw_fd` found for struct `mio::net::TcpStream` in the current scope
   --> src/stream/tcp.rs:261:90
    |
261 | ...                   super::setsockopt(stream.as_raw_fd(), super::RcvBuf, &self.receive_buffer)?;
    |                                                ^^^^^^^^^ method not found in `TcpStream`

error[E0599]: no method named `as_raw_fd` found for struct `mio::net::TcpStream` in the current scope
   --> src/stream/tcp.rs:492:46
    |
492 |                     super::setsockopt(stream.as_raw_fd(), super::SndBuf, &self.send_buffer)?;
    |                                              ^^^^^^^^^ method not found in `TcpStream`

error[E0599]: no method named `as_raw_fd` found for struct `std::net::UdpSocket` in the current scope
   --> src/stream/udp.rs:209:46
    |
209 |                     super::setsockopt(socket.as_raw_fd(), super::RcvBuf, receive_buffer)?;
    |                                              ^^^^^^^^^ method not found in `UdpSocket`

warning: use of deprecated associated function `chrono::NaiveDateTime::from_timestamp`: use `from_timestamp_opt()` instead
   --> src/stream/udp.rs:258:52
    |
258 |             let current_timestamp = NaiveDateTime::from_timestamp(now.as_secs() as i64, now.subsec_nanos());
    |                                                    ^^^^^^^^^^^^^^
    |
    = note: `#[warn(deprecated)]` on by default

warning: use of deprecated associated function `chrono::NaiveDateTime::from_timestamp`: use `from_timestamp_opt()` instead
   --> src/stream/udp.rs:302:55
    |
302 |                 let source_timestamp = NaiveDateTime::from_timestamp(origin_seconds, origin_nanoseconds);
    |                                                       ^^^^^^^^^^^^^^

error[E0599]: no method named `as_raw_fd` found for struct `std::net::UdpSocket` in the current scope
   --> src/stream/udp.rs:475:46
    |
475 |                     super::setsockopt(socket.as_raw_fd(), super::SndBuf, send_buffer)?;
    |                                              ^^^^^^^^^ method not found in `UdpSocket`

Some errors have detailed explanations: E0433, E0599.
For more information about an error, try `rustc --explain E0433`.
warning: `rperf` (bin "rperf") generated 2 warnings
error: could not compile `rperf` (bin "rperf") due to 10 previous errors; 2 warnings emitted

it seems that there are some dependencies on *nix tcp/udp

[flan]

I've added a new branch, windows-cfg_if, which replaces my incorrect assumption that Rust would discard unused symbols during its build process, rather than trying to evaluate everything.

I also incorrectly assumed that nix and other things flagged in your build would still provide those symbols, even if their implementations were effectively just generating errors.

It seems to build in a limited cross-compilation environment here, but I haven't used Windows in over two decades, so I'm afraid I don't have any means of testing it myself.

[ntzb]

thanks for looking into this! I'll happily test.
it compiles fine now. tried a simple usage, but getting an error.
server:

C:\Apps>rperf -s
[2023-09-05T16:50:41Z INFO  rperf::server] server listening on 0.0.0.0:5199
[2023-09-05T16:50:54Z INFO  rperf::server] connection from 192.168.1.150:57412
[2023-09-05T16:50:54Z ERROR rperf::server] error in client-handler: connection lost
[2023-09-05T16:50:54Z INFO  rperf::server] 192.168.1.150:57412 disconnected

client:

C:\Apps\rperf>rperf -c htpc.lan
[2023-09-05T16:50:53Z INFO  rperf::client] connecting to server at htpc.lan:5199...
[2023-09-05T16:50:53Z INFO  rperf::client] connected to server
[2023-09-05T16:50:53Z ERROR rperf] unable to run client: operation would block

p.s. I tried cross compiling for aarch64 (target aarch64-unknown-linux-gnu, but seems to hit brick walls, on macos m1. if it's possible, can you try and build for that target as well?)

[flan]

Can you re-run this with --debug (or -d)?

I suspect the block is happening at

rperf/src/protocol/communication.rs

Line 45 in 14d3826

Ok(stream.write_all(&output_buffer)?)

, though I'm unsure of the circumstances under which the Windows networking subsystem would generate an EWOULDBLOCK response to an established, not-yet-used TCP connection, especially since write_all() is supposed to take care of overflows. So it might be happening somewhere else.

[ntzb]

server:

C:\Apps>rperf -s -d
[2023-09-05T17:22:28Z DEBUG rperf] registering SIGINT handler...
[2023-09-05T17:22:28Z DEBUG rperf] beginning normal operation...
[2023-09-05T17:22:28Z DEBUG rperf::stream::tcp::receiver] using OS assignment for IPv4 TCP ports
[2023-09-05T17:22:28Z DEBUG rperf::stream::tcp::receiver] using OS assignment for IPv6 TCP ports
[2023-09-05T17:22:28Z DEBUG rperf::stream::udp::receiver] using OS assignment for IPv4 UDP ports
[2023-09-05T17:22:28Z DEBUG rperf::stream::udp::receiver] using OS assignment for IPv6 UDP ports
[2023-09-05T17:22:28Z DEBUG rperf::utils::cpu_affinity] enumerated CPU cores: [0, 1, 2, 3, 4, 5]
[2023-09-05T17:22:28Z DEBUG rperf::utils::cpu_affinity] not applying CPU core affinity
[2023-09-05T17:22:28Z INFO  rperf::server] server listening on 0.0.0.0:5199
[2023-09-05T17:22:36Z INFO  rperf::server] connection from 192.168.1.150:57630
[2023-09-05T17:22:36Z DEBUG rperf::protocol::communication] awaiting length-value from 192.168.1.150:57630...
[2023-09-05T17:22:36Z ERROR rperf::server] error in client-handler: connection lost
[2023-09-05T17:22:36Z INFO  rperf::server] 192.168.1.150:57630 disconnected

client:

C:\Apps\rperf>rperf -c htpc.lan -d
[2023-09-05T17:22:35Z DEBUG rperf] registering SIGINT handler...
[2023-09-05T17:22:35Z DEBUG rperf] connecting to server...
[2023-09-05T17:22:35Z DEBUG rperf::stream::tcp::receiver] using OS assignment for IPv4 TCP ports
[2023-09-05T17:22:35Z DEBUG rperf::stream::tcp::receiver] using OS assignment for IPv6 TCP ports
[2023-09-05T17:22:35Z DEBUG rperf::stream::udp::receiver] using OS assignment for IPv4 UDP ports
[2023-09-05T17:22:35Z DEBUG rperf::stream::udp::receiver] using OS assignment for IPv6 UDP ports
[2023-09-05T17:22:35Z DEBUG rperf::utils::cpu_affinity] enumerated CPU cores: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]
[2023-09-05T17:22:35Z DEBUG rperf::utils::cpu_affinity] not applying CPU core affinity
[2023-09-05T17:22:35Z DEBUG rperf::protocol::messaging] preparing TCP upload config...
[2023-09-05T17:22:35Z DEBUG rperf::protocol::messaging] preparing TCP download config...
[2023-09-05T17:22:35Z INFO  rperf::client] connecting to server at htpc.lan:5199...
[2023-09-05T17:22:35Z INFO  rperf::client] connected to server
[2023-09-05T17:22:35Z DEBUG rperf::client] running in forward-mode: server will be receiving data
[2023-09-05T17:22:35Z DEBUG rperf::protocol::communication] sending message of length 172, Object {"family": String("tcp"), "kind": String("configuration"), "length": Number(32768), "receive_buffer": Number(0), "role": String("download"), "streams": Number(1), "test_id": Array [Number(107), Number(61), Number(17), Number(48), Number(58), Number(130), Number(65), Number(229), Number(177), Number(53), Number(253), Number(22), Number(160), Number(115), Number(249), Number(105)]}, to 192.168.1.181:5199...
[2023-09-05T17:22:35Z ERROR rperf] unable to run client: operation would block

[flan]

I'll hopefully be able to do some testing directly against M1/M2 hardware later this year, running native Mac OS, but for now, I'm just trusting that nix provides the facilities I read in its documentation.

aarch64 for Linux should work, though: we do a lot of testing with SBCs, even down to some 8-bit controllers. I'll need to know more about what you mean by "brick walls".

[flan]

Yeah... sending 176 bytes really shouldn't be a problem with the socket in the state I expect it's in.

But it looks like mio's implementation changed since 0.6 (in a manner that may have been later retconned by changing the function's name) and now it expects the stream to already be in non-blocking mode, rather than performing that conversion itself. It expects everything to be non-blocking, actually.

Creating another branch off of the Windows one to build this, though I'm afraid I'm working on a number of other things today, so I can't promise super-quick responses.

[flan]

Ah, it's not that mio's interface changed within 0.6, it's that it behaves very differently on Windows.

And mio 0.7+ removed set_keepalive(), which was the only reason I had introduced it in the first place.

Learning what I can.

[ntzb]

I'll hopefully be able to do some testing directly against M1/M2 hardware later this year, running native Mac OS, but for now, I'm just trusting that nix provides the facilities I read in its documentation.

aarch64 for Linux should work, though: we do a lot of testing with SBCs, even down to some 8-bit controllers. I'll need to know more about what you mean by "brick walls".

I don't think it's rperf related, just probably cross compilation rust stuff (missing linker libs, things like that) - I've already burnt a few hours on it today, it's not exactly a well documented procedure (macos m1 targeting aarch64-unknown-linux-gnu, tried 3 different linkers, ld cannot find needed libs).
it's just if you have cross compilation working, I'd be happy to see a aarch64 test binary as a shortcut :)

[ntzb]

Ah, it's not that mio's interface changed within 0.6, it's that it behaves very differently on Windows.

Learning what I can.

it's fine, surely nothing urgent about this Windows build, any effort is appreciated

[flan]

I've excised mio in the socket2 branch. While everything works (at least under Linux), there's a timing mismatch in the handoff for TCP that doesn't affect total transmission time or data sent, but does skew the data-received window.

I'm trying to figure out how to minimise its impact, but for now, you should be able to build that for Windows to see if it works.

[flan]

I ended up putting bits of mio back in place, but it should all be working now. Possibly a little more accurate than before, though on the order of microseconds, nothing too significant.

[ntzb]

tested with socket2 clean build, same exact logs unfortunately.
testing between two windows machines.
the server is only using port 5199 right?

[flan]

This morning, I observed a Windows build linked through cross-compilation from Linux (same process you were using) succeed as a traffic-sending client without issue, so the "operation would block" issue may be something configuration or policy-based (a firewall, perhaps). It's possible the previous implementation would have worked, too.

However, there is a problem when receiving traffic on Windows: only the first interval (one second) of traffic gets through with TCP and nothing with UDP, seemingly because the sender assesses a timeout, concludes the receiver isn't present, and gives up.

I'll look into this when I have time, but it's a lot less obvious. I think I already have access to all the data I need, though (trace-level logs, packet-captures).

As for aarch64/Linux, I'm still able to build without issue (Debian host), but because there's a dependence on dynamic linking (libc specifically), even if I felt comfortable providing precompiled binaries (and you'd feel comfortable accepting them), there's a high chance of a mismatch, so it's probably best to leave resolving the conflict to your own build environment (maybe use a VM on your M1 if you want to build with a native instruction-set), to avoid desync.

[ntzb]

interesting, I just (cross) compiled for macos, linux and it works between then on my x86 openwrt machine and m1 mac.
Windows machine still fails as both client and server (tried multiple branches).

regarding the aarch64, I get it, it's ok. I'll try cross

[flan]

Windows should now be able to receive UDP; still working on TCP.

I may have also implemented a working handler for the would-block case you're experiencing, though nobody I've asked to help me test has reported it themselves.