feat: allow streaming, line-buffered input and output

[corneliusroemer]

The major shortcoming of sd right now is that it doesn't support streaming stdin to stdout, instead, all input is read into memory which means sd can't be used for bigger-than-memory tasks.

This PR is based on @vtronko's work in #100 but adapted for current main.

It's a bit of a graft and rough around the edges, no validation of multi-line mode switch off etc, but it's a proof of concept and already useful (roughly 3x faster than sed).

Resolves:

• Follow STDIN input #100
• Bug: Memory allocation failure #154

[CosmicHorrorDev]

Just some initial feedback to get things stared

[CosmicHorrorDev]

Suggested change

	#[arg(short = 'l', long = "line-buffered", default_value_t = false)]
	#[arg(short, long, default_value_t)]

Nit: we can use clap's "magic values" for all of these

[CosmicHorrorDev]

Currently this silently ignores the --line-buffered flag when running with files which seems pretty problematic

[CosmicHorrorDev]

Suggested change

	handle
	.write_all(replaced.as_ref())
	.expect("Error writing to standard output");
	handle.flush().expect("Error flushing output");
	handle.write_all(replaced.as_ref())?;
	handle.flush()?;

Failing to write is a pretty normal operation that we handle with the normal failure paths currently

[CosmicHorrorDev]

I'd prefer if this buffer.clear() was moved right before the .read_line(). Forgetting to clear a buffer that you're reusing is a super common mistake that's annoying to debug

[nc7s]

2 cents:

• I'd argue there's no need for a --line-buffered flag. Anyone relying on the full read behavior?
• I "unified" stdin and file scenarios into mmaps. They don't have read_line(). We can maybe implement a BufRead container for it? Or split into separate logic again?

[CosmicHorrorDev]

• I'd argue there's no need for a --line-buffered flag. Anyone relying on the full read behavior?

Very likely people are, yes. Or anyone who wants to match over multiple lines

• I "unified" stdin and file scenarios into mmaps. They don't have read_line(). We can maybe implement a BufRead container for it? Or split into separate logic again?

You can get a slice from it and wrap it in an std::io::Cursor. That being said I'm planning on splitting up at least some of the logic again

[nc7s]

@CosmicHorrorDev:

• I'd argue there's no need for a --line-buffered flag. Anyone relying on the full read behavior?

Very likely people are, yes. Or anyone who wants to match over multiple lines

Multi-line matching isn't tied to full reads, at least theoretically. Practically it might be, yes. But "line buffered" should be implementation detail. Something like --multiline might be better.

I'm curious if a temp file would be useful here.

You can get a slice from it and wrap it in an std::io::Cursor. That being said I'm planning on splitting up at least some of the logic again

TIL std::io::Cursor ;) The reason of splitting being?

[CosmicHorrorDev]

Multi-line matching isn't tied to full reads, at least theoretically. Practically it might be, yes. But "line buffered" should be implementation detail. Something like --multiline might be better.

I'm curious if a temp file would be useful here.

Allowing for line buffering inputs opens the door for streaming stdin and stdout (e.g. someone running just sd can type lines and see the live output after each line). It's more of a conceptual thing since a lot of people think of text files by line

TIL std::io::Cursor ;) The reason of splitting being?

To support streaming reads from stdin. I was looking through ripgreps source and it looks like they still special case streaming stdin, so we'll likely have to too if we want that behavior (which I do)

[nc7s]

Did a little test on Cursor<Mmap>::lines(), it works. We can unify the logic (again?) on iterating through .lines().

(Excuse my "unificationism" XD)