what problems have you run into with redirection in the terminal? so far I have:

1. trying to read and write to the same file overwrites the file with an empty version (like `grep blah file.txt > file.txt`)
2. `sudo echo blah > /etc/file.txt` doesn't work (you can `echo blah | sudo tee file.txt` instead)
3. remembering where you're supposed to put 2>&1 to redirect stderr to stdout is hard

what am I missing?

@b0rk output getting buffered and not immediately landing in your redirection target can sometimes lead to a mismatch in expectations about what's going to be in a file (or how soon).

( related: https://blog.plover.com/Unix/stdio-buffering-2.html )

@brennen thanks, this is a great one

@b0rk bash noclobber is a big help in preventing accidents.
How redirection loses the fancy tty with colors and positioning and stuff, so modern tools like pv or journalctl misbehave.
The 2>&1 thing still confuses me and I've tried every few years to understand it.

@nelson thanks, these are great

@b0rk That there is no reasonable way to pipe stdout and stderr to different things. Not impossible, but unreasonable. Awkward to pipe just stdout to multiple things (daisy chain of "| tee >|(CMD)" or your shell's equivalent for that, need to wrap it in a complicated subshell expression for the other).

@gabe thanks, that's a great one

@b0rk a minor variation on #1: 'ls > list.txt' includes list.txt in the output, when probably you wanted the list of files that existed _before_ that.

(I have a small Unix tool called 'reservoir' that I use to work around this and the original #1)

@simontatham is reservoir inspired by `sponge` from moreutils? (or the reverse?)

@b0rk typos. Fat-finger > when you meant < and poof! Your source file is gone. That's why I always cat instead, and ignore the "useless use of cat" people.

@ksenzee thanks, this is a great one

(the "useless use of cat" thing always feels so silly to me, like what possible impact could using `cat` unnecessarily have?)

@b0rk it's independently invented. I only heard about 'sponge' a month
or two ago, and I've had 'reservoir' kicking around for many years.

https://www.chiark.greenend.org.uk/~sgtatham/utils/reservoir.html

@simontatham nice, a sign of a useful tool! I like that they both have water-themed names

(i also only heard about sponge for the first time last week)

@b0rk Sometimes I accidentally nano … 2>&1 | less and then my terminal is basically useless. I think you can break out with the nano exit sequence but things get weird

@alh haha I just tried this and it's amazing

@b0rk order matters! I had always assumed that `cmd 2>&1 >file` would send both stderr and stdout to the file. But I recently discovered that this is not true; stderr gets redirected to stdout and only stdout gets redirected to the file.

@sarahcodes it's the worst

@b0rk using < for file input always being to me a weird spot in a command invocation. shellcheck always yells about a useless use of cat, but file input redirect just feels wrong for a language that supports pipes reading left to right.

@mgattozzi I really wish shellcheck would remove that check, like who cares. I guess I could probably turn it off somehow.

it's interesting to me how there are a bunch of problems with redirects of the form "this is confusing / hard to remember”, like:

- how the location of 2>&1 changes depending on whether you're redirecting to a file or into a pipe (`cmd 2>&1 | other_cmd` vs `cmd > blah 2>&1`)
- accidentally using > when you meant >>
- using `<` just feels awkward, like `cat blah | ...` always feels nicer to me despite the "useless use" pedants
- redirecting with here docs looks weird `cmd <<EOF >file.txt …`

@b0rk oh yes, another one:

Redirection is set up by the shell before it even _tries_ to run your command. So if the command fails, or never starts in the first place, then the output file has already been created, and ends up zero-sized.

But it has a current datestamp. So if this happened in the context of a failed make rule, the _next_ run of make will believe the file was created successfully, and continue on that assumption, and fail on the step that consumes the file!

@simontatham thanks that's a great one

@b0rk `blah --help | less` sometimes works and sometimes doesn’t, because programs are inconsistent and often send help text to stderr

@b0rk not seeing that the program crashed and/or printed an error when doing `| less`

@ehabkost thanks, that's a classic :)

@b0rk

• <<< and <() only work in bash
• << has weird quoting rules
• handling errors from command substitutions in heredocs is really hard
• some processes behave differently when stdout is a tty; for example ls behaves differently than x=$(ls); echo "$x". you have to trick them with expect or script or something like that, and both of those are non-portable.
• stdout is buffered but stderr is immediately flushed so you get weird overlaps
• | pipelines only work on stdout, so you have to care about whether the command emits to stdout or stderr. this especially matters when there's a lot of error output and you're trying to pipe it to a pager

@jyn thanks these are great!

(also in case you're interested/didn't know, my mastodon client really did not like the ul for some reason, not sure why)

@jyn I genuinely did not know "stdout is buffered but stderr is immediately flushed so you get weird overlaps”, that's wild

@b0rk oh gosh sorry
i guess i need to start turning off markdown formatting :( i thought it would be fine since it just renders to html but apparently many clients just strip the tags

this is what it looks like for me:

@jyn no worries I just clicked through to the view on your instance where it looks normal :)

@b0rk I feel so validated RN about the "useless" thing, `cat blah | ` has always seemed so much clearer to me 😭😭

@hapax haha yeah like the computer's job is to make things convenient for *me*, not the other way around :)

@b0rk #3 is a big one for me I never remember how to do that. Other things:

- I can never figure out when I need `cat foo.txt | whatever` vs `whatever < foo.txt`
- I only sort of understand how/when to use xargs correctly/effectively
- I’ve occasionally used > instead of | and have overwritten things I mean to pipe to as commands 🤦

@jacob @b0rk

I’ve occasionally used > instead of | and have overwritten things I mean to pipe to as commands

zsh without CLOBBER has saved my bacon numerous times for this - it prevents > from overwriting and you need to use >! instead to force an overwrite.