Oliver I'm working on a shared lib to include with coreutils, which should work on any ELF platform at least. Details of the interface are here:
http://lists.gnu.org/archive/html/bug-coreutils/2008-11/msg00178.html
Hello,
Many thanks for your informations about buffering. It helped me with my problem. In my case it was redirecting log from game to my own checking program. Method with unbuffer solved problem.
Piotr
Very useful article, thanks a lot.
It's been a long time since I've had this question:
When we have a command, pipe, and another command, example ...
find . -type f | xargs cat
... does 'find' supply its output to 'cat' command, as soon as it buffers _some_ data, or just it supplies the _whole_ output to 'cat' _after_ it has found _all_ of the files ?
Thanks again.
It depends on the source command. In the `find` case it will write to the output (pipe) as it goes, but for commands like `sort` they necessarily need to buffer all data internally before sending data through the pipe.
Thank you,
Regards.
Thank you for your page, stdbuf was all I needed and I had never heard of it. Thanks for writing the page and working with folks to make stdbuf happen.
This article really helped a lot , which solved all my buffer problems
When I was at Berkeley back around 1980 they solved this problem with "smart buffering". Instead of line buffering stdout, they simply called fflush() in stdout whenever a read was called on stdin. This is more efficient than line buffering, solves the same problem better, and can be used by every program in a pipeline. Unfortunately it didn't catch on, and I never figured out why.
@Peter, "smart buffering" makes a lot of sense. So if you had:
tail -f access.log | cut -d' ' -f1 | uniq
`cut` would fflush(stdout) before each read(stdin).
Note the current line buffering mode does that too on GNU/Linux, but there it also flushes on each line written to stdout too. I guess if one was reading in a byte at a time then you wouldn't want to fflush each time.
I guess that would be OK, if you only flushed before a read that would block. Hmm that reminds me, coreutils cat does just that using ioctl(FIONREAD).
I must look into why this didn't catch on, and whether this is applicable to other coreutils
Thanks so much for posting this article. I figure I should comment in case my situation can help anyone else.
I've been trying to solve my problem of reading continuous output from a file descriptor and piping it through 'tr' and 'tee'. It looked very similar to your example:
exec 3< <(./my-program 2>&1 | tr '\r' '\n' | tee console.log)
I spent countless hours trying to understand why removing the 'tr' command or moving it to the right of the 'tee' would revert the output behaviour back to normal. After reading your article I realized that 'tr' was waiting for 4096 bytes (?) of data to process before sending it onto 'tee'. The fix looks like this:
exec 3< <(./my-program 2>&1 | stdbuf -o0 tr '\r' '\n' | tee console.log)
Hope this helps someone!
stdin is line-buffered if it's a TTY.
stdin is line-buffered if connected to a TTY and as indicated by my program. However line buffering is ineffective on stdin.
I've made this clearer in the summary
Had that problem for a long time on my GNU/Linux-systems ,-(
Found as solution:
TTY_FP=$(tty)
echo ... >>$TTY_FP
Works well for me also with
SSH_OUTPUT=$(ssh -t ...) # '-t' important here
Best regards
Hellmut
Sorry for the lack of precision.
ssh with password login was a similar problem.
Resolved using '-t' option.
Best regards
Hellmut