Tutorial :How do I write stderr to a file while using “tee” with a pipe?



Question:

I have below a command which will print the output of aaa.sh to the screen while also writing stdout to bbb.out; however I would also like to write stderr to a file named ccc.out. Any suggestions on how to modify the piece below?

./aaa.sh | tee ./bbb.out  

Update: stdout and stderr should still both be printed to the screen, regardless.


Solution:1

I'm assuming you want to still see STDERR and STDOUT on the terminal. You could go for Josh Kelley's answer, but I find keeping a tail around in the background which outputs your log file very hackish and cludgy. Notice how you need to keep an exra FD and do cleanup afterward by killing it and technically should be doing that in a trap '...' EXIT.

There is a better way to do this, and you've already discovered it: tee.

Only, instead of just using it for your stdout, have a tee for stdout and one for stderr. How will you accomplish this? Process substitution and file redirection:

command > >(tee -a stdout.log) 2> >(tee -a stderr.log >&2)  

Let's split it up and explain:

> >(..)  

>(...) (process substitution) creates a FIFO and lets tee listen on it. Then, it uses > (file redirection) to redirect the STDOUT of command to the FIFO that your first tee is listening on.

Same thing for the second:

2> >(tee -a stderr.log >&2)  

We use process substitution again to make a tee process that reads from STDIN and dumps it into stderr.log. tee outputs its input back on STDOUT, but since its input is our STDERR, we want to redirect tee's STDOUT to our STDERR again. Then we use file redirection to redirect command's STDERR to the FIFO's input (tee's STDIN).

See http://mywiki.wooledge.org/BashGuide/InputAndOutput

Process substitution is one of those really lovely things you get as a bonus of choosing bash as your shell as opposed to sh (POSIX or Bourne).


In sh, you'd have to do things manually:

out="${TMPDIR:-/tmp}/out.$$" err="${TMPDIR:-/tmp}/err.$$"  mkfifo "$out" "$err"  trap 'rm "$out" "$err"' EXIT  tee -a stdout.log < "$out" &  tee -a stderr.log < "$err" >&2 &  command >"$out" 2>"$err"  


Solution:2

why not simply:

./aaa.sh 2>&1 | tee -a log  

This simply redirects stderr to stdout, so tee echoes both to log and to screen. Maybe I'm missing something, because some of the other solutions seem really complicated.

Note: Since bash version 4 you may use |& as an abbreviation for 2>&1 |:

./aaa.sh |& tee -a log  


Solution:3

This may be useful for people finding this via google. Simply uncomment the example you want to try out. Of course, feel free to rename the output files.

#!/bin/bash    STATUSFILE=x.out  LOGFILE=x.log    ### All output to screen  ### Do nothing, this is the default      ### All Output to one file, nothing to the screen  #exec > ${LOGFILE} 2>&1      ### All output to one file and all output to the screen  #exec > >(tee ${LOGFILE}) 2>&1      ### All output to one file, STDOUT to the screen  #exec > >(tee -a ${LOGFILE}) 2> >(tee -a ${LOGFILE} >/dev/null)      ### All output to one file, STDERR to the screen  ### Note you need both of these lines for this to work  #exec 3>&1  #exec > >(tee -a ${LOGFILE} >/dev/null) 2> >(tee -a ${LOGFILE} >&3)      ### STDOUT to STATUSFILE, stderr to LOGFILE, nothing to the screen  #exec > ${STATUSFILE} 2>${LOGFILE}      ### STDOUT to STATUSFILE, stderr to LOGFILE and all output to the screen  #exec > >(tee ${STATUSFILE}) 2> >(tee ${LOGFILE} >&2)      ### STDOUT to STATUSFILE and screen, STDERR to LOGFILE  #exec > >(tee ${STATUSFILE}) 2>${LOGFILE}      ### STDOUT to STATUSFILE, STDERR to LOGFILE and screen  #exec > ${STATUSFILE} 2> >(tee ${LOGFILE} >&2)      echo "This is a test"  ls -l sdgshgswogswghthb_this_file_will_not_exist_so_we_get_output_to_stderr_aronkjegralhfaff  ls -l ${0}  


Solution:4

To redirect stderr to a file, display stdout to screen, and also save stdout to a file:

./aaa.sh 2>ccc.out | tee ./bbb.out

EDIT: To display both stderr and stdout to screen and also save both to a file, you can use bash's I/O redirection:

#!/bin/bash    # Create a new file descriptor 4, pointed at the file  # which will receive stderr.  exec 4<>ccc.out    # Also print the contents of this file to screen.  tail -f ccc.out &    # Run the command; tee stdout as normal, and send stderr  # to our file descriptor 4.  ./aaa.sh 2>&4 | tee bbb.out    # Clean up: Close file descriptor 4 and kill tail -f.  exec 4>&-  kill %1  


Solution:5

In other words, you want to pipe stdout into one filter (tee bbb.out) and stderr into another filter (tee ccc.out). There is no standard way to pipe anything other than stdout into another command, but you can work around that by juggling file descriptors.

{ { ./aaa.sh | tee bbb.out; } 2>&1 1>&3 | tee ccc.out; } 3>&1 1>&2  

See also How to grep standard error stream (stderr)? and When would you use an additional file descriptor?

In bash (and ksh and zsh), but not in other POSIX shells such as dash, you can use process substitution:

./aaa.sh > >(tee bbb.out) 2> >(tee ccc.out)  

Beware that in bash, this command returns as soon as ./aaa.sh finishes, even if the tee commands are still executed (ksh and zsh do wait for the subprocesses). This may be a problem if you do something like ./aaa.sh > >(tee bbb.out) 2> >(tee ccc.out); process_logs bbb.out ccc.out. In that case, use file descriptor juggling or ksh/zsh instead.


Solution:6

If using bash:

# Redirect standard out and standard error separately  % cmd >stdout-redirect 2>stderr-redirect    # Redirect standard error and out together  % cmd >stdout-redirect 2>&1    # Merge standard error with standard out and pipe  % cmd 2>&1 |cmd2  

Credit (not answering from the top of my head) goes here: http://www.cygwin.com/ml/cygwin/2003-06/msg00772.html


Solution:7

In my case, a script was running command while redirecting both stdout and stderr to a file, something like:

cmd > log 2>&1  

I needed to update it such that when there is a failure, take some actions based on the error messages. I could of course remove the dup 2>&1 and capture the stderr from the script, but then the error messages won't go into the log file for reference. While the accepted answer from @lhunath is supposed to do the same, it redirects stdout and stderr to different files, which is not what I want, but it helped me to come up with the exact solution that I need:

(cmd 2> >(tee /dev/stderr)) > log  

With the above, log will have a copy of both stdout and stderr and I can capture stderr from my script without having to worry about stdout.


Solution:8

The following will work for KornShell(ksh) where the process substitution is not available,

# create a combined(stdin and stdout) collector  exec 3 <> combined.log    # stream stderr instead of stdout to tee, while draining all stdout to the collector  ./aaa.sh 2>&1 1>&3 | tee -a stderr.log 1>&3    # cleanup collector  exec 3>&-  

The real trick here, is the sequence of the 2>&1 1>&3 which in our case redirects the stderr to stdout and redirects the stdout to descriptor 3. At this point the stderr and stdout are not combined yet.

In effect, the stderr(as stdin) is passed to tee where it logs to stderr.log and also redirects to descriptor 3.

And descriptor 3 is logging it to combined.log all the time. So the combined.log contains both stdout and stderr.


Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Previous
Next Post »