Tutorial :Force flushing of output to a file while bash script is still running



Question:

I have a small script, which is called daily by crontab using the following command:

/homedir/MyScript &> some_log.log  

The problem with this method is that some_log.log is only created after MyScript finishes. I would like to flush the output of the program into the file while it's running so I could do things like

tail -f some_log.log  

and keep track of the progress, etc.


Solution:1

bash itself will never actually write any output to your log file. Instead, the commands it invokes as part of the script will each individually write output and flush whenever they feel like it. So your question is really how to force the commands within the bash script to flush, and that depends on what they are.


Solution:2

I found a solution to this here. Using the OP's example you basically run

stdbuf -oL /homedir/MyScript &> some_log.log

and then the buffer gets flushed after each line of output. I often combine this with nohup to run long jobs on a remote machine.

stdbuf -oL nohup /homedir/MyScript &> some_log.log

This way your process doesn't get cancelled when you log out.


Solution:3

script -c <PROGRAM> -f OUTPUT.txt

Key is -f. Quote from man script:

 -f, --flush           Flush output after each write.  This is nice for telecooperation: one person does `mkfifo foo; script -f foo', and another can supervise real-time what is being done           using `cat foo'.  

Run in background:

nohup script -c <PROGRAM> -f OUTPUT.txt  


Solution:4

You can use tee to write to the file without the need for flushing.

/homedir/MyScript 2>&1 | tee some_log.log > /dev/null  


Solution:5

This isn't a function of bash, as all the shell does is open the file in question and then pass the file descriptor as the standard output of the script. What you need to do is make sure output is flushed from your script more frequently than you currently are.

In Perl for example, this could be accomplished by setting:

$| = 1;  

See perlvar for more information on this.


Solution:6

Would this help?

tail -f access.log | stdbuf -oL cut -d ' ' -f1 | uniq   

This will immedidately display unique entries from access.log
http://www.pixelbeat.org/programming/stdio_buffering/stdbuf-man.html


Solution:7

Buffering of output depends on how your program /homedir/MyScript is implemented. If you find that output is getting buffered, you have to force it in your implementation. For example, use sys.stdout.flush() if it's a python program or use fflush(stdout) if it's a C program.


Solution:8

How just spotted here the problem is that you have to wait that the programs that you run from your script finish their jobs.
If in your script you run program in background you can try something more.

In general a call to sync before you exit allows to flush file system buffers and can help a little.

If in the script you start some programs in background (&), you can wait that they finish before you exit from the script. To have an idea about how it can function you can see below

#!/bin/bash  #... some stuffs ...  program_1 &          # here you start a program 1 in background  PID_PROGRAM_1=${!}   # here you remember its PID  #... some other stuffs ...   program_2 &          # here you start a program 2 in background  wait ${!}            # You wait it finish not really useful here  #... some other stuffs ...   daemon_1 &           # We will not wait it will finish  program_3 &          # here you start a program 1 in background  PID_PROGRAM_3=${!}   # here you remember its PID  #... last other stuffs ...   sync  wait $PID_PROGRAM_1  wait $PID_PROGRAM_3  # program 2 is just ended  # ...  

Since wait works with jobs as well as with PID numbers a lazy solution should be to put at the end of the script

for job in `jobs -p`  do     wait $job   done  

More difficult is the situation if you run something that run something else in background because you have to search and wait (if it is the case) the end of all the child process: for example if you run a daemon probably it is not the case to wait it finishes :-).

Note:

  • wait ${!} means "wait till the last background process is completed" where $! is the PID of the last background process. So to put wait ${!} just after program_2 & is equivalent to execute directly program_2 without sending it in background with &

  • From the help of wait:

    Syntax          wait [n ...]  Key        n A process ID or a job specification  


Solution:9

Thanks @user3258569, script is maybe the only thing that works in busybox!

The shell was freezing for me after it, though. Looking for the cause, I found these big red warnings "don't use in a non-interactive shells" in script manual page:

script is primarily designed for interactive terminal sessions. When stdin is not a terminal (for example: echo foo | script), then the session can hang, because the interactive shell within the script session misses EOF and script has no clue when to close the session. See the NOTES section for more information.

True. script -c "make_hay" -f /dev/null | grep "needle" was freezing the shell for me.

Countrary to the warning, I thought that echo "make_hay" | script WILL pass a EOF, so I tried

echo "make_hay; exit" | script -f /dev/null | grep 'needle'  

and it worked!

Note the warnings in the man page. This may not work for you.


Solution:10

I don't know if it would work, but what about calling sync?


Solution:11

I had this problem with a background process in Mac OS X using the StartupItems. This is how I solve it:

If I make sudo ps aux I can see that mytool is launched.

I found that (due to buffering) when Mac OS X shuts down mytool never transfers the output to the sed command. However, if I execute sudo killall mytool, then mytool transfers the output to the sed command. Hence, I added a stop case to the StartupItems that is executed when Mac OS X shuts down:

start)      if [ -x /sw/sbin/mytool ]; then        # run the daemon        ConsoleMessage "Starting mytool"        (mytool | sed .... >> myfile.txt) &       fi      ;;  stop)      ConsoleMessage "Killing mytool"      killall mytool      ;;  


Solution:12

well like it or not this is how redirection works.

In your case the output (meaning your script has finished) of your script redirected to that file.

What you want to do is add those redirections in your script.


Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Previous
Next Post »