Sometimes, you may wonder why my print statement in my Python script not working until the end of my program, especially when you have a fairly heavy computational program that can take up a minute or longer to run. This is due to the way system handles I/O. By default, I/O in programs is buffered which means the IO is not served immediately byte by byte rather are served from a temporary storage to the requesting program. Buffering improves IO performance by reducing the total number of calls.Standard output is buffered because it is assumed there will be far more data going through it. By buffering, the whole block is read into the buffer at once then the individual bytes are delivered to you from the (fast in-memory) buffer area.
The counterpart of buffered output is unbuffered output, which is applied when you want to ensure the output has been written immediately without delay, before continuing. For example, standard error under a C runtime library is usually unbuffered by default. There are mainly two reasons: 1. errors are supposedly infrequent, 2. you want to know it immediately.
The following is a detailed explanation of when buffered vs. unbuffered output should be used:
You want unbuffered output when you already have a large sequence of bytes ready to write to disk and want to avoid an extra copy into a second buffer in the middle.
Buffered output streams will accumulate write results into an intermediate buffer, sending it to the OS file system only when enough data has accumulated (or flush() is requested). This reduces the number of file system calls. Since file system calls can be expensive on most platforms (compared to short memcpy), the buffered output is a net win when performing a large number of small writes. A unbuffered output is generally better when you already have large buffers to send — copying to an intermediate buffer will not reduce the number of OS calls further and introduces additional work.
Unbuffered output has nothing to do with ensuring your data reaches the disk; that functionality is provided by flush(), and works on both buffered and unbuffered streams. Unbuffered IO writes don’t guarantee the data has reached the physical disk — the OS file system is free to hold on to a copy of your data indefinitely, never writing it to disk, if it wants. It is only required to commit it to disk when you invoke flush(). (Note that close() will call flush() on your behalf). — Quote from stackoverflow community wiki.
Here is an example of buffered output:
Now we get an idea of how buffered IO works. How do we force Python’s print function to output to the screen?
If you’re using Python 3.3+, it has added a flush option. By setting flush = True, the stream is forcibly flushed immediately.
print(*objects, sep=”, end = ‘\n’, file = sys.stdout, flush = False)
Another general way is to use sys.stdout.flush().
print “This will be output immediately”
Running using command line, add -u
python -u mypython.py
You can also use an unbuffered file:
f = open(‘file.txt’, ‘a’,0) # 0 is no buffer, 1 is one line, other is the buffer size
sys.stdout = open(‘file.txt’, ‘a’,0)
You can also change the default for the shell operating environment
in Linux or OSX: