pipe - Python subprocesses experience mysterious delay in receiving stdin EOF -

i reduced problem seeing in application down following test case. in code, parent process concurrently spawns 2 (you can spawn more) subprocesses read big message parent on stdin, sleep 5 seconds, , write back. however, there's unexpected waiting happening somewhere, causing code complete in 10 seconds instead of expected 5.

if set verbose=true, can see straggling subprocess receiving of messages, waiting last chunk of 3 chars---it's not detecting pipe has been closed. furthermore, if don't second process (doreturn=true), first process never see eof.

any ideas what's happening? further down example output. in advance.

from subprocess import * threading import * time import * traceback import * import sys verbose = false doreturn = false msg = (20*4096+3)*'a' def elapsed(): return '%7.3f' % (time() - start) if sys.argv[1:]:   start = float(sys.argv[2])   if verbose:     chunk in iter(lambda: sys.stdin.read(4096), ''):       print >> sys.stderr, '..', time(), sys.argv[1], 'read', len(chunk)   else:     sys.stdin.read()   print >> sys.stderr, elapsed(), '..', sys.argv[1], 'done reading'   sleep(5)   print msg else:   start = time()   def go(i):     print elapsed(), i, 'starting'     p = popen(['python','stuckproc.py',str(i), str(start)], stdin=pipe, stdout=pipe)     if doreturn , == 1: return     print elapsed(), i, 'writing'     p.stdin.write(msg)     print elapsed(), i, 'closing'     p.stdin.close()     print elapsed(), i, 'reading'     p.stdout.read()     print elapsed(), i, 'done'   ts = [thread(target=go, args=(i,)) in xrange(2)]   t in ts: t.start()   t in ts: t.join() 

example output:

  0.001 0 starting   0.003 1 starting   0.005 0 writing   0.016 1 writing   0.093 0 closing   0.093 0 reading   0.094 1 closing   0.094 1 reading   0.098 .. 1 done reading   5.103 1 done   5.108 .. 0 done reading  10.113 0 done 

i'm using python 2.6.5 if makes difference.

after way time, figured out, after quote this post jumped out @ me:

see "i/o on pipes , fifos" section of pipe(7) ("man 7 pipe")

"if file descriptors referring write end of pipe have been closed, attempt read(2) pipe see end-of-file (read(2) return 0)."

i should've known this, never occurred me - had nothing python in particular. happening was: subprocesses getting forked open (writer) file descriptors each others' pipes. long there open writer file descriptors pipe, readers won't see eof.

e.g.:

p1=popen(..., stdin=pipe, ...) # creates pipe parent process can write p2=popen(...) # inherits writer fd - long p2 exists, p1 won't see eof 

turns out there's close_fds parameter popen, solution pass close_fds=true. simple , obvious in hindsight, still managed cost @ least couple eyeballs chunks of time.