我想一次启动多个10个作业，然后等待它们完成，然后在后台并行启动10个以上的作业，重复此操作，直到完成所有100个作业。
这是调用shell脚本的python代码
from subprocess import call
# other code here.
# This variable is basically # of jobs/batch.
windowsize = 10
# Here is how I call the shell command. I have 100 jobs in total that I want as 10 batches with 10 jobs/batch.
for i in range (0..100) :
numjobs = i + windowsize
# Start 10 jobs in parallel at a time
for j in range (i..numjobs) :
call (["./myscript.sh", "/usr/share/file1.txt", ""/usr/share/file2.txt"], shell=True)
# Hoping that to wait until the 10 jobs that were recently started in background finish.
call(["wait],shell=True)
在我的shell脚本中，我有这个
#!/bin/sh
# I start the job in background. Each job takes few minutes to finish.
shell command $1 $2 &
...
不幸的是，所有100个作业都已启动，而不是10个批次/批次的10个批次。
没有（直接）等待孙子进程的方法。而是wait在myscript.sh脚本末尾添加。
要限制并发运行的子进程的数量，可以使用线程池：
#!/usr/bin/env python
import logging
from multiprocessing.pool import ThreadPool
from subprocess import call
windowsize = 10
cmd = ["./myscript.sh", "/usr/share/file1.txt", "/usr/share/file2.txt"]
def run(i):
return i, call(cmd)
logging.basicConfig(format="%(asctime)-15s %(message)s", datefmt="%F %T",
level=logging.INFO)
pool = ThreadPool(windowsize)
for i, rc in pool.imap_unordered(run, range(100)):
logging.info('%s-th command returns %s', i, rc)
注意：shell=True已删除。
本文收集自互联网，转载请注明来源。
如有侵权，请联系[email protected] 删除。
我来说两句