I have to download few data as author , title etc. I want to download only two book at once so I create one task for each one. When the download is finished I need to know it.
If I use queue.join() I'll know when tasks are finished but I have to wait for both task, instead, I want to put new item on queue as soon as a worker become 'free'
How Do I know when a worker become available to get a new item ? Below, you find a short code to explain what I trying to do Thanks for any help
nTasks = 2
async def worker(name):
while True:
#Wait for new book item
queue_item = await queue_.get()
#Starts to download author, title etc...
loop = asyncio.get_event_loop()
task = loop.create_task(download_books(queue_item, file))
queue_.task_done()
async def main():
try:
#We create 2 task at once
count = 0
while ( count < nTasks):
#Gets the book file name
mediaGet = ....
#Put on queue
await queue_.put(mediaGet)
#Next download
count = count + 1
contaTask = 0
#Wait until tasks are finished
await queue_.join()
I want to put new item on queue as soon as a worker become 'free'
You shouldn't need to care when a worker becomes free - the whole point of having workers is that you have a fixed number of them (two in your case) and that they drain the queue as fast as they can. You shouldn't use create_task() inside the worker because then you spawn the task in the background and discard the worker limit.
The correct way to work with a queue can look like this:
async def worker(queue):
while True:
queue_item = await queue.get()
await download_books(queue_item, file)
queue.task_done()
async def main():
queue = asyncio.Queue()
# create two workers
workers = [asyncio.create_task(worker(queue)) for _ in 2]
# populate the queue
for media in ...:
await queue.put(media)
# wait for the workers to do their jobs
await queue.join()
# cancel the now-idle workers
for w in workers:
w.cancel()
Collected from the Internet
Please contact [email protected] to delete if infringement.
Comments