Python application multi-threading for dev environment only

Pablosan

I am working on a Python/bottle app that has two components:

  • A REST endpoint that receives messages and drops them on a queue (in Redis)
  • A 'worker', which is an infinite loop that polls for messages and processes them

In production, the two will be run in separate processes, but I'd like to run them both in the same process when running locally for development purposes.

Threading seems like the obvious answer, but threading is unnecessary when running the components in separate processes.

Is there an elegant way to optionally employ a threaded approach only when running both components in the same process?

Luke Wahlmeier

This should not be a problem since it sounds like they are completely isolated. I would just add a new script that imports threading and then either wraps the other starter scripts in functions or calls the functions they use with the new threads.

Here is a simple example:

import threading

def script1():
    import script1
def script2():
    import script2

t1 = threading.Thread(target=script1)
t2 = threading.Thread(target=script2)
t1.start()
t2.start()
t1.join()
t2.join()

That would start anything in the scripts main section of the scripts, more ideally if the script had an actual main function you could run that in the threads.

A few things to note:

  • If they are dealing with any singletons or globals they could end up tromping on each other.
  • Also since python has a GIL they might not run as fast and/or have different timings then the scripts in production.
  • A work around to both problems could be to use the multiprocessing module, they would not be in the same process, but they could be ran from the same script at the same time.

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

Ruby Multi-Threading Sql (in test environment)

From Dev

python - multi-threading in a for loop

From Dev

Multi threading using Python and pymongo

From Dev

Python Feedparser and Multi-threading

From Dev

Multi-threading in Python and shell

From Dev

python - multi-threading in a for loop

From Dev

Multi-threading windows phone application

From Dev

What actual cause the StringBuilder fails in multi threading environment

From Dev

Count number of instances of a class in multi-threading environment?

From Dev

Multi threading in python using parallel threads

From Dev

Multi threading read and write file using python

From Dev

Python how to use Threading in multi download

From Dev

Multi-threading for downloading NCBI files in Python

From Dev

Multi threading in python : issue with parallel processing

From Dev

Using a for loop along with multi threading python

From Dev

Issue with python multi threading and socket connections

From Dev

Multi threading

From Dev

Multi threading

From Java

A small sleep changes application behavior (multi-threading)

From Dev

Multi-threading Outlook Application.ItemSend event?

From Dev

A small sleep changes application behavior (multi-threading)

From Dev

Python threading only launching one extra thread

From Dev

Only first thread is running using python threading

From Dev

multi-threading in python: is it really performance effiicient most of the time?

From Dev

Python multi-threading performance issue related to start()

From Java

Python Proxy Scraper / Checker adding multi-threading trouble

From Dev

Multi-threading safety for python string operation via '%'

From Dev

How display multi videos with threading using tkinter in python?

From Dev

Button problem in multi-threading Python Tkinter Serial Monitor

Related Related

  1. 1

    Ruby Multi-Threading Sql (in test environment)

  2. 2

    python - multi-threading in a for loop

  3. 3

    Multi threading using Python and pymongo

  4. 4

    Python Feedparser and Multi-threading

  5. 5

    Multi-threading in Python and shell

  6. 6

    python - multi-threading in a for loop

  7. 7

    Multi-threading windows phone application

  8. 8

    What actual cause the StringBuilder fails in multi threading environment

  9. 9

    Count number of instances of a class in multi-threading environment?

  10. 10

    Multi threading in python using parallel threads

  11. 11

    Multi threading read and write file using python

  12. 12

    Python how to use Threading in multi download

  13. 13

    Multi-threading for downloading NCBI files in Python

  14. 14

    Multi threading in python : issue with parallel processing

  15. 15

    Using a for loop along with multi threading python

  16. 16

    Issue with python multi threading and socket connections

  17. 17

    Multi threading

  18. 18

    Multi threading

  19. 19

    A small sleep changes application behavior (multi-threading)

  20. 20

    Multi-threading Outlook Application.ItemSend event?

  21. 21

    A small sleep changes application behavior (multi-threading)

  22. 22

    Python threading only launching one extra thread

  23. 23

    Only first thread is running using python threading

  24. 24

    multi-threading in python: is it really performance effiicient most of the time?

  25. 25

    Python multi-threading performance issue related to start()

  26. 26

    Python Proxy Scraper / Checker adding multi-threading trouble

  27. 27

    Multi-threading safety for python string operation via '%'

  28. 28

    How display multi videos with threading using tkinter in python?

  29. 29

    Button problem in multi-threading Python Tkinter Serial Monitor

HotTag

Archive