Using systemd to run airflow inside conda environment

kadu

I configured an Airflow server installed within a conda environment to run some scheduled automations. Currently, I launch the scheduler, workers and webserver directly using nohup, but I'd like to use systemd to manage it more robustly.

However, I'm having trouble launching the system with systemctl start. I added to my .service file the following:

ExecStartPre=. /etc/profile ; /home/shared/miniconda2/bin/conda activate airflow
ExecStart=/home/shared/miniconda2/envs/airflow/bin/airflow webserver --pid /run/airflow/webserver.pid

(where shared/ is not a user, just a folder inside /home/ to which all users have access)

ExecStart requires the airflow conda environment in which airflow is actually installed to be activated. To do this, I added the two commands seen in ExecStartPre: the second one actually activates this environment. Running this alone returns CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'., so I added the first one to ensure /etc/profile.d/conda.sh is loaded.

However, this still fails, seemingly while trying to run the Gunicorn Server:

Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
Traceback (most recent call last):
  File "/home/shared/miniconda2/envs/airflow/bin/airflow", line 28, in <module>
    args.func(args)
  File "/home/shared/miniconda2/envs/airflow/lib/python2.7/site-packages/airflow/bin/cli.py", line 844, in webserver
    gunicorn_master_proc = subprocess.Popen(run_args)
  File "/home/shared/miniconda2/envs/airflow/lib/python2.7/subprocess.py", line 390, in __init__
    errread, errwrite)
  File "/home/shared/miniconda2/envs/airflow/lib/python2.7/subprocess.py", line 1025, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

(timestamps omitted to improve readability a bit)

This raises a number of questions:

  • What might be causing my service to fail?
  • Is my setup nonsensical (systemd + conda + airflow)?
  • If it is nonsensical, is there a better way to run airflow more robustly than launching it directly? I was particularly interested in the enable and restart options that systemd offers.
Antonis Kouzoupis

I hope by now you have solved your problem but for reference here is my solution.

The way I solved it was to add a level of indirection for launching airflow. I have a bash script e.g. airflow_runner.sh with the following

conda activate airflow-env
$AIRFLOW_BIN/airflow $@

Then in my systemd unit file

ExecStart=PATH_TO_SCRIPT/airflow_runner.sh webserver --pid /run/airflow/webserver.pid

Collected from the Internet

Please contact [email protected] to delete if infringement.

edited at
0

Comments

0 comments
Login to comment

Related

From Dev

How to run a command inside a virtualenv using systemd

From Dev

How can I run runsnakerun on Mac OS X inside a conda environment?

From Java

docker run interactive with conda environment already activated

From Dev

cron job to run under conda virtual environment

From Dev

cron job to run under conda virtual environment

From Dev

How to run a conda command inside singularity?

From Dev

How to run a command inside a running systemd container

From Dev

Run gdb inside docker container running systemd

From Dev

Systemd service configuration using environment variable

From Dev

How can you run QtConsole in a specific Conda environment?

From Dev

How to execute python from conda environment by dvc run

From Dev

How can you run QtConsole in a specific Conda environment?

From Dev

Trying to run a python script as a service using systemd

From Dev

pylint failing in conda environment

From Java

Create empty conda environment

From Java

Use Conda environment in pycharm

From Dev

Create a conda environment offline

From Dev

Use Conda environment in pycharm

From Dev

Conda: Creating a virtual environment

From Dev

conda environment not reading dependencies

From Dev

Create a conda environment offline

From Dev

Can i run multiple processes (each with different port) using systemd?

From Dev

Run sudo script at startup under 15.04 using 'systemd'

From Dev

How do I run a single command at startup using systemd?

From Dev

Setting Environment in systemd

From Dev

How to run different tox command inside Jenkins environment?

From Dev

Python: run a python script inside the git-bash environment

From Dev

Using Python to open a shell environment, run a command and exit environment

From Dev

How to run Airflow on Windows

Related Related

  1. 1

    How to run a command inside a virtualenv using systemd

  2. 2

    How can I run runsnakerun on Mac OS X inside a conda environment?

  3. 3

    docker run interactive with conda environment already activated

  4. 4

    cron job to run under conda virtual environment

  5. 5

    cron job to run under conda virtual environment

  6. 6

    How to run a conda command inside singularity?

  7. 7

    How to run a command inside a running systemd container

  8. 8

    Run gdb inside docker container running systemd

  9. 9

    Systemd service configuration using environment variable

  10. 10

    How can you run QtConsole in a specific Conda environment?

  11. 11

    How to execute python from conda environment by dvc run

  12. 12

    How can you run QtConsole in a specific Conda environment?

  13. 13

    Trying to run a python script as a service using systemd

  14. 14

    pylint failing in conda environment

  15. 15

    Create empty conda environment

  16. 16

    Use Conda environment in pycharm

  17. 17

    Create a conda environment offline

  18. 18

    Use Conda environment in pycharm

  19. 19

    Conda: Creating a virtual environment

  20. 20

    conda environment not reading dependencies

  21. 21

    Create a conda environment offline

  22. 22

    Can i run multiple processes (each with different port) using systemd?

  23. 23

    Run sudo script at startup under 15.04 using 'systemd'

  24. 24

    How do I run a single command at startup using systemd?

  25. 25

    Setting Environment in systemd

  26. 26

    How to run different tox command inside Jenkins environment?

  27. 27

    Python: run a python script inside the git-bash environment

  28. 28

    Using Python to open a shell environment, run a command and exit environment

  29. 29

    How to run Airflow on Windows

HotTag

Archive