如何通过Snakefile中的输入并行化规则（具有多个输出）

debugcn 发表于 Dev

CainãMax Couto-Silva

我对Snakemake如何在规则中并行化作业感到非常困惑。我想为每个输入使用一个核心（分别处理输入，它们之间不留空格），为每个输入提供多个输出。

这是我的代码的简化示例：

# Globals ---------------------------------------------------------------------

datasets = ["dataset_S1", "dataset_S2"]
methods = ["pbs", "pbs_windowed", "ihs", "xpehh"]

# Rules ----------------------------------------------------------------------- 

rule all:
    input:
        # Binary files
        expand("{dataset}/results_bin/persnp/{method}.feather", dataset=datasets, method=methods),
        expand("{dataset}/results_bin/pergene/{method}.feather", dataset=datasets, method=methods)


rule bin:
    input:
        "{dataset}/results_bin/convert2feather.R"
    output:
        "{dataset}/results_bin/persnp/{method}.feather",
        "{dataset}/results_bin/pergene/{method}.feather"
    threads:
        2
    shell:
        "Rscript {input}

如果使用来运行上面的代码snakemake -j2，我最终将每个输出方法重新运行每个脚本，这不是我想要的。如果我也将expand()函数用于bin规则中的输入和输出，我最终将使用：

shell:
    """
    Rscript {input[0]}
    Rscript {input[1]}
    """

我认为将无法并行化。

我应该怎么做才能单独使用每个输入，以便每个输入可以使用一个内核？
任何帮助将不胜感激。谢谢！

编辑

试图更好地解释我的脚本的功能以及我对Snakemake的期望。请参阅我的示例文件夹结构：

.
├── dataset_S1
│   ├── data
│   │   └── data.vcf
│   ├── results_bin
│   │   └── convert2feather.R
│   ├── task2
│   │   └── script.py
│   └── task3
│       └── script.sh
└── dataset_S2
    ├── data
    │   └── data.vcf
    ├── results_bin
    │   └── convert2feather.R
    ├── task2
    │   └── script.py
    └── task3
        └── script.sh

如您所见，对于每个数据集，我都有具有相同结构和命名脚本的文件夹（尽管脚本的内容可能有所不同）。在我的示例中，脚本将读取“ data.vcf”文件，进行操作，然后在相应的数据集文件夹中创建新文件夹和文件。它将对两个数据集重复全部任务。我想以某种方式可以对文件夹task2，task3等中的脚本执行相同的操作...

例如，本例中管道的输出将是：

.
├── dataset_S1
│   ├── data
│   │   └── data.vcf
│   └── results_bin
│       ├── convert2feather.R
│       ├── pergene
│       │   ├── ihs.feather
│       │   ├── pbs.feather
│       │   ├── pbs_windowed.feather
│       │   └── xpehh.feather
│       └── persnp
│           ├── ihs.feather
│           ├── pbs.feather
│           ├── pbs_windowed.feather
│           └── xpehh.feather
└── dataset_S2
    ├── data
    │   └── data.vcf
    └── results_bin
        ├── convert2feather.R
        ├── pergene
        │   ├── ihs.feather
        │   ├── pbs.feather
        │   ├── pbs_windowed.feather
        │   └── xpehh.feather
        └── persnp
            ├── ihs.feather
            ├── pbs.feather
            ├── pbs_windowed.feather
            └── xpehh.feather

编辑2

使用的文件和命令：

(snakemake) cmcouto-silva@datascience-IB:~/[email protected]/lab_files/phd_data$ snakemake -j2 -p
# Globals ---------------------------------------------------------------------

datasets = ["dataset_S1", "dataset_S2"]
methods = ["pbs", "pbs_windowed", "ihs", "xpehh"]

# Rules ----------------------------------------------------------------------- 

rule all:
    input:
        # Binary files
        expand("{dataset}/results_bin/persnp/{method}.feather", dataset=datasets, method=methods),
        expand("{dataset}/results_bin/pergene/{method}.feather", dataset=datasets, method=methods)

rule bin:
    input:
        "{dataset}/results_bin/convert2feather.R"
    output:
        expand("{{dataset}}/results_bin/persnp/{method}.feather", method=methods),
        expand("{{dataset}}/results_bin/pergene/{method}.feather", method=methods)
    threads:
        2
    shell:
        "Rscript {input}"

输出日志：

(snakemake) cmcouto-silva@datascience-IB:~/[email protected]/lab_files/phd_data$ snakemake -j2 -p
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 2
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   all
    2   bin
    3

[Wed Sep 30 23:47:55 2020]
rule bin:
    input: dataset_S1/results_bin/convert2feather.R
    output: dataset_S1/results_bin/persnp/pbs.feather, dataset_S1/results_bin/persnp/pbs_windowed.feather, dataset_S1/results_bin/persnp/ihs.feather, dataset_S1/results_bin/persnp/xpehh.feather, dataset_S1/results_bin/pergene/pbs.feather, dataset_S1/results_bin/pergene/pbs_windowed.feather, dataset_S1/results_bin/pergene/ihs.feather, dataset_S1/results_bin/pergene/xpehh.feather
    jobid: 1
    wildcards: dataset=dataset_S1
    threads: 2

Rscript dataset_S1/results_bin/convert2feather.R
  Package "data.table" successfully loaded!
  Package "magrittr" successfully loaded!
  Package "snpsel" successfully loaded!
[Wed Sep 30 23:48:43 2020]
Finished job 1.
1 of 3 steps (33%) done

[Wed Sep 30 23:48:43 2020]
rule bin:
    input: dataset_S2/results_bin/convert2feather.R
    output: dataset_S2/results_bin/persnp/pbs.feather, dataset_S2/results_bin/persnp/pbs_windowed.feather, dataset_S2/results_bin/persnp/ihs.feather, dataset_S2/results_bin/persnp/xpehh.feather, dataset_S2/results_bin/pergene/pbs.feather, dataset_S2/results_bin/pergene/pbs_windowed.feather, dataset_S2/results_bin/pergene/ihs.feather, dataset_S2/results_bin/pergene/xpehh.feather
    jobid: 2
    wildcards: dataset=dataset_S2
    threads: 2

Rscript dataset_S2/results_bin/convert2feather.R
  Package "data.table" successfully loaded!
  Package "magrittr" successfully loaded!
  Package "snpsel" successfully loaded!
[Wed Sep 30 23:49:41 2020]
Finished job 2.
2 of 3 steps (67%) done

[Wed Sep 30 23:49:41 2020]
localrule all:
    input: dataset_S1/results_bin/persnp/pbs.feather, dataset_S1/results_bin/persnp/pbs_windowed.feather, dataset_S1/results_bin/persnp/ihs.feather, dataset_S1/results_bin/persnp/xpehh.feather, dataset_S2/results_bin/persnp/pbs.feather, dataset_S2/results_bin/persnp/pbs_windowed.feather, dataset_S2/results_bin/persnp/ihs.feather, dataset_S2/results_bin/persnp/xpehh.feather, dataset_S1/results_bin/pergene/pbs.feather, dataset_S1/results_bin/pergene/pbs_windowed.feather, dataset_S1/results_bin/pergene/ihs.feather, dataset_S1/results_bin/pergene/xpehh.feather, dataset_S2/results_bin/pergene/pbs.feather, dataset_S2/results_bin/pergene/pbs_windowed.feather, dataset_S2/results_bin/pergene/ihs.feather, dataset_S2/results_bin/pergene/xpehh.feather
    jobid: 0

[Wed Sep 30 23:49:41 2020]
Finished job 0.
3 of 3 steps (100%) done
Complete log: /home/cmcouto-silva/[email protected]/lab_files/phd_data/.snakemake/log/2020-09-30T234755.741940.snakemake.log

(snakemake) cmcouto-silva@datascience-IB:~/[email protected]/lab_files/phd_data$ cat /home/cmcouto-silva/[email protected]/lab_files/phd_data/.snakemake/log/2020-09-30T234755.741940.snakemake.log
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 2
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   all
    2   bin
    3

[Wed Sep 30 23:47:55 2020]
rule bin:
    input: dataset_S1/results_bin/convert2feather.R
    output: dataset_S1/results_bin/persnp/pbs.feather, dataset_S1/results_bin/persnp/pbs_windowed.feather, dataset_S1/results_bin/persnp/ihs.feather, dataset_S1/results_bin/persnp/xpehh.feather, dataset_S1/results_bin/pergene/pbs.feather, dataset_S1/results_bin/pergene/pbs_windowed.feather, dataset_S1/results_bin/pergene/ihs.feather, dataset_S1/results_bin/pergene/xpehh.feather
    jobid: 1
    wildcards: dataset=dataset_S1
    threads: 2

Rscript dataset_S1/results_bin/convert2feather.R
[Wed Sep 30 23:48:43 2020]
Finished job 1.
1 of 3 steps (33%) done

[Wed Sep 30 23:48:43 2020]
rule bin:
    input: dataset_S2/results_bin/convert2feather.R
    output: dataset_S2/results_bin/persnp/pbs.feather, dataset_S2/results_bin/persnp/pbs_windowed.feather, dataset_S2/results_bin/persnp/ihs.feather, dataset_S2/results_bin/persnp/xpehh.feather, dataset_S2/results_bin/pergene/pbs.feather, dataset_S2/results_bin/pergene/pbs_windowed.feather, dataset_S2/results_bin/pergene/ihs.feather, dataset_S2/results_bin/pergene/xpehh.feather
    jobid: 2
    wildcards: dataset=dataset_S2
    threads: 2

Rscript dataset_S2/results_bin/convert2feather.R
[Wed Sep 30 23:49:41 2020]
Finished job 2.
2 of 3 steps (67%) done

[Wed Sep 30 23:49:41 2020]
localrule all:
    input: dataset_S1/results_bin/persnp/pbs.feather, dataset_S1/results_bin/persnp/pbs_windowed.feather, dataset_S1/results_bin/persnp/ihs.feather, dataset_S1/results_bin/persnp/xpehh.feather, dataset_S2/results_bin/persnp/pbs.feather, dataset_S2/results_bin/persnp/pbs_windowed.feather, dataset_S2/results_bin/persnp/ihs.feather, dataset_S2/results_bin/persnp/xpehh.feather, dataset_S1/results_bin/pergene/pbs.feather, dataset_S1/results_bin/pergene/pbs_windowed.feather, dataset_S1/results_bin/pergene/ihs.feather, dataset_S1/results_bin/pergene/xpehh.feather, dataset_S2/results_bin/pergene/pbs.feather, dataset_S2/results_bin/pergene/pbs_windowed.feather, dataset_S2/results_bin/pergene/ihs.feather, dataset_S2/results_bin/pergene/xpehh.feather
    jobid: 0

[Wed Sep 30 23:49:41 2020]
Finished job 0.
3 of 3 steps (100%) done
Complete log: /home/cmcouto-silva/[email protected]/lab_files/phd_data/.snakemake/log/2020-09-30T234755.741940.snakemake.log

贾福斯

我是否正确，您为此规则有两个输入文件（您的脚本，每个数据集一个），并且希望它们并行运行？如果是这样，您需要给snakemake调用两倍于您在规则中定义的核心数量。

threads规则中的字段提供了每个输入/迭代要在此规则上使用的核心数。因此，第一个数据集将使用2个核心，第二个数据集还将使用2个核心。要并行运行它们，您需要致电snakemake -j4。

我希望我能理解您的问题，如果没有，请随时纠正我。

本文收集自互联网，转载请注明来源。

如有侵权，请联系[email protected] 删除。