Mercurial > repos > shellac > sam_consensus_v3

comparison env/lib/python3.9/site-packages/cwltool-3.0.20210319143721.dist-info/METADATA @ 0:4f3585e2f14b draft default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
"planemo upload commit 60cee0fc7c0cda8592644e1aad72851dec82c959"
author shellac
date Mon, 22 Mar 2021 18:12:50 +0000
parents
children
comparison
equal deleted inserted replaced
-1:000000000000 0:4f3585e2f14b
1 Metadata-Version: 2.1
2 Name: cwltool
3 Version: 3.0.20210319143721
4 Summary: Common workflow language reference implementation
5 Home-page: https://github.com/common-workflow-language/cwltool
6 Author: Common workflow language working group
7 Author-email: common-workflow-language@googlegroups.com
8 License: UNKNOWN
9 Download-URL: https://github.com/common-workflow-language/cwltool
10 Platform: UNKNOWN
11 Classifier: Development Status :: 5 - Production/Stable
12 Classifier: Environment :: Console
13 Classifier: Intended Audience :: Developers
14 Classifier: Intended Audience :: Science/Research
15 Classifier: Intended Audience :: Healthcare Industry
16 Classifier: License :: OSI Approved :: Apache Software License
17 Classifier: Natural Language :: English
18 Classifier: Operating System :: MacOS :: MacOS X
19 Classifier: Operating System :: POSIX
20 Classifier: Operating System :: POSIX :: Linux
21 Classifier: Operating System :: OS Independent
22 Classifier: Operating System :: Microsoft :: Windows
23 Classifier: Operating System :: Microsoft :: Windows :: Windows 10
24 Classifier: Operating System :: Microsoft :: Windows :: Windows 8.1
25 Classifier: Programming Language :: Python :: 3
26 Classifier: Programming Language :: Python :: 3.6
27 Classifier: Programming Language :: Python :: 3.7
28 Classifier: Programming Language :: Python :: 3.8
29 Classifier: Programming Language :: Python :: 3.9
30 Classifier: Topic :: Scientific/Engineering
31 Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
32 Classifier: Topic :: Scientific/Engineering :: Astronomy
33 Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
34 Classifier: Topic :: Scientific/Engineering :: Information Analysis
35 Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
36 Classifier: Topic :: System :: Distributed Computing
37 Classifier: Topic :: Utilities
38 Requires-Python: >=3.6, <4
39 Description-Content-Type: text/x-rst
40 Requires-Dist: setuptools
41 Requires-Dist: requests (>=2.6.1)
42 Requires-Dist: ruamel.yaml (<=0.16.5,>=0.12.4)
43 Requires-Dist: rdflib (<5.1.0,>=4.2.2)
44 Requires-Dist: shellescape (<3.5,>=3.4.1)
45 Requires-Dist: schema-salad (<8,>=7.1)
46 Requires-Dist: mypy-extensions
47 Requires-Dist: psutil
48 Requires-Dist: prov (==1.5.1)
49 Requires-Dist: bagit (>=1.6.4)
50 Requires-Dist: typing-extensions
51 Requires-Dist: coloredlogs
52 Requires-Dist: pydot (>=1.4.1)
53 Requires-Dist: argcomplete
54 Provides-Extra: deps
55 Requires-Dist: galaxy-tool-util ; extra == 'deps'
56 Provides-Extra: docs
57 Requires-Dist: sphinx (>=2.2) ; extra == 'docs'
58 Requires-Dist: sphinx-rtd-theme ; extra == 'docs'
59 Requires-Dist: sphinx-autoapi ; extra == 'docs'
60 Requires-Dist: sphinx-autodoc-typehints ; extra == 'docs'
61 Requires-Dist: typed-ast ; (python_version < "3.8") and extra == 'docs'
62
63 ==================================================================
64 Common Workflow Language tool description reference implementation
65 ==================================================================
66
67 |Linux Status| |Debian Stable package| |Debian Testing package| |Windows Status| |Coverage Status| |Downloads|
68
69 .. |Linux Status| image:: https://github.com/common-workflow-language/cwltool/actions/workflows/ci-tests.yml/badge.svg?branch=main
70 :target: https://github.com/common-workflow-language/cwltool/actions/workflows/ci-tests.yml
71
72 .. |Debian Stable package| image:: https://badges.debian.net/badges/debian/stable/cwltool/version.svg
73 :target: https://packages.debian.org/stable/cwltool
74
75 .. |Debian Testing package| image:: https://badges.debian.net/badges/debian/testing/cwltool/version.svg
76 :target: https://packages.debian.org/testing/cwltool
77
78 .. |Windows Status| image:: https://img.shields.io/appveyor/ci/mr-c/cwltool/main.svg?label=Windows%20builds
79 :target: https://ci.appveyor.com/project/mr-c/cwltool
80
81 .. |Coverage Status| image:: https://img.shields.io/codecov/c/github/common-workflow-language/cwltool.svg
82 :target: https://codecov.io/gh/common-workflow-language/cwltool
83
84 .. |Downloads| image:: https://pepy.tech/badge/cwltool/month
85 :target: https://pepy.tech/project/cwltool
86
87 This is the reference implementation of the Common Workflow Language. It is
88 intended to be feature complete and provide comprehensive validation of CWL
89 files as well as provide other tools related to working with CWL.
90
91 This is written and tested for
92 `Python <https://www.python.org/>`_ ``3.x {x = 6, 7, 8, 9}``
93
94 The reference implementation consists of two packages. The ``cwltool`` package
95 is the primary Python module containing the reference implementation in the
96 ``cwltool`` module and console executable by the same name.
97
98 The ``cwlref-runner`` package is optional and provides an additional entry point
99 under the alias ``cwl-runner``, which is the implementation-agnostic name for the
100 default CWL interpreter installed on a host.
101
102 ``cwltool`` is provided by the CWL project, `a member project of Software Freedom Conservancy <https://sfconservancy.org/news/2018/apr/11/cwl-new-member-project/>`_
103 and our `many contributors <https://github.com/common-workflow-language/cwltool/graphs/contributors>`_.
104
105 Install
106 -------
107
108 ``cwltool`` packages
109 ^^^^^^^^^^^^^^^^^^^^
110
111 Your operating system may offer cwltool directly. For `Debian <https://tracker.debian.org/pkg/cwltool>`_, `Ubuntu <https://launchpad.net/ubuntu/+source/cwltool>`_,
112 and similar Linux distribution try
113
114 .. code:: bash
115
116 sudo apt-get install cwltool
117
118 If you are running MacOS X or other UNIXes and you want to use packages prepared by the conda-forge project, then
119 please follow the install instructions for `conda-forge <https://conda-forge.org/#about>`_ (if you haven't already) and then
120
121 .. code:: bash
122
123 conda install -c conda-forge cwltool
124
125 All of the above methods of installing ``cwltool`` use packages which might contain bugs already fixed in newer versions, or be missing features that you desire.
126 If the packaged version of ``cwltool`` available to you is too old, then we recommend installing using ``pip`` and ``venv``::
127
128 .. code:: bash
129
130 python3 -m venv env # Create a virtual environment named 'env' in the current directory
131 source env/bin/activate # Activate environment before installing `cwltool`
132
133 Then install the latest ``cwlref-runner`` package from PyPi (which will install the latest ``cwltool`` package as
134 well)
135
136 .. code:: bash
137
138 pip install cwlref-runner
139
140 If installing alongside another CWL implementation (like ``toil-cwl-runner`` or ``arvados-cwl-runner``) then instead run
141
142 .. code:: bash
143
144 pip install cwltool
145
146 MS Windows users
147 ^^^^^^^^^^^^^^^^
148
149 1. Install `"Windows Subsystem for Linux 2" (WSL2) and Docker Desktop <https://docs.docker.com/docker-for-windows/wsl/#prerequisites>`_
150 2. Install `Debian from the Microsoft Store <https://www.microsoft.com/en-us/p/debian/9msvkqc78pk6>`_
151 3. Set Debian as your default WSL 2 distro: ``wsl --set-default debian``
152 4. Reboot if you have not yet already.
153 5. Launch Debian and follow the Linux instructions above (``apt-get install cwltool`` or use the ``venv`` method)
154
155 ``cwltool`` development version
156 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
157
158 Or you can skip the direct ``pip`` commands above and install the latest development version of ``cwltool``:
159
160 .. code:: bash
161
162 git clone https://github.com/common-workflow-language/cwltool.git # clone (copy) the cwltool git repository
163 cd cwltool # Change to source directory that git clone just downloaded
164 pip install . # Installs ``cwltool`` from source
165 cwltool --version # Check if the installation works correctly
166
167 Remember, if co-installing multiple CWL implementations then you need to
168 maintain which implementation ``cwl-runner`` points to via a symbolic file
169 system link or `another facility <https://wiki.debian.org/DebianAlternatives>`_.
170
171 Recommended Software
172 ^^^^^^^^^^^^^^^^^^^^
173
174 You may also want to have the following installed:
175 - `node.js <https://nodejs.org/en/download/>`_
176 - Docker, udocker, or Singularity (optional)
177
178 Without these, some examples in the CWL tutorials at http://www.commonwl.org/user_guide/ may not work.
179
180 Run on the command line
181 -----------------------
182
183 Simple command::
184
185 cwl-runner [tool-or-workflow-description] [input-job-settings]
186
187 Or if you have multiple CWL implementations installed and you want to override
188 the default cwl-runner then use::
189
190 cwltool [tool-or-workflow-description] [input-job-settings]
191
192 You can set cwltool options in the environment with CWLTOOL_OPTIONS,
193 these will be inserted at the beginning of the command line::
194
195 export CWLTOOL_OPTIONS="--debug"
196
197 Use with boot2docker on macOS
198 -----------------------------
199 boot2docker runs Docker inside a virtual machine and it only mounts ``Users``
200 on it. The default behavior of CWL is to create temporary directories under e.g.
201 ``/Var`` which is not accessible to Docker containers.
202
203 To run CWL successfully with boot2docker you need to set the ``--tmpdir-prefix``
204 and ``--tmp-outdir-prefix`` to somewhere under ``/Users``::
205
206 $ cwl-runner --tmp-outdir-prefix=/Users/username/project --tmpdir-prefix=/Users/username/project wc-tool.cwl wc-job.json
207
208 Using uDocker
209 -------------
210
211 Some shared computing environments don't support Docker software containers for technical or policy reasons.
212 As a work around, the CWL reference runner supports using alternative ``docker`` implementations on Linux
213 with the ``--user-space-docker-cmd`` option.
214
215 One such "user space" friendly docker replacement is ``udocker`` https://github.com/indigo-dc/udocker
216
217 udocker installation: https://github.com/indigo-dc/udocker/blob/master/doc/installation_manual.md#22-install-from-udockertools-tarball
218
219 Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests:
220
221 .. code:: bash
222
223 cwltool --user-space-docker-cmd=udocker https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/main/v1.0/v1.0/test-cwl-out2.cwl https://github.com/common-workflow-language/common-workflow-language/raw/main/v1.0/v1.0/empty.json
224
225 ``cwltool`` can also use `Singularity <https://github.com/hpcng/singularity/releases/>`_ version 2.6.1
226 or later as a Docker container runtime.
227 ``cwltool`` with Singularity will run software containers specified in
228 ``DockerRequirement`` and therefore works with Docker images only, native
229 Singularity images are not supported. To use Singularity as the Docker container
230 runtime, provide ``--singularity`` command line option to ``cwltool``.
231 With Singularity, ``cwltool`` can pass all CWL v1.0 conformance tests, except
232 those involving Docker container ENTRYPOINTs.
233
234 Example:
235 .. code:: bash
236
237 cwltool --singularity https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/main/v1.0/v1.0/v1.0/cat3-tool-mediumcut.cwl https://github.com/common-workflow-language/common-workflow-language/blob/main/v1.0/v1.0/cat-job.json
238
239 Running a tool or workflow from remote or local locations
240 ---------------------------------------------------------
241
242 ``cwltool`` can run tool and workflow descriptions on both local and remote
243 systems via its support for HTTP[S] URLs.
244
245 Input job files and Workflow steps (via the `run` directive) can reference CWL
246 documents using absolute or relative local filesytem paths. If a relative path
247 is referenced and that document isn't found in the current directory then the
248 following locations will be searched:
249 http://www.commonwl.org/v1.0/CommandLineTool.html#Discovering_CWL_documents_on_a_local_filesystem
250
251 You can also use `cwldep <https://github.com/common-workflow-language/cwldep>`
252 to manage dependencies on external tools and workflows.
253
254 Overriding workflow requirements at load time
255 ---------------------------------------------
256
257 Sometimes a workflow needs additional requirements to run in a particular
258 environment or with a particular dataset. To avoid the need to modify the
259 underlying workflow, cwltool supports requirement "overrides".
260
261 The format of the "overrides" object is a mapping of item identifier (workflow,
262 workflow step, or command line tool) to the process requirements that should be applied.
263
264 .. code:: yaml
265
266 cwltool:overrides:
267 echo.cwl:
268 requirements:
269 EnvVarRequirement:
270 envDef:
271 MESSAGE: override_value
272
273 Overrides can be specified either on the command line, or as part of the job
274 input document. Workflow steps are identified using the name of the workflow
275 file followed by the step name as a document fragment identifier "#id".
276 Override identifiers are relative to the toplevel workflow document.
277
278 .. code:: bash
279
280 cwltool --overrides overrides.yml my-tool.cwl my-job.yml
281
282 .. code:: yaml
283
284 input_parameter1: value1
285 input_parameter2: value2
286 cwltool:overrides:
287 workflow.cwl#step1:
288 requirements:
289 EnvVarRequirement:
290 envDef:
291 MESSAGE: override_value
292
293 .. code:: bash
294
295 cwltool my-tool.cwl my-job-with-overrides.yml
296
297
298 Combining parts of a workflow into a single document
299 ----------------------------------------------------
300
301 Use ``--pack`` to combine a workflow made up of multiple files into a
302 single compound document. This operation takes all the CWL files
303 referenced by a workflow and builds a new CWL document with all
304 Process objects (CommandLineTool and Workflow) in a list in the
305 ``$graph`` field. Cross references (such as ``run:`` and ``source:``
306 fields) are updated to internal references within the new packed
307 document. The top level workflow is named ``#main``.
308
309 .. code:: bash
310
311 cwltool --pack my-wf.cwl > my-packed-wf.cwl
312
313
314 Running only part of a workflow
315 -------------------------------
316
317 You can run a partial workflow with the ``--target`` (``-t``) option. This
318 takes the name of an output parameter, workflow step, or input
319 parameter in the top level workflow. You may provide multiple
320 targets.
321
322 .. code:: bash
323
324 cwltool --target step3 my-wf.cwl
325
326 If a target is an output parameter, it will only run only the steps
327 that contribute to that output. If a target is a workflow step, it
328 will run the workflow starting from that step. If a target is an
329 input parameter, it will only run only the steps that are connected to
330 that input.
331
332 Use ``--print-targets`` to get a listing of the targets of a workflow.
333 To see exactly which steps will run, use ``--print-subgraph`` with
334 ``--target`` to get a printout of the workflow subgraph for the
335 selected targets.
336
337 .. code:: bash
338
339 cwltool --print-targets my-wf.cwl
340
341 cwltool --target step3 --print-subgraph my-wf.cwl > my-wf-starting-from-step3.cwl
342
343
344 Visualizing a CWL document
345 --------------------------
346
347 The ``--print-dot`` option will print a file suitable for Graphviz ``dot`` program. Here is a bash onliner to generate a Scalable Vector Graphic (SVG) file:
348
349 .. code:: bash
350
351 cwltool --print-dot my-wf.cwl | dot -Tsvg > my-wf.svg
352
353 Modeling a CWL document as RDF
354 ------------------------------
355
356 CWL documents can be expressed as RDF triple graphs.
357
358 .. code:: bash
359
360 cwltool --print-rdf --rdf-serializer=turtle mywf.cwl
361
362
363 Leveraging SoftwareRequirements (Beta)
364 --------------------------------------
365
366 CWL tools may be decorated with ``SoftwareRequirement`` hints that cwltool
367 may in turn use to resolve to packages in various package managers or
368 dependency management systems such as `Environment Modules
369 <http://modules.sourceforge.net/>`__.
370
371 Utilizing ``SoftwareRequirement`` hints using cwltool requires an optional
372 dependency, for this reason be sure to use specify the ``deps`` modifier when
373 installing cwltool. For instance::
374
375 $ pip install 'cwltool[deps]'
376
377 Installing cwltool in this fashion enables several new command line options.
378 The most general of these options is ``--beta-dependency-resolvers-configuration``.
379 This option allows one to specify a dependency resolver's configuration file.
380 This file may be specified as either XML or YAML and very simply describes various
381 plugins to enable to "resolve" ``SoftwareRequirement`` dependencies.
382
383 To discuss some of these plugins and how to configure them, first consider the
384 following ``hint`` definition for an example CWL tool.
385
386 .. code:: yaml
387
388 SoftwareRequirement:
389 packages:
390 - package: seqtk
391 version:
392 - r93
393
394 Now imagine deploying cwltool on a cluster with Software Modules installed
395 and that a ``seqtk`` module is available at version ``r93``. This means cluster
396 users likely won't have the binary ``seqtk`` on their ``PATH`` by default, but after
397 sourcing this module with the command ``modulecmd sh load seqtk/r93`` ``seqtk`` is
398 available on the ``PATH``. A simple dependency resolvers configuration file, called
399 ``dependency-resolvers-conf.yml`` for instance, that would enable cwltool to source
400 the correct module environment before executing the above tool would simply be:
401
402 .. code:: yaml
403
404 - type: modules
405
406 The outer list indicates that one plugin is being enabled, the plugin parameters are
407 defined as a dictionary for this one list item. There is only one required parameter
408 for the plugin above, this is ``type`` and defines the plugin type. This parameter
409 is required for all plugins. The available plugins and the parameters
410 available for each are documented (incompletely) `here
411 <https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__.
412 Unfortunately, this documentation is in the context of Galaxy tool
413 ``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly.
414
415 cwltool is distributed with an example of such seqtk tool and sample corresponding
416 job. It could executed from the cwltool root using a dependency resolvers
417 configuration file such as the above one using the command::
418
419 cwltool --beta-dependency-resolvers-configuration /path/to/dependency-resolvers-conf.yml \
420 tests/seqtk_seq.cwl \
421 tests/seqtk_seq_job.json
422
423 This example demonstrates both that cwltool can leverage
424 existing software installations and also handle workflows with dependencies
425 on different versions of the same software and libraries. However the above
426 example does require an existing module setup so it is impossible to test this example
427 "out of the box" with cwltool. For a more isolated test that demonstrates all
428 the same concepts - the resolver plugin type ``galaxy_packages`` can be used.
429
430 "Galaxy packages" are a lighter weight alternative to Environment Modules that are
431 really just defined by a way to lay out directories into packages and versions
432 to find little scripts that are sourced to modify the environment. They have
433 been used for years in Galaxy community to adapt Galaxy tools to cluster
434 environments but require neither knowledge of Galaxy nor any special tools to
435 setup. These should work just fine for CWL tools.
436
437 The cwltool source code repository's test directory is setup with a very simple
438 directory that defines a set of "Galaxy packages" (but really just defines one
439 package named ``random-lines``). The directory layout is simply::
440
441 tests/test_deps_env/
442 random-lines/
443 1.0/
444 env.sh
445
446 If the ``galaxy_packages`` plugin is enabled and pointed at the
447 ``tests/test_deps_env`` directory in cwltool's root and a ``SoftwareRequirement``
448 such as the following is encountered.
449
450 .. code:: yaml
451
452 hints:
453 SoftwareRequirement:
454 packages:
455 - package: 'random-lines'
456 version:
457 - '1.0'
458
459 Then cwltool will simply find that ``env.sh`` file and source it before executing
460 the corresponding tool. That ``env.sh`` script is only responsible for modifying
461 the job's ``PATH`` to add the required binaries.
462
463 This is a full example that works since resolving "Galaxy packages" has no
464 external requirements. Try it out by executing the following command from cwltool's
465 root directory::
466
467 cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf.yml \
468 tests/random_lines.cwl \
469 tests/random_lines_job.json
470
471 The resolvers configuration file in the above example was simply:
472
473 .. code:: yaml
474
475 - type: galaxy_packages
476 base_path: ./tests/test_deps_env
477
478 It is possible that the ``SoftwareRequirement`` s in a given CWL tool will not
479 match the module names for a given cluster. Such requirements can be re-mapped
480 to specific deployed packages and/or versions using another file specified using
481 the resolver plugin parameter `mapping_files`. We will
482 demonstrate this using `galaxy_packages` but the concepts apply equally well
483 to Environment Modules or Conda packages (described below) for instance.
484
485 So consider the resolvers configuration file
486 (`tests/test_deps_env_resolvers_conf_rewrite.yml`):
487
488 .. code:: yaml
489
490 - type: galaxy_packages
491 base_path: ./tests/test_deps_env
492 mapping_files: ./tests/test_deps_mapping.yml
493
494 And the corresponding mapping configuraiton file (`tests/test_deps_mapping.yml`):
495
496 .. code:: yaml
497
498 - from:
499 name: randomLines
500 version: 1.0.0-rc1
501 to:
502 name: random-lines
503 version: '1.0'
504
505 This is saying if cwltool encounters a requirement of ``randomLines`` at version
506 ``1.0.0-rc1`` in a tool, to rewrite to our specific plugin as ``random-lines`` at
507 version ``1.0``. cwltool has such a test tool called ``random_lines_mapping.cwl``
508 that contains such a source ``SoftwareRequirement``. To try out this example with
509 mapping, execute the following command from the cwltool root directory::
510
511 cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf_rewrite.yml \
512 tests/random_lines_mapping.cwl \
513 tests/random_lines_job.json
514
515 The previous examples demonstrated leveraging existing infrastructure to
516 provide requirements for CWL tools. If instead a real package manager is used
517 cwltool has the opportunity to install requirements as needed. While initial
518 support for Homebrew/Linuxbrew plugins is available, the most developed such
519 plugin is for the `Conda <https://conda.io/docs/#>`__ package manager. Conda has the nice properties
520 of allowing multiple versions of a package to be installed simultaneously,
521 not requiring evaluated permissions to install Conda itself or packages using
522 Conda, and being cross platform. For these reasons, cwltool may run as a normal
523 user, install its own Conda environment and manage multiple versions of Conda packages
524 on both Linux and Mac OS X.
525
526 The Conda plugin can be endlessly configured, but a sensible set of defaults
527 that has proven a powerful stack for dependency management within the Galaxy tool
528 development ecosystem can be enabled by simply passing cwltool the
529 ``--beta-conda-dependencies`` flag.
530
531 With this we can use the seqtk example above without Docker and without
532 any externally managed services - cwltool should install everything it needs
533 and create an environment for the tool. Try it out with the follwing command::
534
535 cwltool --beta-conda-dependencies tests/seqtk_seq.cwl tests/seqtk_seq_job.json
536
537 The CWL specification allows URIs to be attached to ``SoftwareRequirement`` s
538 that allow disambiguation of package names. If the mapping files described above
539 allow deployers to adapt tools to their infrastructure, this mechanism allows
540 tools to adapt their requirements to multiple package managers. To demonstrate
541 this within the context of the seqtk, we can simply break the package name we
542 use and then specify a specific Conda package as follows:
543
544 .. code:: yaml
545
546 hints:
547 SoftwareRequirement:
548 packages:
549 - package: seqtk_seq
550 version:
551 - '1.2'
552 specs:
553 - https://anaconda.org/bioconda/seqtk
554 - https://packages.debian.org/sid/seqtk
555
556 The example can be executed using the command::
557
558 cwltool --beta-conda-dependencies tests/seqtk_seq_wrong_name.cwl tests/seqtk_seq_job.json
559
560 The plugin framework for managing resolution of these software requirements
561 as maintained as part of `galaxy-tool-util <https://github.com/galaxyproject/galaxy/tree/dev/packages/tool_util>`__ - a small,
562 portable subset of the Galaxy project. More information on configuration and implementation can be found
563 at the following links:
564
565 - `Dependency Resolvers in Galaxy <https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__
566 - `Conda for [Galaxy] Tool Dependencies <https://docs.galaxyproject.org/en/latest/admin/conda_faq.html>`__
567 - `Mapping Files - Implementation <https://github.com/galaxyproject/galaxy/commit/495802d229967771df5b64a2f79b88a0eaf00edb>`__
568 - `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__
569 - `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__
570
571 Use with GA4GH Tool Registry API
572 --------------------------------
573
574 Cwltool can launch tools directly from `GA4GH Tool Registry API`_ endpoints.
575
576 By default, cwltool searches https://dockstore.org/ . Use ``--add-tool-registry`` to add other registries to the search path.
577
578 For example ::
579
580 cwltool quay.io/collaboratory/dockstore-tool-bamstats:develop test.json
581
582 and (defaults to latest when a version is not specified) ::
583
584 cwltool quay.io/collaboratory/dockstore-tool-bamstats test.json
585
586 For this example, grab the test.json (and input file) from https://github.com/CancerCollaboratory/dockstore-tool-bamstats ::
587
588 wget https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fbriandoconnor%2Fdockstore-tool-bamstats/versions/develop/PLAIN-CWL/descriptor/test.json
589 wget https://github.com/CancerCollaboratory/dockstore-tool-bamstats/raw/develop/rna.SRR948778.bam
590
591
592 .. _`GA4GH Tool Registry API`: https://github.com/ga4gh/tool-registry-schemas
593
594 Running MPI-based tools that need to be launched
595 ------------------------------------------------
596
597 Cwltool supports an extension to the CWL spec
598 ``http://commonwl.org/cwltool#MPIRequirement``. When the tool
599 definition has this in its ``requirements``/``hints`` section, and
600 cwltool has been run with ``--enable-ext``, then the tool's command
601 line will be extended with the commands needed to launch it with
602 ``mpirun`` or similar. You can specify the number of processes to
603 start as either a literal integer or an expression (that will result
604 in an integer). For example::
605
606 #!/usr/bin/env cwl-runner
607 cwlVersion: v1.1
608 class: CommandLineTool
609 $namespaces:
610 cwltool: "http://commonwl.org/cwltool#"
611 requirements:
612 cwltool:MPIRequirement:
613 processes: $(inputs.nproc)
614 inputs:
615 nproc:
616 type: int
617
618 Interaction with containers: the MPIRequirement currently prepends its
619 commands to the front of the command line that is constructed. If you
620 wish to run a containerised application in parallel, for simple use
621 cases this does work with Singularity, depending upon the platform
622 setup. However this combination should be considered "alpha" -- please
623 do report any issues you have! This does not work with Docker at the
624 moment. (More precisely, you get `n` copies of the same single process
625 image run at the same time that cannot communicate with each other.)
626
627 The host-specific parameters are configured in a simple YAML file
628 (specified with the ``--mpi-config-file`` flag). The allowed keys are
629 given in the following table; all are optional.
630
631 +----------------+------------------+----------+------------------------------+
632 | Key | Type | Default | Description |
633 +================+==================+==========+==============================+
634 | runner | str | "mpirun" | The primary command to use. |
635 +----------------+------------------+----------+------------------------------+
636 | nproc_flag | str | "-n" | Flag to set number of |
637 | | | | processes to start. |
638 +----------------+------------------+----------+------------------------------+
639 | default_nproc | int | 1 | Default number of processes. |
640 +----------------+------------------+----------+------------------------------+
641 | extra_flags | List[str] | [] | A list of any other flags to |
642 | | | | be added to the runner's |
643 | | | | command line before |
644 | | | | the ``baseCommand``. |
645 +----------------+------------------+----------+------------------------------+
646 | env_pass | List[str] | [] | A list of environment |
647 | | | | variables that should be |
648 | | | | passed from the host |
649 | | | | environment through to the |
650 | | | | tool (e.g. giving the |
651 | | | | nodelist as set by your |
652 | | | | scheduler). |
653 +----------------+------------------+----------+------------------------------+
654 | env_pass_regex | List[str] | [] | A list of python regular |
655 | | | | expressions that will be |
656 | | | | matched against the host's |
657 | | | | environment. Those that match|
658 | | | | will be passed through. |
659 +----------------+------------------+----------+------------------------------+
660 | env_set | Mapping[str,str] | {} | A dictionary whose keys are |
661 | | | | the environment variables set|
662 | | | | and the values being the |
663 | | | | values. |
664 +----------------+------------------+----------+------------------------------+
665
666
667
668 ===========
669 Development
670 ===========
671
672 Running tests locally
673 ---------------------
674
675 - Running basic tests ``(/tests)``:
676
677 To run the basic tests after installing `cwltool` execute the following:
678
679 .. code:: bash
680
681 pip install -rtest-requirements.txt
682 py.test --ignore cwltool/schemas/ --pyarg cwltool
683
684 To run various tests in all supported Python environments we use `tox <https://github.com/common-workflow-language/cwltool/tree/main/tox.ini>`_. To run the test suite in all supported Python environments
685 first downloading the complete code repository (see the ``git clone`` instructions above) and then run
686 the following in the terminal:
687 ``pip install tox; tox``
688
689 List of all environment can be seen using:
690 ``tox --listenvs``
691 and running a specfic test env using:
692 ``tox -e <env name>``
693 and additionally run a specific test using this format:
694 ``tox -e py36-unit -- tests/test_examples.py::TestParamMatching``
695
696 - Running the entire suite of CWL conformance tests:
697
698 The GitHub repository for the CWL specifications contains a script that tests a CWL
699 implementation against a wide array of valid CWL files using the `cwltest <https://github.com/common-workflow-language/cwltest>`_
700 program
701
702 Instructions for running these tests can be found in the Common Workflow Language Specification repository at https://github.com/common-workflow-language/common-workflow-language/blob/main/CONFORMANCE_TESTS.md
703
704 Import as a module
705 ------------------
706
707 Add
708
709 .. code:: python
710
711 import cwltool
712
713 to your script.
714
715 The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory
716
717 .. code:: python
718
719 import cwltool.factory
720 fac = cwltool.factory.Factory()
721
722 echo = fac.make("echo.cwl")
723 result = echo(inp="foo")
724
725 # result["out"] == "foo"
726
727
728 CWL Tool Control Flow
729 ---------------------
730
731 Technical outline of how cwltool works internally, for maintainers.
732
733 #. Use CWL ``load_tool()`` to load document.
734
735 #. Fetches the document from file or URL
736 #. Applies preprocessing (syntax/identifier expansion and normalization)
737 #. Validates the document based on cwlVersion
738 #. If necessary, updates the document to latest spec
739 #. Constructs a Process object using ``make_tool()``` callback. This yields a
740 CommandLineTool, Workflow, or ExpressionTool. For workflows, this
741 recursively constructs each workflow step.
742 #. To construct custom types for CommandLineTool, Workflow, or
743 ExpressionTool, provide a custom ``make_tool()``
744
745 #. Iterate on the ``job()`` method of the Process object to get back runnable jobs.
746
747 #. ``job()`` is a generator method (uses the Python iterator protocol)
748 #. Each time the ``job()`` method is invoked in an iteration, it returns one
749 of: a runnable item (an object with a ``run()`` method), ``None`` (indicating
750 there is currently no work ready to run) or end of iteration (indicating
751 the process is complete.)
752 #. Invoke the runnable item by calling ``run()``. This runs the tool and gets output.
753 #. Output of a process is reported by an output callback.
754 #. ``job()`` may be iterated over multiple times. It will yield all the work
755 that is currently ready to run and then yield None.
756
757 #. ``Workflow`` objects create a corresponding ``WorkflowJob`` and ``WorkflowJobStep`` objects to hold the workflow state for the duration of the job invocation.
758
759 #. The WorkflowJob iterates over each WorkflowJobStep and determines if the
760 inputs the step are ready.
761 #. When a step is ready, it constructs an input object for that step and
762 iterates on the ``job()`` method of the workflow job step.
763 #. Each runnable item is yielded back up to top level run loop
764 #. When a step job completes and receives an output callback, the
765 job outputs are assigned to the output of the workflow step.
766 #. When all steps are complete, the intermediate files are moved to a final
767 workflow output, intermediate directories are deleted, and the output
768 callback for the workflow is called.
769
770 #. ``CommandLineTool`` job() objects yield a single runnable object.
771
772 #. The CommandLineTool ``job()`` method calls ``make_job_runner()`` to create a
773 ``CommandLineJob`` object
774 #. The job method configures the CommandLineJob object by setting public
775 attributes
776 #. The job method iterates over file and directories inputs to the
777 CommandLineTool and creates a "path map".
778 #. Files are mapped from their "resolved" location to a "target" path where
779 they will appear at tool invocation (for example, a location inside a
780 Docker container.) The target paths are used on the command line.
781 #. Files are staged to targets paths using either Docker volume binds (when
782 using containers) or symlinks (if not). This staging step enables files
783 to be logically rearranged or renamed independent of their source layout.
784 #. The ``run()`` method of CommandLineJob executes the command line tool or
785 Docker container, waits for it to complete, collects output, and makes
786 the output callback.
787
788
789 Extension points
790 ----------------
791
792 The following functions can be passed to main() to override or augment
793 the listed behaviors.
794
795 executor
796 ::
797
798 executor(tool, job_order_object, runtimeContext, logger)
799 (Process, Dict[Text, Any], RuntimeContext) -> Tuple[Dict[Text, Any], Text]
800
801 An implementation of the toplevel workflow execution loop, should
802 synchronously run a process object to completion and return the
803 output object.
804
805 versionfunc
806 ::
807
808 ()
809 () -> Text
810
811 Return version string.
812
813 logger_handler
814 ::
815
816 logger_handler
817 logging.Handler
818
819 Handler object for logging.
820
821 The following functions can be set in LoadingContext to override or
822 augment the listed behaviors.
823
824 fetcher_constructor
825 ::
826
827 fetcher_constructor(cache, session)
828 (Dict[unicode, unicode], requests.sessions.Session) -> Fetcher
829
830 Construct a Fetcher object with the supplied cache and HTTP session.
831
832 resolver
833 ::
834
835 resolver(document_loader, document)
836 (Loader, Union[Text, dict[Text, Any]]) -> Text
837
838 Resolve a relative document identifier to an absolute one which can be fetched.
839
840 The following functions can be set in RuntimeContext to override or
841 augment the listed behaviors.
842
843 construct_tool_object
844 ::
845
846 construct_tool_object(toolpath_object, loadingContext)
847 (MutableMapping[Text, Any], LoadingContext) -> Process
848
849 Hook to construct a Process object (eg CommandLineTool) object from a document.
850
851 select_resources
852 ::
853
854 selectResources(request)
855 (Dict[str, int], RuntimeContext) -> Dict[Text, int]
856
857 Take a resource request and turn it into a concrete resource assignment.
858
859 make_fs_access
860 ::
861
862 make_fs_access(basedir)
863 (Text) -> StdFsAccess
864
865 Return a file system access object.
866
867 In addition, when providing custom subclasses of Process objects, you can override the following methods:
868
869 CommandLineTool.make_job_runner
870 ::
871
872 make_job_runner(RuntimeContext)
873 (RuntimeContext) -> Type[JobBase]
874
875 Create and return a job runner object (this implements concrete execution of a command line tool).
876
877 Workflow.make_workflow_step
878 ::
879
880 make_workflow_step(toolpath_object, pos, loadingContext, parentworkflowProv)
881 (Dict[Text, Any], int, LoadingContext, Optional[ProvenanceProfile]) -> WorkflowStep
882
883 Create and return a workflow step object.
884
885