Mercurial > repos > shellac > guppy_basecaller

comparison env/lib/python3.7/site-packages/cwltool-1.0.20191225192155.dist-info/METADATA @ 5:9b1c78e6ba9c draft default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
"planemo upload commit 6c0a8142489327ece472c84e558c47da711a9142"
author shellac
date Mon, 01 Jun 2020 08:59:25 -0400
parents 79f47841a781
children
comparison
equal deleted inserted replaced
4:79f47841a781 5:9b1c78e6ba9c
1 Metadata-Version: 2.1
2 Name: cwltool
3 Version: 1.0.20191225192155
4 Summary: Common workflow language reference implementation
5 Home-page: https://github.com/common-workflow-language/cwltool
6 Author: Common workflow language working group
7 Author-email: common-workflow-language@googlegroups.com
8 License: UNKNOWN
9 Download-URL: https://github.com/common-workflow-language/cwltool
10 Platform: UNKNOWN
11 Classifier: Development Status :: 5 - Production/Stable
12 Classifier: Environment :: Console
13 Classifier: Intended Audience :: Developers
14 Classifier: Intended Audience :: Science/Research
15 Classifier: Intended Audience :: Healthcare Industry
16 Classifier: License :: OSI Approved :: Apache Software License
17 Classifier: Natural Language :: English
18 Classifier: Operating System :: MacOS :: MacOS X
19 Classifier: Operating System :: POSIX
20 Classifier: Operating System :: POSIX :: Linux
21 Classifier: Operating System :: OS Independent
22 Classifier: Operating System :: Microsoft :: Windows
23 Classifier: Operating System :: Microsoft :: Windows :: Windows 10
24 Classifier: Operating System :: Microsoft :: Windows :: Windows 8.1
25 Classifier: Programming Language :: Python :: 2
26 Classifier: Programming Language :: Python :: 2.7
27 Classifier: Programming Language :: Python :: 3
28 Classifier: Programming Language :: Python :: 3.5
29 Classifier: Programming Language :: Python :: 3.6
30 Classifier: Programming Language :: Python :: 3.7
31 Classifier: Programming Language :: Python :: 3.8
32 Classifier: Topic :: Scientific/Engineering
33 Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
34 Classifier: Topic :: Scientific/Engineering :: Astronomy
35 Classifier: Topic :: Scientific/Engineering :: Atmospheric Science
36 Classifier: Topic :: Scientific/Engineering :: Information Analysis
37 Classifier: Topic :: Scientific/Engineering :: Medical Science Apps.
38 Classifier: Topic :: System :: Distributed Computing
39 Classifier: Topic :: Utilities
40 Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4
41 Description-Content-Type: text/x-rst
42 Requires-Dist: setuptools
43 Requires-Dist: requests (>=2.6.1)
44 Requires-Dist: ruamel.yaml (<=0.16,>=0.12.4)
45 Requires-Dist: rdflib (<4.3.0,>=4.2.2)
46 Requires-Dist: shellescape (<3.5,>=3.4.1)
47 Requires-Dist: schema-salad (<5,>=4.5)
48 Requires-Dist: mypy-extensions
49 Requires-Dist: six (>=1.9.0)
50 Requires-Dist: psutil
51 Requires-Dist: scandir
52 Requires-Dist: prov (==1.5.1)
53 Requires-Dist: bagit (>=1.6.4)
54 Requires-Dist: typing-extensions
55 Requires-Dist: coloredlogs
56 Requires-Dist: future (>=0.16)
57 Requires-Dist: pathlib2 (!=2.3.1)
58 Requires-Dist: subprocess32 (>=3.5.0) ; os.name=="posix" and python_version<"3.5"
59 Requires-Dist: typing (>=3.5.3) ; python_version<"3.6"
60 Provides-Extra: deps
61 Requires-Dist: galaxy-tool-util ; extra == 'deps'
62 Provides-Extra: docs
63 Requires-Dist: sphinx (>=2.2) ; extra == 'docs'
64 Requires-Dist: sphinx-rtd-theme ; extra == 'docs'
65
66 ==================================================================
67 Common Workflow Language tool description reference implementation
68 ==================================================================
69
70 CWL conformance tests: |Conformance Status| |Linux Status| |Windows Status| |Coverage Status| |Downloads|
71
72 |CommandLineTool Support| |DockerRequirement Support| |EnvVarRequirement Support| |ExpressionTool Support|
73 |InitialWorkDirRequirement Support| |InlineJavascriptRequirement Support| |MultipleInputRequirement Support| |Core Support|
74 |ResourceRequirement Support| |ScatterRequirement Support| |SchemaDefRequirement Support| |ShellCommandequirement Support|
75 |StepInputRequirement Support| |SubWorkflowRequirement Support| |Workflow Support|
76
77 .. |Conformance Status| image:: https://ci.commonwl.org/buildStatus/icon?job=cwltool-conformance
78 :target: https://ci.commonwl.org/job/cwltool-conformance/
79
80 .. |Linux Status| image:: https://img.shields.io/travis/common-workflow-language/cwltool/master.svg?label=Linux%20builds
81 :target: https://travis-ci.org/common-workflow-language/cwltool
82
83 .. |Windows Status| image:: https://img.shields.io/appveyor/ci/mr-c/cwltool/master.svg?label=Windows%20builds
84 :target: https://ci.appveyor.com/project/mr-c/cwltool
85
86 .. |Coverage Status| image:: https://img.shields.io/codecov/c/github/common-workflow-language/cwltool.svg
87 :target: https://codecov.io/gh/common-workflow-language/cwltool
88
89 .. |Downloads| image:: https://pepy.tech/badge/cwltool/month
90 :target: https://pepy.tech/project/cwltool
91
92 .. |CommandLineTool Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/command_line_tool.json
93 :target: https://ci.commonwl.org/job/cwltool-conformance/
94
95 .. |DockerRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/docker.json
96 :target: https://ci.commonwl.org/job/cwltool-conformance/
97
98 .. |EnvVarRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/env_var.json
99 :target: https://ci.commonwl.org/job/cwltool-conformance/
100
101 .. |ExpressionTool Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/expression_tool.json
102 :target: https://ci.commonwl.org/job/cwltool-conformance/
103
104 .. |InitialWorkDirRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/initial_work_dir.json
105 :target: https://ci.commonwl.org/job/cwltool-conformance/
106
107 .. |InlineJavascriptRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/inline_javascript.json
108 :target: https://ci.commonwl.org/job/cwltool-conformance/
109
110 .. |MultipleInputRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/multiple_input.json
111 :target: https://ci.commonwl.org/job/cwltool-conformance/
112
113 .. |Core Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/required.json
114 :target: https://ci.commonwl.org/job/cwltool-conformance/
115
116 .. |ResourceRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/resource.json
117 :target: https://ci.commonwl.org/job/cwltool-conformance/
118
119 .. |ScatterRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/scatter.json
120 :target: https://ci.commonwl.org/job/cwltool-conformance/
121
122 .. |SchemaDefRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/schema_def.json
123 :target: https://ci.commonwl.org/job/cwltool-conformance/
124
125 .. |ShellCommandequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/shell_command.json
126 :target: https://ci.commonwl.org/job/cwltool-conformance/
127
128 .. |StepInputRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/step_input.json
129 :target: https://ci.commonwl.org/job/cwltool-conformance/
130
131 .. |SubWorkflowRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/subworkflow.json
132 :target: https://ci.commonwl.org/job/cwltool-conformance/
133
134 .. |Workflow Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/workflow.json
135 :target: https://ci.commonwl.org/job/cwltool-conformance/
136
137
138 This is the reference implementation of the Common Workflow Language. It is
139 intended to be feature complete and provide comprehensive validation of CWL
140 files as well as provide other tools related to working with CWL.
141
142 This is written and tested for
143 `Python <https://www.python.org/>`_ ``2.7 and 3.x {x = 5, 6, 7, 8}``
144
145 The reference implementation consists of two packages. The ``cwltool`` package
146 is the primary Python module containing the reference implementation in the
147 ``cwltool`` module and console executable by the same name.
148
149 The ``cwlref-runner`` package is optional and provides an additional entry point
150 under the alias ``cwl-runner``, which is the implementation-agnostic name for the
151 default CWL interpreter installed on a host.
152
153 Install
154 -------
155
156 Your operating system may offer cwltool directly. For [Debian](https://tracker.debian.org/pkg/cwltool "Debian cwltool package tracker") or [Ubuntu](https://launchpad.net/ubuntu/+source/cwltool "Ubuntu Launchpad overview for cwltool") try
157
158 .. code:: bash
159
160 apt-get install cwltool
161
162 Otherwise, to
163 avoid conflicting versions of the same library,
164 it is recommended to do the following:
165
166 .. code:: bash
167
168 virtualenv -p python2 venv # Create a virtual environment, can use `python3` as well
169 source venv/bin/activate # Activate environment before installing `cwltool`
170
171 Installing the official package from PyPi (will install "cwltool" package as
172 well)
173
174 .. code:: bash
175
176 pip install cwlref-runner
177
178 If installing alongside another CWL implementation then
179
180 .. code:: bash
181
182 pip install cwltool
183
184 Or you can install from source:
185
186 .. code:: bash
187
188 git clone https://github.com/common-workflow-language/cwltool.git # clone cwltool repo
189 cd cwltool # Switch to source directory
190 pip install . # Install `cwltool` from source
191 cwltool --version # Check if the installation works correctly
192
193 Remember, if co-installing multiple CWL implementations then you need to
194 maintain which implementation ``cwl-runner`` points to via a symbolic file
195 system link or `another facility <https://wiki.debian.org/DebianAlternatives>`_.
196
197 =====
198 Usage
199 =====
200
201 Run on the command line
202 -----------------------
203
204 Simple command::
205
206 cwl-runner [tool-or-workflow-description] [input-job-settings]
207
208 Or if you have multiple CWL implementations installed and you want to override
209 the default cwl-runner use::
210
211 cwltool [tool-or-workflow-description] [input-job-settings]
212
213 You can set cwltool options in the environment with CWLTOOL_OPTIONS,
214 these will be inserted at the beginning of the command line::
215
216 export CWLTOOL_OPTIONS="--debug"
217
218 Use with boot2docker
219 --------------------
220 boot2docker runs Docker inside a virtual machine and it only mounts ``Users``
221 on it. The default behavior of CWL is to create temporary directories under e.g.
222 ``/Var`` which is not accessible to Docker containers.
223
224 To run CWL successfully with boot2docker you need to set the ``--tmpdir-prefix``
225 and ``--tmp-outdir-prefix`` to somewhere under ``/Users``::
226
227 $ cwl-runner --tmp-outdir-prefix=/Users/username/project --tmpdir-prefix=/Users/username/project wc-tool.cwl wc-job.json
228
229 Using user-space replacements for Docker
230 ----------------------------------------
231
232 Some shared computing environments don't support Docker software containers for technical or policy reasons.
233 As a work around, the CWL reference runner supports using alternative ``docker`` implementations on Linux
234 with the ``--user-space-docker-cmd`` option.
235
236 One such "user space" friendly docker replacement is ``udocker`` https://github.com/indigo-dc/udocker and another
237 is ``dx-docker`` https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images
238
239 udocker installation: https://github.com/indigo-dc/udocker/blob/master/doc/installation_manual.md#22-install-from-indigo-datacloud-repositories
240
241 dx-docker installation: start with the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions).
242
243 Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests:
244
245 .. code:: bash
246
247 cwltool --user-space-docker-cmd=udocker https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/test-cwl-out2.cwl https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/empty.json
248
249 or
250
251 .. code:: bash
252
253 cwltool --user-space-docker-cmd=dx-docker https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/test-cwl-out2.cwl https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/empty.json
254
255 ``cwltool`` can use `Singularity <http://singularity.lbl.gov/>`_ version 2.6.1
256 or later as a Docker container runtime.
257 ``cwltool`` with Singularity will run software containers specified in
258 ``DockerRequirement`` and therefore works with Docker images only, native
259 Singularity images are not supported. To use Singularity as the Docker container
260 runtime, provide ``--singularity`` command line option to ``cwltool``.
261 With Singularity, ``cwltool`` can pass all CWL v1.0 conformance tests, except
262 those involving Docker container ENTRYPOINTs.
263
264
265 .. code:: bash
266
267 cwltool --singularity https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/v1.0/cat3-tool-mediumcut.cwl https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/cat-job.json
268
269 Running a tool or workflow from remote or local locations
270 ---------------------------------------------------------
271
272 ``cwltool`` can run tool and workflow descriptions on both local and remote
273 systems via its support for HTTP[S] URLs.
274
275 Input job files and Workflow steps (via the `run` directive) can reference CWL
276 documents using absolute or relative local filesytem paths. If a relative path
277 is referenced and that document isn't found in the current directory then the
278 following locations will be searched:
279 http://www.commonwl.org/v1.0/CommandLineTool.html#Discovering_CWL_documents_on_a_local_filesystem
280
281 You can also use `cwldep <https://github.com/common-workflow-language/cwldep>`
282 to manage dependencies on external tools and workflows.
283
284 Overriding workflow requirements at load time
285 ---------------------------------------------
286
287 Sometimes a workflow needs additional requirements to run in a particular
288 environment or with a particular dataset. To avoid the need to modify the
289 underlying workflow, cwltool supports requirement "overrides".
290
291 The format of the "overrides" object is a mapping of item identifier (workflow,
292 workflow step, or command line tool) to the process requirements that should be applied.
293
294 .. code:: yaml
295
296 cwltool:overrides:
297 echo.cwl:
298 requirements:
299 EnvVarRequirement:
300 envDef:
301 MESSAGE: override_value
302
303 Overrides can be specified either on the command line, or as part of the job
304 input document. Workflow steps are identified using the name of the workflow
305 file followed by the step name as a document fragment identifier "#id".
306 Override identifiers are relative to the toplevel workflow document.
307
308 .. code:: bash
309
310 cwltool --overrides overrides.yml my-tool.cwl my-job.yml
311
312 .. code:: yaml
313
314 input_parameter1: value1
315 input_parameter2: value2
316 cwltool:overrides:
317 workflow.cwl#step1:
318 requirements:
319 EnvVarRequirement:
320 envDef:
321 MESSAGE: override_value
322
323 .. code:: bash
324
325 cwltool my-tool.cwl my-job-with-overrides.yml
326
327
328 Combining parts of a workflow into a single document
329 ----------------------------------------------------
330
331 Use ``--pack`` to combine a workflow made up of multiple files into a
332 single compound document. This operation takes all the CWL files
333 referenced by a workflow and builds a new CWL document with all
334 Process objects (CommandLineTool and Workflow) in a list in the
335 ``$graph`` field. Cross references (such as ``run:`` and ``source:``
336 fields) are updated to internal references within the new packed
337 document. The top level workflow is named ``#main``.
338
339 .. code:: bash
340
341 cwltool --pack my-wf.cwl > my-packed-wf.cwl
342
343
344 Running only part of a workflow
345 -------------------------------
346
347 You can run a partial workflow with the ``--target`` (``-t``) option. This
348 takes the name of an output parameter, workflow step, or input
349 parameter in the top level workflow. You may provide multiple
350 targets.
351
352 .. code:: bash
353
354 cwltool --target step3 my-wf.cwl
355
356 If a target is an output parameter, it will only run only the steps
357 that contribute to that output. If a target is a workflow step, it
358 will run the workflow starting from that step. If a target is an
359 input parameter, it will only run only the steps that are connected to
360 that input.
361
362 Use ``--print-targets`` to get a listing of the targets of a workflow.
363 To see exactly which steps will run, use ``--print-subgraph`` with
364 ``--target`` to get a printout of the workflow subgraph for the
365 selected targets.
366
367 .. code:: bash
368
369 cwltool --print-targets my-wf.cwl
370
371 cwltool --target step3 --print-subgraph my-wf.cwl > my-wf-starting-from-step3.cwl
372
373
374 Visualizing a CWL document
375 --------------------------
376
377 The ``--print-dot`` option will print a file suitable for Graphviz ``dot`` program. Here is a bash onliner to generate a Scalable Vector Graphic (SVG) file:
378
379 .. code:: bash
380
381 cwltool --print-dot my-wf.cwl | dot -Tsvg > my-wf.svg
382
383 Modeling a CWL document as RDF
384 ------------------------------
385
386 CWL documents can be expressed as RDF triple graphs.
387
388 .. code:: bash
389
390 cwltool --print-rdf --rdf-serializer=turtle mywf.cwl
391
392
393 Leveraging SoftwareRequirements (Beta)
394 --------------------------------------
395
396 CWL tools may be decorated with ``SoftwareRequirement`` hints that cwltool
397 may in turn use to resolve to packages in various package managers or
398 dependency management systems such as `Environment Modules
399 <http://modules.sourceforge.net/>`__.
400
401 Utilizing ``SoftwareRequirement`` hints using cwltool requires an optional
402 dependency, for this reason be sure to use specify the ``deps`` modifier when
403 installing cwltool. For instance::
404
405 $ pip install 'cwltool[deps]'
406
407 Installing cwltool in this fashion enables several new command line options.
408 The most general of these options is ``--beta-dependency-resolvers-configuration``.
409 This option allows one to specify a dependency resolver's configuration file.
410 This file may be specified as either XML or YAML and very simply describes various
411 plugins to enable to "resolve" ``SoftwareRequirement`` dependencies.
412
413 To discuss some of these plugins and how to configure them, first consider the
414 following ``hint`` definition for an example CWL tool.
415
416 .. code:: yaml
417
418 SoftwareRequirement:
419 packages:
420 - package: seqtk
421 version:
422 - r93
423
424 Now imagine deploying cwltool on a cluster with Software Modules installed
425 and that a ``seqtk`` module is available at version ``r93``. This means cluster
426 users likely won't have the binary ``seqtk`` on their ``PATH`` by default, but after
427 sourcing this module with the command ``modulecmd sh load seqtk/r93`` ``seqtk`` is
428 available on the ``PATH``. A simple dependency resolvers configuration file, called
429 ``dependency-resolvers-conf.yml`` for instance, that would enable cwltool to source
430 the correct module environment before executing the above tool would simply be:
431
432 .. code:: yaml
433
434 - type: modules
435
436 The outer list indicates that one plugin is being enabled, the plugin parameters are
437 defined as a dictionary for this one list item. There is only one required parameter
438 for the plugin above, this is ``type`` and defines the plugin type. This parameter
439 is required for all plugins. The available plugins and the parameters
440 available for each are documented (incompletely) `here
441 <https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__.
442 Unfortunately, this documentation is in the context of Galaxy tool
443 ``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly.
444
445 cwltool is distributed with an example of such seqtk tool and sample corresponding
446 job. It could executed from the cwltool root using a dependency resolvers
447 configuration file such as the above one using the command::
448
449 cwltool --beta-dependency-resolvers-configuration /path/to/dependency-resolvers-conf.yml \
450 tests/seqtk_seq.cwl \
451 tests/seqtk_seq_job.json
452
453 This example demonstrates both that cwltool can leverage
454 existing software installations and also handle workflows with dependencies
455 on different versions of the same software and libraries. However the above
456 example does require an existing module setup so it is impossible to test this example
457 "out of the box" with cwltool. For a more isolated test that demonstrates all
458 the same concepts - the resolver plugin type ``galaxy_packages`` can be used.
459
460 "Galaxy packages" are a lighter weight alternative to Environment Modules that are
461 really just defined by a way to lay out directories into packages and versions
462 to find little scripts that are sourced to modify the environment. They have
463 been used for years in Galaxy community to adapt Galaxy tools to cluster
464 environments but require neither knowledge of Galaxy nor any special tools to
465 setup. These should work just fine for CWL tools.
466
467 The cwltool source code repository's test directory is setup with a very simple
468 directory that defines a set of "Galaxy packages" (but really just defines one
469 package named ``random-lines``). The directory layout is simply::
470
471 tests/test_deps_env/
472 random-lines/
473 1.0/
474 env.sh
475
476 If the ``galaxy_packages`` plugin is enabled and pointed at the
477 ``tests/test_deps_env`` directory in cwltool's root and a ``SoftwareRequirement``
478 such as the following is encountered.
479
480 .. code:: yaml
481
482 hints:
483 SoftwareRequirement:
484 packages:
485 - package: 'random-lines'
486 version:
487 - '1.0'
488
489 Then cwltool will simply find that ``env.sh`` file and source it before executing
490 the corresponding tool. That ``env.sh`` script is only responsible for modifying
491 the job's ``PATH`` to add the required binaries.
492
493 This is a full example that works since resolving "Galaxy packages" has no
494 external requirements. Try it out by executing the following command from cwltool's
495 root directory::
496
497 cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf.yml \
498 tests/random_lines.cwl \
499 tests/random_lines_job.json
500
501 The resolvers configuration file in the above example was simply:
502
503 .. code:: yaml
504
505 - type: galaxy_packages
506 base_path: ./tests/test_deps_env
507
508 It is possible that the ``SoftwareRequirement`` s in a given CWL tool will not
509 match the module names for a given cluster. Such requirements can be re-mapped
510 to specific deployed packages and/or versions using another file specified using
511 the resolver plugin parameter `mapping_files`. We will
512 demonstrate this using `galaxy_packages` but the concepts apply equally well
513 to Environment Modules or Conda packages (described below) for instance.
514
515 So consider the resolvers configuration file
516 (`tests/test_deps_env_resolvers_conf_rewrite.yml`):
517
518 .. code:: yaml
519
520 - type: galaxy_packages
521 base_path: ./tests/test_deps_env
522 mapping_files: ./tests/test_deps_mapping.yml
523
524 And the corresponding mapping configuraiton file (`tests/test_deps_mapping.yml`):
525
526 .. code:: yaml
527
528 - from:
529 name: randomLines
530 version: 1.0.0-rc1
531 to:
532 name: random-lines
533 version: '1.0'
534
535 This is saying if cwltool encounters a requirement of ``randomLines`` at version
536 ``1.0.0-rc1`` in a tool, to rewrite to our specific plugin as ``random-lines`` at
537 version ``1.0``. cwltool has such a test tool called ``random_lines_mapping.cwl``
538 that contains such a source ``SoftwareRequirement``. To try out this example with
539 mapping, execute the following command from the cwltool root directory::
540
541 cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf_rewrite.yml \
542 tests/random_lines_mapping.cwl \
543 tests/random_lines_job.json
544
545 The previous examples demonstrated leveraging existing infrastructure to
546 provide requirements for CWL tools. If instead a real package manager is used
547 cwltool has the opportunity to install requirements as needed. While initial
548 support for Homebrew/Linuxbrew plugins is available, the most developed such
549 plugin is for the `Conda <https://conda.io/docs/#>`__ package manager. Conda has the nice properties
550 of allowing multiple versions of a package to be installed simultaneously,
551 not requiring evaluated permissions to install Conda itself or packages using
552 Conda, and being cross platform. For these reasons, cwltool may run as a normal
553 user, install its own Conda environment and manage multiple versions of Conda packages
554 on both Linux and Mac OS X.
555
556 The Conda plugin can be endlessly configured, but a sensible set of defaults
557 that has proven a powerful stack for dependency management within the Galaxy tool
558 development ecosystem can be enabled by simply passing cwltool the
559 ``--beta-conda-dependencies`` flag.
560
561 With this we can use the seqtk example above without Docker and without
562 any externally managed services - cwltool should install everything it needs
563 and create an environment for the tool. Try it out with the follwing command::
564
565 cwltool --beta-conda-dependencies tests/seqtk_seq.cwl tests/seqtk_seq_job.json
566
567 The CWL specification allows URIs to be attached to ``SoftwareRequirement`` s
568 that allow disambiguation of package names. If the mapping files described above
569 allow deployers to adapt tools to their infrastructure, this mechanism allows
570 tools to adapt their requirements to multiple package managers. To demonstrate
571 this within the context of the seqtk, we can simply break the package name we
572 use and then specify a specific Conda package as follows:
573
574 .. code:: yaml
575
576 hints:
577 SoftwareRequirement:
578 packages:
579 - package: seqtk_seq
580 version:
581 - '1.2'
582 specs:
583 - https://anaconda.org/bioconda/seqtk
584 - https://packages.debian.org/sid/seqtk
585
586 The example can be executed using the command::
587
588 cwltool --beta-conda-dependencies tests/seqtk_seq_wrong_name.cwl tests/seqtk_seq_job.json
589
590 The plugin framework for managing resolution of these software requirements
591 as maintained as part of `galaxy-tool-util <https://github.com/galaxyproject/galaxy/tree/dev/packages/tool_util>`__ - a small,
592 portable subset of the Galaxy project. More information on configuration and implementation can be found
593 at the following links:
594
595 - `Dependency Resolvers in Galaxy <https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__
596 - `Conda for [Galaxy] Tool Dependencies <https://docs.galaxyproject.org/en/latest/admin/conda_faq.html>`__
597 - `Mapping Files - Implementation <https://github.com/galaxyproject/galaxy/commit/495802d229967771df5b64a2f79b88a0eaf00edb>`__
598 - `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__
599 - `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__
600
601 Use with GA4GH Tool Registry API
602 --------------------------------
603
604 Cwltool can launch tools directly from `GA4GH Tool Registry API`_ endpoints.
605
606 By default, cwltool searches https://dockstore.org/ . Use ``--add-tool-registry`` to add other registries to the search path.
607
608 For example ::
609
610 cwltool quay.io/collaboratory/dockstore-tool-bamstats:develop test.json
611
612 and (defaults to latest when a version is not specified) ::
613
614 cwltool quay.io/collaboratory/dockstore-tool-bamstats test.json
615
616 For this example, grab the test.json (and input file) from https://github.com/CancerCollaboratory/dockstore-tool-bamstats ::
617
618 wget https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fbriandoconnor%2Fdockstore-tool-bamstats/versions/develop/PLAIN-CWL/descriptor/test.json
619 wget https://github.com/CancerCollaboratory/dockstore-tool-bamstats/raw/develop/rna.SRR948778.bam
620
621
622 .. _`GA4GH Tool Registry API`: https://github.com/ga4gh/tool-registry-schemas
623
624 ===========
625 Development
626 ===========
627
628 Running tests locally
629 ---------------------
630
631 - Running basic tests ``(/tests)``:
632
633 To run the basic tests after installing `cwltool` execute the following:
634
635 .. code:: bash
636
637 pip install -rtest-requirements.txt
638 py.test --ignore cwltool/schemas/ --pyarg cwltool
639
640 To run various tests in all supported Python environments we use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_. To run the test suite in all supported Python environments
641 first downloading the complete code repository (see the ``git clone`` instructions above) and then run
642 the following in the terminal:
643 ``pip install tox; tox``
644
645 List of all environment can be seen using:
646 ``tox --listenvs``
647 and running a specfic test env using:
648 ``tox -e <env name>``
649 and additionally run a specific test using this format:
650 ``tox -e py36-unit -- tests/test_examples.py::TestParamMatching``
651
652 - Running the entire suite of CWL conformance tests:
653
654 The GitHub repository for the CWL specifications contains a script that tests a CWL
655 implementation against a wide array of valid CWL files using the `cwltest <https://github.com/common-workflow-language/cwltest>`_
656 program
657
658 Instructions for running these tests can be found in the Common Workflow Language Specification repository at https://github.com/common-workflow-language/common-workflow-language/blob/master/CONFORMANCE_TESTS.md
659
660 Import as a module
661 ------------------
662
663 Add
664
665 .. code:: python
666
667 import cwltool
668
669 to your script.
670
671 The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory
672
673 .. code:: python
674
675 import cwltool.factory
676 fac = cwltool.factory.Factory()
677
678 echo = fac.make("echo.cwl")
679 result = echo(inp="foo")
680
681 # result["out"] == "foo"
682
683
684 CWL Tool Control Flow
685 ---------------------
686
687 Technical outline of how cwltool works internally, for maintainers.
688
689 #. Use CWL ``load_tool()`` to load document.
690
691 #. Fetches the document from file or URL
692 #. Applies preprocessing (syntax/identifier expansion and normalization)
693 #. Validates the document based on cwlVersion
694 #. If necessary, updates the document to latest spec
695 #. Constructs a Process object using ``make_tool()``` callback. This yields a
696 CommandLineTool, Workflow, or ExpressionTool. For workflows, this
697 recursively constructs each workflow step.
698 #. To construct custom types for CommandLineTool, Workflow, or
699 ExpressionTool, provide a custom ``make_tool()``
700
701 #. Iterate on the ``job()`` method of the Process object to get back runnable jobs.
702
703 #. ``job()`` is a generator method (uses the Python iterator protocol)
704 #. Each time the ``job()`` method is invoked in an iteration, it returns one
705 of: a runnable item (an object with a ``run()`` method), ``None`` (indicating
706 there is currently no work ready to run) or end of iteration (indicating
707 the process is complete.)
708 #. Invoke the runnable item by calling ``run()``. This runs the tool and gets output.
709 #. Output of a process is reported by an output callback.
710 #. ``job()`` may be iterated over multiple times. It will yield all the work
711 that is currently ready to run and then yield None.
712
713 #. ``Workflow`` objects create a corresponding ``WorkflowJob`` and ``WorkflowJobStep`` objects to hold the workflow state for the duration of the job invocation.
714
715 #. The WorkflowJob iterates over each WorkflowJobStep and determines if the
716 inputs the step are ready.
717 #. When a step is ready, it constructs an input object for that step and
718 iterates on the ``job()`` method of the workflow job step.
719 #. Each runnable item is yielded back up to top level run loop
720 #. When a step job completes and receives an output callback, the
721 job outputs are assigned to the output of the workflow step.
722 #. When all steps are complete, the intermediate files are moved to a final
723 workflow output, intermediate directories are deleted, and the output
724 callback for the workflow is called.
725
726 #. ``CommandLineTool`` job() objects yield a single runnable object.
727
728 #. The CommandLineTool ``job()`` method calls ``make_job_runner()`` to create a
729 ``CommandLineJob`` object
730 #. The job method configures the CommandLineJob object by setting public
731 attributes
732 #. The job method iterates over file and directories inputs to the
733 CommandLineTool and creates a "path map".
734 #. Files are mapped from their "resolved" location to a "target" path where
735 they will appear at tool invocation (for example, a location inside a
736 Docker container.) The target paths are used on the command line.
737 #. Files are staged to targets paths using either Docker volume binds (when
738 using containers) or symlinks (if not). This staging step enables files
739 to be logically rearranged or renamed independent of their source layout.
740 #. The ``run()`` method of CommandLineJob executes the command line tool or
741 Docker container, waits for it to complete, collects output, and makes
742 the output callback.
743
744
745 Extension points
746 ----------------
747
748 The following functions can be passed to main() to override or augment
749 the listed behaviors.
750
751 executor
752 ::
753
754 executor(tool, job_order_object, runtimeContext, logger)
755 (Process, Dict[Text, Any], RuntimeContext) -> Tuple[Dict[Text, Any], Text]
756
757 An implementation of the toplevel workflow execution loop, should
758 synchronously run a process object to completion and return the
759 output object.
760
761 versionfunc
762 ::
763
764 ()
765 () -> Text
766
767 Return version string.
768
769 logger_handler
770 ::
771
772 logger_handler
773 logging.Handler
774
775 Handler object for logging.
776
777 The following functions can be set in LoadingContext to override or
778 augment the listed behaviors.
779
780 fetcher_constructor
781 ::
782
783 fetcher_constructor(cache, session)
784 (Dict[unicode, unicode], requests.sessions.Session) -> Fetcher
785
786 Construct a Fetcher object with the supplied cache and HTTP session.
787
788 resolver
789 ::
790
791 resolver(document_loader, document)
792 (Loader, Union[Text, dict[Text, Any]]) -> Text
793
794 Resolve a relative document identifier to an absolute one which can be fetched.
795
796 The following functions can be set in RuntimeContext to override or
797 augment the listed behaviors.
798
799 construct_tool_object
800 ::
801
802 construct_tool_object(toolpath_object, loadingContext)
803 (MutableMapping[Text, Any], LoadingContext) -> Process
804
805 Hook to construct a Process object (eg CommandLineTool) object from a document.
806
807 select_resources
808 ::
809
810 selectResources(request)
811 (Dict[str, int], RuntimeContext) -> Dict[Text, int]
812
813 Take a resource request and turn it into a concrete resource assignment.
814
815 make_fs_access
816 ::
817
818 make_fs_access(basedir)
819 (Text) -> StdFsAccess
820
821 Return a file system access object.
822
823 In addition, when providing custom subclasses of Process objects, you can override the following methods:
824
825 CommandLineTool.make_job_runner
826 ::
827
828 make_job_runner(RuntimeContext)
829 (RuntimeContext) -> Type[JobBase]
830
831 Create and return a job runner object (this implements concrete execution of a command line tool).
832
833 Workflow.make_workflow_step
834 ::
835
836 make_workflow_step(toolpath_object, pos, loadingContext, parentworkflowProv)
837 (Dict[Text, Any], int, LoadingContext, Optional[ProvenanceProfile]) -> WorkflowStep
838
839 Create and return a workflow step object.
840
841