Mercurial > repos > shellac > guppy_basecaller
diff env/lib/python3.7/site-packages/cwltool-1.0.20191225192155.dist-info/METADATA @ 0:26e78fe6e8c4 draft
"planemo upload commit c699937486c35866861690329de38ec1a5d9f783"
author | shellac |
---|---|
date | Sat, 02 May 2020 07:14:21 -0400 |
parents | |
children |
line wrap: on
line diff
--- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/env/lib/python3.7/site-packages/cwltool-1.0.20191225192155.dist-info/METADATA Sat May 02 07:14:21 2020 -0400 @@ -0,0 +1,841 @@ +Metadata-Version: 2.1 +Name: cwltool +Version: 1.0.20191225192155 +Summary: Common workflow language reference implementation +Home-page: https://github.com/common-workflow-language/cwltool +Author: Common workflow language working group +Author-email: common-workflow-language@googlegroups.com +License: UNKNOWN +Download-URL: https://github.com/common-workflow-language/cwltool +Platform: UNKNOWN +Classifier: Development Status :: 5 - Production/Stable +Classifier: Environment :: Console +Classifier: Intended Audience :: Developers +Classifier: Intended Audience :: Science/Research +Classifier: Intended Audience :: Healthcare Industry +Classifier: License :: OSI Approved :: Apache Software License +Classifier: Natural Language :: English +Classifier: Operating System :: MacOS :: MacOS X +Classifier: Operating System :: POSIX +Classifier: Operating System :: POSIX :: Linux +Classifier: Operating System :: OS Independent +Classifier: Operating System :: Microsoft :: Windows +Classifier: Operating System :: Microsoft :: Windows :: Windows 10 +Classifier: Operating System :: Microsoft :: Windows :: Windows 8.1 +Classifier: Programming Language :: Python :: 2 +Classifier: Programming Language :: Python :: 2.7 +Classifier: Programming Language :: Python :: 3 +Classifier: Programming Language :: Python :: 3.5 +Classifier: Programming Language :: Python :: 3.6 +Classifier: Programming Language :: Python :: 3.7 +Classifier: Programming Language :: Python :: 3.8 +Classifier: Topic :: Scientific/Engineering +Classifier: Topic :: Scientific/Engineering :: Bio-Informatics +Classifier: Topic :: Scientific/Engineering :: Astronomy +Classifier: Topic :: Scientific/Engineering :: Atmospheric Science +Classifier: Topic :: Scientific/Engineering :: Information Analysis +Classifier: Topic :: Scientific/Engineering :: Medical Science Apps. +Classifier: Topic :: System :: Distributed Computing +Classifier: Topic :: Utilities +Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4 +Description-Content-Type: text/x-rst +Requires-Dist: setuptools +Requires-Dist: requests (>=2.6.1) +Requires-Dist: ruamel.yaml (<=0.16,>=0.12.4) +Requires-Dist: rdflib (<4.3.0,>=4.2.2) +Requires-Dist: shellescape (<3.5,>=3.4.1) +Requires-Dist: schema-salad (<5,>=4.5) +Requires-Dist: mypy-extensions +Requires-Dist: six (>=1.9.0) +Requires-Dist: psutil +Requires-Dist: scandir +Requires-Dist: prov (==1.5.1) +Requires-Dist: bagit (>=1.6.4) +Requires-Dist: typing-extensions +Requires-Dist: coloredlogs +Requires-Dist: future (>=0.16) +Requires-Dist: pathlib2 (!=2.3.1) +Requires-Dist: subprocess32 (>=3.5.0) ; os.name=="posix" and python_version<"3.5" +Requires-Dist: typing (>=3.5.3) ; python_version<"3.6" +Provides-Extra: deps +Requires-Dist: galaxy-tool-util ; extra == 'deps' +Provides-Extra: docs +Requires-Dist: sphinx (>=2.2) ; extra == 'docs' +Requires-Dist: sphinx-rtd-theme ; extra == 'docs' + +================================================================== +Common Workflow Language tool description reference implementation +================================================================== + +CWL conformance tests: |Conformance Status| |Linux Status| |Windows Status| |Coverage Status| |Downloads| + +|CommandLineTool Support| |DockerRequirement Support| |EnvVarRequirement Support| |ExpressionTool Support| +|InitialWorkDirRequirement Support| |InlineJavascriptRequirement Support| |MultipleInputRequirement Support| |Core Support| +|ResourceRequirement Support| |ScatterRequirement Support| |SchemaDefRequirement Support| |ShellCommandequirement Support| +|StepInputRequirement Support| |SubWorkflowRequirement Support| |Workflow Support| + +.. |Conformance Status| image:: https://ci.commonwl.org/buildStatus/icon?job=cwltool-conformance + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |Linux Status| image:: https://img.shields.io/travis/common-workflow-language/cwltool/master.svg?label=Linux%20builds + :target: https://travis-ci.org/common-workflow-language/cwltool + +.. |Windows Status| image:: https://img.shields.io/appveyor/ci/mr-c/cwltool/master.svg?label=Windows%20builds + :target: https://ci.appveyor.com/project/mr-c/cwltool + +.. |Coverage Status| image:: https://img.shields.io/codecov/c/github/common-workflow-language/cwltool.svg + :target: https://codecov.io/gh/common-workflow-language/cwltool + +.. |Downloads| image:: https://pepy.tech/badge/cwltool/month + :target: https://pepy.tech/project/cwltool + +.. |CommandLineTool Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/command_line_tool.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |DockerRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/docker.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |EnvVarRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/env_var.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |ExpressionTool Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/expression_tool.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |InitialWorkDirRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/initial_work_dir.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |InlineJavascriptRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/inline_javascript.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |MultipleInputRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/multiple_input.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |Core Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/required.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |ResourceRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/resource.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |ScatterRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/scatter.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |SchemaDefRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/schema_def.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |ShellCommandequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/shell_command.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |StepInputRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/step_input.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |SubWorkflowRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/subworkflow.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + +.. |Workflow Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/workflow.json + :target: https://ci.commonwl.org/job/cwltool-conformance/ + + +This is the reference implementation of the Common Workflow Language. It is +intended to be feature complete and provide comprehensive validation of CWL +files as well as provide other tools related to working with CWL. + +This is written and tested for +`Python <https://www.python.org/>`_ ``2.7 and 3.x {x = 5, 6, 7, 8}`` + +The reference implementation consists of two packages. The ``cwltool`` package +is the primary Python module containing the reference implementation in the +``cwltool`` module and console executable by the same name. + +The ``cwlref-runner`` package is optional and provides an additional entry point +under the alias ``cwl-runner``, which is the implementation-agnostic name for the +default CWL interpreter installed on a host. + +Install +------- + +Your operating system may offer cwltool directly. For [Debian](https://tracker.debian.org/pkg/cwltool "Debian cwltool package tracker") or [Ubuntu](https://launchpad.net/ubuntu/+source/cwltool "Ubuntu Launchpad overview for cwltool") try + +.. code:: bash + + apt-get install cwltool + +Otherwise, to +avoid conflicting versions of the same library, +it is recommended to do the following: + +.. code:: bash + + virtualenv -p python2 venv # Create a virtual environment, can use `python3` as well + source venv/bin/activate # Activate environment before installing `cwltool` + +Installing the official package from PyPi (will install "cwltool" package as +well) + +.. code:: bash + + pip install cwlref-runner + +If installing alongside another CWL implementation then + +.. code:: bash + + pip install cwltool + +Or you can install from source: + +.. code:: bash + + git clone https://github.com/common-workflow-language/cwltool.git # clone cwltool repo + cd cwltool # Switch to source directory + pip install . # Install `cwltool` from source + cwltool --version # Check if the installation works correctly + +Remember, if co-installing multiple CWL implementations then you need to +maintain which implementation ``cwl-runner`` points to via a symbolic file +system link or `another facility <https://wiki.debian.org/DebianAlternatives>`_. + +===== +Usage +===== + +Run on the command line +----------------------- + +Simple command:: + + cwl-runner [tool-or-workflow-description] [input-job-settings] + +Or if you have multiple CWL implementations installed and you want to override +the default cwl-runner use:: + + cwltool [tool-or-workflow-description] [input-job-settings] + +You can set cwltool options in the environment with CWLTOOL_OPTIONS, +these will be inserted at the beginning of the command line:: + + export CWLTOOL_OPTIONS="--debug" + +Use with boot2docker +-------------------- +boot2docker runs Docker inside a virtual machine and it only mounts ``Users`` +on it. The default behavior of CWL is to create temporary directories under e.g. +``/Var`` which is not accessible to Docker containers. + +To run CWL successfully with boot2docker you need to set the ``--tmpdir-prefix`` +and ``--tmp-outdir-prefix`` to somewhere under ``/Users``:: + + $ cwl-runner --tmp-outdir-prefix=/Users/username/project --tmpdir-prefix=/Users/username/project wc-tool.cwl wc-job.json + +Using user-space replacements for Docker +---------------------------------------- + +Some shared computing environments don't support Docker software containers for technical or policy reasons. +As a work around, the CWL reference runner supports using alternative ``docker`` implementations on Linux +with the ``--user-space-docker-cmd`` option. + +One such "user space" friendly docker replacement is ``udocker`` https://github.com/indigo-dc/udocker and another +is ``dx-docker`` https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images + +udocker installation: https://github.com/indigo-dc/udocker/blob/master/doc/installation_manual.md#22-install-from-indigo-datacloud-repositories + +dx-docker installation: start with the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions). + +Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests: + +.. code:: bash + + cwltool --user-space-docker-cmd=udocker https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/test-cwl-out2.cwl https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/empty.json + +or + +.. code:: bash + + cwltool --user-space-docker-cmd=dx-docker https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/test-cwl-out2.cwl https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/empty.json + +``cwltool`` can use `Singularity <http://singularity.lbl.gov/>`_ version 2.6.1 +or later as a Docker container runtime. +``cwltool`` with Singularity will run software containers specified in +``DockerRequirement`` and therefore works with Docker images only, native +Singularity images are not supported. To use Singularity as the Docker container +runtime, provide ``--singularity`` command line option to ``cwltool``. +With Singularity, ``cwltool`` can pass all CWL v1.0 conformance tests, except +those involving Docker container ENTRYPOINTs. + + +.. code:: bash + + cwltool --singularity https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/v1.0/cat3-tool-mediumcut.cwl https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/cat-job.json + +Running a tool or workflow from remote or local locations +--------------------------------------------------------- + +``cwltool`` can run tool and workflow descriptions on both local and remote +systems via its support for HTTP[S] URLs. + +Input job files and Workflow steps (via the `run` directive) can reference CWL +documents using absolute or relative local filesytem paths. If a relative path +is referenced and that document isn't found in the current directory then the +following locations will be searched: +http://www.commonwl.org/v1.0/CommandLineTool.html#Discovering_CWL_documents_on_a_local_filesystem + +You can also use `cwldep <https://github.com/common-workflow-language/cwldep>` +to manage dependencies on external tools and workflows. + +Overriding workflow requirements at load time +--------------------------------------------- + +Sometimes a workflow needs additional requirements to run in a particular +environment or with a particular dataset. To avoid the need to modify the +underlying workflow, cwltool supports requirement "overrides". + +The format of the "overrides" object is a mapping of item identifier (workflow, +workflow step, or command line tool) to the process requirements that should be applied. + +.. code:: yaml + + cwltool:overrides: + echo.cwl: + requirements: + EnvVarRequirement: + envDef: + MESSAGE: override_value + +Overrides can be specified either on the command line, or as part of the job +input document. Workflow steps are identified using the name of the workflow +file followed by the step name as a document fragment identifier "#id". +Override identifiers are relative to the toplevel workflow document. + +.. code:: bash + + cwltool --overrides overrides.yml my-tool.cwl my-job.yml + +.. code:: yaml + + input_parameter1: value1 + input_parameter2: value2 + cwltool:overrides: + workflow.cwl#step1: + requirements: + EnvVarRequirement: + envDef: + MESSAGE: override_value + +.. code:: bash + + cwltool my-tool.cwl my-job-with-overrides.yml + + +Combining parts of a workflow into a single document +---------------------------------------------------- + +Use ``--pack`` to combine a workflow made up of multiple files into a +single compound document. This operation takes all the CWL files +referenced by a workflow and builds a new CWL document with all +Process objects (CommandLineTool and Workflow) in a list in the +``$graph`` field. Cross references (such as ``run:`` and ``source:`` +fields) are updated to internal references within the new packed +document. The top level workflow is named ``#main``. + +.. code:: bash + + cwltool --pack my-wf.cwl > my-packed-wf.cwl + + +Running only part of a workflow +------------------------------- + +You can run a partial workflow with the ``--target`` (``-t``) option. This +takes the name of an output parameter, workflow step, or input +parameter in the top level workflow. You may provide multiple +targets. + +.. code:: bash + + cwltool --target step3 my-wf.cwl + +If a target is an output parameter, it will only run only the steps +that contribute to that output. If a target is a workflow step, it +will run the workflow starting from that step. If a target is an +input parameter, it will only run only the steps that are connected to +that input. + +Use ``--print-targets`` to get a listing of the targets of a workflow. +To see exactly which steps will run, use ``--print-subgraph`` with +``--target`` to get a printout of the workflow subgraph for the +selected targets. + +.. code:: bash + + cwltool --print-targets my-wf.cwl + + cwltool --target step3 --print-subgraph my-wf.cwl > my-wf-starting-from-step3.cwl + + +Visualizing a CWL document +-------------------------- + +The ``--print-dot`` option will print a file suitable for Graphviz ``dot`` program. Here is a bash onliner to generate a Scalable Vector Graphic (SVG) file: + +.. code:: bash + + cwltool --print-dot my-wf.cwl | dot -Tsvg > my-wf.svg + +Modeling a CWL document as RDF +------------------------------ + +CWL documents can be expressed as RDF triple graphs. + +.. code:: bash + + cwltool --print-rdf --rdf-serializer=turtle mywf.cwl + + +Leveraging SoftwareRequirements (Beta) +-------------------------------------- + +CWL tools may be decorated with ``SoftwareRequirement`` hints that cwltool +may in turn use to resolve to packages in various package managers or +dependency management systems such as `Environment Modules +<http://modules.sourceforge.net/>`__. + +Utilizing ``SoftwareRequirement`` hints using cwltool requires an optional +dependency, for this reason be sure to use specify the ``deps`` modifier when +installing cwltool. For instance:: + + $ pip install 'cwltool[deps]' + +Installing cwltool in this fashion enables several new command line options. +The most general of these options is ``--beta-dependency-resolvers-configuration``. +This option allows one to specify a dependency resolver's configuration file. +This file may be specified as either XML or YAML and very simply describes various +plugins to enable to "resolve" ``SoftwareRequirement`` dependencies. + +To discuss some of these plugins and how to configure them, first consider the +following ``hint`` definition for an example CWL tool. + +.. code:: yaml + + SoftwareRequirement: + packages: + - package: seqtk + version: + - r93 + +Now imagine deploying cwltool on a cluster with Software Modules installed +and that a ``seqtk`` module is available at version ``r93``. This means cluster +users likely won't have the binary ``seqtk`` on their ``PATH`` by default, but after +sourcing this module with the command ``modulecmd sh load seqtk/r93`` ``seqtk`` is +available on the ``PATH``. A simple dependency resolvers configuration file, called +``dependency-resolvers-conf.yml`` for instance, that would enable cwltool to source +the correct module environment before executing the above tool would simply be: + +.. code:: yaml + + - type: modules + +The outer list indicates that one plugin is being enabled, the plugin parameters are +defined as a dictionary for this one list item. There is only one required parameter +for the plugin above, this is ``type`` and defines the plugin type. This parameter +is required for all plugins. The available plugins and the parameters +available for each are documented (incompletely) `here +<https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__. +Unfortunately, this documentation is in the context of Galaxy tool +``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly. + +cwltool is distributed with an example of such seqtk tool and sample corresponding +job. It could executed from the cwltool root using a dependency resolvers +configuration file such as the above one using the command:: + + cwltool --beta-dependency-resolvers-configuration /path/to/dependency-resolvers-conf.yml \ + tests/seqtk_seq.cwl \ + tests/seqtk_seq_job.json + +This example demonstrates both that cwltool can leverage +existing software installations and also handle workflows with dependencies +on different versions of the same software and libraries. However the above +example does require an existing module setup so it is impossible to test this example +"out of the box" with cwltool. For a more isolated test that demonstrates all +the same concepts - the resolver plugin type ``galaxy_packages`` can be used. + +"Galaxy packages" are a lighter weight alternative to Environment Modules that are +really just defined by a way to lay out directories into packages and versions +to find little scripts that are sourced to modify the environment. They have +been used for years in Galaxy community to adapt Galaxy tools to cluster +environments but require neither knowledge of Galaxy nor any special tools to +setup. These should work just fine for CWL tools. + +The cwltool source code repository's test directory is setup with a very simple +directory that defines a set of "Galaxy packages" (but really just defines one +package named ``random-lines``). The directory layout is simply:: + + tests/test_deps_env/ + random-lines/ + 1.0/ + env.sh + +If the ``galaxy_packages`` plugin is enabled and pointed at the +``tests/test_deps_env`` directory in cwltool's root and a ``SoftwareRequirement`` +such as the following is encountered. + +.. code:: yaml + + hints: + SoftwareRequirement: + packages: + - package: 'random-lines' + version: + - '1.0' + +Then cwltool will simply find that ``env.sh`` file and source it before executing +the corresponding tool. That ``env.sh`` script is only responsible for modifying +the job's ``PATH`` to add the required binaries. + +This is a full example that works since resolving "Galaxy packages" has no +external requirements. Try it out by executing the following command from cwltool's +root directory:: + + cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf.yml \ + tests/random_lines.cwl \ + tests/random_lines_job.json + +The resolvers configuration file in the above example was simply: + +.. code:: yaml + + - type: galaxy_packages + base_path: ./tests/test_deps_env + +It is possible that the ``SoftwareRequirement`` s in a given CWL tool will not +match the module names for a given cluster. Such requirements can be re-mapped +to specific deployed packages and/or versions using another file specified using +the resolver plugin parameter `mapping_files`. We will +demonstrate this using `galaxy_packages` but the concepts apply equally well +to Environment Modules or Conda packages (described below) for instance. + +So consider the resolvers configuration file +(`tests/test_deps_env_resolvers_conf_rewrite.yml`): + +.. code:: yaml + + - type: galaxy_packages + base_path: ./tests/test_deps_env + mapping_files: ./tests/test_deps_mapping.yml + +And the corresponding mapping configuraiton file (`tests/test_deps_mapping.yml`): + +.. code:: yaml + + - from: + name: randomLines + version: 1.0.0-rc1 + to: + name: random-lines + version: '1.0' + +This is saying if cwltool encounters a requirement of ``randomLines`` at version +``1.0.0-rc1`` in a tool, to rewrite to our specific plugin as ``random-lines`` at +version ``1.0``. cwltool has such a test tool called ``random_lines_mapping.cwl`` +that contains such a source ``SoftwareRequirement``. To try out this example with +mapping, execute the following command from the cwltool root directory:: + + cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf_rewrite.yml \ + tests/random_lines_mapping.cwl \ + tests/random_lines_job.json + +The previous examples demonstrated leveraging existing infrastructure to +provide requirements for CWL tools. If instead a real package manager is used +cwltool has the opportunity to install requirements as needed. While initial +support for Homebrew/Linuxbrew plugins is available, the most developed such +plugin is for the `Conda <https://conda.io/docs/#>`__ package manager. Conda has the nice properties +of allowing multiple versions of a package to be installed simultaneously, +not requiring evaluated permissions to install Conda itself or packages using +Conda, and being cross platform. For these reasons, cwltool may run as a normal +user, install its own Conda environment and manage multiple versions of Conda packages +on both Linux and Mac OS X. + +The Conda plugin can be endlessly configured, but a sensible set of defaults +that has proven a powerful stack for dependency management within the Galaxy tool +development ecosystem can be enabled by simply passing cwltool the +``--beta-conda-dependencies`` flag. + +With this we can use the seqtk example above without Docker and without +any externally managed services - cwltool should install everything it needs +and create an environment for the tool. Try it out with the follwing command:: + + cwltool --beta-conda-dependencies tests/seqtk_seq.cwl tests/seqtk_seq_job.json + +The CWL specification allows URIs to be attached to ``SoftwareRequirement`` s +that allow disambiguation of package names. If the mapping files described above +allow deployers to adapt tools to their infrastructure, this mechanism allows +tools to adapt their requirements to multiple package managers. To demonstrate +this within the context of the seqtk, we can simply break the package name we +use and then specify a specific Conda package as follows: + +.. code:: yaml + + hints: + SoftwareRequirement: + packages: + - package: seqtk_seq + version: + - '1.2' + specs: + - https://anaconda.org/bioconda/seqtk + - https://packages.debian.org/sid/seqtk + +The example can be executed using the command:: + + cwltool --beta-conda-dependencies tests/seqtk_seq_wrong_name.cwl tests/seqtk_seq_job.json + +The plugin framework for managing resolution of these software requirements +as maintained as part of `galaxy-tool-util <https://github.com/galaxyproject/galaxy/tree/dev/packages/tool_util>`__ - a small, +portable subset of the Galaxy project. More information on configuration and implementation can be found +at the following links: + +- `Dependency Resolvers in Galaxy <https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__ +- `Conda for [Galaxy] Tool Dependencies <https://docs.galaxyproject.org/en/latest/admin/conda_faq.html>`__ +- `Mapping Files - Implementation <https://github.com/galaxyproject/galaxy/commit/495802d229967771df5b64a2f79b88a0eaf00edb>`__ +- `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__ +- `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__ + +Use with GA4GH Tool Registry API +-------------------------------- + +Cwltool can launch tools directly from `GA4GH Tool Registry API`_ endpoints. + +By default, cwltool searches https://dockstore.org/ . Use ``--add-tool-registry`` to add other registries to the search path. + +For example :: + + cwltool quay.io/collaboratory/dockstore-tool-bamstats:develop test.json + +and (defaults to latest when a version is not specified) :: + + cwltool quay.io/collaboratory/dockstore-tool-bamstats test.json + +For this example, grab the test.json (and input file) from https://github.com/CancerCollaboratory/dockstore-tool-bamstats :: + + wget https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fbriandoconnor%2Fdockstore-tool-bamstats/versions/develop/PLAIN-CWL/descriptor/test.json + wget https://github.com/CancerCollaboratory/dockstore-tool-bamstats/raw/develop/rna.SRR948778.bam + + +.. _`GA4GH Tool Registry API`: https://github.com/ga4gh/tool-registry-schemas + +=========== +Development +=========== + +Running tests locally +--------------------- + +- Running basic tests ``(/tests)``: + +To run the basic tests after installing `cwltool` execute the following: + +.. code:: bash + + pip install -rtest-requirements.txt + py.test --ignore cwltool/schemas/ --pyarg cwltool + +To run various tests in all supported Python environments we use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_. To run the test suite in all supported Python environments +first downloading the complete code repository (see the ``git clone`` instructions above) and then run +the following in the terminal: +``pip install tox; tox`` + +List of all environment can be seen using: +``tox --listenvs`` +and running a specfic test env using: +``tox -e <env name>`` +and additionally run a specific test using this format: +``tox -e py36-unit -- tests/test_examples.py::TestParamMatching`` + +- Running the entire suite of CWL conformance tests: + +The GitHub repository for the CWL specifications contains a script that tests a CWL +implementation against a wide array of valid CWL files using the `cwltest <https://github.com/common-workflow-language/cwltest>`_ +program + +Instructions for running these tests can be found in the Common Workflow Language Specification repository at https://github.com/common-workflow-language/common-workflow-language/blob/master/CONFORMANCE_TESTS.md + +Import as a module +------------------ + +Add + +.. code:: python + + import cwltool + +to your script. + +The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory + +.. code:: python + + import cwltool.factory + fac = cwltool.factory.Factory() + + echo = fac.make("echo.cwl") + result = echo(inp="foo") + + # result["out"] == "foo" + + +CWL Tool Control Flow +--------------------- + +Technical outline of how cwltool works internally, for maintainers. + +#. Use CWL ``load_tool()`` to load document. + + #. Fetches the document from file or URL + #. Applies preprocessing (syntax/identifier expansion and normalization) + #. Validates the document based on cwlVersion + #. If necessary, updates the document to latest spec + #. Constructs a Process object using ``make_tool()``` callback. This yields a + CommandLineTool, Workflow, or ExpressionTool. For workflows, this + recursively constructs each workflow step. + #. To construct custom types for CommandLineTool, Workflow, or + ExpressionTool, provide a custom ``make_tool()`` + +#. Iterate on the ``job()`` method of the Process object to get back runnable jobs. + + #. ``job()`` is a generator method (uses the Python iterator protocol) + #. Each time the ``job()`` method is invoked in an iteration, it returns one + of: a runnable item (an object with a ``run()`` method), ``None`` (indicating + there is currently no work ready to run) or end of iteration (indicating + the process is complete.) + #. Invoke the runnable item by calling ``run()``. This runs the tool and gets output. + #. Output of a process is reported by an output callback. + #. ``job()`` may be iterated over multiple times. It will yield all the work + that is currently ready to run and then yield None. + +#. ``Workflow`` objects create a corresponding ``WorkflowJob`` and ``WorkflowJobStep`` objects to hold the workflow state for the duration of the job invocation. + + #. The WorkflowJob iterates over each WorkflowJobStep and determines if the + inputs the step are ready. + #. When a step is ready, it constructs an input object for that step and + iterates on the ``job()`` method of the workflow job step. + #. Each runnable item is yielded back up to top level run loop + #. When a step job completes and receives an output callback, the + job outputs are assigned to the output of the workflow step. + #. When all steps are complete, the intermediate files are moved to a final + workflow output, intermediate directories are deleted, and the output + callback for the workflow is called. + +#. ``CommandLineTool`` job() objects yield a single runnable object. + + #. The CommandLineTool ``job()`` method calls ``make_job_runner()`` to create a + ``CommandLineJob`` object + #. The job method configures the CommandLineJob object by setting public + attributes + #. The job method iterates over file and directories inputs to the + CommandLineTool and creates a "path map". + #. Files are mapped from their "resolved" location to a "target" path where + they will appear at tool invocation (for example, a location inside a + Docker container.) The target paths are used on the command line. + #. Files are staged to targets paths using either Docker volume binds (when + using containers) or symlinks (if not). This staging step enables files + to be logically rearranged or renamed independent of their source layout. + #. The ``run()`` method of CommandLineJob executes the command line tool or + Docker container, waits for it to complete, collects output, and makes + the output callback. + + +Extension points +---------------- + +The following functions can be passed to main() to override or augment +the listed behaviors. + +executor + :: + + executor(tool, job_order_object, runtimeContext, logger) + (Process, Dict[Text, Any], RuntimeContext) -> Tuple[Dict[Text, Any], Text] + + An implementation of the toplevel workflow execution loop, should + synchronously run a process object to completion and return the + output object. + +versionfunc + :: + + () + () -> Text + + Return version string. + +logger_handler + :: + + logger_handler + logging.Handler + + Handler object for logging. + +The following functions can be set in LoadingContext to override or +augment the listed behaviors. + +fetcher_constructor + :: + + fetcher_constructor(cache, session) + (Dict[unicode, unicode], requests.sessions.Session) -> Fetcher + + Construct a Fetcher object with the supplied cache and HTTP session. + +resolver + :: + + resolver(document_loader, document) + (Loader, Union[Text, dict[Text, Any]]) -> Text + + Resolve a relative document identifier to an absolute one which can be fetched. + +The following functions can be set in RuntimeContext to override or +augment the listed behaviors. + +construct_tool_object + :: + + construct_tool_object(toolpath_object, loadingContext) + (MutableMapping[Text, Any], LoadingContext) -> Process + + Hook to construct a Process object (eg CommandLineTool) object from a document. + +select_resources + :: + + selectResources(request) + (Dict[str, int], RuntimeContext) -> Dict[Text, int] + + Take a resource request and turn it into a concrete resource assignment. + +make_fs_access + :: + + make_fs_access(basedir) + (Text) -> StdFsAccess + + Return a file system access object. + +In addition, when providing custom subclasses of Process objects, you can override the following methods: + +CommandLineTool.make_job_runner + :: + + make_job_runner(RuntimeContext) + (RuntimeContext) -> Type[JobBase] + + Create and return a job runner object (this implements concrete execution of a command line tool). + +Workflow.make_workflow_step + :: + + make_workflow_step(toolpath_object, pos, loadingContext, parentworkflowProv) + (Dict[Text, Any], int, LoadingContext, Optional[ProvenanceProfile]) -> WorkflowStep + + Create and return a workflow step object. + +