Mercurial > repos > shellac > guppy_basecaller
diff env/lib/python3.7/site-packages/cwltool-1.0.20191225192155.dist-info/METADATA @ 5:9b1c78e6ba9c draft default tip
"planemo upload commit 6c0a8142489327ece472c84e558c47da711a9142"
author | shellac |
---|---|
date | Mon, 01 Jun 2020 08:59:25 -0400 |
parents | 79f47841a781 |
children |
line wrap: on
line diff
--- a/env/lib/python3.7/site-packages/cwltool-1.0.20191225192155.dist-info/METADATA Thu May 14 16:47:39 2020 -0400 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000 @@ -1,841 +0,0 @@ -Metadata-Version: 2.1 -Name: cwltool -Version: 1.0.20191225192155 -Summary: Common workflow language reference implementation -Home-page: https://github.com/common-workflow-language/cwltool -Author: Common workflow language working group -Author-email: common-workflow-language@googlegroups.com -License: UNKNOWN -Download-URL: https://github.com/common-workflow-language/cwltool -Platform: UNKNOWN -Classifier: Development Status :: 5 - Production/Stable -Classifier: Environment :: Console -Classifier: Intended Audience :: Developers -Classifier: Intended Audience :: Science/Research -Classifier: Intended Audience :: Healthcare Industry -Classifier: License :: OSI Approved :: Apache Software License -Classifier: Natural Language :: English -Classifier: Operating System :: MacOS :: MacOS X -Classifier: Operating System :: POSIX -Classifier: Operating System :: POSIX :: Linux -Classifier: Operating System :: OS Independent -Classifier: Operating System :: Microsoft :: Windows -Classifier: Operating System :: Microsoft :: Windows :: Windows 10 -Classifier: Operating System :: Microsoft :: Windows :: Windows 8.1 -Classifier: Programming Language :: Python :: 2 -Classifier: Programming Language :: Python :: 2.7 -Classifier: Programming Language :: Python :: 3 -Classifier: Programming Language :: Python :: 3.5 -Classifier: Programming Language :: Python :: 3.6 -Classifier: Programming Language :: Python :: 3.7 -Classifier: Programming Language :: Python :: 3.8 -Classifier: Topic :: Scientific/Engineering -Classifier: Topic :: Scientific/Engineering :: Bio-Informatics -Classifier: Topic :: Scientific/Engineering :: Astronomy -Classifier: Topic :: Scientific/Engineering :: Atmospheric Science -Classifier: Topic :: Scientific/Engineering :: Information Analysis -Classifier: Topic :: Scientific/Engineering :: Medical Science Apps. -Classifier: Topic :: System :: Distributed Computing -Classifier: Topic :: Utilities -Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, <4 -Description-Content-Type: text/x-rst -Requires-Dist: setuptools -Requires-Dist: requests (>=2.6.1) -Requires-Dist: ruamel.yaml (<=0.16,>=0.12.4) -Requires-Dist: rdflib (<4.3.0,>=4.2.2) -Requires-Dist: shellescape (<3.5,>=3.4.1) -Requires-Dist: schema-salad (<5,>=4.5) -Requires-Dist: mypy-extensions -Requires-Dist: six (>=1.9.0) -Requires-Dist: psutil -Requires-Dist: scandir -Requires-Dist: prov (==1.5.1) -Requires-Dist: bagit (>=1.6.4) -Requires-Dist: typing-extensions -Requires-Dist: coloredlogs -Requires-Dist: future (>=0.16) -Requires-Dist: pathlib2 (!=2.3.1) -Requires-Dist: subprocess32 (>=3.5.0) ; os.name=="posix" and python_version<"3.5" -Requires-Dist: typing (>=3.5.3) ; python_version<"3.6" -Provides-Extra: deps -Requires-Dist: galaxy-tool-util ; extra == 'deps' -Provides-Extra: docs -Requires-Dist: sphinx (>=2.2) ; extra == 'docs' -Requires-Dist: sphinx-rtd-theme ; extra == 'docs' - -================================================================== -Common Workflow Language tool description reference implementation -================================================================== - -CWL conformance tests: |Conformance Status| |Linux Status| |Windows Status| |Coverage Status| |Downloads| - -|CommandLineTool Support| |DockerRequirement Support| |EnvVarRequirement Support| |ExpressionTool Support| -|InitialWorkDirRequirement Support| |InlineJavascriptRequirement Support| |MultipleInputRequirement Support| |Core Support| -|ResourceRequirement Support| |ScatterRequirement Support| |SchemaDefRequirement Support| |ShellCommandequirement Support| -|StepInputRequirement Support| |SubWorkflowRequirement Support| |Workflow Support| - -.. |Conformance Status| image:: https://ci.commonwl.org/buildStatus/icon?job=cwltool-conformance - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |Linux Status| image:: https://img.shields.io/travis/common-workflow-language/cwltool/master.svg?label=Linux%20builds - :target: https://travis-ci.org/common-workflow-language/cwltool - -.. |Windows Status| image:: https://img.shields.io/appveyor/ci/mr-c/cwltool/master.svg?label=Windows%20builds - :target: https://ci.appveyor.com/project/mr-c/cwltool - -.. |Coverage Status| image:: https://img.shields.io/codecov/c/github/common-workflow-language/cwltool.svg - :target: https://codecov.io/gh/common-workflow-language/cwltool - -.. |Downloads| image:: https://pepy.tech/badge/cwltool/month - :target: https://pepy.tech/project/cwltool - -.. |CommandLineTool Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/command_line_tool.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |DockerRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/docker.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |EnvVarRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/env_var.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |ExpressionTool Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/expression_tool.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |InitialWorkDirRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/initial_work_dir.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |InlineJavascriptRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/inline_javascript.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |MultipleInputRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/multiple_input.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |Core Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/required.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |ResourceRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/resource.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |ScatterRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/scatter.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |SchemaDefRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/schema_def.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |ShellCommandequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/shell_command.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |StepInputRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/step_input.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |SubWorkflowRequirement Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/subworkflow.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - -.. |Workflow Support| image:: https://flat.badgen.net/https/raw.githubusercontent.com/common-workflow-language/conformance/master/cwltool/cwl_v1.0/cwltool_latest/workflow.json - :target: https://ci.commonwl.org/job/cwltool-conformance/ - - -This is the reference implementation of the Common Workflow Language. It is -intended to be feature complete and provide comprehensive validation of CWL -files as well as provide other tools related to working with CWL. - -This is written and tested for -`Python <https://www.python.org/>`_ ``2.7 and 3.x {x = 5, 6, 7, 8}`` - -The reference implementation consists of two packages. The ``cwltool`` package -is the primary Python module containing the reference implementation in the -``cwltool`` module and console executable by the same name. - -The ``cwlref-runner`` package is optional and provides an additional entry point -under the alias ``cwl-runner``, which is the implementation-agnostic name for the -default CWL interpreter installed on a host. - -Install -------- - -Your operating system may offer cwltool directly. For [Debian](https://tracker.debian.org/pkg/cwltool "Debian cwltool package tracker") or [Ubuntu](https://launchpad.net/ubuntu/+source/cwltool "Ubuntu Launchpad overview for cwltool") try - -.. code:: bash - - apt-get install cwltool - -Otherwise, to -avoid conflicting versions of the same library, -it is recommended to do the following: - -.. code:: bash - - virtualenv -p python2 venv # Create a virtual environment, can use `python3` as well - source venv/bin/activate # Activate environment before installing `cwltool` - -Installing the official package from PyPi (will install "cwltool" package as -well) - -.. code:: bash - - pip install cwlref-runner - -If installing alongside another CWL implementation then - -.. code:: bash - - pip install cwltool - -Or you can install from source: - -.. code:: bash - - git clone https://github.com/common-workflow-language/cwltool.git # clone cwltool repo - cd cwltool # Switch to source directory - pip install . # Install `cwltool` from source - cwltool --version # Check if the installation works correctly - -Remember, if co-installing multiple CWL implementations then you need to -maintain which implementation ``cwl-runner`` points to via a symbolic file -system link or `another facility <https://wiki.debian.org/DebianAlternatives>`_. - -===== -Usage -===== - -Run on the command line ------------------------ - -Simple command:: - - cwl-runner [tool-or-workflow-description] [input-job-settings] - -Or if you have multiple CWL implementations installed and you want to override -the default cwl-runner use:: - - cwltool [tool-or-workflow-description] [input-job-settings] - -You can set cwltool options in the environment with CWLTOOL_OPTIONS, -these will be inserted at the beginning of the command line:: - - export CWLTOOL_OPTIONS="--debug" - -Use with boot2docker --------------------- -boot2docker runs Docker inside a virtual machine and it only mounts ``Users`` -on it. The default behavior of CWL is to create temporary directories under e.g. -``/Var`` which is not accessible to Docker containers. - -To run CWL successfully with boot2docker you need to set the ``--tmpdir-prefix`` -and ``--tmp-outdir-prefix`` to somewhere under ``/Users``:: - - $ cwl-runner --tmp-outdir-prefix=/Users/username/project --tmpdir-prefix=/Users/username/project wc-tool.cwl wc-job.json - -Using user-space replacements for Docker ----------------------------------------- - -Some shared computing environments don't support Docker software containers for technical or policy reasons. -As a work around, the CWL reference runner supports using alternative ``docker`` implementations on Linux -with the ``--user-space-docker-cmd`` option. - -One such "user space" friendly docker replacement is ``udocker`` https://github.com/indigo-dc/udocker and another -is ``dx-docker`` https://wiki.dnanexus.com/Developer-Tutorials/Using-Docker-Images - -udocker installation: https://github.com/indigo-dc/udocker/blob/master/doc/installation_manual.md#22-install-from-indigo-datacloud-repositories - -dx-docker installation: start with the DNAnexus toolkit (see https://wiki.dnanexus.com/Downloads for instructions). - -Run `cwltool` just as you normally would, but with the new option, e.g. from the conformance tests: - -.. code:: bash - - cwltool --user-space-docker-cmd=udocker https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/test-cwl-out2.cwl https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/empty.json - -or - -.. code:: bash - - cwltool --user-space-docker-cmd=dx-docker https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/test-cwl-out2.cwl https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/empty.json - -``cwltool`` can use `Singularity <http://singularity.lbl.gov/>`_ version 2.6.1 -or later as a Docker container runtime. -``cwltool`` with Singularity will run software containers specified in -``DockerRequirement`` and therefore works with Docker images only, native -Singularity images are not supported. To use Singularity as the Docker container -runtime, provide ``--singularity`` command line option to ``cwltool``. -With Singularity, ``cwltool`` can pass all CWL v1.0 conformance tests, except -those involving Docker container ENTRYPOINTs. - - -.. code:: bash - - cwltool --singularity https://raw.githubusercontent.com/common-workflow-language/common-workflow-language/master/v1.0/v1.0/v1.0/cat3-tool-mediumcut.cwl https://github.com/common-workflow-language/common-workflow-language/blob/master/v1.0/v1.0/cat-job.json - -Running a tool or workflow from remote or local locations ---------------------------------------------------------- - -``cwltool`` can run tool and workflow descriptions on both local and remote -systems via its support for HTTP[S] URLs. - -Input job files and Workflow steps (via the `run` directive) can reference CWL -documents using absolute or relative local filesytem paths. If a relative path -is referenced and that document isn't found in the current directory then the -following locations will be searched: -http://www.commonwl.org/v1.0/CommandLineTool.html#Discovering_CWL_documents_on_a_local_filesystem - -You can also use `cwldep <https://github.com/common-workflow-language/cwldep>` -to manage dependencies on external tools and workflows. - -Overriding workflow requirements at load time ---------------------------------------------- - -Sometimes a workflow needs additional requirements to run in a particular -environment or with a particular dataset. To avoid the need to modify the -underlying workflow, cwltool supports requirement "overrides". - -The format of the "overrides" object is a mapping of item identifier (workflow, -workflow step, or command line tool) to the process requirements that should be applied. - -.. code:: yaml - - cwltool:overrides: - echo.cwl: - requirements: - EnvVarRequirement: - envDef: - MESSAGE: override_value - -Overrides can be specified either on the command line, or as part of the job -input document. Workflow steps are identified using the name of the workflow -file followed by the step name as a document fragment identifier "#id". -Override identifiers are relative to the toplevel workflow document. - -.. code:: bash - - cwltool --overrides overrides.yml my-tool.cwl my-job.yml - -.. code:: yaml - - input_parameter1: value1 - input_parameter2: value2 - cwltool:overrides: - workflow.cwl#step1: - requirements: - EnvVarRequirement: - envDef: - MESSAGE: override_value - -.. code:: bash - - cwltool my-tool.cwl my-job-with-overrides.yml - - -Combining parts of a workflow into a single document ----------------------------------------------------- - -Use ``--pack`` to combine a workflow made up of multiple files into a -single compound document. This operation takes all the CWL files -referenced by a workflow and builds a new CWL document with all -Process objects (CommandLineTool and Workflow) in a list in the -``$graph`` field. Cross references (such as ``run:`` and ``source:`` -fields) are updated to internal references within the new packed -document. The top level workflow is named ``#main``. - -.. code:: bash - - cwltool --pack my-wf.cwl > my-packed-wf.cwl - - -Running only part of a workflow -------------------------------- - -You can run a partial workflow with the ``--target`` (``-t``) option. This -takes the name of an output parameter, workflow step, or input -parameter in the top level workflow. You may provide multiple -targets. - -.. code:: bash - - cwltool --target step3 my-wf.cwl - -If a target is an output parameter, it will only run only the steps -that contribute to that output. If a target is a workflow step, it -will run the workflow starting from that step. If a target is an -input parameter, it will only run only the steps that are connected to -that input. - -Use ``--print-targets`` to get a listing of the targets of a workflow. -To see exactly which steps will run, use ``--print-subgraph`` with -``--target`` to get a printout of the workflow subgraph for the -selected targets. - -.. code:: bash - - cwltool --print-targets my-wf.cwl - - cwltool --target step3 --print-subgraph my-wf.cwl > my-wf-starting-from-step3.cwl - - -Visualizing a CWL document --------------------------- - -The ``--print-dot`` option will print a file suitable for Graphviz ``dot`` program. Here is a bash onliner to generate a Scalable Vector Graphic (SVG) file: - -.. code:: bash - - cwltool --print-dot my-wf.cwl | dot -Tsvg > my-wf.svg - -Modeling a CWL document as RDF ------------------------------- - -CWL documents can be expressed as RDF triple graphs. - -.. code:: bash - - cwltool --print-rdf --rdf-serializer=turtle mywf.cwl - - -Leveraging SoftwareRequirements (Beta) --------------------------------------- - -CWL tools may be decorated with ``SoftwareRequirement`` hints that cwltool -may in turn use to resolve to packages in various package managers or -dependency management systems such as `Environment Modules -<http://modules.sourceforge.net/>`__. - -Utilizing ``SoftwareRequirement`` hints using cwltool requires an optional -dependency, for this reason be sure to use specify the ``deps`` modifier when -installing cwltool. For instance:: - - $ pip install 'cwltool[deps]' - -Installing cwltool in this fashion enables several new command line options. -The most general of these options is ``--beta-dependency-resolvers-configuration``. -This option allows one to specify a dependency resolver's configuration file. -This file may be specified as either XML or YAML and very simply describes various -plugins to enable to "resolve" ``SoftwareRequirement`` dependencies. - -To discuss some of these plugins and how to configure them, first consider the -following ``hint`` definition for an example CWL tool. - -.. code:: yaml - - SoftwareRequirement: - packages: - - package: seqtk - version: - - r93 - -Now imagine deploying cwltool on a cluster with Software Modules installed -and that a ``seqtk`` module is available at version ``r93``. This means cluster -users likely won't have the binary ``seqtk`` on their ``PATH`` by default, but after -sourcing this module with the command ``modulecmd sh load seqtk/r93`` ``seqtk`` is -available on the ``PATH``. A simple dependency resolvers configuration file, called -``dependency-resolvers-conf.yml`` for instance, that would enable cwltool to source -the correct module environment before executing the above tool would simply be: - -.. code:: yaml - - - type: modules - -The outer list indicates that one plugin is being enabled, the plugin parameters are -defined as a dictionary for this one list item. There is only one required parameter -for the plugin above, this is ``type`` and defines the plugin type. This parameter -is required for all plugins. The available plugins and the parameters -available for each are documented (incompletely) `here -<https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__. -Unfortunately, this documentation is in the context of Galaxy tool -``requirement`` s instead of CWL ``SoftwareRequirement`` s, but the concepts map fairly directly. - -cwltool is distributed with an example of such seqtk tool and sample corresponding -job. It could executed from the cwltool root using a dependency resolvers -configuration file such as the above one using the command:: - - cwltool --beta-dependency-resolvers-configuration /path/to/dependency-resolvers-conf.yml \ - tests/seqtk_seq.cwl \ - tests/seqtk_seq_job.json - -This example demonstrates both that cwltool can leverage -existing software installations and also handle workflows with dependencies -on different versions of the same software and libraries. However the above -example does require an existing module setup so it is impossible to test this example -"out of the box" with cwltool. For a more isolated test that demonstrates all -the same concepts - the resolver plugin type ``galaxy_packages`` can be used. - -"Galaxy packages" are a lighter weight alternative to Environment Modules that are -really just defined by a way to lay out directories into packages and versions -to find little scripts that are sourced to modify the environment. They have -been used for years in Galaxy community to adapt Galaxy tools to cluster -environments but require neither knowledge of Galaxy nor any special tools to -setup. These should work just fine for CWL tools. - -The cwltool source code repository's test directory is setup with a very simple -directory that defines a set of "Galaxy packages" (but really just defines one -package named ``random-lines``). The directory layout is simply:: - - tests/test_deps_env/ - random-lines/ - 1.0/ - env.sh - -If the ``galaxy_packages`` plugin is enabled and pointed at the -``tests/test_deps_env`` directory in cwltool's root and a ``SoftwareRequirement`` -such as the following is encountered. - -.. code:: yaml - - hints: - SoftwareRequirement: - packages: - - package: 'random-lines' - version: - - '1.0' - -Then cwltool will simply find that ``env.sh`` file and source it before executing -the corresponding tool. That ``env.sh`` script is only responsible for modifying -the job's ``PATH`` to add the required binaries. - -This is a full example that works since resolving "Galaxy packages" has no -external requirements. Try it out by executing the following command from cwltool's -root directory:: - - cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf.yml \ - tests/random_lines.cwl \ - tests/random_lines_job.json - -The resolvers configuration file in the above example was simply: - -.. code:: yaml - - - type: galaxy_packages - base_path: ./tests/test_deps_env - -It is possible that the ``SoftwareRequirement`` s in a given CWL tool will not -match the module names for a given cluster. Such requirements can be re-mapped -to specific deployed packages and/or versions using another file specified using -the resolver plugin parameter `mapping_files`. We will -demonstrate this using `galaxy_packages` but the concepts apply equally well -to Environment Modules or Conda packages (described below) for instance. - -So consider the resolvers configuration file -(`tests/test_deps_env_resolvers_conf_rewrite.yml`): - -.. code:: yaml - - - type: galaxy_packages - base_path: ./tests/test_deps_env - mapping_files: ./tests/test_deps_mapping.yml - -And the corresponding mapping configuraiton file (`tests/test_deps_mapping.yml`): - -.. code:: yaml - - - from: - name: randomLines - version: 1.0.0-rc1 - to: - name: random-lines - version: '1.0' - -This is saying if cwltool encounters a requirement of ``randomLines`` at version -``1.0.0-rc1`` in a tool, to rewrite to our specific plugin as ``random-lines`` at -version ``1.0``. cwltool has such a test tool called ``random_lines_mapping.cwl`` -that contains such a source ``SoftwareRequirement``. To try out this example with -mapping, execute the following command from the cwltool root directory:: - - cwltool --beta-dependency-resolvers-configuration tests/test_deps_env_resolvers_conf_rewrite.yml \ - tests/random_lines_mapping.cwl \ - tests/random_lines_job.json - -The previous examples demonstrated leveraging existing infrastructure to -provide requirements for CWL tools. If instead a real package manager is used -cwltool has the opportunity to install requirements as needed. While initial -support for Homebrew/Linuxbrew plugins is available, the most developed such -plugin is for the `Conda <https://conda.io/docs/#>`__ package manager. Conda has the nice properties -of allowing multiple versions of a package to be installed simultaneously, -not requiring evaluated permissions to install Conda itself or packages using -Conda, and being cross platform. For these reasons, cwltool may run as a normal -user, install its own Conda environment and manage multiple versions of Conda packages -on both Linux and Mac OS X. - -The Conda plugin can be endlessly configured, but a sensible set of defaults -that has proven a powerful stack for dependency management within the Galaxy tool -development ecosystem can be enabled by simply passing cwltool the -``--beta-conda-dependencies`` flag. - -With this we can use the seqtk example above without Docker and without -any externally managed services - cwltool should install everything it needs -and create an environment for the tool. Try it out with the follwing command:: - - cwltool --beta-conda-dependencies tests/seqtk_seq.cwl tests/seqtk_seq_job.json - -The CWL specification allows URIs to be attached to ``SoftwareRequirement`` s -that allow disambiguation of package names. If the mapping files described above -allow deployers to adapt tools to their infrastructure, this mechanism allows -tools to adapt their requirements to multiple package managers. To demonstrate -this within the context of the seqtk, we can simply break the package name we -use and then specify a specific Conda package as follows: - -.. code:: yaml - - hints: - SoftwareRequirement: - packages: - - package: seqtk_seq - version: - - '1.2' - specs: - - https://anaconda.org/bioconda/seqtk - - https://packages.debian.org/sid/seqtk - -The example can be executed using the command:: - - cwltool --beta-conda-dependencies tests/seqtk_seq_wrong_name.cwl tests/seqtk_seq_job.json - -The plugin framework for managing resolution of these software requirements -as maintained as part of `galaxy-tool-util <https://github.com/galaxyproject/galaxy/tree/dev/packages/tool_util>`__ - a small, -portable subset of the Galaxy project. More information on configuration and implementation can be found -at the following links: - -- `Dependency Resolvers in Galaxy <https://docs.galaxyproject.org/en/latest/admin/dependency_resolvers.html>`__ -- `Conda for [Galaxy] Tool Dependencies <https://docs.galaxyproject.org/en/latest/admin/conda_faq.html>`__ -- `Mapping Files - Implementation <https://github.com/galaxyproject/galaxy/commit/495802d229967771df5b64a2f79b88a0eaf00edb>`__ -- `Specifications - Implementation <https://github.com/galaxyproject/galaxy/commit/81d71d2e740ee07754785306e4448f8425f890bc>`__ -- `Initial cwltool Integration Pull Request <https://github.com/common-workflow-language/cwltool/pull/214>`__ - -Use with GA4GH Tool Registry API --------------------------------- - -Cwltool can launch tools directly from `GA4GH Tool Registry API`_ endpoints. - -By default, cwltool searches https://dockstore.org/ . Use ``--add-tool-registry`` to add other registries to the search path. - -For example :: - - cwltool quay.io/collaboratory/dockstore-tool-bamstats:develop test.json - -and (defaults to latest when a version is not specified) :: - - cwltool quay.io/collaboratory/dockstore-tool-bamstats test.json - -For this example, grab the test.json (and input file) from https://github.com/CancerCollaboratory/dockstore-tool-bamstats :: - - wget https://dockstore.org/api/api/ga4gh/v2/tools/quay.io%2Fbriandoconnor%2Fdockstore-tool-bamstats/versions/develop/PLAIN-CWL/descriptor/test.json - wget https://github.com/CancerCollaboratory/dockstore-tool-bamstats/raw/develop/rna.SRR948778.bam - - -.. _`GA4GH Tool Registry API`: https://github.com/ga4gh/tool-registry-schemas - -=========== -Development -=========== - -Running tests locally ---------------------- - -- Running basic tests ``(/tests)``: - -To run the basic tests after installing `cwltool` execute the following: - -.. code:: bash - - pip install -rtest-requirements.txt - py.test --ignore cwltool/schemas/ --pyarg cwltool - -To run various tests in all supported Python environments we use `tox <https://github.com/common-workflow-language/cwltool/tree/master/tox.ini>`_. To run the test suite in all supported Python environments -first downloading the complete code repository (see the ``git clone`` instructions above) and then run -the following in the terminal: -``pip install tox; tox`` - -List of all environment can be seen using: -``tox --listenvs`` -and running a specfic test env using: -``tox -e <env name>`` -and additionally run a specific test using this format: -``tox -e py36-unit -- tests/test_examples.py::TestParamMatching`` - -- Running the entire suite of CWL conformance tests: - -The GitHub repository for the CWL specifications contains a script that tests a CWL -implementation against a wide array of valid CWL files using the `cwltest <https://github.com/common-workflow-language/cwltest>`_ -program - -Instructions for running these tests can be found in the Common Workflow Language Specification repository at https://github.com/common-workflow-language/common-workflow-language/blob/master/CONFORMANCE_TESTS.md - -Import as a module ------------------- - -Add - -.. code:: python - - import cwltool - -to your script. - -The easiest way to use cwltool to run a tool or workflow from Python is to use a Factory - -.. code:: python - - import cwltool.factory - fac = cwltool.factory.Factory() - - echo = fac.make("echo.cwl") - result = echo(inp="foo") - - # result["out"] == "foo" - - -CWL Tool Control Flow ---------------------- - -Technical outline of how cwltool works internally, for maintainers. - -#. Use CWL ``load_tool()`` to load document. - - #. Fetches the document from file or URL - #. Applies preprocessing (syntax/identifier expansion and normalization) - #. Validates the document based on cwlVersion - #. If necessary, updates the document to latest spec - #. Constructs a Process object using ``make_tool()``` callback. This yields a - CommandLineTool, Workflow, or ExpressionTool. For workflows, this - recursively constructs each workflow step. - #. To construct custom types for CommandLineTool, Workflow, or - ExpressionTool, provide a custom ``make_tool()`` - -#. Iterate on the ``job()`` method of the Process object to get back runnable jobs. - - #. ``job()`` is a generator method (uses the Python iterator protocol) - #. Each time the ``job()`` method is invoked in an iteration, it returns one - of: a runnable item (an object with a ``run()`` method), ``None`` (indicating - there is currently no work ready to run) or end of iteration (indicating - the process is complete.) - #. Invoke the runnable item by calling ``run()``. This runs the tool and gets output. - #. Output of a process is reported by an output callback. - #. ``job()`` may be iterated over multiple times. It will yield all the work - that is currently ready to run and then yield None. - -#. ``Workflow`` objects create a corresponding ``WorkflowJob`` and ``WorkflowJobStep`` objects to hold the workflow state for the duration of the job invocation. - - #. The WorkflowJob iterates over each WorkflowJobStep and determines if the - inputs the step are ready. - #. When a step is ready, it constructs an input object for that step and - iterates on the ``job()`` method of the workflow job step. - #. Each runnable item is yielded back up to top level run loop - #. When a step job completes and receives an output callback, the - job outputs are assigned to the output of the workflow step. - #. When all steps are complete, the intermediate files are moved to a final - workflow output, intermediate directories are deleted, and the output - callback for the workflow is called. - -#. ``CommandLineTool`` job() objects yield a single runnable object. - - #. The CommandLineTool ``job()`` method calls ``make_job_runner()`` to create a - ``CommandLineJob`` object - #. The job method configures the CommandLineJob object by setting public - attributes - #. The job method iterates over file and directories inputs to the - CommandLineTool and creates a "path map". - #. Files are mapped from their "resolved" location to a "target" path where - they will appear at tool invocation (for example, a location inside a - Docker container.) The target paths are used on the command line. - #. Files are staged to targets paths using either Docker volume binds (when - using containers) or symlinks (if not). This staging step enables files - to be logically rearranged or renamed independent of their source layout. - #. The ``run()`` method of CommandLineJob executes the command line tool or - Docker container, waits for it to complete, collects output, and makes - the output callback. - - -Extension points ----------------- - -The following functions can be passed to main() to override or augment -the listed behaviors. - -executor - :: - - executor(tool, job_order_object, runtimeContext, logger) - (Process, Dict[Text, Any], RuntimeContext) -> Tuple[Dict[Text, Any], Text] - - An implementation of the toplevel workflow execution loop, should - synchronously run a process object to completion and return the - output object. - -versionfunc - :: - - () - () -> Text - - Return version string. - -logger_handler - :: - - logger_handler - logging.Handler - - Handler object for logging. - -The following functions can be set in LoadingContext to override or -augment the listed behaviors. - -fetcher_constructor - :: - - fetcher_constructor(cache, session) - (Dict[unicode, unicode], requests.sessions.Session) -> Fetcher - - Construct a Fetcher object with the supplied cache and HTTP session. - -resolver - :: - - resolver(document_loader, document) - (Loader, Union[Text, dict[Text, Any]]) -> Text - - Resolve a relative document identifier to an absolute one which can be fetched. - -The following functions can be set in RuntimeContext to override or -augment the listed behaviors. - -construct_tool_object - :: - - construct_tool_object(toolpath_object, loadingContext) - (MutableMapping[Text, Any], LoadingContext) -> Process - - Hook to construct a Process object (eg CommandLineTool) object from a document. - -select_resources - :: - - selectResources(request) - (Dict[str, int], RuntimeContext) -> Dict[Text, int] - - Take a resource request and turn it into a concrete resource assignment. - -make_fs_access - :: - - make_fs_access(basedir) - (Text) -> StdFsAccess - - Return a file system access object. - -In addition, when providing custom subclasses of Process objects, you can override the following methods: - -CommandLineTool.make_job_runner - :: - - make_job_runner(RuntimeContext) - (RuntimeContext) -> Type[JobBase] - - Create and return a job runner object (this implements concrete execution of a command line tool). - -Workflow.make_workflow_step - :: - - make_workflow_step(toolpath_object, pos, loadingContext, parentworkflowProv) - (Dict[Text, Any], int, LoadingContext, Optional[ProvenanceProfile]) -> WorkflowStep - - Create and return a workflow step object. - -