Mercurial > repos > fubar > toolfactory_gtn

annotate toolfactory/README.md @ 5:e2c8c2fa192d draft

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Uploaded
author fubar
date Tue, 27 Apr 2021 23:33:49 +0000
parents 2a46da701dde
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
5
e2c8c2fa192d Uploaded
fubar
parents: 4
diff changeset
1 ## Breaking news! Docker container at https://github.com/fubar2/toolfactory-galaxy-docker recommended as at December 2020
0
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
2
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
3 ### New demonstration of planemo tool_factory command ![Planemo ToolFactory demonstration](images/lintplanemo-2021-01-08_18.02.45.mkv?raw=false "Demonstration inside Planemo")
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
4
5
e2c8c2fa192d Uploaded
fubar
parents: 4
diff changeset
5 ## This is the original ToolFactory suitable for non-docker situations. Please use the docker container if you can because it's integrated with a Toolshed...
e2c8c2fa192d Uploaded
fubar
parents: 4
diff changeset
6
0
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
7 # WARNING
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
8
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
9 Install this tool to a throw-away private Galaxy or Docker container ONLY!
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
10
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
11 Please NEVER on a public or production instance where a hostile user may
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
12 be able to gain access if they can acquire an administrative account login.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
13
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
14 It only runs for server administrators - the ToolFactory tool will refuse to execute for an ordinary user since
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
15 it can install new tools to the Galaxy server it executes on! This is not something you should allow other than
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
16 on a throw away instance that is protected from potentially hostile users.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
17
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
18 ## Short Story
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
19
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
20 Galaxy is easily extended to new applications by adding a new tool. Each new scientific computational package added as
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
21 a tool to Galaxy requires an XML document describing how the application interacts with Galaxy.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
22 This is sometimes termed "wrapping" the package because the instructions tell Galaxy how to run the package
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
23 as a new Galaxy tool. Any tool that has been wrapped is readily available to all the users through a consistent
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
24 and easy to use interface once installed in the local Galaxy server.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
25
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
26 Most Galaxy tool wrappers have been manually prepared by skilled programmers, many using Planemo because it
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
27 automates much of the boilerplate and makes the process much easier.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
28 The ToolFactory (TF) now uses Planemo under the hood for testing, but hides the command
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
29 line complexities. The user will still need appropriate skills in terms of describing the interface between
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
30 Galaxy and the new application, but will be helped by a Galaxy tool form to collect all the needed
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
31 settings, together with automated testing and uploading to a toolshed with optional local installation.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
32
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
33
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
34 ## ToolFactory generated tools are ordinary Galaxy tools
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
35
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
36 A TF generated tool that passes the Planemo test is ready to publish in any Galaxy Toolshed and ready to install in any running Galaxy instance.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
37 They are fully workflow compatible and work exactly like any hand-written tool. The user can select input files of the specified type(s) from their
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
38 history and edit each of the specified parameters. The tool form will show all the labels and help text supplied when the tool was built. When the tool
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
39 is executed, the dependent binary or script will be passed all the i/o files and parameters as specified, and will write outputs to the specified new
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
40 history datasets - just like any other Galaxy tool.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
41
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
42 ## Models for tool command line construction
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
43
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
44 The key to turning any software package into a Galaxy tool is the automated construction of a suitable command line.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
45
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
46 The TF can build a new tool that will allow the tool user to select input files from their history, set any parameters and when run will send the
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
47 new output files to the history as specified when the tool builder completed the form and built the new tool.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
48
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
49 That tool can contain instructions to run any Conda dependency or a system executable like bash. Whether a bash script you have written or
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
50 a Conda package like bwa, the executable will expect to find settings for input, output and parameters on a command line.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
51
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
52 These are often passed as "--name value" (argparse style) or in a fixed order (positional style).
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
53
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
54 The ToolFactory allows either, or for "filter" applications that process input from STDIN and write processed output to STDOUT.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
55
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
56 The simplest tool model wraps a simple script or Conda dependency package requiring only input and output files, with no user supplied settings illustrated by
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
57 the Tacrev demonstration tool found in the Galaxy running in the ToolFactory docker container. It passes a user selected input file from the current history on STDIN
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
58 to a bash script. The bash script runs the unix tac utility (reverse cat) piped to the unix rev (reverse lines in a text file) utility. It's a one liner:
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
59
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
60 `tac | rev`
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
61
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
62 The tool building form allows zero or more Conda package name(s) and version(s) and an optional script to be executed by either a system
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
63 executable like ``bash`` or the first of any named Conda dependency package/version. Tacrev uses a tiny bash script shown above and uses the system
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
64 bash. Conda bash can be specified if it is important to use the same version consistently for the tool.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
65
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
66 On the tool form, the repeat section allowing zero or more input files was set to be a text file to be selected by the tool user and
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
67 in the repeat section allowing one or more outputs, a new output file with special value `STDOUT` as the positional parameter, causes the TF to
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
68 generate a command to capture STDOUT and send it to the new history file containing the reversed input text.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
69
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
70 By reversed, we mean really, truly reversed.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
71
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
72 That simple model can be made much more complicated, and can pass inputs and outputs as named or positional parameters,
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
73 to allow more complicated scripts or dependent binaries that require:
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
74
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
75 1. Any number of input data files selected by the user from existing history data
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
76 2. Any number of output data files written to the user's history
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
77 3. Any number of user supplied parameters. These can be passed as command line arguments to the script or the dependency package. Either
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
78 positional or named (argparse) style command line parameter passing can be used.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
79
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
80 More complex models can be seen in the Sedtest, Pyrevpos and Pyrevargparse tools illustrating positional and argparse parameter passing.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
81
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
82 The most complex demonstration is the Planemo advanced tool tutorial BWA tool. There is one version using a command-override to implement
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
83 exactly the same command structure in the Planemo tutorial. A second version uses a bash script and positional parameters to achieve the same
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
84 result. Some tool builders may find the bash version more familiar and cleaner but the choice is yours.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
85
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
86 ## Overview
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
87
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
88 ![IHello example ToolFactory tool form](files/hello_toolfactory_form.png?raw=true "Part of the Hello world example ToolFactory tool form")
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
89
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
90
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
91 Steps in building a new Galaxy tool are all conducted through Galaxy running in the docker container:
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
92
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
93 1. Login to the Galaxy running in the container at http://localhost:8080 using an admin account. They are specified in config/galaxy.yml and
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
94 in the documentation at
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
95 and the ToolFactory will error out and refuse to run for non-administrative tool builders as a minimal protection from opportunistic hostile use.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
96
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
97 2. Start the TF and fill in the form, providing sample inputs and parameter values to suit the Conda package being wrapped.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
98
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
99 3. Execute the tool to create a new XML tool wrapper using the sample inputs and parameter settings for the inbuilt tool test. Planemo runs twice.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
100 firstly to generate the test outputs and then to perform a proper test. The completed toolshed archive is written to the history
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
101 together with the planemo test report. Optionally the new tool archive can be uploaded
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
102 to the toolshed running in the same container (http://localhost:9009) and then installed inside the Galaxy in the container for further testing.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
103
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
104 4. If the test fails, rerun the failed history job and correct errors on the tool form before rerunning until everything works correctly.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
105
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
106 ![How it works](files/TFasIDE.png?raw=true "Overview of the ToolFactory as an Integrated Development Environment")
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
107
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
108 ## Planning and building new Galaxy tool wrappers.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
109
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
110 It is best to have all the required planning done to wrap any new script or binary before firing up the TF.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
111 Conda is the only current dependency manager supported. Before starting, at the very least, the tool builder will need
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
112 to know the required software package name in Conda and the version to use, how the command line for
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
113 the package must be constructed, and there must be sample inputs in the working history for each of the required data inputs
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
114 for the package, together with values for every parameter to suit these sample inputs. These are required on the TF form
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
115 for preparing the inbuilt tool test. That test is run using Planemo, as part of the tool generation process.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
116
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
117 A new tool is specified by filling in the usual Galaxy tool form.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
118
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
119 The form starts with a new tool name. Most tools will need dependency packages and versions
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
120 for the executable. Only Conda is currently supported.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
121
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
122 If a script is needed, it can be pasted into a text box and the interpreter named. Available system executables
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
123 can be used such as bash, or an interpreter such as python, perl or R can be nominated as conda dependencies
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
124 to ensure reproducible analyses.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
125
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
126 The tool form will be generated from the input data and the tool builder supplied parameters. The command line for the
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
127 executable is built using positional or argparse (named e.g. --input_file /foo/baz) style
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
128 parameters and is completely dependent on the executable. These can include:
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
129
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
130 1. Any number of input data sets needed by the executable. Each appears to the tool user on the run form and is included
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
131 on the command line for the executable. The tool builder must supply a small representative sample for each one as
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
132 an input for the automated tool test.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
133
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
134 2. Any number of output data sets generated by the package can be added to the command line and will appear in
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
135 the user's history at the end of the job
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
136
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
137 3. Any number of text or numeric parameters. Each will appear to the tool user on the run form and are included
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
138 on the command line to the executable. The tool builder must supply a suitable representative value for each one as
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
139 the value to be used for the automated tool test.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
140
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
141 Once the form is completed, executing the TF will build a new XML tool wrapper
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
142 including a functional test based on the sample settings and data.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
143
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
144 If the Planemo test passes, the tool can be optionally uploaded to the local Galaxy used in the image for more testing.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
145
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
146 A local toolshed runs inside the container to allow an automated installation, although any toolshed and any accessible
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
147 Galaxy can be specified for this process by editing the default URL and API keys to provide appropriate credentials.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
148
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
149 ## Generated Tool Dependency management
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
150
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
151 Conda is used for all dependency management although tools that use system utilities like sed, bash or awk
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
152 may be available on job execution nodes. Sed and friends are available as Conda (conda-forge) dependencies if necessary.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
153 Versioned Conda dependencies are always baked-in to the tool and will be used for reproducible calculation.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
154
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
155 ## Requirements
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
156
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
157 These are all managed automagically. The TF relies on galaxyxml to generate tool xml and uses ephemeris and
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
158 bioblend to load tools to the toolshed and to Galaxy. Planemo is used for testing and runs in a biocontainer currently at
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
159 https://quay.io/fubar2/planemo-biocontainer
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
160
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
161 ## Caveats
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
162
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
163 This docker image requires privileged mode so exposes potential security risks if hostile tool builders gain access.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
164 Please, do not run it in any situation where that is a problem - never, ever on a public facing Galaxy server.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
165 On a laptop or workstation should be fine in a non-hostile environment.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
166
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
167
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
168 ## Example generated XML
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
169
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
170 For the bwa-mem example, a supplied bash script is included as a configfile and so has escaped characters.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
171 ```
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
172 <tool name="bwatest" id="bwatest" version="0.01">
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
173 <!--Cite: Creating re-usable tools from scripts doi:10.1093/bioinformatics/bts573-->
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
174 <!--Source in git at: https://github.com/fubar2/toolfactory-->
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
175 <!--Created by admin@galaxy.org at 30/11/2020 07:12:10 using the Galaxy Tool Factory.-->
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
176 <description>Planemo advanced tool building sample bwa mem mapper as a ToolFactory demo</description>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
177 <requirements>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
178 <requirement version="0.7.15" type="package">bwa</requirement>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
179 <requirement version="1.3" type="package">samtools</requirement>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
180 </requirements>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
181 <configfiles>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
182 <configfile name="runme"><![CDATA[
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
183 REFFILE=\$1
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
184 FASTQ=\$2
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
185 BAMOUT=\$3
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
186 rm -f "refalias"
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
187 ln -s "\$REFFILE" "refalias"
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
188 bwa index -a is "refalias"
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
189 bwa mem -t "2" -v 1 "refalias" "\$FASTQ" > tempsam
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
190 samtools view -Sb tempsam > temporary_bam_file.bam
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
191 samtools sort -o "\$BAMOUT" temporary_bam_file.bam
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
192
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
193 ]]></configfile>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
194 </configfiles>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
195 <version_command/>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
196 <command><![CDATA[bash
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
197 $runme
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
198 $input1
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
199 $input2
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
200 $bam_output]]></command>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
201 <inputs>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
202 <param optional="false" label="Reference sequence for bwa to map the fastq reads against" help="" format="fasta" multiple="false" type="data" name="input1" argument="input1"/>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
203 <param optional="false" label="Reads as fastqsanger to align to the reference sequence" help="" format="fastqsanger" multiple="false" type="data" name="input2" argument="input2"/>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
204 </inputs>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
205 <outputs>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
206 <data name="bam_output" format="bam" label="bam_output" hidden="false"/>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
207 </outputs>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
208 <tests>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
209 <test>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
210 <output name="bam_output" value="bam_output_sample" compare="sim_size" format="bam" delta_frac="0.1"/>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
211 <param name="input1" value="input1_sample"/>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
212 <param name="input2" value="input2_sample"/>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
213 </test>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
214 </tests>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
215 <help><![CDATA[
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
216
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
217 **What it Does**
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
218
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
219 Planemo advanced tool building sample bwa mem mapper
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
220
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
221 Reimagined as a bash script for a ToolFactory demonstration
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
222
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
223
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
224 ------
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
225
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
226 Script::
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
227
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
228 REFFILE=$1
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
229 FASTQ=$2
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
230 BAMOUT=$3
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
231 rm -f "refalias"
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
232 ln -s "$REFFILE" "refalias"
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
233 bwa index -a is "refalias"
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
234 bwa mem -t "2" -v 1 "refalias" "$FASTQ" > tempsam
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
235 samtools view -Sb tempsam > temporary_bam_file.bam
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
236 samtools sort -o "$BAMOUT" temporary_bam_file.bam
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
237
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
238 ]]></help>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
239 </tool>
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
240
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
241 ```
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
242
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
243
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
244
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
245 ## More Explanation
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
246
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
247 The TF is an unusual Galaxy tool, designed to allow a skilled user to make new Galaxy tools.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
248 It appears in Galaxy just like any other tool but outputs include new Galaxy tools generated
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
249 using instructions provided by the user and the results of Planemo lint and tool testing using
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
250 small sample inputs provided by the TF user. The small samples become tests built in to the new tool.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
251
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
252 It offers a familiar Galaxy form driven way to define how the user of the new tool will
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
253 choose input data from their history, and what parameters the new tool user will be able to adjust.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
254 The TF user must know, or be able to read, enough about the tool to be able to define the details of
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
255 the new Galaxy interface and the ToolFactory offers little guidance on that other than some examples.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
256
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
257 Tools always depend on other things. Most tools in Galaxy depend on third party
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
258 scientific packages, so TF tools usually have one or more dependencies. These can be
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
259 scientific packages such as BWA or scripting languages such as Python and are
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
260 managed by Conda. If the new tool relies on a system utility such as bash or awk
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
261 where the importance of version control on reproducibility is low, these can be used without
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
262 Conda management - but remember the potential risks of unmanaged dependencies on computational
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
263 reproducibility.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
264
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
265 The TF user can optionally supply a working script where scripting is
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
266 required and the chosen dependency is a scripting language such as Python or a system
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
267 scripting executable such as bash. Whatever the language, the script must correctly parse the command line
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
268 arguments it receives at tool execution, as they are defined by the TF user. The
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
269 text of that script is "baked in" to the new tool and will be executed each time
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
270 the new tool is run. It is highly recommended that scripts and their command lines be developed
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
271 and tested until proven to work before the TF is invoked. Galaxy as a software development
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
272 environment is actually possible, but not recommended being somewhat clumsy and inefficient.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
273
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
274 Tools nearly always take one or more data sets from the user's history as input. TF tools
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
275 allow the TF user to define what Galaxy datatypes the tool end user will be able to choose and what
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
276 names or positions will be used to pass them on a command line to the package or script.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
277
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
278 Tools often have various parameter settings. The TF allows the TF user to define how each
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
279 parameter will appear on the tool form to the end user, and what names or positions will be
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
280 used to pass them on the command line to the package. At present, parameters are limited to
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
281 simple text and number fields. Pull requests for other kinds of parameters that galaxyxml
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
282 can handle are welcomed.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
283
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
284 Best practice Galaxy tools have one or more automated tests. These should use small sample data sets and
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
285 specific parameter settings so when the tool is tested, the outputs can be compared with their expected
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
286 values. The TF will automatically create a test for the new tool. It will use the sample data sets
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
287 chosen by the TF user when they built the new tool.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
288
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
289 The TF works by exposing *unrestricted* and therefore extremely dangerous scripting
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
290 to all designated administrators of the host Galaxy server, allowing them to
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
291 run scripts in R, python, sh and perl. For this reason, a Docker container is
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
292 available to help manage the associated risks.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
293
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
294 ## Scripting uses
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
295
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
296 To use a scripting language to create a new tool, you must first prepared and properly test a script. Use small sample
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
297 data sets for testing. When the script is working correctly, upload the small sample datasets
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
298 into a new history, start configuring a new ToolFactory tool, and paste the script into the script text box on the TF form.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
299
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
300 ### Outputs
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
301
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
302 The TF will generate the new tool described on the TF form, and test it
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
303 using planemo. Optionally if a local toolshed is running, it can be used to
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
304 install the new tool back into the generating Galaxy.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
305
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
306 A toolshed is built in to the Docker container and configured
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
307 so a tool can be tested, sent to that toolshed, then installed in the Galaxy
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
308 where the TF is running using the default toolshed and Galaxy URL and API keys.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
309
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
310 Once it's in a ToolShed, it can be installed into any local Galaxy server
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
311 from the server administrative interface.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
312
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
313 Once the new tool is installed, local users can run it - each time, the
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
314 package and/or script that was supplied when it was built will be executed with the input chosen
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
315 from the user's history, together with user supplied parameters. In other words, the tools you generate with the
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
316 TF run just like any other Galaxy tool.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
317
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
318 TF generated tools work as normal workflow components.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
319
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
320
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
321 ## Limitations
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
322
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
323 The TF is flexible enough to generate wrappers for many common scientific packages
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
324 but the inbuilt automation will not cope with all possible situations. Users can
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
325 supply overrides for two tool XML segments - tests and command and the BWA
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
326 example in the supplied samples workflow illustrates their use. It does not deal with
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
327 repeated elements or conditional parameters such as allowing a user to choose to see "simple"
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
328 or "advanced" parameters (yet) and there will be plenty of packages it just
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
329 won't cover - but it's a quick and efficient tool for the other 90% of cases. Perfect for
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
330 that bash one liner you need to get that workflow functioning correctly for this
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
331 afternoon's demonstration!
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
332
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
333 ## Installation
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
334
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
335 The Docker container https://github.com/fubar2/toolfactory-galaxy-docker/blob/main/README.md
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
336 is the best way to use the TF because it is preconfigured
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
337 to automate new tool testing and has a built in local toolshed where each new tool
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
338 is uploaded. If you grab the docker container, it should just work after a restart and you
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
339 can run a workflow to generate all the sample tools. Running the samples and rerunning the ToolFactory
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
340 jobs that generated them allows you to add fields and experiment to see how things work.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
341
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
342 It can be installed like any other tool from the Toolshed, but you will need to make some
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
343 configuration changes (TODO write a configuration). You can install it most conveniently using the
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
344 administrative "Search and browse tool sheds" link. Find the Galaxy Main
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
345 toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
346 repository in the Tool Maker section. Open it and review the code and select the option to install it.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
347
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
348 If not already there please add:
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
349
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
350 ```
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
351 <datatype extension="tgz" type="galaxy.datatypes.binary:Binary" mimetype="multipart/x-gzip" subclass="True" />
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
352 ```
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
353
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
354 to your local config/data_types_conf.xml.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
355
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
356
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
357 ## Restricted execution
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
358
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
359 The tool factory tool itself will ONLY run for admin users -
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
360 people with IDs in config/galaxy.yml "admin_users".
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
361
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
362 *ONLY admin_users can run this tool*
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
363
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
364 That doesn't mean it's safe to install on a shared or exposed instance - please don't.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
365
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
366 ## Generated tool Security
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
367
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
368 Once you install a generated tool, it's just
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
369 another tool - assuming the script is safe. They just run normally and their
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
370 user cannot do anything unusually insecure but please, practice safe toolshed.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
371 Read the code before you install any tool. Especially this one - it is really scary.
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
372
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
373 ## Attribution
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
374
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
375 Creating re-usable tools from scripts: The Galaxy Tool Factory
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
376 Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
377 Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
378
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
379 http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref
43edf22e8cbc Toolshed seems cranky on very old metadata. Trying a new repo. Again
fubar
parents:
diff changeset
380