|
119
|
1 ## Breaking news! Docker container at https://github.com/fubar2/toolfactory-galaxy-docker recommended as at December 2020
|
|
49
|
2
|
|
119
|
3 ## This is the original ToolFactory suitable for non-docker situations. Please use the docker container if you can because it's integrated with a Toolshed...
|
|
49
|
4
|
|
119
|
5 # WARNING
|
|
28
|
6
|
|
119
|
7 Install this tool to a throw-away private Galaxy or Docker container ONLY!
|
|
28
|
8
|
|
119
|
9 Please NEVER on a public or production instance where a hostile user may
|
|
|
10 be able to gain access if they can acquire an administrative account login.
|
|
28
|
11
|
|
119
|
12 It only runs for server administrators - the ToolFactory tool will refuse to execute for an ordinary user since
|
|
|
13 it can install new tools to the Galaxy server it executes on! This is not something you should allow other than
|
|
|
14 on a throw away instance that is protected from potentially hostile users.
|
|
49
|
15
|
|
119
|
16 ## Short Story
|
|
49
|
17
|
|
|
18 Galaxy is easily extended to new applications by adding a new tool. Each new scientific computational package added as
|
|
119
|
19 a tool to Galaxy requires an XML document describing how the application interacts with Galaxy.
|
|
|
20 This is sometimes termed "wrapping" the package because the instructions tell Galaxy how to run the package
|
|
|
21 as a new Galaxy tool. Any tool that has been wrapped is readily available to all the users through a consistent
|
|
|
22 and easy to use interface once installed in the local Galaxy server.
|
|
|
23
|
|
|
24 Most Galaxy tool wrappers have been manually prepared by skilled programmers, many using Planemo because it
|
|
|
25 automates much of the boilerplate and makes the process much easier.
|
|
|
26 The ToolFactory (TF) now uses Planemo under the hood for testing, but hides the command
|
|
|
27 line complexities. The user will still need appropriate skills in terms of describing the interface between
|
|
|
28 Galaxy and the new application, but will be helped by a Galaxy tool form to collect all the needed
|
|
|
29 settings, together with automated testing and uploading to a toolshed with optional local installation.
|
|
|
30
|
|
|
31
|
|
|
32 ## ToolFactory generated tools are ordinary Galaxy tools
|
|
|
33
|
|
|
34 A TF generated tool that passes the Planemo test is ready to publish in any Galaxy Toolshed and ready to install in any running Galaxy instance.
|
|
|
35 They are fully workflow compatible and work exactly like any hand-written tool. The user can select input files of the specified type(s) from their
|
|
|
36 history and edit each of the specified parameters. The tool form will show all the labels and help text supplied when the tool was built. When the tool
|
|
|
37 is executed, the dependent binary or script will be passed all the i/o files and parameters as specified, and will write outputs to the specified new
|
|
|
38 history datasets - just like any other Galaxy tool.
|
|
|
39
|
|
|
40 ## Models for tool command line construction
|
|
|
41
|
|
|
42 The key to turning any software package into a Galaxy tool is the automated construction of a suitable command line.
|
|
|
43
|
|
|
44 The TF can build a new tool that will allow the tool user to select input files from their history, set any parameters and when run will send the
|
|
|
45 new output files to the history as specified when the tool builder completed the form and built the new tool.
|
|
|
46
|
|
|
47 That tool can contain instructions to run any Conda dependency or a system executable like bash. Whether a bash script you have written or
|
|
|
48 a Conda package like bwa, the executable will expect to find settings for input, output and parameters on a command line.
|
|
|
49
|
|
|
50 These are often passed as "--name value" (argparse style) or in a fixed order (positional style).
|
|
|
51
|
|
|
52 The ToolFactory allows either, or for "filter" applications that process input from STDIN and write processed output to STDOUT.
|
|
|
53
|
|
|
54 The simplest tool model wraps a simple script or Conda dependency package requiring only input and output files, with no user supplied settings illustrated by
|
|
|
55 the Tacrev demonstration tool found in the Galaxy running in the ToolFactory docker container. It passes a user selected input file from the current history on STDIN
|
|
|
56 to a bash script. The bash script runs the unix tac utility (reverse cat) piped to the unix rev (reverse lines in a text file) utility. It's a one liner:
|
|
|
57
|
|
|
58 `tac | rev`
|
|
|
59
|
|
|
60 The tool building form allows zero or more Conda package name(s) and version(s) and an optional script to be executed by either a system
|
|
|
61 executable like ``bash`` or the first of any named Conda dependency package/version. Tacrev uses a tiny bash script shown above and uses the system
|
|
|
62 bash. Conda bash can be specified if it is important to use the same version consistently for the tool.
|
|
|
63
|
|
|
64 On the tool form, the repeat section allowing zero or more input files was set to be a text file to be selected by the tool user and
|
|
|
65 in the repeat section allowing one or more outputs, a new output file with special value `STDOUT` as the positional parameter, causes the TF to
|
|
|
66 generate a command to capture STDOUT and send it to the new history file containing the reversed input text.
|
|
|
67
|
|
|
68 By reversed, we mean really, truly reversed.
|
|
|
69
|
|
|
70 That simple model can be made much more complicated, and can pass inputs and outputs as named or positional parameters,
|
|
|
71 to allow more complicated scripts or dependent binaries that require:
|
|
|
72
|
|
|
73 1. Any number of input data files selected by the user from existing history data
|
|
|
74 2. Any number of output data files written to the user's history
|
|
|
75 3. Any number of user supplied parameters. These can be passed as command line arguments to the script or the dependency package. Either
|
|
|
76 positional or named (argparse) style command line parameter passing can be used.
|
|
|
77
|
|
|
78 More complex models can be seen in the Sedtest, Pyrevpos and Pyrevargparse tools illustrating positional and argparse parameter passing.
|
|
|
79
|
|
|
80 The most complex demonstration is the Planemo advanced tool tutorial BWA tool. There is one version using a command-override to implement
|
|
|
81 exactly the same command structure in the Planemo tutorial. A second version uses a bash script and positional parameters to achieve the same
|
|
|
82 result. Some tool builders may find the bash version more familiar and cleaner but the choice is yours.
|
|
|
83
|
|
|
84 ## Overview
|
|
|
85
|
|
|
86 
|
|
|
87
|
|
|
88
|
|
|
89 Steps in building a new Galaxy tool are all conducted through Galaxy running in the docker container:
|
|
|
90
|
|
|
91 1. Login to the Galaxy running in the container at http://localhost:8080 using an admin account. They are specified in config/galaxy.yml and
|
|
|
92 in the documentation at
|
|
|
93 and the ToolFactory will error out and refuse to run for non-administrative tool builders as a minimal protection from opportunistic hostile use.
|
|
|
94
|
|
|
95 2. Start the TF and fill in the form, providing sample inputs and parameter values to suit the Conda package being wrapped.
|
|
|
96
|
|
|
97 3. Execute the tool to create a new XML tool wrapper using the sample inputs and parameter settings for the inbuilt tool test. Planemo runs twice.
|
|
|
98 firstly to generate the test outputs and then to perform a proper test. The completed toolshed archive is written to the history
|
|
|
99 together with the planemo test report. Optionally the new tool archive can be uploaded
|
|
|
100 to the toolshed running in the same container (http://localhost:9009) and then installed inside the Galaxy in the container for further testing.
|
|
|
101
|
|
|
102 4. If the test fails, rerun the failed history job and correct errors on the tool form before rerunning until everything works correctly.
|
|
|
103
|
|
|
104
|
|
|
105
|
|
|
106 
|
|
|
107
|
|
|
108 ## Planning and building new Galaxy tool wrappers.
|
|
|
109
|
|
|
110 It is best to have all the required planning done to wrap any new script or binary before firing up the TF.
|
|
|
111 Conda is the only current dependency manager supported. Before starting, at the very least, the tool builder will need
|
|
|
112 to know the required software package name in Conda and the version to use, how the command line for
|
|
|
113 the package must be constructed, and there must be sample inputs in the working history for each of the required data inputs
|
|
|
114 for the package, together with values for every parameter to suit these sample inputs. These are required on the TF form
|
|
|
115 for preparing the inbuilt tool test. That test is run using Planemo, as part of the tool generation process.
|
|
|
116
|
|
|
117 A new tool is specified by filling in the usual Galaxy tool form.
|
|
|
118
|
|
|
119 The form starts with a new tool name. Most tools will need dependency packages and versions
|
|
|
120 for the executable. Only Conda is currently supported.
|
|
|
121
|
|
|
122 If a script is needed, it can be pasted into a text box and the interpreter named. Available system executables
|
|
|
123 can be used such as bash, or an interpreter such as python, perl or R can be nominated as conda dependencies
|
|
|
124 to ensure reproducible analyses.
|
|
|
125
|
|
|
126 The tool form will be generated from the input data and the tool builder supplied parameters. The command line for the
|
|
|
127 executable is built using positional or argparse (named e.g. --input_file /foo/baz) style
|
|
|
128 parameters and is completely dependent on the executable. These can include:
|
|
|
129
|
|
|
130 1. Any number of input data sets needed by the executable. Each appears to the tool user on the run form and is included
|
|
|
131 on the command line for the executable. The tool builder must supply a small representative sample for each one as
|
|
|
132 an input for the automated tool test.
|
|
28
|
133
|
|
119
|
134 2. Any number of output data sets generated by the package can be added to the command line and will appear in
|
|
|
135 the user's history at the end of the job
|
|
|
136
|
|
|
137 3. Any number of text or numeric parameters. Each will appear to the tool user on the run form and are included
|
|
|
138 on the command line to the executable. The tool builder must supply a suitable representative value for each one as
|
|
|
139 the value to be used for the automated tool test.
|
|
|
140
|
|
|
141 Once the form is completed, executing the TF will build a new XML tool wrapper
|
|
|
142 including a functional test based on the sample settings and data.
|
|
|
143
|
|
|
144 If the Planemo test passes, the tool can be optionally uploaded to the local Galaxy used in the image for more testing.
|
|
|
145
|
|
|
146 A local toolshed runs inside the container to allow an automated installation, although any toolshed and any accessible
|
|
|
147 Galaxy can be specified for this process by editing the default URL and API keys to provide appropriate credentials.
|
|
|
148
|
|
|
149 ## Generated Tool Dependency management
|
|
|
150
|
|
|
151 Conda is used for all dependency management although tools that use system utilities like sed, bash or awk
|
|
|
152 may be available on job execution nodes. Sed and friends are available as Conda (conda-forge) dependencies if necessary.
|
|
|
153 Versioned Conda dependencies are always baked-in to the tool and will be used for reproducible calculation.
|
|
|
154
|
|
|
155 ## Requirements
|
|
|
156
|
|
|
157 These are all managed automagically. The TF relies on galaxyxml to generate tool xml and uses ephemeris and
|
|
|
158 bioblend to load tools to the toolshed and to Galaxy. Planemo is used for testing and runs in a biocontainer currently at
|
|
|
159 https://quay.io/fubar2/planemo-biocontainer
|
|
|
160
|
|
|
161 ## Caveats
|
|
|
162
|
|
|
163 This docker image requires privileged mode so exposes potential security risks if hostile tool builders gain access.
|
|
|
164 Please, do not run it in any situation where that is a problem - never, ever on a public facing Galaxy server.
|
|
|
165 On a laptop or workstation should be fine in a non-hostile environment.
|
|
|
166
|
|
|
167
|
|
|
168 ## Example generated XML
|
|
28
|
169
|
|
119
|
170 For the bwa-mem example, a supplied bash script is included as a configfile and so has escaped characters.
|
|
|
171 ```
|
|
|
172 <tool name="bwatest" id="bwatest" version="0.01">
|
|
|
173 <!--Cite: Creating re-usable tools from scripts doi:10.1093/bioinformatics/bts573-->
|
|
|
174 <!--Source in git at: https://github.com/fubar2/toolfactory-->
|
|
|
175 <!--Created by admin@galaxy.org at 30/11/2020 07:12:10 using the Galaxy Tool Factory.-->
|
|
|
176 <description>Planemo advanced tool building sample bwa mem mapper as a ToolFactory demo</description>
|
|
|
177 <requirements>
|
|
|
178 <requirement version="0.7.15" type="package">bwa</requirement>
|
|
|
179 <requirement version="1.3" type="package">samtools</requirement>
|
|
|
180 </requirements>
|
|
|
181 <configfiles>
|
|
|
182 <configfile name="runme"><![CDATA[
|
|
|
183 REFFILE=\$1
|
|
|
184 FASTQ=\$2
|
|
|
185 BAMOUT=\$3
|
|
|
186 rm -f "refalias"
|
|
|
187 ln -s "\$REFFILE" "refalias"
|
|
|
188 bwa index -a is "refalias"
|
|
|
189 bwa mem -t "2" -v 1 "refalias" "\$FASTQ" > tempsam
|
|
|
190 samtools view -Sb tempsam > temporary_bam_file.bam
|
|
|
191 samtools sort -o "\$BAMOUT" temporary_bam_file.bam
|
|
28
|
192
|
|
119
|
193 ]]></configfile>
|
|
|
194 </configfiles>
|
|
|
195 <version_command/>
|
|
|
196 <command><![CDATA[bash
|
|
|
197 $runme
|
|
|
198 $input1
|
|
|
199 $input2
|
|
|
200 $bam_output]]></command>
|
|
|
201 <inputs>
|
|
|
202 <param optional="false" label="Reference sequence for bwa to map the fastq reads against" help="" format="fasta" multiple="false" type="data" name="input1" argument="input1"/>
|
|
|
203 <param optional="false" label="Reads as fastqsanger to align to the reference sequence" help="" format="fastqsanger" multiple="false" type="data" name="input2" argument="input2"/>
|
|
|
204 </inputs>
|
|
|
205 <outputs>
|
|
|
206 <data name="bam_output" format="bam" label="bam_output" hidden="false"/>
|
|
|
207 </outputs>
|
|
|
208 <tests>
|
|
|
209 <test>
|
|
|
210 <output name="bam_output" value="bam_output_sample" compare="sim_size" format="bam" delta_frac="0.1"/>
|
|
|
211 <param name="input1" value="input1_sample"/>
|
|
|
212 <param name="input2" value="input2_sample"/>
|
|
|
213 </test>
|
|
|
214 </tests>
|
|
|
215 <help><![CDATA[
|
|
|
216
|
|
|
217 **What it Does**
|
|
|
218
|
|
|
219 Planemo advanced tool building sample bwa mem mapper
|
|
|
220
|
|
|
221 Reimagined as a bash script for a ToolFactory demonstration
|
|
|
222
|
|
|
223
|
|
|
224 ------
|
|
|
225
|
|
|
226 Script::
|
|
|
227
|
|
|
228 REFFILE=$1
|
|
|
229 FASTQ=$2
|
|
|
230 BAMOUT=$3
|
|
|
231 rm -f "refalias"
|
|
|
232 ln -s "$REFFILE" "refalias"
|
|
|
233 bwa index -a is "refalias"
|
|
|
234 bwa mem -t "2" -v 1 "refalias" "$FASTQ" > tempsam
|
|
|
235 samtools view -Sb tempsam > temporary_bam_file.bam
|
|
|
236 samtools sort -o "$BAMOUT" temporary_bam_file.bam
|
|
|
237
|
|
|
238 ]]></help>
|
|
|
239 </tool>
|
|
|
240
|
|
|
241 ```
|
|
|
242
|
|
|
243
|
|
|
244
|
|
|
245 ## More Explanation
|
|
|
246
|
|
|
247 The TF is an unusual Galaxy tool, designed to allow a skilled user to make new Galaxy tools.
|
|
49
|
248 It appears in Galaxy just like any other tool but outputs include new Galaxy tools generated
|
|
|
249 using instructions provided by the user and the results of Planemo lint and tool testing using
|
|
|
250 small sample inputs provided by the TF user. The small samples become tests built in to the new tool.
|
|
|
251
|
|
119
|
252 It offers a familiar Galaxy form driven way to define how the user of the new tool will
|
|
49
|
253 choose input data from their history, and what parameters the new tool user will be able to adjust.
|
|
|
254 The TF user must know, or be able to read, enough about the tool to be able to define the details of
|
|
|
255 the new Galaxy interface and the ToolFactory offers little guidance on that other than some examples.
|
|
28
|
256
|
|
49
|
257 Tools always depend on other things. Most tools in Galaxy depend on third party
|
|
|
258 scientific packages, so TF tools usually have one or more dependencies. These can be
|
|
|
259 scientific packages such as BWA or scripting languages such as Python and are
|
|
119
|
260 managed by Conda. If the new tool relies on a system utility such as bash or awk
|
|
|
261 where the importance of version control on reproducibility is low, these can be used without
|
|
49
|
262 Conda management - but remember the potential risks of unmanaged dependencies on computational
|
|
|
263 reproducibility.
|
|
28
|
264
|
|
49
|
265 The TF user can optionally supply a working script where scripting is
|
|
|
266 required and the chosen dependency is a scripting language such as Python or a system
|
|
|
267 scripting executable such as bash. Whatever the language, the script must correctly parse the command line
|
|
|
268 arguments it receives at tool execution, as they are defined by the TF user. The
|
|
|
269 text of that script is "baked in" to the new tool and will be executed each time
|
|
|
270 the new tool is run. It is highly recommended that scripts and their command lines be developed
|
|
|
271 and tested until proven to work before the TF is invoked. Galaxy as a software development
|
|
|
272 environment is actually possible, but not recommended being somewhat clumsy and inefficient.
|
|
28
|
273
|
|
49
|
274 Tools nearly always take one or more data sets from the user's history as input. TF tools
|
|
119
|
275 allow the TF user to define what Galaxy datatypes the tool end user will be able to choose and what
|
|
49
|
276 names or positions will be used to pass them on a command line to the package or script.
|
|
28
|
277
|
|
49
|
278 Tools often have various parameter settings. The TF allows the TF user to define how each
|
|
|
279 parameter will appear on the tool form to the end user, and what names or positions will be
|
|
|
280 used to pass them on the command line to the package. At present, parameters are limited to
|
|
|
281 simple text and number fields. Pull requests for other kinds of parameters that galaxyxml
|
|
|
282 can handle are welcomed.
|
|
28
|
283
|
|
49
|
284 Best practice Galaxy tools have one or more automated tests. These should use small sample data sets and
|
|
|
285 specific parameter settings so when the tool is tested, the outputs can be compared with their expected
|
|
119
|
286 values. The TF will automatically create a test for the new tool. It will use the sample data sets
|
|
49
|
287 chosen by the TF user when they built the new tool.
|
|
28
|
288
|
|
49
|
289 The TF works by exposing *unrestricted* and therefore extremely dangerous scripting
|
|
|
290 to all designated administrators of the host Galaxy server, allowing them to
|
|
|
291 run scripts in R, python, sh and perl. For this reason, a Docker container is
|
|
|
292 available to help manage the associated risks.
|
|
|
293
|
|
119
|
294 ## Scripting uses
|
|
49
|
295
|
|
|
296 To use a scripting language to create a new tool, you must first prepared and properly test a script. Use small sample
|
|
|
297 data sets for testing. When the script is working correctly, upload the small sample datasets
|
|
|
298 into a new history, start configuring a new ToolFactory tool, and paste the script into the script text box on the TF form.
|
|
|
299
|
|
119
|
300 ### Outputs
|
|
28
|
301
|
|
119
|
302 The TF will generate the new tool described on the TF form, and test it
|
|
|
303 using planemo. Optionally if a local toolshed is running, it can be used to
|
|
|
304 install the new tool back into the generating Galaxy.
|
|
49
|
305
|
|
119
|
306 A toolshed is built in to the Docker container and configured
|
|
|
307 so a tool can be tested, sent to that toolshed, then installed in the Galaxy
|
|
|
308 where the TF is running using the default toolshed and Galaxy URL and API keys.
|
|
49
|
309
|
|
28
|
310 Once it's in a ToolShed, it can be installed into any local Galaxy server
|
|
|
311 from the server administrative interface.
|
|
|
312
|
|
119
|
313 Once the new tool is installed, local users can run it - each time, the
|
|
49
|
314 package and/or script that was supplied when it was built will be executed with the input chosen
|
|
|
315 from the user's history, together with user supplied parameters. In other words, the tools you generate with the
|
|
119
|
316 TF run just like any other Galaxy tool.
|
|
28
|
317
|
|
49
|
318 TF generated tools work as normal workflow components.
|
|
|
319
|
|
|
320
|
|
119
|
321 ## Limitations
|
|
28
|
322
|
|
49
|
323 The TF is flexible enough to generate wrappers for many common scientific packages
|
|
|
324 but the inbuilt automation will not cope with all possible situations. Users can
|
|
|
325 supply overrides for two tool XML segments - tests and command and the BWA
|
|
119
|
326 example in the supplied samples workflow illustrates their use. It does not deal with
|
|
|
327 repeated elements or conditional parameters such as allowing a user to choose to see "simple"
|
|
|
328 or "advanced" parameters (yet) and there will be plenty of packages it just
|
|
|
329 won't cover - but it's a quick and efficient tool for the other 90% of cases. Perfect for
|
|
|
330 that bash one liner you need to get that workflow functioning correctly for this
|
|
|
331 afternoon's demonstration!
|
|
49
|
332
|
|
119
|
333 ## Installation
|
|
56
|
334
|
|
119
|
335 The Docker container https://github.com/fubar2/toolfactory-galaxy-docker/blob/main/README.md
|
|
|
336 is the best way to use the TF because it is preconfigured
|
|
|
337 to automate new tool testing and has a built in local toolshed where each new tool
|
|
|
338 is uploaded. If you grab the docker container, it should just work after a restart and you
|
|
|
339 can run a workflow to generate all the sample tools. Running the samples and rerunning the ToolFactory
|
|
|
340 jobs that generated them allows you to add fields and experiment to see how things work.
|
|
56
|
341
|
|
119
|
342 It can be installed like any other tool from the Toolshed, but you will need to make some
|
|
49
|
343 configuration changes (TODO write a configuration). You can install it most conveniently using the
|
|
28
|
344 administrative "Search and browse tool sheds" link. Find the Galaxy Main
|
|
|
345 toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory
|
|
49
|
346 repository in the Tool Maker section. Open it and review the code and select the option to install it.
|
|
28
|
347
|
|
119
|
348 If not already there please add:
|
|
|
349
|
|
|
350 ```
|
|
|
351 <datatype extension="tgz" type="galaxy.datatypes.binary:Binary" mimetype="multipart/x-gzip" subclass="True" />
|
|
|
352 ```
|
|
|
353
|
|
|
354 to your local config/data_types_conf.xml.
|
|
28
|
355
|
|
|
356
|
|
119
|
357 ## Restricted execution
|
|
|
358
|
|
|
359 The tool factory tool itself will ONLY run for admin users -
|
|
|
360 people with IDs in config/galaxy.yml "admin_users".
|
|
28
|
361
|
|
119
|
362 *ONLY admin_users can run this tool*
|
|
28
|
363
|
|
119
|
364 That doesn't mean it's safe to install on a shared or exposed instance - please don't.
|
|
|
365
|
|
|
366 ## Generated tool Security
|
|
28
|
367
|
|
|
368 Once you install a generated tool, it's just
|
|
|
369 another tool - assuming the script is safe. They just run normally and their
|
|
|
370 user cannot do anything unusually insecure but please, practice safe toolshed.
|
|
|
371 Read the code before you install any tool. Especially this one - it is really scary.
|
|
|
372
|
|
119
|
373 ## Attribution
|
|
28
|
374
|
|
|
375 Creating re-usable tools from scripts: The Galaxy Tool Factory
|
|
|
376 Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team
|
|
|
377 Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573
|
|
|
378
|
|
|
379 http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref
|
|
|
380
|
