|
0
|
1 ## Breaking news! Completely refactored
|
|
|
2
|
|
|
3 ### New demonstration of planemo tool_factory command 
|
|
|
4
|
|
|
5 # WARNING
|
|
|
6
|
|
|
7 Install this tool to a throw-away private Galaxy or Docker container ONLY!
|
|
|
8
|
|
|
9 Please NEVER on a public or production instance where a hostile user may
|
|
|
10 be able to gain access if they can acquire an administrative account login.
|
|
|
11
|
|
|
12 It only runs for server administrators - the ToolFactory tool will refuse to execute for an ordinary user since
|
|
|
13 it can install new tools to the Galaxy server it executes on! This is not something you should allow other than
|
|
|
14 on a throw away instance that is protected from potentially hostile users.
|
|
|
15
|
|
|
16 ## Short Story
|
|
|
17
|
|
|
18 Galaxy is easily extended to new applications by adding a new tool. Each new scientific computational package added as
|
|
|
19 a tool to Galaxy requires an XML document describing how the application interacts with Galaxy.
|
|
|
20 This is sometimes termed "wrapping" the package because the instructions tell Galaxy how to run the package
|
|
|
21 as a new Galaxy tool. Any tool that has been wrapped is readily available to all the users through a consistent
|
|
|
22 and easy to use interface once installed in the local Galaxy server.
|
|
|
23
|
|
|
24 Most Galaxy tool wrappers have been manually prepared by skilled programmers, many using Planemo because it
|
|
|
25 automates much of the boilerplate and makes the process much easier.
|
|
|
26 The ToolFactory (TF) now uses Planemo under the hood for testing, but hides the command
|
|
|
27 line complexities. The user will still need appropriate skills in terms of describing the interface between
|
|
|
28 Galaxy and the new application, but will be helped by a Galaxy tool form to collect all the needed
|
|
|
29 settings, together with automated testing and uploading to a toolshed with optional local installation.
|
|
|
30
|
|
|
31
|
|
|
32 ## ToolFactory generated tools are ordinary Galaxy tools
|
|
|
33
|
|
|
34 A TF generated tool that passes the Planemo test is ready to publish in any Galaxy Toolshed and ready to install in any running Galaxy instance.
|
|
|
35 They are fully workflow compatible and work exactly like any hand-written tool. The user can select input files of the specified type(s) from their
|
|
|
36 history and edit each of the specified parameters. The tool form will show all the labels and help text supplied when the tool was built. When the tool
|
|
|
37 is executed, the dependent binary or script will be passed all the i/o files and parameters as specified, and will write outputs to the specified new
|
|
|
38 history datasets - just like any other Galaxy tool.
|
|
|
39
|
|
|
40 ## Models for tool command line construction
|
|
|
41
|
|
|
42 The key to turning any software package into a Galaxy tool is the automated construction of a suitable command line.
|
|
|
43
|
|
|
44 The TF can build a new tool that will allow the tool user to select input files from their history, set any parameters and when run will send the
|
|
|
45 new output files to the history as specified when the tool builder completed the form and built the new tool.
|
|
|
46
|
|
|
47 That tool can contain instructions to run any Conda dependency or a system executable like bash. Whether a bash script you have written or
|
|
|
48 a Conda package like bwa, the executable will expect to find settings for input, output and parameters on a command line.
|
|
|
49
|
|
|
50 These are often passed as "--name value" (argparse style) or in a fixed order (positional style).
|
|
|
51
|
|
|
52 The ToolFactory allows either, or for "filter" applications that process input from STDIN and write processed output to STDOUT.
|
|
|
53
|
|
|
54 The simplest tool model wraps a simple script or Conda dependency package requiring only input and output files, with no user supplied settings illustrated by
|
|
|
55 the Tacrev demonstration tool found in the Galaxy running in the ToolFactory docker container. It passes a user selected input file from the current history on STDIN
|
|
|
56 to a bash script. The bash script runs the unix tac utility (reverse cat) piped to the unix rev (reverse lines in a text file) utility. It's a one liner:
|
|
|
57
|
|
|
58 `tac | rev`
|
|
|
59
|
|
|
60 The tool building form allows zero or more Conda package name(s) and version(s) and an optional script to be executed by either a system
|
|
|
61 executable like ``bash`` or the first of any named Conda dependency package/version. Tacrev uses a tiny bash script shown above and uses the system
|
|
|
62 bash. Conda bash can be specified if it is important to use the same version consistently for the tool.
|
|
|
63
|
|
|
64 On the tool form, the repeat section allowing zero or more input files was set to be a text file to be selected by the tool user and
|
|
|
65 in the repeat section allowing one or more outputs, a new output file with special value `STDOUT` as the positional parameter, causes the TF to
|
|
|
66 generate a command to capture STDOUT and send it to the new history file containing the reversed input text.
|
|
|
67
|
|
|
68 By reversed, we mean really, truly reversed.
|
|
|
69
|
|
|
70 That simple model can be made much more complicated, and can pass inputs and outputs as named or positional parameters,
|
|
|
71 to allow more complicated scripts or dependent binaries that require:
|
|
|
72
|
|
|
73 1. Any number of input data files selected by the user from existing history data
|
|
|
74 2. Any number of output data files written to the user's history
|
|
|
75 3. Any number of user supplied parameters. These can be passed as command line arguments to the script or the dependency package. Either
|
|
|
76 positional or named (argparse) style command line parameter passing can be used.
|
|
|
77
|
|
|
78 More complex models can be seen in the Sedtest, Pyrevpos and Pyrevargparse tools illustrating positional and argparse parameter passing.
|
|
|
79
|
|
|
80 The most complex demonstration is the Planemo advanced tool tutorial BWA tool. There is one version using a command-override to implement
|
|
|
81 exactly the same command structure in the Planemo tutorial. A second version uses a bash script and positional parameters to achieve the same
|
|
|
82 result. Some tool builders may find the bash version more familiar and cleaner but the choice is yours.
|
|
|
83
|
|
|
84 ## Overview
|
|
|
85
|
|
|
86 
|
|
|
87
|
|
|
88
|
|
|
89 Steps in building a new Galaxy tool are all conducted through Galaxy running in the docker container:
|
|
|
90
|
|
|
91 1. Login to the Galaxy running in the container at http://localhost:8080 using an admin account. They are specified in config/galaxy.yml and
|
|
|
92 in the documentation at
|
|
|
93 and the ToolFactory will error out and refuse to run for non-administrative tool builders as a minimal protection from opportunistic hostile use.
|
|
|
94
|
|
|
95 2. Start the TF and fill in the form, providing sample inputs and parameter values to suit the Conda package being wrapped.
|
|
|
96
|
|
|
97 3. Execute the tool to create a new XML tool wrapper using the sample inputs and parameter settings for the inbuilt tool test. Planemo runs twice.
|
|
|
98 firstly to generate the test outputs and then to perform a proper test. The completed toolshed archive is written to the history
|
|
|
99 together with the planemo test report. Optionally the new tool archive can be uploaded
|
|
|
100 to the toolshed running in the same container (http://localhost:9009) and then installed inside the Galaxy in the container for further testing.
|
|
|
101
|
|
|
102 4. If the test fails, rerun the failed history job and correct errors on the tool form before rerunning until everything works correctly.
|
|
|
103
|
|
|
104 
|
|
|
105
|
|
|
106 ## Planning and building new Galaxy tool wrappers.
|
|
|
107
|
|
|
108 It is best to have all the required planning done to wrap any new script or binary before firing up the TF.
|
|
|
109 Conda is the only current dependency manager supported. Before starting, at the very least, the tool builder will need
|
|
|
110 to know the required software package name in Conda and the version to use, how the command line for
|
|
|
111 the package must be constructed, and there must be sample inputs in the working history for each of the required data inputs
|
|
|
112 for the package, together with values for every parameter to suit these sample inputs. These are required on the TF form
|
|
|
113 for preparing the inbuilt tool test. That test is run using Planemo, as part of the tool generation process.
|
|
|
114
|
|
|
115 A new tool is specified by filling in the usual Galaxy tool form.
|
|
|
116
|
|
|
117 The form starts with a new tool name. Most tools will need dependency packages and versions
|
|
|
118 for the executable. Only Conda is currently supported.
|
|
|
119
|
|
|
120 If a script is needed, it can be pasted into a text box and the interpreter named. Available system executables
|
|
|
121 can be used such as bash, or an interpreter such as python, perl or R can be nominated as conda dependencies
|
|
|
122 to ensure reproducible analyses.
|
|
|
123
|
|
|
124 The tool form will be generated from the input data and the tool builder supplied parameters. The command line for the
|
|
|
125 executable is built using positional or argparse (named e.g. --input_file /foo/baz) style
|
|
|
126 parameters and is completely dependent on the executable. These can include:
|
|
|
127
|
|
|
128 1. Any number of input data sets needed by the executable. Each appears to the tool user on the run form and is included
|
|
|
129 on the command line for the executable. The tool builder must supply a small representative sample for each one as
|
|
|
130 an input for the automated tool test.
|
|
|
131
|
|
|
132 2. Any number of output data sets generated by the package can be added to the command line and will appear in
|
|
|
133 the user's history at the end of the job
|
|
|
134
|
|
|
135 3. Any number of text or numeric parameters. Each will appear to the tool user on the run form and are included
|
|
|
136 on the command line to the executable. The tool builder must supply a suitable representative value for each one as
|
|
|
137 the value to be used for the automated tool test.
|
|
|
138
|
|
|
139 Once the form is completed, executing the TF will build a new XML tool wrapper
|
|
|
140 including a functional test based on the sample settings and data.
|
|
|
141
|
|
|
142 If the Planemo test passes, the tool can be optionally uploaded to the local Galaxy used in the image for more testing.
|
|
|
143
|
|
|
144 A local toolshed runs inside the container to allow an automated installation, although any toolshed and any accessible
|
|
|
145 Galaxy can be specified for this process by editing the default URL and API keys to provide appropriate credentials.
|
|
|
146
|
|
|
147 ## Generated Tool Dependency management
|
|
|
148
|
|
|
149 Conda is used for all dependency management although tools that use system utilities like sed, bash or awk
|
|
|
150 may be available on job execution nodes. Sed and friends are available as Conda (conda-forge) dependencies if necessary.
|
|
|
151 Versioned Conda dependencies are always baked-in to the tool and will be used for reproducible calculation.
|
|
|
152
|
|
|
153 ## Requirements
|
|
|
154
|
|
|
155 These are all managed automagically. The TF relies on galaxyxml to generate tool xml and uses ephemeris and
|
|
|
156 bioblend to load tools to the toolshed and to Galaxy. Planemo is used for testing and runs in a biocontainer currently at
|
|
|
157 https://quay.io/fubar2/planemo-biocontainer
|
|
|
158
|
|
|
159 ## Caveats
|
|
|
160
|
|
|
161 This docker image requires privileged mode so exposes potential security risks if hostile tool builders gain access.
|
|
|
162 Please, do not run it in any situation where that is a problem - never, ever on a public facing Galaxy server.
|
|
|
163 On a laptop or workstation should be fine in a non-hostile environment.
|
|
|
164
|
|
|
165
|
|
|
166 ## Example generated XML
|
|
|
167
|
|
|
168 For the bwa-mem example, a supplied bash script is included as a configfile and so has escaped characters.
|
|
|
169 ```
|
|
|
170 <tool name="bwatest" id="bwatest" version="0.01">
|
|
|
171 <!--Cite: Creating re-usable tools from scripts doi:10.1093/bioinformatics/bts573-->
|
|
|
172 <!--Source in git at: https://github.com/fubar2/toolfactory-->
|
|
|
173 <!--Created by admin@galaxy.org at 30/11/2020 07:12:10 using the Galaxy Tool Factory.-->
|
|
|
174 <description>Planemo advanced tool building sample bwa mem mapper as a ToolFactory demo</description>
|
|
|
175 <requirements>
|
|
|
176 <requirement version="0.7.15" type="package">bwa</requirement>
|
|
|
177 <requirement version="1.3" type="package">samtools</requirement>
|
|
|
178 </requirements>
|
|
|
179 <configfiles>
|
|
|
180 <configfile name="runme"><![CDATA[
|
|
|
181 REFFILE=\$1
|
|
|
182 FASTQ=\$2
|
|
|
183 BAMOUT=\$3
|
|
|
184 rm -f "refalias"
|
|
|
185 ln -s "\$REFFILE" "refalias"
|
|
|
186 bwa index -a is "refalias"
|
|
|
187 bwa mem -t "2" -v 1 "refalias" "\$FASTQ" > tempsam
|
|
|
188 samtools view -Sb tempsam > temporary_bam_file.bam
|
|
|
189 samtools sort -o "\$BAMOUT" temporary_bam_file.bam
|
|
|
190
|
|
|
191 ]]></configfile>
|
|
|
192 </configfiles>
|
|
|
193 <version_command/>
|
|
|
194 <command><![CDATA[bash
|
|
|
195 $runme
|
|
|
196 $input1
|
|
|
197 $input2
|
|
|
198 $bam_output]]></command>
|
|
|
199 <inputs>
|
|
|
200 <param optional="false" label="Reference sequence for bwa to map the fastq reads against" help="" format="fasta" multiple="false" type="data" name="input1" argument="input1"/>
|
|
|
201 <param optional="false" label="Reads as fastqsanger to align to the reference sequence" help="" format="fastqsanger" multiple="false" type="data" name="input2" argument="input2"/>
|
|
|
202 </inputs>
|
|
|
203 <outputs>
|
|
|
204 <data name="bam_output" format="bam" label="bam_output" hidden="false"/>
|
|
|
205 </outputs>
|
|
|
206 <tests>
|
|
|
207 <test>
|
|
|
208 <output name="bam_output" value="bam_output_sample" compare="sim_size" format="bam" delta_frac="0.1"/>
|
|
|
209 <param name="input1" value="input1_sample"/>
|
|
|
210 <param name="input2" value="input2_sample"/>
|
|
|
211 </test>
|
|
|
212 </tests>
|
|
|
213 <help><![CDATA[
|
|
|
214
|
|
|
215 **What it Does**
|
|
|
216
|
|
|
217 Planemo advanced tool building sample bwa mem mapper
|
|
|
218
|
|
|
219 Reimagined as a bash script for a ToolFactory demonstration
|
|
|
220
|
|
|
221
|
|
|
222 ------
|
|
|
223
|
|
|
224 Script::
|
|
|
225
|
|
|
226 REFFILE=$1
|
|
|
227 FASTQ=$2
|
|
|
228 BAMOUT=$3
|
|
|
229 rm -f "refalias"
|
|
|
230 ln -s "$REFFILE" "refalias"
|
|
|
231 bwa index -a is "refalias"
|
|
|
232 bwa mem -t "2" -v 1 "refalias" "$FASTQ" > tempsam
|
|
|
233 samtools view -Sb tempsam > temporary_bam_file.bam
|
|
|
234 samtools sort -o "$BAMOUT" temporary_bam_file.bam
|
|
|
235
|
|
|
236 ]]></help>
|
|
|
237 </tool>
|
|
|
238
|
|
|
239 ```
|
|
|
240
|
|
|
241
|
|
|
242
|
|
|
243 ## More Explanation
|
|
|
244
|
|
|
245 The TF is an unusual Galaxy tool, designed to allow a skilled user to make new Galaxy tools.
|
|
|
246 It appears in Galaxy just like any other tool but outputs include new Galaxy tools generated
|
|
|
247 using instructions provided by the user and the results of Planemo lint and tool testing using
|
|
|
248 small sample inputs provided by the TF user. The small samples become tests built in to the new tool.
|
|
|
249
|
|
|
250 It offers a familiar Galaxy form driven way to define how the user of the new tool will
|
|
|
251 choose input data from their history, and what parameters the new tool user will be able to adjust.
|
|
|
252 The TF user must know, or be able to read, enough about the tool to be able to define the details of
|
|
|
253 the new Galaxy interface and the ToolFactory offers little guidance on that other than some examples.
|
|
|
254
|
|
|
255 Tools always depend on other things. Most tools in Galaxy depend on third party
|
|
|
256 scientific packages, so TF tools usually have one or more dependencies. These can be
|
|
|
257 scientific packages such as BWA or scripting languages such as Python and are
|
|
|
258 managed by Conda. If the new tool relies on a system utility such as bash or awk
|
|
|
259 where the importance of version control on reproducibility is low, these can be used without
|
|
|
260 Conda management - but remember the potential risks of unmanaged dependencies on computational
|
|
|
261 reproducibility.
|
|
|
262
|
|
|
263 The TF user can optionally supply a working script where scripting is
|
|
|
264 required and the chosen dependency is a scripting language such as Python or a system
|
|
|
265 scripting executable such as bash. Whatever the language, the script must correctly parse the command line
|
|
|
266 arguments it receives at tool execution, as they are defined by the TF user. The
|
|
|
267 text of that script is "baked in" to the new tool and will be executed each time
|
|
|
268 the new tool is run. It is highly recommended that scripts and their command lines be developed
|
|
|
269 and tested until proven to work before the TF is invoked. Galaxy as a software development
|
|
|
270 environment is actually possible, but not recommended being somewhat clumsy and inefficient.
|
|
|
271
|
|
|
272 Tools nearly always take one or more data sets from the user's history as input. TF tools
|
|
|
273 allow the TF user to define what Galaxy datatypes the tool end user will be able to choose and what
|
|
|
274 names or positions will be used to pass them on a command line to the package or script.
|
|
|
275
|
|
|
276 Tools often have various parameter settings. The TF allows the TF user to define how each
|
|
|
277 parameter will appear on the tool form to the end user, and what names or positions will be
|
|
|
278 used to pass them on the command line to the package. At present, parameters are limited to
|
|
|
279 simple text and number fields. Pull requests for other kinds of parameters that galaxyxml
|
|
|
280 can handle are welcomed.
|
|
|
281
|
|
|
282 Best practice Galaxy tools have one or more automated tests. These should use small sample data sets and
|
|
|
283 specific parameter settings so when the tool is tested, the outputs can be compared with their expected
|
|
|
284 values. The TF will automatically create a test for the new tool. It will use the sample data sets
|
|
|
285 chosen by the TF user when they built the new tool.
|
|
|
286
|
|
|
287 The TF works by exposing *unrestricted* and therefore extremely dangerous scripting
|
|
|
288 to all designated administrators of the host Galaxy server, allowing them to
|
|
|
289 run scripts in R, python, sh and perl. For this reason, a Docker container is
|
|
|
290 available to help manage the associated risks.
|
|
|
291
|
|
|
292 ## Scripting uses
|
|
|
293
|
|
|
294 To use a scripting language to create a new tool, you must first prepared and properly test a script. Use small sample
|
|
|
295 data sets for testing. When the script is working correctly, upload the small sample datasets
|
|
|
296 into a new history, start configuring a new ToolFactory tool, and paste the script into the script text box on the TF form.
|
|
|
297
|
|
|
298 ### Outputs
|
|
|
299
|
|
|
300 The TF will generate the new tool described on the TF form, and test it
|
|
|
301 using planemo. Optionally if a local toolshed is running, it can be used to
|
|
|
302 install the new tool back into the generating Galaxy.
|
|
|
303
|
|
|
304 A toolshed is built in to the Docker container and configured
|
|
|
305 so a tool can be tested, sent to that toolshed, then installed in the Galaxy
|
|
|
306 where the TF is running using the default toolshed and Galaxy URL and API keys.
|
|
|
307
|
|
|
308 Once it's in a ToolShed, it can be installed into any local Galaxy server
|
|
|
309 from the server administrative interface.
|
|
|
310
|
|
|
311 Once the new tool is installed, local users can run it - each time, the
|
|
|
312 package and/or script that was supplied when it was built will be executed with the input chosen
|
|
|
313 from the user's history, together with user supplied parameters. In other words, the tools you generate with the
|
|
|
314 TF run just like any other Galaxy tool.
|
|
|
315
|
|
|
316 TF generated tools work as normal workflow components.
|
|
|
317
|
|
|
318
|
|
|
319 ## Limitations
|
|
|
320
|
|
|
321 The TF is flexible enough to generate wrappers for many common scientific packages
|
|
|
322 but the inbuilt automation will not cope with all possible situations. Users can
|
|
|
323 supply overrides for two tool XML segments - tests and command and the BWA
|
|
|
324 example in the supplied samples workflow illustrates their use. It does not deal with
|
|
|
325 repeated elements or conditional parameters such as allowing a user to choose to see "simple"
|
|
|
326 or "advanced" parameters (yet) and there will be plenty of packages it just
|
|
|
327 won't cover - but it's a quick and efficient tool for the other 90% of cases. Perfect for
|
|
|
328 that bash one liner you need to get that workflow functioning correctly for this
|
|
|
329 afternoon's demonstration!
|
|
|
330
|
|
|
331 ## Installation
|
|
|
332
|
|
|
333 The Docker container https://github.com/fubar2/toolfactory-galaxy-docker/blob/main/README.md
|
|
|
334 is the best way to use the TF because it is preconfigured
|
|
|
335 to automate new tool testing and has a built in local toolshed where each new tool
|
|
|
336 is uploaded. If you grab the docker container, it should just work after a restart and you
|
|
|
337 can run a workflow to generate all the sample tools. Running the samples and rerunning the ToolFactory
|
|
|
338 jobs that generated them allows you to add fields and experiment to see how things work.
|
|
|
339
|
|
|
340 It can be installed like any other tool from the Toolshed, but you will need to make some
|
|
|
341 configuration changes (TODO write a configuration). You can install it most conveniently using the
|
|
|
342 administrative "Search and browse tool sheds" link. Find the Galaxy Main
|
|
|
343 toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory
|
|
|
344 repository in the Tool Maker section. Open it and review the code and select the option to install it.
|
|
|
345
|
|
|
346 If not already there please add:
|
|
|
347
|
|
|
348 ```
|
|
|
349 <datatype extension="tgz" type="galaxy.datatypes.binary:Binary" mimetype="multipart/x-gzip" subclass="True" />
|
|
|
350 ```
|
|
|
351
|
|
|
352 to your local config/data_types_conf.xml.
|
|
|
353
|
|
|
354
|
|
|
355 ## Restricted execution
|
|
|
356
|
|
|
357 The tool factory tool itself will ONLY run for admin users -
|
|
|
358 people with IDs in config/galaxy.yml "admin_users".
|
|
|
359
|
|
|
360 *ONLY admin_users can run this tool*
|
|
|
361
|
|
|
362 That doesn't mean it's safe to install on a shared or exposed instance - please don't.
|
|
|
363
|
|
|
364 ## Generated tool Security
|
|
|
365
|
|
|
366 Once you install a generated tool, it's just
|
|
|
367 another tool - assuming the script is safe. They just run normally and their
|
|
|
368 user cannot do anything unusually insecure but please, practice safe toolshed.
|
|
|
369 Read the code before you install any tool. Especially this one - it is really scary.
|
|
|
370
|
|
|
371 ## Attribution
|
|
|
372
|
|
|
373 Creating re-usable tools from scripts: The Galaxy Tool Factory
|
|
|
374 Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team
|
|
|
375 Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573
|
|
|
376
|
|
|
377 http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref
|
|
|
378
|
