Mercurial > repos > fubar_too > toolfactory

comparison toolfactory/README.md @ 0:03df06784e56 draft

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Uploaded
author fubar_too
date Tue, 18 May 2021 08:42:29 +0000
parents
children
comparison
equal deleted inserted replaced
-1:000000000000 0:03df06784e56
1 ## Breaking news! Completely refactored
2
3 ### New demonstration of planemo tool_factory command ![Planemo ToolFactory demonstration](images/lintplanemo-2021-01-08_18.02.45.mkv?raw=false "Demonstration inside Planemo")
4
5 # WARNING
6
7 Install this tool to a throw-away private Galaxy or Docker container ONLY!
8
9 Please NEVER on a public or production instance where a hostile user may
10 be able to gain access if they can acquire an administrative account login.
11
12 It only runs for server administrators - the ToolFactory tool will refuse to execute for an ordinary user since
13 it can install new tools to the Galaxy server it executes on! This is not something you should allow other than
14 on a throw away instance that is protected from potentially hostile users.
15
16 ## Short Story
17
18 Galaxy is easily extended to new applications by adding a new tool. Each new scientific computational package added as
19 a tool to Galaxy requires an XML document describing how the application interacts with Galaxy.
20 This is sometimes termed "wrapping" the package because the instructions tell Galaxy how to run the package
21 as a new Galaxy tool. Any tool that has been wrapped is readily available to all the users through a consistent
22 and easy to use interface once installed in the local Galaxy server.
23
24 Most Galaxy tool wrappers have been manually prepared by skilled programmers, many using Planemo because it
25 automates much of the boilerplate and makes the process much easier.
26 The ToolFactory (TF) now uses Planemo under the hood for testing, but hides the command
27 line complexities. The user will still need appropriate skills in terms of describing the interface between
28 Galaxy and the new application, but will be helped by a Galaxy tool form to collect all the needed
29 settings, together with automated testing and uploading to a toolshed with optional local installation.
30
31
32 ## ToolFactory generated tools are ordinary Galaxy tools
33
34 A TF generated tool that passes the Planemo test is ready to publish in any Galaxy Toolshed and ready to install in any running Galaxy instance.
35 They are fully workflow compatible and work exactly like any hand-written tool. The user can select input files of the specified type(s) from their
36 history and edit each of the specified parameters. The tool form will show all the labels and help text supplied when the tool was built. When the tool
37 is executed, the dependent binary or script will be passed all the i/o files and parameters as specified, and will write outputs to the specified new
38 history datasets - just like any other Galaxy tool.
39
40 ## Models for tool command line construction
41
42 The key to turning any software package into a Galaxy tool is the automated construction of a suitable command line.
43
44 The TF can build a new tool that will allow the tool user to select input files from their history, set any parameters and when run will send the
45 new output files to the history as specified when the tool builder completed the form and built the new tool.
46
47 That tool can contain instructions to run any Conda dependency or a system executable like bash. Whether a bash script you have written or
48 a Conda package like bwa, the executable will expect to find settings for input, output and parameters on a command line.
49
50 These are often passed as "--name value" (argparse style) or in a fixed order (positional style).
51
52 The ToolFactory allows either, or for "filter" applications that process input from STDIN and write processed output to STDOUT.
53
54 The simplest tool model wraps a simple script or Conda dependency package requiring only input and output files, with no user supplied settings illustrated by
55 the Tacrev demonstration tool found in the Galaxy running in the ToolFactory docker container. It passes a user selected input file from the current history on STDIN
56 to a bash script. The bash script runs the unix tac utility (reverse cat) piped to the unix rev (reverse lines in a text file) utility. It's a one liner:
57
58 `tac | rev`
59
60 The tool building form allows zero or more Conda package name(s) and version(s) and an optional script to be executed by either a system
61 executable like ``bash`` or the first of any named Conda dependency package/version. Tacrev uses a tiny bash script shown above and uses the system
62 bash. Conda bash can be specified if it is important to use the same version consistently for the tool.
63
64 On the tool form, the repeat section allowing zero or more input files was set to be a text file to be selected by the tool user and
65 in the repeat section allowing one or more outputs, a new output file with special value `STDOUT` as the positional parameter, causes the TF to
66 generate a command to capture STDOUT and send it to the new history file containing the reversed input text.
67
68 By reversed, we mean really, truly reversed.
69
70 That simple model can be made much more complicated, and can pass inputs and outputs as named or positional parameters,
71 to allow more complicated scripts or dependent binaries that require:
72
73 1. Any number of input data files selected by the user from existing history data
74 2. Any number of output data files written to the user's history
75 3. Any number of user supplied parameters. These can be passed as command line arguments to the script or the dependency package. Either
76 positional or named (argparse) style command line parameter passing can be used.
77
78 More complex models can be seen in the Sedtest, Pyrevpos and Pyrevargparse tools illustrating positional and argparse parameter passing.
79
80 The most complex demonstration is the Planemo advanced tool tutorial BWA tool. There is one version using a command-override to implement
81 exactly the same command structure in the Planemo tutorial. A second version uses a bash script and positional parameters to achieve the same
82 result. Some tool builders may find the bash version more familiar and cleaner but the choice is yours.
83
84 ## Overview
85
86 ![IHello example ToolFactory tool form](files/hello_toolfactory_form.png?raw=true "Part of the Hello world example ToolFactory tool form")
87
88
89 Steps in building a new Galaxy tool are all conducted through Galaxy running in the docker container:
90
91 1. Login to the Galaxy running in the container at http://localhost:8080 using an admin account. They are specified in config/galaxy.yml and
92 in the documentation at
93 and the ToolFactory will error out and refuse to run for non-administrative tool builders as a minimal protection from opportunistic hostile use.
94
95 2. Start the TF and fill in the form, providing sample inputs and parameter values to suit the Conda package being wrapped.
96
97 3. Execute the tool to create a new XML tool wrapper using the sample inputs and parameter settings for the inbuilt tool test. Planemo runs twice.
98 firstly to generate the test outputs and then to perform a proper test. The completed toolshed archive is written to the history
99 together with the planemo test report. Optionally the new tool archive can be uploaded
100 to the toolshed running in the same container (http://localhost:9009) and then installed inside the Galaxy in the container for further testing.
101
102 4. If the test fails, rerun the failed history job and correct errors on the tool form before rerunning until everything works correctly.
103
104 ![How it works](files/TFasIDE.png?raw=true "Overview of the ToolFactory as an Integrated Development Environment")
105
106 ## Planning and building new Galaxy tool wrappers.
107
108 It is best to have all the required planning done to wrap any new script or binary before firing up the TF.
109 Conda is the only current dependency manager supported. Before starting, at the very least, the tool builder will need
110 to know the required software package name in Conda and the version to use, how the command line for
111 the package must be constructed, and there must be sample inputs in the working history for each of the required data inputs
112 for the package, together with values for every parameter to suit these sample inputs. These are required on the TF form
113 for preparing the inbuilt tool test. That test is run using Planemo, as part of the tool generation process.
114
115 A new tool is specified by filling in the usual Galaxy tool form.
116
117 The form starts with a new tool name. Most tools will need dependency packages and versions
118 for the executable. Only Conda is currently supported.
119
120 If a script is needed, it can be pasted into a text box and the interpreter named. Available system executables
121 can be used such as bash, or an interpreter such as python, perl or R can be nominated as conda dependencies
122 to ensure reproducible analyses.
123
124 The tool form will be generated from the input data and the tool builder supplied parameters. The command line for the
125 executable is built using positional or argparse (named e.g. --input_file /foo/baz) style
126 parameters and is completely dependent on the executable. These can include:
127
128 1. Any number of input data sets needed by the executable. Each appears to the tool user on the run form and is included
129 on the command line for the executable. The tool builder must supply a small representative sample for each one as
130 an input for the automated tool test.
131
132 2. Any number of output data sets generated by the package can be added to the command line and will appear in
133 the user's history at the end of the job
134
135 3. Any number of text or numeric parameters. Each will appear to the tool user on the run form and are included
136 on the command line to the executable. The tool builder must supply a suitable representative value for each one as
137 the value to be used for the automated tool test.
138
139 Once the form is completed, executing the TF will build a new XML tool wrapper
140 including a functional test based on the sample settings and data.
141
142 If the Planemo test passes, the tool can be optionally uploaded to the local Galaxy used in the image for more testing.
143
144 A local toolshed runs inside the container to allow an automated installation, although any toolshed and any accessible
145 Galaxy can be specified for this process by editing the default URL and API keys to provide appropriate credentials.
146
147 ## Generated Tool Dependency management
148
149 Conda is used for all dependency management although tools that use system utilities like sed, bash or awk
150 may be available on job execution nodes. Sed and friends are available as Conda (conda-forge) dependencies if necessary.
151 Versioned Conda dependencies are always baked-in to the tool and will be used for reproducible calculation.
152
153 ## Requirements
154
155 These are all managed automagically. The TF relies on galaxyxml to generate tool xml and uses ephemeris and
156 bioblend to load tools to the toolshed and to Galaxy. Planemo is used for testing and runs in a biocontainer currently at
157 https://quay.io/fubar2/planemo-biocontainer
158
159 ## Caveats
160
161 This docker image requires privileged mode so exposes potential security risks if hostile tool builders gain access.
162 Please, do not run it in any situation where that is a problem - never, ever on a public facing Galaxy server.
163 On a laptop or workstation should be fine in a non-hostile environment.
164
165
166 ## Example generated XML
167
168 For the bwa-mem example, a supplied bash script is included as a configfile and so has escaped characters.
169 ```
170 <tool name="bwatest" id="bwatest" version="0.01">
171 <!--Cite: Creating re-usable tools from scripts doi:10.1093/bioinformatics/bts573-->
172 <!--Source in git at: https://github.com/fubar2/toolfactory-->
173 <!--Created by admin@galaxy.org at 30/11/2020 07:12:10 using the Galaxy Tool Factory.-->
174 <description>Planemo advanced tool building sample bwa mem mapper as a ToolFactory demo</description>
175 <requirements>
176 <requirement version="0.7.15" type="package">bwa</requirement>
177 <requirement version="1.3" type="package">samtools</requirement>
178 </requirements>
179 <configfiles>
180 <configfile name="runme"><![CDATA[
181 REFFILE=\$1
182 FASTQ=\$2
183 BAMOUT=\$3
184 rm -f "refalias"
185 ln -s "\$REFFILE" "refalias"
186 bwa index -a is "refalias"
187 bwa mem -t "2" -v 1 "refalias" "\$FASTQ" > tempsam
188 samtools view -Sb tempsam > temporary_bam_file.bam
189 samtools sort -o "\$BAMOUT" temporary_bam_file.bam
190
191 ]]></configfile>
192 </configfiles>
193 <version_command/>
194 <command><![CDATA[bash
195 $runme
196 $input1
197 $input2
198 $bam_output]]></command>
199 <inputs>
200 <param optional="false" label="Reference sequence for bwa to map the fastq reads against" help="" format="fasta" multiple="false" type="data" name="input1" argument="input1"/>
201 <param optional="false" label="Reads as fastqsanger to align to the reference sequence" help="" format="fastqsanger" multiple="false" type="data" name="input2" argument="input2"/>
202 </inputs>
203 <outputs>
204 <data name="bam_output" format="bam" label="bam_output" hidden="false"/>
205 </outputs>
206 <tests>
207 <test>
208 <output name="bam_output" value="bam_output_sample" compare="sim_size" format="bam" delta_frac="0.1"/>
209 <param name="input1" value="input1_sample"/>
210 <param name="input2" value="input2_sample"/>
211 </test>
212 </tests>
213 <help><![CDATA[
214
215 **What it Does**
216
217 Planemo advanced tool building sample bwa mem mapper
218
219 Reimagined as a bash script for a ToolFactory demonstration
220
221
222 ------
223
224 Script::
225
226 REFFILE=$1
227 FASTQ=$2
228 BAMOUT=$3
229 rm -f "refalias"
230 ln -s "$REFFILE" "refalias"
231 bwa index -a is "refalias"
232 bwa mem -t "2" -v 1 "refalias" "$FASTQ" > tempsam
233 samtools view -Sb tempsam > temporary_bam_file.bam
234 samtools sort -o "$BAMOUT" temporary_bam_file.bam
235
236 ]]></help>
237 </tool>
238
239 ```
240
241
242
243 ## More Explanation
244
245 The TF is an unusual Galaxy tool, designed to allow a skilled user to make new Galaxy tools.
246 It appears in Galaxy just like any other tool but outputs include new Galaxy tools generated
247 using instructions provided by the user and the results of Planemo lint and tool testing using
248 small sample inputs provided by the TF user. The small samples become tests built in to the new tool.
249
250 It offers a familiar Galaxy form driven way to define how the user of the new tool will
251 choose input data from their history, and what parameters the new tool user will be able to adjust.
252 The TF user must know, or be able to read, enough about the tool to be able to define the details of
253 the new Galaxy interface and the ToolFactory offers little guidance on that other than some examples.
254
255 Tools always depend on other things. Most tools in Galaxy depend on third party
256 scientific packages, so TF tools usually have one or more dependencies. These can be
257 scientific packages such as BWA or scripting languages such as Python and are
258 managed by Conda. If the new tool relies on a system utility such as bash or awk
259 where the importance of version control on reproducibility is low, these can be used without
260 Conda management - but remember the potential risks of unmanaged dependencies on computational
261 reproducibility.
262
263 The TF user can optionally supply a working script where scripting is
264 required and the chosen dependency is a scripting language such as Python or a system
265 scripting executable such as bash. Whatever the language, the script must correctly parse the command line
266 arguments it receives at tool execution, as they are defined by the TF user. The
267 text of that script is "baked in" to the new tool and will be executed each time
268 the new tool is run. It is highly recommended that scripts and their command lines be developed
269 and tested until proven to work before the TF is invoked. Galaxy as a software development
270 environment is actually possible, but not recommended being somewhat clumsy and inefficient.
271
272 Tools nearly always take one or more data sets from the user's history as input. TF tools
273 allow the TF user to define what Galaxy datatypes the tool end user will be able to choose and what
274 names or positions will be used to pass them on a command line to the package or script.
275
276 Tools often have various parameter settings. The TF allows the TF user to define how each
277 parameter will appear on the tool form to the end user, and what names or positions will be
278 used to pass them on the command line to the package. At present, parameters are limited to
279 simple text and number fields. Pull requests for other kinds of parameters that galaxyxml
280 can handle are welcomed.
281
282 Best practice Galaxy tools have one or more automated tests. These should use small sample data sets and
283 specific parameter settings so when the tool is tested, the outputs can be compared with their expected
284 values. The TF will automatically create a test for the new tool. It will use the sample data sets
285 chosen by the TF user when they built the new tool.
286
287 The TF works by exposing *unrestricted* and therefore extremely dangerous scripting
288 to all designated administrators of the host Galaxy server, allowing them to
289 run scripts in R, python, sh and perl. For this reason, a Docker container is
290 available to help manage the associated risks.
291
292 ## Scripting uses
293
294 To use a scripting language to create a new tool, you must first prepared and properly test a script. Use small sample
295 data sets for testing. When the script is working correctly, upload the small sample datasets
296 into a new history, start configuring a new ToolFactory tool, and paste the script into the script text box on the TF form.
297
298 ### Outputs
299
300 The TF will generate the new tool described on the TF form, and test it
301 using planemo. Optionally if a local toolshed is running, it can be used to
302 install the new tool back into the generating Galaxy.
303
304 A toolshed is built in to the Docker container and configured
305 so a tool can be tested, sent to that toolshed, then installed in the Galaxy
306 where the TF is running using the default toolshed and Galaxy URL and API keys.
307
308 Once it's in a ToolShed, it can be installed into any local Galaxy server
309 from the server administrative interface.
310
311 Once the new tool is installed, local users can run it - each time, the
312 package and/or script that was supplied when it was built will be executed with the input chosen
313 from the user's history, together with user supplied parameters. In other words, the tools you generate with the
314 TF run just like any other Galaxy tool.
315
316 TF generated tools work as normal workflow components.
317
318
319 ## Limitations
320
321 The TF is flexible enough to generate wrappers for many common scientific packages
322 but the inbuilt automation will not cope with all possible situations. Users can
323 supply overrides for two tool XML segments - tests and command and the BWA
324 example in the supplied samples workflow illustrates their use. It does not deal with
325 repeated elements or conditional parameters such as allowing a user to choose to see "simple"
326 or "advanced" parameters (yet) and there will be plenty of packages it just
327 won't cover - but it's a quick and efficient tool for the other 90% of cases. Perfect for
328 that bash one liner you need to get that workflow functioning correctly for this
329 afternoon's demonstration!
330
331 ## Installation
332
333 The Docker container https://github.com/fubar2/toolfactory-galaxy-docker/blob/main/README.md
334 is the best way to use the TF because it is preconfigured
335 to automate new tool testing and has a built in local toolshed where each new tool
336 is uploaded. If you grab the docker container, it should just work after a restart and you
337 can run a workflow to generate all the sample tools. Running the samples and rerunning the ToolFactory
338 jobs that generated them allows you to add fields and experiment to see how things work.
339
340 It can be installed like any other tool from the Toolshed, but you will need to make some
341 configuration changes (TODO write a configuration). You can install it most conveniently using the
342 administrative "Search and browse tool sheds" link. Find the Galaxy Main
343 toolshed at https://toolshed.g2.bx.psu.edu/ and search for the toolfactory
344 repository in the Tool Maker section. Open it and review the code and select the option to install it.
345
346 If not already there please add:
347
348 ```
349 <datatype extension="tgz" type="galaxy.datatypes.binary:Binary" mimetype="multipart/x-gzip" subclass="True" />
350 ```
351
352 to your local config/data_types_conf.xml.
353
354
355 ## Restricted execution
356
357 The tool factory tool itself will ONLY run for admin users -
358 people with IDs in config/galaxy.yml "admin_users".
359
360 *ONLY admin_users can run this tool*
361
362 That doesn't mean it's safe to install on a shared or exposed instance - please don't.
363
364 ## Generated tool Security
365
366 Once you install a generated tool, it's just
367 another tool - assuming the script is safe. They just run normally and their
368 user cannot do anything unusually insecure but please, practice safe toolshed.
369 Read the code before you install any tool. Especially this one - it is really scary.
370
371 ## Attribution
372
373 Creating re-usable tools from scripts: The Galaxy Tool Factory
374 Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team
375 Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573
376
377 http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref
378