Mercurial > repos > fubar > toolfactory

annotate README.txt @ 13:d5a5f9c2c108 draft

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
Uploaded
author fubar
date Thu, 20 Mar 2014 06:00:21 -0400
parents 1fcf3fda325f
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
7
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
1 # WARNING before you start
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
2 # Install this tool on a private Galaxy ONLY
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
3 # Please NEVER on a public or production instance
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
4
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
5 Please cite:
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
6 http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
7 if you use this tool in your published work.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
8
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
9 *Short Story*
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
10
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
11 This is an unusual Galaxy tool that exposes unrestricted and therefore extremely dangerous
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
12 scripting to designated administrative users of a Galaxy server, allowing them to run scripts
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
13 in R, python, sh and perl over a single input data set, writing a single new data set as output.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
14
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
15 In addition, this tool optionally generates very simple new Galaxy tools, that effectively
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
16 freeze the supplied script into a new, ordinary Galaxy tool that runs it over a single input file,
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
17 working just like any other Galaxy tool for your users.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
18
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
19 To use the ToolFactory, you should have prepared a script to paste into a text box,
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
20 and a small test input example ready to select from your history to test your new script.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
21 There is an example in each scripting language on the Tool Factory form. You can just
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
22 cut and paste these to try it out - remember to select the right interpreter please. You'll
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
23 also need to create a small test data set using the Galaxy history add new data tool.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
24
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
25 If the script fails somehow, use the "redo" button on the tool output in your history to
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
26 recreate the form complete with broken script. Fix the bug and execute again. Rinse, wash, repeat.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
27
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
28 Once the script runs sucessfully, a new Galaxy tool that runs your script can be generated.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
29 Select the "generate" option and supply some help text and names. The new tool will be
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
30 generated in the form of a new Galaxy datatype - toolshed.gz - as the name suggests,
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
31 it's an archive ready to upload to a Galaxy ToolShed as a new tool repository.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
32
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
33 Once it's in a ToolShed, it can be installed into any local Galaxy server from
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
34 the server administrative interface.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
35
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
36 Once the new tool is installed, local users can run it - each time, the script that was supplied
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
37 when it was built will be executed with the input chosen from the user's history. In other words,
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
38 the tools you generate with the ToolFactory run just like any other Galaxy tool,
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
39 but run your script every time.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
40
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
41 Tool factory tools are perfect for workflow components. One input, one output, no variables.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
42
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
43 *Reasons to read further*
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
44
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
45 If you use Galaxy to support your research;
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
46
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
47 You and fellow users are sometimes forced to take data out of Galaxy, process it with ugly
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
48 little perl/awk/sed/R... scripts and put it back;
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
49
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
50 You do this when you can't do some transformation in Galaxy (the 90/10 rule);
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
51
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
52 You don't have enough developer resources for wrapping dozens of even relatively simple tools;
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
53
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
54 Your research and your institution would be far better off if those feral scripts were all tucked
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
55 safely in your local toolshed and Galaxy histories.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
56
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
57 *The good news* If it can be trivially scripted, it can be running safely in your
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
58 local Galaxy via your own local toolshed in a few minutes - with functional tests.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
59
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
60
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
61 *Value proposition* The ToolFactory allows Galaxy to efficiently take over most of your lab's
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
62 dark script matter, making it reproducible in Galaxy and shareable through the ToolShed.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
63
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
64 That's what this tool does. You paste a simple script and the tool returns
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
65 a new, real Galaxy tool, ready to be installed from the local toolshed to local servers.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
66 Scripts can be wrapped and online literally within minutes.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
67
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
68 *To fully and safely exploit the awesome power* of this tool, Galaxy and the ToolShed,
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
69 you should be a developer installing this tool on a private/personal/scratch local instance where you
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
70 are an admin_user. Then, if you break it, you get to keep all the pieces
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
71 see https://bitbucket.org/fubar/galaxytoolfactory/wiki/Home
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
72
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
73 ** Installation **
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
74 This is a Galaxy tool. You can install it most conveniently using the administrative "Search and browse tool sheds" link.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
75 Find the Galaxy Test toolshed (not main) and search for the toolfactory repository.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
76 Open it and review the code and select the option to install it.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
77
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
78 If you can't get the tool that way, the xml and py files here need to be copied into a new tools
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
79 subdirectory such as tools/toolfactory Your tool_conf.xml needs a new entry pointing to the xml
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
80 file - something like::
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
81
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
82 <section name="Tool building tools" id="toolbuilders">
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
83 <tool file="toolfactory/rgToolFactory.xml"/>
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
84 </section>
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
85
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
86 If not already there (I just added it to datatypes_conf.xml.sample), please add:
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
87 <datatype extension="toolshed.gz" type="galaxy.datatypes.binary:Binary" mimetype="multipart/x-gzip" subclass="True" />
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
88 to your local data_types_conf.xml.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
89
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
90 Ensure that html sanitization is set to False and uncommented in universe_wsgi.ini
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
91
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
92 You'll have to restart the server for the new tool to be available.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
93
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
94 Of course, R, python, perl etc are needed on your path if you want to test scripts using those interpreters.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
95 Adding new ones to this tool code should be easy enough. Please make suggestions as bitbucket issues and code.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
96 The HTML file code automatically shrinks R's bloated pdfs, and depends on ghostscript. The thumbnails require imagemagick .
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
97
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
98 * Restricted execution *
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
99 The new tool factory tool will then be usable ONLY by admin users - people with IDs in admin_users in universe_wsgi.ini
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
100 **Yes, that's right. ONLY admin_users can run this tool** Think about it for a moment. If allowed to run any
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
101 arbitrary script on your Galaxy server, the only thing that would impede a miscreant bent on destroying all your
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
102 Galaxy data would probably be lack of appropriate technical skills.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
103
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
104 *What it does* This is a tool factory for simple scripts in python, R and perl currently.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
105 Functional tests are automatically generated. How cool is that.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
106
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
107 LIMITED to simple scripts that read one input from the history.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
108 Optionally can write one new history dataset,
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
109 and optionally collect any number of outputs into links on an autogenerated HTML
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
110 index page for the user to navigate - useful if the script writes images and output files - pdf outputs
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
111 are shown as thumbnails and R's bloated pdf's are shrunk with ghostscript so that and imagemagik need to
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
112 be avaailable.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
113
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
114 Generated tools can be edited and enhanced like any Galaxy tool, so start small and build up since
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
115 a generated script gets you a serious leg up to a more complex one.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
116
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
117 *What you do* You paste and run your script
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
118 you fix the syntax errors and eventually it runs
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
119 You can use the redo button and edit the script before
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
120 trying to rerun it as you debug - it works pretty well.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
121
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
122 Once the script works on some test data, you can
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
123 generate a toolshed compatible gzip file
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
124 containing your script ready to run as an ordinary Galaxy tool in a
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
125 repository on your local toolshed. That means safe and largely automated installation in any
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
126 production Galaxy configured to use your toolshed.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
127
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
128 *Generated tool Security* Once you install a generated tool, it's just
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
129 another tool - assuming the script is safe. They just run normally and their user cannot do anything unusually insecure
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
130 but please, practice safe toolshed.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
131 Read the fucking code before you install any tool.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
132 Especially this one - it is really scary.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
133
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
134 If you opt for an HTML output, you get all the script outputs arranged
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
135 as a single Html history item - all output files are linked, thumbnails for all the pdfs.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
136 Ugly but really inexpensive.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
137
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
138 Patches and suggestions welcome as bitbucket issues please?
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
139
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
140 long route to June 2012 product
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
141 derived from an integrated script model
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
142 called rgBaseScriptWrapper.py
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
143 Note to the unwary:
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
144 This tool allows arbitrary scripting on your Galaxy as the Galaxy user
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
145 There is nothing stopping a malicious user doing whatever they choose
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
146 Extremely dangerous!!
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
147 Totally insecure. So, trusted users only
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
148
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
149
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
150
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
151
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
152 copyright ross lazarus (ross stop lazarus at gmail stop com) May 2012
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
153
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
154 all rights reserved
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
155 Licensed under the LGPL if you want to improve it, feel free https://bitbucket.org/fubar/galaxytoolfactory/wiki/Home
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
156
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
157 Material for our more enthusiastic and voracious readers continues below - we salute you.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
158
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
159 **Motivation** Simple transformation, filtering or reporting scripts get written, run and lost every day in most busy labs
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
160 - even ours where Galaxy is in use. This 'dark script matter' is pervasive and generally not reproducible.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
161
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
162 **Benefits** For our group, this allows Galaxy to fill that important dark script gap - all those "small" bioinformatics
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
163 tasks. Once a user has a working R (or python or perl) script that does something Galaxy cannot currently do (eg transpose a
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
164 tabular file) and takes parameters the way Galaxy supplies them (see example below), they:
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
165
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
166 1. Install the tool factory on a personal private instance
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
167
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
168 2. Upload a small test data set
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
169
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
170 3. Paste the script into the 'script' text box and iteratively run the insecure tool on test data until it works right -
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
171 there is absolutely no reason to do this anywhere other than on a personal private instance.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
172
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
173 4. Once it works right, set the 'Generate toolshed gzip' option and run it again.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
174
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
175 5. A toolshed style gzip appears ready to upload and install like any other Toolshed entry.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
176
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
177 6. Upload the new tool to the toolshed
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
178
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
179 7. Ask the local admin to check the new tool to confirm it's not evil and install it in the local production galaxy
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
180
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
181 **Simple examples on the tool form**
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
182
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
183 A simple Rscript "filter" showing how the command line parameters can be handled, takes an input file,
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
184 does something (transpose in this case) and writes the results to a new tabular file::
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
185
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
186 # transpose a tabular input file and write as a tabular output file
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
187 ourargs = commandArgs(TRUE)
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
188 inf = ourargs[1]
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
189 outf = ourargs[2]
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
190 inp = read.table(inf,head=F,row.names=NULL,sep='\t')
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
191 outp = t(inp)
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
192 write.table(outp,outf, quote=FALSE, sep="\t",row.names=F,col.names=F)
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
193
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
194 Calculate a multiple test adjusted p value from a column of p values - for this script to be useful,
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
195 it needs the right column for the input to be specified in the code for the
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
196 given input file type(s) specified when the tool is generated ::
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
197
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
198 # use p.adjust - assumes a HEADER row and column 1 - please fix for any real use
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
199 column = 1 # adjust if necessary for some other kind of input
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
200 fdrmeth = 'BH'
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
201 ourargs = commandArgs(TRUE)
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
202 inf = ourargs[1]
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
203 outf = ourargs[2]
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
204 inp = read.table(inf,head=T,row.names=NULL,sep='\t')
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
205 p = inp[,column]
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
206 q = p.adjust(p,method=fdrmeth)
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
207 newval = paste(fdrmeth,'p-value',sep='_')
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
208 q = data.frame(q)
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
209 names(q) = newval
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
210 outp = cbind(inp,newval=q)
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
211 write.table(outp,outf, quote=FALSE, sep="\t",row.names=F,col.names=T)
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
212
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
213
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
214
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
215 Another Rscript example without any input file - generates a random heatmap pdf - you must make sure the option to create an HTML output file is
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
216 turned on for this to work. The heatmap will be presented as a thumbnail linked to the pdf in the resulting HTML page::
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
217
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
218 # note this script takes NO input or output because it generates random data
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
219 foo = data.frame(a=runif(100),b=runif(100),c=runif(100),d=runif(100),e=runif(100),f=runif(100))
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
220 bar = as.matrix(foo)
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
221 pdf( "heattest.pdf" )
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
222 heatmap(bar,main='Random Heatmap')
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
223 dev.off()
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
224
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
225 A Python example that reverses each row of a tabular file. You'll need to remove the leading spaces for this to work if cut
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
226 and pasted into the script box. Note that you can already do this in Galaxy by setting up the cut columns tool with the
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
227 correct number of columns in reverse order,but this script will work for any number of columns so is completely generic::
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
228
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
229 # reverse order of columns in a tabular file
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
230 import sys
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
231 inp = sys.argv[1]
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
232 outp = sys.argv[2]
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
233 i = open(inp,'r')
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
234 o = open(outp,'w')
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
235 for row in i:
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
236 rs = row.rstrip().split('\t')
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
237 rs.reverse()
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
238 o.write('\t'.join(rs))
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
239 o.write('\n')
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
240 i.close()
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
241 o.close()
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
242
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
243
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
244 Galaxy as an IDE for developing API scripts
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
245 If you need to develop Galaxy API scripts and you like to live dangerously, please read on.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
246
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
247 Galaxy as an IDE?
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
248 Amazingly enough, blend-lib API scripts run perfectly well *inside* Galaxy when pasted into a Tool Factory form. No need to generate a new tool. Galaxy+Tool_Factory = IDE I think we need a new t-shirt. Seriously, it is actually quite useable.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
249
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
250 Why bother - what's wrong with Eclipse
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
251 Nothing. But, compared with developing API scripts in the usual way outside Galaxy, you get persistence and other framework benefits plus at absolutely no extra charge, a ginormous security problem if you share the history or any outputs because they contain the api script with key so development servers only please!
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
252
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
253 Workflow
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
254 Fire up the Tool Factory in Galaxy.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
255
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
256 Leave the input box empty, set the interpreter to python, paste and run an api script - eg working example (substitute the url and key) below.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
257
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
258 It took me a few iterations to develop the example below because I know almost nothing about the API. I started with very simple code from one of the samples and after each run, the (edited..) api script is conveniently recreated using the redo button on the history output item. So each successive version of the developing api script you run is persisted - ready to be edited and rerun easily. It is ''very'' handy to be able to add a line of code to the script and run it, then view the output to (eg) inspect dicts returned by API calls to help move progressively deeper iteratively.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
259
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
260 Give the below a whirl on a private clone (install the tool factory from the main toolshed) and try adding complexity with few rerun/edit/rerun cycles.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
261
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
262 Eg tool factory api script
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
263 import sys
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
264 from blend.galaxy import GalaxyInstance
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
265 ourGal = 'http://x.x.x.x:xxxx'
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
266 ourKey = 'xxx'
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
267 gi = GalaxyInstance(ourGal, key=ourKey)
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
268 libs = gi.libraries.get_libraries()
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
269 res = []
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
270 # libs looks like
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
271 # u'url': u'/galaxy/api/libraries/441d8112651dc2f3', u'id': u'441d8112651dc2f3', u'name':.... u'Demonstration sample RNA data',
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
272 for lib in libs:
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
273 res.append('%s:\n' % lib['name'])
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
274 res.append(str(gi.libraries.show_library(lib['id'],contents=True)))
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
275 outf=open(sys.argv[2],'w')
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
276 outf.write('\n'.join(res))
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
277 outf.close()
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
278
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
279 **Attribution**
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
280 Creating re-usable tools from scripts: The Galaxy Tool Factory
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
281 Ross Lazarus; Antony Kaspi; Mark Ziemann; The Galaxy Team
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
282 Bioinformatics 2012; doi: 10.1093/bioinformatics/bts573
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
283
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
284 http://bioinformatics.oxfordjournals.org/cgi/reprint/bts573?ijkey=lczQh1sWrMwdYWJ&keytype=ref
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
285
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
286 **Licensing**
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
287 Copyright Ross Lazarus 2010
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
288 ross lazarus at g mail period com
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
289
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
290 All rights reserved.
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
291
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
292 Licensed under the LGPL
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
293
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
294 **Obligatory screenshot**
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
295
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
296 http://bitbucket.org/fubar/galaxytoolmaker/src/fda8032fe989/images/dynamicScriptTool.png
1fcf3fda325f major fixes with GS and graphicsmagick dependencies included
fubar
parents:
diff changeset
297