annotate smalt/README.md @ 10:b75227b2833e draft default tip

Uploaded
author triasteran
date Tue, 08 Mar 2022 11:44:09 +0000
parents 81f36745bc9d
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
9
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
1 Integration of SMALT into Galaxy and Docker
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
2 ===========================================
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
3
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
4 This describes a way to construct a docker image and modify a tool within Galaxy to work with Docker. The tool I use is SMALT from https://toolshed.g2.bx.psu.edu/repository/find_tools?sort=Repository.name&operation=view_or_manage_repository&id=61631a13f8a13237.
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
5
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
6 1. Building a Docker Image
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
7 ==========================
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
8
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
9 There are two different methods to construct a docker image for a tool:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
10
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
11 1. Interactively
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
12 2. Using a `Dockerfile`
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
13
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
14 1. Building a Docker Image Interactively
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
15 ----------------------------------------
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
16
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
17 The interactive method of constructing a docker image is a good method for installing all the dependencies for a tool if you're not quite sure exactly what commands you need to run to get the tool working. This method involves starting up a docker container, and running commands within this container to get your tool working. Then, you can **commit** this container to an image and save to dockerhub for re-use.
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
18
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
19 ### Step 1: Starting up a Docker base Image
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
20
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
21 To build a docker image we need a starting point. The base image we will use is **debian/wheezy** since it contains `apt-get` for installing more software. To start up this image in an interactive mode please run:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
22
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
23 ```bash
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
24 $ sudo docker run -i -t debian:wheezy
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
25 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
26
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
27 This should bring up a prompt that looks like:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
28
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
29 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
30 root@2c1077a38bce:/#
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
31 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
32
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
33 Note that the id `2c1077a38bce` can be used later for saving this container to an image.
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
34
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
35 ### Step 2: Installing Basic Dependencies
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
36
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
37 A few dependencies are required to install SMALT. These can be installed with:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
38
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
39 ```bash
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
40 $ apt-get update
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
41 $ apt-get install python
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
42 $ apt-get install wget
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
43 $ apt-get install mercurial
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
44 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
45
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
46 ### Step 3: Installing SMALT
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
47
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
48 We will install SMALT by downloading the necessary software and copying to `/usr/bin`. This can be accomplished with:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
49
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
50 ```bash
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
51 $ mkdir tool
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
52 $ cd tool
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
53
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
54 $ wget ftp://ftp.sanger.ac.uk/pub4/resources/software/smalt/smalt-0.7.3.tgz
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
55 $ tar -xvvzf smalt-0.7.3.tgz
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
56
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
57 # because the smalt_wrapper.py finds the binary name based on `uname -i` which is unknown in docker
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
58 $ mv smalt-0.7.3/smalt_x86_64 smalt_unknown
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
59
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
60 $ hg clone https://toolshed.g2.bx.psu.edu/repos/cjav/smalt smalt_deps
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
61 $ cp smalt_deps/smalt_wrapper.py .
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
62
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
63 # add smalt tools to /usr/bin so they're on the PATH
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
64 $ ln -s /tool/smalt_unknown /usr/bin
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
65 $ ln -s /tool/smalt_wrapper.py /usr/bin
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
66
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
67 # make smalt_wrapper executable
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
68 $ chmod a+x /tool/smalt_wrapper.py
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
69 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
70
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
71 You can test out SMALT by trying to run `smalt_unknown` and `smalt_wrapper.py` in the docker container.
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
72
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
73 ### Step 4: Building an Image
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
74
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
75 To build the docker image from the container, please exit the container and run the following:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
76
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
77 ```bash
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
78 $ sudo docker commit -m "make smalt image" -a "Aaron Petkau" 2c1077a38bce apetkau/smalt-galaxy
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
79 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
80
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
81 Please fill in the appropriate information for your image. In particular, make sure the container id `2c1077a38bce` is correct.
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
82
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
83 To push this image to dockerhub you can run:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
84
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
85 ```bash
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
86 $ sudo docker push apetkau/smalt-galaxy
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
87 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
88
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
89 2. Building an image using a `Dockerfile`
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
90 -----------------------------------------
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
91
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
92 Alternatively, instead of building an image interactively, you can build an image with a `Dockerfile`. An example Dockerfile can be found in this repository at [Dockerfile](Dockerfile). To build an image please run:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
93
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
94 ```bash
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
95 $ sudo docker build -t apetkau/smalt-galaxy .
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
96 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
97
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
98 More information on Dockerfiles can be found at https://docs.docker.com/reference/builder/.
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
99
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
100 2. Installing Tool configuration
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
101 ================================
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
102
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
103 ### Installing Example file
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
104
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
105 Once a Docker image has been built, it can be integrated into a tool by modifying the tool configuration XML file. For SMALT, the configuration file is [smalt_wrapper.xml](smalt_wrapper.xml). This is based on https://toolshed.g2.bx.psu.edu/repos/cjav/smalt/file/54855bd8d107/smalt_wrapper.xml. And can be installed by running:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
106
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
107 ```bash
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
108 $ cp smalt_wrapper.xml galaxy-central/tools/docker/smalt_wrapper.xml
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
109 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
110
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
111 Then, please add this tool to the `tool_conf.xml` by adding:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
112
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
113 ```xml
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
114 <tool file="docker/smalt_wrapper.xml"/>
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
115 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
116
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
117 ### List of Changes
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
118
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
119 The exact changes you I needed to make are:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
120
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
121 1. I added the specific docker image name to the requirements by changing:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
122
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
123 ```xml
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
124 <requirements>
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
125 <requirement type="package" version="0.7.3">smalt</requirement>
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
126 </requirements>
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
127 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
128
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
129 To
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
130
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
131 ```xml
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
132 <requirements>
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
133 <container type="docker">apetkau/smalt-galaxy</container>
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
134 </requirements>
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
135 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
136
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
137 2. I had to remove `interpreter` from the command attribute by changing
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
138
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
139 ```xml
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
140 <command interpreter="python">
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
141 smalt_wrapper.py
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
142 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
143
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
144 To
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
145
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
146 ```xml
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
147 <command>
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
148 smalt_wrapper.py
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
149 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
150
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
151 3. Running Galaxy
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
152 =================
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
153
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
154 Once the tool is installed, please run Galaxy. And test out the tool. Some example files (reference.fasta and reads.fastq) are included. To make sure it's running in docker you can look for the following `sudo docker run` in the logs:
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
155
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
156 ```
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
157 galaxy.jobs.runners DEBUG 2014-06-28 16:50:00,930 (18) command is: sudo docker run -e "GALAXY_SLOTS=$GALAXY_SLOTS" -v /home/aaron/Projects/galaxy-central:/home/aaron/Proj
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
158 ects/galaxy-central:ro -v /home/aaron/Projects/galaxy-central/tools/docker:/home/aaron/Projects/galaxy-central/tools/docker:ro -v /home/aaron/Projects/galaxy-central/data
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
159 base/job_working_directory/000/18:/home/aaron/Projects/galaxy-central/database/job_working_directory/000/18:rw -v /home/aaron/Projects/galaxy-central/database/files:/home
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
160 /aaron/Projects/galaxy-central/database/files:rw -w /home/aaron/Projects/galaxy-central/database/job_working_directory/000/18 --net none apetkau/smalt:v3 /home/aaron/Proj
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
161 ects/galaxy-central/database/job_working_directory/000/18/container.sh
81f36745bc9d Uploaded
triasteran
parents:
diff changeset
162 ```