|
1
|
1 # Versioned Data System
|
|
|
2 The Galaxy and command line Versioned Data System manages the retrieval of current and past versions of selected reference sequence databases from local data stores.
|
|
0
|
3
|
|
1
|
4 ---
|
|
0
|
5
|
|
1
|
6 0. Overview
|
|
|
7 1. [Setup for Admins](doc/setup.md)
|
|
|
8 1. [Galaxy tool installation](doc/galaxy_tool_install.md)
|
|
|
9 2. [Server data stores](doc/data_stores.md)
|
|
|
10 3. [Data store examples](doc/data_store_examples.md)
|
|
|
11 4. [Galaxy "Versioned Data" library setup](doc/galaxy_library.md)
|
|
|
12 5. [Workflow configuration](doc/workflows.md)
|
|
|
13 6. [Permissions, security, and maintenance](doc/maintenance.md)
|
|
|
14 7. [Problem solving](doc/problem_solving.md)
|
|
|
15 2. [Using the Galaxy Versioned data tool](doc/galaxy_tool.md)
|
|
|
16 3. [System Design](doc/design.md)
|
|
|
17 4. [Background Research](doc/background.md)
|
|
|
18 5. [Server data store and galaxy library organization](doc/data_store_org.md)
|
|
|
19 6. [Data Provenance and Reproducibility](doc/data_provenance.md)
|
|
|
20 7. [Caching System](doc/caching.md)
|
|
0
|
21
|
|
1
|
22 ---
|
|
|
23
|
|
|
24 ## Overview
|
|
|
25
|
|
|
26 This tool can be used on a server both via the command line and via the Galaxy bioinformatics workflow platform using the "Versioned Data" tool. Different kinds of content are suited to different archiving technologies, so the system provides a few storage system choices.
|
|
0
|
27
|
|
1
|
28 * Fasta sequences - accession ids, descriptions and their sequences - are suited to storage as 1 line key-value pair records in a key-value store. Here we introduce a low-tech file-based database plugin for this kind of data called **Kipper**. It is suited entirely to the goal of producing complete versioned files. This covers much of the sequencing archiving problem for reference databases. Consult https://github.com/Public-Health-Bioinformatics/kipper for up-to-date information on Kipper.
|
|
0
|
29
|
|
1
|
30 * A **git** archiving system plugin is also provided for software file tree archiving, with a particular file differential (diff) compression benefit for documents that have sentence-like lines added and deleted between versions.
|
|
|
31
|
|
|
32 * Super-large files that are not suited to Kipper or git can be handled by a simple "**folder**" data store holds each version of file(s) in a separate compressed archive.
|
|
0
|
33
|
|
1
|
34 * **Biomaj** (our reference database maintenance software) can be configured to download and store separate version files. A Biomaj plugin allows direct selection of versioned files within its "data bank" folders.
|
|
|
35
|
|
|
36 The Galaxy Versioned Data tool below, shows the interface for retrieving versions of reference database. The tool lets you select the fasta database to retrieve, and then one or more workflows. The system then generates and caches the versioned data in the data library; then links it into one's history; then runs the workflow(s) to get the derivative data (a Blast database say) and then caches that back into the data library. Future requests for that versioned data and derivatives (keyed by workflow id and input data version ids) will return the data already from cache rather than regenerating it, until the cache is deleted.
|
|
|
37
|
|
|
38 
|
|
0
|
39
|
|
1
|
40 ## Project goals
|
|
0
|
41
|
|
1
|
42 * **To enable reproducible molecular biology research:** To recreate a search result at a certain point in time we need versioning so that search and mapping tools can look at reference sequence databases corresponding to a particular past date or version identifier. This recall can also explain the difference between what was known in the past vs. currently.
|
|
|
43
|
|
|
44 * **To reduce hard drive space.** Some databases are too big to keep N copies around, e.g. 5 years of 16S, updated monthly, is say, 670Mb + 668Mb + 665Mb + .... (Compressing each file individually is an option but even better we could store just the differences between subsequent versions.)
|
|
0
|
45
|
|
1
|
46 * **Maximize speed of archive recall.** Understanding that the archived version files can be large, we'd ideally like a versioned file to be retrieved in the time it takes to write a file of that size to disk. Caching this data and its derivatives (makeblastdb databases for example) is important.
|
|
0
|
47
|
|
1
|
48 * **Improve sequence archive management.** Provide an admin interface for managing regular scheduled import and log of reference sequence databases from our own and 3rd party sources like NCBI and cpndb.ca .
|
|
0
|
49
|
|
1
|
50 * Integrate database versioning into the Galaxy workflow management software without adding a lot of complexity.
|
|
0
|
51
|
|
1
|
52 * A bonus would be to enable the efficient sharing of versioned data between computers/servers.
|
