Mercurial > repos > scottx611x > data_manager_fetch_gene_annotation

annotate data_manager/data_manager.py @ 46:9346d2955707 draft

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
author iuc
date Tue, 04 Apr 2017 18:07:39 -0400
parents
children 3a02b8ab423a
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
46
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
1 import argparse
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
2 import datetime
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
3 import json
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
4 import os
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
5 import sys
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
6 import uuid
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
7
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
8 import requests
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
9 from requests.exceptions import ContentDecodingError
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
10
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
11
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
12 def url_download(url):
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
13 """Attempt to download gene annotation file from a given url
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
14 :param url: full url to gene annotation file
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
15 :type url: str.
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
16 :returns: name of downloaded gene annotation file
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
17 :raises: ContentDecodingError, IOError
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
18 """
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
19 response = requests.get(url=url, stream=True)
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
20
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
21 # Generate file_name
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
22 file_name = response.url.split("/")[-1]
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
23
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
24 block_size = 10 * 1024 * 1024 # 10MB chunked download
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
25 with open(file_name, 'w+') as f:
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
26 try:
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
27 # Good to note here that requests' iter_content() will
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
28 # automatically handle decoding "gzip" and "deflate" encoding
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
29 # formats
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
30 for buf in response.iter_content(block_size):
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
31 f.write(buf)
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
32 except (ContentDecodingError, IOError) as e:
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
33 sys.stderr.write("Error occured downloading reference file: %s"
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
34 % e)
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
35 os.remove(file_name)
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
36
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
37 return file_name
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
38
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
39
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
40 def main():
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
41 parser = argparse.ArgumentParser(description='Create data manager JSON.')
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
42 parser.add_argument('--out', dest='output', action='store',
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
43 help='JSON filename')
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
44 parser.add_argument('--name', dest='name', action='store',
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
45 default=uuid.uuid4(), help='Data table entry unique ID'
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
46 )
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
47 parser.add_argument('--url', dest='url', action='store',
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
48 help='Url to download gtf file from')
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
49
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
50 args = parser.parse_args()
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
51
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
52 work_dir = os.getcwd()
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
53
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
54 # Attempt to download gene annotation file from given url
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
55 gene_annotation_file_name = url_download(args.url)
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
56
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
57 # Update Data Manager JSON and write to file
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
58 data_manager_entry = {
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
59 'data_tables': {
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
60 'gff_gene_annotations': {
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
61 'value': str(datetime.datetime.now()),
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
62 'dbkey': str(args.name),
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
63 'name': gene_annotation_file_name,
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
64 'path': os.path.join(work_dir, gene_annotation_file_name)
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
65 }
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
66 }
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
67 }
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
68
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
69 with open(os.path.join(args.output), "w+") as f:
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
70 f.write(json.dumps(data_manager_entry))
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
71
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
72
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
73 if __name__ == '__main__':
9346d2955707 planemo upload for repository https://github.com/galaxyproject/tools-iuc/tree/master/data_managers/data_manager_fetch_gene_annotation/ commit 8652f36a3a3838dca989426961561e81432acf4f
iuc
parents:
diff changeset
74 main()