Mercurial > repos > jay > gaiac_heatmap

annotate gaiac_pm_data_pulling/gaiac_pm_data_pulling.py @ 1:e35898450a3c draft

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
author jay
date Thu, 15 May 2025 22:49:12 +0000
parents
children 6f9ed7f12aad
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
1 #!/usr/bin/env python
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
2
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
3
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
4 import argparse
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
5 import logging
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
6 from base64 import urlsafe_b64encode
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
7 from html import escape
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
8 from urllib.parse import urljoin
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
9 import requests
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
10
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
11 log = logging.getLogger("tools.iuc.data_managers.data_manager_refgenie_pull")
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
12
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
13 import urllib.request
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
14 import json
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
15
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
16 def get_file_list(ip):
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
17
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
18 url = "%s/list-files"%(ip)
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
19
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
20 with urllib.request.urlopen(url) as response:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
21 data = response.read()
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
22 file_list = json.loads(data)
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
23
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
24 return file_list
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
25
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
26
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
27 def simulate_dynamic_options(ip):
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
28 try:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
29 rval = []
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
30
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
31 api_data = {ip:{
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
32
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
33 }}
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
34
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
35 for i in get_file_list(ip):
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
36 # print("http://"+ip+'/download/'+i)
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
37 api_data[ip][i] = ip+'/download/'+i
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
38
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
39
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
40 ul = []
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
41 for dataset, url in api_data[ip].items():
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
42
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
43 ul.append({
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
44 'name': dataset,
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
45 'value': url,
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
46 'options': [],
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
47 'selected': False
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
48 })
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
49
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
50 # for urlname, genomes in api_data.items():
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
51 # # URL-safe base64 encoding for the URL name
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
52 # urlname_64 = urlname
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
53 # ul = []
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
54 # for genome, assets in genomes.items():
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
55 # al = []
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
56
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
57 # ul.append({
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
58 # 'name': genome,
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
59 # 'value': genome,
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
60 # 'options': [],
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
61 # 'selected': False
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
62 # })
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
63
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
64 return ul
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
65 except Exception as e:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
66 # Handle any potential errors
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
67 return [{
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
68 'name': escape(str(e)),
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
69 'value': 'ERROR',
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
70 'options': [],
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
71 'selected': False
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
72 }]
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
73
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
74 if __name__ == '__main__':
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
75
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
76 simulate_dynamic_options('http://192.168.1.202:8080')
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
77
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
78 # for i in simulate_dynamic_options('192.168.1.202:8080')[0]['options']:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
79 # print(i)
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
80 # print(i)
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
81 # parser = argparse.ArgumentParser()
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
82 # parser.add_argument('-n', '--names', dest='names', action='store', default=None, help='Table names to reload')
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
83 # parser.add_argument('-u', '--url', dest='url', action='store', default=None, help='Base url for reload')
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
84 # parser.add_argument('-k', '--key', dest='key', action='store', default=None, help='Galaxy API Key')
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
85 # parser.add_argument('-g', '--graceful', dest='graceful', action='store_true', help='Fail gracefully')
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
86
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
87 # args = parser.parse_args()
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
88 # try:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
89 # if not args.names:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
90 # tables = requests.get(urljoin(args.url, "api/tool_data"), params={'key': args.key}).json()
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
91 # args.names = [d.get('name') for d in tables]
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
92 # for name in args.names:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
93 # print(requests.get(urljoin(args.url, "api/tool_data/%s/reload" % (name)), params={'key': args.key}).json())
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
94 # except Exception as e:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
95 # if args.graceful:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
96 # print("Failed to reload data tables:\n%s" % (e))
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
97 # else:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
98 # raise e