Mercurial > repos > jay > gaiac_heatmap

annotate gaiac_pm_data_pulling/gaiac_pm_data_pulling.py @ 3:ea3a11b428d9 draft default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
planemo upload for repository https://github.com/jaidevjoshi83/gaiac commit e9587f93346c7b55e1be00bad5844bf2db3ed03d-dirty
author jay
date Thu, 10 Jul 2025 19:39:56 +0000
parents 6f9ed7f12aad
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
1
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
1 #!/usr/bin/env python
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
2
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
3
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
4 import argparse
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
5 import logging
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
6 from base64 import urlsafe_b64encode
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
7 from html import escape
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
8 from urllib.parse import urljoin
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
9 import requests
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
10 import urllib.request
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
11 import json
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
12
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
13 def get_file_list(ip):
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
14
2
6f9ed7f12aad planemo upload for repository https://github.com/jaidevjoshi83/gaiac commit c31ef5726cd1d2659da9aeb956c22fb834b177ff
jay
parents: 1
diff changeset
15 url = "%s/api/list-files"%(ip)
1
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
16
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
17 with urllib.request.urlopen(url) as response:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
18 data = response.read()
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
19 file_list = json.loads(data)
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
20
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
21 return file_list
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
22
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
23
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
24 def simulate_dynamic_options(ip):
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
25 try:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
26 rval = []
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
27
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
28 api_data = {ip:{
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
29
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
30 }}
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
31
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
32 for i in get_file_list(ip):
2
6f9ed7f12aad planemo upload for repository https://github.com/jaidevjoshi83/gaiac commit c31ef5726cd1d2659da9aeb956c22fb834b177ff
jay
parents: 1
diff changeset
33 api_data[ip][i] = ip+'/api/download/'+i
1
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
34
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
35 ul = []
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
36 for dataset, url in api_data[ip].items():
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
37
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
38 ul.append({
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
39 'name': dataset,
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
40 'value': url,
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
41 'options': [],
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
42 'selected': False
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
43 })
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
44
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
45 return ul
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
46 except Exception as e:
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
47 # Handle any potential errors
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
48 return [{
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
49 'name': escape(str(e)),
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
50 'value': 'ERROR',
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
51 'options': [],
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
52 'selected': False
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
53 }]
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
54
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
55 if __name__ == '__main__':
e35898450a3c planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit a733fec1915a1f6cf5f07092dc33426f765b6cb3-dirty
jay
parents:
diff changeset
56
3
ea3a11b428d9 planemo upload for repository https://github.com/jaidevjoshi83/gaiac commit e9587f93346c7b55e1be00bad5844bf2db3ed03d-dirty
jay
parents: 2
diff changeset
57 print(simulate_dynamic_options('http://iotprojects.ddns.net'))