Mercurial > repos > astroteam > crbeam_astro_tool

annotate cache.py @ 0:f40d05521dca draft default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
author astroteam
date Fri, 25 Apr 2025 19:33:20 +0000
parents
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
1 import os
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
2 import json
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
3
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
4 from minio import Minio
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
5
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
6 # from minio.error import S3Error
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
7
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
8
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
9 class CRbeamCache(object):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
10 credentials_env_var = "S3_CREDENTIALS"
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
11 bucket_name = "crbeam"
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
12 chunk_size = 32 * 1024
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
13
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
14 # default credentials
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
15 endpoint = "play.min.io"
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
16 access_key = "Q3AM3UQ867SPQQA43P2F"
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
17 secret_key = "zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG"
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
18
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
19 def __init__(self) -> None:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
20 credentials = os.getenv(self.credentials_env_var)
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
21 if credentials is not None:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
22 credentials = json.loads(credentials)
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
23 self.endpoint = credentials["endpoint"]
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
24 self.access_key = credentials["access_key"]
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
25 self.secret_key = credentials["secret_key"]
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
26
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
27 self.client = Minio(
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
28 self.endpoint,
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
29 access_key=self.access_key,
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
30 secret_key=self.secret_key,
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
31 )
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
32
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
33 # Make bucket if not exist.
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
34 if not self.client.bucket_exists(self.bucket_name):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
35 self.client.make_bucket(self.bucket_name)
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
36
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
37 def save(self, obj_name, file_path, append_mode=True, **params):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
38 if append_mode:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
39 idx = 0
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
40 for _ in self.client.list_objects(
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
41 self.bucket_name, prefix=obj_name
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
42 ):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
43 idx += 1
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
44 obj_path = obj_name + f"-{idx}"
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
45 else:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
46 obj_path = obj_name
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
47 self.client.fput_object(
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
48 self.bucket_name, obj_path, file_path, metadata=params
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
49 )
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
50 return obj_path
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
51
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
52 def get_cache_size(self, prefix):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
53 size = 0
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
54 for obj in self.client.list_objects(self.bucket_name, prefix=prefix):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
55 size += int(
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
56 obj.object_name[len(prefix):].split("-")[0]
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
57 ) # todo: load number of particles from metadata
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
58 return size
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
59
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
60 def load_file(self, output_path, obj_name):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
61 try:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
62 response = self.client.get_object(self.bucket_name, obj_name)
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
63 except Exception:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
64 raise ValueError("object not found")
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
65 try:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
66 # Read data from response.
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
67 with open(output_path, "wb") as file:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
68 for d in response.stream(self.chunk_size):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
69 file.write(d)
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
70 finally:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
71 response.close()
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
72 response.release_conn()
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
73
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
74 def load_results(self, output_path, prefix, skip_paths=[]):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
75 for obj in self.client.list_objects(self.bucket_name, prefix=prefix):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
76 print("found", obj.object_name)
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
77 if obj.object_name not in skip_paths:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
78 # Get data of an object.
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
79 response = self.client.get_object(
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
80 self.bucket_name, obj.object_name
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
81 )
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
82 try:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
83 # Append the object data to a local file
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
84 with open(output_path, "ab") as file:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
85 # todo: cut first line
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
86 for d in response.stream(self.chunk_size):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
87 file.write(d)
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
88 # Read data from response.
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
89 finally:
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
90 response.close()
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
91 response.release_conn()
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
92
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
93 def detete_results(self, prefix):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
94 for obj in self.client.list_objects(self.bucket_name, prefix=prefix):
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
95 self.client.remove_object(self.bucket_name, obj.object_name)
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
96
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
97
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
98 if __name__ == "__main__":
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
99 # test code
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
100 obj_name_prefix = "photon_z0.1_E_1e+07_1e+13_B1e-15_L0.05-5_N"
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
101 n_particles = 10000
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
102 root_path = "/Users/ok/git/mcray/bin/"
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
103 particle = "photon"
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
104 obj_name = obj_name_prefix + str(n_particles)
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
105 file_dir = root_path + obj_name + "/z0"
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
106 file_path = file_dir + "/" + particle
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
107 c = CRbeamCache()
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
108 c.save(obj_name, file_path, n_particles=n_particles)
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
109 print(file_path + " sent to s3")
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
110 loaded_file_path = file_path + ".loaded"
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
111 c.load_results(loaded_file_path, obj_name)
f40d05521dca planemo upload for repository https://github.com/esg-epfl-apc/tools-astro/tree/main/tools commit de01e3c02a26cd6353a6b9b6f8d1be44de8ccd54
astroteam
parents:
diff changeset
112 print(loaded_file_path + " loaded from s3")