Mercurial > repos > jay > gaiac_precision

annotate gaiac_time_sync/gaiac_time_sync.py @ 0:287d6cc86582 draft

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
author jay
date Thu, 15 May 2025 14:45:45 +0000
parents
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
1 import pandas as pd
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
2 import os
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
3 import argparse
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
4
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
5 def align_sensor_data(file_list, date_time, sep=',', output_mode='multiple', output='aligned.tsv'):
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
6
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
7 file_list = file_list.split(',')
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
8 if len(file_list) < 2:
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
9 print("Please provide at least two files.")
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
10 return
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
11
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
12 # Read all files into a list of dataframes
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
13 dfs = [pd.read_csv(file, sep=sep, parse_dates=[date_time]) for file in file_list]
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
14
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
15 # Get common timestamps by successive inner merges
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
16 common_times = dfs[0][[date_time]]
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
17 for df in dfs[1:]:
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
18 common_times = common_times.merge(df[[date_time]], on=date_time, how='inner')
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
19
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
20 # Now filter each dataframe to contain only common timestamps
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
21 aligned_dfs = [
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
22 df[df[date_time].isin(common_times[date_time])].reset_index(drop=True)
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
23 for df in dfs
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
24 ]
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
25
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
26 # Output files
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
27 if output_mode == 'multiple':
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
28 for i, (file, df) in enumerate(zip(file_list, aligned_dfs)):
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
29 filename = os.path.splitext(os.path.basename(file))[0]
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
30 output_file = f"{output_prefix}_{filename}.csv"
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
31 df.to_csv(output_file, index=False, sep=sep)
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
32 print("Aligned files saved individually.")
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
33 elif output_mode == 'single':
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
34 # Merge all aligned dataframes on date_time
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
35 merged_df = aligned_dfs[0]
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
36 for df in aligned_dfs[1:]:
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
37 merged_df = merged_df.merge(df, on=date_time, how='inner')
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
38
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
39
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
40 merged_df.to_csv(output, index=False, sep=sep)
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
41 print("Single merged file saved.")
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
42 else:
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
43 print("Invalid output mode. Use 'multiple' or 'single'.")
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
44
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
45
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
46 def main():
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
47 parser = argparse.ArgumentParser(description="Align sensor data files on common timestamps.")
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
48
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
49 parser.add_argument(
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
50 '-f', '--files',
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
51 required=True,
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
52 help='List of input file paths (at least two)'
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
53 )
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
54
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
55 parser.add_argument(
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
56 '-s', '--sep',
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
57 default='\t',
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
58 help='Separator used in the input files (default: ,)'
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
59 )
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
60
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
61 parser.add_argument(
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
62 '-m', '--mode',
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
63 choices=['multiple', 'single'],
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
64 default='single',
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
65 help="Output mode: 'multiple' for individual files, 'single' for one merged file (default: multiple)"
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
66 )
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
67
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
68 parser.add_argument(
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
69 '-o', '--output',
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
70 default='aligned',
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
71 help="Output filename"
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
72 )
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
73
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
74 parser.add_argument(
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
75 '-t', '--date_time_column',
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
76 default='date_time',
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
77 help="Provide the name of the date and time column."
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
78 )
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
79
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
80 args = parser.parse_args()
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
81
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
82 align_sensor_data(
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
83 file_list=args.files,
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
84 date_time=args.date_time_column,
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
85 sep=args.sep,
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
86 output_mode=args.mode,
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
87 output=args.output
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
88 )
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
89
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
90 if __name__ == '__main__':
287d6cc86582 planemo upload for repository https://github.com/jaidevjoshi83/gaiac.git commit c29a769ed165f313a6410925be24f776652a9663-dirty
jay
parents:
diff changeset
91 main()