Check data quality per observation at the site - #1676
Conversation
Contains a python script for implementing data quality check and associated config file. Note: Path to the mapmaker on pwg-scripts, given inside the config file under mapmaker_path needs to be modified.
mmccrackan
left a comment
mmccrackan
left a comment
There was a problem hiding this comment.
Overall looks solid, just some starting comments mostly on how it integrates into the site-pipeline work flow.
| parser.add_argument('--config_file', type=str, help='path to config file') | ||
| parser.add_argument('--freq_channel', type=str, help='frequency channel of observation') | ||
| parser.add_argument('--wafer', type=str, help='wafer of observation') | ||
| parser.add_argument('--obs_id', type=str, default=None, help='obs id to be mapped') |
There was a problem hiding this comment.
Typically for the site pipeline scripts, we want to run on all recent obs_ids that haven't been processed yet as it isn't really possible to pass in a single obs_id at a time and have it run automatically. We also don't want to re-run on ones that are already done. So I think passing in an obsdb query string is probably the right way to go with this (i.e. select planet obs only). We have a function that will accept a query and return a list of obs_ids for site-pipeline scripts here:
We would probably to remove the freq and wafer inputs (except as optional overrides maybe) too otherwise we'll need to have multiple separate flows for each one of them. In preprocessing, we have a function that when given a "grouping" like wafer_slot and wafer.bandpass it will find all available options for each obs_id:
| map_file = matching_files[0] | ||
| maps = [map_file] | ||
|
|
||
| qscores, popts = calc_map_quality(maps, band) |
There was a problem hiding this comment.
We'll want this to save the results somehow. I'm thinking a ManifestDb like the other metadata products might be best. You will have some number of hdf5 files containing the output information and a matching ManifestDb sqlite file that tells you which file and dataset the obs_id is in. You can build a ResultSet from you outputs and save using write_dataset. See update_det_cal for an example:
2 participants
Contains a python script for implementing data quality check and associated config file.
Note: Path to the mapmaker on pwg-scripts, given inside the config file under mapmaker_path needs to be modified.