From 8bf2eca0bbdc4dba3e6763a371a666293780202d Mon Sep 17 00:00:00 2001 From: Will-Cooper Date: Wed, 4 Jun 2025 12:19:51 +0100 Subject: [PATCH 1/9] closes #179 with acknowledgement text --- simple_app/templates/about.html | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/simple_app/templates/about.html b/simple_app/templates/about.html index 9827d72..f546199 100644 --- a/simple_app/templates/about.html +++ b/simple_app/templates/about.html @@ -66,8 +66,17 @@

Holdings

Publications:

- Cool Stars 2021 Poster
- ADASS 2021 Poster + Cool Stars 20.5 2021 Poster
+ ADASS XXXI 2021 Poster
+ Cool Stars 21 2022 Poster
+ ADASS XXXIII 2023 Poster
+ Cool Stars 22 2024 Poster +

+

Acknowledgement:

+

+ If the SIMPLE database has been useful for your research, please include the following text. + This work has made use of the SIMPLE Archive of low-mass stars, brown dwarfs, and directly imaged exoplanets: + 10.5281/zenodo.13937301.

Getting Involved

From 9e25e4fd2d68c4e9887af6001b3e5facca5a855c Mon Sep 17 00:00:00 2001 From: Will-Cooper Date: Wed, 4 Jun 2025 12:30:09 +0100 Subject: [PATCH 2/9] closes #178 with contributor section with hyperlinks --- simple_app/templates/about.html | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/simple_app/templates/about.html b/simple_app/templates/about.html index f546199..30838fe 100644 --- a/simple_app/templates/about.html +++ b/simple_app/templates/about.html @@ -78,7 +78,7 @@

Acknowledgement:

This work has made use of the SIMPLE Archive of low-mass stars, brown dwarfs, and directly imaged exoplanets: 10.5281/zenodo.13937301.

-

Getting Involved

+

Getting Involved:

If you'd like to take part or just stay in the loop as this project progresses, please request to join the @@ -87,6 +87,12 @@

Getting Involved

If you are not already in the Astropy Slack, request an account.

+

Contributors:

+

+ See here for the (ever evolving!) list of contributors to the + database and + website. +

Imposter syndrome disclaimer:

We want your help. No, really.
From b8fecbaca5a4945aba01baae523c4bd3f91520b0 Mon Sep 17 00:00:00 2001 From: Will-Cooper Date: Wed, 4 Jun 2025 12:52:12 +0100 Subject: [PATCH 3/9] closes #168 with coordinate query instruction --- simple_app/templates/coordinate_query.html | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/simple_app/templates/coordinate_query.html b/simple_app/templates/coordinate_query.html index 040eeb8..ea4d7c9 100644 --- a/simple_app/templates/coordinate_query.html +++ b/simple_app/templates/coordinate_query.html @@ -23,9 +23,16 @@

Exploring the Database

{{ form.submit(class_="btn btn-primary") }}

- Query in form: "ra dec radius[optional]" + Query in the form: "ra dec radius[optional]" where ra and dec can be in hms/dms or degrees with an optional radius in arcseconds. + Sexagesimal formatting is expected to be with colon separators, HMS (HH:MM:SS.s) and DMS (DD:MM:SS.s), + with arbitrary levels of precision possible, as with a search in degrees.
+ For example, 162.329000 -53.319400, with the default radius of 10 arcseconds, returns both Luhman 16A and B. + 162.329000 -53.319400 0.1, however, with the specified radius of 0.1 arcseconds, only returns Luhman 16A.
+ You could do this same search with sexagesimal coordinates, for example, a lower precision search: + 10:49:14 -53:19:05 44.8 (radius of 44.8 arcseconds) returns both Luhman 16A and B; + 10:49:14 -53:19:05 44.7 returns only Luhman 16B.

- - + + From f2fba66f0b0de9936982fef027096ee25a236b05 Mon Sep 17 00:00:00 2001 From: Will-Cooper Date: Wed, 4 Jun 2025 14:16:57 +0100 Subject: [PATCH 6/9] closes #161 by adding clickable references everywhere they are used --- simple_app/app_simple.py | 12 ++++--- simple_app/plots.py | 10 +++--- simple_app/utils.py | 68 ++++++++++++++++++++++++++++++++++++---- 3 files changed, 75 insertions(+), 15 deletions(-) diff --git a/simple_app/app_simple.py b/simple_app/app_simple.py index 26ba205..7b041d4 100644 --- a/simple_app/app_simple.py +++ b/simple_app/app_simple.py @@ -106,6 +106,7 @@ def coordinate_query(): # submit query results: pd.DataFrame = db.query_region(c, fmt='pandas', radius=radius) # query + results = reference_handle(results, db_file) stringed_results = one_df_query(results) return render_template('coordinate_query.html', form=form, query=query, results=stringed_results, version_str=version_str) @@ -131,7 +132,7 @@ def full_text_search(): # search through the tables using the given query results: Dict[str, pd.DataFrame] = db.search_string(query, fmt='pandas', verbose=False) - resultsout = multi_df_query(results, limmaxrows) + resultsout = multi_df_query(results, db_file, limmaxrows) return render_template('full_text_search.html', form=form, version_str=version_str, results=resultsout, query=query) @@ -169,6 +170,7 @@ def raw_query(): except (ResourceClosedError, OperationalError, IndexError, SqliteWarning, BadSQLError): results = pd.DataFrame() + results = reference_handle(results, db_file, True) stringed_results = one_df_query(results) return render_template('raw_query.html', form=form, results=stringed_results, version_str=version_str) @@ -197,11 +199,11 @@ def solo_result(query: str): except KeyError: abort(404, f'"{query}" does match any result in SIMPLE!') return - everything = Inventory(resultdict) + everything = Inventory(resultdict, db_file) # create camd and spectra plots scriptcmd, divcmd = camd_plot(query, everything, all_bands, all_results_full, all_parallaxes, all_spectral_types, - photometric_filters, all_photometry, js_callbacks, night_sky_theme) + photometric_filters, all_photometry, js_callbacks, night_sky_theme, db_file) scriptspectra, divspectra, nfail, failstr = spectra_plot(query, db_file, night_sky_theme, js_callbacks) query = query.upper() @@ -286,7 +288,7 @@ def create_file_for_download(key: str): # search for a given object and a given key resultdict: dict = db.inventory(query) - everything = Inventory(resultdict, return_markdown=False) + everything = Inventory(resultdict, db_file, return_markdown=False) # writes table to csv if key in resultdict: @@ -331,7 +333,7 @@ def create_spectra_files_for_download(): # search for a given object and specifically its spectra resultdict: dict = db.inventory(query) - everything = Inventory(resultdict, return_markdown=False) + everything = Inventory(resultdict, db_file, return_markdown=False) results: pd.DataFrame = getattr(everything, 'spectra') # write all spectra for object to zipped file diff --git a/simple_app/plots.py b/simple_app/plots.py index 4c6e4a5..4416458 100644 --- a/simple_app/plots.py +++ b/simple_app/plots.py @@ -519,7 +519,7 @@ def colour_absolute_magnitude_diagram() -> Tuple[figure, Toggle, Toggle, Select, def camd_plot(query: str, everything: Inventory, all_bands: np.ndarray, all_results: pd.DataFrame, all_parallaxes: pd.DataFrame, all_spectral_types: pd.DataFrame, photometric_filters: pd.DataFrame, - all_photometry: pd.DataFrame, js_callbacks: JSCallbacks, night_sky_theme: Theme) -> Tuple[Optional[str], + all_photometry: pd.DataFrame, js_callbacks: JSCallbacks, night_sky_theme: Theme, db_file: str) -> Tuple[Optional[str], Optional[str]]: """ Creates CAMD plot into html @@ -546,6 +546,8 @@ def camd_plot(query: str, everything: Inventory, all_bands: np.ndarray, all_resu The javascript callbacks for bokeh night_sky_theme: Theme The theme for bokeh + db_file: str + The connection string to the database Returns ------- @@ -560,7 +562,7 @@ def camd_plot(query: str, everything: Inventory, all_bands: np.ndarray, all_resu # retrieve photometry for given object try: - this_photometry: pd.DataFrame = everything.list_concat('Photometry', False) + this_photometry: pd.DataFrame = everything.list_concat('Photometry', db_file, False) if len(this_photometry) < 4: raise KeyError('Not enough photometric entries') @@ -571,7 +573,7 @@ def camd_plot(query: str, everything: Inventory, all_bands: np.ndarray, all_resu # look for spectral type try: - this_spectral_type: pd.DataFrame = everything.list_concat('SpectralTypes', False) + this_spectral_type: pd.DataFrame = everything.list_concat('SpectralTypes', db_file, False) except KeyError: this_spectral_type = pd.DataFrame.from_dict(dict(spectral_type_code=[np.nan, ], adopted=[np.nan, ])) @@ -590,7 +592,7 @@ def camd_plot(query: str, everything: Inventory, all_bands: np.ndarray, all_resu # attempt to retrieve parallaxes to process absolute magnitudes try: - this_parallaxes: pd.DataFrame = everything.list_concat('Parallaxes', False) + this_parallaxes: pd.DataFrame = everything.list_concat('Parallaxes', db_file, False) except KeyError: pass diff --git a/simple_app/utils.py b/simple_app/utils.py index fa8a483..3b9e45b 100644 --- a/simple_app/utils.py +++ b/simple_app/utils.py @@ -38,6 +38,7 @@ class SimpleDB(Database): # this keeps pycharm happy about unresolved reference Versions = None SpectralTypes = None CompanionRelationships = None + Publications = None def __init__(self, connection_string): super().__init__(connection_string, @@ -53,7 +54,8 @@ class Inventory: ra: float = 0 dec: float = 0 - def __init__(self, d_result: Dict[str, List[Dict[str, List[Union[str, float, int]]]]], **kwargs): + def __init__(self, d_result: Dict[str, List[Dict[str, List[Union[str, float, int]]]]], + db_file: str, **kwargs): """ Constructor method for Inventory @@ -61,6 +63,8 @@ def __init__(self, d_result: Dict[str, List[Dict[str, List[Union[str, float, int ---------- d_result: Dict[str, List[Dict[str, List[Union[str, float, int]]] The dictionary of all the key: values in a given object inventory + db_file: str + The connection string to the database """ self.results: Dict[str, List[Dict[str, List[Union[str, float, int]]]]] = d_result @@ -73,12 +77,12 @@ def __init__(self, d_result: Dict[str, List[Dict[str, List[Union[str, float, int # convert the table result to Markdown low_key: str = key.lower() - markdown_output: str = self.list_concat(key, **kwargs) + markdown_output: str = self.list_concat(key, db_file, **kwargs) setattr(self, low_key, markdown_output) # retrieve ra and dec from the Sources table, if present try: - sources: pd.DataFrame = self.list_concat('Sources', return_markdown=False) + sources: pd.DataFrame = self.list_concat('Sources', db_file, return_markdown=False) self.ra, self.dec = sources.ra[0], sources.dec[0] except (KeyError, AttributeError): pass @@ -122,7 +126,7 @@ def spectra_handle(df: pd.DataFrame, drop_source: bool = True): df['observation_date'] = df['observation_date'].dt.date return df - def list_concat(self, key: str, return_markdown: bool = True) -> Union[pd.DataFrame, str]: + def list_concat(self, key: str, db_file: str, return_markdown: bool = True) -> Union[pd.DataFrame, str]: """ Concatenates the list for a given key @@ -130,6 +134,8 @@ def list_concat(self, key: str, return_markdown: bool = True) -> Union[pd.DataFr ---------- key: str The key corresponding to the inventory + db_file: str + The connection string to the database return_markdown: bool Switch for whether to return either a markdown string or a dataframe @@ -159,6 +165,7 @@ def list_concat(self, key: str, return_markdown: bool = True) -> Union[pd.DataFr if return_markdown: if key == 'Spectra': df = self.spectra_handle(df) + df = reference_handle(df, db_file) df.rename(columns={s: s.replace('_', ' ') for s in df.columns if 'download' not in s}, inplace=True) return markdown(df.to_html(index=False, escape=False, classes='table table-dark table-bordered table-striped')) @@ -964,7 +971,8 @@ def one_df_query(results: pd.DataFrame, table_id: Optional[str] = None, limit_ma return stringed_results -def multi_df_query(results: Dict[str, pd.DataFrame], limit_max_rows: bool = False) -> Dict[str, Optional[str]]: +def multi_df_query(results: Dict[str, pd.DataFrame], db_file: str, + limit_max_rows: bool = False) -> Dict[str, Optional[str]]: """ Handling the output from a query which returns multiple dataframes @@ -972,6 +980,8 @@ def multi_df_query(results: Dict[str, pd.DataFrame], limit_max_rows: bool = Fals ---------- results The dictionary of dataframes + db_file: str + The connection string to the database limit_max_rows Limit max rows switch @@ -986,15 +996,61 @@ def multi_df_query(results: Dict[str, pd.DataFrame], limit_max_rows: bool = Fals # make sources table go first if present if 'Sources' in results.keys(): - d_results['Sources'] = one_df_query(results.pop('Sources'), 'sourcestable', limit_max_rows) + sources_df = reference_handle(results.pop('Sources'), db_file) + d_results['Sources'] = one_df_query(sources_df, 'sourcestable', limit_max_rows) # wrapping the one_df_query method for each table for table_name, df in results.items(): + df = reference_handle(df, db_file) stringed_df = one_df_query(df, table_name.lower() + 'table', limit_max_rows) d_results[table_name] = stringed_df return d_results +def reference_handle(df: pd.DataFrame, db_file: str, multi_dim: bool = False) -> pd.DataFrame: + """ + Handles any references to redirect to ADS via bibcode + + Parameters + ---------- + db_file: str + The connection string to the database + df + Dataframe with references in + multi_dim: bool + Whether the reference values are multidimensional or not + + Returns + ------- + df: pd.DataFrame + Edited dataframe if reference was in columns + """ + if 'reference' not in df.columns: + return df + + if not multi_dim: + old_reference_values = df.reference.values + else: + old_reference_values = df.reference.values[:, 0] + new_reference_values = [] + + db = SimpleDB(db_file) + publications_table: pd.DataFrame = db.query(db.Publications).pandas() + + for old_reference in old_reference_values: + try: + bibcode = publications_table[publications_table.reference == old_reference].bibcode.values[0] + except IndexError: # if no reference match, which only happens on default search view + new_reference_values.append(old_reference) + continue + new_reference = (f'{old_reference}') + new_reference_values.append(new_reference) + + df['reference'] = new_reference_values + return df + + def get_filters(db_file: str) -> pd.DataFrame: """ Query the photometry filters table From f59ff9f286b0111ca4cde4dd7cb09740384d745c Mon Sep 17 00:00:00 2001 From: Will-Cooper Date: Wed, 4 Jun 2025 14:22:40 +0100 Subject: [PATCH 7/9] tests need same update for the reference linking --- simple_app/tests/test_plots.py | 8 ++++---- simple_app/tests/test_utils.py | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/simple_app/tests/test_plots.py b/simple_app/tests/test_plots.py index 663d98e..e3be238 100644 --- a/simple_app/tests/test_plots.py +++ b/simple_app/tests/test_plots.py @@ -52,18 +52,18 @@ def test_camd_plot(db, test_main_plots, test_get_all_photometry, test_get_all_so bad_query = 'thisisabadquery' night_sky_theme, js_callbacks = test_main_plots d_result: dict = db.inventory(good_query) - good_everything = Inventory(d_result) + good_everything = Inventory(d_result, db_cs) d_result = db.inventory(bad_query) - bad_everything = Inventory(d_result) + bad_everything = Inventory(d_result, db_cs) all_results, all_resultsfull = test_get_all_sources all_parallaxes = test_get_all_parallaxes all_spectral_types = test_get_all_spectral_types good_script, good_div = camd_plot(good_query, good_everything, all_bands, all_resultsfull, all_parallaxes, all_spectral_types, photometric_filters, - all_photometry, js_callbacks, night_sky_theme) + all_photometry, js_callbacks, night_sky_theme, db_cs) bad_script, bad_div = camd_plot(bad_query, bad_everything, all_bands, all_resultsfull, all_parallaxes, all_spectral_types, photometric_filters, - all_photometry, js_callbacks, night_sky_theme) + all_photometry, js_callbacks, night_sky_theme, db_cs) assert all([type(s) == str for s in (good_script, good_div)]) assert all([s is None for s in (bad_script, bad_div)]) return diff --git a/simple_app/tests/test_utils.py b/simple_app/tests/test_utils.py index fda6862..e1ca3c3 100644 --- a/simple_app/tests/test_utils.py +++ b/simple_app/tests/test_utils.py @@ -96,7 +96,7 @@ def test_inventory(db): good_query = '2MASS J00192626+4614078' d_results: dict = db.inventory(good_query) assert len(d_results) - everything = Inventory(d_results) + everything = Inventory(d_results, db_cs) assert all([hasattr(everything, s) for s in ('photometry', 'sources', 'names', 'spectra', 'ra', 'dec', 'propermotions')]) return @@ -174,7 +174,7 @@ def test_multi_df_query(db): assert isinstance(results, dict) assert 'Sources' in results assert isinstance(results['Sources'], pd.DataFrame) - results_out = multi_df_query(results) + results_out = multi_df_query(results, db_cs) assert isinstance(results_out, dict) assert 'Sources' in results_out assert isinstance(results_out['Sources'], str) From 74a1fcc5b5d47e19db5a249bcdcb5c45cdeb31c9 Mon Sep 17 00:00:00 2001 From: Kelle Cruz Date: Mon, 28 Jul 2025 15:38:52 -0400 Subject: [PATCH 8/9] Fix Zenodo URL --- simple_app/templates/about.html | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/simple_app/templates/about.html b/simple_app/templates/about.html index 30838fe..c18036d 100644 --- a/simple_app/templates/about.html +++ b/simple_app/templates/about.html @@ -76,7 +76,7 @@

Acknowledgement:

If the SIMPLE database has been useful for your research, please include the following text. This work has made use of the SIMPLE Archive of low-mass stars, brown dwarfs, and directly imaged exoplanets: - 10.5281/zenodo.13937301. + 10.5281/zenodo.13937301.

Getting Involved:

From a18f26040eae5da7b365e543e2c4b2898d71102c Mon Sep 17 00:00:00 2001 From: Will Cooper Date: Wed, 30 Jul 2025 09:48:34 +0100 Subject: [PATCH 9/9] Coordinate text change Co-authored-by: Kelle Cruz --- simple_app/templates/coordinate_query.html | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) diff --git a/simple_app/templates/coordinate_query.html b/simple_app/templates/coordinate_query.html index ea4d7c9..0635a7a 100644 --- a/simple_app/templates/coordinate_query.html +++ b/simple_app/templates/coordinate_query.html @@ -24,15 +24,12 @@

Exploring the Database

Query in the form: "ra dec radius[optional]" - where ra and dec can be in hms/dms or degrees - with an optional radius in arcseconds. - Sexagesimal formatting is expected to be with colon separators, HMS (HH:MM:SS.s) and DMS (DD:MM:SS.s), - with arbitrary levels of precision possible, as with a search in degrees.
- For example, 162.329000 -53.319400, with the default radius of 10 arcseconds, returns both Luhman 16A and B. - 162.329000 -53.319400 0.1, however, with the specified radius of 0.1 arcseconds, only returns Luhman 16A.
- You could do this same search with sexagesimal coordinates, for example, a lower precision search: - 10:49:14 -53:19:05 44.8 (radius of 44.8 arcseconds) returns both Luhman 16A and B; - 10:49:14 -53:19:05 44.7 returns only Luhman 16B. + RA and dec can be provided in decimal degrees or in sexagesimal. These are the expected formats:
+ 162.329000 -53.319400
+ 10:49:18.91 -53:19:10
+ An optional third field can be provided to modify the default search radius of 10 arcseconds. For example,
+ 10:49:18.91 -53:19:10 30
+ performs a 30 arcsecond search around the provided coordinate.