Ingest companion parameters #624

kaseyLee123 · 2025-06-12T00:33:23Z

Ingest Roth24 companion parameters such as age and metalicity

Link to relevant issue: Closes #613

For data ingests:

includes script used for ingest
includes modified JSON files
Add new tests
Update the Versions table

kelle · 2025-06-12T00:44:27Z

Great start!

scripts/ingests/roth24/ingest_companion_parameters.py

kaseyLee123 · 2025-06-12T21:07:52Z

Not really sure what this error means

/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/astrodbkit/astrodb.py:939: SAWarning: Column 'Publications.name' is marked as a member of the primary key for table 'Publications', but has no Python-side or server-side default generator indicated, nor does it indicate 'autoincrement=True' or 'nullable=True', and no explicit value is passed. Primary key columns typically may not store NULL. conn.execute(self.metadata.tables[table].insert().values(data)) Traceback (most recent call last): File "/Users/kasey/Documents/GitHub/SIMPLE-db/scripts/ingests/roth24/ingest_companion_parameters.py", line 22, in <module> db = load_astrodb("SIMPLE.sqlite", recreatedb=RECREATE_DB, reference_tables=REFERENCE_TABLES) File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/astrodb_utils/utils.py", line 77, in load_astrodb db.load_database(data_path) ~~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/astrodbkit/astrodb.py", line 1003, in load_database self.load_table(table, os.path.join(directory, reference_directory), verbose=verbose) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/astrodbkit/astrodb.py", line 939, in load_table conn.execute(self.metadata.tables[table].insert().values(data)) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1416, in execute return meth( self, distilled_parameters, execution_options or NO_OPTIONS, ) File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/sqlalchemy/sql/elements.py", line 516, in _execute_on_connection return connection._execute_clauseelement( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ self, distilled_params, execution_options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1630, in _execute_clauseelement compiled_sql, extracted_params, cache_hit = elem._compile_w_cache( ~~~~~~~~~~~~~~~~~~~~~^ dialect=dialect, ^^^^^^^^^^^^^^^^ ...<4 lines>... linting=self.dialect.compiler_linting | compiler.WARN_LINTING, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/sqlalchemy/sql/elements.py", line 717, in _compile_w_cache compiled_sql = self._compiler( dialect, ...<4 lines>... **kw, ) File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/sqlalchemy/sql/elements.py", line 317, in _compiler return dialect.statement_compiler(dialect, self, **kw) ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/sqlalchemy/sql/compiler.py", line 1429, in __init__ Compiled.__init__(self, dialect, statement, **kwargs) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/sqlalchemy/sql/compiler.py", line 870, in __init__ self.string = self.process(self.statement, **compile_kwargs) ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/sqlalchemy/sql/compiler.py", line 915, in process return obj._compiler_dispatch(self, **kwargs) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^ File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/sqlalchemy/sql/visitors.py", line 141, in _compiler_dispatch return meth(self, **kw) # type: ignore # noqa: E501 File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/sqlalchemy/sql/compiler.py", line 5790, in visit_insert crud_params_struct = crud._get_crud_params( self, ...<4 lines>... **kw, ) File "/opt/miniconda3/envs/simple-db/lib/python3.13/site-packages/sqlalchemy/sql/crud.py", line 335, in _get_crud_params raise exc.CompileError( ...<2 lines>... ) sqlalchemy.exc.CompileError: Unconsumed column names: reference

kelle · 2025-06-13T01:05:21Z

The database isn't being loaded properly because your call to load_astrodb doesn't include the pointer to the schema.

kelle · 2025-06-13T01:06:08Z

scripts/ingests/roth24/ingest_companion_parameters.py

+RECREATE_DB = True  # recreates the .db file from the data files
+# LOAD THE DATABASE


Suggested change

RECREATE_DB = True # recreates the .db file from the data files

# LOAD THE DATABASE

RECREATE_DB = True # recreates the .db file from the data files

SCHEMA_PATH = "simple/schema.yaml"

# LOAD THE DATABASE

scripts/ingests/roth24/ingest_companion_parameters.py

kelle · 2025-06-13T01:09:16Z

And we actually hadn't added the CompanionParameters table yet! I've added it to the schema.yaml file and pushed to your branch.

kelle · 2025-06-18T15:56:46Z

@kaseyLee123: This looks great! Really good work! Please work with @canavarrete01 to merge with main.

@rothermichaustin , please do a spot check of the JSON files and make sure the data looks as expected.

data/reference/Publications.json

data/source/cwise_j002101.45-334631.5.json

This reverts commit ad484d3.

kelle · 2025-06-24T20:52:44Z

Since we're adding the CompanionList table, I wonder if we should also add the other_companion_name column to the CompanionList table also. Currently, it's in CompanionRelationships. I would be ok with just adding other_companion_name to CompanionList and also leaving it CompanionRelationships for the moment.

scripts/ingests/roth24/ingest_companion_parameters.py

kaseyLee123 · 2025-06-24T21:14:21Z

Since we're adding the CompanionList table, I wonder if we should also add the other_companion_name column to the CompanionList table also. Currently, it's in CompanionRelationships. I would be ok with just adding other_companion_name to CompanionList and also leaving it CompanionRelationships for the moment.

I think that makes sense. I can go do that. That would mean I have to reingest the companion list right?

kelle · 2025-06-24T21:31:36Z

I think that makes sense. I can go do that. That would mean I have to reingest the companion list right?

Yep! You'll have to re-run that script.

kelle · 2025-06-24T21:32:10Z

(I would also update the branch using the button here OR git pull upstream main into your branch.)

This reverts commit 2f2a179.

kaseyLee123 · 2025-06-25T15:12:12Z

I think that makes sense. I can go do that. That would mean I have to reingest the companion list right?

Yep! You'll have to re-run that script.

i just reran the script but i'd have to add something to my companion list script to add to the other_companion_names right? For that would I just use what the ingest_companion_relationships function uses or do you recommend something else

…ee123/SIMPLE-db into ingestCompanionParameters

kelle

It's fine with me to keep other_companion_names all blank for the moment. I think this is good to merge!

kelle · 2025-06-26T16:11:04Z

Kasey, one more thing. Add a test that touches the companion parameters you added. It should go in the test/test_data.py module and you can get inspiration from test_modeledparameters_refs and test_companion_relations

kelle · 2025-06-26T17:59:38Z

tests/test_data.py

+
+

delete these lines

kaseyLee123 · 2025-06-26T18:03:10Z

Kasey, one more thing. Add a test that touches the companion parameters you added. It should go in the test/test_data.py module and you can get inspiration from test_modeledparameters_refs and test_companion_relations

I just pushed new code and ran pytest (which worked). Can you take a look over at it to make sure I did it correctly before I can merge?

kelle · 2025-06-26T18:04:57Z

Looks great! Do you have the power to Squash and Merge or should I do it?

kaseyLee123 · 2025-06-26T18:06:16Z

Looks great! Do you have the power to Squash and Merge or should I do it?

I think I do. I can go do that right now

* ingest companion parameters * update + new csv file * commented out unnecessary code * adding CompanionParameters and CompanionList tables to the schema * implement cruz's changes + unconsumed column for reference is fixed * updated schema+parameters.json, updated csv file * comment * companionlist script + further bug testing in parameters * removed unnecessary * added companion list to reference table to fix pytest error * updated companion list json file * edit the ingesting publication logic * finished script + updated json files!! * Revert "finished script + updated json files!!" This reverts commit ad484d3. * changes * publication unicode changed within * schema update * Revert "schema update" This reverts commit 2f2a179. * reingested companion list with changed schema * updated json files with no unicode * update test_data with companion_param cases * remove unnecessart spaces --------- Co-authored-by: kelle <kellecruz@gmail.com>

kaseyLee123 added 2 commits June 11, 2025 16:47

ingest companion parameters

f792230

update + new csv file

80cc13e

kelle reviewed Jun 12, 2025

View reviewed changes

scripts/ingests/roth24/ingest_companion_parameters.py Outdated Show resolved Hide resolved

scripts/ingests/roth24/ingest_companion_parameters.py Outdated Show resolved Hide resolved

kaseyLee123 and others added 2 commits June 12, 2025 17:08

commented out unnecessary code

2d94516

adding CompanionParameters and CompanionList tables to the schema

db677d5

kelle reviewed Jun 13, 2025

View reviewed changes

scripts/ingests/roth24/ingest_companion_parameters.py Outdated Show resolved Hide resolved

kelle reviewed Jun 13, 2025

View reviewed changes

scripts/ingests/roth24/ingest_companion_parameters.py Outdated Show resolved Hide resolved

kaseyLee123 added 9 commits June 13, 2025 10:54

implement cruz's changes + unconsumed column for reference is fixed

bdda6cf

updated schema+parameters.json, updated csv file

d111b9f

comment

a17e035

companionlist script + further bug testing in parameters

e104436

removed unnecessary

cc4d922

added companion list to reference table to fix pytest error

d236694

updated companion list json file

7d9510f

edit the ingesting publication logic

388cfca

finished script + updated json files!!

ad484d3

kelle requested a review from rothermichaustin June 18, 2025 15:55

kelle requested a review from canavarrete01 June 18, 2025 15:57

kelle requested changes Jun 24, 2025

View reviewed changes

data/reference/Publications.json Outdated Show resolved Hide resolved

data/source/cwise_j002101.45-334631.5.json Outdated Show resolved Hide resolved

kaseyLee123 added 3 commits June 24, 2025 15:38

Revert "finished script + updated json files!!"

170cd38

This reverts commit ad484d3.

changes

fc786d7

publication unicode changed within

40afee6

kelle reviewed Jun 24, 2025

View reviewed changes

scripts/ingests/roth24/ingest_companion_parameters.py Outdated Show resolved Hide resolved

kaseyLee123 added 4 commits June 25, 2025 10:31

schema update

2f2a179

Revert "schema update"

41cf748

This reverts commit 2f2a179.

reingested companion list with changed schema

ea7e410

Merge branch 'main' into ingestCompanionParameters

035000b

kaseyLee123 added 2 commits June 25, 2025 11:25

updated json files with no unicode

e93eea3

Merge branch 'ingestCompanionParameters' of https://github.com/kaseyL…

cc18d6a

…ee123/SIMPLE-db into ingestCompanionParameters

kelle approved these changes Jun 25, 2025

View reviewed changes

update test_data with companion_param cases

e284bce

kelle reviewed Jun 26, 2025

View reviewed changes

tests/test_data.py Outdated

Comment on lines 290 to 291

Copy link

Collaborator

kelle Jun 26, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delete these lines

remove unnecessart spaces

b655e4f

kaseyLee123 requested a review from kelle June 26, 2025 18:03

kaseyLee123 merged commit a9a1412 into SIMPLE-AstroDB:main Jun 26, 2025
3 checks passed

		RECREATE_DB = True # recreates the .db file from the data files
		# LOAD THE DATABASE

Ingest companion parameters #624

Ingest companion parameters #624

Uh oh!

Conversation

kaseyLee123 commented Jun 12, 2025 • edited by kelle Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kelle commented Jun 12, 2025

Uh oh!

Uh oh!

Uh oh!

kaseyLee123 commented Jun 12, 2025

Uh oh!

kelle commented Jun 13, 2025

Uh oh!

kelle Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kelle commented Jun 13, 2025

Uh oh!

kelle commented Jun 18, 2025

Uh oh!

Uh oh!

Uh oh!

kelle commented Jun 24, 2025

Uh oh!

Uh oh!

kaseyLee123 commented Jun 24, 2025

Uh oh!

kelle commented Jun 24, 2025

Uh oh!

kelle commented Jun 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kaseyLee123 commented Jun 25, 2025

Uh oh!

kelle left a comment

Choose a reason for hiding this comment

Uh oh!

kelle commented Jun 26, 2025

Uh oh!

kelle Jun 26, 2025

Choose a reason for hiding this comment

Uh oh!

kaseyLee123 commented Jun 26, 2025

Uh oh!

kelle commented Jun 26, 2025

Uh oh!

kaseyLee123 commented Jun 26, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kaseyLee123 commented Jun 12, 2025 •

edited by kelle

Loading

kelle commented Jun 24, 2025 •

edited

Loading