Implement Pan-STARRS manual photometry and proper motion ingestion

[ying2212]

Short description: This pull request is to ingest photometry and proper motion data from Pan-STARRS catalog

Link to relevant issue: #609

For data ingests:

• includes script used for ingest
• includes modified JSON files
• Add new tests
• Update the Versions table

[kelle]

The ingest_photometry script looks good! Is it faster?

Discuss with @canavarrete01 the best way to move forward modifying and reviewing so many JSON files.

[ying2212]

yes! speedup for ~2 seconds per 100 rows.

Also, I didn’t manually ingest the proper motion data as the runtime not change as much.

[ying2212]

I went through the Pan-STARRS data and everything looks good. ~2,080 sources are valid for ingesting into schema.
Would need a review!

[kelle]

keep track of the sources which did not find matches.

[kelle]

Awesome!

Thinking about this, I think the best way to review this data before we ingest to the database is to make a CSV file. So, instead of ingesting the matches, let's add to a CSV file. and then I can review that table and THEN you can ingest the photometry from that table.

[kelle]

It might be faster to write the CSV files as you go through the loop rather than using .append. This is probably most important for the successful_source list since it's the biggest.

eg,

with open('photometry_data.csv', 'w') as phot_data:
        data_writer = csv.DictWriter(phot_data, fieldnames=[blah])
        data_writer.writerows(photometry_data)
        

[ying2212]

Things that have changed and needs review:
collectphot_PS1.py

• save valid sources into valid_photometry.csv, and missing ones to invalid_photometry.csv
• also maps valid source name to SIMPLE source name and saves them in matched_sources.csv for ingesting motion afterwards

ingestphot_PS1.py

• Ingest only valid photometry data from valid_photometry.csv to speed things up.

ingestmotion_PS1.py

• Use the matched source names to ingest_proper_motions
• invalid_proper_motion.csv: store source with missing RA/Dec
• valid_proper_motion.csv: store tested valid proper motions entries

[canavarrete01]

Make sure to update branch and then your branch should be ready to be merged!

[kelle]

Also update the PR name -- it's just photometry,
Not proper motions , correct?

And let's also update the versions table and do a release once this PR is merged.

[kelle]

• Update with main
• add tests for proper motion

[kelle]

Move both Kirk19 and Best18 proper motion test to test_data_astrometry

[kelle]

Work with Kasey to write a test of *adopted" proper motions for Best18, Best20, and GaiaEDR3

[kelle]

Here's an example for parallaxes:

t = (
        db.query(db.Parallaxes)
        .filter(and_(db.Parallaxes.c.reference == ref, db.Parallaxes.c.adopted == 1))
        .astropy()
    )
    assert (
        len(t) == 1077
    ), f"found {len(t)} adopted parallax reference entries for {ref}"

[kelle]

I think the ingestmotion script should be re-run on the updated database. Some of the adopted flags might change now that there are also CatWISE proper motions to compare to.

[ying2212]

Yep you're right!
I've pulled the latest main version with CATWISE proper motion data ingested, and also reran all my ingesting scripts and check the pytest, the adoption numbers are updated for those references are in my latest commit

[kelle]

Does pytest pass locally?

[ying2212]

yep! It passed locally

[kelle]

Instead of running the same test multiple times, try using pytest.mark.paramatrize.

[kelle]

push the JSONs!

[kelle]

and update the Versions table. I'm pretty sure new version is 3.3.2025.9.

[kelle]

I think this is great!