Bulk COPY insert - tickets/INSTRM-2821

[wtgee]

• Add bulk COPY command for insert dataframe

For inserting 65k rows into cobra_target, this, offers a +10x speedup.

[Copilot]

Pull request overview

This PR implements bulk insert optimization using PostgreSQL's COPY command, providing significant performance improvements (claimed 10x speedup for 65k rows) over the previous multi-row INSERT approach.

Key Changes:

• Added new psql_insert_copy function that uses PostgreSQL's COPY command for bulk data loading
• Added use_copy parameter (default True) to insert_dataframe method to enable/disable COPY optimization
• Added performance timing to track insert operations

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[CraigLoomis]

I'll suggest making that WITH (FORMAT csv, HEADER MATCH) (and whatever the equivalent of df.to_csv(...., header=True) is to ward against the worst mistakes.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[wtgee]

I'll suggest making that WITH (FORMAT csv, HEADER MATCH) (and whatever the equivalent of df.to_csv(...., header=True) is to ward against the worst mistakes.

Since this is coming from the to_sql command itself, it has the keys parameter that specifies the exact column, so it should be fine to rely on the specific HEADER FALSE and the ordering of the keys.

I've added some explicit parameters to the csv writer and some other dataframe scrubbing checks that shouldn't interfere with data. It's actually running even faster with these explicit parameters since I guess it doesn't have to do an initial pass or conversions.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 9 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.