Proposal: Separation of Concerns in library.py (Repository Pattern)

[JonathanGlixman]

Hi everyone!

First, I want to say thank you for the great work on TagStudio. I really loved the concept and the vision behind the app, so much so that it inspired me to create a similar project called TagIt for my final university project.

While building my project, I spent a lot of time thinking about architecture, and after diving into TagStudio's core backend recently, I wanted to propose a structural enhancement for src/tagstudio/core/library/alchemy/library.py.

Specifically, I think this refactor could lay a much better basis and cleaner architecture for the upcoming Core Library + CLI milestone.

Aligning with the Core Library + CLI Milestone

Currently, the Library class has grown into a very large, centralized module. It handles several distinct responsibilities all at once:

1. Database connection and session lifecycle.

2. Complex SQLAlchemy queries and CRUD operations.

3. Schema migrations (e.g., __apply_db..._migration methods).

4. Core business logic and file handling.

I saw that the goal for the Core Library milestone is to create a separated core library that could interface with scripts, external programs, and a new CLI.

Having the database lifecycle, migrations, and raw SQL queries tightly coupled with the business logic makes it very difficult to extract a clean, lightweight core. By separating these concerns now, it will be significantly easier to expand the codebase, build out the CLI for bulk operations, and foster wider adoption without getting tangled in the internal SQLAlchemy implementation.

Proposed Architecture (Repository Pattern)

In my project, I implemented a strict Separation of Concerns architecture, and I'd love to propose a refactoring plan to bring a similar pattern to TagStudio to prepare for the above purpose.

We could break library.py down into smaller, specialized components:

• Repositories (e.g., tag_repository.py, entry_repository.py): Pure data-access classes. They would take a SQLAlchemy Session and handle the raw SQL/ORM execution.

• Migration Engine (migrations.py): Extracting the __apply_db_migration functions into a dedicated migration runner to keep the main library initialization clean.

• Library Service (library.py): The main class remains the entry point for the app (and eventually the CLI), but instead of writing SQL and managing the session directly, it delegates to the repositories. It becomes a pure business logic coordinator.

Next Steps

I know rewriting a core file is a massive undertaking and carries risks, so I wanted to open a discussion before touching any code.

1. Is this architectural direction something the core team is interested in to help facilitate the core library + CLI?

2. If so, would you be open to an incremental PR? (For example, we could start by just extracting the Migration logic, or just moving the Tag-specific database queries into a TagRepository to see how it feels).

I'd love to give back to this project. Let me know your thoughts!

[CyanVoxel]

There's a thread in the Discord server currently talks about everything going with proposed refactors to help align with these roadmap goals. For the sake of clarity I'll paste over some of my responses to other proposals and questions and things in general that also address a lot of what you bring up here:

---

Okay, so I've got some stuff in mind that I think will fit in nicely with axing SQLAlchemy but wanted to get some feedback and coordinate things before setting it into motion:

Core Library Split

A core library split has been something that’s been brought up for a while, and is presently a big ticket item on the roadmap that hasn’t had the details ironed out or any timeline given. I’d like to change that here.

First, the “core library” would encompass all the database CRUD operations plus the search query system. The current TagStudio application would import this core and be seemingly unchanged on the surface. Anyone else who wants to make a frontend or add plugin support to another program can also import the same core.

One goal of this separate core library is to make it MIT, which will be a longer-term transition due to the existing code being GPL-3.0. With the desire to remove SQLAlchemy though, this could be coordinated to replace the existing GPL-3.0 SQLAlchemy code with new MIT in-house raw SQL code. The rest of the relicensing can be done over time with the approval of the original contributors where possible, but relicensing is a delicate topic that can be discussed further later.

CLI

A big request (and something that used to exist in the form of a TUI before the Qt frontend took priority) is a CLI for TagStudio. I’ve gone back and forth on this over time, but currently feel that if done properly it would be a major improvement to the TagStudio ecosystem. I’m using Obsidian’s recent CLI as a loose inspiration, but the main thing I’d want the CLI to be is an easy interaction point for users and programs to interact with the TagStudio core for things like bulk or automated actions without the need for a graphical frontend.

My thinking is that this could be done on the core library side of things, so that this CLI would be accessible via normal TagStudio releases (e.g. through a launch flag) as well as alternate frontends. It would also mean that anyone wanting to just use TagStudio as a CLI tool can install the lightweight core instead of a full frontend. I’m very open to hearing other takes on this though.

Carrying Things Out

As for actually carrying out splitting the core, replacing SQLAlchemy, and building a CLI, here’s what I was thinking as a course of action:
0. Make sure all existing PRs that touch the “core” are pulled or addressed, and any urgent changes are done before the switch.

1. Fork the “TagStudio” repo into a new “tagstudio-core” repo under the org. This should serve as a solid jumping off point while preserving the existing Git history.
2. Remove all “non-core” code, update the README, and clarify the license of any new code going forward Xarvex brought up REUSE to help with this).
3. Create “interface” files (in something like src/tagstudio-core/library/) that pass through to the existing GPL-3.0 SQLAlchemy code in src/tagstudio/core/library/alchemy/ but will be replaced over time with new MIT raw SQL implementations.
4. Refactor TagStudio to import this core and use the interface layer instead of the internal alchemy directory. Because the interface will more-or-less mirror the current methods in library.py to start off with, this shouldn’t be as large of an undertaking as it might seem (famous last words...)

This way we can do the core split, remove SQLAlchemy, and develop the CLI at our own pace without too much cross-interference. But I’d really like to get some feedback on this, any holes you all see, or further ideas on some of the details.

As someone who’s often bitten off more than I can chew with this project or found myself in scope-creep hell, I believe that this a decently realistic way to achieve these goals and allows for failures in things like the CLI development or SQLAlchemy refactor to not hold up the rest of the project.

Computerdores then suggested we do the core license split and interface layer inside the current repo before splitting it off, which ultimately I agreed with and is how we've been carrying things out since these messages, including full REUSE compliance.

Responding to comments wanting to start a Rust rewrite now:

I agree with Computerdores here - the difficulty in side-stepping a Python core cleanup first (i.e. removing SQLAlchemy and its specific code accommodations, completing in-progress features and changes, and detaching anything necessary for a core separation) is that we'd be "drawing blueprints for the house while constructing it", which is exactly the sort of problem that occurred with us during the 9.5 migration to SQLite and part of the reason why that migration was plagued with miscommunications and feature implementations that are still being fixed to this day. To reiterate what I usually say whenever a rewrite to another language is proposed: I'd rather have one working version of TagStudio, even if it's in Python, than a bunch of half-finished versions in different languages.

The other problem with moving to Rust right now is that I am currently unfamiliar with the language and would not be able to maintain it or complete development of the roadmap features inside a Rust core if it were added today. Combine not being able to maintain or contribute to a Rust core with not having a cleaned-up finalized Python core to reference from, and that's a deadly combination for the project.

Now with all that being said, I would still like there to be a Rust core rewrite when the time is right. I fully understand the limits of Python and the benefits of compiled languages, the advantages of Rust over other languages, and agree that it is the right move in time. But I also don't believe that there is a rush to write a new core in Rust, as the Python core (despite some quirks) has been plenty performant compared to what I think most people expect when they hear that a program is "written in Python". Believe me, I don't want to be spending time cleaning up and finishing a core in Python just for it to be re-written again in Rust, but I think for the most part it's a necessary evil until a complete reference implementation of the Python core is complete.

And maybe there's a middle-ground where this can be done modularly, where small parts of the core are written in Rust and are exposed to the rest of the Python core using pyo3. I think there's room for stuff like that to be hammered out, and if we're serious about introducing Rust in the future then I'll also make the effort to actively study it to better prepare for that change. But a complete Rust core replacement I feel is out of scope at this time.

---

Ultimately incremental changes are the way to go, as long as they don't currently interfere with other roadmap items being worked on in open PRs, open feature branches, or upcoming milestones. I've recently given the feature roadmap a refresh that should also better align with which release cycles I see new features landing, with 9.6.x involving a lot of heavy lifting on database changes that have been needing to be made for a while.