Resolve Relevancy Task Error

[maxachis]

After uploading URLs from label studio for #161, the relevancy task kicked off and yielded the following error:

Could not parse JSON from subprocess: 
Traceback (most recent call last):
  File "/app/core/classes/task_operators/TaskOperatorBase.py", line 50, in run_task
    await self.inner_task_logic()
  File "/app/core/classes/task_operators/URLRelevanceHuggingfaceTaskOperator.py", line 33, in inner_task_logic
    await self.add_huggingface_relevancy(tdos)
  File "/app/core/classes/task_operators/URLRelevanceHuggingfaceTaskOperator.py", line 49, in add_huggingface_relevancy
    results = await self.huggingface_interface.get_url_relevancy_async(urls_with_html)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/hugging_face/HuggingFaceInterface.py", line 33, in get_url_relevancy_async
    raise RuntimeError(f"Could not parse JSON from subprocess: {raw_output}")
RuntimeError: Could not parse JSON from subprocess: 

Let's look into that, shall we?

[maxachis]

This error may also be affecting Task Agency Identification -- it ran 20 times in a row, causing it to be automatically terminated.

Edit: That was an unrelated error that has been addressed in #266

[maxachis]

So the problem may have something to do with dependencies and subprocesses related to what I did in #185, where I tried to break the application off to a separate subprocess and only initialize it when needed. I haven't yet pinpointed exactly why it's erroring, but this seems to be the cause of the problem.

Reflecting on it now, however, I'm wondering if the Huggingface Task Operator should even be a part of the main app, or if it should be broken into a microservice which sends information to the SC app. My thinking is:

1. Relevancy doesn't really work very well now in the first place
2. Huggingface has a lot of dependencies which slow down setup and increase the size of the docker files to a pretty substantial degree. If we only use it infrequently, maybe it should be something that we spin up and call separately in automation, sending information to the SC via an API, rather than as part of the internal suite of tasks.

To that end, I'm contemplating doing the following:

1. Uninstall the Huggingface dependencies so as to reduce time to startup and overall file size
2. Leave Relevancy Classification inactive for the moment until we
3. Develop a better relevancy classifier
4. And set it up as a microservice in a separate repository.

@josh-chamberlain What do you think?

[josh-chamberlain]

@maxachis your steps sound OK to me, and I agree that it's not core functionality.

Note: our relevancy classifier will get better as we update the training data—we should still have processes for feeding new training data into HF and periodically re-training to see whether it improves. "Develop a better relevancy classifier" is a thing we are passively doing with every label (with room for breakthroughs).

That said—I think we could be interfacing with huggingface via periodically-run scripts which cause data to sync; I'm envisioning something like a source collector app endpoint which, when used, causes huggingface to be updated. Automation could follow. Basically, I see HF as a resource which SC could periodically use, rather than an ongoing dependency, as you way.

Does this generally make sense? I can't tell if I'm just talking past you 😬

[maxachis]

@josh-chamberlain I think we're actually quite aligned! I can see building a microservices architecture to do this! Here's what I'm thinking:

1. We build a huggingface-trainer repository, which handles these HFdependencies
2. We add logic in the SC database indicating when a URL has been submitted into HF
3. We create a /huggingface/urls GET SC endpoint that gets URLs and the associated endpoints that have not yet been added to HF...
4. Then create a corresponding /huggingface/urls POST endpoint indicating that those URLs have been added to HF.
5. The trainer repository is designed to call the endpoint in /huggingface GET, upload the relevant data into HF, and then call /huggingface POST when done
6. We can call this on a periodic basis, or manually, using it as we see fit. ¹

Then, when we're ready to add automatic labeling...

1. Create a /huggingface/relevance GET endpoint, which retrieves URLs that need automatic labeling
2. Create a /huggingface/relevance POST endpoint, which submits automatic labels to the associated URLs
3. Use our huggingface-trainer repository to GET the URLs, train them via HF, and then POST the URLs back to SC

In the meantime, I can get started on removing our existing HF logic from the SC repository, and put it in huggingface-trainer, just so we can store it somewhere for now where it won't take up space in SC.

What do you think of all that?

Footnotes

1. There will be a wrinkle with regards to hosting it, since it's comparatively resource-intensive compared to the API, and doesn't have to be always available. Perhaps we let automation manager start it up every now and then. ↩

[maxachis]

And here is the repository for that trainer!

https://github.com/Police-Data-Accessibility-Project/huggingface-trainer

[maxachis]

Removed the dependencies! The most recent build time was 3/5ths the prior build time 💪

[maxachis]

@josh-chamberlain Following up on this to get your thoughts!

[josh-chamberlain]

@maxachis dang, sorry for my delay! Not sure how this slipped through the cracks in May.

I like the name huggingface-trainer better, as it feels more descriptive; data-analysis could be anything. Is it maybe more accurate to call it huggingface-integration or something? We use it to send URLs back and forth, modify training data, train models.

We actually use the models separately, via the inference API, right? In that case, we should definitely only spin it up as-needed! If we need to have it up in order to use the models, that's different.