fix: runtime services are fixed

[marslanabdulrauf]

Related ticket

https://github.com/mitodl/hq/issues/9621 (MIT Internal)

Discussion

https://discuss.openedx.org/t/recalculate-subsection-grade-v3-is-submitted-with-the-wrong-user-id/17873/12?u=muhammad_arslan

Description

This pull request makes a minor change to how services are handled when creating a runtime for a course. Instead of using the original self.services dictionary directly, the code now uses a copy of it to prevent unintended side effects from modifications.

• Use a copy of the services dictionary in create_runtime to avoid mutating the original self.services when creating a runtime.

Steps to reproduce the issue:

Follow the steps mentioned in the discussion post: https://discuss.openedx.org/t/recalculate-subsection-grade-v3-is-submitted-with-the-wrong-user-id/17873/12?u=muhammad_arslan

Testing instructions

Follow the same steps and now each user should have their own submission

[openedx-webhooks]

Thanks for the pull request, @marslanabdulrauf!

This repository is currently maintained by @openedx/wg-maintenance-edx-platform.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[ormsbee]

I don't understand how this helps. Isn't this just a shallow copy? So the copy and the original still point at the same shared objects?

[marslanabdulrauf]

There were 3-4 common objects, I haven't explored all of them but yeah they might be sharing the same objects. Let me deep copy this service object

[ormsbee]

One of my higher level concerns is that I don't know if this is happening because of something weird that the XBlock runtime or grading system is specifically doing, or whether it's a more general problem with an underlying piece of infrastructure like the RequestCache, which we use in a lot of places.

[ormsbee]

This was discussed in yesterday's BTR meeting, but to reiterate here:

1. If you folks can reliably reproduce the bug in dev and you've verified that this patch fixes it, I'll approve that as a short term fix.
2. If this turns out to be a more general problem that was exposed by MIT's deployment shift to granian (and isn't present in uwsgi or gunicorn), then this bug will not be considered a release blocker for Ulmo.
3. This is important for us to fix, even if it's only exposed by deployment changes, since it implies that we're doing something very wrong with multithreading.

Thank you for investigating this.

[bradenmacdonald]

So, as I understand it, SplitMongoModuleStore is supposed to be a user-agnostic singleton, and when it uses create_runtime it initiates SplitModuleStoreRuntime, which doesn't accept a user parameter but is user-specific and gets the current user using CRUM.

So it makes sense that the services dict should be shallow copied, so the runtime inherits the modulestore user-agnostic services, but cannot add user-specific services to the modulestore services dict. But this services = self.services line has been in place for 9 years, so I don't understand why it ever worked at all? I guess because uwsgi was creating a new modulestore for each request, perhaps?

[ormsbee]

Oh, I see. I was missing the fact that we're adding services for the user (since the user_service is added at the SplitMongoModuleStore level). Then yes, shallow copy makes sense.

@bradenmacdonald: If the issue is around content library content (e.g. this user-specific initialization of the library_tools service in SplitModuleStoreRuntime), it might be that this has been around a while but MIT uses content libraries enough and has enough scale where it shows up more noticeably?

[ormsbee]

But another data point is that something changed around early/mid November that seemed to trigger this. I suppose it could have been content-specific (some big course opening then), but I want to poke around a bit and see if any code landed around then that could have caused this.

[bradenmacdonald]

@ormsbee I suspect now that the reason this services = self.services code looks problematic but hasn't caused bugs before, is that it doesn't generally matter - most of the services get overwritten again afterward here by prepare_runtime_for_user. So the user-specific services may leak into the user-agnostic modulestore, but they're generally going to be overwritten anyways when a new runtime is created. Until they're not - if there's some codepath that re-uses the modulestore singleton to create a runtime but doesn't call prepare_runtime_for_user (?), then we'd see this issue.

[ormsbee]

Okay, if that's really it then, then the shallow copy sounds fine as a short term thing. In the longer term, I wonder if there's really even a need to keep the global around, or if we can always re-instantiate the entire SplitMongoModuleStore on a per-request basis.