fix: race condition in shared runtime services

[marslanabdulrauf]

Related ticket

https://github.com/mitodl/hq/issues/9621 (MIT Internal)

Discussion

https://discuss.openedx.org/t/recalculate-subsection-grade-v3-is-submitted-with-the-wrong-user-id/17873/12?u=muhammad_arslan

Description

This pull request makes a minor change to how services are handled when creating a runtime for a course. Instead of using the original self.services dictionary directly, the code now uses a copy of it to prevent unintended side effects from modifications.

• Use a copy of the services dictionary in create_runtime to avoid mutating the original self.services when creating a runtime.

Steps to reproduce the issue:

Follow the steps mentioned in the discussion post: https://discuss.openedx.org/t/recalculate-subsection-grade-v3-is-submitted-with-the-wrong-user-id/17873/12?u=muhammad_arslan

Testing instructions

Follow the same steps and now each user should have their own submission

[openedx-webhooks]

Thanks for the pull request, @marslanabdulrauf!

This repository is currently maintained by @openedx/wg-maintenance-edx-platform.

Once you've gone through the following steps feel free to tag them in a comment and let them know that your changes are ready for engineering review.

🔘 Get product approval

If you haven't already, check this list to see if your contribution needs to go through the product review process.

• If it does, you'll need to submit a product proposal for your contribution, and have it reviewed by the Product Working Group.
  • This process (including the steps you'll need to take) is documented here.
• If it doesn't, simply proceed with the next step.

🔘 Provide context

To help your reviewers and other members of the community understand the purpose and larger context of your changes, feel free to add as much of the following information to the PR description as you can:

• Dependencies

  This PR must be merged before / after / at the same time as ...

• Blockers

  This PR is waiting for OEP-1234 to be accepted.

• Timeline information

  This PR must be merged by XX date because ...

• Partner information

  This is for a course on edx.org.

• Supporting documentation
• Relevant Open edX discussion forum threads

🔘 Get a green build

If one or more checks are failing, continue working on your changes until this is no longer the case and your build turns green.

Details

---

Where can I find more information?

If you'd like to get more details on all aspects of the review process for open source pull requests (OSPRs), check out the following resources:

• Overview of Review Process for Community Contributions
• Pull Request Status Guide
• Making changes to your pull request

When can I expect my changes to be merged?

Our goal is to get community contributions seen and reviewed as efficiently as possible.

However, the amount of time that it takes to review and merge a PR can vary significantly based on factors such as:

• The size and impact of the changes that it introduces
• The need for product review
• Maintenance status of the parent repository

💡 As a result it may take up to several weeks or months to complete a review and merge your PR.

[ormsbee]

I don't understand how this helps. Isn't this just a shallow copy? So the copy and the original still point at the same shared objects?

[marslanabdulrauf]

There were 3-4 common objects, I haven't explored all of them but yeah they might be sharing the same objects. Let me deep copy this service object

[ormsbee]

FWIW, based on the discussions below, if you folks have verified that it's fine with a shallow copy, let's keep it at that. Thank you.

[farhaanbukhsh]

Hey @ormsbee @marslanabdulrauf if possible, can either of you add a verbose comment about the fix and why are we using copy here.

[marslanabdulrauf]

We are adding a detailed commit message

If you still think, I can add more details here as well

[ormsbee]

@farhaanbukhsh, @marslanabdulrauf: I think a short message above this is reasonable. How about:

A single SplitMongoModuleStore may create many SplitModuleStoreRuntimes, each of which will later modify its internal dict of services on a per-item and often per-user basis. Therefore, it's critical that we make a new copy of our baseline services dict here, so that each runtime is free to add and replace its services without impacting other runtimes.

[farhaanbukhsh]

@ormsbee @marslanabdulrauf that works, thanks a lot

[ormsbee]

One of my higher level concerns is that I don't know if this is happening because of something weird that the XBlock runtime or grading system is specifically doing, or whether it's a more general problem with an underlying piece of infrastructure like the RequestCache, which we use in a lot of places.

[ormsbee]

This was discussed in yesterday's BTR meeting, but to reiterate here:

1. If you folks can reliably reproduce the bug in dev and you've verified that this patch fixes it, I'll approve that as a short term fix.
2. If this turns out to be a more general problem that was exposed by MIT's deployment shift to granian (and isn't present in uwsgi or gunicorn), then this bug will not be considered a release blocker for Ulmo.
3. This is important for us to fix, even if it's only exposed by deployment changes, since it implies that we're doing something very wrong with multithreading.

Thank you for investigating this.

[bradenmacdonald]

So, as I understand it, SplitMongoModuleStore is supposed to be a user-agnostic singleton, and when it uses create_runtime it initiates SplitModuleStoreRuntime, which doesn't accept a user parameter but is user-specific and gets the current user using CRUM.

So it makes sense that the services dict should be shallow copied, so the runtime inherits the modulestore user-agnostic services, but cannot add user-specific services to the modulestore services dict. But this services = self.services line has been in place for 9 years, so I don't understand why it ever worked at all? I guess because uwsgi was creating a new modulestore for each request, perhaps?

[ormsbee]

Oh, I see. I was missing the fact that we're adding services for the user (since the user_service is added at the SplitMongoModuleStore level). Then yes, shallow copy makes sense.

@bradenmacdonald: If the issue is around content library content (e.g. this user-specific initialization of the library_tools service in SplitModuleStoreRuntime), it might be that this has been around a while but MIT uses content libraries enough and has enough scale where it shows up more noticeably?

[ormsbee]

But another data point is that something changed around early/mid November that seemed to trigger this. I suppose it could have been content-specific (some big course opening then), but I want to poke around a bit and see if any code landed around then that could have caused this.

[bradenmacdonald]

@ormsbee I suspect now that the reason this services = self.services code looks problematic but hasn't caused bugs before, is that it doesn't generally matter - most of the services get overwritten again afterward here by prepare_runtime_for_user. So the user-specific services may leak into the user-agnostic modulestore, but they're generally going to be overwritten anyways when a new runtime is created. Until they're not - if there's some codepath that re-uses the modulestore singleton to create a runtime but doesn't call prepare_runtime_for_user (?), then we'd see this issue.

[ormsbee]

Okay, if that's really it then, then the shallow copy sounds fine as a short term thing. In the longer term, I wonder if there's really even a need to keep the global around, or if we can always re-instantiate the entire SplitMongoModuleStore on a per-request basis.

[ormsbee]

Yeah, it's doing a lot less work than it used to. Most of the time spent is spent setting up the MongoDB connection, and PyMongo uses its own connection pooling on the backend, so we don't even really need to worry about that. We should just yank it for Verawood.

[ormsbee]

Maybe another tangent, but there is a big refactoring PR that touches this stuff that merged at the end of October: #35523 I don't see anything in it that I think could cause this, but given that the release where MIT started seeing issues was on Nov 12th, the timing is suspicious. FYI @kdmccormick

(It's also entirely possible that it was triggered by a particular set of content that a course was using at that time, and was not linked to any recent change.)

Yet another tangent: I made a PR to play with removing the global caching of modulestore altogether. It seems to work fine--the tests that I'm seeing break right now are query count related. It's not a serious contender for patching the release though.

[ormsbee]

FYI for folks watching the ticket: @feoh gave an update earlier today saying that the Granian switchover didn't happen until Dec 22, which means it was not involved in these errors.

[ormsbee]

@marslanabdulrauf: Please pull in my commit from #37850

I'm planning to use the following commit message for the squashed commit:

fix: race condition in shared runtime services (#37825)

There is a singleton SplitMongoModuleStore instance that is returned
whenever we call the ubiquitous modulestore() function (wrapped in a
MixedModuleStore). During initialization, SplitMongoModuleStore sets
up a small handful of XBlock runtime services that are intended to be
shared globally: i18n, fs, cache.

When we get an individual block back from the store using get_item(),
SplitMongoModuleStore creates a SplitModuleStoreRuntime using
SplitMongoModuleStore.create_runtime(). These runtimes are intended to
be modified on a per-item, and later per-user basis (using
prepare_runtime_for_user()).

Prior to this commit, the create_runtime() method was assigning the
globally shared SplitMongoModuleStore.services dict directly to the
newly instantiated SplitModuleStoreRuntime. This meant that even though
each block had its own _services dict, they were all in fact pointing
to the same underlying object. This exposed us to a risk of multiple
threads contaminating each other's SplitModuleStoreRuntime services
when deployed under load in multithreaded mode. We believe this led to
a race condition that caused student submissions to be mis-scored in
some cases.

This commit makes a copy of the SplitMongoModuleStore.services dict for
each SplitModuleStoreRuntime. The baseline global services are still
shared, but other per-item and per-user services are now better
isolated from each other.

This commit also includes a small modification to the PartitionService,
which up until this point had relied on the (incorrect) shared instance
behavior. The details are provided in the comments in the
PartitionService __init__().

It's worth noting that the historical rationale for having a singleton
ModuleStore instance is that the ModuleStore used to be extremely
expensive to initialize. This was because at one point, the init
process required reading entire XML-based courses into memory, or
pre-computing complex field inheritance caches. This is no longer the
case, and SplitMongoModuleStore initialization is in the 1-2 ms range,
with most of that being for PyMongo's connection setup. We should try
to fully remove the global singleton in the Verawood release cycle in
order to make this kind of bug less likely.

[feanil]

Changes make sense, I think this is good to go once the small comment suggestion that Dave made in the comment on line 3286 has been added.