gh-142495: make defaultdict keep existed value when racing with __missing__

[LindaSummer]

Issue

#142495

Proposed Changes

• Use PyDict_SetDefaultRef to guard the object update and keep existed value.
• Add a test case for the racing case

Comment

The PyDict_SetDefaultRef would increase the reference count of the setted item, so we must decrement the refrence count on __missing__ created object to avoid memory leak.

• Issue: defaultdict.__missing__ has data races #142495

[serhiy-storchaka]

Does it need threads to reproduce? Would not the following code be enough to reproduce the issue?

        count = 0
        def default_factory():
            nonlocal count
            count += 1
            if count == 1:
                test_dict[key]
            return count

[LindaSummer]

Does it need threads to reproduce? Would not the following code be enough to reproduce the issue?

        count = 0
        def default_factory():
            nonlocal count
            count += 1
            if count == 1:
                test_dict[key]
            return count

Hi @serhiy-storchaka ,

Thanks very much for your suggestion! ❤

Yes, this case could be designed simply without threads.

I have updated the test case as suggested.

Best Regards,
Edward

[serhiy-storchaka]

Great! But similar tests usually use a local function with a nonlocal variable.

[LindaSummer]

Great! But similar tests usually use a local function with a nonlocal variable.

Hi @serhiy-storchaka ,

Thanks very much for your suggestion! 😊
Got it. I will refactor the test case with nonlocal variables for simplicity.

Best Regards,
Edward

[chaope]

If we don't need the return value, could we use PyDict_SetDefault?

[picnixz]

PyDict_SetDefault returns a borrowed reference and I find this easier to read (at least the result is know to be a strong reference and we can directly return it).

[serhiy-storchaka]

Does the new test reproduce the issue? It seems that __missing__() is not called recursively in your current code, unlike to the code in my suggestion.