[WIP] Remove deprecated OpenSSL RSA APIs by PranavSenthilnathan · Pull Request #126034 · dotnet/runtime

PranavSenthilnathan · 2026-03-24T15:59:21Z

Remove deprecated OpenSSL RSA APIs. The RSA type and RSA_* methods have been deprecated in favor of EVP_PKEY. Most of our code already has moved, but this cleans up the function loading and updates the remaining places. The build allows deprecated APIs, but for local testing I've been using -DOPENSSL_API_COMPAT=0x30500000L -DOPENSSL_NO_DEPRECATED to error on deprecated APIs.

dotnet-policy-service · 2026-03-24T16:00:29Z

Tagging subscribers to this area: @bartonjs, @vcsjones, @dotnet/area-system-security
See info in area-owners.md if you want to be subscribed.

Copilot

Pull request overview

This PR continues the migration away from deprecated OpenSSL RSA* APIs toward EVP-based APIs (primarily EVP_PKEY), updating certificate generation and key-validation paths, and adjusting shim/configure logic to better tolerate RSA legacy symbol availability.

Changes:

Update self-signed certificate generation to use/generated EVP_PKEY* directly (removing EVP_PKEY_get1_RSA / EVP_PKEY_set1_RSA usage in that flow).
Add an EVP-based RSA “quick check” (EVP_PKEY_get_bn_param) for OpenSSL 3+ to avoid relying on legacy RSA object access.
Add HAVE_OPENSSL_RSA_PRIMITIVE configure/shim support and relax several RSA-related shim bindings to “lightup” (runtime-detected) functions.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/native/libs/System.Security.Cryptography.Native/pal_ssl.c	Switch self-signed cert helper to return the generated `EVP_PKEY` instead of extracting/setting a legacy `RSA`.
src/native/libs/System.Security.Cryptography.Native/pal_evp_pkey_rsa.c	Factor legacy private-key-availability check into a helper and gate it on `EVP_PKEY_get0_RSA` lightup availability.
src/native/libs/System.Security.Cryptography.Native/pal_evp_pkey.c	Introduce `QuickRsaCheckEvp` using `EVP_PKEY_get_bn_param`; adjust RSA validation flow to prefer EVP-param extraction.
src/native/libs/System.Security.Cryptography.Native/pal_crypto_config.h.in	Add `HAVE_OPENSSL_RSA_PRIMITIVE` feature define.
src/native/libs/System.Security.Cryptography.Native/osslcompat_30.h	Add RSA parameter-name constants needed for EVP param extraction on older headers.
src/native/libs/System.Security.Cryptography.Native/opensslshim.h	Add RSA primitive fallback declarations and convert multiple RSA-related bindings to lightup functions.
src/native/libs/System.Security.Cryptography.Native/configure.cmake	Add compile probe for RSA primitive availability (`RSA_new`/`RSA_free`).

Copilot · 2026-03-24T16:09:18Z

src/native/libs/System.Security.Cryptography.Native/configure.cmake

+check_source_compiles(C "
+#include <openssl/rsa.h>
+int main(void) { RSA *r = RSA_new(); RSA_free(r); return 0; }"
+HAVE_OPENSSL_RSA_PRIMITIVE)


The new HAVE_OPENSSL_RSA_PRIMITIVE probe compiles with only OPENSSL_API_COMPAT=0x10100000L (from CMAKE_REQUIRED_DEFINITIONS). If the build is configured with additional flags like OPENSSL_NO_DEPRECATED / higher OPENSSL_API_COMPAT (as mentioned in the PR description), this check can report a different result than the actual compile, causing HAVE_OPENSSL_RSA_PRIMITIVE to be incorrectly set and the shim to miss needed fallback declarations. Consider making the probe honor the effective build definitions (or explicitly adding the deprecation-related defines to CMAKE_REQUIRED_DEFINITIONS when they are enabled) so the result matches the compilation mode.

Copilot · 2026-03-24T16:09:18Z

src/native/libs/System.Security.Cryptography.Native/pal_ssl.c

@@ -1052,9 +1052,8 @@ int32_t CryptoNative_SslGetCurrentCipherId(SSL* ssl, int32_t* cipherId)
 }

 // This function generates key pair and creates simple certificate.


MakeSelfSignedCertificate now returns the generated key via an out parameter (EVP_PKEY** pEvp), but the comment still describes only certificate generation. Update the comment to reflect the new ownership/contract (e.g., that the caller receives an EVP_PKEY* they must eventually free) to avoid misuse.

Suggested change

// This function generates key pair and creates simple certificate.

// This function generates a key pair and creates a simple self-signed certificate.

// On success (nonzero return), it stores the generated key in *pEvp; the caller then owns

// that EVP_PKEY* and is responsible for freeing it with EVP_PKEY_free when no longer needed.

Copilot · 2026-03-24T16:09:19Z

src/native/libs/System.Security.Cryptography.Native/pal_evp_pkey.c

+        if (API_EXISTS(EVP_PKEY_get_bn_param))
        {
-            //  0 = key check failed
-            //  1 = key check passed
-            // -1 = could not assess private key.
-            int32_t result = QuickRsaCheck(rsa, isPublic);
+            // OpenSSL 3.0+: use EVP_PKEY_get_bn_param to extract RSA components directly.
+            result = QuickRsaCheckEvp(key, isPublic);
+        }


EVP_PKEY_get_bn_param is gated only by API_EXISTS, but in non-distro-agnostic builds API_EXISTS() is always true. This makes the OpenSSL < 3.0 build try to compile/call EVP_PKEY_get_bn_param even though it isn't declared/available there, which can break non-shim builds against OpenSSL 1.0/1.1. Consider wrapping the EVP_PKEY_get_bn_param/QuickRsaCheckEvp path in #ifdef NEED_OPENSSL_3_0 (or an equivalent compile-time OPENSSL_VERSION_NUMBER >= 3.0 || FEATURE_DISTRO_AGNOSTIC_SSL guard) and keep the EVP_PKEY_get0_RSA/EVP_PKEY_check fallback for older headers.

Copilot · 2026-03-24T16:09:19Z

src/native/libs/System.Security.Cryptography.Native/opensslshim.h

    REQUIRED_FUNCTION(EVP_PKEY_set1_DSA) \
    REQUIRED_FUNCTION(EVP_PKEY_set1_EC_KEY) \
-    REQUIRED_FUNCTION(EVP_PKEY_set1_RSA) \
+    LIGHTUP_FUNCTION(EVP_PKEY_set1_RSA) \
    REQUIRED_FUNCTION(EVP_PKEY_sign) \


EVP_PKEY_set1_RSA was changed to a LIGHTUP_FUNCTION, which means the function pointer can be NULL at runtime in distro-agnostic builds (e.g., OpenSSL built with no-deprecated). Any direct call sites need to check API_EXISTS(EVP_PKEY_set1_RSA) and fail gracefully instead of calling through a NULL pointer. Alternatively, keep it as REQUIRED_FUNCTION if the runtime contract is that it must always exist.

bartonjs · 2026-03-25T17:59:11Z

src/native/libs/System.Security.Cryptography.Native/pal_evp_pkey_rsa.c

+{
+    if (!API_EXISTS(EVP_PKEY_get0_RSA))
+    {
+        ERR_PUT_error(ERR_LIB_RSA, RSA_F_RSA_NULL_PRIVATE_DECRYPT, RSA_R_VALUE_MISSING, __FILE__, __LINE__);


If this function is only called for OSSL 1.x, EVP_PKEY_get0_RSA is pretty well guaranteed to exist.

I think there's a different macro in the shim loader to make the shim assert it... LEGACY_FUNCTION, maybe? Or you can just put a comment in this block that it's not expected to be reachable, but if somehow we get a 1.x with no EVP_PKEY_get0_RSA this gives an exception instead of a crash.

LEGACY_FUNCTION I nuked that when I removed support for OpenSSL 1.0 because it was based on the v1_0_sentinel. We could bring something like it back with a 1.1 sentinel function.

Remove deprecated RSA APIs

c4c843a

PranavSenthilnathan added this to the 11.0.0 milestone Mar 24, 2026

PranavSenthilnathan self-assigned this Mar 24, 2026

Copilot AI review requested due to automatic review settings March 24, 2026 15:59

PranavSenthilnathan added the area-System.Security label Mar 24, 2026

Copilot started reviewing on behalf of PranavSenthilnathan March 24, 2026 16:00 View session

Copilot AI reviewed Mar 24, 2026

View reviewed changes

bartonjs reviewed Mar 25, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] Remove deprecated OpenSSL RSA APIs#126034

[WIP] Remove deprecated OpenSSL RSA APIs#126034
PranavSenthilnathan wants to merge 1 commit intodotnet:mainfrom
PranavSenthilnathan:ossl-deprecations

PranavSenthilnathan commented Mar 24, 2026

Uh oh!

dotnet-policy-service bot commented Mar 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

Copilot AI Mar 24, 2026

Uh oh!

bartonjs Mar 25, 2026

Uh oh!

vcsjones Mar 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@@ -1052,9 +1052,8 @@ int32_t CryptoNative_SslGetCurrentCipherId(SSL* ssl, int32_t* cipherId)
		}

		// This function generates key pair and creates simple certificate.

-// This function generates key pair and creates simple certificate.
+// This function generates a key pair and creates a simple self-signed certificate.
+// On success (nonzero return), it stores the generated key in *pEvp; the caller then owns
+// that EVP_PKEY* and is responsible for freeing it with EVP_PKEY_free when no longer needed.

Conversation

PranavSenthilnathan commented Mar 24, 2026

Uh oh!

dotnet-policy-service bot commented Mar 24, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 24, 2026

Choose a reason for hiding this comment

Uh oh!

bartonjs Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

vcsjones Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants