Python bindings do not account for invalid UTF-8 strings

[bdash]

Version and Platform (required):

• Binary Ninja Version: 5.1.7320-dev (aa12054b)
• OS: macOS 15.4.1
• CPU Architecture: arm64

Bug Description:
BnString::as_str panics if the underlying string is not valid UTF-8. Many of these strings come from core which does not make any guarantees about their UTF-8ness. Symbol names, for instance, may not be valid UTF-8.

BnString is described as being analogous to CString. The similar methods on CString are failable:

pub fn into_string(self) -> Result<String, IntoStringError>
pub fn to_str(&self) -> Result<&str, Utf8Error>

IMO BnString::as_str should return Result and let the caller unwrap() if that's what they want.

It might be worth renaming as_str to to_str to match the naming on CStr.

[emesare]

Just had a discussion internally about this, this is also a problem for python, I think that the right answer is what you propose. If we want to unwrap there we need actual guarantees that the string is valid UTF-8 when passed back from the core, some places this is not possible, obviously, but there are definitely quite a few APIs where we could say this true and just return a String for simplicity, the user is unlikely to want to keep the core string around, this is more inline with how our C++ api operates.

[bdash]

This also affects BnString::as_bytes since for some reason that delegates to BnString::as_str.

[emesare]

9f937ea

I went and made something preliminary, limiting the use of BnString purposely is likely the way forward here, we want to give control over invalid UTF-8 in certain limited scopes, particularly around symbols. Basically anything that can conceivably be round-tripped needs to have the option to be invalid UTF-8 so that remaking for example, a symbol, would be lossless.

I still have no idea what we are doing for python, the consensus there was to provide separate API's.

[emesare]

I believe we also might want to adjust InstructionTextToken to contain a BnString as that needs to be lossless.

binaryninja-api/rust/src/disassembly.rs

Line 258 in a7e0392

pub text: String,

[emesare]

Renamed this be broader and capture python API as well, to make my life easier.

[emesare]

From slack: https://binaryninja.slack.com/archives/C1E616ZSQ/p1746115275818749?thread_ts=1745698581.933199&cid=C1E616ZSQ

So all the following method for a function with a non utf-8 character would trigger an exception like 'utf-8' codec can't decode byte 0x88 in position 6: invalid start byte,

• func.hlil (UI correctly handles the culprit line like int32_t hFile = GetStd�and,e(eax_11))
• func.mlil (UI correctly handles the culprit line like hFile = GetStd�and,e(var_5c_2))
• block.disassembly_text (UI correctly handles the culprit line like call dword [GetStd�and,e])

Likely need to make X_bytes versions of the above API's in python. See the CoreSymbol API for an example: https://api.binary.ninja/binaryninja.types-module.html#binaryninja.types.CoreSymbol.raw_bytes

Unlike the Rust API changes, this will be more targeted, maybe just here

binaryninja-api/python/architecture.py

Lines 2865 to 2866 in 12f1e2e

	if not isinstance(text, str):
	text = text.decode("utf-8")

and here, to round-trip invalid utf-8 text tokens:

binaryninja-api/python/architecture.py

Lines 2905 to 2906 in 12f1e2e

	for i in range(len(tokens[j].typeNames)):
	result[j].typeNames[i] = tokens[j].typeNames[i].encode("utf-8")