Skip to content

Commit 3bb2f23

Browse files
anarthalanarthal
committed
Ruben 2025 Q2
1 parent 05b7c65 commit 3bb2f23

File tree

1 file changed

+313
-0
lines changed

1 file changed

+313
-0
lines changed

‎_posts/2025-07-10-Ruben2025Q2Update.md‎

Lines changed: 313 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,313 @@
1+
---
2+
layout: post
3+
nav-class: dark
4+
categories: ruben
5+
title: "Ready, Set, Redis!"
6+
author-id: ruben
7+
author-name: Rubén Pérez Hidalgo
8+
---
9+
10+
I'm happy to announce that I'm now a co-maintainer of [Boost.Redis](https://github.com/boostorg/redis),
11+
a high-level Redis client written on top of Asio, and hence sister of Boost.MySQL.
12+
I'm working with its author, Marcelo, to make it even better than what it is now
13+
(and that's a lot to say).
14+
15+
First of all, we're working on improving test coverage. Boost.Redis was originally
16+
written making heavy use of `asio::async_compose`. If you have a
17+
JavaScript or Python background, this approach will feel natural to you, since it's similar
18+
to `async`/`await`. Unfortunately, this approach makes code difficult to test.
19+
20+
For instance, consider `redis::connection::async_exec`, which enqueues a request to be
21+
executed by the Redis server and then waits for its response. This is a (considerably simplified)
22+
snippet of how this function could be implemented with `async_compose`:
23+
24+
```cpp
25+
struct exec_op {
26+
connection* conn;
27+
std::shared_ptr<detail::request_info> info; // a request + extra info
28+
asio::coroutine coro{}; // a coroutine polyfill that uses switch/case statements
29+
30+
// Will be called by Asio until self.complete() is called
31+
template <class Self>
32+
void operator()(Self& self , system::error_code = {}, std::size_t = 0)
33+
{
34+
BOOST_ASIO_CORO_REENTER(coro)
35+
{
36+
// Check whether the user wants to wait for the connection to
37+
// be stablished.
38+
if (info->req->get_config().cancel_if_not_connected && !conn->state.is_open) {
39+
BOOST_ASIO_CORO_YIELD asio::async_immediate(self.get_io_executor(), std::move(self));
40+
return self.complete(error::not_connected, 0);
41+
}
42+
43+
// Add the request to the queue
44+
conn->state.add_request(info);
45+
46+
// Notify the writer task that there is a new request available
47+
conn->writer_timer.cancel();
48+
49+
while (true) {
50+
// Wait for the request to complete. We will be notified using a channel
51+
BOOST_ASIO_CORO_YIELD info->channel.async_wait(std::move(self));
52+
53+
// Are we done yet?
54+
if (info->is_done) {
55+
self.complete(info->ec, info->bytes_read);
56+
return;
57+
}
58+
59+
// Check for cancellations
60+
if (self.get_cancellation_state().cancelled() != asio::cancellation_type_t::none) {
61+
// We can only honor cancellations if the request hasn't been sent to the server
62+
if (!info->request_sent()) {
63+
conn->state.remove_request(info);
64+
self.complete(asio::error::operation_aborted, 0);
65+
return;
66+
} else {
67+
// Can't cancel, keep waiting
68+
}
69+
}
70+
}
71+
}
72+
}
73+
};
74+
75+
template <class CompletionToken, class Response>
76+
auto connection::async_exec(const request& req, Response& res, CompletionToken&& token)
77+
{
78+
return asio::async_compose<CompletionToken, void(system::error_code, std::size_t)>(
79+
exec_op{this, detail::make_info(req, res)},
80+
token,
81+
*this
82+
);
83+
}
84+
```
85+
86+
The snippet above contains non-trivial logic, specially regarding cancellation.
87+
While the code is understandable, it is difficult to test,
88+
since Asio doesn't include lots of testing utilities.
89+
The end result is usually untested code, prune to difficult to diagnose bugs.
90+
91+
As an alternative, we can refactor this code into two classes:
92+
93+
* A finite state machine object that encapsulates all logic. This should be a
94+
lightweight and should never interact with any Asio I/O object, so we can test
95+
logic easily.
96+
* A dumb `async_compose` function that just applies the actions mandated by the
97+
finite state machine.
98+
99+
The finite state machine for the above code could be like:
100+
101+
```cpp
102+
103+
// The finite state machine returns exec_action objects
104+
// communicating what should be done next so the algorithm can progress
105+
enum class exec_action_type
106+
{
107+
notify_writer, // We should notify the writer task
108+
wait, // We should wait for the channel to be notified
109+
immediate, // We should invoke asio::async_immediate() to avoid recursion problems
110+
done, // We're done and should call self.complete()
111+
};
112+
113+
struct exec_action
114+
{
115+
exec_action_type type;
116+
error_code ec; // has meaning if type == exec_action_type::done
117+
std::size_t bytes_read{}; // has meaning if type == exec_action_type::done
118+
};
119+
120+
// Contains all the algorithm logic. It is cheap to create and copy.
121+
// It is conceptually similar to a coroutine.
122+
class exec_fsm {
123+
asio::coroutine coro{};
124+
std::shared_ptr<detail::request_info> info; // a request + extra info
125+
public:
126+
explicit exec_fsm(std::shared_ptr<detail::request_info> info) : info(std::move(info)) {}
127+
128+
std::shared_ptr<detail::request_info> get_info() const { return info; }
129+
130+
// To run the algorithm, run the resume() function until it returns exec_action_type::done.
131+
exec_action resume(
132+
connection_state& st, // Contains connection state, but not any I/O objects
133+
asio::cancellation_type_t cancel_state // The cancellation state of the composed operation
134+
)
135+
{
136+
BOOST_ASIO_REENTER(coro)
137+
{
138+
// Check whether the user wants to wait for the connection to
139+
// be stablished.
140+
if (info->req->get_config().cancel_if_not_connected && !st.is_open) {
141+
BOOST_ASIO_CORO_YIELD {exec_action_type::immediate};
142+
return {exec_action_type::done, error::not_connected, 0};
143+
}
144+
145+
// Add the request to the queue
146+
st.add_request(info);
147+
148+
// Notify the writer task that there is a new request available
149+
BOOST_ASIO_CORO_YIELD {exec_action_type::notify_writer};
150+
151+
while (true) {
152+
// Wait for the request to complete. We will be notified using a channel
153+
BOOST_ASIO_CORO_YIELD {exec_action_type::wait};
154+
155+
// Are we done yet?
156+
if (info->is_done) {
157+
return {exec_action_type::done, info->ec, info->bytes_read};
158+
}
159+
160+
// Check for cancellations
161+
if (cancel_state != asio::cancellation_type_t::none) {
162+
// We can only honor cancellations if the request hasn't been sent to the server
163+
if (!info->request_sent()) {
164+
conn->state.remove_request(info);
165+
return {exec_action_type::done, asio::error::operation_aborted};
166+
} else {
167+
// Can't cancel, keep waiting
168+
}
169+
}
170+
}
171+
}
172+
}
173+
};
174+
```
175+
176+
`exec_op` no longer contains any logic:
177+
178+
```cpp
179+
struct exec_op {
180+
connection* conn;
181+
exec_fsm fsm;
182+
183+
// Will be called by Asio until self.complete() is called
184+
template <class Self>
185+
void operator()(Self& self, system::error_code = {}, std::size_t = 0)
186+
{
187+
// Call the FSM
188+
auto action = fsm.resume(conn->state, self.get_cancellation_state().cancelled());
189+
190+
// Apply the required action
191+
switch (action.type)
192+
{
193+
case exec_action_type::notify_writer:
194+
conn->writer_timer.cancel();
195+
(*this)(self); // This action doesn't involve a callback, so invoke ourselves again
196+
break;
197+
case exec_action_type::wait:
198+
fsm.get_info()->channel.async_wait(std::move(self));
199+
break;
200+
case exec_action_type::immediate:
201+
asio::async_immediate(self.get_io_executor(), std::move(self));
202+
break;
203+
case exec_action_type::done:
204+
self.complete(action.ec, action.bytes_read);
205+
break;
206+
}
207+
}
208+
};
209+
```
210+
211+
With this setup, `exec_fsm` is now trivial to test, since it doesn't invoke any I/O.
212+
213+
We're migrating most of the algorithms towards this approach, and we're
214+
finding and fixing many subtle problems in the process. There is still lots to do,
215+
but efforts are already paying off.
216+
217+
## Boost.Redis features and docs
218+
219+
While I tend to get very excited about this new sans-io approach, I've also
220+
made other contributions that had likely had more impact on users. For instance,
221+
I've implemented support for UNIX sockets, which had been something recurrently
222+
requested by users that want to squeeze the last bit of performance from their setup.
223+
224+
I've also worked on logging. Since the reconnection algorithm is complex, Boost.Redis
225+
logs some messages by default to simplify diagnostics. This is now performed through
226+
a simple, extensible and well-documented API, allowing users to integrate third-party
227+
logging libraries like `spdlog`.
228+
229+
Last (but not least), I've migrated Boost.Redis docs to the new asciidoc/antora/mrdocs
230+
toolchain. I'm pretty impressed with [the results](https://www.boost.org/doc/libs/develop/libs/redis/doc/html/redis/index.html),
231+
and would like to thank the MrDocs and Boostlook people for their efforts.
232+
233+
I know that comparisons are odious, but...
234+
235+
* [Old docs for `basic_connection`](https://www.boost.org/doc/libs/1_88_0/libs/redis/doc/html/classboost_1_1redis_1_1basic__connection.html).
236+
* [New docs for `basic_connection`](https://www.boost.org/doc/libs/develop/libs/redis/doc/html/redis/reference/boost/redis/basic_connection.html).
237+
238+
## A great MySQL comes with a great responsibility
239+
240+
It's not all been Redis these days. Boost.MySQL users also deserve some attention.
241+
I've rewritten the MySQL handshake algorithm, so the `caching_sha2_password` plugin
242+
can work without TLS.
243+
244+
For most of you, the sentence above will likely say nothing, so let's provide some context.
245+
When a client connects to the MySQL server, it performs a connection establishment packet
246+
exchange where several connection parameters are negotiated, the client is authenticated,
247+
and the TLS layer is optionally installed. This is collectively called the MySQL handshake,
248+
and it's not very clean in design.
249+
250+
Clients can authenticate using several authentication mechanisms, called authentication
251+
plugins. The most widespread one is `mysql_native_password`, a challenge/response mechanism
252+
where the client sends a hashed password to the server. It doesn't require a TLS layer,
253+
and it's supported by MySQL 5.x, MySQL 8.x and MariaDB.
254+
255+
The problem with `mysql_native_password` is that it uses `SHA1`, which is considered nowadays weak.
256+
MySQL 8.x introduced `caching_sha2_password` and deprecated `mysql_native_password`, and MySQL 9.x
257+
has removed the latter. `caching_sha2_password` uses `SHA256`, but it also introduces a cache in the server.
258+
If your user is in the cache, the password is sent hashed, as with `mysql_native_password`.
259+
But if it's not, things get more complex:
260+
261+
* When using a TLS layer, the password is sent in plain text.
262+
* When not using a TLS layer, the server supplies an RSA key, and the password is sent encrypted using it.
263+
264+
I never got to implement the second point, since most people were just using
265+
the simpler `mysql_native_password`. With the advent of MySQL 9, this was becoming a problem,
266+
since it meant using TLS even for local network connections (like the ones between Docker containers),
267+
with the overhead it implies.
268+
269+
Implementing this new exchange has required a big refactor and many tests,
270+
but it has paid off, as it unravelled some buggy edge-cases.
271+
Remember the `async_compose` vs sans-io discussion at the beginning of this post?
272+
For such complex exchanges, going sans-io has been key.
273+
274+
## Three databases are better than two
275+
276+
Why MySQL and not Postgres? Well, I found myself asking this question, too.
277+
Following what I've learnt with Boost.MySQL, I'm writing a new library
278+
to interact with Postgres. It's not usable yet, but it's made some
279+
progress. You can check it out [here](https://github.com/anarthal/nativepg).
280+
281+
## New Boost citizens: OpenMethod and Bloom
282+
283+
I've also had the pleasure to participate in the review of two wonderful
284+
libraries that have been accepted into Boost: [OpenMethod](https://github.com/jll63/Boost.OpenMethod),
285+
which allows defining virtual functions outside classes; and [Bloom](https://github.com/boostorg/bloom),
286+
which implements Bloom filters. The family keeps growing.
287+
288+
## C++20 modules and Boost
289+
290+
I'm happy to see some more Boost authors adding support for C++20 modules
291+
in their libraries. Concretely, I've reviewed PRs for Boost.Pfr and Boost.Any.
292+
293+
This is really exciting for me, and I hope to be able to dedicate some time soon
294+
to progress my C++20 prototype for Boost.Core and Boost.Mp11.
295+
296+
## Lightweight test context
297+
298+
Boost.Core contains a small component to write unit tests: the [lightweight test](https://live.boost.org/doc/libs/master/libs/core/doc/html/core/lightweight_test.html)
299+
framework. It's extremely simple, and that makes it fast, both at runtime and compile-time.
300+
301+
It's sometimes too simple. I'm a big fan of parametric tests, where you run a test case
302+
over a set of different values. You can do so with lightweight test by just using a loop,
303+
but that makes failures difficult to diagnose.
304+
305+
I'm implementing an equivalent to [`BOOST_TEST_CONTEXT`](https://live.boost.org/doc/libs/1_88_0/libs/test/doc/html/boost_test/test_output/test_tools_support_for_logging/contexts.html)
306+
for lightweight test. I'm still on the middle of it, so I'll dive deeper onto this
307+
in my next post. All I can say is that this addition makes lightweight test
308+
a perfect fit for most of my testing needs!
309+
310+
## Next steps
311+
312+
It looks like databases, Asio and modules are definitely part of my future.
313+
So many exciting things that I sometimes struggle to decide on which one to focus!

0 commit comments

Comments
 (0)