Skip to content

PS-10246 feature: Implement fixing next_event_position fields / checksums in the event common_header / footer in case of reconnecting to another node (part 4)#100

Merged
percona-ysorokin merged 1 commit intomainfrom
binlog_rewrite
Mar 10, 2026
Merged

PS-10246 feature: Implement fixing next_event_position fields / checksums in the event common_header / footer in case of reconnecting to another node (part 4)#100
percona-ysorokin merged 1 commit intomainfrom
binlog_rewrite

Conversation

@percona-ysorokin
Copy link
Collaborator

https://perconadev.atlassian.net/browse/PS-10246

Implemented binlog event data rewrite in the 'gtid' replication mode. When user continues streaming from a different MySQL server, there is no guarantee that binlog files on the new server are physically identical to those on the old one and therefore may have different values of 'next_event_position' in the common header and 'crc' in the footer. That is why we had to implement the logic of updating certain fields in the serialized event data blocks before writing them to our own binlog files. Moreover, the old and the new servers may have different names for their binlog files and that is why in this mode we simply cannot react to file names extracted from the ROTATE events. As the result, we implemented our own binlog file name generation and event data splitting.

We can now specify a new section '<replication.rewrite>' in the main configuration file to enable this rewrite mode. If it is not present, then this mode is turned off. '<replication.rewrite>' has 2 parameters:

  • '<replication.rewrite.base_file_name>' - base binlog file name (e.g. 'binlog' if we want to get 'binlog.000001')
  • '<replication.rewrite.file_size>' - the file size after reaching which we will perform binlog file rotation on the receiving side.

Added new class 'binsrv::rewrite_config' that represents this '<replication.rewrite>' section in the configuration file.

Significantly refactored 'binsrv::events::reader_context' class.

  • It now operates on instances of 'binsrv::events::event_view' instead of 'binsrv::events::event'.
  • It now materialized required post-headers / bodies only when necessary.
  • It now has a helper method method for generating event post-header length array in the FDE post-header by providing hardcoded data extracted from 8.0 and 8.4 MySQL Servers ('get_hardcoded_post_header_lengths()'). The old existing 'get_post_header_lengths()' method renamed into 'get_known_post_header_lengths()'.
  • Reworked the way how we identify "info-only" events (those that should not be written to the binary log) - instead of keeping this value for the last processed event in the context, we now return it immediately in the 'process_event_view()' method.

Added new class 'binsrv::composite_binlog_name' that helps with parsing / generating of the binlog file names in the 'binlog.000001' format. 'binsrv::storage' reworked to store binlog names as instances of this class instead of strings.

Added new 'binsrv::events::materialize()' free function that is supposed to copy data from the binlog events represented via 'binsrv::events::event_view' into new buffers performing some basic adjustments (like adding / removing footers with checksums).

Changed the way how binlog events interacts with 'binsrv::events::reader_context' - instead of automatically invoking 'process_event()' method in the 'binsrv::events::event' constructor, we now do this manually via 'process_event_view()' method.

In the 'set_up_binsrv_environment.inc' MTR include file it is now possible to specify '$binsrv_rewrite_file_size' parameter before including that will be translated into '<replication.rewrite.file_size>' configuration parameter.

'binlog_streaming.resume_streaming' MTR test case extended with two additional combinations: 'gtid_rewrite_buffered' and 'gtid_rewrite_unbuffered' that check streaming resume operations in the gtid mode with rewrite enabled.

Updated README.md documentation file : we now have new section describing 'rewrite' configuration parameters - '<replication.rewrite>' section.

'event_test' unit test (BOOST_TEST_MODULE EventTests) extended with additional test case 'EventMaterialization' that checks various combinations of the 'event_view' materializations (from a view with / without a footer to a new data block with / without a footer).

…sums in the event common_header / footer in case of reconnecting to another node (part 4)

https://perconadev.atlassian.net/browse/PS-10246

Implemented binlog event data rewrite in the 'gtid' replication mode. When user continues
streaming from a different MySQL server, there is no guarantee that binlog files on the new
server are physically identical to those on the old one and therefore may have different values
of 'next_event_position' in the common header and 'crc' in the footer. That is why we had to
implement the logic of updating certain fields in the serialized event data blocks before
writing them to our own binlog files. Moreover, the old and the new servers may have
different names for their binlog files and that is why in this mode we simply cannot react to
file names extracted from the ROTATE events. As the result, we implemented our own binlog
file name generation and event data splitting.

We can now specify a new section '<replication.rewrite>' in the main configuration file
to enable this rewrite mode. If it is not present, then this mode is turned off.
'<replication.rewrite>' has 2 parameters:
- '<replication.rewrite.base_file_name>' - base binlog file name (e.g. 'binlog' if we want to
  get 'binlog.000001')
- '<replication.rewrite.file_size>' - the file size after reaching which we will perform binlog
  file rotation on the receiving side.

Added new class 'binsrv::rewrite_config' that represents this '<replication.rewrite>' section in
the configuration file.

Significantly refactored 'binsrv::events::reader_context' class.
- It now operates on instances of 'binsrv::events::event_view' instead of 'binsrv::events::event'.
- It now materialized required post-headers / bodies only when necessary.
- It now has a helper method method for generating event post-header length array in the
  FDE post-header by providing hardcoded data extracted from 8.0 and 8.4 MySQL Servers
  ('get_hardcoded_post_header_lengths()'). The old existing 'get_post_header_lengths()'
  method renamed into 'get_known_post_header_lengths()'.
- Reworked the way how we identify "info-only" events (those that should not be written to
  the binary log) - instead of keeping this value for the last processed event in the context,
  we now return it immediately in the 'process_event_view()' method.

Added new class 'binsrv::composite_binlog_name' that helps with parsing / generating of
the binlog file names in the 'binlog.000001' format. 'binsrv::storage' reworked to store binlog
names as instances of this class instead of strings.

Added new 'binsrv::events::materialize()' free function that is supposed to copy data from
the binlog events represented via 'binsrv::events::event_view' into new buffers performing
some basic adjustments (like adding / removing footers with checksums).

Changed the way how binlog events interacts with 'binsrv::events::reader_context' - instead of
automatically invoking 'process_event()' method in the 'binsrv::events::event' constructor, we
now do this manually via 'process_event_view()' method.

In the 'set_up_binsrv_environment.inc' MTR include file it is now possible to specify
'$binsrv_rewrite_file_size' parameter before including that will be translated into
'<replication.rewrite.file_size>' configuration parameter.

'binlog_streaming.resume_streaming' MTR test case extended with two additional
combinations: 'gtid_rewrite_buffered' and 'gtid_rewrite_unbuffered' that check streaming
resume operations in the gtid mode with rewrite enabled.

Updated README.md documentation file : we now have new section describing 'rewrite'
configuration parameters - '<replication.rewrite>' section.

'event_test' unit test (BOOST_TEST_MODULE EventTests) extended with additional test case
'EventMaterialization' that checks various combinations of the 'event_view' materializations
(from a view with / without a footer to a new data block with / without a footer).
@percona-ysorokin percona-ysorokin merged commit a14c99e into main Mar 10, 2026
7 checks passed
@percona-ysorokin percona-ysorokin deleted the binlog_rewrite branch March 10, 2026 11:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant