PS-10246 feature: Implement fixing next_event_position fields / checksums in the event common_header / footer in case of reconnecting to another node (part 4)#100
Merged
percona-ysorokin merged 1 commit intomainfrom Mar 10, 2026
Conversation
…sums in the event common_header / footer in case of reconnecting to another node (part 4) https://perconadev.atlassian.net/browse/PS-10246 Implemented binlog event data rewrite in the 'gtid' replication mode. When user continues streaming from a different MySQL server, there is no guarantee that binlog files on the new server are physically identical to those on the old one and therefore may have different values of 'next_event_position' in the common header and 'crc' in the footer. That is why we had to implement the logic of updating certain fields in the serialized event data blocks before writing them to our own binlog files. Moreover, the old and the new servers may have different names for their binlog files and that is why in this mode we simply cannot react to file names extracted from the ROTATE events. As the result, we implemented our own binlog file name generation and event data splitting. We can now specify a new section '<replication.rewrite>' in the main configuration file to enable this rewrite mode. If it is not present, then this mode is turned off. '<replication.rewrite>' has 2 parameters: - '<replication.rewrite.base_file_name>' - base binlog file name (e.g. 'binlog' if we want to get 'binlog.000001') - '<replication.rewrite.file_size>' - the file size after reaching which we will perform binlog file rotation on the receiving side. Added new class 'binsrv::rewrite_config' that represents this '<replication.rewrite>' section in the configuration file. Significantly refactored 'binsrv::events::reader_context' class. - It now operates on instances of 'binsrv::events::event_view' instead of 'binsrv::events::event'. - It now materialized required post-headers / bodies only when necessary. - It now has a helper method method for generating event post-header length array in the FDE post-header by providing hardcoded data extracted from 8.0 and 8.4 MySQL Servers ('get_hardcoded_post_header_lengths()'). The old existing 'get_post_header_lengths()' method renamed into 'get_known_post_header_lengths()'. - Reworked the way how we identify "info-only" events (those that should not be written to the binary log) - instead of keeping this value for the last processed event in the context, we now return it immediately in the 'process_event_view()' method. Added new class 'binsrv::composite_binlog_name' that helps with parsing / generating of the binlog file names in the 'binlog.000001' format. 'binsrv::storage' reworked to store binlog names as instances of this class instead of strings. Added new 'binsrv::events::materialize()' free function that is supposed to copy data from the binlog events represented via 'binsrv::events::event_view' into new buffers performing some basic adjustments (like adding / removing footers with checksums). Changed the way how binlog events interacts with 'binsrv::events::reader_context' - instead of automatically invoking 'process_event()' method in the 'binsrv::events::event' constructor, we now do this manually via 'process_event_view()' method. In the 'set_up_binsrv_environment.inc' MTR include file it is now possible to specify '$binsrv_rewrite_file_size' parameter before including that will be translated into '<replication.rewrite.file_size>' configuration parameter. 'binlog_streaming.resume_streaming' MTR test case extended with two additional combinations: 'gtid_rewrite_buffered' and 'gtid_rewrite_unbuffered' that check streaming resume operations in the gtid mode with rewrite enabled. Updated README.md documentation file : we now have new section describing 'rewrite' configuration parameters - '<replication.rewrite>' section. 'event_test' unit test (BOOST_TEST_MODULE EventTests) extended with additional test case 'EventMaterialization' that checks various combinations of the 'event_view' materializations (from a view with / without a footer to a new data block with / without a footer).
ef0a2e4 to
3983340
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://perconadev.atlassian.net/browse/PS-10246
Implemented binlog event data rewrite in the 'gtid' replication mode. When user continues streaming from a different MySQL server, there is no guarantee that binlog files on the new server are physically identical to those on the old one and therefore may have different values of 'next_event_position' in the common header and 'crc' in the footer. That is why we had to implement the logic of updating certain fields in the serialized event data blocks before writing them to our own binlog files. Moreover, the old and the new servers may have different names for their binlog files and that is why in this mode we simply cannot react to file names extracted from the ROTATE events. As the result, we implemented our own binlog file name generation and event data splitting.
We can now specify a new section '<replication.rewrite>' in the main configuration file to enable this rewrite mode. If it is not present, then this mode is turned off. '<replication.rewrite>' has 2 parameters:
Added new class 'binsrv::rewrite_config' that represents this '<replication.rewrite>' section in the configuration file.
Significantly refactored 'binsrv::events::reader_context' class.
Added new class 'binsrv::composite_binlog_name' that helps with parsing / generating of the binlog file names in the 'binlog.000001' format. 'binsrv::storage' reworked to store binlog names as instances of this class instead of strings.
Added new 'binsrv::events::materialize()' free function that is supposed to copy data from the binlog events represented via 'binsrv::events::event_view' into new buffers performing some basic adjustments (like adding / removing footers with checksums).
Changed the way how binlog events interacts with 'binsrv::events::reader_context' - instead of automatically invoking 'process_event()' method in the 'binsrv::events::event' constructor, we now do this manually via 'process_event_view()' method.
In the 'set_up_binsrv_environment.inc' MTR include file it is now possible to specify '$binsrv_rewrite_file_size' parameter before including that will be translated into '<replication.rewrite.file_size>' configuration parameter.
'binlog_streaming.resume_streaming' MTR test case extended with two additional combinations: 'gtid_rewrite_buffered' and 'gtid_rewrite_unbuffered' that check streaming resume operations in the gtid mode with rewrite enabled.
Updated README.md documentation file : we now have new section describing 'rewrite' configuration parameters - '<replication.rewrite>' section.
'event_test' unit test (BOOST_TEST_MODULE EventTests) extended with additional test case 'EventMaterialization' that checks various combinations of the 'event_view' materializations (from a view with / without a footer to a new data block with / without a footer).