Skip to content

Conversation

@gkousouris
Copy link

@gkousouris gkousouris commented Aug 18, 2023

Problem

_unpack_message_set was modified in #72 to skip over transactional control batches.

Because of this, unpacked = _unpack_message_set may return an empty list.
However, later in the code we reference the last element of that list, unpacked[-1], to find how many messages have been consumed. This causes a list index out of range exception

Solution

To solve this, we do not want to reference unpacked[-1], but skip the message.

Testing

I have manually edited the file to this from within the venv and this error was fixed:

{...}/venv/lib/python3.10/site-packages/kafka/consumer/fetcher.py", line 830, in _parse_fetched_data
    last_offset = unpacked[-1].offset
IndexError: list index out of range
exception 'IndexError('list index out of range')' encountered running paastorm.{..}`

@gkousouris gkousouris force-pushed the u/georgios/KAFKA-36342_fix_control_batches_bug branch from ae6cace to c4ee776 Compare August 18, 2023 15:17
@gkousouris gkousouris requested a review from JayH5 August 18, 2023 15:18
@JayH5
Copy link

JayH5 commented Aug 21, 2023

LGTM. Do you want to bump the version in this PR?

@gkousouris
Copy link
Author

LGTM. Do you want to bump the version in this PR?

ah, might as well bump it now

@gkousouris gkousouris merged commit 5e508ed into master Aug 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants