Bug in Async flush when sending >20,000 events with batch size of 2,000 #4
Description
There is a bug in the async flush in an asp.net core web api, where only 2,000-3,000 events get sent to the Event Hub when logging >10,000 events with batch size of 2,000.
My local solution was to change the bold lines below. Now after the web api Controller returns it's instant response, the async log events write to the Event Hub in a background process on the web server until they are complete or the edge case where the web api process is intentionally terminated.
private async Task ReadChannelAsync()
{
**var tasks = new List<Task>();**
while (await this.channel.Reader.WaitToReadAsync())
{
// No client is available, so briefly pause before retrying.
if (this.eventHubClient is null)
{
await Task.Delay(TimeSpan.FromSeconds(1)).ConfigureAwait(false);
continue;
}
var eventDataBatch = this.eventHubClient.CreateBatch();
while (this.channel.Reader.TryRead(out var eventData))
{
// Attempt to add the current event data to existing batch.
if (eventDataBatch.TryAdd(eventData))
{
// There was space available, try to read more event data.
continue;
}
// There was not enough space available, so send the current batch and create a
// new one.
**tasks.Add(TrySendAsync(eventDataBatch));**
eventDataBatch = this.eventHubClient.CreateBatch();
// Attempt to add the current event data to new batch.
eventDataBatch.TryAdd(eventData);
}
// No more event data is currently available, so send the current batch.
**tasks.Add(TrySendAsync(eventDataBatch));**
}
**await Task.WhenAll(tasks.ToArray());**
}