Improve Avro implementation #332
Description
The current Avro writer and reader implementations are not that efficient because we need to use avro::GenericDatum as the intermediate layer:
-
Avro writer:
ArrowArray->avro::GenericDatum->avro::Encoder -
Avro reader:
avro::Decoder->avro::GenericDatum->ArrowArray(feat: eliminate GenericDatum in Avro reader for performance #374)
It would be more efficient if we can get rid of conversion from/to avro::GenericDatum by directly using the encoder/decoder in the Avro writer/reader. It can greatly improve the performance of manifest file I/O.