Add boosting support to linfa-ensemble

[yfnaji]

I was looking into the linfa-ensemble crate and noticed that it currently does not include a boosting mechanism. The current ensemble implementation appears to be quite generalised, so we could either extend EnsembleLearner to support boosting methods or (preferably) create a new struct dedicated to boosting, while keeping EnsembleLearner as the dedicated bagging struct (and rename EnsembleLearner -> Bagging).

I would like your thoughts on this - I’d be happy to work on this and write an initial implementation.

[relf]

In my mind EnsembleLearner is indeed the most generalized and more specialized algorithms composes with it.
You might be interested in discussing the implementation in #410 to target a smooth integration of your boosting method implementation.

[yfnaji]

A gradient boosting machine (using decision trees) effectively requires regression trees since the algorithm operates on continuous residuals rather than discrete class labels. It appears that this package does not yet support regression trees as mentioned in the documentation and this issue.

I can work on adding regression tree support via a separate PR. Once that’s done, we can proceed with implementing gradient boosting. To keep things manageable, I will focus on implementing gradient boosting specifically for decision trees first, and then generalise to other base learners later.

[rathideep22]

can you please close this issue the pr is merged

[yfnaji]

Hi @rathideep22 - your PR addresses Adaboost, while this issue is focused on gradient boosting which I am still currently implementing.