[tribuo-devel] [EXT]:Re: [External] : Re: Tribuo Non Numerical Features Support

Fri Jun 16 17:48:06 UTC 2023

Hi Sandeep,

Firstly, we always welcome contributions. In this case the difficulty depends on if you only want inference support or if you’d also like to train models.

To add pure inferencing support (similar to our ONNX, OCI and XGBoost external model support) the Tribuo bits would be relatively simple as you don’t need to interact with the provenance system much. You’d subclass org.tribuo.interop.ExternalModel and implement the necessary methods to get things out of the Tribuo representation and into the appropriate LightGBM one. It would be pretty similar to the existing org.tribuo.common.xgboost.XGBoostExternalModel. However it looks like the LightGBM Java interface is autogenerated and very low level (https://central.sonatype.com/artifact/com.microsoft.ml.lightgbm/lightgbmlib/3.3.510), so it would need a fair amount of work to get that C-like API into something that’s easy to use from Java. The SWIG wrapper is basically passing around raw pointers so it needs actual objects and methods built on top of that. Microsoft built a lot of that for their SynapseML API to LightGBM (https://github.com/microsoft/SynapseML/tree/master/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm), but that is for Apache Spark integration and I don’t think it would be reusable outside of Spark & Scala (and we don’t want Scala in Tribuo’s builds it adds too much complexity). Alternatively you could look at Amazon’s DJL which has a LightGBM interface (https://github.com/deepjavalibrary/djl/tree/master/engines/ml/lightgbm), but as that’s focused on deep learning I don’t know if it would give you the necessary control over the incoming feature representation. DJL’s wrapper is less than 1000 lines though, so it probably isn’t too hard to make an equivalent for Tribuo.

Basically on the Tribuo side you’d need to write LightGBMExternalModel (e.g. https://github.com/oracle/tribuo/blob/main/Common/XGBoost/src/main/java/org/tribuo/common/xgboost/XGBoostExternalModel.java), a short protobuf definition for serialization (e.g. https://github.com/oracle/tribuo/blob/main/Common/XGBoost/src/main/resources/protos/tribuo-xgboost.proto#L54), then some kind of output converter mechanism if you wanted it to work for both classification & regression (https://github.com/oracle/tribuo/blob/main/Common/XGBoost/src/main/java/org/tribuo/common/xgboost/XGBoostOutputConverter.java), or just make it only do regression. Then you’d need to map from Tribuo’s internal representation of a sparse vector/matrix of feature values to the internal representation LightGBM wants (which shouldn’t be too bad as it supports CSR and Tribuo’s representation is basically that with more objects, e.g. https://github.com/oracle/tribuo/blob/main/Common/XGBoost/src/main/java/org/tribuo/common/xgboost/XGBoostTrainer.java#L711), write a mapping from LightGBM’s output to Tribuo’s output (which if it’s an array is easy, otherwise will require some unpicking), and then the glue to stick that to the SWIG wrapper Microsoft provide. That glue is the worst part as the SWIG isn’t very well documented, but it would be pretty similar to the glue DJL have here - https://github.com/deepjavalibrary/djl/blob/master/engines/ml/lightgbm/src/main/java/ai/djl/ml/lightgbm/jni/JniUtils.java.

Training support would add more complexity on the Tribuo side, though it would be pretty similar to the rest of our XGBoost wrapper so you could base it off that. The main difficulty would still be in getting the LightGBM wrapper up from JNI as now you’ve got to pass parameters and configuration across as well, so it’ll be quite a bit more effort.

Thanks,

Adam
--
Adam Pocock
Principal Member of Technical Staff
Machine Learning Research Group
Oracle Labs, Burlington, MA

On 16 Jun 2023, at 12:23, Sikka, Sandeep <sandeep.sikka at staples.com> wrote:

Thanks Adam for the feedback. In your opinion, how difficult would it be for one or more experienced Java Engineers to implement support for wrapping the LightGBM Java API in Tribuo? Assume no prior experience working in the Tribuo codebase. I am wondering if we take it on for our needs and contribute the developments back to Tribuo, how much time/effort will that take.

Sandeep.
From: Adam Pocock <adam.pocock at oracle.com>
Date: Friday, June 16, 2023 at 11:25 AM
To: "Sikka, Sandeep" <sandeep.sikka at staples.com>
Cc: "Parthasarathy, Bharadwaj" <bharadwaj.parthasarathy at staples.com>, "tribuo-devel at oss.oracle.com" <tribuo-devel at oss.oracle.com>, "Kumar, Navdeep" <navdeep.kumar at staples.com>, Jack Sullivan <jack.t.sullivan at oracle.com>
Subject: Re: [EXT]:Re: [External] : Re: Tribuo Non Numerical Features Support

Hi Sandeep,

Tribuo doesn’t support inference for LightGBM models, aside from those which have been exported in ONNX format. You’d have to look at how the ONNX format model expects the categorical inputs to be passed in, I’m not sure if ONNX’s TreeEnsembleRegressor (https://github.com/onnx/onnx/blob/main/docs/Operators-ml.md#aionnxmltreeensembleregressor<https://urldefense.com/v3/__https://github.com/onnx/onnx/blob/main/docs/Operators-ml.md*aionnxmltreeensembleregressor__;Iw!!ACWV5N9M2RV99hQ!NqMIzH8F7USIbjeG7iyv2eQ2TKagOulH5hFoK7SAtnilTSkPEh4ZzfTlmypE7r5ljs9cF2l_u7aMjD0knPgubeY58A$>) supports special treatment of categorical inputs so it might not be possible. It looks like there is support for LightGBM in things like JPMML, but JPMML is AGPL licensed and so we can’t use it in Tribuo. We haven’t looked at directly wrapping the LightGBM Java API, as we already had XGBoost support for tree ensembles.

Thanks,

Adam
--
Adam Pocock
Principal Member of Technical Staff
Machine Learning Research Group
Oracle Labs, Burlington, MA

On 16 Jun 2023, at 11:08, Sikka, Sandeep <sandeep.sikka at staples.com> wrote:

Hi Adam
Thank you for the quick response. We are looking to score (perform inference) on a LGBMRegressor<https://urldefense.com/v3/__https:/lightgbm.readthedocs.io/en/latest/pythonapi/lightgbm.LGBMRegressor.html__;!!ACWV5N9M2RV99hQ!MaIMJSbVSuFi4H4SnXBIfhxdZCWgg2GBNhEefwQDhe4vef4HJeHRLEbwfLxphLQ8iFNWOqXkTwp7VTblc7Jy9qPBBw$> trained externally with the categorical features applied using the categorical_features parameter: https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html#categorical-feature-support<https://urldefense.com/v3/__https:/lightgbm.readthedocs.io/en/latest/Advanced-Topics.html*categorical-feature-support__;Iw!!ACWV5N9M2RV99hQ!MaIMJSbVSuFi4H4SnXBIfhxdZCWgg2GBNhEefwQDhe4vef4HJeHRLEbwfLxphLQ8iFNWOqXkTwp7VTblc7LbplBZGQ$>. This uses a technique different from OHE: https://lightgbm.readthedocs.io/en/latest/Features.html#optimal-split-for-categorical-features<https://urldefense.com/v3/__https:/lightgbm.readthedocs.io/en/latest/Features.html*optimal-split-for-categorical-features__;Iw!!ACWV5N9M2RV99hQ!MaIMJSbVSuFi4H4SnXBIfhxdZCWgg2GBNhEefwQDhe4vef4HJeHRLEbwfLxphLQ8iFNWOqXkTwp7VTblc7L6i0QwjQ$>. Do you have any thoughts on how we can achieve inference in Tribuo (we train the model in Python scikit-learn pipelines)?

Thanks
Sandeep.
From: Adam Pocock <adam.pocock at oracle.com<mailto:adam.pocock at oracle.com>>
Date: Friday, June 16, 2023 at 9:34 AM
To: "Sikka, Sandeep" <sandeep.sikka at staples.com<mailto:sandeep.sikka at staples.com>>, "Parthasarathy, Bharadwaj" <bharadwaj.parthasarathy at staples.com<mailto:bharadwaj.parthasarathy at staples.com>>
Cc: "tribuo-devel at oss.oracle.com<mailto:tribuo-devel at oss.oracle.com>" <tribuo-devel at oss.oracle.com<mailto:tribuo-devel at oss.oracle.com>>, "Kumar, Navdeep" <navdeep.kumar at staples.com<mailto:navdeep.kumar at staples.com>>, Jack Sullivan <jack.t.sullivan at oracle.com<mailto:jack.t.sullivan at oracle.com>>
Subject: [EXT]:Re: [External] : Re: Tribuo Non Numerical Features Support

Hi Sandeep, Bharad,

The Tribuo mailing list is monitored, but I can’t see the email you sent to the list in the moderator console nor the archives, did you get a bounceback from the mailserver? It may be configured only to accept emails from people who have joined the mailing list. Anyway, the Tribuo development team have their email addresses listed in the pom file visible on Github.

As to the question of feature support, in Tribuo all features are converted to numerical values during the data loading step. Categoricals can be encoded as one hot features or directly as doubles by choosing an index, and ordinals can be encoded as doubles. Very few ML algorithms directly support categorical variables without one hot-encoding or vectorization, I think of the popular ones the only kind that do are trees. Tribuo’s tree package doesn’t currently have specializations for categoricals though there is enough information in the feature domain to support that if we do add the algorithmic support. In tree algorithms ordinals are naturally supported as the splits don’t care about the distance between any feature values, only their relative ordering (or at least all the splitting algorithms we have in Tribuo are of that kind). You can see how to perform featurization into doubles in a number of ways in the Columnar Data (https://tribuo.org/learn/4.3/tutorials/columnar-tribuo-v4.html<https://urldefense.com/v3/__https:/tribuo.org/learn/4.3/tutorials/columnar-tribuo-v4.html__;!!ACWV5N9M2RV99hQ!MaIMJSbVSuFi4H4SnXBIfhxdZCWgg2GBNhEefwQDhe4vef4HJeHRLEbwfLxphLQ8iFNWOqXkTwp7VTblc7J1rg3Sxg$>) and Document Classification (https://tribuo.org/learn/4.3/tutorials/document-classification-tribuo-v4.html<https://urldefense.com/v3/__https:/tribuo.org/learn/4.3/tutorials/document-classification-tribuo-v4.html__;!!ACWV5N9M2RV99hQ!MaIMJSbVSuFi4H4SnXBIfhxdZCWgg2GBNhEefwQDhe4vef4HJeHRLEbwfLxphLQ8iFNWOqXkTwp7VTblc7LtOLJzVw$>) tutorials.

Thanks,

Adam
--
Adam Pocock
Principal Member of Technical Staff
Machine Learning Research Group
Oracle Labs, Burlington, MA

On 16 Jun 2023, at 00:13, Sikka, Sandeep <sandeep.sikka at staples.com<mailto:sandeep.sikka at staples.com>> wrote:

Adding a few individual emails on the project from Github. The devel email list doesn’t appear to have any activity.

Thanks
Sandeep Sikka
Distinguished Engineer
500 Staples Dr., Framingham MA
508.253.7977
sandeep.sikka at staples.com<mailto:sandeep.sikka at staples.com>

<image001.png>

<image002.png><image003.png> <image004.png> <image005.png>

From: "Parthasarathy, Bharadwaj" <bharadwaj.parthasarathy at staples.com<mailto:bharadwaj.parthasarathy at staples.com>>
Date: Thursday, June 15, 2023 at 2:46 PM
To: "tribuo-devel at oss.oracle.com<mailto:tribuo-devel at oss.oracle.com>" <tribuo-devel at oss.oracle.com<mailto:tribuo-devel at oss.oracle.com>>
Cc: "Sikka, Sandeep" <Sandeep.Sikka at Staples.com<mailto:Sandeep.Sikka at Staples.com>>, "Kumar, Navdeep" <navdeep.kumar at staples.com<mailto:navdeep.kumar at staples.com>>
Subject: Tribuo Non Numerical Features Support

Hi,

This is Bharadwaj from one of Staples' Engineering teams. We are exploring Tribuo java framework for one of our use cases. We are trying to use Tribuo for runtime inferencing with a regressor model.

We have non numerical features that we are trying to integrate with Tribuo. However, all the classes or samples for regressor points us to passing feature values as only double. Feature names as String and Feature Values as Double is what we see.

Does Tribuo support non numerical features (categorical and ordinal features) such as String, Boolean etc or is it strictly only double values? How can we train and infer supervised models that consume categorical and ordinal features?

I am not sure if we are missing something here, it would be helpful if you can provide your inputs.

Regards,
Bharad

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://oss.oracle.com/pipermail/tribuo-devel/attachments/20230616/66e22530/attachment-0001.html>