From adam.pocock at oracle.com Thu Aug 10 19:57:19 2023 From: adam.pocock at oracle.com (Adam Pocock) Date: Thu, 10 Aug 2023 19:57:19 +0000 Subject: [tribuo-devel] [EXT]:Re: [External] : Re: Tribuo Non Numerical Features Support In-Reply-To: <98629166-284A-4C84-A1D1-8D5AD4B37625@staples.com> References:

<9004C1F2-7D35-498A-AE7A-93C81A12D9DC@oracle.com>

<68C369ED-D0A0-4D84-9C89-1F543853F1CB@staples.com>

<98629166-284A-4C84-A1D1-8D5AD4B37625@staples.com> Message-ID: Hi Sandeep, Ok, I understand how that goes. Thanks, Adam ________________________________ From: Sikka, Sandeep Sent: 09 August 2023 22:49 To: Adam Pocock Cc: Parthasarathy, Bharadwaj ; tribuo-devel at oss.oracle.com ; Kumar, Navdeep ; Jack Sullivan Subject: Re: [EXT]:Re: [External] : Re: Tribuo Non Numerical Features Support Hi Adam I realized I never replied to you. The business committee decided to not pursue the project that necessitated the conversation below. We have deferred the work. If anything changes or a new need comes up, I will reach out. Thanks for your help. Sandeep. Sandeep Sikka Distinguished Engineer 500 Staples Dr., Framingham MA 508.253.7977 sandeep.sikka at staples.com [signature_1764920163] [signature_1743869139][signature_1068797335] [signature_107582287] [signature_66462932] From: "Sikka, Sandeep" Date: Tuesday, June 20, 2023 at 3:00 PM To: Adam Pocock Cc: "Parthasarathy, Bharadwaj" , "tribuo-devel at oss.oracle.com" , "Kumar, Navdeep" , Jack Sullivan Subject: Re: [EXT]:Re: [External] : Re: Tribuo Non Numerical Features Support Thank you, Adam, for the detailed feedback. Let me have some discussions internally and I will get back to you. Sandeep. From: Adam Pocock Date: Friday, June 16, 2023 at 1:48 PM To: "Sikka, Sandeep" Cc: "Parthasarathy, Bharadwaj" , "tribuo-devel at oss.oracle.com" , "Kumar, Navdeep" , Jack Sullivan Subject: Re: [EXT]:Re: [External] : Re: Tribuo Non Numerical Features Support Hi Sandeep, Firstly, we always welcome contributions. In this case the difficulty depends on if you only want inference support or if you?d also like to train models. To add pure inferencing support (similar to our ONNX, OCI and XGBoost external model support) the Tribuo bits would be relatively simple as you don?t need to interact with the provenance system much. You?d subclass org.tribuo.interop.ExternalModel and implement the necessary methods to get things out of the Tribuo representation and into the appropriate LightGBM one. It would be pretty similar to the existing org.tribuo.common.xgboost.XGBoostExternalModel. However it looks like the LightGBM Java interface is autogenerated and very low level (https://central.sonatype.com/artifact/com.microsoft.ml.lightgbm/lightgbmlib/3.3.510), so it would need a fair amount of work to get that C-like API into something that?s easy to use from Java. The SWIG wrapper is basically passing around raw pointers so it needs actual objects and methods built on top of that. Microsoft built a lot of that for their SynapseML API to LightGBM (https://github.com/microsoft/SynapseML/tree/master/lightgbm/src/main/scala/com/microsoft/azure/synapse/ml/lightgbm), but that is for Apache Spark integration and I don?t think it would be reusable outside of Spark & Scala (and we don?t want Scala in Tribuo?s builds it adds too much complexity). Alternatively you could look at Amazon?s DJL which has a LightGBM interface (https://github.com/deepjavalibrary/djl/tree/master/engines/ml/lightgbm), but as that?s focused on deep learning I don?t know if it would give you the necessary control over the incoming feature representation. DJL?s wrapper is less than 1000 lines though, so it probably isn?t too hard to make an equivalent for Tribuo. Basically on the Tribuo side you?d need to write LightGBMExternalModel (e.g. https://github.com/oracle/tribuo/blob/main/Common/XGBoost/src/main/java/org/tribuo/common/xgboost/XGBoostExternalModel.java), a short protobuf definition for serialization (e.g. https://github.com/oracle/tribuo/blob/main/Common/XGBoost/src/main/resources/protos/tribuo-xgboost.proto#L54), then some kind of output converter mechanism if you wanted it to work for both classification & regression (https://github.com/oracle/tribuo/blob/main/Common/XGBoost/src/main/java/org/tribuo/common/xgboost/XGBoostOutputConverter.java), or just make it only do regression. Then you?d need to map from Tribuo?s internal representation of a sparse vector/matrix of feature values to the internal representation LightGBM wants (which shouldn?t be too bad as it supports CSR and Tribuo?s representation is basically that with more objects, e.g. https://github.com/oracle/tribuo/blob/main/Common/XGBoost/src/main/java/org/tribuo/common/xgboost/XGBoostTrainer.java#L711), write a mapping from LightGBM?s output to Tribuo?s output (which if it?s an array is easy, otherwise will require some unpicking), and then the glue to stick that to the SWIG wrapper Microsoft provide. That glue is the worst part as the SWIG isn?t very well documented, but it would be pretty similar to the glue DJL have here - https://github.com/deepjavalibrary/djl/blob/master/engines/ml/lightgbm/src/main/java/ai/djl/ml/lightgbm/jni/JniUtils.java. Training support would add more complexity on the Tribuo side, though it would be pretty similar to the rest of our XGBoost wrapper so you could base it off that. The main difficulty would still be in getting the LightGBM wrapper up from JNI as now you?ve got to pass parameters and configuration across as well, so it?ll be quite a bit more effort. Thanks, Adam -- Adam Pocock Principal Member of Technical Staff Machine Learning Research Group Oracle Labs, Burlington, MA On 16 Jun 2023, at 12:23, Sikka, Sandeep wrote: Thanks Adam for the feedback. In your opinion, how difficult would it be for one or more experienced Java Engineers to implement support for wrapping the LightGBM Java API in Tribuo? Assume no prior experience working in the Tribuo codebase. I am wondering if we take it on for our needs and contribute the developments back to Tribuo, how much time/effort will that take. Sandeep. From: Adam Pocock Date: Friday, June 16, 2023 at 11:25 AM To: "Sikka, Sandeep" Cc: "Parthasarathy, Bharadwaj" , "tribuo-devel at oss.oracle.com" , "Kumar, Navdeep" , Jack Sullivan Subject: Re: [EXT]:Re: [External] : Re: Tribuo Non Numerical Features Support Hi Sandeep, Tribuo doesn?t support inference for LightGBM models, aside from those which have been exported in ONNX format. You?d have to look at how the ONNX format model expects the categorical inputs to be passed in, I?m not sure if ONNX?s TreeEnsembleRegressor (https://github.com/onnx/onnx/blob/main/docs/Operators-ml.md#aionnxmltreeensembleregressor) supports special treatment of categorical inputs so it might not be possible. It looks like there is support for LightGBM in things like JPMML, but JPMML is AGPL licensed and so we can?t use it in Tribuo. We haven?t looked at directly wrapping the LightGBM Java API, as we already had XGBoost support for tree ensembles. Thanks, Adam -- Adam Pocock Principal Member of Technical Staff Machine Learning Research Group Oracle Labs, Burlington, MA On 16 Jun 2023, at 11:08, Sikka, Sandeep wrote: Hi Adam Thank you for the quick response. We are looking to score (perform inference) on a LGBMRegressor trained externally with the categorical features applied using the categorical_features parameter: https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html#categorical-feature-support. This uses a technique different from OHE: https://lightgbm.readthedocs.io/en/latest/Features.html#optimal-split-for-categorical-features. Do you have any thoughts on how we can achieve inference in Tribuo (we train the model in Python scikit-learn pipelines)? Thanks Sandeep. From: Adam Pocock > Date: Friday, June 16, 2023 at 9:34 AM To: "Sikka, Sandeep" >, "Parthasarathy, Bharadwaj" > Cc: "tribuo-devel at oss.oracle.com" >, "Kumar, Navdeep" >, Jack Sullivan > Subject: [EXT]:Re: [External] : Re: Tribuo Non Numerical Features Support Hi Sandeep, Bharad, The Tribuo mailing list is monitored, but I can?t see the email you sent to the list in the moderator console nor the archives, did you get a bounceback from the mailserver? It may be configured only to accept emails from people who have joined the mailing list. Anyway, the Tribuo development team have their email addresses listed in the pom file visible on Github. As to the question of feature support, in Tribuo all features are converted to numerical values during the data loading step. Categoricals can be encoded as one hot features or directly as doubles by choosing an index, and ordinals can be encoded as doubles. Very few ML algorithms directly support categorical variables without one hot-encoding or vectorization, I think of the popular ones the only kind that do are trees. Tribuo?s tree package doesn?t currently have specializations for categoricals though there is enough information in the feature domain to support that if we do add the algorithmic support. In tree algorithms ordinals are naturally supported as the splits don?t care about the distance between any feature values, only their relative ordering (or at least all the splitting algorithms we have in Tribuo are of that kind). You can see how to perform featurization into doubles in a number of ways in the Columnar Data (https://tribuo.org/learn/4.3/tutorials/columnar-tribuo-v4.html) and Document Classification (https://tribuo.org/learn/4.3/tutorials/document-classification-tribuo-v4.html) tutorials. Thanks, Adam -- Adam Pocock Principal Member of Technical Staff Machine Learning Research Group Oracle Labs, Burlington, MA On 16 Jun 2023, at 00:13, Sikka, Sandeep > wrote: Adding a few individual emails on the project from Github. The devel email list doesn?t appear to have any activity. Thanks Sandeep Sikka Distinguished Engineer 500 Staples Dr., Framingham MA 508.253.7977 sandeep.sikka at staples.com From: "Parthasarathy, Bharadwaj" > Date: Thursday, June 15, 2023 at 2:46 PM To: "tribuo-devel at oss.oracle.com" > Cc: "Sikka, Sandeep" >, "Kumar, Navdeep" > Subject: Tribuo Non Numerical Features Support Hi, This is Bharadwaj from one of Staples' Engineering teams. We are exploring Tribuo java framework for one of our use cases. We are trying to use Tribuo for runtime inferencing with a regressor model. We have non numerical features that we are trying to integrate with Tribuo. However, all the classes or samples for regressor points us to passing feature values as only double. Feature names as String and Feature Values as Double is what we see. Does Tribuo support non numerical features (categorical and ordinal features) such as String, Boolean etc or is it strictly only double values? How can we train and infer supervised models that consume categorical and ordinal features? I am not sure if we are missing something here, it would be helpful if you can provide your inputs. Regards, Bharad -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 3186 bytes Desc: image001.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 719 bytes Desc: image002.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.png Type: image/png Size: 809 bytes Desc: image003.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.png Type: image/png Size: 931 bytes Desc: image004.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.png Type: image/png Size: 728 bytes Desc: image005.png URL: