LightGBM - An In-Depth Guide [Python API]¶

> What is LightGBM?¶

LightGBM is a framework that provides an implementation of gradient boosted decision trees. The gradient boosted decision trees is a type of gradient boosted machines algorithm that uses decision trees as estimators of an ensemble. An ensemble consists of many weak models/estimators (decision trees) whose predictions are combined to make final prediction.

It was created by the researchers and developers team at Microsoft.

> Why "LightGBM" over Other Python Gradient Boosted Trees Implementations?¶

Light GBM is known for its

• Faster-training speed
• Good accuracy with default parameters
• Parallel and GPU learning
• Low memory footprint
• Capability of handling large datasets which might not fit in memory.

LightGBM provides API in C, Python, and R Programming.

LightGBM even provides CLI (Command Line Interface) which lets us use the library from the command line.

LightGBM estimators provide a large set of hyperparameters to tune the model. It even has a large set of optimization/loss functions and evaluation metrics already implemented.

> What Can You Learn From This Article?¶

As a part of this tutorial, we have explained how to use Python library LighGBM to solve machine learning tasks (Regression and Classification). Tutorial explains majority of Python API of library with simple and easy-to-understand examples.

Apart from training model and making predictions, it explains many different concepts like cross-validation, saving & loading model, visualizing features importances, early stopping training to avoid overfitting, how to create custom loss functions, how to create a custom evaluation metrics, how to use callbacks, etc.

All our examples have lightgbm models trained on toy datasets (structured - tabular) available from scikit-learn.

The main aim of this tutorial is to make readers aware of the majority of functionalities available through lightgbm and get them started with the framework.

> Which Other Python Libraries Provides Implementation Of Gradient Boosted Trees?¶

• XGBoost
• CatBooost
• Scikit-Learn

> How to Install LightGBM?¶

• PIP
  • pip install -U lightgbm
• Conda
  • conda install lightgbm

Below, we have listed important sections of tutorial to give an overview of the material covered. We know that the list below is big but you can skip some sections of tutorial which has a theory or repeat example of some concepts. We have included NOTE in those sections so you can skip them to complete tutorial faster. You can then refer to those sections in your free time or as per need.

Important Sections Of Tutorial¶

1. Load Datasets for Tutorial
   • Boston Housing Dataset for Regression Tasks
   • Breast Cancer Dataset for Binary Classification Tasks
   • Wine Dataset for Multi-Class Classification Tasks
2. LightGBM Models at High-Level (High-Level API)
3. Booster: train() - Core API to Train Model
   • Important Parameters of train() Function
   • Dataset: LightGBM Data Structure to Represent Data
   • Regression Example
   • Binary Classification Example
   • Multi-Class Classification Example
4. List of Important Parameters of LightGBM Estimators
5. LGBMModel (Scikit-Learn like API)
   • Regression Example
   • Binary Classification Example
6. LGBMRegressor (Scikit-Learn like API)
7. LGBMClassifier (Scikit-Learn like API)
8. Saving and Loading Model
9. Cross Validation
10. Plotting Functionality
    • Visualize Features Importance using "plot_importance()"
    • Visualize ML Metric using "plot_metric()"
    • Visualize Feature Values Split using "plot_split_value_histogram()"
    • Visualize Individual Boosted Tree using "plot_tree()"
11. Early Stopping Training
12. Feature Interaction Constraints
13. Monotonic Constraints
14. Custom Objective/Loss Function
15. Custom Evaluation Function
16. Callbacks

We'll start by importing the necessary Python libraries and printing the versions that we have used in our tutorial.

1. Load Datasets ¶

We'll be using the below-mentioned three different datasets which are available from sklearn as a part of this tutorial for explanation purposes.

1. Boston Housing Dataset: It's a regression problem dataset which has information about the various attribute of houses in Boston and their price in dollar. This will be used for regression tasks.
2. Breast Cancer Dataset: It's a classification dataset that has information about two different types of tumor. It'll be used for explaining binary classification tasks.
3. Wine Dataset - It's a classification dataset that has information about ingredients used in three different types of wines. It'll be used for explaining multi-class classification tasks.

We have loaded all three datasets mentioned one by one below. We have printed descriptions of datasets which gives us an overview of dataset features and size. We have even loaded each dataset as a pandas data frame and displayed the first few samples of data.

Boston Housing Dataset¶

Breast Cancer Dataset¶

Wine Dataset¶

2. LightGBM Models at High-Level ¶

LightGBM provides four different estimators to perform classification and regression tasks.

1. Booster - It is a universal estimator created by calling train() method. It can be used for regression as well as classification tasks. All other estimators are wrapper around it.
2. LGBMModel - Its a universal estimator with scikit-learn like API that can handle both classification and regression datasets with settings.
3. LGBMRegressor - It is an estimator with scikit-learn like API designed to work with regression datasets.
4. LGBMClassifier - It is an estimator with scikit-learn like API designed to work with classification datasets.

3. Booster (train()): Core API to Train Model ¶

The simplest way to create an estimator in lightgbm is by using the train() method. It takes as input estimator parameter as dictionary and training dataset. It then trains the estimator and returns an object of type Booster which is a trained estimator that can be used to make future predictions.

3.1 Important Parameters of "train()" Function¶

Below are some of the important parameters of the train() method.

• params - This parameter accepts dictionary specifying parameters of gradient boosted decision trees algorithm. We just need to provide an objective function to get started with based on the type of problem (classification/regression). We'll later explain a commonly used list of parameters that can be passed to this dictionary.
• train_set - This parameter accepts lightgbm Dataset object which holds information about feature values and target values. It's an internal data structure designed by lightgbm to wrap data.
• num_boost_round - It specifies the number of booting trees that will be used in the ensemble. The group of gradient boosted trees is called ensemble to whom we generally refer as an estimator. The default value is 100.
• valid_sets - It accepts list of Dataset objects which as validation sets. These validation sets will be evaluated after each training round.
• valid_names - It accepts a list of strings of the same length as that of valid_sets specifying names for each validation set. These names will be used when printing evaluation metrics for these datasets as well as when plotting them.
• categorical_feature - It accepts list of strings/ints or string auto. If we give a list of strings/ints then those columns from the dataset will be treated as categorical columns.
• verbose_eval - It accepts bool or int as value. If we set the value to False or 0 then it won't print metrics evaluation results calculated on validation sets that we passed. If we pass True then it'll print results for each round. If we pass an integer greater than 1 then it'll print results repeatedly after that many rounds.

3.2 Dataset: LightGBM Data Structure to Represent Data¶

The dataset is a lightgbm internal data structure for holding data and labels. Below are important parameters of the class.

• data - It accepts numpy array, pandas dataframe, scipy sparse matrix, list of numpy arrays, h2o data table’s frame as input holding feature values.
• label - It accepts numpy array, pandas series, pandas one column dataframe specifying target values. We can even set this parameter to None if we don't have target values. The default is None.
• feature_name - It accepts a list of strings specifying feature names.
• categorical_feature - It has the same meaning as that mentioned in the train() method parameter above. We can handle categorical feature here or in that method.

3.3 Regression Example¶

The first problem that we'll solve using lightgbm is a simple regression problem using the Boston housing dataset which we loaded earlier. We have divided the dataset into train/test sets and created a Dataset instance out of them. We have then called the lightgbm.train() method giving it train and validation set. We have set the number of boosting rounds to 10 hence it'll create 10 boosted trees to solve the problem. After training completes, it'll return an instance of type Booster which we can later use to make future predictions on the dataset. As we have given the validation set as input, it'll print the validation l2 score after each iteration of training. Please make a note that by default lightgbm minimizes l2 loss for regression problems.

Below we have made predictions on train and test data using a trained booster. We have then calculated R2 metrics for both using the sklearn metric method. Please make a note that the predict() method accepts numpy array, pandas dataframe, scipy sparse matrix, or h2o data table’s frame as input for making predictions.

If you are interested in learning the list of available metrics in scikit-learn then please feel free to check our tutorial on the same.

• Scikit-Learn : Model Evaluation and Scoring Metrics

The predict() method of a few important parameters which can be used to make a different kind of predictions.

• raw_score - It a boolean parameter which if set to True will return raw predictions. For regression problems, this won't make any difference but for classification problems, it'll return function values as output than probabilities.
• pred_leaf - This parameter accepts boolean values which if set to True will return an index of leaf in each tree that was predicted for a particular sample. The size of the output will be n_samples x n_trees.
• pred_contrib - It returns an array of features contribution for each sample. It'll return an array of size (n_features + 1) for each sample of data where the last value is the expected value and the first n_features values are the contribution of features in making that prediction. We can add the contribution of each feature to the last expected value and we'll get an actual prediction. It's commonly referred to as SHAP values.

If you are interested in learning about SHAP values and our tutorial on the awesome SHAP package which lets us visualize these SHAP values in different ways to understand the performance of the model then check our tutorial on the same.

• SHAP - Explain Machine Learning Model Predictions using Game-Theoretic Approach

We can call the num_trees() method on the booster instance to get a number of trees in the ensemble. Please make a note that if we don't stop training early then a number of trees will be the same as num_boost_round. But if we are stopping training early then a number of trees will be different from num_boost_round. We have explained later in this tutorial how we can stop training if the ensemble's performance is not improving when evaluated on the validation set.

The booster instance has another important method named feature_importance() which can return us the importance of features based on gain and split values of the trees.

3.4 Binary Classification Example¶

In this section, we have explained how we can use the train() method to create a booster for a binary classification problem. We are training the model on the breast cancer dataset and later evaluating the accuracy of it using a metric from sklearn. We have set an objective to binary for informing the train() method that we'll be giving data for binary classification problem. We have also set the verbosity parameter value to -1 in order to prevent training messages. It'll still print validation set evaluation results which can be turned off by setting the verbose_eval parameter to False.

Please make a note that for classification problems predict() method of booster return probabilities. We have included logic to convert probabilities to the target class.

LightGBM evaluates binary log loss function by default on the validation set for binary classification problems. We can give the metric parameter in the dictionary which we are giving to the train() method with any metric names available with lightgbm and it'll evaluate that metric. We'll later explain the list of available metrics with lightgbm.

3.5 MultiClass Classification Example¶

NOTE: Please feel free to skip this section if you are in hurry and have understood how to use LightGBM for classification tasks using our previous binary classification example.

As a part of this section, we have explained how we can use the train() method for multi-class classification problems. We are using it on the wine dataset which has three different types of wine as the target variable. We have set an objective function to multiclass. We need to provide the num_class parameter with an integer specifying a number of classes whenever we are using the method for multi-class classification problems.

The predict() method returns the probabilities of each class in case of multi-class problems. We have included logic to select the class with maximum probability as a prediction.

LightGBM evaluates multi-class log loss function by default on the validation set for binary classification problems.

4. List of Important Parameters of LightGBM Estimators (train() Function) ¶

NOTE: Please feel free to skip this section if you are in hurry. It is theoretical section listing parameters of "train()" constructor. You can refer them later as you need to tweak model.

We'll now list down important parameters of lightgbm which can be provided in a dictionary when calling the 'train()' method. We can provide the same parameters to estimators (LGBMModel, LGBMRegressor, and LGBMClassifier) that are readily available in lightgbm with the only difference that we don't need to provide them as a dictionary but we can provide them directly when creating an instance. We'll be introducing those estimators from the next section onwards.

• objective - This parameter lets us define an objective/loss function to use for the task. The default value of this parameter is regression. Below is a list of commonly used values for this parameter.
  • regression
  • regression_l1
  • tweedie
  • binary
  • multiclass
  • multiclassova
  • cross_entropy
  • Other available objective functions
• metric - This parameter accepts metrics to be evaluated on evaluation datasets if evaluation datasets are provided as eval_set/validation_sets parameter value. We can provide more than one metric and all will be evaluated on validation sets. Below is a list of the commonly used values of metrics.
  • rmse
  • l2
  • l1
  • tweedie
  • binary_logloss
  • multi_logloss
  • auc
  • cross_entropy
  • Other available metrics
• boosting - This parameter accepts one of the below-mentioned string specifying which algorithm to use.
  • gbdt - Default. Gradient Boosting Decision Tree
  • rf - Random Forest
  • dart - Dropouts meet multiple additive regression trees
  • goss - Gradient-based on side sampling
• num_iterations - This parameter is an alias to num_boost_round which lets us specify the number of trees to the ensemble to create an estimator. The default is 100.
• learning_rate - This parameter accepts a learning rate to use for the training process. The default is 0.1.
• num_class - If we are working with multi-class classification problems then we need to provide a number of classes to this parameter.
• num_leaves - This parameter accepts integer specifying the number of max leaves allowed per tree. The default is 31.
• num_threads - It accepts integer specifying the number of threads to use for training. We can set it to the same number of cores of the system.
• seed - This lets us specify the default seed for training which lets us regenerate the same results.
• max_depth - This parameter lets us specify the maximum depth allowed for trees in the ensemble. The default is -1 which let trees grow as deep as possible. We can restrict this behavior by setting this parameter.
• min_data_in_leaf - This parameter accepts integer value specifying a minimum number of data points that can be kept in one leaf. This parameter can be used to control overfitting. The default value is 20.
• bagging_fraction - This parameter accepts float value between 0-1 letting us specify randomly select that much part of the data when training. This parameter can help prevent overfitting. The default is 1.0.
• feature_fraction - This parameter accepts a float value between 0-1 that informs the algorithm to select that fraction of features from the total for training at each iteration. The default is 1.0 hence selecting all features.
• extra_trees - This parameter accepts boolean values specifying whether to use an extremely randomized tree or not.
• early_stopping_round - This parameter accepts integer specifying we should stop training if evaluation metric is not improving on last evaluation set for iterations specified by this parameter.
• monotone_constraints - This parameter lets us specify whether our model should enforce increasing, decreasing, or no relation of an individual feature with the target value. We have explained the usage of this parameter in a section named monotonic constraints.
• monotone_constraints_method - This parameter accepts one of the below-mentioned string specifying the type of monotonic constraints to impose.
  • basic - Basic monotone constraints method which can over constrain the model.
  • intermediate - It’s a little advanced constraints method which is a little less constraining than the basic method but can take a little more time.
  • advanced - - It’s an advanced constraints method that is less constraining than basic and intermediate methods but can take more time.
• interaction_constraints - This parameter accepts a list of lists where individual list specify feature indices which are allowed to interact with one another. We have explained feature interaction in detail in section feature interaction constraints.
• verbosity - This parameter accepts integer value for controlling logging message when training.
  • <0 - Only Fatal Errors are displayed.
  • 0 - Only Error/Warning messages are displayed.
  • 1 - Only info messages are displayed.
  • >1 - Only debug information is displayed.
• is_unbalance - This is a boolean parameter that should be set to True if data is imbalanced. It should be used with binary and multi-class classification problems.
• device_type - It accepts one of the below string specifying device type of training.
  • cpu
  • gpu
• force_col_wise - This parameter accepts boolean value specifying whether to force column-wise histogram building when training. If data has too many columns then setting this parameter to True will improve training process speed by reducing memory usage.
• force_row_wise - This parameter accepts boolean value specifying whether to force row-wise histogram building when training. If data has too many rows then setting this parameter to True will improve training process speed by reducing memory usage.

Please make a NOTE that this is not the full list of parameters available with lightgbm but only a few important parameters list. If you are interested in learning about all parameters then please feel free to check the below link.

• LightGBM Full Parameters List

5. LGBMModel (Scikit-Learn like API) ¶

LGBMModel class is a wrapper around Booster class that provides scikit-learn like API for training and prediction in lightgbm. It let us create an estimator object with a list of parameters as input. We can then call the fit() method giving train data for training and the predict() method for making a prediction. The parameters which we had given as a dictionary to params parameter of train() can now directly be given to the constructor of LGBMModel to create a model. LGBMModel let us perform both classification and regression tasks by specifying the objective of the task.

5.1 Regression Example¶

Below we have explained with a simple example of how we can use LGBMModel to perform regression tasks with Boston housing data. We have first created an instance of LGBMModel with the objective as regression and number of trees set to 10. The n_estimators parameter is an alias of num_boost_round parameter of train() method.

We have then called the fit() method for the training model giving train data to it. Please make a note that it accepts numpy arrays as input and not lightgbm Dataset object. We have also given a dataset to be used as an evaluation set and metrics to be evaluated on the evaluation dataset. The parameter of the fit() method is almost the same as that of the train() method.

At last, we have called the predict() method to make predictions.

5.2 Binary Classification Example¶

Below we have explained with a simple example of how we can use LGBMModel for classification tasks. We have a trained model with a breast cancer dataset. Please make a note that the predict() method returns probabilities. We have included logic to calculate class from probabilities.

6. LGBMRegressor (Scikit-Learn like API) ¶

LGBMRegressor is another wrapper estimator around the Booster class provided by lightgbm which has the same API as that of sklearn estimators. As its name suggests, it’s designed for regression tasks. LGBMRegressor is almost the same as that of LGBMModel with the only difference that it’s designed for only regression tasks. Below we have explained the usage of LGBMRegressor with a simple example using the Boston housing dataset. Please make a note that LGBMRegressor provides the score() method which evaluates the R2 score for us which we used to evaluate using the sklearn metric method till now.

7. LGBMClassifier (Scikit-Learn like API) ¶

LGBMClassifier is one more wrapper estimator around the Booster class that provides a sklearn-like API for classification tasks. It works exactly like LGBMModel but for only classification tasks. It also provides a score() method which evaluates the accuracy of data passed to it.

Please make a note that LGBMClassifier predicts actual class labels for the classification tasks with the predict() method. It provides the predict_proba() method if we want probabilities of target classes.

7.1 Binary Classification Example¶

Below we have explained with a simple example of how we can use LGBMClassifier for binary classification tasks. We have explained its usage with the Breast cancer dataset.

7.2 Multi-Class Classification Example¶

NOTE: Please feel free to skip this section if you are in hurry and have understood how to use LightGBM for classification tasks using our previous binary classification example.

Below we have explained the usage of LGBMClassifier for multi-class classification tasks using the Wine classification dataset.

8. Saving and Loading Model ¶

We'll now explain how we can save the trained model to a disk to use later for predictions. Lightgbm provides the below-mentioned methods for our purpose of saving and loading models.

• save_model() - This method takes as an input file name to which save the model.
• model_to_string() - This method returns a string representation of the model which we can then save to a text file.
• lightgbm.Booster() - This constructor lets us create an instance of the Booster class. It has two important parameters that can help us load a model from a file or from a string.
  • model_file - This parameter accepts the file name from which to load the trained model.
  • model_str - This parameter accepts a string that has information about the trained model. We need to give a string that was generated using model_to_string() to this parameter after loading from the file.

Below we have explained with simple examples how we can use above mentioned methods to save models to a disk and then load it.

Please make a note that in order to save model trained using LGBMModel, LGBMRegressor, and LGBMClassifier, we first need to get their Booster instance by using the booster_ attribute of an estimator and then save it. LGBMModel, LGBMRegressor, and LGBMClassifier do not provide saving and loading functionalities. It’s only available with the Booster instance.

Save Model using LightGBM "save_model()" Method¶

Load Model through "Booster()" Constructor¶

Save Model as a String to File using "model_to_string()" Method¶

Load Model from String through Booster() Constructor¶

9. Cross Validation Example ¶

Lightgbm let us perform cross-validation using cv() method. It accepts model parameters as a dictionary like the train() method. We can then give a dataset on which to perform cross-validation. It performs 5-fold cross-validation by default. We can change the number of folds by setting the nfold parameter. It also accepts sklearn's data splitter like KFold, StratifiedKFold, ShuffleSplit, and StratifiedShuffleSplit. We can provide these data splitters to the folds parameter of the method.

The cv() method returns a dictionary that has information about the mean and standard deviation of loss for each round of training. We can even ask the method to return an instance of CVBooster by setting the return_cvbooster parameter to True. CVBooster object has information about cross-validation.

10. Plotting Functionality ¶

Lightgbm provides a list of the below-mentioned plotting functions.

10.1 Visualize Features Importance using "plot_importance()"¶

This method accepts a booster instance and plots feature importance using it. Below we have created a feature importance plot using the booster trained earlier for the regression task. The method has a parameter named importance_type which can be set to string split will plot the number of times feature was used for split and plots gains of splits if set to string gain. The value of parameter importance_type is split. The plot_importance() method has another important parameter max_num_features which accepts an integer specifying how many features to include in the plot. We can limit the number of features using this parameter as it'll include only that many top features in the plot.

10.2 Visualize ML Metric using "plot_metric()"¶

This method plots the results of an evaluation metric. We need to give a booster instance to the method in order to plot an evaluation metric evaluated on the evaluation dataset.

10.3 Visualize Feature Values Split using "plot_split_value_histogram()"¶

This method takes as input booster instance and feature name/index. It then plots a split value histogram for the feature.

10.4 Visualize Individual Boosted Tree using "plot_tree()"¶

This method lets us plot the individual tree of the ensemble. We need to give a booster instance and index of the tree which we want to plot to it.

11. Early Stopping Training to Avoid Overfitting ¶

Early stopping training is a process where we stop training if the evaluation metric evaluated on the evaluation dataset is not improving for a specified number of rounds. Lightgbm provides parameter named early_stopping_rounds as a part of train() method as well as fit() method of lightgbm sklearn-like estimators. This parameter accepts integer value specifying that stop the training process if the evaluation metric result has not improved for that many rounds.

Please make a note that we need an evaluation dataset in order for this to work as it’s based on evaluation metric results evaluated on the evaluation dataset.

Below we have explained the usage of the parameter early_stopping_rounds for regression and classification tasks with simple examples.

How to Stop Training Early using "early_stopping()" Callback?¶

Lightgbm provides early stopping training functionality using the early_stopping() callback function as well. We can give number of rounds to early_stopping() function and give that function to callbacks parameter of train()/fit() method. We have explained callbacks in an upcoming section.

12. Feature Interaction Constraints ¶

When lightgbm has completed training trees of the ensemble on a dataset, the individual node of trees represents some condition based on some value of the feature. When we are making predictions using an individual tree, we start from the root node of the tree, checking the feature condition specified in the node with our sample feature values. We make decisions based on the feature values in our sample and the condition present in the tree. This way we follow a particular path reaching the leaf of the tree to make the final prediction. By default, there is no restriction on which node can have which feature as a condition. This process of making a final decision by going through nodes of tree checking feature condition is called feature interaction because predictor has come to the particular node after evaluating the condition of the previous node. Lightgbm can let us define restrictions on which feature to interact with which another feature. We can give a list of indices and only that many features will interact with one another. Those features won't be allowed to interact with other features and this restriction will be forced when creating trees during the training process.

Below we have explained with a simple example of how we can force feature interaction constraint on estimator in lightgbm. Lighgbm estimators provide a parameter named interaction_constraints which accepts a list of lists where individual lists are indices of parameters that are allowed to interact with one another.

13. Monotonic Constraints ¶

Lightgbm let us specify monotonic constraints on a model that specifies whether the individual feature has increasing, decreasing, or no relation with the target value. It let us specify monotone values of -1, 0, and 1 forcing model to impose decreasing, none, and increasing relationship of the feature with the target. We can provide a list with the same length as a number of features specifying 1,0 or -1 for the monotonic relationship by using the monotone_constraints parameter. We have explained below with a simple example of how we can enforce monotonic constraints in lightgbm.

14. Custom Objective/Loss Function ¶

Lightgbm let us define custom objective function as well. We need to define a function that takes a list of prediction and actual labels as input and returns the first derivative and second derivative of the loss function. We need to return the first derivative and the second derivative of loss function evaluated using predictions and actual values. We can give a custom-defined objective/loss function to the objective parameter of the estimator. If we are using the train() method then we need to give this function to the fobj parameter.

Below we have designed the mean squared error objective function. We have then given this function to an objective parameter of LGBMModel for an explanation.

15. Custom Evaluation Function ¶

Lightgbm lets us define our own evaluation metric if we don't want to use evaluation metrics available with lightgbm. We need to define a function that takes an input list of predictions and actual target values and returns a string specifying metric name, metric evaluation value, and boolean value specifying whether higher is better or not. The value higher is better should be returned True if we want the metric value to be maximized else it should be False if we want the metric value to be minimized.

We need to give reference to this function as the value of parameter feval if we are using train() method to design our estimator. If we are using a sklearn-like estimator then we need to give this function to the eval_metric parameter of the fit() method.

Below we have explained with simple examples of how we can use custom evaluation metrics with lightgbm.

16. Callbacks ¶

Lightgbm provides users with a list of callback functions for a different purpose that gets executed after each iteration of training. Below is a list of available callback functions with lightgbm:

• early_stopping(stopping_rounds) - This callback function accepts an integer specifying whether to stop training if evaluation metric results on the last evaluation set are not improved for that many iterations.
• print_evaluation(period, show_stdv) - This callback function accepts integer values specifying how often to print evaluation results. Evaluation metric results are printed at every that many iterations as specified.
• record_evaluation(eval_result) - This callback function accepts a dictionary in which evaluation results will be recorded.
• reset_parameter() - This callback function lets us reset the learning rate after each iteration of training. It accepts an array of size the same as the number of iterations or callback returning the new learning rate for each iteration.

The callbacks parameter which is available with the train() method and the fit() method of estimators accepts a list of callback functions.

Below we have explained with simple examples of how we can use different callback functions. The explanation of the early_stopping() callback function has been covered in the early stopping training section of this tutorial.

How to Use "print_evaluation()" Callback?¶

How to Use "record_evaluation()" Callback?¶

How to Use "reset_parameter()" Callback?¶

This ends our tutorial explaining the Python API of LightGBM end to end.

References ¶

Other Gradient Boosted Machine Implementations¶

• CatBoost - An In-Depth Guide (Python)
• XGBoost - An In-Depth Guide (Python)
• Scikit-Learn - Ensemble Learning: Boosting

LightGBM Documentation¶

• LightGBM Full Parameter List
• LightGBM Performance Experiments

Other Useful Tutorials¶

• SHAP - Explain Machine Learning Model Predictions using Game-Theoretic Approach
• Scikit-Learn : Model Evaluation and Scoring Metrics