Optiver - Trading at the Close

We used MAE for everything, including tuning and model training and evaluation @tztang

In our local validation, grouping by stock_id is better than grouping by both stock_id and date_id. We did not use any postprocessing in our evaluation. We used dates 436-480

For online learning, we saved the data in each date_id and time_id in a CSV file, then loaded them one by one for prediction. Evaluation was done by each group of date_id and time_id and concatenated together to calculate the MAE. Same dates 436-480. Whether online learning or not we used this same method to evaluate when we want to be in sync with the API

Hey,

Our approach was different and based on the .refit() and .train(init_model) on LightGBM. .refit() got us the better score (5.4344):

We retrained daily at the start of the day, based on the previous day's data and the same day the previous week. I.e. if we were to predict a Friday - we would refit with data from Thursday and Friday the week before (assuming no holidays etc.) checking on this condition:

    #If start of new day
    if test.seconds_in_bucket.unique()[0] == 0:
        print(f"Day Completed in : {time.time()-start}")
        start = time.time()

        #Process revealed target
        revealed_targets_df = process_revealed_targets(new_revealed_targets=revealed_targets, old_revealed_targets=revealed_targets_df)

        #Update model if new day + after 480
        if test.date_id.max() > 476:
            UPDATE_MODELS=True
        else:
            UPDATE_MODELS=False

    #Don't update if not new day 
    else:
        UPDATE_MODELS=False
content_copy

And then our actual online learning was pretty simple for refit - we found this was better than booster.train(init_model):

                ######################################################INFERENCE REFITTING START######################################################
                if UPDATE_MODELS:
                    try:
                        input_start_time = time.time()
                        #NUM DAYS NEESD TO BE T+1 -> starts at current day  
                        lgb_input = get_raw_lgb_input(X_input = train, num_days =6, feature_names = feature_names, _revealed_targets_df = revealed_targets_df)
                        dt = revealed_targets_df['date_id'][-1]
                        X_refit_input = lgb_input.filter((pl.col('date_id') == dt-4) | (pl.col('date_id') == dt))[feature_names]
                        y_refit_input = lgb_input.filter((pl.col('date_id') == dt-4) | (pl.col('date_id') == dt))['target'].to_numpy()

#                         X_refit_input = lgb_input.filter((pl.col('date_id') == dt))[feature_names]
#                         y_refit_input = lgb_input.filter((pl.col('date_id') == dt))['target'].to_numpy()
                        print(f"Completed getting 5 day input in {time.time() - input_start_time}")

                        for idx in range(len(lgb_models)):
                            input_start_time = time.time()
                            lgb_models[idx] = lgb_models[idx].refit(data = X_refit_input, label = y_refit_input, feature_name = feature_names, decay_rate=0.993)
                            print(f"Completed 1 day refit for fold {idx} in {time.time() - input_start_time}")
                    except Exception as e:
                        print(e)
                        print('Failed to refit')

                ######################################################INFERENCE REFITTING END######################################################
                #If no need to update, then just take the last 1 day
                else:
                    lgb_input = get_raw_lgb_input(X_input = train, num_days = 1, feature_names = feature_names, _revealed_targets_df = revealed_targets_df)

                try:
                    input_start_time = time.time()
                    for mdl in lgb_models:
                        lgb_prediction = mdl.predict(lgb_input[feature_names][-n_inputs : ])
                        ensemble_predictions.append(lgb_prediction)
                    print(f"Completed LGB prediction loop in {time.time() - input_start_time}")
                except Exception as e:
                    print(e)
content_copy

And our worse submission (5.4270) was the same flavour but with a different decay parameter:

lgb_models[idx] = lgb_models[idx].refit(data = X_refit_input, label = y_refit_input, feature_name = feature_names, decay_rate=0.993)

refit will effectively keep the tree structures the same but update the values according to the new data so initial fitting parameters are extremely important for regularization.

Our validation process was to log everything via MLFlow and look at impact of different decay rates over a testing period of days 440-480 to validate.

What postprocessing did you use in your submission ? I wonder if it made a difference, for us, there is barely any difference between our zero-sum and no postprocessing submission. We did not use the zero-mean as the local CV and the public LB was worse than zero-sum with refitting.

EDIT: the zero-sum postprocessing does help. It’s the feature set having different base accuracies. Our EWM and bollinger band features have worsened the private LB even though the CV (436-480) and public LB was better

| PostProcess | Feature Set 1 | Feature Set 2 |
| None | 5.4475 | 5.4457 |
| Zero Sum | 5.4458 | 5.4440 |

@yusefkaggle thanks for the mention! I try to help others to the best I can. These responses motivate me to share better and help!
Best wishes!

BTW, our solution also used Catboost and LightGBM. We discarded XGBoost as it did not give a good CV score. My gold medal submission notebook highlights this approach.

As mentioned by @yeoyunsianggeremie, post-processing did not make a significant change for us. Both our chosen submissions would have given us the same rank. We did not try the mean submission and used zero-sum and no post-processing.

--> 650 with open(filename, 'rb') as f:
651 with _read_fileobject(f, filename, mmap_mode) as fobj:
652 if isinstance(fobj, str):
653 # if the returned file object is a string, this means we
654 # try to load a pickle file generated with an version of
655 # Joblib so we load it with joblib compatibility function.

FileNotFoundError: [Errno 2] No such file or directory: '/kaggle/input/optivermodels/LGBM1R_V27.model'