Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How XGBoost sample_weights alternate the objective function formula? #10297

Closed
geogreko opened this issue May 17, 2024 · 4 comments
Closed

How XGBoost sample_weights alternate the objective function formula? #10297

geogreko opened this issue May 17, 2024 · 4 comments

Comments

@geogreko
Copy link

From my understanding and reading post #144

when sample_weights are used, then first and second gradient

g_i and h_i are multiplied with sample_weights so that for each instance i:

grad_xgb[i] = grad[i] * case_weight[i]
hess_xgb[i] = hess[i] * case_weight[i]

The 2016 paper ‘XGBoost: A Scalable Tree Boosting System’ states that,
if we remove the constant terms, the objective function at step t is given in eq.3

image

The question is: How eq.3 is alternated when using sample weights?
Is it as simple as that:

(g_i*case_weight[i])*ft(xi)+1/2(h_i * case_weight[i]) (ft)power of 2(xi) + Ω(ft)

In other words, just substituting g_i, and h_i with grad[i] * case_weight[i] and hess[i] * case_weight[i] respectively?

Unfortunately, the links provided in post 144 are broken.

@trivialfis
Copy link
Member

Yes, it's multiplied into gradients. It's also used in some other places like sketching.

@geogreko
Copy link
Author

Great, thanks for the response!

@trivialfis
Copy link
Member

Feel free to reopen if there are further questions.

@geogreko
Copy link
Author

One more question.
In the case of sample_weights, do we also multiply the gradients (gi,hi) to the sample_weights when calculating the Gain function? Like g_i * case_weight_i, h_i * case_weight_i, for all gradients in the gain function below?
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants