How Does XGBoost Use Curvature and Regularization in Its Objective?

Objective: tree t minimizes sum l(y_i, yhat_i + f_t(x_i)) + Omega(f_t). Expand the loss around the current prediction with a second-order Taylor expansion, giving an approximate objective that only needs first-order gradients g_i and second-order gradients h_i.

Optimal leaf weight has a closed form: w* = -G / (H + lambda), where G and H are sums of g and h in the leaf, and lambda comes from Omega = gamma*T + 0.5*lambda*sum w^2.

Why second order: it captures both direction and curvature, so steps are more accurate and convergence is faster. First order only degrades to plain GBDT. Second-order info also gives a closed-form split Gain, enabling exact greedy or histogram-based split finding.