There's a gap in the ML world that doesn't get discussed enough. LLMs, through pretraining on enormous amounts of text, have absorbed a kind of domain knowledge that's completely invisible to a gradient boosted tree. A model like LLaMA knows that a 45-year-old married professional working 50 hours a week in a managerial role probably has different financial patterns than a 22-year-old in a part-time service job. XGBoost just sees integers.
But the flip side is equally true: LLMs can't actually learn from a table of 1,000 rows the way a gradient boosted tree can. They don't update weights. They don't find non-linear interactions between credit inquiries and debt-to-income ratios. Classical ML is built for exactly that.
So the question became: what if you let an LLM describe and contextualise each row, extract those descriptions as features, and then pass everything into XGBoost? The LLM handles semantic enrichment. XGBoost handles the actual learning. Best of both worlds, at least in theory.
The core pipeline looks like this: for each data row, three different LLM prompt framings generate a structured description of the person, as a labour economist, credit analyst, or sociologist. Critically, the LLM is never told the target variable. It's asked to reason about the person, not to predict the outcome. This avoids leaking the label through the back door.
The LLM outputs are parsed into structured fields (career stage, stability score) and a behaviour note, which is then encoded into a 384-dimensional embedding using all-MiniLM-L6-v2. This embedding captures richer semantic content than the parsed fields alone. The full hybrid feature matrix, raw features plus LLM-derived features, is then fed to XGBoost.
Three variants are evaluated per dataset: Baseline (XGBoost on raw features only), LLM-only (XGBoost on LLM features only), and Hybrid (XGBoost on both).
Two things will silently break this kind of experiment. First: PCA leakage. If you fit PCA on all 1,000 rows before splitting into folds, the PCA has seen the test fold, even subtle leakage matters when AUC differences live in the 0.01 range. The fix is fitting PCA exclusively on the training fold inside each outer loop iteration, and transforming the test fold with that fitted PCA. Second: hyperparameter tuning on evaluation data. The fix is nested CV, 3 repeats of 5-fold, 15 outer scores per variant, with 30 Optuna trials using the TPE sampler in the inner loop.
Statistical evaluation used Wilcoxon signed-rank tests on the 15 paired outer fold scores (paired because the same folds apply to all variants), and Cohen's d for effect size with magnitude labels.
Two datasets were used, and the choice of the second one was deliberate. The UCI Adult Income dataset is the obvious first pick, but it's extremely well-known and the LLM almost certainly encountered it during pretraining. Any positive result there has a contamination problem.
The HMEQ Loan Default dataset was chosen as the contamination control. It originates from SAS Institute educational materials, has minimal Kaggle presence, and is genuinely obscure. The job category and loan reason features carry real semantic meaning that LLM pretraining can enrich, but the chance of the model having seen this dataset in a labelled ML context is substantially lower than for Adult.
| Dataset | Variant | AUC Mean | AUC Std | Cohen's d | Significant |
|---|---|---|---|---|---|
| Adult | Baseline | 0.9010 | 0.0291 | N/A | N/A |
| Adult | LLM-only | 0.8411 | 0.0314 | −1.975 | Yes (worse) |
| Adult | Hybrid | 0.8969 | 0.0287 | −0.139 | No |
| HMEQ | Baseline | 0.9044 | 0.0268 | N/A | N/A |
| HMEQ | LLM-only | 0.5587 | 0.0630 | −7.144 | Yes (worse) |
| HMEQ | Hybrid | 0.8654 | 0.0601 | −0.837 | Yes (worse) |
Adult Income: hybrid not significantly different from baseline (p = 0.12, Cohen's d = −0.14)
HMEQ Loan Default: hybrid significantly worse than baseline (p = 0.004, Cohen's d = −0.84)
Across all valid prompt framings and both datasets, no hybrid outperformed its baseline.
There are a few plausible mechanisms, and they're not mutually exclusive. The first is dimensionality: even with PCA compression, adding 10–30 embedding dimensions introduces noise that XGBoost has to learn to ignore. With only 1,000 training rows, the noise-to-signal ratio is unfavourable.
The second is that the LLM features probably aren't orthogonal to the raw features. The job category information is already in the feature matrix as an encoded integer. The LLM reasoning about that same job category may just be adding a noisy correlated version of what XGBoost already knew.
The HMEQ LLM-only result is particularly striking: an AUC of 0.56 is barely above random. Whatever the LLM reasons about financial stability and job trajectory is not capturing the actual default risk structure in the data. The HMEQ hybrid also shows higher variance (std 0.0601 vs 0.0268 for baseline), suggesting the LLM features aren't just adding noise, they're introducing instability.
SHAP analysis on a separately fitted hybrid model (for interpretability only, not reported as the main result) showed that on Adult, LLM PCA components ranked between 10th and 14th out of 49 total features, not completely ignored, but not dominant either. The top features remained raw structured columns like capital gain and education level.
On HMEQ, LLM features ranked similarly in the middle of the pack. The model was using them, but they weren't adding predictive value at the macro level, which is consistent with the significance tests.
The hypothesis had genuine theoretical appeal. The methodology was sound. The negative result isn't an artefact of bad experimental design, it's a genuine signal that at this scale, with these datasets, LLM-derived features don't bridge the gap between semantic reasoning and discriminative tabular learning.
A few things might change this: a larger dataset would reduce the noise-to-signal problem; a private proprietary dataset would eliminate contamination concerns entirely; a distillation approach where a small model is trained to predict LLM features from raw inputs might reduce the dimensionality problem; active learning where the LLM is only called on samples where the baseline is uncertain might concentrate the contribution where it actually matters.
For now, the conclusion is that the gap between LLM reasoning and tabular ML is harder to bridge than the theoretical motivation suggests, and that's worth knowing.
Full source code and experiment scripts on GitHub →