Information-Geometric Perspectives on Merging Variational Foundation Models

We propose an information-geometric framework for merging variational foundation models that preserves global robustness while integrating domain-specific knowledge in a principled manner. Assuming that the foundation models have been pretrained or fine-tuned using the Improved Variational Online Newton (IVON) optimizer, matching Adam’s computational cost while providing Bayesian advantages, we formulate the merging problem between the pretrained and fine-tuned models as an information-geometric projection. Under mild assumptions, this reduces to computing a barycenter in the variational parameter space, yielding a computationally efficient and theoretically grounded merging rule. The framework naturally extends to multi-model barycentric merging, minimizing the average discrepancy among fine-tuned models.