FedSIT: Efficient federated fine-tuning with model splitting and importance-based tuning

The rapid scalability of large language models (LLMs) has driven significant advancements across various natural language processing tasks. However, the immense size of LLMs and the growing demand for large-scale datasets present challenges in fine-tuning these models in resource-constrained environments. Federated learning (FL) has emerged as a promising solution, enabling collaborative model fine-tuning on distributed private data without requiring data sharing. Despite its potential, the heavy computational and communication burdens imposed by LLMs hinder the widespread adoption of FL-based fine-tuning. To mitigate these challenges, we propose FedSIT (Federated Split Importance-Based Tuning), a novel federated fine-tuning framework designed to optimize LLM training in environments with limited computational resources. FedSIT splits the pretrained model into Bottom, Trunk, and Top layers, offloading the computationally intensive Trunk layer to the server while distributing the Bottom and Top layers to client devices. Additionally, FedSIT leverages layer importance scores to selectively finetune the most critical layers, reducing the number of parameters to be fine-tuned. Our extensive experiments demonstrate that FedSIT achieves comparable performance to existing methods while significantly reducing resource requirements, offering an efficient and scalable solution for federated fine-tuning of LLMs in real-world settings.