Systematic and portable optimisation of hybrid parallelism for large-scale distributed training

color:#002060;mso-ansi-language:EN-US">Modern AI services rely on large language models (LLMs) trained before deployment. Scaling these models often improves predictions, but it also grows model state, activations, communication, and computation beyond the local resources of one accelerator or one node. Frontier training therefore uses large accelerator clusters. A larger cluster, however, does not automatically produce useful model throughput: communication, pipeline waiting, fragmented memory, and imbalanced work can leave capacity unused. The central problem is therefore how to choose parallelisation strategies that make the model fit in memory while converting cluster capacity into realised training throughput. This thesis develops a systematic workflow for hybrid-parallel planning across this full stack. The workflow forms a methodology for reasoning about training plans before they are executed. It is realised by five planning methods that share a layer-level cost abstraction. This abstraction links model structure, hardware properties, and parallelism choices, and it supports four steps: (i) symbolic cost modelling, (ii) joint strategy search, (iii) re-planning across model changes, and (iv) targeted operator-level refinement inside a fixed plan. Together, these methods provide four connected capabilities: predictability, interoperability, evolvability, and adaptability. The result is a portable planning workflow for improving the efficiency of large-scale LLM training on accelerator clusters. mso-ansi-language:EN-US">