Probabilistic clustering based on mixture models is widely used in biomedical studies. With the increasing availability of multi-site data, joint analyses across multiple sites offer opportunities to improve the efficiency and generalizability of clustering methods. However, data-sharing restrictions and between-site heterogeneity present significant challenges for multi-site probabilistic clustering. To address between-site heterogeneity, we propose a heterogeneous mixture model in which different sites share the same mixture components but have site-specific mixing proportions. This structure accounts for site-level heterogeneities while leveraging shared information across sites. Due to data-sharing restrictions, distributed inference of this heterogeneous mixture model is challenging. One potential approach is the use of distributed Expectation-Maximization (EM) algorithms for federated maximum likelihood estimation (MLE). Nevertheless, the existing distributed EM algorithm requires cross-site communication in every iteration, leading to considerable communication overhead in terms of personnel and time. In this talk, we will introduce a novel one-shot distributed learning algorithm for communication-efficient inference of the heterogeneous mixture model. We provide theoretical guarantee that our approach achieves the full-sample efficiency via only one single round of cross-site communication. The effectiveness of the approach are demonstrated through an application to multi-site electronic health records (EHR) data, illustrating how they enable efficient probabilistic clustering and facilitate targeted interventions.