Multimeric cargo adaptors such as AP2 play central roles in intracellular membrane trafficking. We recently discovered that the assembly of the AP2 adaptor complex, a key player in clathrin-mediated endocytosis, is a highly organized process controlled by alpha- and gamma-adaptin-binding protein (AAGAB, also known as p34). In this study, we demonstrate that besides AP2, AAGAB also regulates the assembly of AP1, a cargo adaptor involved in clathrin-mediated transport between the trans-Golgi network and the endosome. However, AAGAB is not involved in the formation of other adaptor complexes, including AP3. AAGAB promotes AP1 assembly by binding and stabilizing the γ and σ subunits of AP1, and its mutation abolishes AP1 assembly and disrupts AP1-mediated cargo trafficking. Comparative proteomic analyses indicate that AAGAB mutation massively alters surface protein homeostasis, and its loss-of-function phenotypes reflect the synergistic effects of AP1 and AP2 deficiency. Taken together, these findings establish AAGAB as an assembly chaperone for both AP1 and AP2 adaptors and pave the way for understanding the pathogenesis of AAGAB-linked diseases.