The clathrin-associated, heterotetrameric adaptor protein (AP) complexes, AP-1, AP-2, and AP-3, recognize signals in the cytosolic domains of transmembrane proteins, leading to their sorting to endosomes, lysosomes, lysosome-related organelles, and/or the basolateral membrane of polarized epithelial cells. One type of signal, referred to as "dileucine-based," fits the consensus motif (D/E)XXXL(L/I). Previous biochemical analyses showed that (D/E)XXXL(L/I) signals bind to a combination of two subunits of each AP complex, namely the AP-1 γ-σ1, AP-2 α-σ2, and AP-3 δ-σ3 hemicomplexes, and structural studies revealed that an imperfect variant of this motif lacking the (D/E) residue binds to a site straddling the interface of α and σ2. Herein, we report mutational and binding analyses showing that canonical (D/E)XXXL(L/I) signals bind to this same site on AP-2, and to similar sites on AP-1 and AP-3. The strength and amino acid requirements of different interactions depend on the specific signals and AP complexes involved. We also demonstrate the occurrence of diverse AP-1 heterotetramers by combinatorial assembly of various γ and σ1 subunit isoforms encoded by different genes. These AP-1 variants bind (D/E)XXXL(L/I) signals with marked preferences for certain sequences, implying that they are not functionally equivalent. Our results thus demonstrate that different AP complexes share a conserved binding site for (D/E)XXXL(L/I) signals. However, the characteristics of the binding site on each complex vary, providing for the specific recognition of a diverse repertoire of (D/E)XXXL(L/I) signals.