Protein tyrosine phosphatases (PTPs) are emerging drug targets for many diseases, including type 2 diabetes, obesity, and cancer. However, a high degree of structural similarity between the catalytic domains of these enzymes has made the development of selective pharmacological inhibitors an enormous challenge. Our previous research uncovered two unfunctionalized terpenoid inhibitors that selectively inhibit PTP1B over TCPTP, two PTPs with high sequence conservation. Here, we use molecular modeling with experimental validation to study the molecular basis of this unusual selectivity. Molecular dynamics (MD) simulations indicate that PTP1B and TCPTP contain a conserved h-bond network that connects the active site to a distal allosteric pocket; this network stabilizes the closed conformation of the catalytically influential WPD loop, which it links to the L-11 loop and α 3 and α 7 helices-the C-terminal side of the catalytic domain. Terpenoid binding to either of two proximal allosteric sites-an α site and a β site-can disrupt the allosteric network. Interestingly, binding to the α site forms a stable complex with only PTP1B; in TCPTP, where two charged residues disfavor binding at the α site, the terpenoids bind to the β site, which is conserved between the two proteins. Our findings indicate that minor amino acid differences at the poorly conserved α site enable selective binding, a property that might be enhanced with chemical elaboration, and illustrate, more broadly, how minor differences in the conservation of neighboring-yet functionally similar-allosteric sites can have very different implications for inhibitor selectivity.