OBJECTIVE: Alzheimer's disease (AD) is highly heritable, and AD polygenic risk scores (AD-PRSs) have been derived from genome-wide association studies. However, the nature of genetic influences very early in the disease process is still not well known. Here we tested the hypothesis that an AD-PRSs would be associated with changes in episodic memory and executive function across late midlife in men who were cognitively unimpaired at their baseline midlife assessment.. METHOD: We examined 1168 men in the Vietnam Era Twin Study of Aging (VETSA) who were cognitively normal (CN) at their first of up to three assessments across 12 years (mean ages 56, 62, and 68). Latent growth models of episodic memory and executive function were based on 6-7 tests/subtests. AD-PRSs were based on Kunkle et al. (Nature Genetics, 51, 414-430, 2019), p < 5×10-8 threshold. RESULTS: AD-PRSs were correlated with linear slopes of change for both cognitive abilities. Men with higher AD-PRSs had steeper declines in both memory (r = -.19, 95% CI [-.35, -.03]) and executive functioning (r = -.27, 95% CI [-.49, -.05]). Associations appeared driven by a combination of APOE and non-APOE genetic influences. CONCLUSIONS: Memory is most characteristically impaired in AD, but executive functions are one of the first cognitive abilities to decline in midlife in normal aging. This study is among the first to demonstrate that this early decline also relates to AD genetic influences, even in men CN at baseline.