In our aging society, diseases in the elderly come more and more into focus. An important issue in research is Mild Cognitive Impairment (MCI) and Alzheimer's Disease (AD) with their causes, diagnosis, treatment, and disease prediction. We applied the Brain Age Gap Estimation (BrainAGE) method to examine the impact of the Apolipoprotein E (APOE) genotype on structural brain aging, utilizing longitudinal magnetic resonance image (MRI) data of 405 subjects from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. We tested for differences in neuroanatomical aging between carrier and non-carrier of APOE ε4 within the diagnostic groups and for longitudinal changes in individual brain aging during about three years follow-up. We further examined whether a combination of BrainAGE and APOE status could improve prediction accuracy of conversion to AD in MCI patients. The influence of the APOE status on conversion from MCI to AD was analyzed within all allelic subgroups as well as for ε4 carriers and non-carriers. The BrainAGE scores differed significantly between normal controls, stable MCI (sMCI) and progressive MCI (pMCI) as well as AD patients. Differences in BrainAGE changing rates over time were observed for APOE ε4 carrier status as well as in the pMCI and AD groups. At baseline and during follow-up, BrainAGE scores correlated significantly with neuropsychological test scores in APOE ε4 carriers and non-carriers, especially in pMCI and AD patients. Prediction of conversion was most accurate using the BrainAGE score as compared to neuropsychological test scores, even when the patient's APOE status was unknown. For assessing the individual risk of coming down with AD as well as predicting conversion from MCI to AD, the BrainAGE method proves to be a useful and accurate tool even if the information of the patient's APOE status is missing.