Vascular contributions to dementia and Alzheimer’s disease are increasingly recognized^{, , , , –}. Recent studies have suggested that breakdown of the blood–brain barrier (BBB) is an early biomarker of human cognitive dysfunction, including the early clinical stages of Alzheimer’s disease^{,, –}. The E4 variant of apolipoprotein E (APOE4), the main susceptibility gene for Alzheimer’s disease^{, , –}, leads to accelerated breakdown of the BBB and degeneration of brain capillary pericytes^{, , , –}, which maintain BBB integrity^{, –}. It is unclear, however, whether the cerebrovascular effects of APOE4 contribute to cognitive impairment. Here we show that individuals bearing APOE4 (with the ε3/ε4 or ε4/ε4 alleles) are distinguished from those without APOE4 (ε3/ε3) by breakdown of the BBB in the hippocampus and medial temporal lobe. This finding is apparent in cognitively unimpaired APOE4 carriers and more severe in those with cognitive impairment, but is not related to amyloid-β or tau pathology measured in cerebrospinal fluid or by positron emission tomography. High baseline levels of the BBB pericyte injury biomarker soluble PDGFRβ^, in the cerebrospinal fluid predicted future cognitive decline in APOE4 carriers but not in non-carriers, even after controlling for amyloid-β and tau status, and were correlated with increased activity of the BBB-degrading cyclophilin A-matrix metalloproteinase-9 pathway in cerebrospinal fluid. Our findings suggest that breakdown of the BBB contributes to APOE4-associated cognitive decline independently of Alzheimer’s disease pathology, and might be a therapeutic target in APOE4 carriers.