George Allen is set to relaunch his political career this week, returning from several years in the private sector working for oil and gas interests to run for the Senate seat he lost in 2006.

Several years later, the popular narrative of that election has been condensed into one word: "macaca." Allen, seemingly unbeatable and already eyeing a presidential campaign in 2008 —some even considered him a frontrunner—called an Indian-American volunteer for his opponent a "macaca" at a rally and welcomed him to "America and the real world of Virginia." He subsequently plummeted in the polls, flailed about for the rest of his campaign looking to regain momentum, and ended up losing by a heartbreakingly narrow margin.

Certainly the "macaca" moment, a term that has become a synonym for politicians' racial gaffes in general, set Allen down the path to defeat. But it didn't just appear out of nowhere and it's worth remembering just what Allen has to answer for as he returns to public life. While "macaca" launched Allen onto the national stage for many Americans, he already was on probation with Virginia voters for a number of previous racial flaps. A major feature in The New Republican months before that incident documented his long interest in the Confederate flag, which he kept in his home as part of a collection and wore on a pin in his high school graduation picture as part of a self-described "rebel" image (the school was in California).

Then there were his Civil War proclamations. You may recall that recently Virginia Governor Bob McDonnell apologized—with Allen's support—for issuing a proclamation acknowledging Confederate history without mentioning slavery. But Allen made this an annual ritual as governor himself, referring to the Civil War in his own statements as a “four-year struggle for independence and sovereign rights” and asking Virginians to acknowledge "the honorable sacrifices of Confederate leaders, soldiers and citizens to the cause of liberty." His language didn't go unnoticed at the time either, attracting plenty of criticism from civil rights groups at the time. In his recent book, Allen apologized, but also accused his political opponents of exploiting the proclamations to attack him. Still, they “could have been worded better, I suppose," he told The Daily Beast last year.

It's important to note that these weren't significant issues only in retrospect after "macaca" —plenty of articles before the incident described Allen's race issues as one of his top liabilities and he was already trying to repair the damage with public apologies and efforts to reach out to the African-American community. That Allen was clearly aware of the reputation he had cultivated for racial insensitivity and was actively on his best behavior is what made the "macaca" moment and subsequent flare-ups—which included a strange and defensive response to news of his mother's Jewish ancestry and claims he used the n-word frequently in college—so irreparably damaging. It gave the impression that Allen's race problem was so deeply entrenched in his political persona that he simply couldn't let it go and still be George Allen.

Many conservative commentators fretted after Harry Reid was quoted in a book gushing over Obama's lack of "Negro dialect" that Democrats were hypocritical for not forcing him out given their condemnation of Allen. But, fairly or unfairly, individual gaffes only do real damage when they fit an established pattern and Reid had no history of inflammatory racial remarks—even the one quote came in the context of proudly supporting a colleague's bid to become the first African-American president. Backstory matters, and before Allen can move past his one "macaca moment" he needs to convince voters to forgive his several "Confederate decades" as well.