First, regarding one's ethnic or racial identity as some sort of unbearable burden sets one up for shame and distortion. To regard everything white and male as toxic is to fall into the same sort of binary and rigid thinking so often practiced toward people of color. All cultures and societies have much to be ashamed and to be proud of. In terms of the latter, I think of my father's ferocious work ethic and commitment to hospitality. If I simply write him and people like him off, I do violence to the complexity and the richness, the goodness in his life and the considerable gifts that he left me as he struggled to give me a better childhood than his father had been able to give him.
Second, divesting oneself of maleness or whiteness is impossible. Claiming that sort of renunciation is, in my opinion, cheap and theatrical. I could, as a Portland participant in a "World Have Your Say" radio forum on race put it some years ago, say that I am "willing to drop my whiteness." But I would still have all the benefits that whiteness had brought me during my lifetime and before. I would still often be viewed by students and others as the "smartest person in the room" simply based on my appearance. Pretending that I have not been handed a bunch of privileges would simply be an attempt to deny my advantages.
Third, many people of color and white women would be delighted to have the sort of power that white men enjoy. Rather than trying to divest ourselves of it, to somehow lose our whiteness or our gender, why not instead use that power for good, like a superhero? Simply pointing out that we have accrued privileges we haven't earned could quality as a radical act. Of course part of what we should be advocating for are opportunities for people of color and white women to speak. But refusing to speak or to act ourselves simply because we have enjoyed undeserved privileges seems more self-indulgent than progressive.