I mentioned recently, in my post "A FEW BRIEF NOTES ON SPACETIME IN PALE FIRE," that "twice removed" might in part be a joke about John and Sybil Shade being related. Two pertinent lines:
"[...] and you would be
In your own study, twice removed from me,”
(Lines 365-366)
“John Shade’s wife [...] was a few months his senior. I understand she came of Canadian stock, as did Shade’s maternal grandmother (a first cousin of Sybil’s grandfather, if I am not greatly mistaken).”
(From note to line 247)