From the XOR truth table, we can find that any bit that gets XORed with a 0 keeps its value, and any bit that gets XORed with 1 gets flipped.
Therefore to toggle the Xth bit, all we have to do is to evaluate N XOR (1 << X) where N is our input.
Since X is always 5 in this problem, we can hard code (1 << 5) to 0x20 and then all we have to do is return N XOR 0x20.