Selecting another view (even of the same type) has a 2 seconds properties refresh delay.
As suggested a possible speed up might be creating property panels per type and just hide them and show them again when needed. (if this is not already the case?)