From your description and your layout, the only thing that I can think of is that your panel is hidden firstly and secondly, the placement of your labels and buttons is problematic.
If you are using your panel to "switch on/off" the view then make sure that your labels and button are attached to the panel. You can check this in your Designer by clicking eg. on your button and checking the parent.
Having checked your project whilst writing this post, this is exactly where your problem lies.
Your project shows that the items have the "Activity" as parent - select the item first (eg. your label) - then just click on the pulldown, choose the panel name and attach to it like that - you might need to re-align your item because it is now placed relative to it's parent.
This should solve your problem.