I'm using a 2012 mini (see specs here:
http://www.everymac.com/systems/app...-mini-core-i7-2.3-late-2012-server-specs.html). As per the spec, the processor in this box supports "dual channel" memory. This is only active with two memory modules (in this case, since that is all this mini supports) that match. The speed gain of dual channel configuration may be debatable (of the cuff, 6-8%?), so this is the only thing you would be loosing out on when having only one memory module or different sized modules (if your system/processor supports it to begin with).
Originally, this box came with 4GB of RAM and 2 x 1TB 7200rpm hard drives. This configuration felt dog slow. After going to a 256GB SSD for the boot drive and 16 Gigs of memory, it works like a charm. The funny part is that I use the mini more on the virtualized Windows (10) side (via Parallels), then the OSX side. I'm using a triple display 1080p setup (2 native, 1 via USB) and love it. I still can tell that I'm virtualized when using B4X (be it a, i, J, not using R yet), since I can out-type the autocomplete (it's actually a little slow in this configuration).