I gave big images a go here, tried creating larger and larger ones until something broke:
18600 x 18600 x 24 bit color = at least 1,037,880,000 bytes, or perhaps one-third more if is RGBA.
Then I looked to see if the Java heap could be made larger, and sure enough somebody suggested giving #VirtualMachineArgs a burl:
so I did, and that was the last I heard from my computer for about 8 minutes. It sank to its knees and I was starting to worry.
Anyway, when it blinked I managed to capture this:
and then when it properly regained consciousness I had this:
both of which suggest that the maximum object size, for the JVM on my machine at least, is limited by the maximum value of Java's default signed 32-bit Integer, so the maximum image size would be like 23000 RGBA pixels square.