[Assuming that you should not modify official libraries - they are created by Anywhere Software and are in the Libraries folder - but if you really want to do it, it is better to make a copy, modify it, change its name and put it in the additional libraries folder (/AdditionalLibs/B4X)]...
What's the proper way to go about modifying a B4XLib?
Just as simply as I suppose you did. Unzip the B4XLib file, add your new Public Sub to a code or class module, zip again the library and, if the IDE is open, refresh the Libraries Manager tab.
If you don't see the change you made, you probably haven't placed the B4XLib in the right folder. In the case of this DesignerUtils.b4xlib there are 3 copies of it, one for each platform, in the Libraries folder of each installation (but, again, it is much better that you create your own version as I wrote at the beginning of this post ).
It is also possible that you have put your modified DesignerUtils.b4xlib in the /AdditionalLibs/B4X folder; in this case in the Libraries Manager tab you would see only the original, internal one, which is prevalent.