it is not possible to select by group when multiple groups are assigned to one object/s.
Confirmed from what I can tell. If the active object is only in one group it automatically selects it's group. If the active object is in more then one group it pops up a list box so you can choose which group to select, but when you select a group it doesn't do anything.
I'll take some time to investigate and possibly create a patch.
Added a patch to fix the issue. Problem was comparing full ID name to just the name of the group, which didn't match so it couldn't find the group.
There are still some other things to maybe work on in this area, such as if you use Shift+G then select group but cancel selecting a group it deselects everything. The "Select Same Group" operator isn't really useful by itself from the interface because it requires a group name rna prop but doesn't provide a popup to select a group, so it really is only used through the Shift+G operator, which has a function to provide a popup.
But none of those things are really related to simply fixing this so it at least works.