Open the XML Editor (Shift+Ctrl+X), and then look through your objects until you find a "svg:g" node that also has a name that starts with Territory_. This is the one causing the problem. Select it and then right-click and select ungroup
When you select it in the XML editor, it will highlight it on the real map. Then you'll know which one it is and can right click it on the map to ungroup it.
After ungrouping you'll also need to clean it up. You'll have multiple objects -- one of which you probably want to be the territory (and give it an ID) and the others you can decide if you want to delete or merge with the previous ones.
And another question: the reason of this is the same border color of groups? Border colors are easier to change, so I just change the border colors of all territories if this is the reason.