Everywhere I go I read that the territories need to be at least 20x20 px in Inkscape so they can fit the 2 digit numbers. But in the editor it doesn't fit and I need to make the territories at least 70 px to fit the numbers.
Does anyone know what causes this and how to fix it?

Use the magnifying tool on the editor then the numbers should fit. The bigger the map is, the more you’ll need to zoom in on the map.

Is scale set to 1? Check document properties in Inkscape

Yes scale is 1.
This is what a 20x20 looks like all the way zoomed in with example numbers on.
https://imgur.com/gallery/dvoy8kr