I believe it happens because I'm not using curves, just polygonal chains.
That does sound plausible. I looked around in Inkscape preferences but it looks like it doesn't have an option to force Simplify to avoid introducing curves.
You can try messing with the simplify threshold through, found in File -> Preferences -> Misc -> Simplification Threshold. Although I'm not sure this will help you, it's worth trying.
Another thing you can do is look for another program to simplify the SVG. Inkscape probably isn't the best optimizer since that's not its primary focus. Just googling "svg optimizer" finds a whole bunch of promising results. I haven't tried any of them, but there's no harm in trying each one on a copy of your SVG and seeing if they help.
If you open your SVG in a text editor, you can get a feel for which objects are using the most space just by seeing which of them have the most data in them. If you're a programmer it's easy to write a tool that would list out how much space is being used by each object. If you e-mail me or post your SVG file somewhere I could probably write this. It might not help, but there's a chance that the problem objects aren't the ones you think they are, which would help you narrow your efforts.