Sounds like you're hitting bug 394103
, which does appear to be related to the groundwork for the stuff I was wiffling about in my earlier comment. (fx: slaps self for not looking in bugzilla first)
It sounds like they've turned stuff on in FF3 to pay attention to the DPI setting reported by some part of the operating system, and scale stuff based on that value. From the look of things, this can lead to things going wonky on various flavours of Linux and/or Gnome. There are lots of comments in that bug talking about fiddling with DPI settings in X and various other areas of Linux config, which I can't follow at all. (I'm a Java/Webserver/CSS/HTML weenie - I know what X *is*, but that's about it, and my Linux experience is mostly from the other end of ssh, where X isn't really an issue)
After some brief fiddling here, I reckon that if you want to confine any tweaks to Firefox itself, and not clobber your entire desktop, try this:
(apologies for any grandmother-egg-sucking-education incidents that may occur in the following)
1. launch 2 firefox windows, side-by-side
2. get one of 'em (A) to a problem site
3. in the other (B) type "about:config" in the location bar, then confirm your way past the warning
4. still in window B, type "layout" in the filter box
5. look for "layout.css.dpi" (it might not be there)
6. if you can't see it, right click in the window somewhere and select "new->integer", then enter "layout.css.dpi" as a preference name, hit "ok" and go to step 8.
7. if it is there already, right click on it and select "modify"
8. enter a value approximating your actual screen dpi (probably somewhere between 80 and 150) and hit "ok".
9. observe effect in window A
10. (if the property wasn't there originally in step 6, you may need to tweak the "filter" value entered in step 4 to get it to show up at this point)
11. repeat steps 7-9 until you're happy. Or crank it up to 250 for super-ginormo-vision.