Are you using version 1 or 2?
In v2, you can set the font size as part of the contour layer. Open up the Layers panel and choose the required contour layer. The edit options will be shown and you can choose a font size here (and also toggle label visibility). Relative font size is also affected by the overall resolution of the exported image - all else being equal, higher resolution images will have a smaller relative label size as the font size remains the same.