My font file was about 1.2mb, I ended up removing the need for it from the home page by designing around it. But I wanted to figure out a way to reduce the file size. One option was to remove the characters I didn't need from the file itself. That sort of worked.
For some reason, the font wasn't being compressed. Although ttf was listed in mime.types (but for some reason it still wasn't being compressed or cached).
I made a small modification to /etc/nginx/mime.types to cover every sort of font I could use. You may already have these, or just some of these declared in your mime.types:
The font being served is now down to 389kb, which has brought my home page down to less than 1.5mb. I hope this might be useful to someone else that is facing the same problem.