つい最近 is more colloquial and often implies 'just the other day'; ごく最近 is slightly more formal and emphasizes a very short time ago.
Standard spelling; ごく is usually written in kana.
Kanji spelling for ごく; less common and may be confused with other readings.
Compound of ごく (極, 'extremely') and 最近 (さいきん, 'recently'). The exact historical derivation is straightforward.