Fine-tuned a local LLM instead of paying for API calls, saved $180 this month

I run a small side gig analyzing product reviews for local shops. Was using GPT-4 API and the bill hit $214 in June. Decided to try fine-tuning Llama 3 on my own data using a used RTX 3090 I got off Facebook Marketplace for $600. Spent three weekends learning LoRA and stuff. First run was garbage, kept repeating the same phrases. Second attempt with better cleaning of my training data actually worked. Now I run it locally for most jobs. Still hit the API for hard cases but my July bill was $34. The tradeoff is setup time and having to troubleshoot my own GPU drivers at 2am. Has anyone else made the jump to self-hosting for cost reasons? Wondering if bigger fine-tunes are worth the hardware upgrade.

1 comments

1 Comment

hannahcraig21d ago

Saw a similar story on a self-hosted AI subreddit last week, some guy running a legal doc summarizer off a pair of old 2080s. He said the same thing about midnight driver fights. The $180 gap is real though, that covers a lot of troubleshooting. Biggest thing I've heard is the training data quality matters way more than the model size, so maybe stick with the 3090 before chasing a bigger card. Also heard used 3090s are still a steal, so you did fine there.