Beginner - Help with self hosted LLM build

[u/crpt_conch]

Looking into self-hosting a LLM. I have an elitedesk mini G9 with 32gb ram and an intel i7 that I want to add a eGPU to and I’m trying to decide what GPU/setup is the right one.

I see that Newegg is selling an Aorus box with an RTX 5070 Ti 16gb for about 1000 bucks. Is this worth the deal? I’m trying to budget for this amount exactly and this deal seems too good to be true. But I figure, even if I buy a used GPU, the cost of the additional components would take to my budget limit and beyond anyway. I am an absolutely beginner at this home lab stuff. Just need some help.

I was recommended a V100 but I’ve read that they require a fan that will sound like a lawn mower/vacuum to prevent overheating and I don’t want that in my house

[u/asimovs-auditor]

Expand the replies to this comment to learn how AI was used in this post/project.

[u/WranglerTop5624]

I’d be careful with those Aorus boxes, the cooling on the enclosures is pretty middling and you’re stuck with whatever PSU they slapped in there

For a beginner setup an RTX 3060 12gb used would leave you with cash left over and it’s way easier to get running properly than trying to make a V100 behave in a mini PC

[u/Weekly-Offer-4172]

First thing to check before buying anything: whether that EliteDesk actually has Thunderbolt. The Aorus eGPU enclosures only connect over Thunderbolt 3/4, and a lot of 1L minis ship with a plain USB-C port - if yours is one of those, an eGPU isn't on the table at all. Even when it works, Thunderbolt gives you effectively PCIe 3.0 x4, so a 5070 Ti takes a noticeable haircut versus sitting in a real slot. For LLMs specifically, VRAM capacity matters more than raw speed: a used RTX 3090 with 24GB is the usual budget pick because 24GB is where the larger quantized models start fitting, and used ones land well under your thousand-dollar budget even after adding an enclosure. You read right about the V100 too - they're passive-cooled datacenter cards that expect screaming chassis fans, have no display outputs, and only do FP16, so no BF16 or the newer quant formats. Fine in a rack, miserable on a desk.

[u/crpt_conch → u/Weekly-Offer-4172]

Where can I find a 3090 at that price point? Seems like everything is double that price when I look online

[u/crpt_conch → u/Weekly-Offer-4172]

I mentioned to someone earlier that it has an additional m.2 slot that I’ve seen people connect a pcie-to-m.2 cable to. Maybe that would be a viable option. It’s sounding like I might as well go with a considerably lower tier GPU like a 3060. I could see about getting a sonnet box or a razor box. Any thoughts?

[u/anonymousart3]

Disclaimer: I'm still pretty new to LLMs myself.

Have you done any setup with an LLM yet?

If it were me, that's where I would start. Get an ollama install going, and I've been using openwebui to actually interact with the models.

You can get a pretty low end GPU (by today's standards that is), and learn about how LLMs work on it. Dolphin is supposed to be a good one to start with.

I personally jumped in with gemma3:4b. Not exactly great to start with I think, but...I learned a lot as a result of trying to use it.

I have a gtx 1060, 6gb VRAM, card. Gemma3 fit on that card, so I was able to feel how quick a GPU accelerated model actually was. Now I'm trying out some bigger models, like Qwen3 7b, and I'm seeing how much slower CPU spillover is. It's really wild to see/feel the difference.

You can play around with that as you learn more about what hardware you actually want, and look for deals.

There's a huge issue right now with everyone rushing to get hardware that runs AI better, so I don't think it's a bad idea to play around with weaker hardware for now, learning how the AI stuff works, and waiting to get some used hardware that can still run AI, but... Not as well.

If I had the money, I'd definitely be upgrading to a 5070, but... That's more because it's just faster. I actually want a GPU with much more vram so that I could run actually bigger models without it spilling over to CPU... But, yeah.

[u/crpt_conch → u/anonymousart3]

I appreciate your point. I’ve been considering an RTX 3060 for this reason as a starting point. It’s inexpensive compared to other options I’ve see and can reasonably handle a lower B LLM. But there’s a lot to consider with power consumption compared to newer models. It’s honeslty why I splurged a bit to get a G9 rather than a cheaper G6. That might be unreasonable but in the long run, I hope the extra expense will save me on my electric bill

[u/Anshu6666]

for a first self-hosted LLM start with llama.cpp + a quantized 7B (q4_k_m) on CPU - it runs on 8GB RAM and teaches you the moving parts (context window, prompt template, sampler) before you buy a GPU. skip ollama; it hides the config you need to learn.

[u/RiceEvening4211]

The real question is what size models you want to run, because this whole decision is a VRAM math problem.

16GB (5070 Ti) = comfortable up to ~30B models at Q4 quantization, tight beyond that. 32GB (V100) = 70B at Q4. So the V100 literally doubles your headroom for the same money, on paper.

The V100 catches, since you asked about noise: it’s a blower card designed for server airflow, so yes, in a room it will be loud under load, it pulls ~250W, and it’s a compute-only card (no display out, which is fine for headless inference but worth knowing). It’s also old enough that you’re buying used with no warranty. The lawn mower fear is directionally right.

The eGPU part: Thunderbolt bandwidth is mostly a non-issue for inference. Weights load onto the card once, then it’s all compute. You lose a bit vs internal PCIe, but not enough to change the decision.

My actual advice: if your goal is “run the best open models I can,” the V100’s 32GB wins and you accept the noise (or set a fan curve and put the box in another room). If your goal is “a quiet box on my desk that runs 30B-class models well,” the 5070 Ti is the sane pick and the eGPU tax is worth it for your mini. Don’t buy the 5070 Ti expecting to run 70B models, that’s where people get disappointed.

[u/crpt_conch → u/RiceEvening4211]

Thanks for this breakdown. I’m definitely leaning away from the V100 at this point.

What could I accomplish with a 30B model? Is there a relatable metric that you can use to describe what I’d get? I understand quantization a bit but still trying to grasp the parameters part of it

[u/ethansky → u/crpt_conch]

Don't bother with any of the other comments. They're all from AI spam bots with outdated info because this sub is barely moderated. You're better off going to r/LocalLLaMA as they're dedicated to local hosting LLMs.

Honestly, the only two use cases for self-hosting an LLM in Oct 2026 is either absolute privacy at the price of capability or a true love for hardware/software tinkering, regardless of the cost. The hardware to run LLMs at home has exploded in price over the last year, from memory to GPUs to storage. The amount you would have to spend in components to run a decent model at decent speeds could pay for years of frontier AI lab subscriptions.

If you really want to self host, 32GB of vram is probably the ideal starting point to run solid dense models like Qwen 3.8 27B and Gemma 4 31B at q4 with decent context. If you get your hands on more ram, that could open up bigger MoE models like Qwen3.8 flash next by offloading experts to RAM. So the common GPU setups would be 2x 5060 ti 16GB, 2x 5090, 2x 3090, etc. You could go with older cards like the V100, but between power, noise, mediocre performance, and it soon being EOL, it'd be at the bottom of my recommendation list.

[u/ethansky → u/RiceEvening4211]

AI spam bot ^

[u/Schwitters]

Don’t shy away from at least looking into Intel. Their arc pro b70 will give you 32gb vram. There is ASRock build on Newegg for $1300. 16gb just leaves you capable for a lot of things but the open weights models are getting better and better, bigger and bigger. You’ll regret not grabbing more vram if you actually get into the LLM thing.

[u/ethansky → u/Schwitters]

How is the sycl backend these days? I kinda regret not picking some up while they were still $950, but the lackluster software support at the time pushed me towards 2x 5060 ti 16GB.

[u/Schwitters → u/ethansky]

It’s really only lacking when you get into really optimizing models. There are some efficiency losses when quantizing context that don’t happen on cuda. It also doesn’t get the full bang out of things like MTP. I see like 35 tps with a Q4 Qwen3.8 35B MoE, or 10 tps with a IQ3 Qwen 27B dense. Don’t really use either of those though.

All that to say I literally run the models I need to for writing/editing, scrubbing ads from podcasts, and text to image. I use an $900 beelink mini pc with an i9 32gb ram and the arc pro b50 via the beelink EX dock. I’m more than happy for what I’ve spent, how energy efficient it is, and the amount of work it’s done for me.

[u/ComprehensiveShake76]

First thing is the one someone already raised: check the exact G9 model for a Thunderbolt port. A plain USB C port won't drive an eGPU at all.

If it has one, VRAM is the number that matters for local LLMs, and 16GB on a 5070 Ti is a solid amount with good speed for models that fit fully on the card. The Thunderbolt link mostly slows loading the model, not generating once it's loaded, so it hurts less than people expect. If the bundle eats the whole budget, a separate enclosure plus a used 3090 with 24GB is worth pricing, since more VRAM means bigger models, though it draws more power and runs louder.

[u/sharebeam]

One thing on the m.2 slot idea: those m.2-to-PCIe adapters do work, but go in knowing the caveats. You'll be running at PCIe Gen3 x4-ish speeds, which is honestly fine for inference since the weights just sit in VRAM, but you lose the clean enclosure and cooling of the Aorus box and need a way to power the card. If you go that route, an RTX 3060 12GB is still the sweet spot - cheap used, sips power compared to bigger cards, and 12GB VRAM is enough for 20B-class models at Q4. Keep the G9 stock as your 24/7 box and treat the GPU as an add-on.