I’m in the process of wiring a home before moving in and getting excited about running 10g from my server to the computer. Then I see 25g gear isn’t that much more expensive so I might was well run at least one fiber line. But what kind of three node ceph monster will it take to make use of any of this bandwidth (plus run all my Proxmox VMs and LXCs in HA) and how much heat will I have to deal with. What’s your experience with high speed homelab NAS builds and the electric bill shock that comes later? Epyc 7002 series looks perfect but seems to idle high.
You must log in or register to comment.
You most likely won’t utilize these speeds in a home lab, but I understand why you want them. I do too. I settled for 2.5GBit because that was a sweet spot in terms of speed, cost and power draw. In total, I idle at about 60W for following systems:
- Lenovo M90q (i7 10700, 32GB, 3 x 1 TB SSD) running Proxmox, 15W idle
- Custom NAS (Ryzen 2400G, 16GB, 4x12TB HDD)v running Truenas (30W idle)
- Firewall (N5105, 8GB) running OPNsense (8W idle)
- FritzBox 6660 Cable, which functions as a glorified access point, 10W idle
With 25 GbE, even 10, I’d be tempted to PXE boot client systems. Maybe still have a local PCIe SSD for windows game files.
Dunno how that would actually work with Windows, but it was fun when I did it for beowulf nodes. Setting RPis to netboot is a little involved, but you can create an OSMC image and give all your TVs a consistent ‘smart’ interface. You don’t even need 10GbE to be pretty functional for the Pi, but my experience is that WiFi is not fast enough.
deleted by creator
I use a Ryzen 5900x, RTX 3080, 2x 10Gbit sfp+ NIC, 128GB ECC RAM, and only 2x 20TB drives at the moment.
For my gateway, I have an Intel N6005 box, I have a managed 2.5/10Gbit switch, and I have a wifi AP.
I have a ton of Proxmox VMs and containers.
All that hovers between 140W to 180W
The load on my UPS is around 100-140 watts. That includes my server, firewall, switch, starlink and a unifi access point. I would love to get that power consumption down. I only get 4-5 hours of runtime on battery. Also, the room it’s in is small and it gets really hot in the summer time.
5950x in an matx board with 15 x 3.5in drives 1 x sata sad 1 x optane u.2 drive (pulls like 10watts) 1 x Nvidia A2000 1 x Lsi 9305 16i 1 x 2.5gbe intel nic 3 x 140 mm fans at full tilt
Runs at like 120 watts at idle, like 220 watts with a good amount of work and peaks at like 320 watts if I make it do a lot of work
What in the world…
Dafuq you doing over there?
The last time I checked, mine runs at about 5-10 watts usually.
- Intel i7-3770
- 16gb DDR3
- 2 1TB SSDs
Are you sure. I was thinking those specs you would be more in the 50-80 watts range.
Yep, my homeserver spends most of it’s time idling, so power management kicks in.
Now when one of my build VMs are running, it’ll get up to that range, but that’s why I said it runs at 10 watts usually
I’ve got a 3 node Proxmox/ceph cluster with 10G, plus a separate Nas. They are all rack mount with dual PSU. Add in the necessary switching, and my average load is about 800w. Throw my desktop (also on 10G) into the mix and it runs 1.1kw.
That’s roughly $50-60 extra in electricity costs for me monthly.
Would be around 300€ in Germany, on a cheap contract. Limiting myself to one combined NAS/application server atm, with the others turned on only if I want to try sth out.
deleted by creator
Average load 800W is 0.8kW24h30d=576kWh/M
Which is over 172€ on a 30ct/kWh contract.
Wow! I’m paying 10.5¢/kWh for electricity at home here in the US; it’s a little below the national average but not dramatically.
Yeah, we pay a lot. We also got one of the lowest downtimes regarding electricity, on average approximately 10minutes per year…so that’s kind of a (small) advantage you get for the premium price
I ise about the same. But that is more due to the hardware I got being a bit older. 2 dell R710s 1 R510 and a custom build server. Everything is still 1g. In my case electricity is not a big deal due to solar. We produce much more then we can use our self.
I’m afraid of dumping 500+ watts into a (air conditioned) closet. How are you able to saturate the 10g? I had some idea that ceph speed is that of the slowest drive, so even SATA SSDs won’t fill the bucket. I imagine this is due to file redundancy not parity/striping spreading the data. I’d like to stick to lower power consumer gear but ceph looks CPU, RAM, and bandwidth (storage and network) hungry plus low latency.
I ran proxmox/ceph over 1GB on e-waste mini PCs and it was… unreliable. Now my NAS is my HA storage but I’m not thrilled to beat up QLC NAND for hobby VMs.
My 10G is far from saturated, but I do try and keep things using RAM where possible. I figure that with 100gb of DDR4 in my main server, that should be able to provide enough speed for a 10G link.
I’ve got ceph running on Intel Enterprise SSDs, so they are pretty quick.
I also tried running ceph on 1G. I found it unreliable as well.
I’m running my smart home entirely from a single NUC running proxmox with VMs and LXCs for my services. It’s pulling ~7W on average
Around 100 Watts for
- NAS with 4x3.5" HDD,
- Minisforum HM90 for Proxmox with 2x2.5" HDDs,
- 16 Port TP Link PoE Switch,
- TP Link router
- 2x Raspberry Pi 4b
But everything with gigabit speed. Doesnt need more at home
Minisforum HM90 for Proxmox with 2x2.5" HDDs
How did you connect HDDs to that thing?
Inside the bottom tray you have two cutouts in order to put two 2.5" HDDs/SSDs in
Well look at that, it sure does. Too bad they don’t make those anymore.
Thinkcenter tiny, 4 external HDDs, a DAC, a raspi3b+, was like 25W I think.
Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I’ve seen in this thread:
Fewer Letters More Letters AP WiFi Access Point DHCP Dynamic Host Configuration Protocol, automates assignment of IPs when connecting to a network DNS Domain Name Service/System HA Home Assistant automation software ~ High Availability LXC Linux Containers NAS Network-Attached Storage NUC Next Unit of Computing brand of Intel small computers NVMe Non-Volatile Memory Express interface for mass storage PCIe Peripheral Component Interconnect Express PSU Power Supply Unit PiHole Network-wide ad-blocker (DNS sinkhole) PoE Power over Ethernet RPi Raspberry Pi brand of SBC SATA Serial AT Attachment interface for mass storage SBC Single-Board Computer SSD Solid State Drive mass storage Unifi Ubiquiti WiFi hardware brand VPS Virtual Private Server (opposed to shared hosting)
[Thread #782 for this sub, first seen 4th Jun 2024, 04:35] [FAQ] [Full list] [Contact] [Source code]
About 30 watts for a old Lenovo Thinkcentre with a i5-6500T and 8 GB RAM in combination with a DAS and 2x2TB HDD’s. I’m currently waiting for parts for my new server I’m building, a small N100 Mini-ITX board with 4x4TB HDD’s that hopefully has a similar power consumption.
82.2W average for which I pay 144.6€/a at the moment. That’s for a Ryzen 7 3700X, some hard drives and SSDs and the fiber connection to my basement. I outsourced 90% of media consumption to a VPS though, that’s another 84€/a.
From the wall I’m pulling 120w
Ryzen 5700G
128GB ram
2tb + 4tb NVMe drive
2 x 20tb HDDs
Unifi Enterprise 24 PoE
Mikrotik RB5009
2 access points
3 cameras
Fiber runs cooler than copper all of my SFP+ are fiber.
I feel almost obliged to ask: what are you running on this monster of a setup?
Mostly for PiHole.
You know he’s just running docker.
- Fujitsu motherboard
- Intel pentium G5600
- 6 HDD (4 x 4 TB 2 x 8 TB) spinned down
- 2 SSD for proxmox
- 6 CT and no VM for now
it runs at 16W mostly idle
