Upgrading Our Home Network for At&t 2.5 Gig Fiber
Well, life has been crazy and it has been way too long since the last post. Since then, we have moved, both taken new jobs, and the little guy keeps getting bigger. However, all that aside, there have been some major changes in our home internet service. The first being that we ditched Comcast when we moved and couldn't be happier. In the beginning, the At&t service was a little spotty, but we moved to a neighborhood that is still under construction and the houses on our street were not finished when we moved in. Since then, it has been absolutely solid and I cannot remember the last time it was down for even a minute. This was not the case with Comcast and we had down time at least once per week. Having said that, I hope I didn't just curse my internet service in the future.
Now that the everyone is caught up, let's get into it. If you have home or business internet service that is above 1gb, you will quickly find that your options for a router are severely limited. Before we moved, I dove head first into the Unifi ecosystem. I purchased the Unifi Dream Machine(UDM), and at the old house with Comcast service, it did a great job. It was able to support 80+ wifi devices and allowed us to reliably receive 900 mbit+ download speeds to all of the wired devices. For the first 4 months after we moved, we continued to use it with 3 additional Unifi Wifi 6 access points. By the way, the access points are the best I have even used and I cannot say enough good things about them. If you have a home network, the Wifi 6 Lite is plenty for daily use and is a definite upgrade if you have devices that support the new standard. Even though the UDM has a built in access point, I went ahead and bought 3 to more than cover our home, because I knew that I would need them later on. My research led me to only one viable option, and that was to run pfSense or OPNsense, I settled on pfSense, and run the Unifi controller in a docker container to manage the Wifi Access Points. Frankly, I felt as if I was getting to the edge of that the UDM was capable of even without needing it to support the increase in speed. Now, if you are looking for a single router to serve just a few devices in a small radius, there is a review over on Dong Knows (yea I know) that will get you where you are looking to go. Here's the link. If you are like me and you want something that will be more reliable, faster, and allow you to cover a larger area more reliably, keep reading.
The first thing that I needed was hardware with network cards that supported more than 1 gig speeds. I looked at the prebuilts from Netgate, but to get one that has multi-gigabit nics you would need at least the 4100. That piece of tech will set you back $599 with 4gb of ram and 16gb of storage with an Intel Atom CPU. While I understand this is probably very power efficient, I felt like I could do better with a few years old i5 or similar. For the record, I did try virtualizing pfSense and while it worked well, I just didn't like the idea of taking the network down every time I rebooted the desktop or home server. I chose a Dell Optiplex 7040 with an i5-6500 with 16gb of ram and I installed a 128gb nvme drive that I had from a previous laptop. Here's a search on ebay for similar. You can find one for between $100 and $120 depending on the day. In my search, I discovered that if I was willing to take a 500 gig spinning rust drive installed, the price was a little less. I don't know why, but I am now the proud owner of a 500 gig HGST spinning hard drive that will live in the back of a drawer until I find some use for it. Anyway, this particular Dell model comes with a 1 gig nic on the motherboard, so I needed to find something a little faster. Right now, the prices on 10 gig nics are just too high. I'm sure in about a year that will change, but I needed something now. So, I searched my two favorite places to cause damage to my credit card, Amazon and eBay, and found something to fit my needs. This dual 2.5Gbe (Realtek) nic for the pfSense box, this single 2.5Gbe (Realtek) nic for the home server, and this TRENDnet 2.5Gbe POE switch to connect the Unifi APs and most everything else. If I had it to do over again, I would probably buy a managed switch for the VLAN support, but for now this one is great and I can separate things with a few firewall rules. Now, I know that most people would recommend running Intel nics on pfSense, however, the price of intel nics was just too much for me personally, especially having no experience with building my own router. If you choose to go the route that I am in the process of outlining, you will need to install a driver for the dual 2.5Gbe nic on pfSense or OPNsense. I know that the later claims to support Realtek nics out of the box, but believe me, it does not support this particular, rtl8125, chipset. That being said, the fact that the Optiplex came with a 1 gig nic onboard came in super handy, in that I didn't have to figure out how to mount a usb device on freeBSD. The process of adding the driver is very simple. Here's a link to drivers and here's a link to a forum post on serverbuilds.net outlining the process of adding the driver. The post mentions using the drivers to solve some stability issues with Realtek nics, but in my case, the card simply didn't work without a proper driver. With the driver, this nic has been absolutely reliable 24/7 and under heavy load. I tried to break it with sustained high load from multiple clients and couldn't. In fact, knowing what I now know, there is no way I would spend the money on an Intel nic. In my opinion, it would just be wasting money that could be used elsewhere. Like more disk space for my media server. Also, with this setup, when I get ready to move to and the price comes down on 10Gbe gear, I won't feel quite as bad if it doesn't sell on eBay. However, if you are looking for a nic that works out of the box without needing to install any drivers, I have heard that this one, with the Intel i225 chipset, is great. Here are couple screenshots using iperf betweeen pfSense and my media server, as well as my desktop.
As you can see, the difference in speed using two Realtek NICs is negligible in a home network environment. Maybe, and that's a really big maybe, I might care about a difference of 200 kbit/sec in a business environment.
As of now, I have been running pfSense with the new 2.5Gbe gear at home for about a month. In that time period, I have 100% noticed the upgrade in speed between my devices and the media server, as well as on the desktop and using the web. Besides, if you're going to pay for the upgraded fiber service, you need to have the networking gear to be able to use it to the full potential. As of right now, At&t does offer 5 gig fiber service at our home, but I will wait until they bump that up to 10 before making any more upgrades. For now, 2.5 gig is more than enough to backup and share all of the "Linux ISOs" qBittorrent can handle.
If you are looking for a great setup video to walk you through setting up pfSense, Tom over at Lawrence Systems has the best of the best on YouTube. That playlist will walk you through getting started, all the way to setting up firewall rules and VLANs.
