@Prydzen In 5/10 years that number will be way higher. 4K used to be very rare in households 10 years ago and crazy expensive like 8K today. And now in 2022 you can get a cheap 4k Tv voor $400/500 this was impossible 10 years ago.
In the grim darkness of 2025, Linus tech tips finds itself barely recognizable as thousands of servers coat their buildings like a 2 meter thick layer of armor, shielding the world from the madman at it's heart. Inside the asylum, a monument to the sins of Linus' technology addiction, he slaves away streaming 24/7 as he tries in vain to "save the universe" in sever project after server project. His employees are wired directly into everything as little more than motherboards waiting to be dropped by Linus. As time goes by, Linus will himself finally become the master of the internet as he wires himself into his temple of tech tips, continually giving tips about tech so arcane that they are lost upon the ears of the masses. In the year 2025, Linus tech tips becomes unknowable, unrecognizable, feared, and worshipped.
Should they ever have that many servers, one they would be in some sort of business relation with ServeTheHome, and two their location would be called a datacenter not a studio.
These videos are truly genius. As a company they need to upgrade things, so they go way overboard and it makes an awesome video showing some unique tech configs we wouldn't otherwise see. They get sponsors to send in components for free, sponsorspots sold on the video and ad-revenue from the video it self. The epitome of efficiency.
Linus, I am local (Burnaby) and would love the help do your storage management. Get in touch I have my FreeNAAS home server but no backup due to money. But I can make your system drastically reduce the chances of data loss. Get in touch
I am working in the IT department of a bigger company. For people like us it is always interesting what other companies are using and how they are using it. So it has a use for a lot of people 😉
What do you do that's not overkill? Well, almost everything I'd say, and I'm a bit of an expert on the subject :) I say this with love, but LTT never fails to make me cringe by how absolutely wrong they're doing an expensive project in order to avoid spending additional money. Even after your sh** broke and you've gotten to this point, you're still doing SATA drives and ZFS instead of SAS with Ceph. The sadist in me looks forward to watching you have to deal with drive failures and subsequent replacements in the future!
It's very cool but from what I've heard LTT has like 50 employees. This is like 26 Terabytes per person if you are doing a 1-to-1 backup (and not all employees will utilize this fully). Perhaps LTT has a larger data inefficiency/clean-up issue? A media company of this size should not/does not need this much storage. It's definitely cool but it seems like for the past year or so we've been getting server upgrades pretty often. It seems like a bandaid solution to a larger problem IMO. It may be in LTT's best interest moving forward to hire someone to establish standardized procedures around data retention and general operational efficiency. If these things weren't sponsored like they are LTT would be breaking the bank just trying to keep up with its data problem.
you are aware that they keep all old video footage for archival purposes, right? A media company this large, i'm actually surprised they're *only* using those Petabytes. yes, they could delete their old footage, and to be honest, for archival really going to tape would be cheaper and in their best interest, but they've not got a data retention problem, they just have a lot of data that is considered to be relatively valuable to LTT
Installing these in Lab 2 is quite an ingenious idea considering one of Lab 2's key features is its insane power infrastructure. Then just run 40Gbit or something fiber back to Lab 1 :D
Finally, they are doing it right, I've seen so many of their setups and thought, well if that works best for you and the team, then cool. But I always thought in the back of my head, that they really should separate the storage and compute and have the storage compatible to where you can swap out the compute unit and continue to roll without downtime. Your storage is starting to look strong now LTT, sweet !! ❤️
Going from something like glusterfs to what's effectively just a single file server really isn't an upgrade. However if downtime isn't a particular issue and there's no need for clustering etc then it's an ok solution that's much easier to manage, and sometimes that's more important than how good the tech is.
i love how linus is the owner of this business, but acts like an employee that dgaf when handling the extremely expensive equipment he comes across while making videos. funny watching the employees actually cringe while the owner well acts like he dgaf lol.
Next year, I'm expecting to see the same thing, in flash storage 🤪 The crazy thing is, you could put more storage per unit right now. It would just cost about 30x more lol
@Cheesy Duck so what I'm hearing is "we bought the entire block" video then they just move to a bigger building turn existing HQ into a storage farm with all the backup servers and employ full building water-cooling 🤣
I'm expecting by end of 2030 Linus will rip out the server room and just install a Very super computer or maybe one of the first Quantum cpu based servers with petabyte SSDs.
Finally, they are doing it. Hope this solution works for you. I've been using a slightly more refined version of this and it's been working OK. Really glad you are finally considering merging the vault and wannick together. Once you've set that up correctly it should be much better/easier to manage.
To be fair that's a bit like saying that the courier company down the road goes through more tyres than the local racing team in a year. The use cases that an ISP has for storage solutions is completely different to these guys. The amount of video content they produce (and need multiple people working on) is pretty full on. A better comparison is to look at media production companies, and let me tell you: those guys really REALLY pour through drives. Especially for their achival stuff now that tape backups are actually more expensive than hard drives are now.
i love watching the server videos these guys make cause it got me interested in doing a career in servers. Currently in taking my classes now, so exciting.
23:18 - I remember having my mind absolutely blown with the NetApp E5600A and DE6600’s years ago that had this same feature. Not only the density at the time (60 drives in 4U), but also the drawer that was still in operation when pulled was insane. Also had optical LED channels from each individual drive interface board going to the front panel. So cool.
What boggles my mind is that despite it being a known issue for years and despite spending so much money on so many other projects... you guys still haven't hired a full time system administrator?
If their gross turnover is a couple mil a year, wages a mil, other costs another mil or so, why would they spend 50 to 80k gross (benefits and tax etc) on another employee, when they dont need a SA? Heck, it would be crap, but they could reclaim old footage off YT, a SA shouldnt be their priority when they are too small to need it and alot of these guys have familiarity with system administration.
@bigpod so you've got no idea what it takes to be a sysadmin. Got it, thanks for confirming. You like Linus, we get it. That doesn't make him immune to criticism. He's not a sysadmin and his hacked together solutions based on his sponsorships and what he can make videos on doesn't change that.
Connect that sm drives directly to the APC battery backup and do (9) disk vdev array of zfs3 aka raid-z3. 10 vdevs Oracle has a good white paper of optimum zfs. Also don't use windows use proxmox. Zpool1, Zpool2,Zpool3,...Zpool9
It would be awesome if you did a video on how you organize ~2PB worth of files. It seems everyone has their own scheme, and to see what you all have come up with would be useful.
you know, jake is awsome, he seems like a guy ur telling how u wanna rob a bank and suddenly he is more into it than you were before. i like his understatement energy. calm, but with FORCE.
One day it’d be cool if you guys made a video covering ALL the servers you guys have ever had dating back to the original. Covering all the names, capacity, reason for upgrading, all that stuff, that’d be awesome!
Relative to the size of the system, it's not that much power since it's literally just spinning up disks plus minimal power for the SAS IO module and expanders, IPMI board etc. You can use more than that in a 1U server or a workstation if you want to. Couple of Xeons or Epycs and a couple of GPUs and you're already past that.
In terms of power density per RU, this is small bikkies. Start adding Instinct or Tesla cards and you can triple or more that desnity. Two racks of servers is the rough equivalent of the power available to a small electric farm tractor.
Recently, I haven't been liking the non-descriptive titles. However, I find that the video itself is usually very good. I love how you used an ethernet cable as a make shift whip!
@BikeLife o for sure, but without it, I just don't see a massive advantage for the ratcheting mechanism. But I'm not constantly assembling computers, and id imagine it was designed mostly for that where tourqe settings wouldn't be hugely advantageous, especially given the cons and complexity of adding a tourqe mechanism,
I love that you have finally done a episode on jbod and head server. I have been looking into this for a home or small business. There is not much on you tube on this that clearly show you how to build a home or a small business level. Would you ever considering on doing a episode on how to build both jbod and a head server and how to connect them.
Obviously depends on capacity or number of drives but for home or small business you would normally use internal disks. A server chassis with say 12x 3.5" drive bays and internal SAS or SATA controllers is way cheaper to buy than a server + JBOD + external SAS + cabling, it's less to worry about, less space, less power, etc. Also to be blunt if it's just file storage, then unless you want to nerd out for the sake of it - just go and buy a NAS from someone like QNAP or Synology.
Depending on the controller, you might be able to at least light up a particular drive. If you rely on labeling, you're relying on anyone that puts their hand on it to properly label.
Honestly these segways to the sponsor just never get old. I still just get blindsided by them and dont know when theyre coming. And I laugh about everytime. And then I skip through the ad but i mean its still good stuff.
Scaling up will keep you busy every year. Go scaleout like ceph, or go home. With all its incredible merits, you will hit a brick wall with single node ZFS. You not having ceph or glusterfs staff is a valid reason, but your troubles with ZFS are not exactly encouraging. You could opt to let professionals take over. Yeah they cost $$$, so does fucking around with your precious data.
8:56 SAMTEC and many others make connectors like those, they are usually called board-to-board interconnects or backplane connectors. Supermicro no doubt made (or had someone make) them custom for that and their other servers.
I enjoy watching these videos, but with all your building expansions, how have you not built a legit server room? Talking about all the airflow, heat storage, power capacity, proper cable management? Sounds like a whole video SERIES in itself. ;)
12:19 Biggest thing I have to deal with as a datacenter Storage Admin is constantly checking people that assume Storage doesn't need lots of host resources. Perfectly competent technical people somehow miss that processing and handling IO isn't free in terms of system resources, both at the Storage System AND Host Initiator sides. 20:20 Metadata dedicated is a godsend for huge fileservers like yours -- The idea is that filesystem metadata lookup is actually a really big performance hit because metadata reads/writes are very small, very random, and usually with a sync_required status so they hold up the line. With a metadata tier, you put all your metadata in a dedicated high performance tier/pool/spam/whatever and it not only speeds up your metadata activity, but it also keeps your other storage open for regular IO without having to shudder around working those small metadata IO packets in the middle of regular workload.
@Drew we run JBOF/all flash, so each drive is 200.000 iops 4k for older systems, and more for newer, from a hw perspective, with zfs they just hang around chilling waiting for cpu, 7f52, ice lake.. same same, breaking 15-20GBs is hard even for larger files. (qsfp28/56 backend). On spinning drives its another story
@Paul Hughes Problem with that theory is that they have over many videos showed a total lack of understanding about PB scale storage. What you see as they lurch from one disaster to the next is them slowly learning stuff that those like myself with the hard won experience have known for years or even decades. Consequently the only reasonable conclusion is they indeed don't know better. However as I said previously that's not surprising as when doing PB scale storage almost everything you think you know about storage is wrong.
So... you don't have redundancy AC in the server room? that's basic! and a simple second ar/or third mini spliter inverter would be enough to never have to worry about it. Just let it on, and when the temps are ok it will turn off the AC and just blow air, not making any huge difference in the energy costs. I would just use two or three and over dimension them so they run on minimal power always and if one or two fail one would still be enough. Much cheaper than risking all the expensive hardware because of temps.
If they go with trueness core rather than scale, cinder is an option which would then potentially open the door to an internal openstack deployment, I can think of a few interesting applications for such a thing at LTT.
The funny thing is, the Clustering Tech Linus specifically said they wouldn't be using (Ceph) is the gold standard for Cloud Storage, scaling to dozens of PB. At the moment, Technology is keeping up with Linus' demands; he can keep scaling up harder to get more storage. At some point he'll have to scale out. And there, Ceph will be waiting
Longhorn is actually very nice for k8s cluster storage. Its very user friendly, even if you don't have the certification in clusterology that is required for Ceph.
Everytime @Linus Tech Tips does these videos I always remember the image of a huge 5mb disk in 1960 compared to a 1TB micro sd today and wonder if in the future they will be looking at Linus build Petabytes of storage server while they have thousands at the palm of their hands 😆
Pretty much every larger enterprise storage requires 208v. In example, Pure's C60 qlc nvme and Dell's Powerstore exclusively run 208v. Honestly, economically it also makes sense on both power usage and being able to run lower amps (and thus less costly power equipment) especially when running 208v 3Phase
I wonder what the cost difference would be vs using cloud storage with a dedicated connection? There's the cost of storage and the dedicated connection, but what does this equipment cost with power and hours spent? I know AWS has a different storage option where data frequently accessed is cached locally.
The standard deal I have seen with ZFS is to do 8 drive vdevs of RAIDZ level 2. It seems to be a good break point when dealing with lots of data. I suppose 10 drives isn't too far from that. I would not recommend doing deduplication. It eats a lot of resources, really slows things down, and unless you have a whole lot of duplicate data, it doesn't really help. When you look at all of the resources deduplication eats up and what those resources cost, maybe just consider buying more drives and call it a day. For compression, zstd seems to be a pretty good deal. May still want to think about how compressible your data is and if you want to be spending so much CPU on compression. For an archive server, even if it just helps a little, may consider it.
I am happy to announce that I built my first pc exactly 1 year ago, that means I did not miss any single video of LTT from 1.8 years almost, it was a new space for me I remember what horrible parts I chose while making the first list of parts then after getting knowledge From here and other creators also i selected the best bang for the buck components. My budget was very low so it had to be good. While my first selection was like just looks and names of the brand but the when I made the second list which was around 6-7months later it was much better. All and all thanks LMG for helping me love computers
At this point, you SHOULD be running a robotic tape library by now if you're running 3.6 PB of raw storage for archive. There's literally NO excuse for you NOT to be running a robotic tape library by now. But yes, at startup, because the system doesn't know the state of your environment nor the thermal conditions of the drives, it will run the fans at 100% (which is not unusual/uncommon) until it gets the environmental sensor data and the drive temp data and it will start ramping down the fans. The old Sun Microsystems SunFire X4200 was I think rated at something like 106 dBA for the front six fans that it had. You're getting into data center class/hyper scaler class hardware and leaving the realm of "normal" business/enterprise grade hardware.
Use a labeler and put the drive serial number on top to make double sure you are pulling the correct drive. Would be easy to pull the 2nd drive on the bottom of the row to later find out you pulled the 2nd drive from the top of the row.
Oooof! I feel for the bearings in that standing table! Also, goo advert for it, most cap out ar around 80-90kg!!! Although mine is rated to 120kg. LLT store suggestion, can we get a bundle for all of the t-shirts, not to save money, bust to save me having to hunt them all out (LTT store needs improved product filters FYI).
@Administrator WSV What are you talking about? HPE, Dell all still sell LTO tape drives or libraries. What we've seen is a consolidation in the manufacturers, a number of these IT vendors just resell another OEM's drives & libraries like Lenovo and Dell do with their badge on it. IBM & HPE still have a wide range of libraries. HPE's biggest LTO library can expand to a 56,400 slot, 144 drive 2.53EB behemoth. IBM has something a bit smaller. The other consolidation is that HPE & Quantam have left the development of the drives to IBM. Those three companies control the LTO consortium.
@Xanderplayz Yeah you jolted my memory, it was 2018 trvid.com/video/video-alxqpbSZorA.html Since then LTO-9 (18 TB) has arrived and LTO-12 (144 TB) has been teased.
The connector is call AirMax. Used a lot on datacenter blade servers. treat them super gentle. They jam and break really easily. I recommend never disconnecting them.
As someone who's trying to get into TrueNAS, I would *love* to see a video explaining ZFS. I've been trying to wrap my head around why vdev setup is important and haven't really figured it out.
The only silver lining about the whole server situation at LMG is the fact that you get to build a shiny new one for content & better than ever, complete with sponsors.
Regarding connector @ 8:44, I am fairly certain those are some variant of an Impact connector. Lots of companies, including TE Connectivity and Molex make them. They're used for routing high-speed differential pairs from a daughtercard to a backplane.
Oh sweeet! All the stuff I'm working on right now. Discussing a vdev type special with 45Drives to add to our Storinator, because last week we got f*cked trying to process folders with 90K files. And this Supermicro is nifty, still I'm inclined to go with Seagate Corvault, but those look like direct competitors. Got a meeting with our consultant tomorrow, to discuss planning 208V power and extra-deep racks for our new server room, because those suckers are 46" deep! No I don't work for LTT, but LTT can work for me. ;-)
Why a Z2? With this much data the resilvering will need ages if one drive fails. A mirror+stripe would be much faster in daily use and in recovery… Yes, with Z2 it‘s not important which disks will fail, you‘ll always have 2 disks which can fail, but resilver is high risk… I would at least use Z3, but would try to avoid it.
@Mark Butler I didn’t talk about the (bad) speed, I think the most important thing here should be security and availability and this is also really bad with ZX and disks with many devices and/or high capacity because of the resilver process.
It's all about capacity. Since this is archival storage they don't need the performance improvement that mirroring provides and Z2/Z3 allows for a lot more usable storage.
I love how at their scale they could pay a company to manage their data professionally but they instead just YOLO it for the sake of making more content...
@PWN NATION I think you could be right. With the growth of floatplane and the lab, soon they are going to need more than a few petabytes and a logical step could be having their own branch dedicated to data management. Something like Floatplane or Creator Warehouse...
@Chiron maybe less than an Full Time Sysadmin BUT a Sysadmin would not only maintain the Server but also all other Servers, maybe even Clients and users, the Network. An Storage Server that is well setup and running does not need much maintenance (hours/week) setting it up sure that takes some time but that would be like a 1 or 2 Week project. After that you need to spend a few hours per month to make sure that everything stays up to date and no data is corrupted (redundancy is key for something like that, but as Linus stated in an older video if the data is lost its lost not great but fine they could get everything back from youtube). And the key advantage on having everything on Prem is latency
A petabyte at something like backblaze b2 would be around $5-6k per month. Pretty expensive, but also probably less than a full time Sysadmin... I don't think they could expect anywhere close to the current performance though even with their 10G fiber connection.
At this point, if you don't have a scissor lift right by your server room to hold that front-loaded weight when you pull the drives, you're going to have the most EPIC episode of "Linus drops stuff" in the history of Linus dropping stuff.
