This seems more daunting than dedicated resources on the Cloud. Imagine having DDR2 RAM and then needing to scale up to better performance on DDR3 (I know we're coming up on the release of DDR6, but just using my past experience with bare metal shifting from DDR2 to DDR3 to make this example), but you couldn't just do that without swapping out a motherboard either. If you didn't have 2 or more machines, you would also have significant downtime as the server was prepped.
Not going to happen with provider supplied equipment. You will be obtaining a new server that supports the hardware configuration you want.
The only time you will get motherboard "hands-on" swap-out is generally when their is a MB failure on the hardware that the vendor is supplying. And if they no longer have that MB to do such with, they will frequently bump you into a newer server that is near the specs of the old one at the same cost.
If you are doing co-location, unless you are paying for hands-on services, you will be the one doing the swap out. That's why generally it's not a motherboard swap, but a total server swap. If it's time to upgrade the motherboard, the other components are usually going to be lacking also, so you are going to drive a new(er) server into place. When you are at that level, you are rarely piecing your server together like that.
You also can't load balance with a dedicated server if you suddenly got peak traffic that it couldn't handle... it'd just tug along
Yes, you can. You simply install the required number of servers you need if you are doing co-location. If you are obtaining your servers from a provider, you set your system up before that by having the hardware in place.
And yes, there are offerings that are not SaaS based that will also let you scale up like you are referring to. They cost more than a regular VPS, but they are out there.
With Cloud, you can pop up instances around the globe to handle the heavier load in a region closer to where they are in a matter of minutes, then destroy the instance when the resources are no longer needed, only paying for the hourly cost of doing that as opposed to a monthly cost for a dedicated machine.
That's what CDN's like CloudFlare are for, to help with the data delivery more localized source, reducing the loads on the servers depending on the cache level you have configured. There is no need to have a server instance in a particular geographic region when the "heavy" data you deliver is already sitting out on the CDN.
And you can do the same thing with providers. OVH has (and has had) that capability for a few years in their offering(s).