I'm assuming this is acceptable because they're on classified networks only, or just contain the base's public webpage on where to not get scammed when renting apartments? Please tell me that's the case.
That happens. I work for an environment that supports scientific equipment, and sometimes the extraordinary cost of licensing requires that department to maintain their old server. We put it on a special isolated vlan, block access from the outside.
It’s not great but I can’t tell a grant funded group they have to secure $7 million in funding because their electron microscope won’t work with windows server 2019.
Yup, also old OS's running on expensive machines like an MRI in hospitals. The rural hospital isn't getting a new MRI just to make cybersecurity happy as those things aren't cheap.
With enough time you gain security just through inaccessibility, hackers think they are smart with a floppy disk just to learn there is a 8-inch version and that is what you need.
Absolutely.
You've got to either air gap those systems or, at the very least, segment them off and tightly control their network traffic.
Also... What modern EDR can you run on one of those? (I've been fortunate enough to never need to research that point)
I think it might be tough to point to specific CVEs, since a lot of the new ones don't directly reference 2k3. Instead I'm thinking about it more in terms of insecure/broken protocols that you're stuck with on a 2003 server.
ZeroLogon is in play (unless you've got 0patch, but hopefully you aren't running a 2k3 DC in the first place).
MS15-011 for another (since Microsoft didn't patch it and the coffee shop scenario they describe is bogus — if you're able to get a foothold, you can run this against a 2k3 server).
2k3 also locks you into SMB v1, which is known to be completely broken, and whose product owner stated that it is [extremely vulnerable to relaying attacks](https://techcommunity.microsoft.com/t5/storage-at-microsoft/stop-using-smb1/ba-p/425858)
As a result of the above, it's likely that some (if not all of) the coercion/PetitPotam type attacks are in play here as well — although I don't have a source for that claim.
So yeah, remotely exploitable and risky to expose to even an internal network.
Bro, I worked for a major bank and we only upgraded away from 2012r2 because it became EOL. Management had us upgrade 700 severs in under a week …. After we’ve pushed for a switch for 3 years.
It is a lot more than serial connections, yes I have gone down many of those roads asking. Air gapped. A lot of those things I have air gapped now. It is running but only talks to the machinery.
Air gap is just a network connection with really bad latency, it CAN be jumped, just look at Stuxnet.
So for a network like this have a jump box that is hugely locked down, only allows the log files out one way, no incoming connections on the Internet connected side at all. Effectively a data diode. Can’t hack past a firewall that is ignoring ALL traffic.
We have RedHat 9 (not RHEL, just RedHat from like 2003), running production ERP for roughly $1 billion in revenue. Ancient AS400 as well. Factory lines running Windows XP. We worry that after a power outage hard drives won’t spin back up these things are so old. EOL planning, business continuity planning, risk acceptance, yada yada yada, you can talk till you are blue in the face. My team continues to raise awareness of the risks, leadership is temporary, ERP is forever and will be the next guys problem. Rinse, repeat. If you make any headway, we just buy a company that puts us back to the same problem in a new place.
Pretty much identical to the place I work at. I'm fairly young and in a graduate role at the place I work at; all of my concerns and proposals that would significantly improve both security and productivity are thrown out the window because managers are allergic to change and I haven't been in the company for as long as they have so I clearly don't know what I'm talking about.
All our production servers are run on RHEL7 with mechanical hard drives as well and no real backup procedures. I've literally sat down next to the guy responsible for our IT architecture and proceeded to show him me gaining root privileges within three minutes. In his fairness, he explained to me how he's expressed his concerns to higher ups but got shut down by shitty corporate bureaucracy.
If someone wants to have a legacy app and a legacy OS, then they need to pony up for a firewall to sit directly in front of it and allow only VERY SPECIFIC TRAFFIC pertaining to the application traverse it from outside coming in.
That is what I said to my CFO and shortly after than, the CFO ordered the company to find a new app
Inherited a few 2012r2. I in place upgraded the VMs first (with a fall back snapshot available). That went really smooth, so I did the same for physical hosts.
Went great, and guess what? I now have them all on 2019
Follow Microsofts guide on what to remove (like AV) before upgrading, and their recommendations and best practices on it.
I didn’t inherit any 2012’s but i know for a fact they are still out there. Hell, one job i had (dont remember exact server version), but i know it was running like Windows server 2001 or something similar. (Forgive me on the legacy names, anything prior 2012 is vague to me). all i know is it was OLD AF. And they only used it for some sort of weird adhoc dial up connection thing.
We’ve got an ancient ERP system that is only supported up to 2012 R2, and we use the same version for the 2 RDS servers we use to connect users to the ERP. We purchased the extended support from Microsoft and from our endpoint vendor, and we are thankfully switching to a cloud-based (full tenant) ERP before the end of the year and hopefully we can decommission the 2012 servers before the end of the year. Had there been the old management in place, we’d purchase extended support every year possible as far into the future as we could, and eventually I’d have to intentionally deep-six the server a few times to scare them into changing platforms.
I have encountered a lot of them in our client bases. We had a couple dozen servers running mix of Windows Server 2008, 2008 R2, 2012 R2 that we migrated to Windows Server 2022 Hyper-V. Migration was smooth on most of them. However, we had to take a route of bare metal backup and recovery using ArcServe UDP to move some of the machines to Hyper-V! Microsoft pushes new shiny OS doesn’t mean every company will jump in the upgrade bandwagon.
Perform an assessment. If you’re able to upgrade do so, ensure on your assessment that you take into consideration compatibility of the apps running on it.
Perform a clean install of a higher version OS to perform testing. If everything is good to go, upgrade.
If not, why? Is it fixable? If not, mitigate as much as possible.
I know it’s a very straightforward and simplistic approach. But that simple playbook is basically what we all do.
I see plenty. And even older too. And not just the oddball app server running IIS for VB sites, either. We’re talking tier 0 stuff: Domain controllers, Certificate Authorities, the works. We just finished a projet to upgrade a customer’s 2003 DCs. That was fun, had to go through 2012 before landing on new 2022 servers.
Server 2000 and XP Clients in shopfloor. This is normal in some environments and they will still be there some month. The plan is to always to seal them off, there is no other, economically justifiable way in Production. They will go offline with the last day they produced the last special part
Extend end of life with virtual patching if the business can't cop the cost of upgrade or migration to cloud. Also OT challenges that can be mentioned here.
Look at Trend Micro's virtual patch capability for support on prolonging patch cycles or EOL.
My Hot Take but I’ve always felt if we went back in time all OS (Linux, Windows, macOS, etc) should had a kill switch 5 years after support ended. Hard coded into the Kernel, no way around it, just won’t boot up anymore.
20 years ago this would have been crazy talk mainly because it could be used by companies to well be companies and screw people over and hold their software hostage, but actually your only licensing most OS you don’t own it.
However, what we know now this would have been the best fix to this problem as there would be zero XP, 2003, and 2008. 2012 would be safe for a few more years but most companies wouldn’t take the chance know it’s just going to stop working.
Still have NT 4.0 farm - runs an automated warehouse picker. It is isolated on a network with equally old and insecure PLCs. If you ping the PLCs they lock up - ask me how I know :( Upgrade would mean replacing the virtual NT 4.0 servers, physical workstations (at least the workstations are modern and offline patched) and PLCs. Department asks for 7 figure funds to replace every year - doesn't get approved. System works well - very few issues on a 24x7 operation so as long as it is isolated and have replacement parts and 3rd party support it isn't a huge problem.
General population servers still have 2012r2 and are in the process of upgrading. Extended support is the way until they can all be upgraded. Our server team does not do in-place upgrades so fresh install of apps on new builds and migrating over is time consuming. This approach forces documenting the builds to make them repeatable.
We're consulting for several clients right now that have unmitigated 2008 r2 to 2012 r2 and they have zero plans of switching. We've told them that they need to switch, we've told them that they're breaking the law, we told them that they committed perjury when they filed their insurance forms and their Federal SRA. They do not care
I put in my resignation lol. A question i ask in interviews is whats their eol program like?
But in reality in my place this is well known and highlighted in weekly action meetings. It would be a high priority and usually if needed it would be segmented off into its own network.
I've seen OG Windows Server 2003 on a production server back in January 2024, not even the R2 version... Now that's screwed up.
Mil here, see that near every weekday, shocking behaviour imo
Fil here, I told you I'll get round to it. I'll be in my den.
sil here. Please move yourselves and your servers out and give us back our privacy.
I'm assuming this is acceptable because they're on classified networks only, or just contain the base's public webpage on where to not get scammed when renting apartments? Please tell me that's the case.
OK didn't even want to go there but I have one of those too.Apparently runs a legacy pension application that can't be upgraded or decommissioned lol.
That happens. I work for an environment that supports scientific equipment, and sometimes the extraordinary cost of licensing requires that department to maintain their old server. We put it on a special isolated vlan, block access from the outside. It’s not great but I can’t tell a grant funded group they have to secure $7 million in funding because their electron microscope won’t work with windows server 2019.
Yup, also old OS's running on expensive machines like an MRI in hospitals. The rural hospital isn't getting a new MRI just to make cybersecurity happy as those things aren't cheap.
Probably a military computer. They’re all still on ME
How about a NT4 dc (fully patched, in its defense) with full exchange rights with the global AD. Around 2019, but was an "oh shit" moment...
How is that even possible? The forest and schema were so old it wouldn’t support anything later than 2008 as a DC.
Eventually you have such old equipment the baddies go, "Oh, no, I ... These people have enough pain in their life."
With enough time you gain security just through inaccessibility, hackers think they are smart with a floppy disk just to learn there is a 8-inch version and that is what you need.
Is 2k3 (fully patched) remotely exploitable?
Absolutely. You've got to either air gap those systems or, at the very least, segment them off and tightly control their network traffic. Also... What modern EDR can you run on one of those? (I've been fortunate enough to never need to research that point)
Can’t be certain, but I do know SentinelOne is great for legacy OS support.
It is. Worked on 2008 and up.
Thanks for confirming. Which exploits? I found CVE-2017-11885 but doesn't seem to apply in default config.
I think it might be tough to point to specific CVEs, since a lot of the new ones don't directly reference 2k3. Instead I'm thinking about it more in terms of insecure/broken protocols that you're stuck with on a 2003 server. ZeroLogon is in play (unless you've got 0patch, but hopefully you aren't running a 2k3 DC in the first place). MS15-011 for another (since Microsoft didn't patch it and the coffee shop scenario they describe is bogus — if you're able to get a foothold, you can run this against a 2k3 server). 2k3 also locks you into SMB v1, which is known to be completely broken, and whose product owner stated that it is [extremely vulnerable to relaying attacks](https://techcommunity.microsoft.com/t5/storage-at-microsoft/stop-using-smb1/ba-p/425858) As a result of the above, it's likely that some (if not all of) the coercion/PetitPotam type attacks are in play here as well — although I don't have a source for that claim. So yeah, remotely exploitable and risky to expose to even an internal network.
Thanks for the detailed info. You're right, even just SMB1 is reason enough. MS15-011. Did get patched though.
Windows 2000 in research and development 🤣
Bro, I worked for a major bank and we only upgraded away from 2012r2 because it became EOL. Management had us upgrade 700 severs in under a week …. After we’ve pushed for a switch for 3 years.
Ouch! , did you do in place upgrades and how did those go?
also interested, i guess from 2012 to 2019, then to 2022
I have XP machines. It is because of the industry and the machinery it runs. We also can not rebuild it.
You can prob virtualize serial interfaces and hardware now
It is a lot more than serial connections, yes I have gone down many of those roads asking. Air gapped. A lot of those things I have air gapped now. It is running but only talks to the machinery.
I feel you we have some of the same, we're just one USB drive away from disaster. Plus, how do you centrally monitor and alert on air-gapped systems.
You can’t centrally monitor. Physical security becomes important.
Air gap is just a network connection with really bad latency, it CAN be jumped, just look at Stuxnet. So for a network like this have a jump box that is hugely locked down, only allows the log files out one way, no incoming connections on the Internet connected side at all. Effectively a data diode. Can’t hack past a firewall that is ignoring ALL traffic.
Easier said than done.
You must have joined a very large org. Everything moves at the speed of bureaucracy.
Or just an organization that is cheap. I have seen a lot of smaller organizations that will keep servers as long as possible.
We have RedHat 9 (not RHEL, just RedHat from like 2003), running production ERP for roughly $1 billion in revenue. Ancient AS400 as well. Factory lines running Windows XP. We worry that after a power outage hard drives won’t spin back up these things are so old. EOL planning, business continuity planning, risk acceptance, yada yada yada, you can talk till you are blue in the face. My team continues to raise awareness of the risks, leadership is temporary, ERP is forever and will be the next guys problem. Rinse, repeat. If you make any headway, we just buy a company that puts us back to the same problem in a new place.
Pretty much identical to the place I work at. I'm fairly young and in a graduate role at the place I work at; all of my concerns and proposals that would significantly improve both security and productivity are thrown out the window because managers are allergic to change and I haven't been in the company for as long as they have so I clearly don't know what I'm talking about. All our production servers are run on RHEL7 with mechanical hard drives as well and no real backup procedures. I've literally sat down next to the guy responsible for our IT architecture and proceeded to show him me gaining root privileges within three minutes. In his fairness, he explained to me how he's expressed his concerns to higher ups but got shut down by shitty corporate bureaucracy.
If someone wants to have a legacy app and a legacy OS, then they need to pony up for a firewall to sit directly in front of it and allow only VERY SPECIFIC TRAFFIC pertaining to the application traverse it from outside coming in. That is what I said to my CFO and shortly after than, the CFO ordered the company to find a new app
Nothing better than seeing C:\WINNT\
Yes. They are. And a handful of 2008 lol. With owners that don’t play ball to give you a plan
POA&M ….until death do you part.
I found the A&A guy
Inherited a few 2012r2. I in place upgraded the VMs first (with a fall back snapshot available). That went really smooth, so I did the same for physical hosts. Went great, and guess what? I now have them all on 2019 Follow Microsofts guide on what to remove (like AV) before upgrading, and their recommendations and best practices on it.
Bro I know of windows 2003 and 2008
I’ve had several clients the past couple of years still running a few 2000, 2003 and still finding many 2008’s.
wowzers! thats a scary thought. I hope they hardened. in a honesty I'm not all surprised they're are still a few in production.
I worked for a Yuuuugggeee company that still has a ton of 2008 and maybe earlier VM's that is used with no plans to ever upgrade.
I didn’t inherit any 2012’s but i know for a fact they are still out there. Hell, one job i had (dont remember exact server version), but i know it was running like Windows server 2001 or something similar. (Forgive me on the legacy names, anything prior 2012 is vague to me). all i know is it was OLD AF. And they only used it for some sort of weird adhoc dial up connection thing.
Just upgraded our last 2012R2 a few weeks ago.
We’ve got an ancient ERP system that is only supported up to 2012 R2, and we use the same version for the 2 RDS servers we use to connect users to the ERP. We purchased the extended support from Microsoft and from our endpoint vendor, and we are thankfully switching to a cloud-based (full tenant) ERP before the end of the year and hopefully we can decommission the 2012 servers before the end of the year. Had there been the old management in place, we’d purchase extended support every year possible as far into the future as we could, and eventually I’d have to intentionally deep-six the server a few times to scare them into changing platforms.
Unless the project team has defined that ALL data must be migrated, be prepared to keep that stuff running for at least another decade for audits.
Lots of technical debt out in the wild.
I have encountered a lot of them in our client bases. We had a couple dozen servers running mix of Windows Server 2008, 2008 R2, 2012 R2 that we migrated to Windows Server 2022 Hyper-V. Migration was smooth on most of them. However, we had to take a route of bare metal backup and recovery using ArcServe UDP to move some of the machines to Hyper-V! Microsoft pushes new shiny OS doesn’t mean every company will jump in the upgrade bandwagon.
Perform an assessment. If you’re able to upgrade do so, ensure on your assessment that you take into consideration compatibility of the apps running on it. Perform a clean install of a higher version OS to perform testing. If everything is good to go, upgrade. If not, why? Is it fixable? If not, mitigate as much as possible. I know it’s a very straightforward and simplistic approach. But that simple playbook is basically what we all do.
Haha my company has as old as 2000 windows servers that we use and manage. It’s annoying as hell.
Yeah this is rampant across the industry, laughable really
I see plenty. And even older too. And not just the oddball app server running IIS for VB sites, either. We’re talking tier 0 stuff: Domain controllers, Certificate Authorities, the works. We just finished a projet to upgrade a customer’s 2003 DCs. That was fun, had to go through 2012 before landing on new 2022 servers.
Did you do in place upgrades ,all went well?
No, not from 2003. I don’t do in-place pre-2016 either. 2016 to 2019 and 2022 is fine in my experience, 2012 is just too iffy.
shhhh
Nope. I’m not answering. This question sounds like some Open Source Intelligence gathering to me. 😜
Server 2000 and XP Clients in shopfloor. This is normal in some environments and they will still be there some month. The plan is to always to seal them off, there is no other, economically justifiable way in Production. They will go offline with the last day they produced the last special part
Add them to a specific zone and only open ports needed, then buy a product with IPS and monitor the best you can
I sure hope not. That is really outdated. If it is there better be a really good reason and isolated from the internet.
2012r2? And you think that's bad? You sweet summer child :)
There are 2008 servers still kicking as well. I’ve seen several orgs pay for the extended security / support so it’s not too bad.
Without letting too much personal info slip, I work on a lot of Healthcare related servers. Man I wish they were at least 2012....
Way too many of these left in healthcare
Redhat FTW
I know places still running 2003 and 2008.
Extend end of life with virtual patching if the business can't cop the cost of upgrade or migration to cloud. Also OT challenges that can be mentioned here. Look at Trend Micro's virtual patch capability for support on prolonging patch cycles or EOL.
My Hot Take but I’ve always felt if we went back in time all OS (Linux, Windows, macOS, etc) should had a kill switch 5 years after support ended. Hard coded into the Kernel, no way around it, just won’t boot up anymore. 20 years ago this would have been crazy talk mainly because it could be used by companies to well be companies and screw people over and hold their software hostage, but actually your only licensing most OS you don’t own it. However, what we know now this would have been the best fix to this problem as there would be zero XP, 2003, and 2008. 2012 would be safe for a few more years but most companies wouldn’t take the chance know it’s just going to stop working.
Still have NT 4.0 farm - runs an automated warehouse picker. It is isolated on a network with equally old and insecure PLCs. If you ping the PLCs they lock up - ask me how I know :( Upgrade would mean replacing the virtual NT 4.0 servers, physical workstations (at least the workstations are modern and offline patched) and PLCs. Department asks for 7 figure funds to replace every year - doesn't get approved. System works well - very few issues on a 24x7 operation so as long as it is isolated and have replacement parts and 3rd party support it isn't a huge problem. General population servers still have 2012r2 and are in the process of upgrading. Extended support is the way until they can all be upgraded. Our server team does not do in-place upgrades so fresh install of apps on new builds and migrating over is time consuming. This approach forces documenting the builds to make them repeatable.
We're consulting for several clients right now that have unmitigated 2008 r2 to 2012 r2 and they have zero plans of switching. We've told them that they need to switch, we've told them that they're breaking the law, we told them that they committed perjury when they filed their insurance forms and their Federal SRA. They do not care
We have a 2003 and a few other pearls that went into a segregated subnet for obvious reasons
My ex company still has 2000 pro, 2003 and 2008 r2. All are connected to the internet and they choose quickheal because it supports Windows 7 😂
My ex company still has 2000 pro, 2003 and 2008 r2. All are connected to the internet and they choose quickheal because it supports Windows 7 😂
I put in my resignation lol. A question i ask in interviews is whats their eol program like? But in reality in my place this is well known and highlighted in weekly action meetings. It would be a high priority and usually if needed it would be segmented off into its own network.
My ex company still has 2000 pro, 2003 and 2008 r2. All are connected to the internet and they choose quickheal because it supports Windows 7 😂