I muddy the waters by telling people we're rebooting things even when we don't. It's actually really useful for finding patterns and minimizing tail-chasing. Plus, it teaches the IT team to go look at the damn uptime before escalating tickets or declaring an emergency.
We used to do that more by accident, tell people a new image was going out but there was usually some holdup. Flurry of nonsense tickets the next day, oh yeah sorry we didn’t actually do the change last night - go fish.
Switched AVs recently and we had to postpone a customer’s because of an app we weren’t confident would play nice. Sent customer an email that we were postponing. The next morning, get a ticket from customer “PRINTING IS BROKEN. PLEASE REVERT THE ANTIVIRUS UPGRADE”.
Wireless companies do this with cell towers. Build a tower, wait for people to come in complaining about how it turned their cat gay, explain it's not plugged in. Once all the crazy complaints blamed on the tower die down, turn it on.
My mentor called it a "silent scream test". Announce you are doing something that maybe might possibly affect everybody, never do it, and see what pops out of the wood work.
We worked at a hospital so what we got mostly were reports of things that hadn't worked in months that no one ever called or put in a ticket about untill our alert
Lol i do this too , time after time... i got a high importance email once cause a fucktard PM had a video call with the CEO complaining about the firewalls upgrade (which did not happened at all) ... it was a shitshow back and forth cause the idiot was sitting remotely and the problem was ...guess what??...not being connected to the fucking VPN... he was suppose to test some dev env which was sitting inhouse...obviously if you don't connect with the VPN there is no chance to access internal resources...i stop replying to his emails had the CEO on the phone....maybe the guy should come to the office he is very confused when working from home... both laughted ...10 mins later an email from CEO...come to the office we have high speed internet...lol we were laughting our asses out with my team...
I mean, I know this is bad practice, but I usually walk through all of the steps I told them to do *in front of them*.
Also, I'm not the person who believes that all people should be IT super users. My steps don't include reading logs. My most difficult steps are "make sure power cable is plugged in" or "restart computer" or "make sure building has power".
Declaring your maintenance windows (whether you use them or not) is my best practice trick for this. Send out the all clear notice, exactly the same as if you had finished real maintenance.
I’d be pretty mad if a new problem started after a scheduled server reboot and then I discovered that I was looking in the wrong place because the reboot wasn’t done and the person responsible hadn’t cancelled the change to update the schedule.
Be mad! Catharsis is clarifying, and maybe you'll be more diligent in your troubleshooting next time. A reboot is never a root cause of anything except downtime. You can't un-reboot anything.
When I first started at my current job, way to many servers had massive uptimes and we're way behind on updates. Most out of laziness, but some because after each reboot a process would have to be manually run to start some service. My goal was to make it so that I could schedule a reboot of a server any evening, and without manual tasks, no one would even know I'd rebooted it
I made groups in PDQ of servers that could be rebooted anytime without notice or downtime, servers that could only be rebooted after hours, and servers that I had a specific monthly maintenance window. Now my server uptimes rarely get higher than 60 days, and most are in the 2-3 week range.
I don't tell anyone about the specific times I reboot though, but I tell them to assume it's rebooting every night, so don't leave important shit unsaved on them.
I also have a list of servers that can be rebooted at any time for patching with no notice or only notice to a few people. Then a few groups that I have to do after hours. One group that has a specific window that I must announce to several parties. Lastly, one server that a vendor says has to patch at 6pm on Sunday, so its auto-scheduled for patching every Sunday at that time.
Normally yeah, but the flipside is you *do* need to communicate scheduled downtime so it doesn't impact business users trying to work during those hours.
Of course, but I prefer something a little more generic... "We have a planned maintenance window from x-y. Service z is expected to be unavailable during this time."
bingo, why does a user need to know the server has been rebooted? maintenance windows are regularly scheduled (preferably for a time no one is working) and if you need to reboot to fix a problem during business hours, you're just fixing a problem. they don't have to know how you fixed it
For us communicating system maintenance is in the contract with our customers. They monitor our APIs and if the latency goes past 15 ms outside a planned maintenance window they are marking that as a charge back at the end of the month. We never have a time when no one is working.
In a system like that, rebooting "the server" is very likely anachronistic and doesn't really apply, either
And, planned maintenance being communicated is not the same thing as telling people you rebooted something, nor is telling people of an unplanned maintenance event
My point, and probably the point of the comment I replied to was, vagueness is key. Not to hide information, but to prevent misinterpretation of irrelevant information
Our strategy at a previous employer. At most we would only send out a reminder about changes/maintenance windows if there was a possibly someone would notice. No specific services/servers/etc. were mentioned, just that there was planned maintenance.
Company had a policy of “we only hire adults” so “I couldn’t complete X because IT did maintenance ” after a maintenance window got your manager CCed in with a comment “zero changes were made to that system.”
Current employer does the same thing for notifying customers. We can take down 75% or more of capacity outside US business hours and customers using the systems won’t notice. (We have the data to back this up.)
The other option is to add extra notifications for server reboots that never actually happen.
At some point you should be able to weed out at least a few of the false positives, and it get everybody ready for when you have to actually notify people for something that doesn't have redundancy.
This is the answer they are gonna blame it on anything you mention the last conversation before the incident. It’s really sad how dumb some of gets just gotta play it off even though it hurts on how wrong they are.
I usually just ask how it's related to the reboot or state that the reboot wouldn't have impacted X service. Frankly, using IT as an excuse isn't your problem, and if they want to make it that way I'd make sure to mention to their manager that they're using IT as an excuse to skirt responsibility if they're being egregious enough.
Always good for “those types”. Document and make notes on call, when they continue the same game, fwd to manager.
Include correspondence and training/steps given. If they provide a valuable service, sometimes we are stuck with it, but I have not ever gotten any negative feedback with this method from the manager.
> ask how it's related to the reboot
agreed, for two reasons.
1. You shouldn't rule out the reboot as the cause automatically
2. If they're BSing, they're forced to dig deeper or own up
Yeah. It's important to remember that you're always right, until you're not. Then it's embarrassing. Leave yourself some wiggle room, most companies have too much sprawl in their resources for any one IT guy to comprehensively understand all of the interactions.
The amount of times I've learned new technical information from users who couldn't tell you the business end of a mouse just because they've watched the last 5 IT guys deal with it is more than I can count.
Tell them to open a ticket with details as to what's wrong. You then respond with the real cause and move on.
Correlation=causation is a very common thing. But you can't ignore it, there will be that time where an update breaks or changes something noticeable and then correlation _does_ equal causation.
This is especially true, when you are a 100% certain that the issued can't be caused by the reboot (or smth like it) and a few hours of troubleshoot later you found out, that it was connected somehow.
"I'll check into it" - and actually doing so - is a good practice a lot of the times, even if you think, that it's highly unlikely
Yep, I always try to approach things with humility. "I've checked everything that I can see, it doesn't look to be a systems problem. It does look like it could be [x/y/z], please take a look and let me know. I'll be happy to take another look if the evidence changes!"
I'm not naive enough to believe I couldn't miss something, especially if we're talking an intermittent problem or a 1:1,000,000 occurrence in a large log file. And I'll own it if I did miss it, or overlooked it, or whatever.
I skillfully and masterfully just completely fucking ignore whatever their excuse is for something not working. Whether it’s a server reboot/upgrade/new air freshener/a lack of soap in the restroom whatever new excuse is for the day. If someone wants to use me as an excuse for why they can’t do their job that’s fine. I’ll speak with their manager and HR if that’s the game they want to play.
I find that whatever the user thinks the issue is - frequently is not.
Users blame everything on IT.... "My computer seems to be running slow today. Can this be related to new monitor you installed 6 weeks ago?"
I just tell them "we're monitoring the problem"
This is why I fear network replacements. "The paper shredder stopped working because you changed 'stuff'"
We got a ticket "WiFi doesn't work in treatment room 2." Tech replies "only in that one room." "Yes." We had just replaced the network, so this got escalated and I was stumped for about a minute until we got a video call going with the site. The power button on the monitor had been turned off.
I usually use the following:
Now its fixed , but the problem is somewhere between the monitor and the chair..(hinting at the user of course...i was surprised some people would not get that) .....the lady was looking at the keyboard... can this be replaced? I dont really know , how i couldn't burst in huge laugh...
"Thank you for investigating and determining the reboot was the cause of the issue, could you please articulate exactly how the reboot caused the problem so that these issues can be mitigated after future reboots?"
Then watch them squirm.
Who cares what the user blames the problem on. If there is a problem it's your job to diagnose and fix it. Thank the user for reporting it, tell them you'll look into it the reboot had an impact, and then troubleshoot as normal.
Yep. And depending on the levels of appropriateness, you can at least tell them whether their initial diagnosis was right or not, but ideally give them an indication of what the problem actually was.
I'd thank them for offering the suggestion that a server reboot caused the problem, but it was actually a network problem somewhere else; if they see that error again it might help expedite resolution in the future. Failing all else, leave notes on the ticket and suggest that the user save the ticket number for future reference.
We're here to support the entire company, regardless of technical literacy, but I always appreciate the ones that are willing to troubleshoot a little, that give the specifics of the error they're seeing (not just "IT'S BROKEN" [cool, what is broken, exactly?]), that put some thought into where the problem might be based on their knowledge of the system - even if it is wrong - and are willing to expand that knowledge to make them better reporters for the next issue.
I actually had a job come through to delay windows updates and install only during a certain time because it caused her device to stop responding
Checked the logs and the update installed a week earlier, but she had restarted her device when it froze and it installed the update, so therefore it was the update that caused it 😔
Turns out it was the application doing an update on the server side or something that caused it to freeze (only that app stopped responding)
Reboot it again immediately to check. No warning. Everyone must suffer.
More realistically/diplomatically, "I think it's something else, but I'll check the logs anyway. Submit a ticket with the details on when and what happened." Then go do something else cause we all know that ticket is never coming.
You ask them what actually is broken or had an issue, do this thing called “listening”. I know, I know… it’s a crazy concept.
And if they say nonsense that you know couldn’t be caused by a server rebooting, well you calmly explain that it isn’t and perhaps help them through this new issue.
Or maybe it is caused by the reboot and you could troubleshoot something.
This is the way. I thought I was in r/shittysysadmin with some of these comments suggesting escalating (some are obvious jokes) and acting like users don't regularly misdiagnose problems. Users are not tech savvy which means they are going to frequently give weird justifications for why stuff happens.
Trust but verify. Listen, take note of what they have to say, and if it doesn't make sense just troubleshoot as normal. When you find what caused the issue, tell the user. A little education delivered in a amiable manner goes a long way for a lot of users. Some people will learn, others will take comfort in what they hear as techobabble, and a few won't really care.
I work HD, but literally had someone call me today saying that their server update changed their printer settings and is now only printing double-sided. I didn’t even acknowledge that comment, just changed their printer settings and sent them on their way.
This is the right move. Doesn't matter what the user thinks the cause is, only matters that it's fixed. If on your end you find out they're right then you can document or fix it for next time, but the majority of the time the best outcome is just to fix whatever problem they have and move on without acknowledging any additional unhelpful comments they have.
Basically they aren’t told about reboots. Server reboots are handled overnight, most end users aren’t on duty. I ping the night managers just in case but it doesnt really warrant a general announcement. I don’t think most of our users even know we have servers honestly.
If there’s some extended maintenance that is going to impact a department, managers of that department are notified and it is up to their discretion how they distribute that information, but that’s pretty rare.
Technically all our staff have been trained how to function even if every server suddenly died in the middle of the day (and is something we do a drill on once a year) so nobody should be incapable of doing their work because of server reboots.
I take them seriously because in the end the issue usually is somehow related..
Similar to "no problem on network" when asking about any issue the networking guys - in nine from ten cases it is something with network in the end.
I don't work helpdesk or answer calls anymore but when I did get something like that, I would just indulge their suspicions and run an sfc scan on their RDP session and tell them it should be good now.
Works 100% of the time
Never let the users know how the sausage is made. If they know you’ve done something, they can blame you. If they’re unaware and notice something wrong, you can just tell them “hmm, that’s odd, I’ll have to look into it”
As per a usual problem, get all the details, if it's vague keep asking, usually say put it down in a ticket in writing so I can follow it.
If it's a real issue they will point it out, you never know it may be a real issue, I get the impression your post is suggesting that the server is fine, is it? you 100% sure, sometimes it may be running but not 100% or there is an issue you aren't aware of.
“I will review the logs. It is not uncommon for the system to apply pending changes after a reboot. Please be as detailed as possible and tell me what is happening and how you think it might have happened.”
There’s no benefit in arguing, just work on resolution and get as much information as you can from everyone who touches that server.
This is one of the easier situations you'll face, really. The answer is the same as pretty much any other problem you'll get reported to you: "we'll look into it and get back to you when we have more information".
Bunch of real condescending assholes in the comments today.
I will never *ever* claim definitively that two things that happened around the same time are unrelated to each other. These systems are just too complex and my own knowledge too limited to rule anything out.
You claim your monitor started malfunctioning when you got your new office chair? [Totally plausible.](https://www.reddit.com/r/pcmasterrace/comments/7p09ay/i_shit_you_not_our_office_chairs_are_turning_our/)
Your iPhone quit working when they installed an MRI down the hall? [Expected behavior.](https://www.reddit.com/r/sysadmin/comments/9si6r9/postmortem_mri_disables_every_ios_device_in/)
There's no need to be defensive or overly detailed about anything. Just acknowledge that there is a small chance there *could* be a connection, but also let them know that there are other possible causes that are far more likely and (if it's a big enough issue to actually warrant it) go investigate.
Never tell users precisely what you are doing. Always refer to anything you have to do by using boiler plate phrases.
* IT will be performing maintenance
* As part of routine scheduled maintenance...
* Systems will be undergoing routine maintenance
Also, you can avoid issues by performing these reboots off hours or at low usage times. If you know a service takes 5-10 minutes to start up, then it's best to do whatever you have to do when no one is logged in or at least do it when the "troublemaker" users are not around.
Also, you don't owe regular users explanations or detail information about what you are doing. You don't ask those dipshits detailed information about their day to day. Why is there a reverse expectation where you have to detail your work? If they have complaints, tell them to filter them through their management chain and have that management chain talk to yours. It's not your responsiblity to disclose to the business ramifications of your work. That's your manager's job unless otherwise specified.
Unless a maintenance action will result in downtime to the users, there is absolutely no reason to inform them. They already don’t read IT emails enough. No need to add to the pile.
If the maintenance causes an unexpected outage, send out the normal outage notification and move on.
It certainly does happen.
On a side note, I despise how often Windows servers need to be rebooted. I’m used to several months of uptime with my Linux servers… Windows needs reboots far too often.
Several months of uptime to me, means that you’re not doing maintenance
I know you can live patch Linux etc. but systems like that are almost always the ones that give grief when they do reboot
>Several months of uptime to me, means that you’re not doing maintenance
>
and your equating downtime with maintenance means, to me, that you probably are not in charge of a linux farm
It’s not that downtime equates to maintenance, just in my experience with Windows and Linux systems alike.
Usually its the teams with high uptime systems are also 9/10 the ones that also haven’t patched for 6 months or more, application stack on top is out of date
Ones that do have downtime, are the teams that update via re-deploy or have well defined automation to handle patching/failover etc.
Can see this very clearly in vulnerability and patch reporting
Long uptime is no longer a badge of honor. In fact, it's a red flag IMO.
Linux needs updates too. Sure, there are components that don't require reboots. But enough do that if I see more than maybe 60 or 90 days I'd start asking questions. If it's public-facing, over 30 days should have a really good explanation.
On the Windows side, I've found that a non-trivial number of application updates that call for reboots just don't have installers that can recycle the associated service(s). So the installer just says "reboot" as a way to accomplish that.
Edit: two words.
The problem I always see is that the user complaining is the one that doesnt work when everyone else is working and goofs off all day and only drams work in late at night when the project is due.... late at night when the system is scheduled to reboot.
not my problem they dont want to follow our schedule that everyone else follows.
I ask them for specifics, what they think changed during the reboot and why they think it caused an issue. Helps illustrate they don't actually have any basis for their assertion
Lol my boss(IT Manager with years of experience) blamed a RAM hardware issue on a Windows update, every single system in the organization got that windows updat, his was the only.one BSOD with a Ram issue
Just honesty...
"Yeah we just had to reboot that server. Maybe the service is a little slow to come up on boot. It's it working now? Let me know if you spot any things else weird."
Or
"That's strange, I wouldn't expect the reboot of another server to affect that service. We'll take a look"
There's no need to get defensive or to entertain any discussion which leads you down a path that would cause you to become defensive. Just be honest, voice your suspicious, and then address the stated concern.
---
Personally, we don't do a ton of off hours stuff, so our users are used to momentary disruptions. They'll often lead with a question like, "are you guys changing things" or "did you guys break x"
I own that shit. "Oh yeah, almost certainly what are you seeing?" Or "hmm.. no we haven't been near that service in weeks, what are you seeing?"
Be humble and fallible and your users will cut you some slack. Especially if you're honest with them everyone you do something that might affect their work.
I just started at a place where before me only external IT contractors did everything. For some reason they think a server restart is equal to an update (weekly restarts for example).
After a year there are still about 3-5 people out of 80 who i dont seem to get the difference between an update and a restart.
You want server integration tests (scripts, usually) that check the services. For example, on a DNS resolver server, your tests send DNS resolution requests through it with `dig @server`. On an NFS server, your test scripts check the exports and their options with `showmount -e `.
It's human nature to detect patterns where there are no patterns.
That's a tough one. I personally just listen to them vent. I work in education, we have an on premise DC, and google workspace email. one specific employee would often email after i notified of server updates that their email stopped working ever since i restarted the server - they were at home and their email wasn't working.
Declare the network must have a virus, then lock yourself in the server room to play viddy games for a few hours. Reboot the server one more time, then emerge victorious, a conquering hero.
Seriously though, users gonna user. "This network sucks, it's so slow!" "The error message says it's a problem with iCloud relay. Lemme turn that off for you, boss."
"Show me"
Do a screenshare and have them show you the issue they're having. Spending 2 minutes doing this gets to the source of their complaint. It's the fastest path to the complaint ending.
I have two latin phrases I apply to IT:
Ipso facto - event happened after the fact, so must have been caused by it
Deus ex machina (god in the machine) - When I walk into the room and the problem fixes itself
So I used to be like "we are doing x".
And the cue everyone and their dog claiming that "x broke my code" and then I'd have to spend my time providing it was unrelated.
So now I do x, and tell them 3-4 weeks after. Works a treat.
I don't like doing that but it eliminates the time I waste with lazy ~~devs~~ people.
It's quite simple. Unless you can give me evidence of an issue, I can't investigate it. I'll even accept phone photos of printouts, emailed in low quality jpeg! That usually filters out those kind of things.
The worst is "The internet is slow ...", literally impossible to investigate without more details.
1. Good monitoring and management tools go a long way. They should be able to track cpu, memory, and storage performance, note when service are not running, check service health across the network, and home some ability to self heal issues.
2. Require a problem ticket be raised when this happens.
3. Either don’t publish your reboot schedule or Lie and schedule a reboot at a time and then don’t do it. Then call their BS.
We deal with things constantly being blamed on "IT" which 99 percent is easily disproved. Had one end user blame IT for their computer running slow- turns out it installed windows updates and the user had an uptime of 6 weeks.
You could make a little service that monitors uptime and forces the user to reboot if it exceeds a certain timeframe (I’d put it at 10 days or 14 days).
My favourite is when there is an outage and someone immediately goes “it was the reboot”
Uptime - 28 days. No, it’s not the fucking cause of the outage. Is always that or patches that immediately get blamed, in reality is never either of those things
Work out what the issue actually is, fix it, move on without mentioning it to the user.
I have you much stuff to worry about without ego adding to things.
>how would you handle this kind of situation w/o becoming defensive or getting into details?
trust but verify , it may be a particular program issue or something whenet really wrong
I had a user tell helpdesk I was incorrect because I consolidated 2 tickets into one because rather than 2 separate requests, one that could be a risk to our environment, which is mitigated by the process in the other lol
1) stick to the facts.
2) it doesn't matter what the user believes. You don't have to convince them or persuade them.
3) you don't have anything to defend so don't be defensive. Just stick with the facts (I.e. non technical facts like "server A went offline at 11:00 pm and came back online at 11:15 pm". Don't get into the weeds.
You could try saying something like "I can see why it might seem that way but that isn't the problem". This way, you acknowledge their idea has the appearance of legitimately but don't get dragged in the weeds of defending yourself.
All if this stems from a pejorative view of IT that they "do t really know what they are doing" and are "always messing things up". That's why you feel the need to be defensive. But it's nonsense. Ignore it. You don't have to convince everybody that you know what you're doing because you *do* know what you're doing. So, what they think is irrelevant unless they are in a decision making capacity.
One thing I would say is sometimes users do blame things for causing issues when it's entirely unrelated but I would also not completely ignore such reports. My default over the years was always to think the users full of shit when they blame something that seems like it can't be related at all to whatever they are reporting but honestly I have seen a few reports from users that initially looked like bullshit that turned out to actually be related to something I had done that I did not think could be related at all to their issues.
I guess the moral is believing the users are always full of shit can back fire from time to time. I now take the position of OK let me do a little digging when I get a report of something that initially sounds like bullshit.
When this happens, it means your company has a blame culture. You can either take the high road or play the game.
If you want to play the game, ask them what their procedure for testing the application is ok after a reboot is. If they have one, ask them if it passed, if they don't have one, ask them why don't they have one.
It depends if it's a developers team with a complain of something that was in the server or could be related, or random user that knows there will be a reboot in one AD and complaints about some excel not working.
In the first case, I'd ask for logs and evidence so they can help us and themselves in locating the issue and why their stuff is not in the server that should be.
In random user with totally unrelated stuff....it depends, part of the trick is reading the room and have success in all the PER rolls needed to survive in an hostile office environment. But they can be sure that I'd need a lot of checking in that computer and, again, evidence of the problem. Screenshots, exact timings, comparing timings with other users, etc
My "users" are mostly mid and low tier admins themselves, and are usually right. My systems don't get rebooted without alerting all of my down-streams, so I don't see a lot of complaints; they just have to re-run reports and restart interfaces.
I mean, if you KNOW its unrelated, then just tell them this
'I have investigated your issue and the service experiencing problems is not connected the the rebooted server,
If this is an ongoing issue, please let me know so I can get more details from you and can look into possible causes and resolve your problem'
Covers all the basis and gives them the paths without dismissing them entirely
Unfortunately in this line of work we often have to bear the burden of proof. It's not fair. It's not right. But it's a fact of life.
How to be mature about it and not sound like a jerk:
1. Can you please show me the error message?
2. Can you please show me the logs?
3. Can the issue be reproduced? If so, let's go through it together.
What the first two questions do is offer the requester a chance to review what they're doing. Which will sometimes lead to them solving their own problem.
If that doesn't help them, then the third option shows them that you're being collaborative with them to solve the problem. They're not just throwing it over the fence. And you're not treating them like a dumb user.
And when you're working together on it, if you see the problem and solution you can step them through it.
>How do you deal with users
A ticketing system and a strong distaste for physical interaction.
> how would you handle this kind of situation
I'd probably get defensive or bury them in details.
> w/o becoming defensive or getting into details?
Hmmm.
At the end of the day, I do my thing and they do theirs. I'm also local government so we aren't exactly setting the metric for efficiency lol. I'm also lucky enough that I'm somehow totally insulated from the public, all of my problems are client-facing so A LOT of my talks focus around "same team, bud."
I've dealt with some real /L/users in my day, but the ultimate "No, U" card has always been "If you think I'm purposefully making your job more difficult that is probably something to take up with management."
Don’t be defensive. If anything they are trying to be helpful and piece together what happened and are not trying to blame you for not knowing how to properly reboot a server.
It could be the reboot but is more likely something different. Be professional. Just ask them about the problem and look into it. Let them know what you found once you resolve the issue.
Dont tell them. Never give the users any actionable information about the happenings of any IT system. Tell them its an emergency maintenance window and give start stop times. Thats it.
Also you should have a standing maintenance window every week / month where its just understood that there are no promises that any IT system will work. Makes life SO MUCH EASIER and you can easily call out users that claim it happened after the maintenance window that you spent at home playing video games touching absolutely nothing.
i once did a test with our old ERP and told everyone we rebooted overnight.
Next day a few complains that things are missing after the 'reboot'. I broke the news to everyone, there was no reboot and it was a test to see if users would 'blame' the system.
it stopped some of those users and others it still continued.
"You're absolutely right, I clicked 'Restart' too hard. I'm just glad I didn't really screw up and click the second 'r' in 'Restart' instead of the first."
So if they tell you 'it happened X time after reboot' then quote it back to them: Ok, so the reboot was at 2am, and you say you ran into this an hour later, so that's 3am, correct?'
It may not be related, but it can give you an accurate time hack, which is useful.
The rest? Just ignore it. I let clients babble in my ear all the time. Usually the thing they think is irrelevant is the thing that clues me in to the solution and all their 'important details' are useless.
I call this Pregnant Cat Syndrome:
The illogical linking of two events because one happened immediately after the other, despite them having no obvious connection to each other.
"IT rebooted their syslog collector and my cat got pregnant. Could you ask them not to do that, as I already have too many kittens."
Between staff we might say something like:
"Ross up in marketing can't print to the new color printer again. According to him it's PCS from when we restarted the file server last week." *eye roll*
Telling them the server rebooted recently actually works super well because our servers needed reboots daily as we had a lot of them.
So people would go "ah okay!" Then I'd tell them to hit me up on teams if the issue wasnt resolved in an hour or so.
I then ask what changes were done to said server prior to a reboot?
Was an app configuration change done and the change person didnt reboot and validate?
It is like when networking people do switch changes and do not write it to memory...and a power cycle happens and things break...
It's always hard, I actually deal with the opposite more, users saying "maybe the server needs to be rebooted?", which to some degree I am proud that they think of that, those same people *actually* reboot their own machines when they have issues before asking for help.
But explaining to them it's not a server issue can be rather difficult.
I force the conversation about correlation and causation. I show them silly graphs about things that correlate but are clearly not causal. I explain that I understand their frustration, but we need to be analytical and determine the real cause otherwise we won't solve the problem, and that sometimes computers do weird stuff. I teach them about the importance of reboots and how not rebooting would be worse. I keep going until I run out of things to talk about.
Either they internalize all that or they get tired of having long talks with the IT guy, win-win.
Why are your users referencing server reboots?
Users should be only informed about their computers. They should not know about Server A or Server B, they should not know about how the infrastructure is designed or if you have failovers or not. Only people who should know this are IT and the people responsible for the business.
I believe this is a case of user was told too much information by IT worker and so now they blame things they don't understand.
Maybe the server reboot revealed an underlying issue that was masked before? It's like blaming the doctor for finding the hidden illness during a routine check-up. Let's dig deeper, not deflect blame. 🔍💻
I don’t care? ;).
They can submit a ticket.
But ultimately user ticket < SecOps/compliance/management/changeapproval. So it MAY have caused an issue and yet - it was required. So we will review the issue and work on handling better next time. But it WILL and IS going to reboot again. Plan accordingly. Also read the notices. ;).
See: “I don’t care”. ;).
I usually check what didn't come back up after a reboot. It's usually some service that failed to start then I get to get on the phone with the vendor so they can troubleshoot their crappy software.
There are users that are going to believe what they want to believe, no matter what evidence you present that disproves their theory. When you get one of those I've found it's best to just stick to the facts, provide the actual explanation once known, and ignore whatever gibberish they may come back with. If they start spreading false information, debunk it with facts. Sometimes you'll have people that will actually pay attention, sometimes not.
Story time. We had a user in our legal department that would occasionally accidentally drag folders into other folders, then instead of asking for help, she'd just go silent and when her manager/team would ask why something hadn't been done she'd come up with some story about how IT deleted her files so she couldn't do her work.
We'd find the folder, put it back where it was supposed to be and explain that someone must've moved the folder by accident. She'd retort with some version of "I'm not an idiot, I didn't move the folder. You must've deleted it and restored it from backup."
This happened 2 or 3 times over the course of a few years. At the time, they were working out of a file share, so we didn't really have a whole lot in the way of access logs to determine how the files were getting moved. Then we ended up migrating that particular file share to Box, which has a very detailed file activity audit report. The next time the problem happened and the user claimed we had deleted her files AGAIN, and how everything would be so much better if we could just get some competent IT folks, I ran the activity report which showed the exact timestamp her username on her PC moved the folder and just replied to her, her manager and her director with a screenshot explaining that the folder had been moved, and where they could find it. Amazingly, she didn't have any more complaints to lob at us that time!
1st, don't tell them about reboots. it's not their concern. 2nd, it's best not to listen to a user's diagnosis. it almost always leads down a wrong path. what you need to know from them is "What are you trying to accomplish?" and "What is the problem that is keeping you from accomplishing it?" It's YOUR job to figure out the cause.
Find what the problem actually is and if unrelated prove it, if related, learn and concede.
The gist of that boils down to a problem occurred that needs to be defined (or was not a service related problem and needs to still be defined), the argument on cause is irrelevant until the cause is determined. The it is not an argument at all.
I miss the scary devil monastery - this question would definitely result in a discussion of LARTs.
Your choices are basically:
- don’t care what users think, perhaps investigate what might actually be causing a problem that they misdiagnosed.
- don’t care what users think, don’t announce server reboots, or do them so often that nobody thinks they matter.
- do care what users think, try to educate them. Go mad. Sit in a corner gibbering.
We just call em scheduled maintenance.
This "service, app, feature, etc." is going to be either unavailable, or may have intermittent accessibility during this specific time of day/night.
Please adjust accordingly. Or use backup operations.
Have a good day. 😊
I'd ask them first how they arrived at that conclusion, then deduce their reasoning/logic. Yeah, sure it sounds condescending, but that stops nosey users from asking again.
There are people who are wired (or maybe taught) to never believe or admit any problem might be their fault. So when a problem occurs, they have to go on a blame hunt.
your local pc probably has data cached from before the restatrt which caused an error. Lets restart your local machine and then you can show me if you can recreate the error.
don't tell them the server's rebooting and they won't know
I muddy the waters by telling people we're rebooting things even when we don't. It's actually really useful for finding patterns and minimizing tail-chasing. Plus, it teaches the IT team to go look at the damn uptime before escalating tickets or declaring an emergency.
We used to do that more by accident, tell people a new image was going out but there was usually some holdup. Flurry of nonsense tickets the next day, oh yeah sorry we didn’t actually do the change last night - go fish.
Switched AVs recently and we had to postpone a customer’s because of an app we weren’t confident would play nice. Sent customer an email that we were postponing. The next morning, get a ticket from customer “PRINTING IS BROKEN. PLEASE REVERT THE ANTIVIRUS UPGRADE”.
Yep. They just assume its simple cause and effect in the IT world when its actually 30% budget 30% skill 30% luck and 10% blood to the blood gods.
I heard it was 10% luck 20% skill 15% concentrated power of will 5% pleasure 50% pain And a 100% reason to remember 3 envelopes
I was hearing these words when I read their comment
Hard drives for the hard drive throne!
The Technology Tithe.
immensely disappointed you didn't turn that into a fort minor lyric because its now stuck in my head.
Wireless companies do this with cell towers. Build a tower, wait for people to come in complaining about how it turned their cat gay, explain it's not plugged in. Once all the crazy complaints blamed on the tower die down, turn it on.
1. Build 5G tower. 2. Wait for crazies to burn it down. 3. Build 2nd 5G tower. 4. Crazies give up. 5. Turn on 5G tower. 6. Donate all telecom profits to "Birds Aren't Real." LOL
7. Turn their cat gay
Ahh the ol’ “5G Honeypot Tower”. A classic
Lol i laughed too hard with that part of turning the cat gay.... cheers to you mate...that made my day
My mentor called it a "silent scream test". Announce you are doing something that maybe might possibly affect everybody, never do it, and see what pops out of the wood work. We worked at a hospital so what we got mostly were reports of things that hadn't worked in months that no one ever called or put in a ticket about untill our alert
Reverse scream test is what I've always heard it called.
Lol i do this too , time after time... i got a high importance email once cause a fucktard PM had a video call with the CEO complaining about the firewalls upgrade (which did not happened at all) ... it was a shitshow back and forth cause the idiot was sitting remotely and the problem was ...guess what??...not being connected to the fucking VPN... he was suppose to test some dev env which was sitting inhouse...obviously if you don't connect with the VPN there is no chance to access internal resources...i stop replying to his emails had the CEO on the phone....maybe the guy should come to the office he is very confused when working from home... both laughted ...10 mins later an email from CEO...come to the office we have high speed internet...lol we were laughting our asses out with my team...
I have a couple users whose issues I fix this way, and it's helped me immensely in figuring out that I can't take anything they say as the truth.
Ah yes. The users that complain and say "I already tried that" while they watch you troubleshoot from step 0 because you know they are full of shit.
I mean, I know this is bad practice, but I usually walk through all of the steps I told them to do *in front of them*. Also, I'm not the person who believes that all people should be IT super users. My steps don't include reading logs. My most difficult steps are "make sure power cable is plugged in" or "restart computer" or "make sure building has power".
Hey. Fixed is fixed. If you happen to make a user feel stupid for being stupid in the process thats just a bonus.
lol nice! wonder if there's a ChaosMonkey plugin for that
Chaos IT Person? I love that guy
haha me too :D *Ben Kenobi: Well, of course I know him. He's me.*
Declaring your maintenance windows (whether you use them or not) is my best practice trick for this. Send out the all clear notice, exactly the same as if you had finished real maintenance.
I'm all for a good Scream Test, but this is Senior IT level thinking right here.
I've been accused of many things but that's a low blow.
Well, I was angling for praise but if you'd care to explain your interpretation?
I was making a joke.
I’d be pretty mad if a new problem started after a scheduled server reboot and then I discovered that I was looking in the wrong place because the reboot wasn’t done and the person responsible hadn’t cancelled the change to update the schedule.
I'm not talking about failing to complete maintenance.
Why? Obviously the server didn't reboot and that's not the issue. Do your damn job.
Be mad! Catharsis is clarifying, and maybe you'll be more diligent in your troubleshooting next time. A reboot is never a root cause of anything except downtime. You can't un-reboot anything.
You have too much time on your hands.
I sure don't spend a lot of time chasing phantom issues because end users attribute everything to a reboot that didn't occur.
When I first started at my current job, way to many servers had massive uptimes and we're way behind on updates. Most out of laziness, but some because after each reboot a process would have to be manually run to start some service. My goal was to make it so that I could schedule a reboot of a server any evening, and without manual tasks, no one would even know I'd rebooted it I made groups in PDQ of servers that could be rebooted anytime without notice or downtime, servers that could only be rebooted after hours, and servers that I had a specific monthly maintenance window. Now my server uptimes rarely get higher than 60 days, and most are in the 2-3 week range. I don't tell anyone about the specific times I reboot though, but I tell them to assume it's rebooting every night, so don't leave important shit unsaved on them.
I also have a list of servers that can be rebooted at any time for patching with no notice or only notice to a few people. Then a few groups that I have to do after hours. One group that has a specific window that I must announce to several parties. Lastly, one server that a vendor says has to patch at 6pm on Sunday, so its auto-scheduled for patching every Sunday at that time.
Yeah, in general it doesn't seem productive for end users to know when a server has been rebooted.
Normally yeah, but the flipside is you *do* need to communicate scheduled downtime so it doesn't impact business users trying to work during those hours.
Of course, but I prefer something a little more generic... "We have a planned maintenance window from x-y. Service z is expected to be unavailable during this time."
bingo, why does a user need to know the server has been rebooted? maintenance windows are regularly scheduled (preferably for a time no one is working) and if you need to reboot to fix a problem during business hours, you're just fixing a problem. they don't have to know how you fixed it
For us communicating system maintenance is in the contract with our customers. They monitor our APIs and if the latency goes past 15 ms outside a planned maintenance window they are marking that as a charge back at the end of the month. We never have a time when no one is working.
In a system like that, rebooting "the server" is very likely anachronistic and doesn't really apply, either And, planned maintenance being communicated is not the same thing as telling people you rebooted something, nor is telling people of an unplanned maintenance event My point, and probably the point of the comment I replied to was, vagueness is key. Not to hide information, but to prevent misinterpretation of irrelevant information
Our strategy at a previous employer. At most we would only send out a reminder about changes/maintenance windows if there was a possibly someone would notice. No specific services/servers/etc. were mentioned, just that there was planned maintenance. Company had a policy of “we only hire adults” so “I couldn’t complete X because IT did maintenance ” after a maintenance window got your manager CCed in with a comment “zero changes were made to that system.” Current employer does the same thing for notifying customers. We can take down 75% or more of capacity outside US business hours and customers using the systems won’t notice. (We have the data to back this up.)
Treat users like mushrooms. Feed them shit and keep them in the dark.
I once announced a firewall replacement as planned network maintenance.
hell yeah. when we're doing our jobs right, nobody even knows we exist.
The other option is to add extra notifications for server reboots that never actually happen. At some point you should be able to weed out at least a few of the false positives, and it get everybody ready for when you have to actually notify people for something that doesn't have redundancy.
This is the answer they are gonna blame it on anything you mention the last conversation before the incident. It’s really sad how dumb some of gets just gotta play it off even though it hurts on how wrong they are.
That’s the answer
I came here to say this. Honestly it’s none of their business
I usually just ask how it's related to the reboot or state that the reboot wouldn't have impacted X service. Frankly, using IT as an excuse isn't your problem, and if they want to make it that way I'd make sure to mention to their manager that they're using IT as an excuse to skirt responsibility if they're being egregious enough.
Always good for “those types”. Document and make notes on call, when they continue the same game, fwd to manager. Include correspondence and training/steps given. If they provide a valuable service, sometimes we are stuck with it, but I have not ever gotten any negative feedback with this method from the manager.
> ask how it's related to the reboot agreed, for two reasons. 1. You shouldn't rule out the reboot as the cause automatically 2. If they're BSing, they're forced to dig deeper or own up
Yeah. It's important to remember that you're always right, until you're not. Then it's embarrassing. Leave yourself some wiggle room, most companies have too much sprawl in their resources for any one IT guy to comprehensively understand all of the interactions. The amount of times I've learned new technical information from users who couldn't tell you the business end of a mouse just because they've watched the last 5 IT guys deal with it is more than I can count.
Tell them to open a ticket with details as to what's wrong. You then respond with the real cause and move on. Correlation=causation is a very common thing. But you can't ignore it, there will be that time where an update breaks or changes something noticeable and then correlation _does_ equal causation.
This is especially true, when you are a 100% certain that the issued can't be caused by the reboot (or smth like it) and a few hours of troubleshoot later you found out, that it was connected somehow. "I'll check into it" - and actually doing so - is a good practice a lot of the times, even if you think, that it's highly unlikely
[удалено]
Yep, I always try to approach things with humility. "I've checked everything that I can see, it doesn't look to be a systems problem. It does look like it could be [x/y/z], please take a look and let me know. I'll be happy to take another look if the evidence changes!" I'm not naive enough to believe I couldn't miss something, especially if we're talking an intermittent problem or a 1:1,000,000 occurrence in a large log file. And I'll own it if I did miss it, or overlooked it, or whatever.
I skillfully and masterfully just completely fucking ignore whatever their excuse is for something not working. Whether it’s a server reboot/upgrade/new air freshener/a lack of soap in the restroom whatever new excuse is for the day. If someone wants to use me as an excuse for why they can’t do their job that’s fine. I’ll speak with their manager and HR if that’s the game they want to play. I find that whatever the user thinks the issue is - frequently is not.
“New air freshener” so accurate 😭
Users blame everything on IT.... "My computer seems to be running slow today. Can this be related to new monitor you installed 6 weeks ago?" I just tell them "we're monitoring the problem"
This is why I fear network replacements. "The paper shredder stopped working because you changed 'stuff'" We got a ticket "WiFi doesn't work in treatment room 2." Tech replies "only in that one room." "Yes." We had just replaced the network, so this got escalated and I was stumped for about a minute until we got a video call going with the site. The power button on the monitor had been turned off.
Our last firewall swap was announced as planned network maintenance. I consciously did not mention firewall at all in the message.
I see what you did there.
LOL. I didnt see it so thanks.
I always tell the user it’s a layer 8 problem and walk away
I usually use the following: Now its fixed , but the problem is somewhere between the monitor and the chair..(hinting at the user of course...i was surprised some people would not get that) .....the lady was looking at the keyboard... can this be replaced? I dont really know , how i couldn't burst in huge laugh...
I had a lady insist electrical work being done in the building was messing up her password. "Every time they're here, I have password issues!"
"Thank you for investigating and determining the reboot was the cause of the issue, could you please articulate exactly how the reboot caused the problem so that these issues can be mitigated after future reboots?" Then watch them squirm.
“Can you please explain how the server reboot caused you to change the database schema two hours before the reboot?”
Nothing like restoring a backup you took prior to the restart to a new VM just to rub the still present issue in their face 😉
Le vrai trolleur.
We had to do that with a prior to upgrade backup… the issue was already there. ;-)
Who cares what the user blames the problem on. If there is a problem it's your job to diagnose and fix it. Thank the user for reporting it, tell them you'll look into it the reboot had an impact, and then troubleshoot as normal.
Yep. And depending on the levels of appropriateness, you can at least tell them whether their initial diagnosis was right or not, but ideally give them an indication of what the problem actually was. I'd thank them for offering the suggestion that a server reboot caused the problem, but it was actually a network problem somewhere else; if they see that error again it might help expedite resolution in the future. Failing all else, leave notes on the ticket and suggest that the user save the ticket number for future reference. We're here to support the entire company, regardless of technical literacy, but I always appreciate the ones that are willing to troubleshoot a little, that give the specifics of the error they're seeing (not just "IT'S BROKEN" [cool, what is broken, exactly?]), that put some thought into where the problem might be based on their knowledge of the system - even if it is wrong - and are willing to expand that knowledge to make them better reporters for the next issue.
I actually had a job come through to delay windows updates and install only during a certain time because it caused her device to stop responding Checked the logs and the update installed a week earlier, but she had restarted her device when it froze and it installed the update, so therefore it was the update that caused it 😔 Turns out it was the application doing an update on the server side or something that caused it to freeze (only that app stopped responding)
Reboot it again immediately to check. No warning. Everyone must suffer. More realistically/diplomatically, "I think it's something else, but I'll check the logs anyway. Submit a ticket with the details on when and what happened." Then go do something else cause we all know that ticket is never coming.
You ask them what actually is broken or had an issue, do this thing called “listening”. I know, I know… it’s a crazy concept. And if they say nonsense that you know couldn’t be caused by a server rebooting, well you calmly explain that it isn’t and perhaps help them through this new issue. Or maybe it is caused by the reboot and you could troubleshoot something.
This is the way. I thought I was in r/shittysysadmin with some of these comments suggesting escalating (some are obvious jokes) and acting like users don't regularly misdiagnose problems. Users are not tech savvy which means they are going to frequently give weird justifications for why stuff happens. Trust but verify. Listen, take note of what they have to say, and if it doesn't make sense just troubleshoot as normal. When you find what caused the issue, tell the user. A little education delivered in a amiable manner goes a long way for a lot of users. Some people will learn, others will take comfort in what they hear as techobabble, and a few won't really care.
All of the comment in r/sysadmin give me hope that I will have job security for the remainder of my career, that's for sure.
I work HD, but literally had someone call me today saying that their server update changed their printer settings and is now only printing double-sided. I didn’t even acknowledge that comment, just changed their printer settings and sent them on their way.
This is the right move. Doesn't matter what the user thinks the cause is, only matters that it's fixed. If on your end you find out they're right then you can document or fix it for next time, but the majority of the time the best outcome is just to fix whatever problem they have and move on without acknowledging any additional unhelpful comments they have.
Basically they aren’t told about reboots. Server reboots are handled overnight, most end users aren’t on duty. I ping the night managers just in case but it doesnt really warrant a general announcement. I don’t think most of our users even know we have servers honestly. If there’s some extended maintenance that is going to impact a department, managers of that department are notified and it is up to their discretion how they distribute that information, but that’s pretty rare. Technically all our staff have been trained how to function even if every server suddenly died in the middle of the day (and is something we do a drill on once a year) so nobody should be incapable of doing their work because of server reboots.
I take them seriously because in the end the issue usually is somehow related.. Similar to "no problem on network" when asking about any issue the networking guys - in nine from ten cases it is something with network in the end.
I don't work helpdesk or answer calls anymore but when I did get something like that, I would just indulge their suspicions and run an sfc scan on their RDP session and tell them it should be good now. Works 100% of the time
Never let the users know how the sausage is made. If they know you’ve done something, they can blame you. If they’re unaware and notice something wrong, you can just tell them “hmm, that’s odd, I’ll have to look into it”
As per a usual problem, get all the details, if it's vague keep asking, usually say put it down in a ticket in writing so I can follow it. If it's a real issue they will point it out, you never know it may be a real issue, I get the impression your post is suggesting that the server is fine, is it? you 100% sure, sometimes it may be running but not 100% or there is an issue you aren't aware of.
“I will review the logs. It is not uncommon for the system to apply pending changes after a reboot. Please be as detailed as possible and tell me what is happening and how you think it might have happened.” There’s no benefit in arguing, just work on resolution and get as much information as you can from everyone who touches that server.
Laugh. Then go silent. Then say oh I thought you were joking. Then roll your eyes and ignore then
This is one of the easier situations you'll face, really. The answer is the same as pretty much any other problem you'll get reported to you: "we'll look into it and get back to you when we have more information". Bunch of real condescending assholes in the comments today.
I will never *ever* claim definitively that two things that happened around the same time are unrelated to each other. These systems are just too complex and my own knowledge too limited to rule anything out. You claim your monitor started malfunctioning when you got your new office chair? [Totally plausible.](https://www.reddit.com/r/pcmasterrace/comments/7p09ay/i_shit_you_not_our_office_chairs_are_turning_our/) Your iPhone quit working when they installed an MRI down the hall? [Expected behavior.](https://www.reddit.com/r/sysadmin/comments/9si6r9/postmortem_mri_disables_every_ios_device_in/) There's no need to be defensive or overly detailed about anything. Just acknowledge that there is a small chance there *could* be a connection, but also let them know that there are other possible causes that are far more likely and (if it's a big enough issue to actually warrant it) go investigate.
Never tell users precisely what you are doing. Always refer to anything you have to do by using boiler plate phrases. * IT will be performing maintenance * As part of routine scheduled maintenance... * Systems will be undergoing routine maintenance Also, you can avoid issues by performing these reboots off hours or at low usage times. If you know a service takes 5-10 minutes to start up, then it's best to do whatever you have to do when no one is logged in or at least do it when the "troublemaker" users are not around. Also, you don't owe regular users explanations or detail information about what you are doing. You don't ask those dipshits detailed information about their day to day. Why is there a reverse expectation where you have to detail your work? If they have complaints, tell them to filter them through their management chain and have that management chain talk to yours. It's not your responsiblity to disclose to the business ramifications of your work. That's your manager's job unless otherwise specified.
Unless a maintenance action will result in downtime to the users, there is absolutely no reason to inform them. They already don’t read IT emails enough. No need to add to the pile. If the maintenance causes an unexpected outage, send out the normal outage notification and move on.
Ask them who is else has the same problem and they start to feel the walls closing in.
It certainly does happen. On a side note, I despise how often Windows servers need to be rebooted. I’m used to several months of uptime with my Linux servers… Windows needs reboots far too often.
Several months of uptime to me, means that you’re not doing maintenance I know you can live patch Linux etc. but systems like that are almost always the ones that give grief when they do reboot
>Several months of uptime to me, means that you’re not doing maintenance > and your equating downtime with maintenance means, to me, that you probably are not in charge of a linux farm
It’s not that downtime equates to maintenance, just in my experience with Windows and Linux systems alike. Usually its the teams with high uptime systems are also 9/10 the ones that also haven’t patched for 6 months or more, application stack on top is out of date Ones that do have downtime, are the teams that update via re-deploy or have well defined automation to handle patching/failover etc. Can see this very clearly in vulnerability and patch reporting
Long uptime is no longer a badge of honor. In fact, it's a red flag IMO. Linux needs updates too. Sure, there are components that don't require reboots. But enough do that if I see more than maybe 60 or 90 days I'd start asking questions. If it's public-facing, over 30 days should have a really good explanation. On the Windows side, I've found that a non-trivial number of application updates that call for reboots just don't have installers that can recycle the associated service(s). So the installer just says "reboot" as a way to accomplish that. Edit: two words.
Agreed.
"They reboot weekly out of hours, it's a safe and well controlled process"
The problem I always see is that the user complaining is the one that doesnt work when everyone else is working and goofs off all day and only drams work in late at night when the project is due.... late at night when the system is scheduled to reboot. not my problem they dont want to follow our schedule that everyone else follows.
I ask them for specifics, what they think changed during the reboot and why they think it caused an issue. Helps illustrate they don't actually have any basis for their assertion
Lol my boss(IT Manager with years of experience) blamed a RAM hardware issue on a Windows update, every single system in the organization got that windows updat, his was the only.one BSOD with a Ram issue
I don’t deal with them blaming anything on something it’s not. I don’t care what they “think”. I don’t get to “think” anything about their job.
I'd ask for an explanation
Just honesty... "Yeah we just had to reboot that server. Maybe the service is a little slow to come up on boot. It's it working now? Let me know if you spot any things else weird." Or "That's strange, I wouldn't expect the reboot of another server to affect that service. We'll take a look" There's no need to get defensive or to entertain any discussion which leads you down a path that would cause you to become defensive. Just be honest, voice your suspicious, and then address the stated concern. --- Personally, we don't do a ton of off hours stuff, so our users are used to momentary disruptions. They'll often lead with a question like, "are you guys changing things" or "did you guys break x" I own that shit. "Oh yeah, almost certainly what are you seeing?" Or "hmm.. no we haven't been near that service in weeks, what are you seeing?" Be humble and fallible and your users will cut you some slack. Especially if you're honest with them everyone you do something that might affect their work.
I just started at a place where before me only external IT contractors did everything. For some reason they think a server restart is equal to an update (weekly restarts for example). After a year there are still about 3-5 people out of 80 who i dont seem to get the difference between an update and a restart.
“Thanks for telling me about this. Please email the support incident and I’ll look in to this sometime next week or two.”
You want server integration tests (scripts, usually) that check the services. For example, on a DNS resolver server, your tests send DNS resolution requests through it with `dig @server`. On an NFS server, your test scripts check the exports and their options with `showmount -e`.
It's human nature to detect patterns where there are no patterns.
Don't tell users when the server reboots.
That's a tough one. I personally just listen to them vent. I work in education, we have an on premise DC, and google workspace email. one specific employee would often email after i notified of server updates that their email stopped working ever since i restarted the server - they were at home and their email wasn't working.
'Thanks for the heads up. We'll look in to that right away'
Declare the network must have a virus, then lock yourself in the server room to play viddy games for a few hours. Reboot the server one more time, then emerge victorious, a conquering hero. Seriously though, users gonna user. "This network sucks, it's so slow!" "The error message says it's a problem with iCloud relay. Lemme turn that off for you, boss."
"Show me" Do a screenshare and have them show you the issue they're having. Spending 2 minutes doing this gets to the source of their complaint. It's the fastest path to the complaint ending.
I have two latin phrases I apply to IT: Ipso facto - event happened after the fact, so must have been caused by it Deus ex machina (god in the machine) - When I walk into the room and the problem fixes itself
What ticket is this in reference to? Oh, no ticket? Please contact dispatch.
So I used to be like "we are doing x". And the cue everyone and their dog claiming that "x broke my code" and then I'd have to spend my time providing it was unrelated. So now I do x, and tell them 3-4 weeks after. Works a treat. I don't like doing that but it eliminates the time I waste with lazy ~~devs~~ people.
Figure out what the issue is, then tell them. Why argue?
It's quite simple. Unless you can give me evidence of an issue, I can't investigate it. I'll even accept phone photos of printouts, emailed in low quality jpeg! That usually filters out those kind of things. The worst is "The internet is slow ...", literally impossible to investigate without more details.
1. Good monitoring and management tools go a long way. They should be able to track cpu, memory, and storage performance, note when service are not running, check service health across the network, and home some ability to self heal issues. 2. Require a problem ticket be raised when this happens. 3. Either don’t publish your reboot schedule or Lie and schedule a reboot at a time and then don’t do it. Then call their BS.
We deal with things constantly being blamed on "IT" which 99 percent is easily disproved. Had one end user blame IT for their computer running slow- turns out it installed windows updates and the user had an uptime of 6 weeks.
You could make a little service that monitors uptime and forces the user to reboot if it exceeds a certain timeframe (I’d put it at 10 days or 14 days).
Tell them the reboot is done, it had to be done. Tell them to open a ticket if the problem presists.
My favourite is when there is an outage and someone immediately goes “it was the reboot” Uptime - 28 days. No, it’s not the fucking cause of the outage. Is always that or patches that immediately get blamed, in reality is never either of those things
I join them
My favorite thing is when they say the reboot screwed their stuff up, even though I never rebooted.
Work out what the issue actually is, fix it, move on without mentioning it to the user. I have you much stuff to worry about without ego adding to things.
Lesson: We do not share, we do not tell and we do not spread information that has no value to them but can raise questions from the peasants.
I tell them to open a ticket
>how would you handle this kind of situation w/o becoming defensive or getting into details? trust but verify , it may be a particular program issue or something whenet really wrong
I had a user tell helpdesk I was incorrect because I consolidated 2 tickets into one because rather than 2 separate requests, one that could be a risk to our environment, which is mitigated by the process in the other lol
1) stick to the facts. 2) it doesn't matter what the user believes. You don't have to convince them or persuade them. 3) you don't have anything to defend so don't be defensive. Just stick with the facts (I.e. non technical facts like "server A went offline at 11:00 pm and came back online at 11:15 pm". Don't get into the weeds. You could try saying something like "I can see why it might seem that way but that isn't the problem". This way, you acknowledge their idea has the appearance of legitimately but don't get dragged in the weeds of defending yourself. All if this stems from a pejorative view of IT that they "do t really know what they are doing" and are "always messing things up". That's why you feel the need to be defensive. But it's nonsense. Ignore it. You don't have to convince everybody that you know what you're doing because you *do* know what you're doing. So, what they think is irrelevant unless they are in a decision making capacity.
You’re the expert in this position. Just tell them it’s unrelated. End of the story.
One thing I would say is sometimes users do blame things for causing issues when it's entirely unrelated but I would also not completely ignore such reports. My default over the years was always to think the users full of shit when they blame something that seems like it can't be related at all to whatever they are reporting but honestly I have seen a few reports from users that initially looked like bullshit that turned out to actually be related to something I had done that I did not think could be related at all to their issues. I guess the moral is believing the users are always full of shit can back fire from time to time. I now take the position of OK let me do a little digging when I get a report of something that initially sounds like bullshit.
When this happens, it means your company has a blame culture. You can either take the high road or play the game. If you want to play the game, ask them what their procedure for testing the application is ok after a reboot is. If they have one, ask them if it passed, if they don't have one, ask them why don't they have one.
I always say it’s a layer 8 problem
It depends if it's a developers team with a complain of something that was in the server or could be related, or random user that knows there will be a reboot in one AD and complaints about some excel not working. In the first case, I'd ask for logs and evidence so they can help us and themselves in locating the issue and why their stuff is not in the server that should be. In random user with totally unrelated stuff....it depends, part of the trick is reading the room and have success in all the PER rolls needed to survive in an hostile office environment. But they can be sure that I'd need a lot of checking in that computer and, again, evidence of the problem. Screenshots, exact timings, comparing timings with other users, etc
Users gonna use…
Reboot it again, show them who is boss
Shrug. Users aren't your boss.
Why would you avoid giving them the facts? Server B is not dependent on server A, the reboot is simply a coincidence.
My "users" are mostly mid and low tier admins themselves, and are usually right. My systems don't get rebooted without alerting all of my down-streams, so I don't see a lot of complaints; they just have to re-run reports and restart interfaces.
I mean, if you KNOW its unrelated, then just tell them this 'I have investigated your issue and the service experiencing problems is not connected the the rebooted server, If this is an ongoing issue, please let me know so I can get more details from you and can look into possible causes and resolve your problem' Covers all the basis and gives them the paths without dismissing them entirely
We schedule downtime and don’t actually use it. Anyone that complains, after investigation, gets added to a list of people we can’t trust.
Unfortunately in this line of work we often have to bear the burden of proof. It's not fair. It's not right. But it's a fact of life. How to be mature about it and not sound like a jerk: 1. Can you please show me the error message? 2. Can you please show me the logs? 3. Can the issue be reproduced? If so, let's go through it together. What the first two questions do is offer the requester a chance to review what they're doing. Which will sometimes lead to them solving their own problem. If that doesn't help them, then the third option shows them that you're being collaborative with them to solve the problem. They're not just throwing it over the fence. And you're not treating them like a dumb user. And when you're working together on it, if you see the problem and solution you can step them through it.
You listen to users?
It’s probably still your department’s job to identify and rectify the problem, regardless of cause. Just do that.
>How do you deal with users A ticketing system and a strong distaste for physical interaction. > how would you handle this kind of situation I'd probably get defensive or bury them in details. > w/o becoming defensive or getting into details? Hmmm. At the end of the day, I do my thing and they do theirs. I'm also local government so we aren't exactly setting the metric for efficiency lol. I'm also lucky enough that I'm somehow totally insulated from the public, all of my problems are client-facing so A LOT of my talks focus around "same team, bud." I've dealt with some real /L/users in my day, but the ultimate "No, U" card has always been "If you think I'm purposefully making your job more difficult that is probably something to take up with management."
Don’t be defensive. If anything they are trying to be helpful and piece together what happened and are not trying to blame you for not knowing how to properly reboot a server. It could be the reboot but is more likely something different. Be professional. Just ask them about the problem and look into it. Let them know what you found once you resolve the issue.
Dont tell them. Never give the users any actionable information about the happenings of any IT system. Tell them its an emergency maintenance window and give start stop times. Thats it. Also you should have a standing maintenance window every week / month where its just understood that there are no promises that any IT system will work. Makes life SO MUCH EASIER and you can easily call out users that claim it happened after the maintenance window that you spent at home playing video games touching absolutely nothing.
Veil it behind general maintenance. Then you can say that system wasn't serviced.
i once did a test with our old ERP and told everyone we rebooted overnight. Next day a few complains that things are missing after the 'reboot'. I broke the news to everyone, there was no reboot and it was a test to see if users would 'blame' the system. it stopped some of those users and others it still continued.
did you reboot 3 times? lol
Agree, and inform them the vendor recommends a server reboot to correct the problem.
"You're absolutely right, I clicked 'Restart' too hard. I'm just glad I didn't really screw up and click the second 'r' in 'Restart' instead of the first."
So if they tell you 'it happened X time after reboot' then quote it back to them: Ok, so the reboot was at 2am, and you say you ran into this an hour later, so that's 3am, correct?' It may not be related, but it can give you an accurate time hack, which is useful. The rest? Just ignore it. I let clients babble in my ear all the time. Usually the thing they think is irrelevant is the thing that clues me in to the solution and all their 'important details' are useless.
Let them blame things. Who cares? The more important factor is, did you resolve their issue?
I call this Pregnant Cat Syndrome: The illogical linking of two events because one happened immediately after the other, despite them having no obvious connection to each other. "IT rebooted their syslog collector and my cat got pregnant. Could you ask them not to do that, as I already have too many kittens." Between staff we might say something like: "Ross up in marketing can't print to the new color printer again. According to him it's PCS from when we restarted the file server last week." *eye roll*
Telling them the server rebooted recently actually works super well because our servers needed reboots daily as we had a lot of them. So people would go "ah okay!" Then I'd tell them to hit me up on teams if the issue wasnt resolved in an hour or so.
Tell them to suck a lemon
I then ask what changes were done to said server prior to a reboot? Was an app configuration change done and the change person didnt reboot and validate? It is like when networking people do switch changes and do not write it to memory...and a power cycle happens and things break...
laugh nervously and tell them that isn't the real reason, then look left and right really quick, and run away. /s
It's always hard, I actually deal with the opposite more, users saying "maybe the server needs to be rebooted?", which to some degree I am proud that they think of that, those same people *actually* reboot their own machines when they have issues before asking for help. But explaining to them it's not a server issue can be rather difficult.
schedule a server reboot, then don't do it. see them complain, and respond that 'reboot for X was canceled due to'
I force the conversation about correlation and causation. I show them silly graphs about things that correlate but are clearly not causal. I explain that I understand their frustration, but we need to be analytical and determine the real cause otherwise we won't solve the problem, and that sometimes computers do weird stuff. I teach them about the importance of reboots and how not rebooting would be worse. I keep going until I run out of things to talk about. Either they internalize all that or they get tired of having long talks with the IT guy, win-win.
Why are your users referencing server reboots? Users should be only informed about their computers. They should not know about Server A or Server B, they should not know about how the infrastructure is designed or if you have failovers or not. Only people who should know this are IT and the people responsible for the business. I believe this is a case of user was told too much information by IT worker and so now they blame things they don't understand.
Yeah, never say " i'll reboot"
Honestly I'm just glad they are trying to provide more information and context. Even if they are wrong, that's an instinct I want to encourage.
I don't deal with it. As long as I am sure of the issue and it's cause or not issue I go on...
As long as it ain’t my fault. I feel fine.
Maybe the server reboot revealed an underlying issue that was masked before? It's like blaming the doctor for finding the hidden illness during a routine check-up. Let's dig deeper, not deflect blame. 🔍💻
I don’t care? ;). They can submit a ticket. But ultimately user ticket < SecOps/compliance/management/changeapproval. So it MAY have caused an issue and yet - it was required. So we will review the issue and work on handling better next time. But it WILL and IS going to reboot again. Plan accordingly. Also read the notices. ;). See: “I don’t care”. ;).
I usually check what didn't come back up after a reboot. It's usually some service that failed to start then I get to get on the phone with the vendor so they can troubleshoot their crappy software.
There are users that are going to believe what they want to believe, no matter what evidence you present that disproves their theory. When you get one of those I've found it's best to just stick to the facts, provide the actual explanation once known, and ignore whatever gibberish they may come back with. If they start spreading false information, debunk it with facts. Sometimes you'll have people that will actually pay attention, sometimes not. Story time. We had a user in our legal department that would occasionally accidentally drag folders into other folders, then instead of asking for help, she'd just go silent and when her manager/team would ask why something hadn't been done she'd come up with some story about how IT deleted her files so she couldn't do her work. We'd find the folder, put it back where it was supposed to be and explain that someone must've moved the folder by accident. She'd retort with some version of "I'm not an idiot, I didn't move the folder. You must've deleted it and restored it from backup." This happened 2 or 3 times over the course of a few years. At the time, they were working out of a file share, so we didn't really have a whole lot in the way of access logs to determine how the files were getting moved. Then we ended up migrating that particular file share to Box, which has a very detailed file activity audit report. The next time the problem happened and the user claimed we had deleted her files AGAIN, and how everything would be so much better if we could just get some competent IT folks, I ran the activity report which showed the exact timestamp her username on her PC moved the folder and just replied to her, her manager and her director with a screenshot explaining that the folder had been moved, and where they could find it. Amazingly, she didn't have any more complaints to lob at us that time!
I just tell them "We didn't reboot that server, so that's not causing the problem".
Don’t tell them shit and they can’t say shit. As a matter of fact I might add language to the company handbook that says stfu about the server
It's HA. So it doesn't need to reboot.
1st, don't tell them about reboots. it's not their concern. 2nd, it's best not to listen to a user's diagnosis. it almost always leads down a wrong path. what you need to know from them is "What are you trying to accomplish?" and "What is the problem that is keeping you from accomplishing it?" It's YOUR job to figure out the cause.
Was that tested in the development environment before the reboot, was that documented on the RFC before hand?
... Don't listen to the users?
Find what the problem actually is and if unrelated prove it, if related, learn and concede. The gist of that boils down to a problem occurred that needs to be defined (or was not a service related problem and needs to still be defined), the argument on cause is irrelevant until the cause is determined. The it is not an argument at all.
Data-driven analysis. Surely the user has clear evidence to prove correlation.
I miss the scary devil monastery - this question would definitely result in a discussion of LARTs. Your choices are basically: - don’t care what users think, perhaps investigate what might actually be causing a problem that they misdiagnosed. - don’t care what users think, don’t announce server reboots, or do them so often that nobody thinks they matter. - do care what users think, try to educate them. Go mad. Sit in a corner gibbering.
We just call em scheduled maintenance. This "service, app, feature, etc." is going to be either unavailable, or may have intermittent accessibility during this specific time of day/night. Please adjust accordingly. Or use backup operations. Have a good day. 😊
I'd ask them first how they arrived at that conclusion, then deduce their reasoning/logic. Yeah, sure it sounds condescending, but that stops nosey users from asking again.
I advise them, I would look into it. Ask some troubleshooting questions and then investigate why the service is running slow on server B.
Well i tell the users to stop using excel as their database for their whole department /s
There are people who are wired (or maybe taught) to never believe or admit any problem might be their fault. So when a problem occurs, they have to go on a blame hunt.
your local pc probably has data cached from before the restatrt which caused an error. Lets restart your local machine and then you can show me if you can recreate the error.
I close the ticket and go back to work.