IT Department Venting (1 Viewer)

[coopersdad]

So a quick backstory, I'm director of financial planning and analysis for a large brand under a corporate umbrella. We got purchased about 5 years ago, and in the "transition" most of our IT department was let go, and the corporate IT department took over. Then about 50% of that work was eventually outsourced. So there is no going down the hall to talk to someone, or having them come to your desk. You open a ticket and hope someone gets back to you (and not at 3am OUR TIME).

Over the last 3-4 years all the other brands have transitioned to a new ERP / Operation / Planning module. Then our stock price tanked, sales decreased, so our transition was put on hold. So we are using an old (10+ year legacy system that is not even supported by Oracle any more).

Over the last 3/4 months IT has been doing a "lot" of stuff and I always ask them..............."in no way is this going to effect our system or access.......correct?".

So last week I get an email stating "we need to move your applications from one server to another". I ask to delay a week until our 6 month forecast is done. They say no. I run it up the chain, and I get told "no".

I say "make sure you follow the instructions provided by our 3rd party administration to a T" because our system CAN NOT BE DOWN and it seems everytime our IT does this process something gets effed up.

"No problem Mr. Cooper"

On friday, we can not access our system. Can not use excel as a querrying tool. They can not figure out what they did wrong.

We get our outside admin to look at the issue. Of course there are issues in the restart left and right. They get us back up and running late Friday afternoon. SEV1 ticket that will be outside of our normal support agreement.

This morning open up our applicaton where there is supposed to be an Icon where I can launch updates to the forecast. Not only is there not an icon. There is not a system.

Contact our outside Admin. They say "This is strictly an access / server issue"

So all day today my manager and 2 analysts have spent their day in zoom meetings trouble shooting the issue.

Forecast is due tomorrow. STILL NO SYSTEM.

YES I have the emails trail of me saying "no".

I digress, I know we've got bigger things going on in the US right now, but this just chaps my ***.

 

[SpurrierisisGod]

"Did you turn it off and turn it back on?"

 

[SpurrierisisGod]

Don't mean to be snide in what's obviously a frustrating spot but that seems to be the extent of "IT" now.

 

[coopersdad]

"Did you turn it off and turn it back on?"

the funny thing is we had an "old" IT guy that took the package when we were purchased and his 1st question always was "have you rebooted, if not do it, and call me back afterwards". And amazingly I'd say that fixed 50% of most issues.

 

[Penguin]

It's a bad finnegan pin that plugs into the flux capacitor, easy fix. Seriously, they can't rollback the changes?

 

[coopersdad]

It's a bad finnegan pin that plugs into the flux capacitor, easy fix. Seriously, they can't rollback the changes?

something about the amount of other stuff that they'd have to undo also. Don't get me started.....................

 

[Mister CIA]

 

[Penguin]

It's a bad finnegan pin that plugs into the flux capacitor, easy fix. Seriously, they can't rollback the changes?

something about the amount of other stuff that they'd have to undo also. Don't get me started.....................

Crazy, Every "ASK YOURSELF", SMOP & MOP needs a back out plain, and that's after extensive lab testing. I'm not saying it's 100% all the time but it sounds like someone dropped the ball

 

[Kanil]

So a quick backstory, I'm director of financial planning and analysis for a large brand under a corporate umbrella. We got purchased about 5 years ago, and in the "transition" most of our IT department was let go, and the corporate IT department took over. Then about 50% of that work was eventually outsourced. So there is no going down the hall to talk to someone, or having them come to your desk. You open a ticket and hope someone gets back to you (and not at 3am OUR TIME).

Over the last 3-4 years all the other brands have transitioned to a new ERP / Operation / Planning module. Then our stock price tanked, sales decreased, so our transition was put on hold. So we are using an old (10+ year legacy system that is not even supported by Oracle any more).

Over the last 3/4 months IT has been doing a "lot" of stuff and I always ask them..............."in no way is this going to effect our system or access.......correct?".

So last week I get an email stating "we need to move your applications from one server to another". I ask to delay a week until our 6 month forecast is done. They say no. I run it up the chain, and I get told "no".

I say "make sure you follow the instructions provided by our 3rd party administration to a T" because our system CAN NOT BE DOWN and it seems everytime our IT does this process something gets effed up.

"No problem Mr. Cooper"

On friday, we can not access our system. Can not use excel as a querrying tool. They can not figure out what they did wrong.

We get our outside admin to look at the issue. Of course there are issues in the restart left and right. They get us back up and running late Friday afternoon. SEV1 ticket that will be outside of our normal support agreement.

This morning open up our applicaton where there is supposed to be an Icon where I can launch updates to the forecast. Not only is there not an icon. There is not a system.

Contact our outside Admin. They say "This is strictly an access / server issue"

So all day today my manager and 2 analysts have spent their day in zoom meetings trouble shooting the issue.

Forecast is due tomorrow. STILL NO SYSTEM.

YES I have the emails trail of me saying "no".

I digress, I know we've got bigger things going on in the US right now, but this just chaps my ***.

At the end of the issue, figure out how much they cost you and send their director a bill. It won't work and you'll catch a ton of **** for it but it will get your point across.

 

[Riffraff]

Good IT Sys Admin: Drop network cables to old servers, do not turn off until all up and running. Bring up on new servers. If there's an issue, reconnect the comms and troubleshoot the new servers in the background.

Bad IT Sys Admin: Turn off old server and turn on new one and pray that it all works. Hope the newest guy archived the back-up for a rollback. Pray some more to the deity of choice.


It's called the 'scream test'. If no one screams, it's working. If anyone screams... oopsies.

 

[Mister CIA]

Good IT Sys Admin: Drop network cables to old servers, do not turn off until all up and running. Bring up on new servers. If there's an issue, reconnect the comms and troubleshoot the new servers in the background.

Bad IT Sys Admin: Turn off old server and turn on new one and pray that it all works. Hope the newest guy archived the back-up for a rollback. Pray some more to the deity of choice.


It's called the 'scream test'. If no one screams, it's working. If anyone screams... oopsies.

Couldn't the bad IT admin just turn the old machine back on?

 

[Riffraff]

Couldn't the bad IT admin just turn the old machine back on?

Cold boots can be less predictable.
For each component of the system the Hardware, Firmware, OS, and Software all have to initialize from the no power condition. Lots of room for issues - moreso for older servers that haven't been turned off since the Clinton Administration (kidding... sort of). Modern hot-swappable servers are less of an issue. Most people think of Windows systems, but there are plenty of UNIX / Linux servers out there - slower boot times as they run data checking scans to find/fix errors.

Hardware components do a power draw all at the same time and it taxes the power supply/UPS. Brown outs (lower voltage than optimum) cause havoc on random parts. BIOS/Firmware need to pass Power On Self Tests (POST). Operating Systems need to run through start-up processes and stabilize. Start up whatever mission systems/applications after that - could be plenty of them. Third party/proprietary software can have quirks of their own. Need to create virtual servers to run some processes/apps.

Or leave it up and running instead until you get the thumbs up.
If it's a government entity, they likely have redundancy (duplicate system setups) - switch over to the alternate while working on the primary.

--
Pretty much the same concept as most companies saying to reboot (warm) the desktop computer, but leave it powered on (don't turn off) when you leave for the day. Laptops going home turn off due to the battery life - your traffic time may vary.

 

[notoriousbill]

Did you reboot/restart?

 

[arrow1]

 

[coopersdad]

So I'm in the office now..................AND EVERYTHING IS WORKING. They've done nothing. They think it is a Zscaler / security issue.

At least we'll get the forecast done. SMH.

 

[Rich Conway]

I've been an IT ERP programmer/analyst/tester for 27 years.

All I needed to hear you say was "outsourced".

The newish term is, of course, "managed services". Which means that the business sees the IT department as a liability on the ledger and anything they can do to minimize the number is good. So a third party salesperson had a willing accomplice to murder internal technology departments because their folks can "do the same thing at a lower price".

Which naturally is untrue.

I'm just going to call one particular group out by name - if Accenture ever shows up at your company...run.

As in all things, there's a balance.

Outsourcing and Managed Services are necessities for smaller shops. A 50-user company can't possibly keep the necessary IT expertise in house. IT has become so specialized that it's no longer "the one IT guy doing everything", but rather specialists in Security, Infrastructure, Desktop and Applications, Monitoring, Compliance and Governance, Messaging and Communications, etc. Small companies can't afford that kind of expertise in multiple areas.

Large companies are different, of course. In those cases, institutional knowledge is quite valuable, although outside specialists will still be needed for large projects. Even if you're maintaining specialists in house, those specialists generally won't have hands-on experience doing the SD-WAN implementation, cloud-based infrastructure migration, or Exchange upgrade because they've worked in the current company that has yet to do that project, whereas an outside consultant may have done that exact project for 50 different companies.

 

[dickey moe]

This sounds complicated.

 

[Andy Dufresne]

I've been an IT ERP programmer/analyst/tester for 27 years.

All I needed to hear you say was "outsourced".

The newish term is, of course, "managed services". Which means that the business sees the IT department as a liability on the ledger and anything they can do to minimize the number is good. So a third party salesperson had a willing accomplice to murder internal technology departments because their folks can "do the same thing at a lower price".

Which naturally is untrue.

I'm just going to call one particular group out by name - if Accenture ever shows up at your company...run.

As in all things, there's a balance.

Outsourcing and Managed Services are necessities for smaller shops. A 50-user company can't possibly keep the necessary IT expertise in house. IT has become so specialized that it's no longer "the one IT guy doing everything", but rather specialists in Security, Infrastructure, Desktop and Applications, Monitoring, Compliance and Governance, Messaging and Communications, etc. Small companies can't afford that kind of expertise in multiple areas.

Large companies are different, of course. In those cases, institutional knowledge is quite valuable, although outside specialists will still be needed for large projects. Even if you're maintaining specialists in house, those specialists generally won't have hands-on experience doing the SD-WAN implementation, cloud-based infrastructure migration, or Exchange upgrade because they've worked in the current company that has yet to do that project, whereas an outside consultant may have done that exact project for 50 different companies.

Sorry I pulled the rug out on you by deleting my post. I was thinking I hadn't really contributed much with it.

You're probably right that my perspective is skewed from working at big companies and primarily with a big ERP package (PeopleSoft).

 

[MTskibum]

We recently had to change VM providers and do a massive server migration. I am not sure how much of a raise they would have to give me to do that type of position full time. Everyone hates the server infrastructure team, they are incompetent, blah, blah, blah.

I feel bad for them. Often times management gives them unrealistic deadlines as well and then they get stuck working more overtime than anyone else in the "IT" world.

Management telling them they have to move servers, unruly users telling them they can't move servers. Sucks to be them.