r/MaliciousCompliance • u/t4pf • 9d ago
M Do not provision computational resources without approval? As you say!
I work at a small, local company which uses Microsoft Azure heavily.
For those not working in IT or related fields, think of Azure as a computer rental service. We rent computers from them (but the computers stay in Microsoft’s warehouses). We control said computers remotely to run our software, which our customers then use for their own needs.
During the COVID-19 pandemic, as we started expanding and building more software products, we began to rent more computers, which rapidly drove our Azure costs up. This didn’t go down well with the C-suite (even though the revenue from the new products far outstripped the rise in costs). Someone in senior management whose only job is to watch the expense ticker issued a directive to ‘contain the Azure costs’. Suddenly, a team originally responsible for customer support was pulled into an initiative to auto-release any rented computer which had been idle, with no clear definition of ‘idle’.
When a rented computer is released (or a container or VM is shut down, for those familiar with Azure), all software running on it is terminated. This meant that our websites, our tools and our products running in Azure and in use by our customers would go down!
We were told to get approvals if we wanted to keep idle computers active. For the moment, none of them were idle (because our customers were always using our products running on them). But this was an anvil hanging on our heads: our services could go down any time they went idle for any reason, and we’d have a flood of incoming customer complaints. Hence, after reaching a consensus with my team, I started loading our websites at regular intervals to simulate activity. By the end of the day, I had built an automated script which would keep pinging our websites and products.
Simultaneously, we filed keep-active requests for all our critical rented computers. Those requests went unanswered for a month so my team was rather glad about my automated script. However, the ticker-watcher was checking the Azure activity logs and found my little ploy. He posted a message in our common Teams chat (which included my manager and my teammates), publicly berating me and asking me not to do such irresponsible things in the future and to file keep-active requests if we really needed the computers to stay.
All right, I thought—let’s do it your way. I terminated my script and re-submitted all keep-active requests (including both critical and non-critical computers this time), quoting the Teams message in the notes section of the form (in line with CYA rules). To no one’s surprise, these requests also went unanswered. One by one, as our computers hit the idle timeout for whatever reason, complaints from customers started coming in. Most of our websites were down. Few of our products delivered to customers worked. (I’m sure the genius who invented SaaS thought of this scenario.) And we spent amost a week in the office chatting and drinking coffee. There was no work to do because only a handful of Azure computers were running, and there were technical problems beyond my pay grade with re-renting everything we needed.
There isn’t a truly happy ending here. The ticker-watcher was not fired. For what it’s worth, he was removed from his management position and made an individual contributor (where he’d actually have to work) because he had demonstrated a lack of prioritisation and delegation skills. After the incident root cause analysis meeting, he was officially handed this demotion as a disciplinary action. He left the company soon after.
Lots of commenters have pointed out that we weren’t using Azure right, that we needed auto-scaling, and that we should’ve had monitoring systems. You’re correct: we were using Azure VMs as if they were our on-prem servers … we had automatic deployment scripts which would set everything up on-prem, but management with a fetish for the cloud didn’t give us time to migrate properly. They told us to run those scripts directly in Azure VMs instead. So, we weren’t using Azure services (scale sets, API management, functions, etc.) they way they should be. Everything ran in VMs.
I excluded these details in the original post in order to keep it non-technical.
271
u/ceciltech 9d ago
that’s like a hamburger joint selling tons of burgers to be told to stop buying beef because they’re just looking at the cost of the beef. Think of the money we could save if we didn’t buy all this beef.
92
57
u/curtludwig 9d ago
My favorite is "We can't keep that thing on the shelf, we're not going to stock it anymore..."
38
34
u/RetroRodent 8d ago
"Oh we still have loads of these, very popular, bring in more! Oh we only sold one of this one (because we only had one in stock, total, on release day), stop all orders."
- A large american bookstore chain who decided the actual booksellers didn't know which books in their department would sell and that one person could handle all stores from head office.
17
5
u/Ok-Satisfaction-7821 3d ago
Same problem exists in grocery stores. Kroger used to sell 1/2% milk - purple top. But it was slightly less popular than others. They failed to adjust orders, so they overstocked. Final result - it was nearly expired when you bought it. People bought less.
So due to incompetence, they no longer carry it.
2
u/login0false 3d ago
I see this shit way too often. Some variants/flavors of a product are less popular than others, but all of them seem to be ordered in equal amount, so some get sold out and the others have trouble doing so. So they either a) order less of the product on the whole, making the popular flavors even harder to come by, thus reducing reason to go to that store, or b) axe the less popular flavors entirely, so you get less choice (seems less common). It's as if the providers just refuse to sell to retailers any proportion of flavors that is not the one they want to sell.
•
u/Medium-Presence-8008 10h ago
Lol, reminds me of when I took over the product portfolio of a colleague that left. I familiarized myself with what else our suppliers stocked, not just looking for a supplier if someone calls, asking for such a product and reverting with 2-3 month lead times.
"Why are you buying in all these drums of X-grade Y? We haven't even sold any in the last year!" - Upper who just figured out our inventory system but doesn't know default search only lists the last 12 months.
Guess who people come to when they need local, drum-lots of uncommon Y with urgent requirements. Crazy how now we're selling more in a quarter than we've sold since the company was established 30 years ago, now that we actually stock it.
31
u/Chaosmusic 9d ago
I thought the same thing. I run a retail business and am perfectly happy buying more inventory due to increased sales.
21
u/Jonnylotto 9d ago
I hate this (opposite)
I can’t buy it if you don’t stock it. You stopped stocking it so I have to go online. Since I’m online anyway I might as well shoot for the free shipping threshold.9
u/TygerTung 7d ago
Just In Time manufacturing for a burger bar. Don't have any ingredients in the kitchen, just order them from the supplier when a customer orders a hamburger with chips.
7
189
u/tantalor 9d ago
"uptime? what's that?"
Seriously though, this is a framing problem. Don't think of them as "rented idle computers". Those are the servers the fucking business runs on. It's critical they are available.
-12
u/Altruistic_Base_7719 9d ago
Yeah this seems truly malicious because any sane person would not characterize critical production servers as "idle".
I have a feeling OP is more focused on achieving MC by any means possible, than just doing the job in a reasonable way... like people who drive on the freeway at exactly the speed limit, and do their best to ensure no one can pass them.
47
u/Buznik6906 9d ago
What more could they reasonably have done?
They saw the problem coming; they flagged the problem in advance; they tried to do it the right way and got ignored; they tried to end-run the block and got blasted in public. What other option is left?
-28
u/Altruistic_Base_7719 9d ago
We were told to get approvals if we wanted to keep idle computers active
OP is in the wrong here, because as a person employed for the sole purpose of technical support, they wrongly characterized their "azure workloads" as "idle" which is flat wrong, this is entirely in the fault of the OP and no one else.
33
u/zem 9d ago
an important point is that the OP didn't classify the servers as idle. the servers get automatically classified as idle if nothing is running on them, which is what OP was trying to prevent by having a script that just ran some test job on the servers continuously so they could never be seen to be idle.
-19
u/Altruistic_Base_7719 9d ago
... a production server is never idle because web services are always running. Whatever the service is, the server is not idle because it's running the services for the customer... the only way you can misunderstand this is if A) you aren't a technical person or IT or B) you are being willfully ignorant C) completely ignorant
29
u/zem 9d ago
from the OP:
Hence, after reaching a consensus with my team, I started loading our websites at regular intervals to simulate activity.
there was clearly some threshold below which a server was considered idle and shut down, otherwise how would you interpret this detail?
-5
u/Altruistic_Base_7719 9d ago
This completely glossed over detail is the most important one to the whole story, funny how they choose to leave it out.
18
u/invalidConsciousness 8d ago
It was not glossed over. OP explicitly stated that the servers were shut down by a separate team according to some nebulous definition of "idle" that wasn't approved by the other teams.
OPs team had no input into whether a server would be considered idle.
17
u/Buznik6906 9d ago
OP didn't phrase it the best in their explanation here for non-techies, still pretty safe to assume they phrased it better in their official requests for the websites to not go down
-1
u/Altruistic_Base_7719 9d ago
a team originally responsible for customer support was pulled into an initiative to auto-release any rented computer which had been idle, with no clear definition of ‘idle’.
This meant that our websites, our tools and our products running in Azure and in use by our customers would go down!
By the end of the day, I had built an automated script which would keep pinging our websites and products.
This is the exact opposite of "flagging the problem in advance"
OP created a jury-rigged solution to a problem INSTEAD of flagging it and escalating it up to management
and lastly, this seems fake because by definition, servers running critical services on Azure are not idle, would not be detected as idle, and would not be flagged as being idle.
OP is clearly in the wrong
26
u/Zestyclose_Space7134 9d ago
Did you read the story?
OP did everything they could to ensure server availability, was berated and ignored, fallout ensued.
16
238
u/Aggressive-Fig-8734 9d ago
"What is this expense for stuff I don't understand? I don't understand it, so it can't be important."
Narrator's voice-over "...oh but it was."
76
u/modianos 9d ago
And all he had to do was ask why they were doing it this way. Probably a five minute conversation.
57
u/anomalous_cowherd 9d ago
Why would he? They were beneath him and obviously had no grasp on the commercial realities!
Really though it's a failure of visibly mapping the Azure costs to the customer income.
22
24
u/uncanny_taverna 9d ago
I’ve seen this exact management logic: “I don’t understand the line item, therefore it’s waste.” Then the line item disappears and suddenly nobody understands why the entire product is offline.
1
17
1
109
u/Dal-Ron 9d ago
I feel sorry for the next company he goes to. He's definitely going to lie about why he left and worm his way into management again.
5
u/ShireNomad 8d ago
I heard he got a sweet position at some government department run by a tech genius...
48
u/froot_loop_dingus_ 9d ago
which rapidly drove our Azure costs up. This didn’t go down well with the C-suite (even though the revenue from the new products far outstripped the rise in costs)
Typical. The sales department makes money, every other department costs money in the view of management
23
u/Physical_Piglet_47 9d ago
So all they really need is just a sales team. One big, giant sales team...
5
3
1
64
u/biffbobfred 9d ago
Reminds me of the Dilbert where the Intern had to go to various meeting rooms and wave his hands to keep the lights on when the “motion detectors don’t detect movement for a while” turned lights off.
39
u/TheFilthyDIL 9d ago
It's not fun when you're sitting in the ladies room minding your own business and the lights automatically turn off.
11
u/din_the_dancer 9d ago
I have the opposite problem when going into the ladies room, the lights are off when I go in so I'm standing there flailing wondering why it's taking so long for the lights to turn back on.
2
u/SilverIndustry2701 6d ago
I have that with the staircase at home. The lights wont turn on unless I entertain them with a little dance.
I tried fiddling with the sensor, but it just seems to be placed inconveniently for my flat.4
u/dazzlezak 9d ago
This is why your phone has a flashlight. /S
6
u/Top-Interaction-6729 9d ago
I rarely take my phone to the washroom, because, ew.
So this would be of no use.
24
u/anomalous_cowherd 9d ago
Been there done that. If there were non-coders present in our open plan office we had a rota of who had to get up and walk round the room to keep the lights on.
If not we let it go dark, coders like it in the dark and hurled abuse and soft objects at anyone who moved and woke the lights up.
9
u/din_the_dancer 9d ago
There was a job I had where I occasionally did overtime on the weekends, and on the weekends it seemed like the sensors for the lights were way more aggressive. There were multiple times that I guess I was so still at my desk that the lights turned off in my office.
6
u/johndcochran 9d ago
Doubt they were more aggressive. It's just that during normal working hours, there's more bodies to trigger "motion detected" than there is off hours. So, while you may be still during normal hours, someone else is moving about to keep the lights on. No such benefit during off hours.
2
u/din_the_dancer 9d ago
There were times I was in there on a weekend with someone else in the same room and the light would still shut off. And it definitely felt like a more aggressive setting because I had to go out of the room to hit a light switch to turn it back on instead of simply flailing my arms around for motion.
4
u/Cloudy_Automation 9d ago
Where I worked, they didn't use a motion sensor. Out of working hours, they would turn off the lights for the entire floor once every 4 hours. We would need to call an automated system which would turn the floor back on.
2
u/FeelMyBoars 8d ago
We had that as well. Luckily there were physical buttons to use.
Ours was only 2 hours though. It felt like you were constantly getting up to hit the damn things, but really it was only 2 or 3 times a day. You get to play the game where everyone assumes someone else is going to get it so you sit in the dark for a minute, then there's usually 2 people who go to the buttons at the same time.
20
u/Magical_Savior 9d ago
I hate Dilbert, and the author of Dilbert. But I really hate bathrooms with motion sensors that automatically turn off the lights like I've had in several hospital staff bathrooms. ... So he might have had a point, there.
17
u/Techgruber 9d ago
About two weeks after the original Dilbert strip of this was published, electricians came in on the weekend to install these very switches in all 200 offices I was net admin/tech support for. All set for 15 second timeout. Much hilarity and drama ensued. Months later the facility manager who implemented this would still fly into a foaming rage if he heard the word Dilbert.
3
u/SomeOtherPaul 9d ago
Had a motion sensor switch also been installed in the facility manager's office, or was he Too Important for that?
5
u/Rainy_Grave 9d ago
I once had to deal with that in an ER bathroom. Definitely felt like a God damn it, I’m not dead yet moment.
21
u/NoelBeautiful 9d ago
You know that manager is going to tout his cost savings initiatives and show the "savings" he was responsible for.
9
u/Afraid_Baseball_3962 9d ago
At least he won't be causing any more problems at your company. Good luck to the rest of us. 😐
3
u/SilverIndustry2701 6d ago
More often than anyone likes, these kind of colleagues reappear as managers at one of your clients.
24
u/bi_polar2bear 9d ago
We use Azure, and except for having a DR backup, I'm convinced owning services is a much better, and cheaper option. I've yet to see Azure be as great as a data center.
11
u/toxicatedscientist 9d ago
Azure IS a data center, you just don’t know where
5
u/BentGadget 9d ago
Yes I do. I just looked up the closest Azure data center to me, and easily determined that it is located in California. You just didn't try hard enough.
I can drive over there and take a picture if you don't believe me.
6
u/N3rdr4g3 9d ago
I don't believe you
(I'm definitely not just saying that because I want to see your photo of the datacenter)
5
6
3
u/bi_polar2bear 9d ago
I know both of my locations,, though due to OPSEC, I won't say where.
I meant data center as in server farm, not virtual servers owned by some business. Great idea for a DR, not so great for cost savings. It's much cheaper to have a SQL server for database conversions than to have Azure host that server.
23
u/TransportationJaded8 9d ago
Honestly this sounds like your infrastructure team is just doing cloud entirely wrong, probably not their fault a lot of the time management demands they do it wrong and then is horrified by the bill
15
u/t4pf 9d ago
You’re right. We were doing cloud wrong. As other commenters pointed out below, we moved to Azure because someone higher up was attracted to this shiny thing called ‘the cloud’. There was no time given to do it right. We asked for a dedicated Azure platform and infra team (we were all new to Azure then), but we were instead told to spin up resources ourselves with the understanding that we’d do it responsibly.
It worked very well, because everyone was conscious about using only as much as was necessary for the business. Until this incident happened.
2
u/pgn674 9d ago
Was it a lift and shift? Where you end up running on the cloud, but you are not by any means cloudy?
12
u/t4pf 9d ago
We were not ‘cloudy’ at all.😭 Things have improved now, but back then, we were deploying the products as they ran on our systems directly on Azure VMs. For instance, instead of using APIM and application gateways, we ran our in-house API gateway and a traffic router (that were originally run on-prem) in VMs and exposed APIs from them.😂
This was possible because we had self-contained deployment scripts which would set up our APIs automatically. Of course, management said that since we had the scripts, we could just run them in Azure (with minor modifications)! No need to use Azure the way it is intended to be used!
4
u/dreaminginteal 8d ago
Oh Gord, this sounds like where I used to work... One of my last projects over there was to help set up such a monstrosity. They wanted a modular approach so they could spin up VMs on Azure or AVS or whatever, with VMs being added automatically and shut down automatically based on whatever criteria the user wanted to specify...
11
11
3
u/Own_Caterpillar_6178 9d ago
My thoughts exactly. There are many, many ways to optimise workloads in Azure to reduce costs that don't involve just shutting down the entire service when it's "idle". I suspect their infrastructure team is more comfortable racking and stacking and doing cable management than they are working in the cloud.
2
u/Grillmeister5000 9d ago
Since i don't really get the kind and size of the operation, i would be rather careful with such assumptions.
When i worked at a software company (early covid era) that also was basically 80% azure-based, we had customers with higher standards and legal implications which drove costs insanely high.
E.g. most didn't even need a dedicated firewall back then which alone (per client) easily did cost about 10 times than the rest of the whole app structure together. I don't know about modern prizing but a firewall in the lowest tier back then was listed at about 1k/month.
And firewalls weren't the only things that the rather demanding companies wished to add to their structures. I'm looking at stuff like active monitoring and advanced protection services for the databases.
The most basic tier of the app structure was quite efficient though with somewhere between 60 and 80$/month
2
u/TransportationJaded8 9d ago
Things have definitely changed a lot since then and your mentioning of firewalls is demonstrative of that, you can completely isolate things to an extremely high standard exceeding any banking industry or other regulated industry while still optimizing resources and automating away most servers.
If you do truly require dedicated hardware for a customer environment:
1 the cloud is wrong for what you are doing rent space in a CoLo or something else.
2 your executives would be aware of the necessary SLAs included in such a contract and this would never happen or at least if it did happen OP would be laughing at the tens of millions in penalties they’d have to pay out.
0
u/Cloudy_Automation 9d ago
There could be client segregation requirements which don't allow for shared infrastructure which can scale up/down or in/out. The cost of failing to implement segregation (such as medical records) may be much higher than the cost of cloud resources. Or, the cost of changing the implementation was too expensive. I had one case where a team kept a 64 core server used for compilation active while they were thinking. One might think that turning off the server would be cost effective. However, reloading all the source and binaries back into RAM delayed the next compilation starting by so long that it wasn't worth to turn the machine off while there were working.
3
u/PHXkpt 9d ago
It's sad that the business proposals for new/existing products will typically talk to total costs and projected revenue. I'm sure if anyone in the C-suite bothered to look they'd see the increase in Azure costs was expected and accounted for in the "go" decision. Always someone trying to look good by cutting "unnecessary costs"!
4
u/asada_burrito 9d ago
How does somebody who doesn't understand the implications of terminating idle instances have the intelligence/ability to detect keep-alive requests from activity logs?
4
u/t4pf 9d ago
That’s a good question. In my experience, incompetent people high up on the corporate ladder who know nothing about business usually know everything about monitoring and control.
Then again, maybe he knew the implications but, because of his incompetence, didn’t think that scenario would happen.
2
u/MohnJaddenPowers 5d ago
A lot of companies have their IT infrastructure group report to the CFO, with the logic that IT is always a cost center - it doesn't generate revenue, thus it's subject to financial oversight.
Thus, people like OP's bean counter get thrown into the mix.
3
3
u/Grillmeister5000 9d ago
Is there still a team there by now? (A rather competent) One that actually tries restructuring the azure-system to run more cost-efficient? Or does C-level say "better run with higher cost than deal with fallouts"?
3
u/Remarkable-Intern-41 6d ago
We rent computers from them (but the computers stay in Microsoft’s warehouses). We control said computers remotely to run our software, which our customers then use for their own needs.
I loved this story but that's a lot of words when you could just say 'cloud'.
4
u/Axman6 3d ago
Plenty of people have no idea what “the cloud” actually is. They think it’s some magic machine that just runs software, with no idea there’s still often A Computer with an operating system behind most tasks. IMO this was one of the better explanations of the reality of what cloud infrastructure looks like for people with little technical background.
1
u/Just_Aioli_1233 5d ago
Yeah, the opening paragraph seemed a little patronizing. Maybe 10 years ago, but I have a hard time imagining anyone here reading this story doesn't have enough familiarity with "cloud" to know what that means when someone says "Azure is Microsoft's cloud service and we use them for our company's computing needs" instead of "think of it like we're renting computers from a warehouse, but the computers stay in the warehouse" like geez man.
5
4
u/garden-wicket-581 9d ago
ohh, but cloud is a CAPEX not an OPEX (or vice versa, I don't care, I'm not a bean-counter, and thought it was smoke/mirrors money shuffling.. )
10
5
u/PapaDuckD 9d ago
Come on now, it's not hard. And learning how the world works is a Very Good ThingTM for an IT person to know.
CAPital EXpenses are things you buy one time and become assets that you own (and have to maintain/manage). Buying a new car is a CAPEX expense. You buy it once, it lasts for years, it's something you can sell if you wanted to, but you have to care for it while you own it.
OPerating EXpenses are services that you buy on an ongoing basis and can scale up or down as needed. Leasing a car is an OPEX expense. I agree to lease for a certain term and when that term is up, I'm done. I can lease a new car or just walk away.
Same with internet access. The term is typically much shorter (often month-to-month) and I can make it go faster or slower by paying more or less money. And when I'm done, I stop paying and it goes away.
2
u/TheFluffiestRedditor 9d ago
The only thing I would have changed is not creating the activity script. That's a legitimate no-no. Watch it burn sooner and faster, to create an issue for manglement.
3
u/t4pf 9d ago
It was a collective team decision. I had received the go-ahead for it from my tech lead. The script was a defensive measure to be removed the moment the requests were approved.
However, if something like this were to happen again in the future, I’d definitely push to let it burn sooner and faster, as you put it.
2
u/RenouxMarais 8d ago
So excuse my incompetence, but could you not have installed or split everything on 1 or 2 servers? Do you need that many devices? It feels like having 1 IIS server could potentially save them money and you headaches. I dont know the setup but in my mind you should have 1 server running all services and you just scale that server up or down as required...
2
2
u/ScorchingFalcon 9d ago
no auto-scaling or request queue for cold starts? route requests through whatever azure's equivalent of lambda is that checks for/spinsup server first before forwarding requests (granted the customer triggering scale up probably will complain about latency)
1
u/t4pf 8d ago
I mentioned the details in another comment, but the gist is that we were using Azure VMs the same way we used on-prem servers: by deploying our in-house tools directly on it instead of using Azure services like APIM and application gateways. We weren’t given the time to move to the cloud the right way. We were told to just use our auto-deployment scripts which were meant for on-prem.
3
1
u/Bane-o-foolishness 9d ago
Microsoft’s warehouses.
One of their management team that is responsible for their multi-billion dollar investment in DC facilities is going to put out a contract on you.
1
u/RIP_Sinners 9d ago
Rather than cover up a shit decision, better to let it fail quickly where it reflects the least on you and most on the decider. This hopefully results in the source of poor decisions being identified and removed quickly (usually via promotion to a position of least consequence).
1
1
1
u/ricksebak 3d ago
> even though the revenue from the new products far outstripped the rise in costs
I’ve found it helpful to create a report which expresses cloud cost as a ratio along with some other metric which is important to your business. For example, if you work in cloud operations at Dominos Pizza, then your metric might be the number of orders placed in a month. You can create a report for the finance team which says “our cloud costs last month were $100k, and we served 500k orders.” Then a few months later if the report says “our cloud costs were $200k and we served 1MM orders”, your finance team can understand why the cost went up and they may even view it as not going up at all.
794
u/Frequent-Road-5686 9d ago
I'd say the demotion and leaving are pretty satisfactory. Bean counter had to actually work and didn't like it, and now the demotion will follow him and he'll have to explain it wherever he tries to work.