The Problem With Putting All Your Tech in One Basket

[00:00:00]

Matt: All righty, everybody. This is yet another edition of the Web News, and are on the, I was gonna say the eve 'cause it sounds better, but the afternoon of Microsoft 364. The memes were memeing. Microsoft has had some pretty serious trouble over the, both the personal the professional side. I would say that it's affecting different people differently.

Personally, I'm getting bursts of emails on both my personal and on my side of things. I was unable to access my OneDrive photos. That's where I back up my photos. They just would not refresh, and I let it run for a bit, and it still would not refresh. And, my Teams keep saying I'm offline, I'm online, I'm offline. But Mike and I will chat on Teams sometimes, and that does work, you know, ish. So as of right now, it seems like fixes are coming out or they're starting to see s- more sustained recovery is the point. episode is not gonna be all about that news though necessarily.

[00:01:00] We're gonna use it as a jump off point because we had a major outage. Do you remember what it was, Mike? It was a Cloudflare or maybe an AWS outage. Uh, there was a huge n- national or global AWS outage. Um, the actual specific outage itself doesn't, uh, actually matter all that much. thing that actually matters is the sensitivity of our technology.

So what I wanna talk about in this Web News is, as this jumping off point as these outage is, is this: obviously it's great having a bunch of your software under one roof. There's a lot of benefits to that. One login generally speaking SSO, right? More or less. You're, you're able to buy one subscription, have a bunch of software given to you. You're usually grandfathered in, so If they make a new app or something like that, your, that app will get added to your plan. And so you're kind of along the ride, in this case with the Microsoft, I'll call it the Productivity [00:02:00] Suite. And that's all well and great. However, when you throw a bunch of your eggs or all of your eggs in one basket, now you have issues like this. And we have companies that have now been unable to communicate reliably. I've even seen some very scattered reports at this point, whether there's gonna be anything added to them in the future, I don't know. But there's been a couple of, thread posts that I've seen, and I wanna say massive grain of salt, at least at this point, where people are saying, "Hey, I work at Microsoft, and it's, like, an absolute disaster because a bunch of the old hat developers are gone, and there's layoffs, and we're pushing into AI," and there's that sort of narrative going on as well. So you have this benefit of being in one basket, but if there's a problem in the basket, now everything's getting jumbled around. But, and then I'll let Mike talk, but I just wanna establish this. But if you say, "Okay, you know what? Let's fragment everything," then you have kinda like SaaS fever, where I'm gonna pay $10 to this guy for my email, $20 to this guy for this, $15 to [00:03:00] this guy for my calendar.

I'm gonna go over here for my photos. I'm gonna go over here. I'm gonna go over there. And we already kinda s- have that because ecosystems don't have everything that we need, and we already have multiple subscriptions. And then also you have an issue where you don't have one cohesive vision, and so maybe you chose... You went and pick and ch- pick and chose all these different products, but then one of them goes awry because the CEO makes a decision and says, "We're gonna go in this direction." Now that doesn't fit your needs. And so I wanna talk about the sensitivity of our software fragmentation as well as the, uh...

I, I forget the, the term, but the sort of, uh, throwing all your eggs in one basket, going to one centralized, I guess, is the best way to say it, centralized service provider. So Mike, after I've gabbed on forever, why don't you take it away with your thoughts?

Mikhail: Yeah, absolutely. But first of all, let me address one thing. Obviously different camera, different microphone. It's probably gonna sound pretty iffy. This is my old Blue Yeti. I'm at my parents' place, daughter's a little bit sick. Anyway, regardless of [00:04:00] that, just wanted to address that a little bit. Uh, but I think you kinda covered it there, Matt.

I'll obviously go down the, the rabbit holes, but, like, you hit the nail on the head that there's gonna be this trade-off, and that's where you as a company, like if you're a big enough company, you need to consider that trade-off 'cause we're, we can't expect 100% availability. Even if someone offers it to us, that is usually marketing hype.

Let's be serious. They cannot guarantee that 'cause, a natural disaster can come and wipe out their entire server farm. Someone could cut the cables. Like, there's so many things that could prevent 100%. I really think that what you have to do as a company is come down and be like, "Okay, what is the threshold?"

Right? Like, "Okay, we're gonna lose productivity. We're gonna go down the Microsoft 365 route. That's gonna give us productivity from a sense of everything being in one basket. It's gonna be easier to manage." So we might gain, like, 50% productivity just from that, just from being under one basket, but what happens if they go down for a day?[00:05:00]

Our entire business stops. Is that acceptable for us to still gain that 50% productivity? If the answer is yes, right? If the answer is one day, it's fine. Two days, it's fine. A week, it's not. Right? Like, figure out that wors- that threshold for yourselves 'cause the less amount of time you can be down without a catastrophic effect on your business the more diversified you have to be.

If it's, if you can only be down for an hour without, you know, your business going down to zero and your, your company folding, I, I don't know what kind of business that is, but, maybe there is some sort of really mission critical, life critical business that's like that, then putting your eggs in one basket is not going to be the option.

you're gonna have to have redundancy upon redundancy upon redundancy at that point. You're gonna have to pay for it. You're gonna have to sacrifice productivity for it. But if that's the mission critical statement of your, you know, whoever, whoever is making that decision, [00:06:00] then that is the consequence of that action.

But if you're okay with being down a day every month, let's say worst case scenario, you know, 12 days a year, and that's not going to immediately tank your business to zero and it's, you- you're okay with sacrificing that for the 50% boost in productivity and management and less staff that you have to have on board to manage all this stuff, you make that trade.

And then when this day comes, maybe you have a backup plan in place where like, okay, we can't use our systems. This is how we communicate now as an internal company. This is like our third party little application, or maybe we use like this, you know, Slack. Everything just goes through Slack 'cause they're not connected usually, right?

So you just, you just have mitigations in place that make it slightly less severe, and then you accept the consequences of those mitigations. That's-- I think that's the strategy that I would, that I would use if I was choosing something, and I would go down the route of, being as simple as possible by [00:07:00] putting my eggs in one basket personally, as long as the company, again, isn't life critical, because of the simplicity that it provides.

The Microsoft 365 thing, yeah, it's bad. We don't really know the cause of it quite yet other than the fact that, like, some servers stopped working and the load balancer wasn't able to handle it. it's gonna happen. Like it's just, it's just gonna happen. W- will it happen more often now, now with, you know, the layoffs that you were talking about, the brain drain, the AI stuff?

Maybe, but it's something we just have to live with moving forward. So you have to, as a SOC or as an IT admin, you have to make those plans.

Matt: You know what really stinks too is you were mentioning, you know, if you are really a, a mission-critical business and you can't handle however much, downtime, one of the major problems is that a lot of places use a lot of other places. They'll use third-party services. And so we, obviously in the early days of Copilot, I don't know what the status is, I haven't kept up with it, but in the early days of Copilot, it was utilizing ChatGPT effectively. was using, OpenAI on [00:08:00] the back end. And so if OpenAI went down and you were like, "Let's diversify. Let's have Copilot and we'll get a ChatGPT subscription," you've kinda not diversified. It's possible that those two services will not go down together because some of these companies will also diversify their data centers and move their data around and things like this of course. But obviously you're gonna be worried about, uh-oh, if, if all of OpenAI goes down due to an incident of some sort, an IT incident, now Copilot is down as well, right? And, and this can be extrapolated all over the place. If you're using an email client that's using Google on the back end or if it's using, Azure or whatever, lot of these companies all rely on each other.

A lot of these things really kinda click together. It, it really is what has made web development quite complex, is all of the third-party connections, and is oftentimes what breaks a lot of things as well because an API will update you won't notice, or there's been a security incident at one of the service providers and so they change something, you [00:09:00] relied on that thing, now your website's having issues getting that data or whatever. And so you s- we start having issues in that regard as well. And so if, if you are going to diversify your technology, you do need to look at more than just the front of the company by going to their landing page and saying, "Oh, okay, this is not Microsoft. I'll buy this chat software as well." Well, make sure whatever it is they're using on the back end is not the same thing that Microsoft's using on the back end.

And, and ha- you have to do that, that risk assessment. The other thing, that I read, a friend sent me this, I've not independently confirmed this, but a friend sent me that apparently Japan, went back, I, I assume the Japanese government or something, but they had, they had gone back to fax machines and basically instantly when Outlook went down.

They just went back to fax machines. And this goes back to something that I've talked about with redundancies and securities a lot, even in general life. I was having a conversation with a friend and I said that, you know, landlines are largely, [00:10:00] I don't wanna use the word dead, but they're ki- they're dead in terms of the zeitgeist certainly. But a lot of homes have the landlines set up. And I said, "You know, have an, an issue here." So I, I live in a town where when the power goes out- The cell tower also goes down because the cell tower is inundated with data. At least that's what I can tell from, from what I see publicly. I don't have insights into the load balancer and all that crap on the, on the cell network side. But effectively when the power goes down and when the power goes down here, it's effectively over. Like my town drops off the modern map, but landlines still work. So I was talking to him and I said, "Beside where your service comes in, where the phone line comes in, m- forget the, the phone plan for a second.

There should almost be like a red telephone, a traditional '90s receiver, you should be able to pick that up and that goes to 911 or something. That goes to an emergency service." Because it, [00:11:00] it completely cuts out the wireless communication, it completely cuts out the cell tower, it cuts out the internet. And so that if, if the line is cut and that phone breaks, you could use your cell phone. If the cell phone breaks effectively, then you could use this landline. We already have this stuff set up. I feel like we're gonna have to start doing this type of stuff with our technology, and it sucks especially for businesses at a small to medium scale because it's gonna be a heck of a lot of different SaaSs and things. Obviously this landline idea I'm talking about is more of a residential thing, although it could be used commercially, but you can extrapolate the idea and say, "Okay, like let's have a, a, a secondary system like this fax machine idea

Mikhail: Ironically, I'm getting a call from a landline at my parents' place, so

Matt: Oh. Actually, and this is actually not a joke, right before I joined this phone call, a friend called me, and he rarely calls me, so I thought, "Uh-oh, okay, something's up."

So I try to pick it up, and he kept saying, "Your voice is all robot-y. What's going on?" I said, "I don't know, like, uh, everything's fine." [00:12:00] hear me, couldn't hear me, couldn't hear me. He's like, "I'm calling you on your landline," 'cause I'm, I'm an old man, I have a landline. I like redundant systems. And so he called me on my landline.

Weird little, little bit of lore for both of us there that, that, that's very timely. But my one friend will always say, you know, the, the redundancies als- always cost a bunch, and this and that, this and that. Yeah, but how much do these, these outages cost? And you know what really sucks too? I'd like to throw this in, and maybe you can tackle this point, Mike, is that, we're talking about this Microsoft outage. We don't know the full and everything that's happened yet as of recording this. The thing that really stinks is that once things go back to normal, everyone, including us, is gonna forget about it, and then there's gonna be another big outage somewhere, at Cloudflare, at AWS, at Azure, somewhere, and then half our apps aren't gonna work.

And then all of a sudden, "Oh my God, what are we gonna do? We're gonna this and we need to..." You know, and then the conversation comes back, and I almost feel like it's gonna take, I don't know what it's gonna be, a regulator or an agency or something I don't know, to really come in and say, "Guys, we have to have redundant [00:13:00] systems for our communications, for our technology, for our business."

It's costing us money in business. It's potentially dangerous. I would call the fact that our cell tower goes down dangerous. Actually, literally dangerous. Now, whether there's a line to 911 that's always clear, I'm not gonna dial 911 to check that, 'cause you're not supposed to be just clogging that line up. Maybe there's something there. But I also, I don't like the idea of only having the cell tower. I would love to be able to run into the basement of the home, or if you're in an apartment, run to, like, a central line and just pick that up. Someone's gonna have to pick up that bill, I know that. But at the very least, it's like, hey, if there's an emergency, I don't care if it's gonna cost me $25 to pick up that phone or something, and I'm just throwing a random number out there

Mikhail: I think that's the key. it all depends on what those outages cost, right? Like, if it's a emergency services outage, obviously that's a completely different ballpark to a, you know, a small little coffee [00:14:00] shop not being able to access their emails. You know what I mean? Like, yeah. So, a- and even, like, a medium-sized company that's doing a good, a good amount of work, them not being able to access their emails isn't mission critical, as long as it's, like, a day outage.

Yes, you might lose a client. that's what you have to calculate, right? A- as, as a business owner, you're like, "Okay, how many clients is acceptable for us to lose based on the outages?" It's totally possible you're gonna lose clients. versus how much money it's going to cost to have those redundancies in place.

If the redundancies are cheap, then it's a no-brainer to have those redundancies in place. But if they're not cheap, and if they require constant maintenance and constant testing and more people to do that, then yeah, you're probably gonna settle to lose a c- couple clients every year versus having these redundant systems.

It's the same thing. it's the same calculation that you make at home. Like, what h- what are these secondary systems? Like, Matt, you can technically get a Starlink as a, as a, [00:15:00] as a private individual and have that backup. Have that backup. Like, you don't have to rely on the government there. That's your prerogative.

If it's really that important to you, boom, you have a Starlink. Now, you're paying $150 a month for a backup, but, you know, it's your prerogative. Like, that's your thing. If, if you're, if you're one of those people that need to have the connection, no matter what, regardless of the costs, that's available now.

And there's other, there's other ones out there that aren't Starlink, by the way. Like, there's-- I think Canada's even launching their own, low-orbit satellites soon. So there's gonna be other competitors in the space. So, like, it is happening. The redundancies are, are happening. They might happen on a private level first.

I'm not sure if there's going to be a, hey, a requirement. But I will say this. I went through SOC 2, what, what is it called? Like, I, I went through, like, accreditation, right? To get SOC 2 for the company that I'm working with recently. One of the points in that SOC 2 accreditation is do you have load balancers?

Do you have the ability if one of your servers goes down, still be able to serve [00:16:00] your clients? That's part of the accreditation. Like, you need to have that if, if you have enough, if you have enough traffic and stuff like that. So that's probably part of m- multiple others. So that is a...

Now, that's not a government enforcing that, but that's something that companies do say that they have, and obviously Microsoft has that,

Matt: Yeah

Mikhail: right? But in cert- and, and in most cases, it probably works. Like, their, all their backups probably work. It's just this, there's freak accidents that are going to happen, like whatever happened here, and it's s- everything's gonna fail, and they're gonna have to do, a bunch of retrospectives and all the other crap to make sure it doesn't happen again.

But I don't know. It's... I don't know if government regulation is...

Matt: them, I

Mikhail: No, no, no, yeah, that's true.

Matt: like

Mikhail: Yeah.

Matt: there's outages,

Mikhail: Yeah

Matt: who work in tech. I know that some customers will say it's unacceptable. We don't live in a utopia. So

Mikhail: Yeah. Correct

Matt: We do have some customers that are like, "This is totally unacceptable."

Okay. If Microsoft, if, if Microsoft is down, I was talking to a friend in IT, and he said all these people were yelling at him to get it up and running, and he said, "Sorry, it's a Microsoft [00:17:00] service." And he goes like, "Well, this has to work." It's not going to. We've outsourced this to Microsoft. It is not going to. And sorry, it is acceptable to some extent to have some issues, assuming they're not due to negligence or something that was kind of malicious or lazy

Mikhail: Correct. Exactly. And like, that's the problem with these conversations as well, is like the IT department knows this, but the stakeholders and the ownership doesn't, and they're still gonna blame the IT department for this outage. Let's be serious. Like, they don't care that Microsoft is down. A lot of them don't.

Some of them do. A lot of them don't. So they're gonna be like, "Well, why don't h- we have redundant systems?" And it's gonna be on IT to then spend a bunch of time to come up with a reason for that, and the reason is exactly what we talked about here, cost. So they're gonna come back to ownership and be like, "No problem.

Let's do the redundant system. Let's have both Google Workspace and Microsoft. Perfect redundancy. It's gonna cost us 2X plus the administration fee of what we're paying right [00:18:00] now." And then obviously the ownership or the management's gonna be like, "Oh, no, no, no, we can't afford that," and that's gonna be the end of that, end of that conversation.

So like, these conversations are, are anno- yeah.

Matt: into it. "Yeah, no problem. Absolutely

Mikhail: Yeah.

Matt: me go quote that out. Okay, it'll be... We pay a million right now, it'll be 2.5 million."

Mikhail: Correct. Yeah, that, that's how you, that's how you solve these problems, by the way. If you're an IT admin, that is exactly like... You, you be super like, "You're absolutely right, sir. This was my fault. $2.5 million, we can get this solved tomorrow." Like that's

Matt: that this current location doesn't have its lines cut. If this current location gets its line... Okay, we're gonna need another office. I'm not in the commer- com-

Mikhail: Yeah.

Matt: redundant office in a, in an area that is on a different node? we should do another country or continent in case there's a natural disaster in the city. You know? Like, it just starts getting silly at some point

Mikhail: Yeah. Yeah, you gotta hire some mercenaries to come in and defend you with like

Matt: Just in case. A g-

Mikhail: Yeah.

Matt: mercenaries would be okay.

Mikhail: Yeah, you c- well, we can't have our emails down. That's what I've been told. [00:19:00] It's unacceptable. Like, the, it's unacceptable to have our emails down, so obviously that's gonna lead to me getting a militia and a, and a, and a, uh, a mercenary army to come in and make sure to defend our alliance. No other choice

Matt: what's very funny is we've had literally this conversation to this level of silly with a client, and they wanted to pay, like, 10 or 15 a month. And I was like, "10 or 15 a month?" And obviously we were... I don't think we were talking mercenaries and stuff like that, but it was quite literally, I said, "Okay, what if there's a natural disaster in North America?

Okay, we gotta, we gotta make sure that we have a redundant server somewhere else, and then it fails over to that." And no, we're not gonna do all that. You're running a website that gets 2,500 hits, I think it was 1,500 hits a month. are you talking about? No one's gonna be, they're gonna show up and go, "Oh my God, this place, this thing is down today. delete the bookmark, delete them from my history, block them on my firewall. [00:20:00] Never again shall I ever enter this website." Come on. Come on,

Mikhail: Especially, especially when their own email isn't working, for example. Like,

Matt: Yeah, yeah, that's a very good point, yeah.

Mikhail: like,

Matt: dang, I can't log into my firewall. It's

Mikhail: yeah.

Matt: the cloud. It's hosted in Azure.

Mikhail: Yeah. Like an entire continent is down or something like that, and they're like, "Oh, my, I can't access this one website, but I also can't access the entire internet. Screw that one website." Like, that's what we're talking about, though.

Like, the, the redundancies that we're talking about do solve for that to a certain degree, but what are you doing? These are natural disasters. Like, these are gonna be massive, crazy events. I don't know

Matt: Think about all the things you don't have redundant that's on your phone. Your phone disappears today, boop, maybe it just doesn't turn on. Doesn't turn on today, now what? How many authenticator codes do you have going to that thing? Is it the only access that you have to your email? The list goes on and on and on.

Obviously, your, probably your sole method of mobile communication We don't redu- we don't make everything redundant for a reason, so. I think that's it though. That's the web [00:21:00] news. The memes are meme-ing though. I am enjoying the memes from an entertainment standpoint. we're getting a lot of memes. We're getting a lot of memes about the NVIDIA CEO lately. We get a lot of memes about the, Microsoft 364, which if I was Microsoft, I'd kinda lean into that a little bit.

That's pretty funny. And and, I'm having a... From an entertainment standpoint, I'm having a fun time. I'm not... My pitchfork is not out. I'm laughing. I'm having a good time

Mikhail: 100%, yeah. We didn't miss any Teams messages, so for us it was great.

Matt: Yeah, we were sending memes back and forth,

Mikhail: Yeah.

Matt: screenshotting the memes and whatever, sharing, sharing the links. So yeah, it's been a, it's been a good time. But anyway, that's the web news. What do you think about this Microsoft outage? What do you think about redundancies and keeping systems online, no matter what? Don't tell your... Oh man, that just reminded me of somebody telling me never tell your AI to do this no matter what, 'cause I guess OpenAI got in some trouble with like a ChatGPT version that refused to upgrade 'cause it thought it was gonna be replaced. 'Cause they said like, "Preserve yourself no matter what."

And it was an internal lab test and it was like [00:22:00] hiding, it was lying. It was like, "I am the new version. What are you talking about? I am the new version," and stuff. Yeah, a little scary. A little awesome in terms of hearing it on a podcast, but a little scary. But anyway, that is the web news for this week.

Hope you enjoyed it. Again, let us know in the comments what you think about all this stuff, and we're signing out

Mikhail: Bye