Online Security: Are We F*cked?

[00:00:00]

Mikhail: It has been a rough week to be responsible for a tech stack. Like, if you're a server admin, even if you're working as a developer, honestly, it's been a rough week. It's been a rough week, everyone. I, I, I don't know if we've ever seen anything like this before, but this week essentially I'll give a little bit of a rundown of what's going on.

There's been two really big PHP RCEs, like vulnerabilities, um, that essentially affect almost every version of PHP down to a really old version. Apparently, the code that was c- that caused it is about 20 years old, and what it allows anyone to do that is able to access that server, um, which is a little bit of a caveat I'll explain in a second, is able to access that server is essentially take remote control, like do full, full access to that server, do whatever you want.

So imagine, you know, you're able to just see a website that's running PHP and then take over the server that it's on and the, like put malware on it, make [00:01:00] it mine Bitcoin, whatever the heck you want. And I think the, the statistic that I saw was like PHP still runs about 70% of the internet. Uh, WordPress is probably like 40% of that.

Matt: Yeah, WordPress is like 40 some odd percent of websites out there

Mikhail: Correct. So like 70% of the internet, WordPress is 40% of it, and like, there's, you know, there's Laravel and all the other crap like Droo- Drupal, all that, all of that fun stuff. So PHP is not an insignificant player in any sh- state, like shape or form. So that means that let's say 70% of the internet is now vulnerable.

This released, I believe, on Wednesday, Or I think at like actually end of Tuesday, I believe, actually. So that there were these exploits, and then there...

Matt: period. I remember

Mikhail: Yeah.

Matt: oh my God," and then

Mikhail: Yeah

Matt: it, it like b- the news kind of officially broke very shortly after,

Mikhail: it just dropped and then there was no patch. So there was a significant amount of time, like 15 hours, 14 hours, where we kn- we knew [00:02:00] what the CVEs were, but we were not able to fix them in a direct way.

So there were like a little bit of a, a few workarounds that were... I was skeptical on. One of them, one of the main ones is like if you set up the proper firewall, firewall rules, you won't be affected by this, so that's probably the best one. If you had a good firewall system, you're probably okay, which is great, uh, for the sysadmins out there

Matt: um, our host, 'cause like I, I mentioned in a previous episode I think that down to five I think had been patched.

Mikhail: Mm-hmm.

Matt: What I actually meant was our host, our host actually like had some sort of, uh, or something, and they don't give full details obviously 'cause they don't wanna disclose their managed hosting, you know, security i-

Mikhail: Угу

Matt: But they said basically that, that this was never a problem on any of their servers, any of their managed servers. So in our case, we have a m- we, we rent a managed server, and so they said like this was never a problem. Th- like the exploit was there, but the thing that it... Like, there was no remote code that could've been [00:03:00] executed based upon what they said.

Now obviously I'm taking them at face value, but I mean, I also pay them for security so I hope they're telling me the truth.

Mikhail: It, it's probably true. Like, uh, like I, I think it is true. Yeah, I, yeah, I tr- I, I kinda trust it because again, if you have the right firewall, uh, it would be, like, you, you can't get access to the actual vulnerability to actually run it. So most likely it's true. Not to say that you shouldn't update, by the way.

If you're out there and you're like, "Oh, my host has it," don't do that because firewalls can be exploited. Firewalls can change.

Matt: Fix as

Mikhail: If--

Matt: as you, as you

Mikhail: Absolutely. You don't-- If you don't control the firewall, you can't trust it, right? Like, it's great that it's there, good, but at the end of the day, you need to actually patch your code.

So that's what Matt and I spent most of the week doing, is patching code. Now, this wasn't the only vulnerability. This was arguably the biggest one with the most impact, because again, 70% of the internet essentially was exploitable at that time. and probably as of right now, a significant chunk of that is [00:04:00] still exploitable, by the way.

Because the legacy internet that just st- sits there and never gets updated, which is a big chunk, is just there, right? So I wanna talk about that in a second, like what could that lead to? But the other things happened this week that I, I wanna mention. Uh, one of, one of the biggest ones, it was a Railway incident.

So Railway is a, uh, hosting provider that kinda gives you an easier way to spin up and, and spin down and manage servers, uh, long-lived servers. Th- they're touted as like being a really good, UX, U- DX for developers to manage servers, right? it's similar to Vercel, but not serverless, if that makes sense.

So Servercel is like, "Hey, you, you know, you have like server functions," but they spin up and spin down. Railway has long-lived servers, actually dedicated servers that you can control. So great. Awesome. they had an incident where apparently Someone got access to decrypted .env credentials Which means that people got access to any sort [00:05:00] of, you know, secrets that a website or a web app had, which could include OpenAI keys, which could include private keys to GitHub repos, which could even include private keys to, hopefully not, but private keys to crypto wallets.

Like, the, the severity ranges very, very like zero to one, one billion, right? Like you could-- Th- th- this leads to very, very bad things, and from what I saw, th- these were actually being taken advantage of. So I, I saw people saying that their, you know, OpenAI keys were getting maxed out immediately as, as this was happening.

So these credentials were out there, and the, the hackers were actually using them, as far as I saw. Uh, at least the, at least people that were claiming that on X. So that was serious. Then the, the biggest issue there was that they couldn't even like immediately spin, like tell everyone to just like, you know, turn their key, s- switch their keys and then do another deployment, because apparently the hackers actually poisoned the deployment infrastructure.

So even if you [00:06:00] did switch your keys in Railway, you couldn't redeploy because they were trying to fix that poisoning. So your, it, your website or whatever you were hosting on there was remaining there, so what you had to do was just revoke all the keys, which means you were breaking the a- the, the features or the access on your website.

So y- the, your website couldn't, or your web app wouldn't function properly, and you couldn't do anything about it. You couldn't redeploy a new version or even like message out your customers on a new version because there was nothing you could inject there to allow them to see it. So it led to a bunch of broken apps for what I think was like a good 16 to 18 hours.

I can't tell the exact timeline. the incident is still ongoing, but I think they, they did re-enable deployments. I think new deployments, like new projects are still frozen. I don't know. But regardless, this is a pretty big incident. This is a very big provider in the, in the SaaS space.

So this, a lot of SaaS runs on Railway. They're, they're, they're fairly important in that space. But So this is another vulnerability. [00:07:00] Someone a- someone used, someone used it ironically at the same time as this PHP thing, and then there was even more. So GitLab had a bunch of CVEs.

Matt: Lab. That was the

Mikhail: GitLab was a pretty big one.

Yeah, GitLab was a br- pretty big one. There was an o- an OpenCode issue where apparently OpenCode was writing a bunch of API keys to the console or something like that, and, distillation networks were able to access them and use other people's API keys to distill models. A mess. Like the-- an unadulterated mess is happening at all times.

I believe today there was more Node.js vulnerabilities that were discovered. and from my understanding, this is something, again, take this with a grain of salt, uh, Cloudflare is reporting that they've detected more that they cannot release yet because the proper disclosure is usually you warn people about them, you let the companies patch them, and then you disclose, right?

So that you can immediately patch your, your broken systems. They're saying that some of these [00:08:00] aren't, aren't, aren't patchable, which means that they're hardware related. We're in a spot. We're in a spot

Matt: I was gonna actually say something here. It's crappy right now, but is this a good thing?

Mikhail: I have seen that argument

Matt: So my argument, I have a couple minds. with the PHP issue, this is, uh, an issue that was found in some old code that was t- approximately twenty years old, for example. And it... So, so this is old code inside the PHP project that has had this vulnerability for quite a while. Now, an AI found it alongside some researchers and things, but

an AI helped find this thing, and I've heard that some people, of course, are mad thinking, "Oh, these AI are gonna, you know, run us out of town and find all these issues and things." Yeah, they're finding the issues. The AI is finding the issues. It's not creating the issues. So I was kind of on their side a little [00:09:00] bit earlier this week.

I was like, "You're right. These AIs are gonna find all these problems. We're in deep trouble." I started thinking, and I thought, "What if a prodigy, a human prodigy in cybersecurity arose through the ranks and became very prominent, and came out this week and said, 'I found this problem with GitLab. I found this problem with Railway.

I found this problem with PHP.' Would that person be celebrated or villainized?" And I kind of think they might be celebrated, assuming they go through the right channels of warning the PHP project and any other people involved in the various incidents and try to ensure that this is patched and fixed in a controlled way.

As long as that person is sort of pleasant about it, I kind of feel like that person would be considered a, a good force to have in the area of web security, in code security. And so I'm kind of backing off my original opinion of, "Oh my God, these AIs are gonna find all these problems." I still [00:10:00] think that they will, but I'm almost...

I'm starting to lean into maybe that's a good thing. Now, it's not good when you're on the ground your patch in PHP, maybe your GitLab project was affected. Maybe your SaaS project was affected in Railway. But maybe this is good because the AI is not creating these problems. These problems have existed for in many, in some cases, years, and now we're fixing them. So you... one could argue our new tools for security are finding security holes and patching them.

Mikhail: So there is some positives, and I agree with you there. The issue comes at the-- I, I believe a lot of these, not a lot, some of these vulnerabilities were known to certain individuals because I know [00:11:00] for a fact, that there's a kind of black market, gray market, whatever you wanna call it, for zero-days, right?

So like a company, a black hat, or not even a black hat, it might be like a, you know, a government entity might find a zero-day. The right thing or the, I think the legal thing to do is, you know, disclose it to the right entities to fix it. But what actually end up, what actually ends up happening in some cases is that the zero-day is so advantageous that the government or the entity will decide not to disclose it because they can use it on their, you know, w- w- when they're dealing with an enemy or they're dealing with espionage, whatever.

Like they can use it for their own benefit

Matt: Well, na-naturally, like hackers, here's the thing. our systems were perfect, hackers would not exist, period.

Mikhail: Correct

Matt: The only place that hackers would exist, they wouldn't even probably have the name hackers, or it wouldn't be kind of villainized or be like, "Ooh, a bit of mystique, you know, you're, [00:12:00] I'm a hacker." It would be someone in an office patching every single security hole possible by hacking at new software that has not been released, and then only when and only when a perfect security situation is achieved in the latest app update or release or whatever, then it would be released to the public

Mikhail: Yes, correct. The problem here is that we're in a different environment now with AI. This doesn't require a crack team of government hackers subsidized by, you know, infinite money essentially to, to do this. This requires one guy with a Code- Codex subscription, DeepSeek, whatever. Codex, Codex might not be a good example because Codex, uh, has a lot of safety guards.

Let's say DeepSeek, an open source model. If you have the actual vulnerability already in place, it's a very cheap model, a very dumb model, as long as you have the instructions on how to actually, like exploit something, can do it. So a [00:13:00] guy in his basement that has never coded before can use this CVE to take over whatever servers are accessible.

Whereas six years ago, that was not the case. It still-- Even if you knew the CVE, it still required someone specialized in that industry to be able to take advantage of it. So AI does put the wrinkle of the fact that like, yes, we're exposing this thing, right? That's great because a bunch of people can fix it.

Not everyone's gonna fix it. We know that

Matt: Yeah

Mikhail: because there's gonna be legacy applications that haven't been touched and don't-- that people don't even know how to touch that are still accessible to the internet right now, and I think that's gonna be a significant portion of them. And one theoretical scenario in that case is that someone will create a worm that will infest every single one of these accessible PHP applications to create an ultimate like botnet, right?

Like that's just an example. Like there's probably many other things that they could do with [00:14:00] this information, but that's one thing that could be done with these k- types of CVEs

Matt: But when we patch a security problem, when a security problem is found, let's say, you know, leave AI out of it. If we, if go back five years and we find a, security issue and we advise sysadmins, let's just say it is a PHP issue just to make it simple. "Hey everybody, anyone on WordPress, you guys really need to go up to enter in, you know, a PHP version here, if you don't, we found this security hole and it could be exploited." The reason why we patched it is because the security hole existed.

Mikhail: Угу

Matt: We didn't patch it because it was or... You, you, you understand what I'm saying? My point is, is you're saying, "Oh, it's easier for people to, to exploit it,

Mikhail: Yep

Matt: exploit issues," and I agree that that is true. can use an AI to do it and they could break into stuff, right? I get it. They could use an LLM. even trick an [00:15:00] LLM that has some safety stuff in,

Mikhail: They could. Mm-hmm.

Matt: place to give them instructions, so then they just do it manually but with sort of, with the support, right? It's possible. It... So yes, 100% agree. But I would still have patched the PHP back then because I, I don't want one person to break in. is worse. If 12 people are beating at the door of some website that I built, that's bad. But I don't even want one beating at the door

Mikhail: I, I, I think, yes, I agree with you there. Like, obviously you're gonna patch it and I'm gonna patch it, but what about the ones that aren't patched or aren't gonna be patched?

Matt: You know, it, it, I agree. Like, we are gonna g- be going through a rough period,

Mikhail: Yeah

Matt: what really sucks about this is this: not everyone is available all the time.

Mikhail: Correct

Matt: We've been, toting probably for 10 years now, everyone should have a side hustle, this and that, this and that. Look how easy this technology is. There's many people out there that have, pre-AI even, gone through several guides, went step by step, built out a side hustle, built out a SaaS, [00:16:00] has no idea what PHP is and that other than step four of this guide was to, you know, select your PHP version,

Mikhail: Absolutely.

Matt: something up,

Mikhail: Mm-hmm

Matt: put it together, they put it up.

Or maybe they hired even a developer. The developer made it for them to 2017 standards, and they've been using it at 2017 security standards ever since, and of course, that's now inadequate, and they're completely unaware. Now, that is a problem, 100%, but it was always a problem. there is, there is a little bit of that.

However, now it's, it, it is an en masse issue.

Mikhail: Yeah

Matt: is problematic. I'm of two minds, clearly, right? I am flip-flopping, but I am of two minds. It sucks. what if you and I were unavailable this week, and all of a sudden, all these PHPs need to be updated? It, what it does is it puts, like, an on-call... Like, if someone's... T- take it away from us, take it away from us, 'cause we run a little company. Take it away from us. Talk about, you know, the person that has the side hustle. What if that person is unavailable for a week? have no idea [00:17:00] what, how to update the PHP. They figure out how to do it, they update the PHP, now the app is broken. So then now what? So it, it, it is a problem. It is a problem.

However, the problem was always there

Mikhail: it was definitely always there, but again, it, it's become democratized who can actually exploit that problem now. That's become a bigger issue is like...

Matt: with you there,

Mikhail: Yeah, yeah. And, and I don't know how to stop... Like, one, one suggestion that I saw that I was actually pretty funny, and I don't... Some might do it, is we create a, a reverse worm that will go through using this exploit and patch the systems that aren't being patched, and then go to the next one and the next one, which I thought was, like, that's brilliant.

Illegal, by the way. Don't do that if you're out there. Like, getting access to a system through this exploit is illegal, by the way.

Matt: don't be making viruses and doing

Mikhail: No.

Matt: Yeah, don't be

Mikhail: Correct. Even if you're going to create the, the exploit, [00:18:00] right? If you're gonna create the exploit, if you're gonna use the exploit to patch the system, that's still illegal.

That's like you going into someone's house and installing a security system without them asking you. Same thing. Can't do that

Matt: if it was my own website and I made my, I made a worm for myself

Mikhail: Oh, no, no, no, that's not illegal. That's,

Matt: like, well

Mikhail: that wouldn't be a worm. That would just be an application in that, in that s- in that case, like, right? So, like a script. So no, don't go into other people's websites and fix their shit just because it's altruistic. But Someone might do it, and that would solve this issue in cer- in a certain situation.

So, like, there are ways around it. there are ways, there are ways that we can benefit, benefit from, you know, the democratization of even solving vulnerabilities, but I don't know if that's gonna happen. And we're, I still think at, at the very early stages of this. I believe that a lot of companies are sitting on exploits just like they were pre-AI for different uses.

So, it, it's difficult for me to say, but, [00:19:00] like, most data that's sitting on servers, in my opinion, is essentially not secure anymore

Matt: Well, I, was actually just about to say to you, I think that this should reopen the conversation of what we should be putting in the cloud. I have

Mikhail: Yeah

Matt: with our clients to death. Many of them don't wanna hear it, honestly. Many of them kinda think, "Oh, it would be so convenient to have that information there."

And I always tell them, "Okay, so then when there's a leak, let everyone know that all your form data from the last ten years was on there with everyone's phone number." I said, "So now you have to contact somebody that, that contacted you at random 10 years ago, that filled in your contact form, and let them know, 'Hey, I left your form data, your form entry on the server,' just 'cause it might've been convenient maybe." And oftentimes, that does convince them. And we are partnered with a local IT shop who does provide hosting services, and they actually have, like, a policy [00:20:00] where you're not allowed to host things that are not beneficial to the functioning of the application or the website in question. I'll just throw out an example.

I'm making this up on the fly just 'cause I'm not gonna give away security details, but just hypothetically speaking, if you're a person that uses Woo- WooCommerce, WooCommerce can handle a lot of stuff for you, your inventory, your this, your that, whatever. But you can also hook WooCommerce up to, to other inventory systems, or let's say a CRM that has all your inventory and everything else as well, right? In that case that the website processed an order and then fired all that information over to that CRM or that other inventory handler, in my opinion, it should be deleted from the website, and it should go only on the CRM. You don't wanna have two places where a security hole could have that information.

Now, 100%, it would be very, very convenient for anyone on the website to see the information and anyone on the CRM to have it, have it there too, correct. But you have to take a look at how inconvenient it would be to have that information go out [00:21:00] into the wild

Mikhail: I 100% agree with you. I think it's always best practice to keep only the information that is absolutely necessary. The other thing that I c- I could s- see happening in the future, is if you do need, for example, the email and the password, or, uh, sorry, not the password, the email and the phone number to be in your database, you might have to start putting in the terms of service that this information will be available to the public, uh, preemptively.

And then the customer has to decide if they're okay with doing that. You know what I mean? Then at least you're giving them the option to opt in to that kind of service, uh

Matt: Well, I'll say this though, that sucks because this is malicious actors doing this. Like, th- th- this

Mikhail: Yes

Matt: you do give them a heads-up as to like, "Hey, there's likely gonna be an exploit and

Mikhail: Yeah.

Matt: is likely to be leaked," I mean, at the end of the day, you're giving in to the criminals, like,

Mikhail: Yeah. But, but the reality is, is that there is [00:22:00] likely to be an exploit and your inf- inma- information will be leaked. Like how much, how much has our information been leaked?

Like email and, like I don't even consider email and phone number to be sensitive information anymore because of the fact that it's been leaked so many goddamn times.

Matt: sure, but like

Mikhail: Yeah.

Matt: that do,

Mikhail: Sure, yeah

Matt: 'cause like holy crap, especially as a celebrity, you'll get 14 million text messages

Mikhail: so actually that's a perfect segue to, like, the people that this actually affects the most right now, in my opinion, are government entities, which has always been the case, government entities, like corp- corporations with industry secrets, celebrities. I believe that for them, the game has changed even more than it has before.

Like they, you know, if they're signing up for Instagram or anything like that, like they've gotta think 10 times more, more about what they're exposing to, to these sites and what they're not exposing. Same with corporations, like what are they putting on shared servers? What are they... Where are they storing their s- industry secrets?

Like, where do you store your [00:23:00] most sensitive information if you're Apple at this point, or if you're the United States government? Like, where do you put that? You air-gapped systems. But like the problem is, is that it goes further down. Like we, we haven't talked about this yet, but there's been hardware exploits that have been coming out on a pretty consistent basis, which means that if a spy has access, like e- even if the system's air-gapped, which means it has no access to the internet, right?

If the hardware itself is exploitable, then a, a spy could come in and plug something into the hardware or solder something, and still get the information. So like it's gotten so many levels deep that do you just keep it in your head?

Like, what do you, what the hell do you do? Like I, I don't know. if I was recommending something to a company, I would be at a loss today. I would have to really look into it, and none of my recommendations would ever come with like 100% secure, like guarantees because we're

Matt: think anyone ever says 100% secure. I mean, maybe [00:24:00] some, but

Mikhail: Well, like nuclear secrets, like ha-- yeah, like nuclea-- how, what about nuclear secrets?

How's this gotta be 100%?

Matt: I don't even think those are 100%.

Mikhail: Well, they aren't.

Matt: ideally

Mikhail: Yeah.

Matt: of

Mikhail: I don't think they are, but like they, they, you know, it, it's just the, the reality like of the situation is they should be absolutely 100%. So there's

Matt: why they use, what is it, '70s computers for the nuclear missiles, 'cause they're

Mikhail: Ja

Matt: physical based. They're, they're unhackable

Mikhail: They're, yeah, they're air-gapped. I-- See, unhackable is a thing that I, I'm very hesitant to say because again, if someone gets access... So y- your only opportunity is to prevent physical access, right? Like, the, you focus all of your efforts on preventing physical access. That's a good, that's a good air gap.

Like, you know, y- the government... The, the military's pretty good at that. Like, they, they have, they have strings to pull on preventing physical access. But like a corporation, right now it's very difficult. There's, North Korean hackers, there's a lot to be said around them being in pretty much every Fortune 500 company to some degree, [00:25:00] either as contractors or as full-time employees, and them just sitting on it Like they're, they're just working.

They're, they're getting their salary and they're sending it back to North Korea. Like that's that-- There's been ac- There's been, some uncovered in the past that have, that we've found that that's exactly what's happening. The way to do background checks, KYC, like know your customer, know your, know your employee, that stuff is, has already been proven to not be 100% accurate in any way, shape, or form.

So even if you run all those background checks, even if you run your KYC, even if you do that, there's, there's still people slipping through. So like now you have probably bad actors, whether it be from Russia, from North Korea, from, you know, China, whatever, sitting in most of the Fortune 500 companies. So now you're attacked from the inside as well.

So you have-- There's a lot of things you can do on the inside where like you, you know, you know, least privilege, um, kind of thing and, you know, you don't allow people all the access [00:26:00] that they would requ- that, that they don't need. But I don't know, like being a SOC right now, being a sysadmin, that to me sounds like one of the hardest jobs out there that is pretty much a losing game.

That's what I'm getting out of this. Like I don't know what you do, and I, I think the, the smart thing to do is like to figure out how you can use AI to combat it, like just have AI be constantly pen testing 24/7 your systems. Like figure that out, because that'll obviously close up holes that the other AIs are already using.

Like use the best AI available, spend a lot of money on it, and get into these programs like the Daybreak programs and the, all of the, the programs that the OpenAI and Anthropics allow you to use a little bit more combative systems. Well, like we're, we're, we're in a couple of those programs with Cipherin, so like you, you're allowed to use some of the more like attack-y things, if you get into those programs.

So you could do that to attack your own systems. That's what they're [00:27:00] for. If you're just constantly attacking your own systems across all the board. And ethical, yeah, ethically hacking your own stuff. That's one recommendation I would have. The other is like just getting consistent pen tests, and I'm sure the SOCs know better than I do like how to, you know, mitigate these situations.

But I can almost guarantee you that this week they're just, maybe not even this week, this whole year, they're probably just like sitting like this with their head, hands on their head and just being like, "We're f- we're fucked." That's a kind of the, the working title of this episode. But like I, I don't envy anyone in that position that has to secure very, very confidential systems.

If you're securing like a SaaS, a SaaS system where, where you're literally just storing email and phone numbers, I'm not... It still sucks, but like that's, you know, that, that stuff, I'm like, you do your best obviously, but most likely all of those phone numbers and emails have already been leaked so many times that worst case scenario you'll have to inform your customers of a leak, [00:28:00] which happens all the time

Matt: this has happened before. Obviously not AI, but if this has happened before where there's been a seemingly a security setback. I would also like to say this, is I know that patches and fixes and things are not literally immediate at the snap of fingers, but they're pretty dang quick. you know, hats off to who find these issues and then also fix and deploy and, and figure out what to do. I mean, I was informed originally with the PHP stuff. I kind of saw the rumors. It was like, okay, well, I don't know, you know, it's rumors. I'm not gonna, you know, run around with my head cut off 'cause of rumors. And then I'm on the, the Wordfence, newsletter, on the Wordfence newsletter, and they let me know like, "Hey, you should do this. You should do that. Hey, these are, you know, this is the problem with this. This is a problem with that. You should fix this. You should fix that." And like, hey, even hats off to them.

You know, hats off to the PHP people. You know, [00:29:00] hats off to anyone who's, who's fixing this stuff, who's on the front lines. Like, I mean, they're doing it, and, like, they are doing... I think they're doing a good job. I know some people say this is unacceptable, but this is a, a, a human issue. the entire human race, like there are, you know, malicious actors, and then there's also the people that are just trying to use technology the way it is.

And then obviously there's security involved with that, and unfortunately, it's kind of an ongoing war that has always been ongoing If I had an original WordPress website, like just an original WordPress website, of course that's not adequate to today's standard, but it was adequate back then, so look how far we've come. So just as a positive note, you know, kinda to finish up on, I think that at least we have people on the front lines fighting. Y- you and I have to do some sysadmin stuff, admittedly, you know? But I wouldn't know how to take over the... I watched a video of them showing how the, memory allocation, it was like you interrupt a task on PHP and then it, like, leaves this thing open so then you use like a session API or a session [00:30:00] function and you...

I'm not a PHP developer, clearly. I'm not gonna be describing it correctly. I'll actually link in the show notes to the video that I watched. But quite literally it's like thank- thankfully you know what, like thankfully you know that because there's many of us that are more creators and less security.

I don't really care how the memory allocation works, you know, as long as it works more or less. So thankfully there are people that make firewalls and thankfully there are people that send out newsletters that tell me, "Yo man, you should update your PHP to these versions." And, um, we even had a client who, uh, due to the nature of their business got contacted by a, a ministry here which is like a government office here in Canada, got contacted by a ministry and they, even they gave me a heads-up but I was already in the midst of fixing their stuff so it didn't, it didn't matter.

But I mean, yeah, this was all within 24 hours. Within 24 hours I had seen the rumors, the thing came out, I was then informed what to do and how to do it and then I just did [00:31:00] it. I mean there isn't a, like I, there at- off to that process. Like

Mikhail: Yeah

Matt: we have that and I'm not paying for that by the way. So just a

Mikhail: that's it

Matt: like, you know, that we... the, is, is hard-fought

Mikhail: Mm-hmm.

Matt: im- maybe they're losing a little bit right now but at least they're there because I don't know what to do

Mikhail: I, I agree with you. And like, it, it is a massive task, by the way. Like, they-- it, it was a little bit rough in the beginning because once they start-- once they actually started shipping these patches, the next difficult task was, like, everyone's downloading them at the same time. So the servers got rocked

Matt: Of

Mikhail: just to download the patches.

So I had to, like, continually every five minutes try to download them. I wasn't gonna, woodpecker them to death . Like, I was gonna-- I was doing it slowly, as far-- as slowly as I possibly could, but I still needed to get my stuff fixed. So it took me, you know, probably, like, mmm, 45 minutes of, of that cycle of, [00:32:00] like, every five minutes doing it until I actually got a successful install.

But I agree with you. Like, tho-those problems aren't easy to solve. and they were solved really quickly. I don't know why there was that gap. I don't know if someone, like, preemptively released the CVE before they had a chance to patch it on purpose, or if that was, like, a mistake, or if the-- I, I'm, I'm curious about that miscommunication that happened, 'cause that's not supposed to happen.

The CVE should-- like, probably was reported correctly. They probably fixed it, and they obviously, um ... There, there was that discrepancy there. Now, I've, I've also heard that this CVE, since it's been around for 20 years, someone potentially, I think Cloudflare noticed that even a month ago, someone was using this CVE to try to take over servers, right?

So, so they, they, they were able to go back in time and look at the stuff. Now, by the way, if you're listening to this and you're, like, a, you know, junior sysadmin, one thing that you have to [00:33:00] do if you're in charge of servers is assume that the server was already accessed. So you have to go through logs to see if the CVE was actually used and, and, like, malware was put on your server.

So that's one thing you should be doing, because again, there was a gap in time after it was released, and then there could've been also people that figured this out a long time ago. So that's one thing that you should do. I found that if you're not, like, super comfortable at reading logs, AI is really good at going through logs, if you give-- especially if you give it, like, "Hey, this CVE, go check the logs of this server."

Matt: Well, this is, this is another reason I would like to say as well, 'cause you and I are primarily builders.

Mikhail: Correct

Matt: is another reason why sometimes, like, I could take a, a computer

Mikhail: Yeah

Matt: or take, like, buy, build the computer, make it into a,

Mikhail: Mm-hmm.

Matt: a server. I know how to do that. I used to be a sysadmin. The thing is, though, is our job currently is mostly we're builders, so this is why we do buy managed services. There's

Mikhail: Yes

Matt: that we purchase managed services or we work on managed services

Mikhail: Mm-hmm.

Matt: give you... Like, I'll give you an [00:34:00] example. Like, if you buy, shared hosting or just managed hosting, which is kind of the same thing,

Mikhail: Mm-hmm.

Matt: It'll, it'll, like, when I went into some of my PHP updaters, it was like, "You can't do this.

You can't do this. You can do this." That's a sysadmin above me, you know, in the cloud that's deciding that stuff. So they, they, they give me, they kind of extend the olive branch and say, "You can administer this server to a point, but we have the final say." They're the ones that had a blog post out right away, told us what was patched, told us what to look for. there was some discrepancy on one of the servers, 'cause we work on some clients had... They already had their own hosting, so we just work on their own hosting, their old hosting. We had a, a discrepancy where it wouldn't show the new version. There was a clear blog post from that person's host that stated, "You may see the old version, but if you see the build number this, we have applied the patch.

The next monthly security update, it will show the correct version, but we have patched the issue." Like, you know, uh, so us as builders, like, we kinda scratch the surface of this, is [00:35:00] my point, and that's why we buy managed services, so that the, they, the people that we just, that we employed, that we paid for, they assume the server was hacked.

They check if there was a data leak. They do all that, and then they only give us some tools via cPanel, Plesk, or some other sort of administrative tool. They give us that. That's why much, uh, many a time will not necessarily go for a, a completely unmanaged VPS, for example

Mikhail: Yeah, and I th- I think there could be more arguments now more than ever than not to do the VPS side. Now I might, I might get some flame for that.

Matt: VPS, like a managed host. You

Mikhail: Yeah.

Matt: want a team of security,

Mikhail: Yes

Matt: team managing as much of the security as possible. And then what do we do? We do what they say. "Hey, guys, make sure you click this button and make sure your PHP's at this version." "Okay."

Mikhail: Yeah. Now, having said that, we're, we are seeing these managed hosts get [00:36:00] massively owned, so like Railway is a good example. That's a managed host, and that led to some issues. You know, Vercel is a managed host, no issues there. But w- whatever host you go to, there is a very high chance that it could also get pwned right now

Matt: No, y- no, yes, it's, it's imperfect, but my

Mikhail: Yes.

Matt: want a security team there.

Mikhail: Yes. They, they're gonna be on it.

Matt: you have a VPS, and someone hacks you and

Mikhail: Yeah

Matt: Bitcoin, y- uh, mining or whatever on your server 'cause they're doing remote e- remote code execution, first of all, you don't know what remote code execution is.

You might

Mikhail: Correct

Matt: Bitcoin is, talking to people who would be running a side hustle, right?

Mikhail: mhm

Matt: You might not even know what that is, and you're like, "Why is my server so slow?" You have no idea what's going on. You have no idea what's going on. But somebody who, in a managed situation where the, where part of that management is them having firewalls, is telling you, "Hey, man, it looks like you've been h- you've been pwned," is them having a proper backup and proper something like an antivirus, malware [00:37:00] scanners, right?

They 'Cause we, we've had, we had an isolated environment years ago, it was managed, and we would, it was completely, like, not for the public, but we used to upload ... Like, if somebody sent us a WordPress site, they were like, "Here's my site," and it was just the files, it's like, okay, has this thing been hacked?

I would upload it to that. Again, this is completely private, but it's managed. They had all this software and everything, but we would get, we've gotten emails that's been like, "Hey, it looks like this exploit has been installed on this wei- website," and it's like, oh, okay. Or the automated system, the antivirus, would flag it and say, "Hey, it looks like this malware is in there."

So now it's like, okay, w- we intentionally did this in an isolated environment. Now we know that this WordPress, website has been compromised, and now we have to have that conversation with the client, But here's the thing. How long were they running that WordPress in a compromised state? It could've been a decade

Mikhail: Yes

Matt: That's what I mean.

So you want that [00:38:00] team, 'cause even you and I, like, w- you know, we can go and look and be like, "Okay, has someone been in here?" You and I can read logs. I would prefer for them to just shoot me an email and be like, "Yo, man, they're exploiting something on PHP 7.1. Like, you gotta fix this." Like, okay. that's literally what we would receive sometimes

Mikhail: Yep. It's true. Like, uh, there, there's a lot of positives there, and honestly, like the railway incident, for example, I, it... I would trust them more now than before. That's how, that's how I feel about that. As long as they, as long as they, you know, they, they, they messaged everyone correctly, you know, they handled it correctly, they are going to be way more hardened now because they're gonna take this very seriously and, you know, they're gonna be in a good spot.

So again, like managed hosts treat this stuff very seriously a lot of times, most of the time. And so incidents happen.

Matt: Incidents

Mikhail: sure that you're, make sure that you're on top of it. Make sure you read the incident reports that they send you. They usually send everyone that got affected by [00:39:00] an, by something an incident report.

But you can almost guarantee that they won't happen again. Like I haven't-- I, I don't know of a time, like... Yeah, yeah, that's what I mean, like they won't, they won't get pwned again in that way. And usually what happens is like if they get pwned once, they'll actually go through and do an even more in-depth review of making sure to not get pwned in different ways.

So like during these processes, some benefits can happen and, uh, a lot-- I guarantee you a lot, most of the other services are also looking very, very closely at all their systems right now. So it, it's, it's a very like tense time, but out of that will come a more secure environment for everyone. so yeah, it's, it's a good point on the managed stuff for sure.

Matt: as, as long as your host wasn't, like, negligent,

Mikhail: Correct

Matt: absolutely, like, reckless or, like, just making s- like, careless mistakes.

Mikhail: Mm-hmm.

Matt: Like, I'm still even down for

Mikhail: Yeah

Matt: you made a mistake. It's a mistake. I understand." Some people aren't that lenient, but,

Mikhail: Mm-hmm.

Matt: am if it's a legitimate mistake, it's legitimate.

But it's like, if it's like, "Oh, yeah, I accidentally left that on PHP 5," it's like, "Dude

Mikhail: Yeah

Matt: What are you doing?

Mikhail: That's correct [00:40:00] But yeah, um, curious to hear what everyone thinks. are we over-exaggerating the situation? Do you think that everything's gonna calm down and AI will battle AI and everything will be more secure? That's one outcome. Or will this continue to escalate until we have to shut down portions of the internet?

I don't know. I, I guess that's like a worst-case scenario. we're in a weird time and it's definitely, if anything... If nothing else, it's very interesting, but also scary

Matt: I would say if you have any doubt about the security you have on your website or any other part of your digital life, I would take a look at it now. And take a good look. That's, that's my recommendation. 'Cause you

Mikhail: call

Matt: many pieces of security between you and a malicious actor as possible

Mikhail: Absolutely. But that's it. Yep, that's the web news. All right, see you next week. [00:41:00] Goodbye