Transcripts
This week on the backup wrap-up we dive into the disastrous data loss
Speaker:event that hit dedoose a cloud-based research data management platform.
Speaker:Back in 2014.
Speaker:Dedoose lost access to their primary data and their backups simultaneously
Speaker:resulting in the loss of over three weeks worth of customer research.
Speaker:We discussed what caused the crash as well as some questionable backup practices.
Speaker:These practices resulted in a ton of data loss for doctoral researchers who
Speaker:lost countless hours of painstaking work, they will never get back.
Speaker:We discuss what dedoose did right, and what they could have done
Speaker:better so that we can all learn.
Speaker:If you're new to the pod, I'm w Curtis Preston, AKA Mr.
Speaker:Backup.
Speaker:And there's a reason I'm so passionate about this topic.
Speaker:When I first started in backups over 30 years ago, my company lost an important
Speaker:database that I couldn't restore.
Speaker:Since then I've dedicated my career to making sure that would never again happen
Speaker:to me or anyone who will listen to me.
Speaker:We turn unappreciated backup admins into cyber recovery heroes.
Speaker:This is the backup wrap-up.
Speaker:Welcome to the show.
Speaker:W. Curtis Preston: Welcome to the show.
Speaker:I'm your host, w Curtis Preston, AKA, Mr.
Speaker:Backup, and I have with me my tax non-ad advisor, Prasanna Malaiyandi.
Speaker:How's it going?
Speaker:I, I've needed you a lot over the last few days is
Prasanna Malaiyandi:It is that time of the year, Curtis, you know,
Prasanna Malaiyandi:I wouldn't expect anything else.
Prasanna Malaiyandi:For, for our international listeners who don't know , april
Prasanna Malaiyandi:15th IS a US' day that you have to file your yearly tax income tax
Prasanna Malaiyandi:with the government, with the IRS
Prasanna Malaiyandi:W. Curtis Preston: You can file an extension, but any money that you
Prasanna Malaiyandi:owe has to be paid by that time.
Prasanna Malaiyandi:This was, I'm gonna say one of the, if not the most complicated years that
Prasanna Malaiyandi:I've ever done tax, because halfway through the year I was laid off.
Prasanna Malaiyandi:And so then I went, you know, self-employment and I went like nine
Prasanna Malaiyandi:different ways of self-employment.
Prasanna Malaiyandi:And um, you know, and I had, I had some carryovers from previous years and I had
Prasanna Malaiyandi:some, I had some of this, some of that, and I'm like, I think I'm gonna be okay.
Prasanna Malaiyandi:I might not be okay.
Prasanna Malaiyandi:Turned out I was way okay.
Prasanna Malaiyandi:I was very okay.
Prasanna Malaiyandi:But you may recall that there was a day there or so where I was not having any.
Prasanna Malaiyandi:I recall that.
Prasanna Malaiyandi:And, and for people who may not understand the complexities of the US
Prasanna Malaiyandi:tax system, they're probably thinking, oh, don't you just like take your
Prasanna Malaiyandi:income and they just take a check for based on a certain percentage.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:That's not how the US system works.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:And well, and then when you add in the complexities of, um, some of the
Prasanna Malaiyandi:different types of self-employment that I did, there's some very specific
Prasanna Malaiyandi:deductions that you have to take and then you have to really record stuff
Prasanna Malaiyandi:to, to, to make those deductions.
Prasanna Malaiyandi:And, and, uh, there was, um.
Prasanna Malaiyandi:It was just,
Prasanna Malaiyandi:I remember the picture you sent of just like, Hey, here
Prasanna Malaiyandi:are the things I have to be tracking in order to be able to do this deduction.
Prasanna Malaiyandi:I was like, oh my gosh, Curtis, there must be a better way to
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:I, and, and I, I will say I did figure out a better way for that one.
Prasanna Malaiyandi:I've only done that, that particular one for like seven years.
Prasanna Malaiyandi:And, uh, I never had figured out a better way.
Prasanna Malaiyandi:And actually this year, for the first time, I figured out a better way.
Prasanna Malaiyandi:So 2024 will be better for
Prasanna Malaiyandi:W. Curtis Preston: 2024.
Prasanna Malaiyandi:And also, uh, I, you know, uh, you helped me, uh, you know,
Prasanna Malaiyandi:come up with a spreadsheet.
Prasanna Malaiyandi:I mean, I did it on my own, but, you know, there was some,
Prasanna Malaiyandi:definitely some advice in there.
Prasanna Malaiyandi:And man, you're right, the US tax code is super complicated.
Prasanna Malaiyandi:Um, you know, because, you know, we have a progressive tax system, so like
Prasanna Malaiyandi:they tax this percentage on that amount and this percentage on that amount and,
Prasanna Malaiyandi:and, uh, so that, so it's not just.
Prasanna Malaiyandi:Figure out how much you made and take X percentage of it.
Prasanna Malaiyandi:It is so not that.
Prasanna Malaiyandi:And of course, I live in California and we have a state income tax,
Prasanna Malaiyandi:so we have that in addition.
Prasanna Malaiyandi:And just in my tax bracket from the lowest, you know, there
Prasanna Malaiyandi:are, what, what did we find it?
Prasanna Malaiyandi:Six, six different layers of that.
Prasanna Malaiyandi:It starts at 1%, then 2%, 3%, 4%, you know, all the way up to like 9%.
Prasanna Malaiyandi:I think, um, you know, once you go.
Prasanna Malaiyandi:12.3.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Is
Prasanna Malaiyandi:W. Curtis Preston: Oh, well, no.
Prasanna Malaiyandi:Well, I, well, I know it goes past that, but like, but like, you know,
Prasanna Malaiyandi:if you make over a certain amount, you end up paying 9% on that final, uh,
Prasanna Malaiyandi:on the, on the marginal, that's what's called the marginal tax rate, uh,
Prasanna Malaiyandi:So it's like every dollar you make
Prasanna Malaiyandi:additional gets taxed at that rate.
Prasanna Malaiyandi:And yeah.
Prasanna Malaiyandi:And then the other thing challenge is you have kids.
Prasanna Malaiyandi:They have different deductions, credits and deductions change
Prasanna Malaiyandi:every year, depending on what gets passed, what doesn't get passed.
Prasanna Malaiyandi:Your brackets don't stay fixed because some are adjusted based on the rate
Prasanna Malaiyandi:of inflation, but some are based on one inflation rate, others is
Prasanna Malaiyandi:based on a different inflation rate.
Prasanna Malaiyandi:So it's a giant mess.
Prasanna Malaiyandi:You basically need a PhD in order to be able to do this stuff and keep up with
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Yeah, but through, but through it all, I have contacted my
Prasanna Malaiyandi:financial non-ad advisor.
Prasanna Malaiyandi:Prasanna Malaiyandi
Prasanna Malaiyandi:let's just say you are not the only person
Prasanna Malaiyandi:who talks to me about, uh, taxes.
Prasanna Malaiyandi:W. Curtis Preston: Is that right?
Prasanna Malaiyandi:I, that, that does not surprise me.
Prasanna Malaiyandi:'cause you're a, you're a smart, you're a smart little person.
Prasanna Malaiyandi:It's just random bits of knowledge that I have in my head.
Prasanna Malaiyandi:W. Curtis Preston: yeah.
Prasanna Malaiyandi:I love, I love how you can just, you can just quote the marginal tax rates for
Prasanna Malaiyandi:different, for different levels of income.
Prasanna Malaiyandi:Um, yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:And you're like, oh, yeah, with the self-employment tax, you know,
Prasanna Malaiyandi:you could deduct half of it, right?
Prasanna Malaiyandi:I'm like, no, I didn't know that.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:And of course, I, I bet a significant portion of our listeners go, yeah,
Prasanna Malaiyandi:no, that's why I have a guy, right?
Prasanna Malaiyandi:I just give the money to the guy and the guy just tells me, um, you know, uh, yeah.
Prasanna Malaiyandi:Um.
Prasanna Malaiyandi:Prasanna Malaiyandi: useful to learn, right?
Prasanna Malaiyandi:Because then you could figure out the impact of certain things that
Prasanna Malaiyandi:you're gonna be doing or like how you should be withholding.
Prasanna Malaiyandi:And honestly, like it's great to have a guy, but the guy
Prasanna Malaiyandi:charges money and it's expensive
Prasanna Malaiyandi:W. Curtis Preston: charge money.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:I, I think one meeting with the guy, and by the way, it
Prasanna Malaiyandi:could also be a girl, right?
Prasanna Malaiyandi:Just, you know, one meeting with the guy and you, uh, you've spent
Prasanna Malaiyandi:like the entire thing of just what I spent with TurboTax for the year.
Prasanna Malaiyandi:And that's just one meeting, right?
Prasanna Malaiyandi:Um, yeah.
Prasanna Malaiyandi:So.
Prasanna Malaiyandi:My guy's free, except he is not allowed to be my guy.
Prasanna Malaiyandi:He's, he is my non-ad advisor, legally is not giving me financial advice.
Prasanna Malaiyandi:Um, yeah, our financial advice is get a guy.
Prasanna Malaiyandi:Um, anyway.
Prasanna Malaiyandi:Prasanna Malaiyandi: Well, at least it's done.
Prasanna Malaiyandi:W. Curtis Preston: At least.
Prasanna Malaiyandi:Yeah, it is done both last year and this year.
Prasanna Malaiyandi:I have filed my estimated taxes for Q1, like a week early.
Prasanna Malaiyandi:I
Prasanna Malaiyandi:Look at that, Curtis.
Prasanna Malaiyandi:And I'm hoping that a lot of people who have complicated
Prasanna Malaiyandi:taxes don't wait till the end.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:I'm just so super happy to be done with my taxes, both for the last
Prasanna Malaiyandi:year and for this, this quarter.
Prasanna Malaiyandi:I'm so super happy, super excited.
Prasanna Malaiyandi:Uh, but we're gonna talk about something not so exciting.
Prasanna Malaiyandi:We have yet another episode of, you know, cloud disasters and this one.
Prasanna Malaiyandi:This
Prasanna Malaiyandi:I don't know about you.
Prasanna Malaiyandi:W. Curtis Preston: what, go ahead.
Prasanna Malaiyandi:I don't know about you, but I've kind of enjoyed this series.
Prasanna Malaiyandi:I know it's sort of taking joy in other people's misery, but
Prasanna Malaiyandi:I've found it to be kind of, I.
Prasanna Malaiyandi:Like interesting, right?
Prasanna Malaiyandi:To kind of look and figure out, hey, what did people do?
Prasanna Malaiyandi:What happened?
Prasanna Malaiyandi:Because these are things that could happen if you do something wrong or
Prasanna Malaiyandi:misconfigure something, or maybe it's not even your fault and you just ended up
Prasanna Malaiyandi:getting attacked by a malicious user and
Prasanna Malaiyandi:W. Curtis Preston: Yeah, I, I.
Prasanna Malaiyandi:I will say that this story, there's one particular part in the middle of it that
Prasanna Malaiyandi:is the crux of what happened that is so bad backup design that we're gonna,
Prasanna Malaiyandi:we're gonna get to right at the core.
Prasanna Malaiyandi:At the core, they had some other mistakes that, you know, the, the same mistakes
Prasanna Malaiyandi:that everybody else makes and, um, and, and sadly it appears that this was
Prasanna Malaiyandi:one where they stepped on their own.
Prasanna Malaiyandi:Toe or whatever, right?
Prasanna Malaiyandi:Um, but this was not an attack.
Prasanna Malaiyandi:This was just, uh, you know, some of the headlines, you know, use phrases
Prasanna Malaiyandi:like fat fingering and stuff like that.
Prasanna Malaiyandi:Uh, we are of course talking about a company who I'd never even heard of prior
Prasanna Malaiyandi:to this story, and it's called Dedoose.
Prasanna Malaiyandi:At least I think that's how it's pronounced because they are a, um,
Prasanna Malaiyandi:uh, you know, what do they describe?
Prasanna Malaiyandi:They're, they're a cloud-based, qualitative and mixed methods research,
Prasanna Malaiyandi:data management and analysis application.
Prasanna Malaiyandi:That's a mouthful.
Prasanna Malaiyandi:I.
Prasanna Malaiyandi:And they were in Manhattan Beach, California, which for the record is
Prasanna Malaiyandi:right up the road from me and right down the road from our friend, uh,
Prasanna Malaiyandi:Jeff Rochlin, uh, that's literally like right next door to him.
Prasanna Malaiyandi:And they, they were designed to help researchers organize, analyze,
Prasanna Malaiyandi:and collaborate on qualitative and mixed methods research data.
Prasanna Malaiyandi:And one particular group of people that used this platform a lot were people
Prasanna Malaiyandi:doing their doctoral dissertation.
Prasanna Malaiyandi:I.
Prasanna Malaiyandi:And so this is gonna
Prasanna Malaiyandi:a bunch of research.
Prasanna Malaiyandi:W. Curtis Preston: bunch of research and that, that you're just getting
Prasanna Malaiyandi:ready to file to, to, to, you know, to what, what do you call, do you,
Prasanna Malaiyandi:what do you do with your dissertation?
Prasanna Malaiyandi:You file it, you.
Prasanna Malaiyandi:You defend.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Um, present and defend.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, and you're just at that moment where you're about to do that.
Prasanna Malaiyandi:And, um, the um, uh, and, and this, then this bad thing happened.
Prasanna Malaiyandi:So they were, they had some competitors, uh, I'll just finish this part.
Prasanna Malaiyandi:They had some competitors, Atlas Ti NVivo.
Prasanna Malaiyandi:And Max does, they all have these, these interesting names, but they were
Prasanna Malaiyandi:apparently the only SaaS based, um, you know, web-based, uh, portal, which made
Prasanna Malaiyandi:them the only game in town for this, for doing it the way that they were doing it.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:And my guess is people, like if you're an academic, you're probably
Prasanna Malaiyandi:like, yeah, I'll just throw it into spreadsheets or throw it into Microsoft
Prasanna Malaiyandi:Access and just run it locally.
Prasanna Malaiyandi:Um, but that doesn't really scale.
Prasanna Malaiyandi:And then you have to manage it, and then you have to worry about how
Prasanna Malaiyandi:do I protect it and back it up, and what happens if my laptop crashes?
Prasanna Malaiyandi:How do I sync it over to OneDrive or wherever else it has to go?
Prasanna Malaiyandi:And yeah, so I could see the appeal of this company.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, absolutely.
Prasanna Malaiyandi:You mentioned it was a cloud provider.
Prasanna Malaiyandi:They are hosted on top of Azure,
Prasanna Malaiyandi:W. Curtis Preston: Yes, they are hosted on Azure.
Prasanna Malaiyandi:And this was one of the reasons that we wanted to cover this because we were
Prasanna Malaiyandi:trying to get a story from each of the major cloud providers, and this was the
Prasanna Malaiyandi:biggest story that we could find on Azure.
Prasanna Malaiyandi:But I don't know if the fact that they were on Azure was
Prasanna Malaiyandi:really part of the story.
Prasanna Malaiyandi:It almost was as we're gonna cover it in the story.
Prasanna Malaiyandi:But, um, but in the end, I think this was all them.
Prasanna Malaiyandi:Prasanna Malaiyandi: Actually, I take it back.
Prasanna Malaiyandi:It's Azure, not Azure.
Prasanna Malaiyandi:W. Curtis Preston: Oh, it depends on who you ask.
Prasanna Malaiyandi:Um, we could go on, we could go on Microsoft copilot and
Prasanna Malaiyandi:ask them how to pronounce it.
Prasanna Malaiyandi:Do you want to describe what, what actually happened?
Prasanna Malaiyandi:Yeah, so everything was going fine, right?
Prasanna Malaiyandi:They had the service up and running, and then sometime in 2014
Prasanna Malaiyandi:they encountered an event, right?
Prasanna Malaiyandi:And it was basically a failure of a service.
Prasanna Malaiyandi:Running on top of Azure, which initially they said was a Microsoft service that
Prasanna Malaiyandi:had an issue, but turns out it was just their service that they had been
Prasanna Malaiyandi:operating that just leverages Microsoft component infrastructure to run on.
Prasanna Malaiyandi:W. Curtis Preston: right.
Prasanna Malaiyandi:And so, uh, what ended up happening is that failed and they
Prasanna Malaiyandi:basically lost access to their databases, which hosted the data for the end users.
Prasanna Malaiyandi:They also happened to lose access to their data itself and
Prasanna Malaiyandi:the backups at the same time.
Prasanna Malaiyandi:And so a lot of bad things happened all at once.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:I mean, this is why we back up, right?
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Because bad things happen.
Prasanna Malaiyandi:Um, what I, what I don't, just don't fundamentally understand is
Prasanna Malaiyandi:they go, they describe it as we were in the middle of our database
Prasanna Malaiyandi:encryption and backup process.
Prasanna Malaiyandi:What does that mean to you?
Prasanna Malaiyandi:I read that as they're encrypting the database
Prasanna Malaiyandi:before they write out a backup.
Prasanna Malaiyandi:W. Curtis Preston: Right.
Prasanna Malaiyandi:But
Prasanna Malaiyandi:And so my,
Prasanna Malaiyandi:W. Curtis Preston: place
Prasanna Malaiyandi:no, I
Prasanna Malaiyandi:W. Curtis Preston: as an event.
Prasanna Malaiyandi:my guess is, well, it's almost like streaming, right?
Prasanna Malaiyandi:So I think as they're reading data out, they're encrypting it and writing
Prasanna Malaiyandi:W. Curtis Preston: Okay.
Prasanna Malaiyandi:Okay.
Prasanna Malaiyandi:It's just that it's just that the event damaged the database.
Prasanna Malaiyandi:That's why, that's what I don't understand is why would a backup
Prasanna Malaiyandi:event damage the database?
Prasanna Malaiyandi:Well, it's like the old days of Oracle, right?
Prasanna Malaiyandi:And I know you've been in this situation so many times, right, Curtis, where
Prasanna Malaiyandi:it's like, Hey, you did a backup, you took down my production database, right?
Prasanna Malaiyandi:They might have been an issue.
Prasanna Malaiyandi:And I know that in all your scenarios, Curtis, that's
Prasanna Malaiyandi:always been not the case, right?
Prasanna Malaiyandi:W. Curtis Preston: Right.
Prasanna Malaiyandi:That backup doesn't take down production, right?
Prasanna Malaiyandi:Maybe they just didn't code for it properly, or maybe there
Prasanna Malaiyandi:was just some race condition,
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:some reason while they were doing a backup.
Prasanna Malaiyandi:Our production database also had a failure.
Prasanna Malaiyandi:W. Curtis Preston: Right.
Prasanna Malaiyandi:And I think this was good that they were encrypting the backup, but
Prasanna Malaiyandi:based on what I'm hearing, what I'm inferring from this is that this was
Prasanna Malaiyandi:a homegrown backup system, right?
Prasanna Malaiyandi:Because they, they, they seem to.
Prasanna Malaiyandi:Make a big deal about the fact that they're encrypting and, and
Prasanna Malaiyandi:then, and then, you know, and then backing it up, it's like, okay,
Prasanna Malaiyandi:well that's sort of standard.
Prasanna Malaiyandi:You know, that's SOP for most people, right?
Prasanna Malaiyandi:But, um, so they had a failure of the primary database as they were backing
Prasanna Malaiyandi:it up, they were backing up and this, this was the crucial part that I
Prasanna Malaiyandi:was talking about earlier, is that.
Prasanna Malaiyandi:It's apparently they didn't have enough space or whatever that they
Prasanna Malaiyandi:ba they're overriding the last good backup of the database as they're
Prasanna Malaiyandi:backing it up and slash or they're only backing up like once a month because,
Prasanna Malaiyandi:a bunch of incrementals
Prasanna Malaiyandi:W. Curtis Preston: well, but if they were, but if they were doing incrementals,
Prasanna Malaiyandi:they would've been able to restore to a better point in time because.
Prasanna Malaiyandi:So maybe we should talk about that point in time that they
Prasanna Malaiyandi:actually were able to restore four.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:So after, after, yeah.
Prasanna Malaiyandi:I guess we, we kind of jumped ahead, so we're doing the research afterwards.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So, um, yeah.
Prasanna Malaiyandi:So let's back up to, you talked about the event.
Prasanna Malaiyandi:So after that event, they, they went out and they did what we tell people to do.
Prasanna Malaiyandi:They put out a blog post and, uh, they're like, here's the situation.
Prasanna Malaiyandi:Uh, we believe we can go back to, this was May of 2014 . We know for a fact
Prasanna Malaiyandi:we can, that the March backup is good.
Prasanna Malaiyandi:We think we can also restore the April backup.
Prasanna Malaiyandi:We unfortunately believe that we are going to be unable to restore.
Prasanna Malaiyandi:The, the, the May backup.
Prasanna Malaiyandi:And so every, all of the changes and anything that you put in the system for
Prasanna Malaiyandi:the last two to three weeks will be gone.
Prasanna Malaiyandi:So I am, I am inferring from that, that they only backed up once a
Prasanna Malaiyandi:month because otherwise they would've been able to get a lot closer
Prasanna Malaiyandi:and it wasn't just the database, like I was reading Reddit
Prasanna Malaiyandi:posts where they're like, Hey, we talked to a service person or a support person.
Prasanna Malaiyandi:They basically said if you created a new account, even in that May
Prasanna Malaiyandi:timeframe, you're basically out of luck.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:So it
Prasanna Malaiyandi:W. Curtis Preston: Like the entire application, right?
Prasanna Malaiyandi:So it's possible that they weren't overriding the last good backup.
Prasanna Malaiyandi:It's just that the last good backup was from over a month ago.
Prasanna Malaiyandi:Or it could be that like their more
Prasanna Malaiyandi:recent backups were corrupt.
Prasanna Malaiyandi:W. Curtis Preston: Uh, well, we don't have any information to suggest that,
Prasanna Malaiyandi:don't have any information.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:But
Prasanna Malaiyandi:W. Curtis Preston: I I also don't understand why if it, if all backups
Prasanna Malaiyandi:were equal, why didn't they immediately try restoring the one from April?
Prasanna Malaiyandi:They, they, they said, we know for a fact we can get the one from March.
Prasanna Malaiyandi:We think we're gonna get to one.
Prasanna Malaiyandi:And they go and, and, and they made a comment about, you know, we realize
Prasanna Malaiyandi:that this is, it reminds me a little bit of the Rackspace event where.
Prasanna Malaiyandi:What we can do, the quickest is we can get the March one up and running.
Prasanna Malaiyandi:Why wasn't the April one also just as good as the March one?
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:So I'm not, I'm not sure what happened there, but there was something that
Prasanna Malaiyandi:was not as good about the April backup.
Prasanna Malaiyandi:Uh, but, but it did appear that they eventually were able to get
Prasanna Malaiyandi:that up, but they lost two to three weeks worth of, of effort.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:And even for the April backup, they didn't actually restore it.
Prasanna Malaiyandi:So using the way back machine right, we were able to go back and
Prasanna Malaiyandi:find the original blog post for
Prasanna Malaiyandi:W. Curtis Preston: Right.
Prasanna Malaiyandi:Why did we have do that Prasanna?
Prasanna Malaiyandi:Why did we have to use the, the, the internet archive?
Prasanna Malaiyandi:because the company took down that post and created a
Prasanna Malaiyandi:new post at that same, for that same date, that shed different light on
Prasanna Malaiyandi:what had happened and their steps.
Prasanna Malaiyandi:W. Curtis Preston: Interesting, right?
Prasanna Malaiyandi:Um,
Prasanna Malaiyandi:so, so from that or may, uh, blog post, they basically
Prasanna Malaiyandi:said, or soon thereafter, right, I think it was a couple weeks later, they
Prasanna Malaiyandi:were like, Hey, we're able to get you back up and running for your March and
Prasanna Malaiyandi:for April, we were able to restore the data, but they didn't wanna merge the
Prasanna Malaiyandi:data back into the original database.
Prasanna Malaiyandi:So they actually created a staging site where people had to go to manually
Prasanna Malaiyandi:in order to be able to pull down their data if they want to access it.
Prasanna Malaiyandi:W. Curtis Preston: this is so much like the Rackspace event, right?
Prasanna Malaiyandi:And so what that, what that tells me, you know, and again, we're, we're, we're not
Prasanna Malaiyandi:taking joy in this, but what that tells me is that they didn't have a regular
Prasanna Malaiyandi:tested procedure because they didn't, they didn't know that the April one was good.
Prasanna Malaiyandi:If they knew the April one was good, they would've just restored the April
Prasanna Malaiyandi:one, because by restoring the March one.
Prasanna Malaiyandi:They may, you know, you couldn't just do an incremental restore
Prasanna Malaiyandi:on top of the March database.
Prasanna Malaiyandi:So, um, they created this extra work for
Prasanna Malaiyandi:themselves and their
Prasanna Malaiyandi:and especially their entire thing was we want to get people
Prasanna Malaiyandi:up and running as quickly as possible.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:That's why they did the March one.
Prasanna Malaiyandi:But like you said, Curtis, if April was good, they should have just used April and
Prasanna Malaiyandi:it would've been the same amount of time.
Prasanna Malaiyandi:W. Curtis Preston: There had to be something up with the April one.
Prasanna Malaiyandi:You know, you could, you'd sort of have to read between the lines.
Prasanna Malaiyandi:Um, there was something up with the April one and then,
Prasanna Malaiyandi:we've talked about this before.
Prasanna Malaiyandi:We urge people not to do homegrown backup and when I said this last time,
Prasanna Malaiyandi:you said sometimes you have no choice.
Prasanna Malaiyandi:Yeah,
Prasanna Malaiyandi:W. Curtis Preston: I would like to, I would like to.
Prasanna Malaiyandi:say it again.
Prasanna Malaiyandi:W. Curtis Preston: Well, I'm, say it again, but I would like to point out the
Prasanna Malaiyandi:fact that they were running on Azure, and Azure has backup features built into it.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:They could have taken a snapshot of their entire environment every day, every hour.
Prasanna Malaiyandi:They could have done that.
Prasanna Malaiyandi:They would've had essentially, you know, you know, a copy of
Prasanna Malaiyandi:their database from two hours ago.
Prasanna Malaiyandi:You know, they could have, and that wouldn't have been homegrown.
Prasanna Malaiyandi:I'm just saying they could have done that.
Prasanna Malaiyandi:Well.
Prasanna Malaiyandi:The other thing they could have done is why don't they
Prasanna Malaiyandi:have a disaster recovery copy?
Prasanna Malaiyandi:Like if this is a SaaS service, wouldn't you have expected a Dr.
Prasanna Malaiyandi:Copy to be available somewhere where even if the production instance went down for
Prasanna Malaiyandi:some reason, they should have been able to come back up even if it wasn't a backup.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, so this, this is where, you know, we often talk about
Prasanna Malaiyandi:that, that if your data is sitting on a SaaS service, I think it's your
Prasanna Malaiyandi:responsibility to back it up, or at least to make sure that it's backed up.
Prasanna Malaiyandi:You can, you can make an argument as to whether or not you should use
Prasanna Malaiyandi:their backup service if they have one.
Prasanna Malaiyandi:I think we've got three stories now that show that sometimes their backup
Prasanna Malaiyandi:service is as good as their it, um, and.
Prasanna Malaiyandi:The, so you, I think you should be backing up your own data, but
Prasanna Malaiyandi:generally speaking, some of these SaaS services, they will have a dr.
Prasanna Malaiyandi:Copy that isn't any good for you if you do something stupid, but, but if
Prasanna Malaiyandi:they do something stupid or there's a fire, or there's a, an attack,
Prasanna Malaiyandi:they've got a copy that they can restore the entire environment that
Prasanna Malaiyandi:that appears to be what they had here, but it was just woefully out of date.
Prasanna Malaiyandi:Um, and so the vast majority of their customers did get their data back, but
Prasanna Malaiyandi:there were a few, and I wanna, I want, I want to just point out a few that
Prasanna Malaiyandi:they were, uh, quoted in the story.
Prasanna Malaiyandi:There was, uh, Margaret Fry, a postdoctoral researcher at Harvard
Prasanna Malaiyandi:University, lost about 60 annotated texts and over a hundred hours worth of coding
Prasanna Malaiyandi:work related to her research on AIDS and sexual attraction Southern Africa.
Prasanna Malaiyandi:Jason Richardson, an associate professor at the University of Kentucky and his
Prasanna Malaiyandi:colleagues lost around a hundred hours of work on a project, analyzing job
Prasanna Malaiyandi:advertisements for school principles to assess how well Kentucky prepares
Prasanna Malaiyandi:aspiring school administrators.
Prasanna Malaiyandi:Uh, there's one comment on Dedoose Facebook page that the company had managed
Prasanna Malaiyandi:to destroy their wife's $10,000 research project, so there were real people.
Prasanna Malaiyandi:And these weren't just companies, these were just, these were just regular
Prasanna Malaiyandi:people trying to get their doctorates done and uh, they lost, you know,
Prasanna Malaiyandi:yeah, but, but to their credit though, dedoose actually
Prasanna Malaiyandi:had no, I don't know if it was before this issue or after the event, but
Prasanna Malaiyandi:they did say they have functionality.
Prasanna Malaiyandi:Actually, no, it was even before because for staging, this is
Prasanna Malaiyandi:how you get your data out.
Prasanna Malaiyandi:There is an export functionality.
Prasanna Malaiyandi:W. Curtis Preston: Right.
Prasanna Malaiyandi:That a user could use to pull all their data
Prasanna Malaiyandi:out of Dedoose for doing, like what you talked about, a local backup.
Prasanna Malaiyandi:So it is possible that these users could have been using that to protect
Prasanna Malaiyandi:themselves from these situations, and Dedoose is still around, and so I highly
Prasanna Malaiyandi:recommend people who are using Dedoose.
Prasanna Malaiyandi:Maybe you should periodically export your own data out.
Prasanna Malaiyandi:W. Curtis Preston: Can I, can I edit you and take the word maybe out there?
Prasanna Malaiyandi:Yes.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, so you should periodically back up your work no
Prasanna Malaiyandi:matter where it's residing, you know, a, a periodic, you know, inefficient
Prasanna Malaiyandi:backup that has a poor recovery point.
Prasanna Malaiyandi:Actual is still better than, than nothing.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, yeah.
Prasanna Malaiyandi:I remember, I, I have a, a niece that came to me with her, she had a laptop
Prasanna Malaiyandi:and she had her masters on this laptop.
Prasanna Malaiyandi:And the laptop, the drive was making those noises.
Prasanna Malaiyandi:You know
Prasanna Malaiyandi:Oh, the click, click, click, click,
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:The hard drive should not make.
Prasanna Malaiyandi:We were able to get off her master's thesis, uh, off of that
Prasanna Malaiyandi:drive before it just went up in smoke and it was killing me.
Prasanna Malaiyandi:'cause I was like, you don't have a backup of her.
Prasanna Malaiyandi:That's really important work.
Prasanna Malaiyandi:Uh, you know, I feel for these people At the same time, you know, uh, my,
Prasanna Malaiyandi:my statement is gonna remain the same.
Prasanna Malaiyandi:That if your data is sitting in, in a SaaS service somewhere, it's
Prasanna Malaiyandi:your responsibility to, to make sure that it's being properly backed up.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:W. Curtis Preston: and and I still say even if the SaaS vendor offers
Prasanna Malaiyandi:backup, which this one apparently did, um, I still think that you
Prasanna Malaiyandi:should do it some other way.
Prasanna Malaiyandi:Um, yeah.
Prasanna Malaiyandi:Well, do we know if they offered backup or
Prasanna Malaiyandi:was this just sort of their own
Prasanna Malaiyandi:W. Curtis Preston: right.
Prasanna Malaiyandi:It this was their, this was their internal dr.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, what I don't know is whether or not they advertise this as part of the, um.
Prasanna Malaiyandi:You know, now the CEO of Dedoose said this is a horrible moment for our company.
Prasanna Malaiyandi:We have never lost data or had a breach.
Prasanna Malaiyandi:It's impossible to say how many customers have lost data because
Prasanna Malaiyandi:of the firm doesn't monitor how customers are using their accounts.
Prasanna Malaiyandi:Only users can access, can assess losses in their projects, not Dedoose.
Prasanna Malaiyandi:Yeah, it was a, it was a
Prasanna Malaiyandi:Yeah, if I
Prasanna Malaiyandi:W. Curtis Preston: a.
Prasanna Malaiyandi:Oh, oh yeah.
Prasanna Malaiyandi:Like, and especially researchers, like coming back and getting
Prasanna Malaiyandi:that data is so difficult.
Prasanna Malaiyandi:And then just being like, yep, it's gone.
Prasanna Malaiyandi:Just like that puff of
Prasanna Malaiyandi:W. Curtis Preston: And, and some of that data is irreplaceable.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:W. Curtis Preston: It's interviews of subjects.
Prasanna Malaiyandi:Um, you know, I don't even want to try to, when you're in the middle of
Prasanna Malaiyandi:something that complicated, there is some data that is simply irreplaceable.
Prasanna Malaiyandi:Um, and, um, so it, you know, it's, it's a real shame what happened.
Prasanna Malaiyandi:Did you wanna talk about what they said they're going
Prasanna Malaiyandi:to do and what they've actually done?
Prasanna Malaiyandi:W. Curtis Preston: yeah.
Prasanna Malaiyandi:That's a great, yeah, that's a great segue.
Prasanna Malaiyandi:So they said they're now going to be doing seven things and when, when I
Prasanna Malaiyandi:say now we, we mean, uh, 10 years ago.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, after the event they said they were going to, it sounds like they
Prasanna Malaiyandi:went from not much to way overboard.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Of course, I would never say that any backup system is overboard.
Prasanna Malaiyandi:But here's what they said they were gonna do.
Prasanna Malaiyandi:They were gonna develop a database mirror, slave, and Azure.
Prasanna Malaiyandi:A database mirror slave in Amazon S3, keeping a mirror copy of the entire
Prasanna Malaiyandi:blob storage, you know, including all the stuff in an encrypted volume
Prasanna Malaiyandi:storing nightly database backups on the V-H-D-V-H-D, virtual hard drive, I guess,
Prasanna Malaiyandi:and Azure blob storage and Amazon three.
Prasanna Malaiyandi:So they're gonna keep three nightly backups in three separate places mirroring
Prasanna Malaiyandi:all Azure file data into an S3 bucket.
Prasanna Malaiyandi:Carrying out weekly restore exercise, uh, for the database backups and
Prasanna Malaiyandi:a monthly bare bones restore.
Prasanna Malaiyandi:That all sounds great.
Prasanna Malaiyandi:Now I do, I do wanna comment, um, that anything that refers to mirroring to me.
Prasanna Malaiyandi:Isn't a backup, right?
Prasanna Malaiyandi:Um, that is a dr.
Prasanna Malaiyandi:Copy that will only be useful if what happens is like a fire, right?
Prasanna Malaiyandi:If there's any sort of attack, any of those mirrors are totally worthless
Prasanna Malaiyandi:because, or if, if you do something stupid like drop a table, all those
Prasanna Malaiyandi:mirrors are gonna immediately, uh, you know, get corrupted as
Prasanna Malaiyandi:but it would've helped in their current,
Prasanna Malaiyandi:in the event that it happened.
Prasanna Malaiyandi:W. Curtis Preston: Uh, I'm gonna say it's unclear as to whether
Prasanna Malaiyandi:or not it would help in it,
Prasanna Malaiyandi:Okay.
Prasanna Malaiyandi:Sorry.
Prasanna Malaiyandi:It could help.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:W. Curtis Preston: It could have helped.
Prasanna Malaiyandi:Yes, yes.
Prasanna Malaiyandi:Um, mirroring is not bad just because they're putting it here and
Prasanna Malaiyandi:they're listing all these things.
Prasanna Malaiyandi:Um, you know, it's a shame that they, that they had to wait.
Prasanna Malaiyandi:Uh, after the thing.
Prasanna Malaiyandi:What I also would've liked to seen in that list is I would've liked
Prasanna Malaiyandi:to have seen that one or more of these copies, uh, was immutable.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:Um, and, uh, and maybe they are, I don't know, you know?
Prasanna Malaiyandi:It's a great list, but it's also one of those
Prasanna Malaiyandi:things like once a horse has left the barn, like what do you do?
Prasanna Malaiyandi:You know, it's sort of like, Hey, uh, we're just gonna throw
Prasanna Malaiyandi:everything at this and just say, this is everything we're going to do.
Prasanna Malaiyandi:W. Curtis Preston: yeah, it
Prasanna Malaiyandi:than what
Prasanna Malaiyandi:W. Curtis Preston: yeah.
Prasanna Malaiyandi:what is actually makes sense from a business perspective,
Prasanna Malaiyandi:because sure, you could do all these things, but just imagine the cost
Prasanna Malaiyandi:you would be spending for backup.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:Your backup copies are probably more than your production
Prasanna Malaiyandi:copies at that point, right?
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:It depends on, depends on the way they're doing them.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:To take, to take your, to take your analogy, you know, the horse is laying
Prasanna Malaiyandi:out dead in the field, and you're like, you know what we're gonna do?
Prasanna Malaiyandi:We're gonna lock the, we're gonna lock the barn door now, and then we're
Prasanna Malaiyandi:gonna put another padlock on top of that lock, and then we're gonna put
Prasanna Malaiyandi:a fence around the area in front of the barn, just in case the lock fails.
Prasanna Malaiyandi:And then, uh, you know, and then we're gonna have a guy
Prasanna Malaiyandi:standing, standing there with a.
Prasanna Malaiyandi:With a tran gun, uh, to shoot the horse.
Prasanna Malaiyandi:If the horse comes outta the bar, it's like, yeah, that's all great, but your
Prasanna Malaiyandi:horse is dead, so you need another horse.
Prasanna Malaiyandi:Yeah,
Prasanna Malaiyandi:W. Curtis Preston: we're some, I'm, I'm sorry that was a little
Prasanna Malaiyandi:morbid, but I apologize to the horse lovers in the, in the group.
Prasanna Malaiyandi:but the other thing we found though, so that's what they
Prasanna Malaiyandi:had promised 10 years ago, but they do have in their, what is this called in
Prasanna Malaiyandi:their security article on Dedoose, sort of what they've actually done, right?
Prasanna Malaiyandi:So.
Prasanna Malaiyandi:What they have done is they are using redundant storage volumes on Azure.
Prasanna Malaiyandi:W. Curtis Preston: mm-Hmm.
Prasanna Malaiyandi:All project data backed up in full on, on a nightly basis.
Prasanna Malaiyandi:A volume is onsite with two being offsite and replicated across geographic regions.
Prasanna Malaiyandi:Um, and then the storage volume is encrypted and mirrored in
Prasanna Malaiyandi:real time to Amazon S3 storage.
Prasanna Malaiyandi:They've automated everything such that on a weekly basis, it'll download the most
Prasanna Malaiyandi:recent backup from each storage volume.
Prasanna Malaiyandi:Verify they're using the correct version of the backup file, do a full
Prasanna Malaiyandi:test restoration of the database, and email reporting to all backup
Prasanna Malaiyandi:and restore process results to key members of the Dedoose team.
Prasanna Malaiyandi:W. Curtis Preston: Yeah, it's, it's still, to me, it still reads
Prasanna Malaiyandi:like a homegrown system, but, um.
Prasanna Malaiyandi:You know, because like one of the things like, you know, I, I did, I did a lot of
Prasanna Malaiyandi:homegrown systems over the years, right?
Prasanna Malaiyandi:Like one of the things like with email reporting is.
Prasanna Malaiyandi:D does your system report if it doesn't run right, is is there,
Prasanna Malaiyandi:is there a thing that, that's, you know, like what happened?
Prasanna Malaiyandi:You know, 'cause sometimes you don't notice that you're not getting an email.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:Hey, has anybody noticed that?
Prasanna Malaiyandi:We don't get the, we stopped getting the backups, you know,
Prasanna Malaiyandi:uh, like three weeks ago.
Prasanna Malaiyandi:Um, usually when you, when you discover that is right after the, um, the
Prasanna Malaiyandi:bad thing happens, whatever, it's
Prasanna Malaiyandi:but even though it might be sort of homegrown from a
Prasanna Malaiyandi:automation process perspective, like we don't have enough information.
Prasanna Malaiyandi:I think the fact that there are at least.
Prasanna Malaiyandi:Doing nightly backups, right?
Prasanna Malaiyandi:Verifying it on a weekly basis like these are, and also making sure that
Prasanna Malaiyandi:the data is stored offsite, right?
Prasanna Malaiyandi:I think maybe it's going a bit for like, they may not have to do like multi-cloud.
Prasanna Malaiyandi:Maybe that's going a stretch too far, but I could see that they really are
Prasanna Malaiyandi:worried about data integrity, so,
Prasanna Malaiyandi:W. Curtis Preston: There's no such thing as a stretch too
Prasanna Malaiyandi:far for backup and recovery.
Prasanna Malaiyandi:My friend.
Prasanna Malaiyandi:well,
Prasanna Malaiyandi:W. Curtis Preston: They're doing the thing that I wished everybody did, but
Prasanna Malaiyandi:nobody does because it's too expensive,
Prasanna Malaiyandi:Yes.
Prasanna Malaiyandi:W. Curtis Preston: right?
Prasanna Malaiyandi:Um, so I, I can't fault 'em for doing that,
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So it looks like they are doing the right things,
Prasanna Malaiyandi:W. Curtis Preston: mm-hmm.
Prasanna Malaiyandi:um, from what we can gather.
Prasanna Malaiyandi:So it looks like they have learned and they're still around, right?
Prasanna Malaiyandi:People still continue to use 'em.
Prasanna Malaiyandi:They do have an active subreddit.
Prasanna Malaiyandi:They are continuing to release new features.
Prasanna Malaiyandi:W. Curtis Preston: Mm-Hmm.
Prasanna Malaiyandi:Yeah, and I think, I think what was in their favor is that they
Prasanna Malaiyandi:were the only SaaS game in town.
Prasanna Malaiyandi:And for a lot of people that, that, that's probably a significant.
Prasanna Malaiyandi:Reason to use their service.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Um, I would also hope that this is a significant reason for anyone to do a,
Prasanna Malaiyandi:a regular export of their data, right.
Prasanna Malaiyandi:When I'm using a service like that, where I tend to do is I do, I do an export.
Prasanna Malaiyandi:Like if I'm doing a manual export, I'm doing it after significant events
Prasanna Malaiyandi:like finishing my taxes, right?
Prasanna Malaiyandi:I did, you know, I did an export.
Prasanna Malaiyandi:I, I did a PD, I didn't do exports of.
Prasanna Malaiyandi:My intuit data as I was doing it right.
Prasanna Malaiyandi:But once I finished my return, I printed a PDF of that.
Prasanna Malaiyandi:Um, I have a paper copy.
Prasanna Malaiyandi:I have a PDF copy that's on my laptop.
Prasanna Malaiyandi:It's backed up using my backup software.
Prasanna Malaiyandi:And uh, I put it in, um, Google Docs, right?
Prasanna Malaiyandi:So it's all over the place, right?
Prasanna Malaiyandi:Um, but I don't do that every day.
Prasanna Malaiyandi:It's just you do it when you, when you get to that moment where you're
Prasanna Malaiyandi:like, ah, oh, this is really good.
Prasanna Malaiyandi:I hope I don't lose
Prasanna Malaiyandi:Prasanna Malaiyandi: a good stopping point.
Prasanna Malaiyandi:W. Curtis Preston: It's a good stopping point.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Um, it's like also I use voice recognition software, right.
Prasanna Malaiyandi:And sometimes it crashes and you lose whatever you said, uh, in that thing.
Prasanna Malaiyandi:And so I just got into the habit of anytime I finish, like
Prasanna Malaiyandi:a page, I just hit, you know?
Prasanna Malaiyandi:I just say, um, yeah.
Prasanna Malaiyandi:Uh, 'cause I, I can actually say.
Prasanna Malaiyandi:Um, I almost said, Hey, Siri, you know, click file save.
Prasanna Malaiyandi:That's what I say.
Prasanna Malaiyandi:Click file, save.
Prasanna Malaiyandi:And, um, and I can do that or you can just type control s right?
Prasanna Malaiyandi:Uh, just do you know, this is, I guess this is another lesson to learn
Prasanna Malaiyandi:is, is that should be a part of your mental process when you do stuff and
Prasanna Malaiyandi:it's storing that data in some place.
Prasanna Malaiyandi:At the end, at that stopping moment, do you, do you think to yourself, Hey,
Prasanna Malaiyandi:is this data backed up in a way that protects me from all stupid things?
Prasanna Malaiyandi:You know
Prasanna Malaiyandi:W. Curtis Preston: are you
Prasanna Malaiyandi:funny when you're, when you're talking
Prasanna Malaiyandi:about the control save, right?
Prasanna Malaiyandi:Or Control S to save it.
Prasanna Malaiyandi:So I'm so used to working on Excel spreadsheets that I would always do
Prasanna Malaiyandi:like Control S, control S, to save
Prasanna Malaiyandi:W. Curtis Preston: Right, right,
Prasanna Malaiyandi:Even when I use Google shoots, I still do it anyway,
Prasanna Malaiyandi:W. Curtis Preston: right.
Prasanna Malaiyandi:just out of the force of habit.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:That's funny.
Prasanna Malaiyandi:, Prasanna Malaiyandi: So, lessons learned.
Prasanna Malaiyandi:Lessons
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:So we, you know, so obviously we talk about, you know, it's still your
Prasanna Malaiyandi:responsibility to backup up SaaS.
Prasanna Malaiyandi:The other thing is a month RPO?
Prasanna Malaiyandi:A month backup frequency.
Prasanna Malaiyandi:That's, that's unacceptable.
Prasanna Malaiyandi:That's just, I can't imagine that being acceptable, uh, to
Prasanna Malaiyandi:in, in any world, by the way.
Prasanna Malaiyandi:That is determined by your business, right?
Prasanna Malaiyandi:I, I have met at least one company where a month long RPO was fine.
Prasanna Malaiyandi:They were a paper mill, right?
Prasanna Malaiyandi:They were, they were like, we don't, the, the, the, the computers, they just, they
Prasanna Malaiyandi:don't have anything that we care about.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:They were fine with a month.
Prasanna Malaiyandi:That should be determined by your business.
Prasanna Malaiyandi:But a month seems really long to me.
Prasanna Malaiyandi:Um, and the other thing is, if your backup system.
Prasanna Malaiyandi:Is this is, we don't know if this is actually the case,
Prasanna Malaiyandi:but I'm inferring from it.
Prasanna Malaiyandi:If your backup system overwrites the last good backup as part of
Prasanna Malaiyandi:the process because you're so starved for storage, that is bad.
Prasanna Malaiyandi:That is bad.
Prasanna Malaiyandi:Bad, bad, bad, bad.
Prasanna Malaiyandi:That is basic
Prasanna Malaiyandi:storage is cheap.
Prasanna Malaiyandi:W. Curtis Preston: Bad.
Prasanna Malaiyandi:Well, it's cheap, but I'm just saying it's cheap.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:So much.
Prasanna Malaiyandi:So much we do or don't do is because it's, it's inexpensive.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:Well, given that they are keeping all these
Prasanna Malaiyandi:copies and dealing with cross site replication and everything else,
Prasanna Malaiyandi:W. Curtis Preston: Mm-Hmm.
Prasanna Malaiyandi:Well, they, yeah, they're not doing that now.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:I'm just saying there are people, there are people that their backup system
Prasanna Malaiyandi:is overriding the last good backup, and I just, that, that is a, that is
Prasanna Malaiyandi:a really, basically, it means that if you're in the middle of the backup and
Prasanna Malaiyandi:something happens, you have no backup.
Prasanna Malaiyandi:Yep.
Prasanna Malaiyandi:W. Curtis Preston: Unless you're backing up the backup.
Prasanna Malaiyandi:If you're backing up the backup right, then, then, then, right.
Prasanna Malaiyandi:But if this is your only backup and you're overriding it with the next
Prasanna Malaiyandi:good backup because you're too cheap to buy enough storage for multiple
Prasanna Malaiyandi:backups, um, I dunno what to tell you.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Please do not please step away from the keyboard and have someone
Prasanna Malaiyandi:else design your backup system.
Prasanna Malaiyandi:W. Curtis Preston: step.
Prasanna Malaiyandi:That
Prasanna Malaiyandi:is true though.
Prasanna Malaiyandi:W. Curtis Preston: perfect.
Prasanna Malaiyandi:It is true.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:Yeah.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Uh, can you think of any other, um, lessons from this one?
Prasanna Malaiyandi:I know that we talked about how Dedoose may not have
Prasanna Malaiyandi:had the best backup systems in place,
Prasanna Malaiyandi:but this is also one of those things like it was.
Prasanna Malaiyandi:Unfortunate that they had these multiple issues that led to such a big failure.
Prasanna Malaiyandi:But it is good that they did see, okay, here are gaps in our system.
Prasanna Malaiyandi:Here's how we're gonna fix it.
Prasanna Malaiyandi:They were transparent with the users and it looks like they've gotten
Prasanna Malaiyandi:their act together, which is good.
Prasanna Malaiyandi:Right?
Prasanna Malaiyandi:So I think it's just a matter of making sure they're continuing to stay up to
Prasanna Malaiyandi:date and reevaluating their architecture and infrastructure of their backup.
Prasanna Malaiyandi:Especially like, I don't know how they deal with ransomware.
Prasanna Malaiyandi:W. Curtis Preston: Right, right,
Prasanna Malaiyandi:Like are they really using separate accounts,
Prasanna Malaiyandi:like when they're replicating to different buckets, different regions,
Prasanna Malaiyandi:or is it all like the same username, password, or whatever across the two?
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:W. Curtis Preston: You're killing me
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:Because hopefully they're keeping their Azure or their AWS system isolated.
Prasanna Malaiyandi:W. Curtis Preston: Right, right.
Prasanna Malaiyandi:And hopefully again,
Prasanna Malaiyandi:than backups.
Prasanna Malaiyandi:Yep.
Prasanna Malaiyandi:W. Curtis Preston: somewhere, along the way they're using immutable storage.
Prasanna Malaiyandi:Right.
Prasanna Malaiyandi:And we, and we do like the way that they, that they had the blog.
Prasanna Malaiyandi:They were, they were immediately, um, you know, transparent with what was going on.
Prasanna Malaiyandi:They were, they were a little bit of, uh, you know, mea culpa, right?
Prasanna Malaiyandi:They're like, Hey, this is our fault.
Prasanna Malaiyandi:You know, this is all us, you know, uh, we're so, so sorry.
Prasanna Malaiyandi:Uh, I'm a little disappointed that they deleted the blog after the fact.
Prasanna Malaiyandi:Um,
Prasanna Malaiyandi:Put a link to the.
Prasanna Malaiyandi:W. Curtis Preston: referenced in mo.
Prasanna Malaiyandi:What's that?
Prasanna Malaiyandi:Yeah, we will put a log to a, a link to the archive version of the blog.
Prasanna Malaiyandi:'cause the internet never forgets.
Prasanna Malaiyandi:Um, yeah.
Prasanna Malaiyandi:So.
Prasanna Malaiyandi:All right, well thanks for helping me work our way through
Prasanna Malaiyandi:another cloud disaster Prasanna.
Prasanna Malaiyandi:No, thank you Curtis.
Prasanna Malaiyandi:This was fun.
Prasanna Malaiyandi:W. Curtis Preston: Yeah.
Prasanna Malaiyandi:Fun, fun, fun and a sad way that unfortunately involved
Prasanna Malaiyandi:a, you know, a deceased
Prasanna Malaiyandi:horse.
Prasanna Malaiyandi:but.
Prasanna Malaiyandi:W. Curtis Preston: Um,
Prasanna Malaiyandi:Uh, poor horsey.
Prasanna Malaiyandi:W. Curtis Preston: yeah.
Prasanna Malaiyandi:Um, well subscribe folks so that you don't miss all of this amazingness.
Prasanna Malaiyandi:Uh, and, uh, that is a wrap.