In this livestream, Frank and Andy discuss the timeless nature of backend enterprise tech, that, much like a Christmas special from decades ago, is still very much celebrated.

Moments

00:00 Exploring SSIS future in a festive episode.

08:28 Data engineering evolved from business intelligence systems.

10:57 Social networks project before Facebook's popularity.

19:19 SSIS training informed data engineering concepts teaching.

24:56 Bill Gates moved project to immature Microsoft tooling.

29:10 Data engineering possible in 2024 using T-SQL.

35:23 Huge cloud companies surpass previous brick-and-mortar giants.

40:10 Old technologies endure; misconceptions about their age.

46:03 Evaluate change benefits: technical ease, business growth.

52:30 Cloud departure interests rise, SSIS assistance sought.

55:47 Big government agency utilizing diverse cloud platforms.

01:00:59 Security is crucial; clients' preferences vary.

01:08:56 Certification issues hinder software updates and compliance.

01:10:02 People stick with older systems for reasons.

01:15:15 Proper GPU driver drastically improved loading time.

01:22:16 Repost increased engagement and communication with author.

01:25:45 Data scientists should learn SQL for simplicity.

01:31:06 Obsolete systems cause issues without quotes.

In this special holiday themed episode, we're diving into a topic

that's as classic as Christmas Carols, but just as divisive as fruitcake.

And that topic is the future of SQL Server Integration

Services, SSIS. But wait, there's a

twist. This episode was recorded live, so if you notice

a different vibe, some festive banter, and maybe even a change in

our usual musical interludes, that's why. Think of it as

the holiday party version of our usual data driven discussions.

Together, we'll explore why SSIS, despite its vintage

status, remains a cornerstone of data engineering and why

dismissing it might just be a data driven mistake. So grab your

cocoa, settle in by the fire or your nearest CPU,

and let's get festive with some data talk.

Well, hello, and welcome to franksworld.comstream.

And, with me today is Andy, and I'm

looking for the lower third that has us both. There we

go. Frank and Andy Frank Lavinia and Andy Leonard, host of Data

Driven, which I might turn this into a podcast. I might take the

audio and and turn it into a podcast. What do you think about that? That'd

be good kind of festive stream and also kind

of up to date on things. And it gives me some more time that to

put together another episode that I had a really great

conversation with a guy who does red teaming for LLMs.

Nice. So which I think is a growth industry

and certainly a wise career move.

Speaking of career moves. Good thing.

Oh, we have a first comment. SQL dev d b a.

Hey, SQL dev. Awesome. So

this is this may be the first time we've done this. This feature's been around

for a while. No. We did it once or twice before. Did we do it

on recent? Like, months. Yeah. That we've done. So we're sharing

our so Frank's audience, people that are connected to Frank,

they're seeing this. People connected to me are seeing this. It's like it'll

because it told me Frank started this, and, then he sent me

the link. And as I joined in, it it said, hey. You can

share this with with your on your channels as well. So I

was like, oh, yeah. Click that. Oh, you know what it is? We did it

the other way. You were the main, and then I shared it on my channels.

That's what happened. That's what happened. Yeah. Yeah.

Well, it's cool, though. If you've never met me before hello?

That's Frank. Frank digs data on the socials and,

franksworld.com, datadriven.tv, which hopefully you know about that,

and impactquantum.com. So that's me.

And, yeah. So back to the segue.

Yeah. I was talking about how security and AI is a

good career move. And we were talking about, speaking of

career moves, 'tis the season for SSIS is the title of the

stream. And this kind of goes,

I'm sorry. Come on, man. It's fun. Right? It is.

It's awesome. So so and I had kind of done,

2 livestreams on this already, but one of them for, like, 10 minutes, I didn't

catch the fact that I had no audio. And then yesterday, I did one for

2 minutes, so I didn't catch the fact that I didn't do the audio. So

I figured I'd bring the troublemaker himself onto here. Although, strictly

speaking, you're not the original troublemaker on this. Well,

I participated in it. I'll I'll own my my part of the

trouble. You'll own your part of the trouble. So so I definitely will. Yeah.

What's the background here? Well and and I'll

I'll do a plug for, for andylehner.blog.

And if you go there, you can sign up for my newsletter over on the

right side. It's kinda hard to read because the widget is a little

narrower than it needs to be. But if you if you do that or if

you just look up engineer of data, I think it's

engineer of data dot substack.com.

But I I put a newsletter out today kinda talking about it. Yeah.

There's the site. Thanks, Frank. No problem. And, you see the subscribe

to my newsletter down there on the right, and there's a box on

the left where you type your email address and then on the right, you click

it's free. And it'll it should take you, right over to

Subsec, which by the way, I started using this year. And so far,

I'm pretty impressed. It's it's been a a

really interesting, experience for me. So the

trouble here here's, here's where the

trouble happened. I I have been, reading. I

caught a couple of articles just every now here and then, mostly

on LinkedIn, where people

would express an opinion about, you know,

SSIS stinks. I don't like it. It's old. It's was

so much trouble. And, you know, and they would just

kind of kind of poo poo share their their negative thoughts about

Azure sorry. About SSIS. And

I've, of course, I've worked in SSIS since

really before it came out, I got to work on that Rocks book project

with Brian Knight and I remember that book. Yeah. Yeah. 10

of us. Yes. Back when Rocks would put your picture on the cover of the

book. And have a copy around here somewhere. Yeah. That

yeah. Thank you, Frank. You know, it just but it's

so I got yes. I got kind of a boost out of my career,

and I did an awful lot in SSIS for a long time. And

every now and then, I still do. I used to

deliver training, as part of solid,

solid quality learning is what it was called when I joined it. Solid

queue. After that, I worked with them for a few years and I delivered

training developed by Eric Veerman and

also did consulting gigs. And I learned a lot,

about both data engineering and SSIS while I was

doing both those things. When I left solid q, I

think I put about a year or 2 between me and,

you know, and the business. Actually, it was about two and a half years because

I went to work for Unisys then as a ETL architect. I remember

that. You're up in Reston quite a bit because that's where it was. Oh, yeah.

Yeah. Frank. Now an apartment complex now, that building. Oh, is

it? Okay. I think so. Yeah. Okay. So Frank and I

have been friends since the before times, even before SSIS came

out. And, Well, no. I think you had just written the book at the

time. I I'm trying to remember. So Just moved to Richmond just when I

met. It was November of 2005. December 2005.

Yeah. Yeah. November 2005 is when we met. And,

another mutual friend that I won't name, but we're all still friends now.

And the book actually was published in

January, I think, of 2006.

Yes. That's right. So it wasn't it wasn't quite

ready for for prime time. But oh, sorry. The the

book wasn't out. It was going through the process, and it takes a couple of

months from the from the time all of the drafts are finished

until they they make a book out of it. It was my very first,

book project. And, yeah, I I'm pretty sure I was I was so

excited. I was telling everybody, I worked on a book. Oh, yeah. Yeah. Because it

was for the Richmond Code Camp, which was in May, April of

Yep. 2006. Yeah. 2000

Yeah. It was 2006. You and I. Where I did A team. A team.

Developers on a plane, and I had the guy I photoshopped the

guy carrying your book. That's right. I do remember that.

Yeah. I have to find that picture somewhere. I've been I've been using

SSIS for a a long time. I would say I

learned more about data engineering, the

field and did more projects probably

in, in data warehousing where I used SSIS

for for the data engineering, data integration. I think it's important to to

to, 1, explain for those who may not know what SSIS

is, and 2, explain that data engineering was not always seen as a

discrete,

profession or or Yeah. It's a data engineering's a

relatively new word to describe what we do. It was called

the part of business intelligence.

Back even before all that, I think the first term I

heard was data acquisition,

and it was in it was sometimes that was that phrase

was used standalone. The most often,

at the time when I and this is what got me into databases

was doing system control and data acquisition or SCADA

systems. These were manufacturing systems where you collected data from

instruments on the floor. You gotta remember, IoT

was, you know, still somebody's dream back, you know, in

the 19 nineties. IoT. It was just OT back then. It just was

OT. You're right. It's funny. Yeah. But

but we still did acquire, data from,

plant floors and instruments that were mounted all over, but they weren't

Internet enabled at that time. They were, most of them were hardwired. A

few were using wireless. And so that's kinda what led me

into this whole this whole field. And the idea of the

field, is of data engineering, data

integration as we called it back then, is we do that data acquisition part.

We go find wherever the data lives, we go find it there.

And sometimes the data is a very static

list. It it could be even a text

document, created in notepad that

is tab separated or, you know, delimited

by character position or something like that. And a lot of old

old lookups, lookup data was that way. And I'm not making

that up. It was maintained in a a text EDI.

EDI. Yeah. So electronic data interchange. Yeah.

So, yeah, EDI is I have an interesting stories about EDI, but but one of

the things that really kept me away from the data space for a long time

was I didn't wanna be DBA. And this work, I think, had traditionally

been kind of merged with DBAs. Oh, absolutely.

But at some point, I don't know exactly when it really

evolved into its own discipline. And I remember.

Go ahead. Because I remember I tried to get you a job at a particular

company. I remember that. And what do they do? And what was

it? Why do we need a DBA? You don't need a DBA.

Right. And I think that I'm not DBA. That

was the funny part. Well, that was the fun. Well, we clearly did because at

the time there was a project going on,

and I think the term data architect is what you just said. You were I'm

not a DBM data architect. And then that fell

on deaf ears. And, ironically, like,

not like a couple months later, there was a project that we worked on that,

so many stories, and I'm just trying to protect the innocent and the guilty,

and myself, from from from libels. But, basically,

there was a project going on that when it was basically kind of

behavioral analysis of social networks. Right? This is before

Facebook. I think Myspace was around that sort of thing. But it was basically the

idea of organizational networking as a discipline. And it turned out that the

the software that we bought off the shelf would actually query the

database, bring everything back in from the database,

and then run through the filtering on the C Sharp

components on the web server. Gotcha. So

long story short, there was 0 optimization, hardly an

index. I mean, it was just a mess. A data architect

will use the terms of the day, would have slot spotted this right away. We

didn't. And it was just a massive disaster. And it's kind of one of those

things where there were a number of projects that that company was taking

on. Basically, one of their one of their core

business models was was brilliant actually was software maintenance. So you have an

existing application offshore or outsource it outsource it to

us, and we'll take care of it for you. And,

you know, it was really like an an an education

in kind of Jenga programming. Right? Where you had they wanted updates

to this stuff, but they didn't wanna pay to redo it. So you kinda, like,

had to replace rip and replace stuff. And there's one particular

instance where there was a SQL query that took like 14

minutes to bring back an answer. And I'm like, it's only like

like 30,000 records. Like, what what's the deal here?

And turns out there was no indexes, no nothing.

Well, you know, those indexes take up space. Right? Exactly.

Exactly. I mean, this is like why you should save space.

Joke. That's a joke. For one reason or the other, like, there was there was

no index. And I was like, well, let's add indexes. And like, no, no, no.

We can't change the schema. Okay. So what I end up what I end

up doing was creating temporary tables with indexes

and then copying all the data, and I still got it down to 2 minutes.

Nice. Which 1 minute and 59 seconds was copying the data,

and then one second was actually changing. Yeah. So, like, it was it was kind

of like what I call Jenga programming or Jenga architecture. You had to like they

wanted updates, couldn't touch too much, couldn't change anything,

couldn't improve anything because it was just it was a

time in my career that I think back of and I've kind of learned

many lessons, both hard lessons and soft skill

lessons. But Sure. But we digress. But, I'm just

gonna I'm just gonna take that answering your question in my usual

long winded way. SQL Server Integration

Services, came along, and it was probably

the, again, it was the thing that

spanned the longest part of my career. Before that, I worked with something called

data mirror. That was the first, I'd I'd

say the the first system like that. First bit of software that

way. Before that, I was writing my own. So I

was reading from these plant networks and writing to all

sorts of stuff. And I got into SQL Server because

I crashed access back in the nineties. So I ran,

I collected a 1,000 points of data every second for a long

weekend. And I wanna say the access file grew to about 4 gigs.

When I went to open it and start doing some analysis on it, it turned

out it wouldn't open. So 4 gigs is nothing now. Right? You

can do that on a smartwatch. But back then, a server

struggled, to open the file system. If you go back far

enough, would have freaked out or anything over certain size unless it was, like, NTFS

or something like that. Right? Yeah. And this this

wasn't. This was, one of the other OSs. But

so, you know, I went I went to, altavista.digital.com

and typed in Microsoft database, and I saw this

listing for something called SQL Server, and that's how it all started.

Well, then I I got got in as, working

on a data warehouse, and part of my job moved

into the database part of it. I actually was hired to do the reporting piece

of it, and lots of cool lessons learned there as

well. But on the database side, they use Data Mirror. I

think that company is still around. I'm not sure. But this is like 25 years

ago. And it was it was so cool,

and I was fascinated that somebody had built software to

orchestrate this collection of data. I was like, wow.

That is a good idea. You know, it always makes me feel better, Frank, when

smart people come up with an idea that I've also come up with independently.

It makes me feel like, okay. Maybe I'm onto something. Go through all of

that, data transformation services or DTS, and then finally, SSIS

and this big block. And

what I've noticed and I kinda noticed this trend started

maybe 4 or 5 years ago. I people complained about SSIS before

that. Don't get me wrong. And a lot of it is

because are you sitting down? It's

hard. We're not making it up.

Comparatively, though, like, I I remember when I was at

barnesandnoble.com and which just goes

back a ways. So if you bought a magazine at Barnes and

Noble between 1996 and probably about

2012, 13, you

you interacted with the system I wrote, nice in the late

nineties or at least part of it anyway. So, you

know, that's how I learned EDI, right? Because we get these feeds

from publishers, literally a mainframe would dial up another

mainframe, download the file over a modem.

And, and this is how it worked. And what we did was we pulled down

the raw EDI files and I parsed it

and I had to do that and drop it into an informix database. So it

was a cool writing for GL scripts to to to take

that data in text format and then dump it

into an actual. You were doing data engineering. I was

doing data engineering, which is kind of funny. But like, you know, data engineering as

a discipline is not easy. Right? So SSIS being hard.

I mean, you know, brain surgery brain surgery is hard too. Right?

You you make a good point about it. And it, you know, it took me

a while, especially teaching it. And I would do

4 or 5 day course, originally with solid q and then

eventually on my own. I I wrote my own course. I

found myself adding to Eric's content when I would deliver the

material here. And don't get me wrong. Eric

is still a genius. He was then and he still

is. I just I I had a way of approaching

some, demos and examples that I felt kinda added

to the clarity of the information we were sharing. I

kind of expanded that out and wrote all my own material, my own I

use my own data, that I collect as part of my, weather station

here. And to this day, there are students that

are going through, recordings of that class.

The last recordings I made were back in December 2020,

and I recorded 3 courses on SSIS. The 4

day from 0 to SSIS course was, you know, will take you

from if you can spell SSIS to being a

functional, advanced beginner, low

end intermediate developer. And it was built for

that. It's got labs 13 12, 13 labs

that you do in 2 days, of that course. And then it talks

it kinda changes gears and goes to the care and feeding of SSIS

and ancillary topics. So

I learned a ton about the concepts

of data engineering on while as

while doing SSIS training and consulting and

development. So when I teach it, Frank,

I share these concepts that

I learned. Because you gotta keep in mind, this all came out around the same

time as a data warehouse toolkit book, by,

Kimball and his crew. And the in

fact, I don't know what the relationship was between Microsoft

and Kimbell, but I do know from the horse's mouth

that the, data flow task in SSIS

was modeled to load, Kimball data

warehouses. There's just a lot of functionality baked right in

that, you know, targets those star schemas, and, you know,

it's it's built to do that. There's so, you know, there

was that aspect of it. So at the same time, I'm reading

and learning and, you know, and then going out and teaching

and, you know, and and consulting. There's

this nice amalgam going on. I'm getting information from books.

I'm applying that information on consulting gigs. I'm

figuring out new ways to solve, you know, problems I hadn't

seen before, And then I'm training. So I'm just

rolling all that together. When I do the training, I'm sharing with people, hey. Here's

some first principles, if you will Right. Of data engineering.

And we call it data integration and BI back then.

And star schemas and why you use them and how they work and, you

know, kind of the trade offs that you get. Data explodes a little

bit. Talking about concepts like staging, data,

the benefits of it, why you like, how

you would wanna build your staging tables.

If you're reading from a flat file, everything in

that file is text. Now the text may be

numbers. It may be dates, but it's really just text.

So you built the stage tables with and bar charts.

So at, you know, stuff like that because you wanna get in and get out

just quickly. Memory than the way to do it would be in memory and then,

like, do validation as you do the insert and things like that. There's a there's

a 100 different ways to slice that. Yeah. There really are.

But, you know, when you did, that was that was just pieces and parts of

saying, okay. You know, Tim, I'm teaching you how to use this mechanism,

if you will. Right. SSIS. But I'm also sharing with

you how you would use it and then why you would

use it that way. And, you know, so there's more to

it than just the data engineering. And the point I wanted to make thanks,

Hector. Merry Christmas to you too, Hector. The data

engineering all by itself, just that world,

that's hard all by itself. Yeah. Absolutely. And then the tool

itself was extremely

flexible. And, you know, from the years that you and I have been sharing

about stuff, anytime you say it's it's flexible, you're

also saying, the the it's a sonic way of

saying it's complex. Right. And if

it wasn't flexible, people would say that it's too simple. And, like, it's just

one of those things where now that I'm in a job where I am in

a on the product group, what Microsoft would call PG, a product group or or

team. Yeah. We call it a BU. I I understand. Like, there's

only so many hours in a day that you have engineers and

there's time to market. You have to kind of make these trade offs.

And, you know. That's it. I mean, that's that that I mean,

I had this real eye opening moment with with I think suspect was the guy

who introduced us, who was an evangelist at Microsoft back in the day.

And, you know, I wanted some new shiny feature in Visual

Studio 2005. And, you know, I was complaining about it.

And he kind of pointed out like, look, even Microsoft has limited

resources in terms of people, time and testing and material and

things like that. And I was like, you know, I mean, my god, if Microsoft

has that problem, then I guess everyone has that problem. You know? It turns out

they're just a bunch of software developers just like the rest of us. Turns out

they're all humans. Although maybe now it's mostly AI. Who knows? But,

it's getting there. So so so, you know, I think we both kind

of set the stage for the controversy here. Right. SSI has

been around for at least 20 years,

maybe 25 and SQL Server itself. Let's let's remind folks the

history of SQL Server. It was originally who was it a

partnership with? Sybase? Yes. I believe it was a

Sybase product, completely. And I don't know if it was like And it was like

version 6. Got into the mix, and there was a collaboration

or something, and then they ended up with it,

owning it. That's my best guess on it. I actually

I I know I haven't spoken to her in a while, but I was I'm

friends with and and have co worked with, with

Caitlin Delaney. And she was

with Sybase. Oh, okay. Yep.

So, you know and did we have her as a guest on the show? I

know we wanted to. We we totally need to because that would be Yeah. Interesting

story because I first heard of SQL Server when I was at Barnes and Noble

because at the time we were ready to launch in 19 this is why I

left Barnes and Noble. We're ready to launch by Christmas of 96 with a

Yeah. Linux or Unix based based system based on Spark, Oracle,

and a few other things. No. I'm sorry. 4 g l. It was Ultimate

Formics. And, you know, we had the hardware. We had

everything set up. And then as the story goes, Bill

Gates and, one of the Riggio brothers who was the CEOs

of kind of co CEOs of Barnes and Noble at the time.

Bill Gates had kind of I don't know what he'd done, Jedi Mind Trick.

In August, September of, like, 96,

basically said, no, we're ripping everything we've built so far and we're moving

it over to Microsoft tooling, which at the time was not really mature. I

mean, it was this is like inter dev. I think we had a beta version

of visual inter dev. Yeah. Yeah. Which

was not the best product at the time. Right? It was, you know You know,

I used it At the time. At the time.

Yeah. I I used it, and if you came

from, like, cold fusion or some other development platform.

Yes. Was also awful. Yes. But yeah. So

So I started on inter dev. In fact, that was the first tool

that I I remember downloading for, Visual

Studio. I don't think I downloaded it. I think I went somewhere and bought a

CD or something. Yeah. Yeah. I think I found it in her dev 97 CD,

which was the the second or third version. But, I mean, I we we had

everything written in per on CGI Pearl scripts. Like, we had everything,

and it was just a very different era. But my

take was and this was my I was at the meeting with the CEO and

everyone else. Like, if we don't launch by this Christmas,

people are not going to use us as a habit. Amazon will

take the mindshare and this and that. And then then the

CEO said, sit down, s t f u. Basically, you don't know

how to sell books. You may know technology, but you don't know how

to sell books. Now we can look back at Jeff

Bezos' super yacht and his, you know, moon

missions and all that. These guys have super yachts and moon missions.

Right? They do not, actually. Oh. And my well, I

mean, I'm pretty sure they live in an oceanfront thing in Long Island. But,

he didn't know anything about selling books online either. So I can kinda I

can sit back here, you know, some, you know, good God almost 30 years

later and kind of be smug about it. Right? Right. But

it's just it's just funny. Right? Like, so so what's interesting is and I think

this really cuts to the bone of what this controversy is. And I

have the thing queued up. I can kind of show the screen where you posted

it, where

the fundamentals haven't really changed. Not at all.

Right. Yeah. Binary is still binary.

The debates about schema optimization and things like that are still

very much the same today as they were

20 years now. The numbers are bigger. The stakes are arguably bigger.

But for the most part, the fundamentals haven't changed. And and

I would say this is really where it kind of boiled down to. And this

is this is where the controversy starts. So buckle up, kids.

Let's see. I will share the screen. There's actually

2. I think you talked about one of them. The choices? I'm only

aware of 1. This is the this is the one post,

and I'll drop you the link, to to one of

the others. Right? Yeah. I'll put it in a chat.

I'll I'll send that to you here. Just a second. Along can can understand. So

this is what I saw. And it was basically Kendra

Little, who was a I would say legendary. Scary

smart. She's legendary in in in in the sequel

kind of family, right? Hashtag sequel family. Is that still a thing? I

think so. She's legendary. She used to work at Redgate. I think she worked at

Microsoft, too, at a time.

I think so. But I'm not positive. Well, we can look at LinkedIn. If only

we had that information. But anyway,

so you basically so if you read this and she says

so it says strong disagree. Don't run after every shiny

thing. Again, that is good advice. But, Lord, I would assume

that is her saying. But Lord, don't learn SQL Server and

SSIS if you want to be a data engineer. That's 2 decades too

out of date. Sincerely, a SQL Server expert. I think that's

a bit harsh. She's right about this part. Don't try to change chase out

there if you show anything. So apparently, I can't, and I

can't select a thing. So I read that,

and and I know there's more controversies that are in there as I as

I look at the thing. And you said I humbly submit

data engineering may be accomplished even in the year of our Lord, 2024,

using T SQL, this foul year of our

Lord, 2024. To borrow a phrase from Hunter Thompson,

T SQL, SIS, ADF, Fabric Data Factory

and other technologies supported by Microsoft, which I thought

clearly Microsoft's not going anywhere. Right? Yeah.

And so I basically said

fundamentals never grow out of style. Then I think I wrote again somewhere

like when I looked at the context of it because that's

not what you're supposed to do apparently in social media. You're supposed to react right

away. I did that, by the way, Frank.

I'm guilty. I did not go look at the context. So this is the

original context. I well and, you know,

you pointed that out and I'll I'll be honest, I I'm still

running on second hand information. I have not yet clicked it and gone back

to, to our guest post. Now I can see

it. Now you can see. So so this is what struck me is, well. This

is what struck me as odd. And I know we had talked about it and

I had talked about it. You talked about it. We talked to each other about

it. You know, we talked to our dogs about it. I don't know. Like, but

like, it was kind of like so so when I read the thing, it gets

even stranger. Right? So Yeah. He was talking to

someone, and I guess strictly speaking, even this is secondhand knowledge. Right?

But, so that's the data

scientist in there. Like, well, strictly speaking, this data is also all right. So

so look looking to someone to get a job as a data engineer. Okay?

Right. Unfortunately, he was learning about LLMs and other ML stuff.

I'm like, that's not data engineering. That's a

AI engineering or data science type work. That's more like I think he's he's trying

to set him straight from that. He's like, you're learning the wrong things. That's how

I read those two sentences. I mean, I would say you're learning the right things

if you wanna be an AI practitioner.

Yeah. But I wouldn't call I wouldn't, you know, read up on Langchain,

you know, Ollama and anything LLM and all that stuff

and then call myself a data engineer. I mean, that's Yeah.

That's like a cardiologist cutting up you know, doing your taxes. You

know what I mean? Like Sure. Or or cutting open your brain. Like,

I mean, I suppose there's some similarities, but it's not

the same. Well, I I do like bullet number 1.

Yeah. You know, let's see that. Yeah. This is something I think

that you point out quite a bit. So when you give your talks, either on,

SSIS or ADF, you ask

people, like, how many people here have workloads running in the

in the cloud or right? And then only a quarter of the hands

go up. Well, it's it grew to about

40% the last time I did it, but it's been over a year

since I since I spoke live and asked that question, ran

that little survey. There's a slide usually hidden in,

all of my presentations that has survey up near the very

top. Right. You know, it just and that's that's what the survey is about.

And often, especially

say the last, I said it's been over a year. So let's say from a

year ago and then back maybe 4 years of asking that

question. Almost every time I did that and people

didn't see everyone else's hand go up with theirs,

the those people would come up to me at the end. And usually, their

first comment was, I didn't know

that it was most of the people here were not doing

production jobs in the cloud at this point point with data. I thought we

were way behind and we're the only ones. And my response would be

2 fold. The first would be, that's because Microsoft

marketing is doing an astounding job. That is not a swipe

at Microsoft Marketing. If anything, they deserve a raise

because they were so effective at communicating

how cool this is Right. And how these larger

companies are doing it. You all of the big shows, keynotes,

There's some list of big companies, and they're almost all of them or

companies that you'd wanna work for because it's prestigious.

That's so I don't know if you want it on my personal market. Seem like

everybody's doing it. And I I know I know for a fact it's not always

true because when I worked in the sales for Microsoft, we

would encounter them and there was a pejorative term used internally called server

huggers. Okay. Right. Because like, oh, they're

server huggers. They'll never go to Azure. Right.

So so now, you know, I used to see that it's server hugger as a

pejorative. Now in light of kind of maturity and

working, with more customers and being

more aligned in the open source kind of realm and dealing with

international customers who have very real regulatory concerns.

You're right. Call them smart. Right. It's not, you know,

I didn't so much drink the Kool Aid is I became one with

the Kool Aid. You couldn't tell where I ended and where it began, where I

kind of had this deep programing experience

of. Yeah. That's not always the answer. Right. And I

think that dealing with LLMs and AI and things like

that, I think really makes that more obvious.

Right? Yeah. I totally agree with that. And, you know,

to be fair, and I wanna start with, you know, with being as

positive as I can about this. If I was It's not a negative on any

from scratch. Wasn't. No. I I'm just saying. But if I'm starting

today, day 1, and I wanna go, be a

starter company and and work with data, I it

would be foolish. Foolish to start

today and not go to the cloud. Absolutely.

So and and the reasons are numerous. Yeah. Here's the

thing. The companies there are a handful of

companies, really large companies, mind you,

that have started sent in the cloud age. Let's just call it

that, or the Internet age. There's a small number of them

that have gone on to be huge, but they are really huge.

They're overpowering, oversized. They're larger than the

companies that are previous to the Internet age companies

that have made their way into the Internet. And that's that's

not an accident. However, those

companies, the brick and mortar companies, are the companies calling consultants like

me and asking me to help them either

transition from a purely on premises

environment, managing their data into a cloud environment

or the and back before that, in 20 years ago, when I was first

getting called to do this kind of work, they were just trying to figure out

how to collect their data and then analyze it. And

so, you know, SSIS was a great way to do that. T

SQL was everywhere. Azure Data Factory didn't exist. Yes.

Much less Fabric Data Factory. And so we were just trying to solve

this business problem. And I was trying to couch couch my responses,

especially there was a thread that that got combative, I

would say. And, you know, as we went went down

through that, and I kept trying to say, and

I did. I said over and over again that, you know,

my job is to go help solve these business problems.

And what I meant by that opening line,

that T SQL, SSIS, Azure Data Factory, Fabric Data

Factory, even in 2024 of viable ways to accomplish

data engineering. I I meant that, and I'm not back backing

off that for one minute. I I misunderstood the context of the question,

and I didn't really understand until I listened to your stream

last night where you had gone back and done what I should have done and

read the original post. And you said, yeah. It's kind of a mixed mesh

post. The guy's talking about data engineer, but he's also talking about LLMs

and machine learning. And in the middle of that, he

throws out, you know, this comment about SSIS,

how 90 99.5% of the companies are still using. I

think that estimate is high. I I think it was more of

a, let's make this point that there's still a lot of companies out there

using, T SQL and SSIS to

accomplish this. And this is something that I can't find the comment that I put

in there. I'm looking for it now, but. Yeah. Some of the

comments I can't get to anymore. I don't know why. Maybe they were

reported or maybe they're. Who knows? Right. I mean, social media

does weird things to people psychology. But the

point that I think that I wanna say

that Kendra overlooks. I think everyone overlooks it.

Data and back end systems have a

longer shelf life. And I say

this as someone who was, what, 10, 15 years ago,

strongly ensconced in client development. Right? Whether it was your

Windows, Windows Phone, or other types of Windows

based devices. Right. Or web development. Right.

Those technologies turn over pretty quickly.

Right. You know, you're likely to get

multiple updates per year on a device phone, like an app on

a device, but you're likely to never see,

a radical change or redesign. You'll you'll see a

radical change or web redesign of a website or portions of a website

couple times a year maybe. Right? But you're never gonna see

a radical redesign of a data back end system,

but once or twice a decade. And It's true.

Yeah. And mostly what drives that is scale,

not features. Right. Not features. It's just date or just tend yeah.

Exactly. Right? So if you a 100 x and who could who could accounted for

that, you know, going to the project started. It's a problem. Still a

wonderful problem to have, but but a problem nonetheless. Well, and there's

also the fact that, you know, it's 2,000 whatever now, and

there's still mainframes running. Right? There are still not not not to to

knock on IBM too hard because they are the company of Red Hat. But,

d b 2 is still around, still getting updates. Still backbone of

many Fortune 100 companies that also share the stage with Satya

at these big Microsoft events too. Right? Like Which was mind blowing

for people from the old days of Microsoft. Right? Well,

that's a whole other thing. But, like, you know but, I

mean, it it really boils down to, like, these technologies have a longer shelf

life. So if something is 20 I think we get

hung up. 1 of the threads sub threads in here gets hung up on, you

know, 30, 20 year old technology. We're thinking that, well, you know,

there's a meme of the the little monkey puppet, like, you know,

giving a side eye and then goes like a cringe face, like, and a side

eye where it's like, oh, Windows is, you know, I don't know, 40 year old

technology. And I'm thinking, like, some, you know, Unix people or Linux slash

Linux people are, like, 40 years old is old. You

know? I mean, this stuff goes back much further. So it's but

it's still like and that's not a knock. It's just

No. It's just now that we're in this

industry now for as long as we've been in it and the

industry's been around longer this long, there's just

stuff that is gonna just start aging out, but it doesn't age out as

quickly as we think it does. It's not like it's not like the iPhone. Sure.

Right? Where you the iPhone I don't know what number they up

to. 16, 17. Right? Oh, well, suddenly my iPhone 15 looks bad, and that

happens every year or 2. You this you don't see that in

database systems. Right? The only impetus to really move, say, from, like, SQL Server

2,005 to 2019 is updates stop going. Right?

And that's a whole big project. Yeah. The maintenance cycle. So it goes out

of maintenance. And then you worry that if something crazy happens,

you can't get support for it. And that's

kinda like, you know, it's it it's sort of it I'll say this.

It's analogous to your phone starting to run slow for some unknown

reason. That's funny. Something something on SQL.

Yeah. Something something. Sybase something. Well, and you think about all the

I mean, I mean, and contrary to this, contrary to that statement of these things

have long life shelf lives. Yeah. Is the fact that I mentioned

Informix earlier. Raise your hand if you heard of Informix. Right?

So I've heard of Informix. You've heard of Informix? I mean, we don't count.

But but, like no. But, like, I remember my first

experience with Informix was because some alum of Fordham had

because it was a big shot at Informix. And, and I think we had somebody

who was also a big shot at Silicon Graphics. So we had SGI machines

running at Formix. Right. So I remember my first UNIX I used was an

IRIX system. Right. Which most people today

wouldn't even know what what that means. Right. And, you know, but Informix is

out of business. Sybase is gone.

I can't even think of other names. I know there's more.

Right. But really, the only things that it those have probably been

migrated to SQL Server or Oracle. Well,

or some form of Postgres or something like that. And I

I hear you. You know, there's there's an argument to be made

for, you know, the the cost of maintaining

old software. Right. There there definitely is.

I'll say this about SSIS. I if you learned

SSIS in probably in 2,005 era, between

2,005 and 2,008, that engine

I I don't know how many lines of code were

changed before it was upgraded to 2008

or r two, but it changed. There were some performance tweaks in there. It was

obviously, faster. And then again, that happened in

the 2012, error when we saw

I love that SQL tab.

You know, it's dead. Long live Crystal. So was Crystal ever database or was it

just Crystal Reports? Reports is all I knew. I didn't know about it as a

database. That's all I I I use Crystal Reports and

my favorite thing was it filling up the drive because

it kept caching things. But I

remember the whole idea of just because you place it somewhere, it doesn't

mean it's actually gonna end up there. Like, the whole thing is

but sorry to cut off. That's okay. But SSIS in general,

if you learned it even in, you know, 2006, came out

in November, I think, of 2,005. Even if

you learned it then, it at at a fundamental level, it

hasn't changed that much. And whereas you'll see other software

you Visual Studio is, you know, a software development platform

that allows you to do c sharp and v v and, you know, all of

the stuff. And it allows it still supports for us. I know. I haven't tried

v v in 9 years now. So

it's been a while. But if you look at

how much most software changes from a developer

perspective, and SSIS is software development. So as your

data factory, and any data engineering, that software development,

SSIS is probably in the 95%

of what it was. If if you knew the fundamentals

in 2006, you know those fundamentals in

2024. And Right. Part of the decision

to go make the upgrade, we talked about, you know, maintenance wonders and stuff, and

I I get it. And it's not the same as your phone slowing down. I

said that, but that's a bad analogy. But Well, it is also

it's also, I think, also very relevant to Windows 10. Right? If you're

on Windows 10, your updates are gonna stop in October. That's

true. I don't wanna get on that soapbox and rant. Sure. No. But I

I mean, there's I get reasons for that as well. I don't

like that it's gonna change because I like Windows 10. But,

but yeah. Well, there's there's I'm gonna join you in not

going down that road. But I'll say this. Hey,

Maddie. How are you? The, the

just the fundamentals of data engineering haven't changed. And the

tool itself, you know, if you knew it back then.

And it's, you know, you know it now. And so if you

learned it now, you could go back then and still work

in the previous versions of it with, very little headache. And

that speaks a lot speaks volumes to the,

the team that designed and built that. And

so in addition to the technical reasons for doing this, the business

reasons, kind of revolve around one of my favorite

phrases. I mentioned this in today's newsletters. It's a compelling

reason. Do you have a compelling reason to make this change?

And business people think about this all day every day because

the amount of money that they make, the profit is based directly

on the amount of money that they, you know, they spend and are they getting

this value for it. So if they can improve the performance of

something, say 10 times, and a result of

that is they get, 5 times as many customers,

then that's not a bad investment. That'll just work. But

if you're coming to me and I'm a business that existed

before the Internet, if you're coming to me and saying,

I want you to change to this completely different model,

where, you know, and and the way it's presented

often is you can save money. And that's

true because if I start a new business today, I'm I

couldn't even compete. I'm not gonna be able to stand up the

servers, you know, take that time and buy that hardware and float

that that inventory that I need to manage all that. Whereas,

I can I can pay rent essentially every month

on that service? Right? Right. I I always like to say I

always like to say if I wanted to start a bookstore today,

right, versus 1996,

right, or 1995, depending on when you wanna say when they started.

I mean, Barnes and Noble spent a ton of money. I don't have the exact

number, but I can there's probably tens of 1,000,000 of dollars, probably closer to a

$100,000,000 to just before they had their first customer.

Right? Wow. And that but that was the heyday of the

dotcom. Right? Because they were you know? But then

but if you wanted to start a bookstore today, whether or not it's a good

idea, let's let's just suspend our disbelief for a

second. You can probably do it on a on an average credit

card limit. Because

because your IT is enabled. Right. And and you pay like I

said, you pay fractions of what you would have to do in the brick and

mortar. And most of the initial spend isn't gonna be your servers or hardware.

It's gonna be in development and marketing. Right? Getting the word out

because it's such a noisy market. It did the the market has radically changed.

And I also think imagine go ahead. I'm sorry.

What? Imagine? I was gonna say imagine that you've built

Right. This infrastructure on premises already. You've got all of this done.

It's a sunk cost. We can debate about how to feel about sunk cost.

Right. But it's there. You spent the money and it's there. And

you're not gonna get that 5 x income boost when you move to

the cloud. In fact, in some cases, not

all by by any stretch, but in enough cases, you

move to the cloud and it costs you money. Because when

you're getting the presentation about starting using the metrics of this

new company being started today Right. You're, you know, you're told the

truth. You're not being lied to at all in in any of this.

But often, systems that were designed software

and front end back end systems that were designed, you know, from

the nineties through the mid early 2000.

Those systems were architected in a whole different mindset

of what's the prevalent mindset for today. And as a result of

that Yeah. Yeah. As a result of that, one of the things missing from

the spreadsheet calculation that you're gonna get the ROI

from moving off your on premises servers to the cloud is

that couple of $1,000,000 and about 18

months, of the hit that you're gonna have to

spend rearchitecting Yep. All of your systems so that they

now fit today's paradigm. And frankly,

if you are interested in doing that, you you could go do that

at any you could have done that at any time in the last 10 years

and made that shift. But people didn't do

it because the business people didn't do it because the ROI was not dead. There

was not enough return on that investment. If they wanted to, they

would have spent that money then, but it wasn't gonna improve the bottom line.

In fact, it was gonna hurt the bottom line. And so you see

companies now make this move into the cloud and

then, yeah. Yes. That is a

that that's an astute question to ask. So for those who may be

listening and not viewing this, it says is SQL SQL dev

d b a says, I use Brent's and I'm assuming Brent Ozarks. Brent

Ozars. Problem are you trying to solve by changing this for justifying

upgrades? Brilliant. That's that is brilliant. And he's

right. And the you know, but it's compelling to hear and

read the case studies of of companies that, you

know, were able to do use to to access

$10,000,000 worth of hardware, like you said, on a credit card. And think,

wow, what would that do for us? And the answer is sometimes, yeah, it'll

revolutionize your business. You'll 10 x coming out of this. But other

times, it's like, no. You'll point 8 x.

You know, this isn't as compelling. So it's interesting because, like, I think

there's a number of and I found the article. I'll pull it up. But but

one of the examples, it was either Dropbox or Box. I forget which company it

was. But but they had basically started off, I think, in

AWS. Mhmm. And then they got to a certain size. They actually

figured out it's cheaper for them to design their own servers that are optimized for

mass storage Mhmm. Than doing it. So they started building their own hardware and their

own stuff. But I could tell you, if they were a startup and they went

to a VC saying, we want to start with this on prem, they would have

been laughed out of the building. Yep. Today. Yes. Today they would

have. They mean, you know, and it's

it just shows that the the shifting economics of cloud versus on

prem and and other types of things that I don't think people really have figured

out yet. So this was a really interesting I'm gonna share this tab if

I can show it on the screen. Sources. But that that

use case is you can't, you know, having the compelling

reason to migrate to the cloud, and you can do that upfront.

It's harder. But exactly what you're showing there, you're

sharing that that idea of leaving the cloud, that's

growing. And it it's growing across the board. And I

one of the, metrics for that that's

directly related to what we're talking about here today with

with data engineering, is that there

there's been an increase in 2024 in the

number of, people that reach out to me to talk

about, SSIS help with their systems.

And, I mean, I do consulting in, you know, ADF and fabric, and

most of my consulting has been in ADF. When SSIS was involved,

it was in lifting and shifting SSIS into an Azure SSIS,

integration runtime. But all of a sudden, after

2, 3, 4 years of that, that shifted this

year. And people started reaching out to me with SSIS on

premises consulting things, and I kept up with it. So I was

able to do it. And but there's other

evidence that I will not share. I probably I may be able to, but I'm

just not going to. But it's even better evidence than my

anecdotes about people more people reaching out to me. Right. That

the amount of SSIS being executed in

the world has increased, and it's a double digit percentage

increase just in the past few months.

And I I think I now this is where I start speculating, and I

don't know the answer to that. But we have a our mutual

friend that we, another mutual friend you and I connected with

in November of 2025. Sorry. 2005. Like in the

future. Recently recently worked for a

year and a half, 2 years for this large agency that's

not part of the government, but does money supply stuff.

Oh, okay. I know. I know. Yeah. After getting his MBA from Sloan, you

know, which no. Sloan. No. Sloan is

important. The, you know MIT. School with MIT.

Right. He's a graduate with that. Super smart.

He shares with me when I'm telling him this story, I give him that stat,

and he says, here's what's going on. Economically,

money is more expensive today than it was. And

so he said he said that as he's telling me this as a

cautionary tale because he says it's gonna change. It's good. Money's gonna get

cheap again, and people are gonna flock back to the cloud. That's his

theory that it's all being driven by money, and I don't think he's wrong,

especially I think that's one level. I think that's one lever. I think there's

more than one lever. That is certainly a big one. But I you

know, as someone who I you know, my previous role at Red

Hat and my current role at Red Hat, I have to think globally. Right? And

we don't again, not a commercial for Red Hat even though the

fedora is there. You know, one of the things we do

is we basically provide a data platform end to end that

can run-in any cloud on

prem or, you know, one of the hyperscale. Or hybrid. Yeah. Yeah. Or

hybrid. Right? Where and there was one customer that I spoke with

before I won opportunity to leave. They were, big government agency.

And this big government agency, you know, they have

their own data centers, even though there was a push to

get rid of them all. But they also have because of way contract

government contracts work in the US, they had, foot

you know, money to spend in AWS, money to spend on Azure, and I think

even money to spend on Google Cloud. So the one

advantage that we had that the other ones couldn't is that the he called them

the soft costs of training people how to do he'd do the same

thing to do linear regression in SageMaker and

push them out of production in SageMaker and Azure and in Google.

Right? Yeah. And this was one tool. You learn it

once. You administer it once. The same glass.

It was the same thing. I think those environments are very real. Now those are

probably limited to large customers or kind of the government

agencies that have these kind of contracts and things like that.

Yeah. But also Mhmm. You have a number of

countries that it's just not a good look to move

your data out of country. Right now, in the

US and Canada, we don't have this issue because there's plenty of all the hyperscales

have footprints in Canada and the US. But if you're in

Latin America, which is this where this example comes from, right, there's only

at least as Red Hat defines Latin America, includes Mexico, and basically all the

way down to Antarctica. Mhmm. And only just,

like, 30 some odd countries. Right? Someone's gonna write me hate mail saying that this

is the exact number, but let's just keep the math simple. It's 30. We

love those mail. We do. We love we love the mail. We learn things every

time. Right. I talk about them personally when I get corrected at

at the dinner table because I wanna share that with No. I mean, it's it's

good. I'm not saying don't do it. I'm just trying to keep the math simple

because it's Friday before, you know, basically, we're sure holidays. Sorry, Frank.

I I derailed you. No. That's fine. Only 3 countries

from Mexico down to Antarctica have hyperscaler presences.

Now, the 4th one in the but out of 30.

Right. So it's actually 10% or less realistically. I think it's like

37 countries. I asked Wikipedia and stuff like that.

So less than 10%. Right? Right. If you're in a

country that doesn't have a footprint. If you're in that 90%.

You have to ship it out as a country. You have to be okay with

that or do roll your own solution on a thing. So there was a

government we we won a big contract because they

wanted to do advanced AI and they wanted

to keep it in country. Right? Doesn't necessarily have to be on prem. Could just

be, like, you know, an Equinox data center down the street or something like that.

Right. But within their thing. And it was a government agency, so it wasn't

computer science or even data science. It was political science that really kinda was the

driver there. Right? Because if I'm in country x and I have to move

my and I'm a government agency in country x, I have to move my data

to a sovereign country y. Not a good look.

Yeah. Right? And,

you know, would it really matter? I don't think so. Like, in a but

from a legal point of view, it kinda does. Like, where the data resides in

the residency. And I think if you go to the Azure website

now, they'll actually tell you where the data resides. And they actually interestingly

enough, they get down to granular, at least on the US side to the

state. Right? So, like, it'll say, you know, Virginia and

stuff like that. We have a comment from, or not. I'm gonna hide my

screen so I can look up the Azure map and kind of

demonstrate that. And then I have to figure out where the sources are. There we

go. So you wanna read the comment while I do that while I'm distracted? Sure.

So so many comments, but there are clients who are

considering the technology used in the software package, and they

may escape when they see the old school,

stuffs. I'm so maybe.

And and that may be, you know, I hate to be that guy that says,

oh, that use case is invalid. I don't think so. I'm not aware of it.

That doesn't mean it's invalid. I'm not aware of a lot of things.

But but maybe. And, you know, definitely

have mixed emotions, about that. If you're buying

because the the if you're a client and you're

buying from, some company and you

decide to go to a different company because of the

technology stack that's being used behind there, I

don't know. I think that says more about you as a client than it does

about the company. If they're delivering the service and it's, you know,

the the the rules of data engineering are you get accurate

data as fast as possible, and those priorities

are in that order. Yeah. I don't think the old school stuff second.

The only risk of the old school stuff is it's still maintained or there's still

security packages. That would I mean, if I were in that

position, I would be like, oh, I mean I mean, if you're still using, say,

Sybase 6. Right? You know, like Yeah.

You know, got a little there's definitely a line there, and it's drawn

based on it's drawn different places first. And some of the

reasons, that it is drawn in different places is

security is huge these days. I mean, that's gotta be your number one

concern. And and, you know, it it

goes from there. But if you're delivering the service securely,

I'll just pick that one, then I would say

that, you know, that if if you

lose a client because you're not using the new shiny,

I don't know what you can do about that. I'm trying to think I'm not

gonna say the client's wrong for feeling that way. They probably have valid reasons

for feeling that way. But if, you know, if they wanna if they

wanna make that decision based on that. And I'm looking at Frank's graphic here

of the is that the data centers? This is the one I was telling you

about. Right? So this is this is just Azure, but I would say it's a

pretty good proxy for the other hyperscalers. Right? Mhmm. I would

say Azure at one point had more. I I haven't kept up.

Well, that was one of our talking points when I worked at Microsoft. Right? We

have more than to be honest. Right? But I would say if it's not exactly

more, it's close enough. Right? So there's Mexico. You see the United

States is pretty well covered. So is Canada. Right? So if you were a Canadian

company, you had to keep it in Canada. You had an option. Right?

Yeah. If you're an American company, you have pretty good choices.

If you're Mexico, yep. But if you're in any of these countries in Latin America,

down through South Wales. We only had okay. So now there's Chile.

Okay. Gotcha. Right? So I'm sorry. Now there's

4.

So my math is going to get more complicated right away. Right. So, there's

Brazil. Actually, no, there's still 3.

So there is no footprint for Azure in Argentina.

These little blue things you see, that is Colombia. Those are networking

pops. So basically, from a

networking point of view, if computer science were the only thing that would matter,

then that be that would be acceptable. But data residency is

the issue. So if I go here, the US East 1.

Right? It'll tell you that its

location is Virginia and it's stored at rest in the United States.

Like like here.

Is there any more details? There isn't. US.

Yeah. They used to. Yeah. They don't talk much about that,

about where they are. But one of the

US East Georgia. 1 of the Yep. The US

East 2 is down in Danville, isn't it?

They are or Mecklenburg. I forget which. Well, essentially, they chose a

picture of Richmond. Right.

I mean, these are, you know, with

we kinda touched briefly on on politics in a geopolitical

slash, sovereignty strategic way.

Right. These are huge. And I I know I'm not the one, thinking

about it. But Look at this. Yeah. There's one in Israel. Those satellites, Frank,

are getting it, by the way. Those those

satellites on the graphic, those things are moving at way faster than

normal satellites. Oh, yeah. Yeah. I mean, satellites are going to

change things, but, like, in terms of where the data sits at rest is really

where because ultimately, I think it really boils down to when are the

local police going to barge down a door with a with a court hopefully

with a court order and basically copy everything. Right. That's really what

matters. Right. That turns out that's really what ended up mattering. But

if you look at the Middle East, right, like 1, 2,

3 countries have it.

Right. And that entire region, you

know, obviously with geopolitical tensions being what they have been for a number of

years. Yeah. Moving your data center to any one of these countries may be an

issue for you for your organization or your

regulatory. Right? Europe is kind of the same thing, right, where, you

know, there's Switzerland, there's Italy. And I

know that there's different kind of things in terms of Germany.

It was actually I don't know if it still is now, but it

might have been a, I used to live in

Frankfurt, actually. Yeah. There was actually what they call a

sovereign cloud because there was concern that if it was a US company owning

a data center, that US courts would have jurisdiction there, which is a

brilliant move by a a past administration. I say

brilliant sarcastically in case you're didn't get pick up on that.

Where they thought that they could basically issue a a court order to

demand something from here in Ireland. And

Microsoft fought that because they realized, like, wait a minute. That would mess up our

entire that would cause a lot of problems. Yeah.

And, ultimately, they dropped the case before it was finally

decided. But in order that they could, thing, at one point,

anyway, this is actually owned by a German

company, managed by a German company, and it's leased to Microsoft to to

to have that concern. I think China also operates the same way.

Frank, I was commenting on the, satellites on the graphic

there. Oh, that they were moving around. Yeah. Yeah. They are moving very, very

fast. And it keeps they're moving with us.

But, I mean, keep in mind, though, like, keep in mind, though, that

we're just talking about data residency. There's other things that if you're building a real

solutions, other things to consider. Yeah. Right? Like And there's

a whole lot to that. And Yeah. Yeah. Oh, absolutely. And,

you know, part of the part of it is,

part of what what happens when you start kinda going back to

the data engineering, platforms and stuff that you use.

There are sound business reasons for not making a change,

and there are some unsound business reasons that will

confine you to not making a change. And I I think about this. I'll put

it in context of, of SQL Server.

Companies will come up and they this has happened, and I still have clients

running applications on old servers

because the company that so they

they serve, their clients include

enterprises that care an awful lot about

checking boxes and auditability and all of that stuff. Regulatory

type things, which is not bad. It's just the

way that it is. That's their their business demands something

like this. These companies were formed. They were stood up, and they've got SQL

Server 2,005 running or 2,008 or stuff that's been

out of the maintenance cycle at Microsoft for a long,

long time. And unknown it's also not a well known

fact that if you don't wanna upgrade to version x or

y, you can pay extra money, and Microsoft will maintain

and provide you patches. Right? There there's rumors that

there's at least as of a few years ago, there were still Windows 95 systems

that were, you know and that sounds absurd.

Yeah. But Well, you walked down the, entry to,

Delta flight, and there was one that's what was it? One is 97, I

think, sitting there. 98. Yeah. 98, was it? Yeah. One is 98.

Sorry. Yeah. One is 98, boxes sitting there for the longest time. They're still

I believe they're 1 to 7 now. Still. I I

saw XP. XP. You're right. It is XP. Yeah.

So, you know, you just I kinda noticed this, like, wow. I hadn't seen

that in a while. But it's it's not about

will the new technology run that

SQL Server 2,005 database. The answer is clearly yes.

Well, you can always virtualize something. Right? Like, that's something that, like I

mean, there's that compatibility levels help. Right. There's a number of things that

do it. But here's the kicker. If the application is

not certified to run on that

and you're for you can change it, and you be maybe you have changed

it and tested it and go, yeah. It works. We'll just move it to, you

know, SQL Server 2019 or 2022. We

know it works. But the people you're serving,

people who care way more about checking all the boxes and the regulations

being a 100% and auditable, they won't

allow you to. And it gets even more complex when that company that

originally sold you that software 20 years ago is no longer in

business. Right. So you have no path forward.

I mean, the only Or they get bought by another company that you

don't really like. Exactly. That's happened too.

Exactly. Then A lot of mainframe companies were brought up by I don't wanna

name names, but, like, were brought up, and they it really was, like,

ironically, because they what they do, they they knew they had them. And

ironically, a lot of mainframe migrations happened because of

that. Like, it was And so you've got, you know, you've got that

angle where people are sticking with older systems for whatever reason.

And it's, you know, it goes like I'm saying, my point is that this goes

beyond just the data engineering realm. There are

there are compelling reasons to use,

older software. It may not be anybody's, you

know, satisfactory answer, but it is, you know, those

reasons exist. And if you're the, you know, if you're a

developer who likes using the new shiny and learning the new

stuff, I'm one of those. That's why I'm teaching courses on fabric data

factory right now and watching as it kind of some

days it works and some days it doesn't. We've had that happen on a

number of deliveries this year, with that.

So if you read what I wrote about this and

you come away with Andy's against the new stuff, well, you're just

as wrong as wrong can be. That's not the case at all.

You know, it's an

interesting I mean, so back to the lecture at hand, what kind of kicked this

all off and inspired the stream was

this post where I think the short

answer is everybody's a little right. Everybody's a little

wrong. And as a consultant, you can appreciate these two

words. It depends.

Right? Because, like, you may want to upgrade to the new shiny. I know every

developer wants to do that. And I think the comment for some reason I can't

say is like basically hiring managers will put in a job description. All

there's also the other matter of job descriptions and, you know, job requirements

are. They're always a 100% accurate. Disconnecting

from consensus reality. Yes. I like to say.

But they may want someone with, like, say, ADF

and and and this, but then actually have them working on systems and SSIS

because the hiring manager knows that he or she may not have that open req

for a while and has a in the back of

the mind the idea of moving to that someday.

Sure. But realistically, for the next 2 years, you're gonna work in this site.

Yeah. I mean And there are still large

consulting companies out there that develop brand new

applications in SSIS, brand brand new data

engineering data warehouse. One worse than that or one better depending on your

point of view. A few years ago, I think it was on dotnetrocks. They

were talking about telemetry from Visual Studio. And this back

when I cared about Windows client development.

Basically, WPAF came out in 2006,

2007. Right? XAML.

No. Not XAML. Metro or modern

applications, UWP came out in 2011,

2012. Right? So there's been multiple

frameworks to write when and it's been a number since. But, again, don't really care

about those client development anymore.

Windows Forms is still the number one of

all those, like, ways to develop Windows applications that run on Windows

Mac. Windows Form is still accounts for 80% of development.

Wow. Something someone's going to like, please email me in

hate with hate mail, not hate mail, but like tell me the exact number. But

it was still. He's off. Well, and they kept saying in Visual

Studio 2005 came out. They said, this is it. This is the end of the

line for Windows Forms. We're not updating. We're not adding anything.

And the future is from now on. Right?

Until the future became something else. And then when

I last installed, I think it was Visual

Studio 2016, 2019.

There was improved. They added stuff to Windows Forms, which is kind of funny because

they said they never would. Yeah. But it just because they.

Demand. Right? Customer demand. Ultimately, customer demand

is what pays the bills. So you've got to be very mindful of that.

And, you know, if you if it's 2024 and you're writing

a Windows Forms app, I have questions.

You know, I'm not saying I disagree, but I have many questions.

Arguably, you could say the same thing for UWP or

WPF. Right? You know,

But, again, it really depends. Like, in the last time I had worked with

WPF professionally was it was for when I

was at a, between my stints

at Microsoft, and there was a, you know, there was

a customer who was a mortgage company, and they basically had their mortgage

intake form written in WPF. Right?

And they were having performance issues with it. And I I looked at the code,

and I was like, this is a good lesson, I think, is

that, you know, they loaded up, like, some 6, 700 controls

all at once. Right? Wow. Because there were a lot of

fields and but they were all collapsed and things like that. And I was like,

well, I'm looking at this, and I'm, like, testing it. And I'm like, oh,

dear god. This is gonna be a nightmare because it's 600 controls. You could do

lazy loading and things like that, but then Sure. There could be unintended

consequences there. And then then I happen to notice when I load the

app, the CPU spikes, but the GPU was

hardly touched, which the whole promise of WPF was

that it would offload as much of the rendering

Yeah. Over to the GPU as possible because it was basically built on

XNA, which was a gaming framework. But that, again,

different different sidetrack, and different lifetime ago.

So I'm like, what the heck is going on? So then I'm like, I looked

at the some of the machines. I'm like, they had the generic

GPU driver. So I'm like, just

for grins, let me see if I

can get the proper driver for this device.

All of a sudden the 30, 40 seconds it took to load that initial

screen went down to 5 seconds.

Wow. And that's a big jump. That was a

big jump. And that was like, and I said to the guy,

it's like, look, just installing this driver, you get

a, you know, massive increase in it. Right? Do you

really wanna architect it or are you trying to just you want this to be

faster? Like, what's considered acceptable? And he said, well, under 10 seconds would be

acceptable. And I was like, I could do you better. How about 5 and a

half? Right. And I showed him

and he goes, well, not everybody has a GPU in their device. And I was

like, well, like, you know, this GPU costs

about, I think, 189. For some reason, 189 is stuck

under $200. Yeah. And I'm like,

so you'd have to install it. You have to think about the labor of installing,

like, this cheap GPU. Right? And this, he goes, that's

fine. He goes, look. The cost of redeveloping and

rearchitecting and retesting this versus $200 per

box, plus whatever it takes for somebody to go in with a screwdriver

and update the drivers. Right. It was so like we

ended up not having to touch the code at all. Right? Yeah. It was just

a matter of a driver update, which nice. Because it was like a fixed

price kind of support contract. I was the hero because I

solved the problem with about 3 out 3 to

5 hours of work. Nice. And the customer was happy because

they didn't have to re architect anything. Everybody wins.

Everybody wins. I love it. Happens. But but it it's just it

just goes to show you, like, sometimes the most cost

effective approach isn't is to not

touch anything. You know, and it's although it was a

relatively small amount of money, and and,

manageable amount of time for the client,

sometimes in if you look at the, you know, kind of the the

big, performance tuning

picture. And I I ran into some of this, in the past where

Well, it's not always a happy ending. As as an engineer,

I want to, you know, to fix the the thing that I'm engineering. And if

it's software, then I wanna make the software perform better. And

if it's, you know so I went through one of those experiences and

then, number of circumstances where

but the end result was we we

threw money at it and

bought, better disks.

And I remember telling me about this. I remember you telling me this. Oh, it's

a specific long story, but, yeah, it's back from about 12 years ago. Yeah.

You were, like, just a long SSDs, and it got fast

enough. And so when I did the math on that was the

enterprise level project, there were dozens of people

tuning on a tiger team, and we did make it

go faster. And as a test, what I did was I

rolled the code back to where it was when we

started. I will say now though, tiger the term tiger

team after my Nobody knows what I mean. Sorry. I know what you mean. I

know what you mean. Of individuals, dozens of individuals focused on solving

a particular problem. It's it's like, Although in some cases

slurring. Focused on not solving a particular problem, but that's a story

for another day. Different different story. You're right. But, yeah, we

we did and and I rolled everything back and ran the old

code that we had optimized. You know, it ran super

fast on the SSDs, and it was running okay. It was acceptable,

barely, on, the spindles. But when we rolled it back, it

was, it was the same difference. We actually got a touch more

performance just off the SSDs. And when it you know, you do the math on

that. At that time, SSDs were rather new, and the amount

that I wanted was, not trivial.

I asked for it on a lark, and I was surprised when it showed

up. So she did work for Microsoft. Okay. I'm not

surprised. Kendra is, scary as much. She did consulting. We

Brent's name came up a a few times. She worked with Brent for a while.

She worked at Redgate. Not everyone knows what Redgate is, but they're kind of a

big deal in this in this situation. Yeah. Yeah. They're a big deal. So, like,

clearly, like, I I just find I'd love to get her, like, initial

opinion after factoring in all kind of all of this is that,

you know, ironically, Sync don't run after every shiny thing.

But the the thing that that guy that was originally learning was is

the new shiny thing, ironically. Like, so there's a lot of layers to this.

There's even if you take this kind of at face value of not knowing the

context, there's a lot of layers. But, like, beyond that, there's even more layers. Like,

it becomes this multidimensional problem. It's true. It's like an

ogre. Many layers. Many I was wondering when we're

gonna have a movie reference because it's been a while. There we go. Boom. Shrek's

a classic. Shrek is a classic. The

sequel is not so much, but that's a common case.

Sure. Very, very few Matrix movies looking at here. It was

you know, I I saw the, activity on this and it's best because

since I posted this, I can see how many people looked at that. The the

link I sent you for the other one is kinda the one that started it.

I don't know if you have that link in chat. I think If you wanted

to click on that. I have That was that was a

few days, maybe a week before this. That was the first one I

commented. It was similar sentiment and I shared, you know, again, I

joined the conversation and then, I can't believe is

SSIS dead? Yeah. That's the one. So that

particular one has gotten, like, probably 12

to 15 times as many views. It's happened to You know, you mentioned And

it was the first time I had piped up about anything like this. I'm I

commented on the original one. LinkedIn will pop this up if you write a

lengthy comment like like that one. And

I said, that window exist. Looks like Windows XP

era. Well, that's SSIS. That is

a funny. Like, it's just funny. Yeah. But, And I

commented when this LinkedIn popped up and said, hey. Do you wanna repost this

since you wrote this comment? And I said, sure. And it just pastes the

comment up at the top of the repost. But I can see, like, the number

of people that and that one drew a lot more,

a lot more comments. And like I said, 12 to 15 times

the views. And, and and I communicated

with the original author just to touch about that at all. It it

went nowhere near as I'm

trying to think of the right word. No nowhere. It it

didn't get nearly as heated. I'll say it that way. And it

may just be that, you know, that I'm saying heated, but there was

there was one individual in particular who just very passionate about the tools that they

that they used for. I I I wanted to ask,

that individual, you know, how

many SSIS packages they developed and put into production.

Because I I I I would again, I think I know the answer

to that. This in the this individual conversations like that.

Say that again? Which individual? Not in that one.

Oh. Not it's in the other range. The one we've been looking at. This angle

here, though. Right? Nothing against Kiwi

ETL pools. What do you Yeah. Appreciate so this is this

is an argument of GUI versus Yeah. Straight

code. I think that's also an interesting concept. Low code, no

code versus you know? I like them both, but,

you know I do too. It's for me, it's which solution

matches best and as a consultant who goes to work with other companies a

lot Right. A big factor for me is What they have. You know, what

happens when I'm gone? Can you support it? What what are you most

familiar with? Even if I'm working in a tech a technology

technician, a technology like SSIS, there's really a

couple of ways you can, float it. It. That's good, John.

2 was good. I'm referring to a 3, and I think there was a 4th

one. Oh, really? I didn't even know there was a 4th one. So there you

go. But in SSIS, you can take a more And

the matrix 2 was good as well. Matrix 2 was good as well. I was

That's true. Yeah. You you could take, like, a more

DBA approach, a more T SQL driven approach, or you

can take a more dot net driven approach because you have both

execute SQL tasks and SQL sources and destinations and transformations,

and you have a script task and a script component.

And, you know, which way do you go? Which is the right way to go?

Well, it depends. If you've got a bunch

of dot net developers being tasked with maintaining

the, ETL after I leave, no way. 5.

Good god. Shrek 5 or matrix 5.

Either one sounds like a terrible idea. You know,

whoever's gonna be left behind support net, you gotta make sure that Right. You've done

a a fair enough job of representing their preferences,

on that. And, you know, I I know

I know consultants who come in and say, you know, we gotta do it

this way and you're bad and wrong and you're, you know, you're gonna go out

of business in 18 months if you don't listen to me. Right. That

if I also think too, like, when I first read it I'm sorry. I don't

say hire somebody else. Right. I mean, I when I first read

it, I first read it as should you learn SQL?

Okay. Right? And that's a big debate in the data science community is should should

data scientists learn SQL because, you know, Python can do everything. Just your no. No

knock on my Python can do everything. Should you use Python for everyone everything, I

think, is another question. Right? And I think the answer is no. Yeah. I think

if you're a data scientist if you're a

data scientist or even an AI kinda engineer, whatever the word is

this week, you should still learn SQL

because this one, it's way less complicated than anything else you're doing. And

2, it's kind of the the the the lingua

franca of anything data or data interaction. Right? And

again, Frank, the context comes into play here. Right. There's the there's the

mechanical tool. It's describing a software, you

know, a language is mechanical is one one way of looking at it. But Right.

There's also the problem you're trying to solve. And if if you're gonna do

data exploration or managing or whatever you wanna call it,

then I don't really care which mechanism you use. But don't

tell me one of these mechanisms is better than the other. It

Without a qualifier. For this particular task.

Yeah. So it could be that, you know, one

of them is is good at this one particular thing, and I I would argue

this. I did in my newsletter. Oh, strike 5. Okay. In my

newsletter, I argued, every single tool has something

that they're stronger and there's there's some

feature or some aspect of it that's better than all of

the others. And they also have some weakness

that's worse than all of the others. It's true. And and so

you gotta, you know, you gotta strike that balance. It's gonna depend

on your use case, the parameters, the things that are important to you.

Sometimes it's cash on hand.

Sometimes it's the servers, hardware that you're forced to work

with because you're owned by somebody and they they're not upgrading. They're not

giving you that staffing concerns. The people, their

experience, their languages, that their Well, how easy is it to find someone that

necessarily knows SQL versus knows Python?

Or, the bill rate for someone that knows SQL is probably gonna be different than

the person who knows Pandas. I think so.

I I think there are differences there, but, again, it's gonna depend on the company.

That's true. You're worth our job market. Right. You're worth

more to, you know, this company as a SQL developer than, you

know, maybe than a Pandas developer is to that that company. That's a

possibility. Go go where you get paid the

most, but realize especially if you're new, and I think that

person that in Kendra's original, post that

she was, she was jamming on, that

that was a different scenario. You're right. There's a number of squirrelly things about that

original post, but if you're

a young developer and just getting started, you're in college. I've got a

daughter studying computer science in college right now. And my

recommendation to her is first, learn everything that you

can. Do as much as absolutely as you can.

Pick up that knowledge. But,

be aware that there's more to it than just the, you

know, the the the brain exercise you get and the

thrill you get from seeing the code execute. Keep that thrill. Keep that spark

alive. Right. Right. Right. Right. But, you know, real

realize there's often more to it. And some of those factors you have no control

over, and the person you're working for has no control over.

And, Ed, you know, there's just a number of things totally external to the experience

of writing code that often

impact the experience of writing code. That's very true.

That's very true. K. Actually, going on for,

like, 90 minutes. So Wow.

Goodness. It doesn't seem like it. I've had this much coffee, and I still don't

have to go to the bathroom. That's amazing. Christmas miracle.

Of you. It's a festive miracle or a Christmas miracle. I don't

know. Awesome. But, so

this is this has been great. I think we kinda got to the bottom

of this is that basically, it's a nuanced conversation.

There's no simple answer. I have many

questions though. Why if you're learning LLMs and now, why are you

calling yourself a data engineer? But Yeah. That's a different thing.

Could just be semantics at that point. Could be.

But thanks, John. Thanks, Merdad.

Thanks, Hector and SQL Dev

DBA. I'm sure that's not the

name on on the driver's license, but

you never know. You never know. I wanna get a license plate holder

that has, like a, like, drop table.

Like, so that way when they

take a picture, they do the OCR. Boom. I've seen

the bumper stickers with longer Right. Right. Right. Right. Right. Secret

hacks on them, you know, secret injection attacks. No. Hey, man. Thanks

for tuning in. And, he legally changed his name. That's

cool. Apparently, there's a number of people

who have, like, a last name, and their

last name is Null. Oh, wow. And I

was looking up on YouTube. Like, there's, like, a number of, like, problems that people

have. And my first thought was, wait. Wouldn't that only be the case if

they encased it in quotes or single quotes? Like, wouldn't it?

Apparently, no. Some of these systems Depends. I mean, you

wanna talk old systems. I'm sure DMVs have some pretty

ancient technologies that are still running.

I mean, I can only imagine. Robert

Tables. Yes. Little Bobby Tables. Little Bobby Tables. That's the

one. I tried to name my kids something like that. But

You got overruled if I remember correctly. I did in so many ways. I'm

glad I didn't go with the, initials of, x

a m l. Yeah. That

that You could have. Xavier Anthony

Marcus was, on the table. See and

Lavinia? That would just Yep. That would just flow. That would be

your role. Though, since XAML kinda died,

it's probably a good thing. That's true.

So XML is even not as in vogue as it once

was. Well, you know our mutual friend, mister Kevin

Hazard, aka the Duke of Hazard. The

Duke. He says he says JSON is just hipster

XML. He's right, though. And there's another format called

JSONL. What? Yeah. JSONLines.

Never even heard of that. It I it's only the

I didn't hear about it until the product I work on, Rel AI, actually

uses it to store our data. And, basically, it's a different

between JSON and JSONL? Basically, it's a long line

where each line is a record of or a chunk of

JSON. That's how I interpret it. I'm sure I'll get

corrected, but please correct me on that one. Interesting. Yeah. I'm like, Jason

now? Like, what the heck?

Jason is an acceptable one. That's true. Jason

would've been a good name. But I didn't want him to get

teased on Friday 13th for the rest of his life.

So so,

thanks everyone for tuning in, man. This was awesome. We should do it more often.

Thumbs up on, or, you know, be sure to like, share, subscribe. I gotta do

this to all the things. Like, share, and subscribe. I can show off,

this little graphic.

You you know, Frank, we could do this, we could do this

a lot more often if there anytime I stir up some trouble, we do a

a live stream Oh, god. We do it every day. I Yeah. That's what I'm

saying. You know? No. I think it's interesting. I think it's good because, like, you

know, it the controversy

I think people are starting I just look last time I looked at the thread

thoroughly, people were talking past each other. Yeah. Which I

guess defines all of Reddit, mostly Internet.

You know, it is what it is. Yeah. But

cool, man. I gotta actually, now that I mentioned it, I do have to go

to the restroom. See. You did it to yourself. I did do it to myself.

And thank you, Miranda, for turning in. And

I will play the outro graphic. Excellent.