Steve Germano of IMEG joins the show to talk about developing an in-house AI assistant named Meg, which functions as a conversational chatbot and search engine for company data running within Microsoft Teams.

We talk about the goals and capabilities of the first release and how the dev team is considering adding workflows to Meg's capabilities. We also discuss the platform they have chosen to build on, what the biggest challenges have been getting to this point, early feedback from staff and leadership, what the potential ROI on a project like this is, and more.

Episode Sponsor:

Chaos

Enscape is soon launching something special that will make your 3D workflow the best 3D workflow for a special price. In the meantime, you can experience it for yourself for free at https://chaos-enscape.com/trial-14 or simply by Googling “try Enscape”.

Episode Links:

• Steve on LinkedIn
• Steve on Twitter
• IMEG website
• IMEG on LinkedIn
• IMEG on Twitter

Watch this episode on YouTube or Spotify.

-----

The Confluence podcast is a collaboration between TRXL and AVAIL, and is produced by TRXL Media.

• Learn more about them at https://trxl.co/media and https://getavail.com
• Learn more about the Confluence podcast at https://confluence.getavail.com/podcast
• Learn about the Confluence events at https://confluence.getavail.com

Welcome to the Confluence podcast.

2

Uh, this is a fun one today, I'll probably

say that about everyone, but we just, uh,

3

I've ended up building a great

network of people that I've met in

4

the industry and it's, uh, just always

a lot of fun to, to have 'em on here

5

and talk about what the exciting

kinds of things that they're doing.

6

But, uh, today we've

7

got Steve Germano uh, I've

8

known Steve for over a decade.

9

Uh, he was at Unifi, uh, uh, ended

up, you know, being a, you know.

10

People probably always think it's weird.

11

It's like, aren't they a

competitor of yours or were they

12

a competitor before Autodesk?

13

And it's like, yeah, but we were all,

you know, in this industry, everybody's

14

friends and we all get along and,

and do a lot of things together.

15

So known Steve for a long time, uh,

he ended up, most recently is he's

16

at an, in a large engineering firm

called IMEG and um, I just noticed, uh,

17

it's probably been about a month ago.

18

He, uh, posted on LinkedIn, a little

clip, uh, of a internal, uh, chatbot AI

19

project that they were doing and kind of

showing off some of the success of that.

20

So I just thought it was really cool.

21

Uh, obviously it's very topical.

22

Uh, this year's, uh, Confluence, uh, three

day event that we did back in October

23

was all around AI and machine learning.

24

So I immediately reached out to

Steve and was like, oh, Steve,

25

this is like the coolest thing.

26

Would you come on?

27

And, and, and you know,

let's talk about, show us.

28

Show us.

29

You know, what, what you've done and,

and then a little bit of the behind the

30

scenes about how all this came about.

31

So I think everybody will enjoy, uh,

he, he agreed to come on and do it.

32

I'm still trying to twist his arm to

get him to come to our, uh, our New

33

York event that we're doing, Confluence

event that we're doing in April.

34

So, uh, hopefully we'll get

him out for that as well.

35

But I think everybody

will enjoy, uh, you know.

36

Hearing this episode and, and hearing him

kind of dig into what's behind, uh, being

37

able to actually, uh, pull off, you know,

the technology, but more importantly,

38

the, the people aspect of it and how

it works inside of a firm like IMEG.

39

Evan Troxel: I think every firm is kind of

thinking about doing this, and so for him

40

to come on the show and show how they've

done it so far, what things to look out

41

for, what platform they're building on

top of all of that was really valuable.

42

And I think people will get a lot out

of this because we all have experience

43

seeing different firms in all the

different departments and how they

44

could potentially benefit from ingesting

their data and then having it answer

45

questions in real time to people who have.

46

Questions.

47

Right?

48

So, uh, I think that it's, it's

really valuable and IMEG is, is at

49

the forefront of this, it seems like,

because they're talking about it

50

and, and showing it off, and, and it

seems like a, a really valuable tool.

51

And, uh, I, I can only IMEGine

the alternative, right?

52

It's like searching the intranet,

looking, going to the HR department,

53

going to the graphics department,

trying to find the right person.

54

And like this actually cuts

down on a lot of that and it.

55

Gives you that single source of truth.

56

But, but him also talking about

accuracy and integrity of the data

57

and bias and all of the things that

are potentially in there as well.

58

And what they're testing against, um,

is, is also the other part of the story

59

that everybody has questions about and

not quite sure how to approach this.

60

And he's giving us some answers

here today, so it's pretty

61

Randall Stevens: Yeah.

62

And I think, you know, Steve has a,

you know, he'll, he'll, when we kick

63

into the episode, he'll give a little

bit more of his background from a,

64

uh, you know, he, he was actually

an engineer, but he's turned into

65

a, being able to write code right.

66

All along the way.

67

So now he's

68

kind of wearing both these

hats and inside the industry.

69

So, um, he's, uh.

70

He's a smart guy, and as you'll

see, uh, his willingness to share,

71

you know, what he's learned.

72

And I think it's just a huge,

benefit to the industry.

73

So appreciate that.

74

So let's just dive right in.

75

uh, Welcome Steve to

the Confluence podcast.

76

Um, for those in the audience that

don't know Steve, um, been an AC

77

industry guy for several years.

78

Steve and I met each other,

probably known each other 10 years.

79

I was thinking back, you know, we,

uh, Steve and his team formerly when

80

he was with Unify, uh, were involved.

81

In helping to get the first

building content summit going, and

82

that's, that's the time I kind of

remember I ran down on the floor

83

at uh, au maybe back in 2013 or 14.

84

And, and, uh, we, we have known

each other at least since then.

85

So, uh, anyway, welcome to the podcast.

86

Um, part of, uh, part of this has been

a series, you know, this, um, we've

87

been talking about a AI and machine

learning and, um, caught my attention

88

A couple of months ago Steve posted

something on LinkedIn that was kinda

89

showing off a new initiative that

he's got at Imec where he works now.

90

So we thought we'd have you on

Steve and kind of talk through

91

what's going on on that front and,

uh, so welcome to the podcast.

92

Steve Germano: Yeah.

93

Thanks guys.

94

Appreciate

95

Randall Stevens: Yeah.

96

So,

97

uh, I, I'll, I guess I'll just kind

of kick off and ask the question.

98

You, you, you posted a video, uh, as I

said, over on LinkedIn and you've got a,

99

a chat bot that

100

you've been developing, your team's

been developing inside of IMEG,

101

which is a large engineering firm.

102

Maybe you can tell a little bit

about what the firm does, but, um,

103

uh, Meg, I think is what you named

your, your bot, which is interesting.

104

We're doing some work here.

105

And actually just earlier today

somebody was asking, it's like,

106

what are we gonna call this?

107

It's like, should we

make it, you know, human?

108

Should it be like it's a human,

or are you talking, you know,

109

generically to something.

110

So we can dive into all these

things and kind of the, the

111

backstory of what's going on there.

112

So maybe you can just kind of kick it off

with a little, a little bit more about

113

yourself and how you ended up at IMEG and,

and the team that you're running there.

114

And, and let's talk about Meg.

115

Steve Germano: Sure, sure.

116

Yeah.

117

So yeah, Meg's, Meg's a a

funny story behind that one.

118

But yeah, so, um, a little bit about me.

119

So I'm originally a, a

mechanical engineer by trade.

120

Um, started my industry or started in

the AC industry for, uh, working for

121

RG Vander Oil Engineers in Boston.

122

And I was actually hired, um, as

a mechanical engineer in their

123

group, but actually specialized

on their CAD department.

124

And so we were doing CAD setup

and all these things for every

125

project and the whole company

while also doing mechanical design.

126

And I was kind of, I.

127

Really just not wanting to

do the CAD side of things.

128

So, um, naturally kind of

gravitated into programming.

129

Had done some programming classes

previously in school and dove in and,

130

you know, six months later or so, kind

of pretty much automated the CAD setup

131

process within the, within the company.

132

Um, and was able to kind of

focus more on engineering side.

133

Um, and while I was at Vanderweil,

uh, actually developed what was known

134

today as Unify, but it was a very

ugly version of Unify back then.

135

It was a, you know, pretty simple UI

and a, and a SQL database just to manage

136

our BIM content amongst our offices.

137

'cause that was just a need

that we found that we needed.

138

Um, and then kind of fast forwarding that

became a business, um, you know, had some

139

great partners and we all kind of built

up this unified product and brand and.

140

You know, just, uh, recently, uh, a year

or so ago, got acquired by Autodesk.

141

Um, and then I had left, uh, unify in

20, I wanna say 20, around around 20, 20,

142

20, 20 21, and went back in the industry,

um, working for MSA, uh, consulting

143

engineers, uh, smaller firm, about

three offices, and, um, and basically

144

was, uh, got back in the industry

as a director of design technology

145

there, and was helping kind of revamp

that business from design technology

146

perspective, IT, and all those things.

147

Um, and then lo and behold, during Covid,

uh, we end up getting acquired by iMac.

148

And I'm like, well, you guys already

have a design tech of technology

149

director, so, uh, where do I go?

150

Right?

151

Um, and so it was a really good fit.

152

We had some, I had some great

conversations, a lot of leadership,

153

and at that point they were, they

had done some programming and had

154

some automations and things going on.

155

A lot of dynamo as every firm has.

156

And they really wanted to dive in and,

and structure a software development team.

157

So I said, Hey, I'm up for the challenge.

158

So I came on board as the, uh, the

head of the software development team.

159

And, uh, we now have a team of,

uh, six engineers today and, um,

160

actively hiring some more as we speak.

161

So it's growing, it's growing rapidly,

and it's been, it's been just an

162

Randall Stevens: Is that,

163

uh, uh, Steve, is that

is that rare to see?

164

I mean, a team of six is a

pretty good team for whatever

165

size you're.

166

Steve Germano: it's, it's, and, and it's

actually larger than that if you count

167

in, so there's six core developers today,

but there's also product owners, right?

168

And so there's, uh, product

owners, and then there's also

169

what we call pims, which are, um.

170

They are product, uh, innovation

managers, kinda a new thing.

171

We just developed, uh, the

beginning of this year.

172

And each department now has a PIM and

all they do is think about workflows.

173

And that's like their core competencies.

174

Like, Hey, how do we do X, y, Z workflows?

175

And then where can we automate this?

176

Where can we share data

with other departments?

177

And how do those workflows and

interactions work with other departments?

178

So they're just thinking about

innovation, which is awesome.

179

So they come up with use cases

and ideas, oh, we need a widget

180

for this, or a website for that.

181

And then those requirements will

kind of flow down to a product owner,

182

and then they'll get more technical

and spec things out and work with

183

myself to, you know, kind of quote

those things and estimate 'em.

184

And then we actually go and develop 'em.

185

So it's a, it's a, there's,

there's kind of a, a big investment

186

from, from I make, I would say.

187

And, you know, their goals

and how, and their growth, uh,

188

their growth has been amazing.

189

And they grow through acquisition.

190

So they're, they're active, very actively

purchasing two to three firms a quarter.

191

Um, sometimes upwards to four

or five, depending on the size

192

and kind of filling in the dots

across the nationwide map here.

193

So when I started, they were

very much Midwest where they had

194

begun, and then they purchased

our firm for the Southwest area.

195

They had some West coast and

they, they added more there.

196

They then added south and Southeast.

197

And then just recently

they've added the Northeast.

198

So we now kind of have a good, um,

nationwide presence and, and they keep

199

growing through acquisitions, so it just

kind of snowball the effect of how many

200

more apps and integrations and things

people need across the organization.

201

So it's been a really, really fun

challenge and great place to work.

202

Randall Stevens: Yeah, that makes sense.

203

So let's, uh, let's dive into the, uh,

this AI bot that you've been working on.

204

Steve Germano: Meg?

205

Randall Stevens: Yeah.

206

So, uh, so,

207

you know, was this something that,

that came through that process from

208

the top down or was this something

you guys were experimenting with and,

209

Steve Germano: No.

210

Randall Stevens: yeah.

211

Steve Germano: Yeah, this

was an interesting one.

212

Um, so.

213

The, the original need.

214

So, so two years ago I was kind

of dabbling around with LLMs

215

when they first came out, right?

216

Da Vinci oh three comes out and

I'm instantly, it, it, you know,

217

it got my interest and I'm like,

we can now turn language into

218

numbers that computers understand.

219

And that was really the big thing about

LLMs is like, Hey, now this sentence

220

means these 1500 numbers, and now I can

compare those numbers with other numbers,

221

which are sentences, other sentences.

222

And it really just, it was awesome.

223

Like I, I just dove in.

224

I started messing with stuff in

my own time, built my own little

225

chat bot way back in the day.

226

And, um, and no real

use case at work, right?

227

It was just, just dibble and dabbling.

228

And then I had a product owner come

to me and he goes, you know, I'm going

229

out, I'm doing these site visits and,

and talking to our, um, you know,

230

per office talking to our offices.

231

And we're doing some

trainings on our Revit apps.

232

And we have a, we have a Revit

app suite of about probably

233

60 different types of tools.

234

Um, a lot of will small, little push a

button, executes some things, uh, from

235

small apps to some pretty larger ones

that do full electrical calculations

236

for entire building, arc flash

calculations, some pretty cool stuff.

237

Um, and, you know, folks

need training, right?

238

So we have these guys

go out and do trainings.

239

And he comes to me, he goes,

Hey, I got done with the training

240

and I'm talking to this team.

241

It's a fairly new acquisition.

242

I think they were about six months,

uh, into the acquisition process.

243

And they didn't know about half the tools.

244

They're like, oh man, I had to do this

whole workflow and it would be great.

245

We had a tool for that.

246

And he's like, oh yeah, we have one.

247

It's right here.

248

And he's like, and then another

person was like, Hey, it'd be

249

great if I had a tool for this.

250

Yeah, we have that.

251

It's right here.

252

So we wanted to, he was like, Hey, can we

just make a search engine so that people

253

can just say, Hey, I wanna do X, Y, Z.

254

Let, let me just find the

tool that we already have.

255

And so that's where it started.

256

So I just like, oh, I could do that.

257

I'll throw that into a LLM.

258

Well, you know, we'll do some AI stuff.

259

No big deal.

260

And it became really easy.

261

It was like a, you know, two, 3000

line of code app, like really small.

262

I think I did it over a weekend just

messing around to see if I could do it.

263

And then I showed him next week

and he was like, oh, that is cool.

264

I said, what else could we put in there?

265

I'm like, oh, oh, okay.

266

Here we go.

267

Right.

268

So, so that, that kind of was

the beginning of it, and it was

269

just a prototype and we were

actively working on the project.

270

So it kind of just stuck.

271

It just sat there for a couple

months and then, um, you know, the,

272

the techs started maturing, right?

273

And then there was that big aha moment

when G PT three came out, right.

274

And Da Vinci oh three came

out and ev and all of a sudden

275

everybody's like, oh my god.

276

Chat bot, chat bot chat bots, right?

277

Uh, opening AI was blowing up, uh,

leadership's hearing about this stuff.

278

And, um, you know, and, and our CIO

and I were having a conversation

279

about, you know, we have.

280

Um, we have a whole data team that

works at IMEG and they do amazing work.

281

They're, they're basically doing a lot

of ETL pipelines, so they're transforming

282

data, pulling it from one database,

putting into another place, merging data

283

together, and building power BI reports

for all these different use cases and

284

people throughout the business, which

every a c firm is doing about these days.

285

And they're extremely busy and

they're always busy, right?

286

They're always new reports and new

data and insights to get from our data.

287

And so we have this great team

that's really actively involved in

288

our data and one of the first things

our CIO did when he came on board

289

was, Hey, we gotta clean our data.

290

We can't really make any insights

in our information unless we have

291

it structured and we know how

to actually go and access it.

292

So I was having a conversation

with him and you know, we've got.

293

Content on SharePoint.

294

We've got database for, you know,

VantagePoint Salesforce, and every firm

295

has 15 different places data lives.

296

And there's no real consolidated location

where we kind of bring that stuff together

297

and then be able to query it, right?

298

Uh, without having to go bother data

team to make me new Power bi, right?

299

And so where I was like, well,

I think I can consume that

300

data into a chat bot, right?

301

I think there's new things coming out

and I'm, I was always reading white

302

papers as soon as they launched.

303

I'm a big white paper nerd.

304

Like yeah, like I need to get a life.

305

But yeah, so I love reading

those white papers, right?

306

So they come out and it's like, oh, we

just developed, you know, uh, I think

307

it was Stanford, um, that developed

the first rag pattern where RAG is

308

a retrieval augmented generation.

309

So the LLMs are, um, when they

first came out, a lot of problems

310

with hallucinations, making up

things they couldn't do math.

311

They're really, really kind of,

uh, not really trustworthy, right?

312

And so this rag pattern came out.

313

It was like, Hey, I wanna ask

questions about this set of

314

information, whether that's a, a

paragraph or a book or whatever it is.

315

And if we took that information, we put

it in line with the user's prompt and the

316

LLM would read the user's question and

see that information and answer from just

317

that information and not hallucinate.

318

And there's some things you, you

know, some, some dials and knobs

319

of turn with the LM to make sure it

doesn't hallucinate when you're given

320

those types of rag instructions.

321

And so that, when that came out, I read

that white paper and I was like, you know

322

what, I think we can actually do this now.

323

Because my primary concern was,

well, I can't release a chat bot

324

that's gonna one, say bad things

to people and get us in trouble.

325

Right.

326

Especially when it's, you know,

so, and, and two, we can't

327

be making up stuff, right?

328

We're talking building

engineering, structural, right.

329

I always used to make a joke

as an HVC guy, if I made a

330

mistake, somebody's hot or cold.

331

If a structural engineer makes a

mistake there, there's, there's

332

bigger liabilities there.

333

So we wanna make sure the data

that's retrieving is accurate, right?

334

First and foremost.

335

And with that pattern, it

opened up that opportunity.

336

So in that conversation, I was like,

I think we got an opportunity here

337

where we can actually execute on this.

338

And we're like, okay,

let's go build a prototype.

339

So we did that prototype

and it just kept going.

340

And now here

341

we are.

342

So.

343

Randall Stevens: really, uh, the, the

moment was the, the data's there and

344

it's really just freeing up the front

end and making it accessible without, you

345

know, having to have a data scientist or

somebody even, you know, even at the level

346

that knows Power bi and it's like, um, you

know, of how to pull all this together.

347

So it sounds like, you know, exciting

opportunities, I think, across the

348

industry to begin tapping into these

data sources like you guys are.

349

What are, what are all the different,

um, what are the different places where

350

the data is living now being sourced?

351

Steve Germano: Yeah,

that's a great question.

352

Um, there's a lot of 'em today and,

um, there's kind of two buckets.

353

If you kind of think about data

in an organization today, there's

354

structured data, which is living

in a database somewhere, right?

355

Um, that could be a SQL database,

that could be via APIs to something

356

like VantagePoint or Salesforce.

357

Um, those data is still in the

underlying locations, the database.

358

Then there's unstructured data,

which is our documents, right?

359

There's, you know, I don't know how

many engineering memos we have on

360

SharePoint or PDFs, you know, code, code

documents, things of that nature, and

361

that, that data's very unstructured.

362

And so how do you get the data from

all these different sources and

363

bring that in was a real challenge.

364

And that was probably the biggest

challenge of this whole tech stack, is

365

not only how do we consume it, right?

366

But then as it updates and updates

live, I don't wanna be asking a

367

question, getting outdated data, right?

368

So we had to build a system and

architect and infrastructure.

369

That as things change live, it can

update that data in real time, so you're

370

never getting out of data information.

371

And that was a real challenge.

372

That was probably the, the most

critical piece to this actually

373

successfully working in the organization.

374

Um, and that involved, you know,

being able to, to get all of our,

375

our information data off SharePoint,

which is mostly the unstructured data,

376

um, as well as tap into, you know,

Salesforce databases and our data lake,

377

which our data team ETLs a lot of API

data together into a single location.

378

Um, so yeah, a lot of different places.

379

Uh, VantagePoint today, uh, employee

directory, which is data structured.

380

Um, you know, all of the,

the SharePoint documents.

381

Uh, Salesforce, a vantage point, and

I think that's about it right now.

382

Uh, oh, project database,

which I think is a mix.

383

And this is another fun part.

384

Half of our data for projects

lived in Salesforce and the

385

other half lived in VantagePoint.

386

And had you meld those two together

with a singular question and

387

had you search across both those

singular questions, it was very

388

complicated structures to the data.

389

So, so again, this, this whole thing

though, with this structured data,

390

would've never been able to, like we

would've never been able to execute

391

it if we didn't have our data team.

392

And, um, one of the things we were able to

do with working with that team is saying,

393

Hey, if we can, I know this data lives

in two or three databases, but if we can

394

make a singular location for that and ETL,

that data in it makes it much easier for

395

us to consume and the LLM to consume it.

396

So we did a lot of work with those

folks and, and really made that, that,

397

you know, that opportunity possible.

398

So.

399

Randall Stevens: You were, uh, uh,

uh, kind of touching on earlier that.

400

Ultimately with these new,

any new technology like this,

401

if people don't trust it.

402

Know we were talking about the

hallucinations and the challenges

403

of, you know, overcoming those

early stage kind of obstacles.

404

And, uh, I'm, I, I'm the same.

405

It's like, I, I don't

want to do something.

406

You only get one bite at the apple

or one swing swing at that, right?

407

It's like, uh, as soon as

somebody has a bad experience.

408

So maybe you can kind of dig in a

little bit about how did you all, you,

409

you had to prove it out to yourself

that it would work, but how did

410

you start to

411

show it to others?

412

What were the data sources?

413

How did you all kind of prove out and

make sure that it, that one you were

414

putting something out there that could

be trusted and wasn't hallucinating.

415

And then I'm sure once people got a taste

of that, they, they were asking right.

416

To kind of broaden the

sources of that info.

417

But maybe you can talk

a little bit about that.

418

Steve Germano: Yeah, the, um, so, so

Meg, it can, Meg basically has access

419

to these different locations for data.

420

So as you ask a question, it goes and

gets it and retrieves it, and then

421

responds in a human-like way, right?

422

And so, um, making sure and how

you do your prompt engineering,

423

that you're not, that the LLM you,

you, you, you make it really safe.

424

So it doesn't hallucinate on that data.

425

Does it make up numbers?

426

Um, that, that's a kind

of a prompt Engineering.

427

Prompt engineering is tricky.

428

It's, it is, right now, there's, there's

continual research being done on it.

429

So almost every week there's new white

papers coming out with new techniques.

430

And so by telling the LLM very,

very kind of simple things like

431

let's think step by step, right?

432

Or, um, you know, no pros, which

don't be verbose, just gimme

433

the information I'm asking for.

434

Those little things in the prompt

engineering make a big difference.

435

Um, but as you start to tweak with

some of the LM settings on temperature

436

and top p and those types of,

you know, technical settings, I.

437

You can basically tell the inference

engine, Hey, don't be creative.

438

There's like a

439

creative dial in there, if you will.

440

Right?

441

And, um, as you kind of tweak

these dials and you do some of

442

this prompt experimenting, you

can find that right marriage.

443

And that is not an easy thing.

444

I'm, I was prompt engineering

something this morning, like,

445

you know, it's a continual thing.

446

And as you start, now that we're into

kind of a, you know, a, a company-wide

447

rollout, we're getting a lot more

analytics so you can start to tweak

448

even a little bit more and fine tune.

449

Um, but that's really the, the, the

key is, you know, can you get your

450

data in one, but then can you make

sure the LLM respond correctly?

451

And if it can't answer the question with

the data you provided, most importantly.

452

Say you don't

453

know the answer, right?

454

So we've taken the approach, and

I'll show you in some examples when

455

we get to it, but we've taken the

approach of, we might not have the

456

information to answer your question,

that would just maybe a knowledge gap.

457

And so our product owners

really focused on that.

458

Now that we have good analytics

coming in, like, oh, this guy asked a

459

question but it didn't have the answer.

460

So that's a knowledge gap.

461

Let's go get our librarians and

let's get that information added

462

to the data sources, right?

463

And so making sure it just says,

Hey, I don't know the answer,

464

but here's what I did find, and

lemme summarize what I found.

465

That seemed to be really

adequate for people.

466

Um, because what they

ask may be very vague.

467

And that's, uh, you know, to get to our

early conversation, uh, before we went on.

468

You know, how do people search?

469

That's a, that's a really ingrained

thing over the past 20 years

470

of working with Google, right?

471

We, we, we search by keyword

types of lexicon search, right?

472

Where it's like one word and add

another word, add another word.

473

We don't typically search in

a conversational way, right?

474

So LLMs work with a conversation, right?

475

So the, you, i, I always say to

our product owner kind of a joke.

476

I say, ask, don't get dumb.

477

Right?

478

Ask smart.

479

Get smart.

480

So if you actually put more information

into your text query, right?

481

As that gets vectorized, there's

more information to compare it.

482

With the associated data to get

more accurate results coming back.

483

So if you put in one or two words and

it's vague, you're not really asking

484

a question, we're doing a search.

485

It's, it's harder for the LLM to

figure out what your intent is.

486

And there's lots of tricks out there.

487

There's like a, hey, well as soon as

a question comes in, pass it through

488

an AI that just does intent and then

pass it to the other things, right?

489

So, lots of tricks out there.

490

I've tried a lot of 'em.

491

We've settled on, um, we have

a couple different ais that

492

are interacting at the time.

493

You do a question.

494

Uh, acronyms is a big hassle.

495

Oh, man.

496

Acronyms, man.

497

Whew.

498

That's, that's giving me a lot of trouble.

499

You know, we are, we

live in acronym soup as

500

architects and Engineers,

501

Randall Stevens: It makes me, uh, think

about just, you know, and I hadn't

502

even started thinking about this,

but you know, how do you write a test

503

plan if you're developing software

with this kind of a front end engine?

504

How do you write a test plan?

505

You'll now, you gotta start to say, what

kinds of questions are people gonna ask?

506

And that's

507

basically infinite.

508

And, uh, so how do

509

you really test this?

510

Have you all started thinking about

that from a, what, what's a test

511

plan look like for, for a bot, right?

512

Steve Germano: We thought about it.

513

We don't know that we have

a great answer for it today.

514

Yeah, it's, I've seen some examples, um,

on kind of the, uh, Microsoft development

515

team that I'm part of that does, uh,

the semantic kernel, which is kind

516

of like their AI engine, if you will.

517

Um, those folks are, are trying to do the

same things we're doing right now, right?

518

Everyone's all in the same boat.

519

Everyone's developing, getting into

production at the same time, and

520

everyone has the same questions.

521

Well, how do we, how do

we do testing of LLMs?

522

Um, there has been some good white

papers put out, and OpenAI released

523

a update to one of their models, um,

I wanna say the November release last

524

year, if I remember correctly, where

you can now pass in a, uh, a grid

525

for all intent and purposes, and it's

basically like an identification token.

526

So when this exact string is asked.

527

And I use this token,

the, here's the answer.

528

So then when you run your test, same

question, same token should get the

529

very same answer or close to it.

530

Um, and that was actually a key because

of this exact question, their developers

531

and open ais, um, you know, third

party developer apps, app developers

532

we're asking for the same thing.

533

How do we guarantee, um, you know,

consistency from question to question?

534

And we still struggle with that, right?

535

Like, you can ask the same question twice

and get, you're gonna get two different

536

answers, but the intent might be the same,

but they're gonna be worded differently.

537

Um, you know, so there, there are

some struggles that, that you are

538

gonna deal with that I don't think the

industry really has a hundred percent,

539

um, you know, kind of worked out.

540

But from a unit testing perspective, using

that key is supposed to help the inference

541

engine stay consistent if you will go

through the same neural network pass.

542

Um, but it's,

543

it's a

544

Randall Stevens: We ran, uh,

545

Steve Germano: a lot of, a

546

Randall Stevens: we ran some of those very

similar kind of tests with some of the

547

work that we're doing here internally.

548

And it was, well, one of my first

questions was like, if you ask, if

549

you feed the same, very same text

string in over and over and over, do

550

you get the same answer back or not?

551

And, you know, you don't, uh, you, you

552

might get nine, nine times out of

10 the same thing and then all of

553

a sudden something a little bit,

uh, you know, varied comes back.

554

So it is.

555

Evan Troxel: There's some interesting

expectations around that, right?

556

Because we, we've, we expect a computer

to answer it the same every single

557

time, but it's actually like you're

asking nine different people nine

558

the same question.

559

And, and it's, it's just like sitting

in architectural design studio and

560

everybody has the same brief and you

get 16 different outcomes, right?

561

It's

562

everybody is applying their,

563

and so it's, it's kind

of like this, right?

564

It's like if it's searching across all

of the possible information that could

565

apply to this, and it's this LLM, right?

566

So it's just looking for patterns in

words and concepts and things that I,

567

I see why it's coming back differently,

but we have this expectation that

568

we're asking a computer something

and that we will always, it's

569

like we expect it to be like math.

570

Right.

571

And not

572

like a conversation because,

573

and this

574

has come up before, right?

575

But I don't know what the

next word I'm gonna say is.

576

I, I don't know.

577

I don't know what you're gonna say next.

578

And you don't know what you're gonna

say next, but you respond in real time.

579

And that's exactly what's

going on with this.

580

And so it's not different, but

we have this, this different,

581

we think differently about it.

582

Randall Stevens: Yep.

583

Steve Germano: Yeah.

584

When this, when this comes up, and it

comes up quite often right now, and,

585

uh, the, the technical term is called,

uh, deterministic or non-Deterministic.

586

Right.

587

And LLMs are non-deterministic.

588

So every time somebody tries to gimme a

bug report, I just, I have a meme say, and

589

I just send it up to 'em, non-determinism.

590

And it's actually from open AI's quote,

what does non-deterministic mean?

591

Right?

592

So it, but it's challenging.

593

So how do you, how, and as a programmer,

we're almost kind of retraining our

594

brains because our goal, our job right, is

595

to get deterministic outputs.

596

Yeah.

597

A hundred percent.

598

So how do you, as a programmer

retrain your brain to say, Hey, you

599

know, now you're gonna start using

AI and, and not just chat bots.

600

Everybody thinks chat bots,

but you can use a AI planner.

601

Right.

602

In a program that may have,

I'll give you a great example of

603

talking about something right now.

604

Um, when I place an outlet, how do I

know what host hosting behaviors should

605

go into in Revit when there's 50 of 'em?

606

Right?

607

Does it go on a wall?

608

Does it go in the floor?

609

Does it go in the ceiling?

610

Does it go to the work plan?

611

Like, there's all these

different scenarios.

612

Well, when you can't code all those

code branches, that's where AI has a

613

great advantage because it's like, Hey,

I can give you the inputs, let you give

614

you instructions on how to determine

your thought process, and then give

615

me what you think is the right output.

616

And that, that is where I feel like

programming is changing right now.

617

Right?

618

Everyone's all enamored with chatbots,

and that's the first, first location.

619

But moving forward, how do we use a

small bit of AI that may never even be

620

seen by the end user, but it's doing

things behind the scenes to just make.

621

Decisions, right?

622

From a, a, a, a very large amount

of potential outcomes, right?

623

That's, there's a lot of ml that can do

that in a mathematical manner, right?

624

When we start talking about ML algorithms

and different things like that.

625

But when you're talking

about reasoning, right?

626

That's a little bit different.

627

And LLMs are great at reasoning,

especially if you give them good

628

instructions on what to reason

over and set parameters coming

629

in and set outputs going out.

630

And so OpenAI wasn't really their

models, uh, kind of, they're

631

always on the forefront, right?

632

They're always like the best

models out, the newest features.

633

They added something last

summer called functional.

634

And with function calling, you can

say, Hey, uh, you can make a decision

635

and call functions based on inputting

parameters, and then those functions

636

can give me back set parameters.

637

So no longer are you trying to

just parse through text strings and

638

figure out numbers and texturing.

639

It's now actually structured data inputs

and structured data outputs that kind

640

of changed the game for programming

where now we can actually use this to

641

make decisions inside a deterministic

type of method that is actually a

642

non-deterministic type of SLU or problem

that you're trying to solve that I

643

can't code 50 different code paths for.

644

Right.

645

So it becomes really interesting

and it, it is tough and I, I talk to

646

even my, my dev team about this and.

647

You know, you almost have to kind

of retrain your brain as to, okay,

648

we're not gonna use AI in this tool or

this new app coming out just for the

649

sake of, Hey, let's just use your ai.

650

You gotta be smart, you gotta

use the right tool for the job.

651

And it's not great for everything, right?

652

You know, square peg and

around a whole scenario.

653

It just doesn't fit everywhere.

654

But as we get, you know, the models

get better, and as we get better, as

655

we learn more, as we retrain ourselves

on how to think, we're gonna find more

656

applications for where an LLM can be used,

where things like stable diffusion can

657

be used, these different bits of, uh, AI

models that are coming out these days.

658

So it's really exciting time.

659

Um, probably the biggest change in

programming I've seen in my entire career.

660

So it's, it's really

661

exciting.

662

Evan Troxel: so is Meg built

on top of open AI's platform.

663

Steve Germano: Yes.

664

So today, um, we have a couple

different models that we use, but we

665

are primarily using open AI's models.

666

Um, and a couple reasons for that.

667

One, they're the most trusted, right?

668

They have uh, they have

a bunch of filters.

669

So if you just go to OpenAI and you

chat today, it goes through a bunch

670

of behind the scenes filters there,

sexism, racism, all the isms, right?

671

Um, it has a political filter,

never to make jokes about politics

672

and all these different things.

673

It's just got a lot of behind the scenes

things because they've come under so

674

much fire and heat since they've started.

675

They have a ton of protections.

676

You could use open source models

for free for cheap, right?

677

That can do about the same performance

wise, but you lose all that.

678

So from an enterprise, right?

679

Hey, money's kind of cheap in

that perspective, where why

680

not pay for the best product.

681

And it's what I've been

talking with other developers.

682

Most people are going that

route just because it's easier

683

and it's safer to today.

684

That may change over time.

685

There's a French model that came out

called Mistral that is, uh, amazing.

686

It's, it's performing better

than GPT-4 in some areas.

687

Um, definitely better than 3.5.

688

And no one's been able to say

that with an open source model.

689

They also have a paid model as well

that's, uh, comparable to GT four.

690

Um, but, uh, you know, the protections

are, they fall a little bit short on the

691

protections right now, but everyone's

692

Randall Stevens: What kind of goes back,

693

Steve Germano: it's gonna be a,

there's gonna be a lot of competition

694

in time.

695

For

696

Randall Stevens: kind of goes back to

that, you know, early stages, obviously

697

with all of this, that you, you don't

want to do anything that's gonna break

698

trust because you, you can, you know, one,

699

one misfire of some

700

type right?

701

Freak freaks people out and you just wanna

try to avoid it if you can, especially.

702

Steve Germano: Yeah.

703

I mean, especially a

company wide chat, right?

704

Uh, it makes one.

705

You know, racial slur or something,

you're getting sued, right?

706

You gotta be really

careful with these things.

707

That was my biggest concern.

708

I was, you know, two years

ago, I'm like, it's not ready.

709

It's not ready, it's not ready.

710

And then finally I'm like, okay,

I think we're, we're there, right?

711

The tech's catching up.

712

Um, and that's kind of cool right now

it's like all, you wanna be bleeding

713

edge, but you don't wanna be too

bleeding edge because, you know, one,

714

you can waste a lot of investment.

715

Right?

716

Um, and we were kind of really, um,

intricately thinking about exactly when is

717

the tech gonna catch up in different areas

for us to release different features.

718

And so we have some new plugins

that are on hold right now.

719

We're kind of waiting for

some things to catch up.

720

Um, but there's just, there's a

lot of movement going on right now.

721

And even, even Microsoft themselves are,

they're launching their co-pilots in

722

all their product stacks and, uh, you

know, they've got issues too, right?

723

So I, I was just watching yesterday,

um, someone demoing the preview

724

version of the co-pilot in Excel.

725

And, uh, someone had asked me in my

firm, well, how are they doing that?

726

And we're not doing

Excel stuff today, right?

727

And I'm like, eh, I don't know.

728

That's gonna be really feasible today.

729

And so, and, and I'm watching this

video and this guy's got a million

730

rows of data in Excel file, right?

731

And the co-pilot chokes and just

completely locks up and crashes.

732

Then he goes, okay, lemme

try it with less data.

733

He goes down to half a million rows,

crashes, a hundred thousand rows,

734

crashes, goes down to 5,000 rows crash.

735

He had to get all the way down to

400 rows before it would actually

736

be able to talk over that data.

737

So, so there's just a lot of, I mean,

if Microsoft, you know, has obviously

738

more resources, they, if they're

still struggling with some of this

739

tech and how to achieve it, right?

740

Probably good to go put that on hold

for a little while and we'll come

741

back to Excel files later, right?

742

So, uh, it's just being smart about where

you kind of think the industry is and kind

743

of, you know, just, just being tuned in

with things so you don't forge too far

744

ahead and waste time and money and effort.

745

Randall Stevens: Yeah.

746

Well, I think,

747

Evan Troxel: Maybe before you just

jump into, into whatever we're gonna

748

do next, I'm hoping, I want, I want

to take a look at this, but I'm just

749

wondering how much of the sh has,

has your team shifted from before two

750

years ago to what you're doing now?

751

How much has it shifted

to ai, uh, focus on that.

752

Steve Germano: Um, not

much to be honest with you.

753

Not, not much outside of the MEG product.

754

That's really the only product where

we've actively integrated AI today.

755

Um, actually, no, I'm sorry.

756

There is one more, but it was a

legacy project that our data team's

757

working on and that's doing, you

know, points in buildings for, um.

758

Um, it's more natural language

processing and it's for points and

759

buildings for, uh, control systems.

760

So, so those are really the only

two projects I know of today.

761

They're actually utilizing any

type of ai, if you will, with LLMs.

762

Um, and so like we have other projects

that are act inactive development that

763

we will be adding some elements into, but

I don't think there'll be public facing.

764

There'll more be, like I was

saying before, behind the scenes

765

to do some decisions and things.

766

So, um, so yeah, I guess not a ton today.

767

Everyone's conscious of it, but

we gotta have the, it is gotta be

768

the right tool for the job, so,

769

Evan Troxel: I appreciate you saying

that, just because I think, because

770

we're talking about this topic, the,

the, the mind immediately goes to

771

like, this is everything and this is

all that we should be focusing on.

772

And, and so it's, it's, uh,

important I think to hear what

773

the answer to that question.

774

Steve Germano: Yeah, cer certainly not,

I mean, that's a common trap, right?

775

Of, of development teams.

776

They find this new shiny object

and they're like, oh man,

777

I want to go work on this.

778

So they can learn it and, and

it might not just, might not

779

be the right tool for the job.

780

So you gotta be really smart with that.

781

Randall Stevens: so do you get to,

782

can you show us, can you

783

show us something?

784

Yeah,

785

let's see

786

a

787

Steve Germano: yeah, absolutely.

788

so this is Meg, and you can see

it's kind of my chat history.

789

Been asking her some

questions here and there.

790

Um, let's ask, we've got some questions.

791

I'll just copy and paste in here.

792

So one of the things

793

Evan Troxel: say her, you say her.

794

I just wanna point out that

like you are personifying

795

this and, and I'm just

796

wondering is it natural?

797

Like, is

798

that, was that

799

important?

800

Is that

801

something that comes up

802

in conversation like

culturally in the firm?

803

Steve Germano: it's a, it's a funny thing.

804

So I'll tell you the backstory.

805

Um, there was no name and I had

somehow in my early version of

806

it, I had named it Megatron.

807

Right.

808

Just from, you know, being a,

being a, a nerd there, right?

809

So, and then when we sent off some

information to marketing, like, can you

810

give us a logo or, you know, I, I, there

was no persona, there was no Meg name.

811

Our market, one of our marketing

people came up with it and just

812

like, Hey, what do you think of this?

813

This is a first iteration,

only one iteration.

814

And we're like, it's better than Megatron.

815

And, and then it just took

on a life of its own right.

816

And now people personify, hi, thanks Meg.

817

That, you know, they're talking to it.

818

And the analytics, it's really

funny to watch the analytics.

819

'cause you can see people have

conversations, which is really cool.

820

They're appreciative when they

get an answer and they say

821

Randall Stevens: We're nice to it.

822

Steve Germano: you're welcome.

823

Let me know.

824

Yeah.

825

It's really, it's

826

Evan Troxel: not the ones who are

gonna go kick the robot because

827

they know the robot will come back.

828

Yeah.

829

Steve Germano: It's, it's a smart move.

830

I'm very polite to make.

831

Evan Troxel: Right.

832

Steve Germano: Yeah.

833

Um, so, so you can ask, uh, you can

just start typing and ask a question.

834

Um, in the, in the, there's two

ways to kinda interact with it.

835

You can just type, like, you

can see here, can you explain

836

ary ventilation six 2.1, right?

837

Um, or you can call a plugin directly.

838

And so what we did was we

built a plugin architecture.

839

And those plugin architectures are for

different purposes, like our corporate

840

directory, our corporate policies, right?

841

It's all of our bucket of data for

corporate information and health

842

benefits and all those things.

843

Um, our directory is kind

of like, you know, employee

844

directory, those types of things.

845

Project databases, token's.

846

Really interesting.

847

Uh, token and VDC probably two of

my favorite, um, plugins because

848

there's so much data there.

849

Gigs and gigs and gigs

of unstructured data.

850

Um, VDC is just all of

our Revit knowledge.

851

Just, you know, uh, it's all

currently lives in like a, um.

852

Uh, it lives in a massive, uh, OneNote

notebook that all the BDCs use across

853

the company, and they search in there

and, oh, search, it's okay in there.

854

It's not too bad, right?

855

In OneNote.

856

Um, but this is just much

faster, much easier, right?

857

And because that OneNote got

so big, there's hundreds and

858

hundreds of sections and pages.

859

It, it can be complicated to find stuff.

860

And then Token is uh, what we call

our tech Ops Knowledge network.

861

And this is a location on SharePoint

with just thousands and thousands

862

of design memos, code change

announcements, all the things that

863

our engineering folks, um, need to

get out there to, to our engineers.

864

Um, and this could be hey scenario

situations where, Hey, I know local

865

jurisdiction code calls for this and

this and this, but we recommend upsizing

866

because of x, y, Z scenario, right?

867

Lessons learned from, from

building engineering designs

868

and things of that nature.

869

So there's a plethora of our

most senior engineers brains.

870

Focused in that area of unstructured data.

871

Um, so it's really, really exciting

to kind of have that level, level

872

of information available to anybody.

873

Randall Stevens: Steve, a a quick,

874

Steve Germano: so you could fire,

875

Randall Stevens: oh, I'll

876

ask, just ask

877

a quick question along those lines, I'm

sure you know, inev, inevitably there's

878

gonna be some data that's either out

of date or potentially wrong, right?

879

Even, you know, there's always

bad information somewhere.

880

Have you all, uh, have you put in

any kind of a feedback loop where

881

if, if, if that information is

identified, it can be flagged and

882

then the system can learn from that?

883

Steve Germano: That's a great question.

884

That's a great question.

885

Um, so we use a, for all of our

software, we use a feedback board, right?

886

Where folks can go in there,

report bugs, you know, report,

887

um, different feature requests.

888

We don't have something live iterative

in here with the exception of, excuse me.

889

Um, anyone can get a good

answer and then hit I.

890

Thumbs up or love, you know, these are

the first two things that kind of pop up.

891

And those positive sentiments are what

we'll use in the future to kind of do

892

some fine tuning of the model, right?

893

Like, hey, you've answered

some great things.

894

Here's this.

895

Uh, we don't have a negative

sentiment in there today.

896

They have to go to can, uh, to our

feedback board and actually submit that.

897

So, um, but this is still in beta and

that's something we're considering.

898

You know, can I tell Meg, hey, that

previous answer was actually outta

899

date and, you know, submitted as a

bug report or something like that.

900

Um, those are all certainly, uh, possible.

901

So I'll show you a really easy plugin.

902

So we just typed help.

903

This is kind of like our initial help

plugin where this comes in really fast.

904

It's static text.

905

It's not actually using any AI

or anything, but it just kind

906

of gives people, like, first

thing people are always gonna

907

ask how do you use stuff, right?

908

So it gives 'em some examples

of what to search, how to

909

search, those types of things.

910

You know, pretty, pretty standard stuff.

911

Um, if I come in here and

let's ask, you know, how do I

912

submit, um, an expense report?

913

So this will go and hopefully get

us to the right location for data.

914

So that's the first decision point, right?

915

Which location bucket is the right one

to go to based on user's question, right?

916

So it'll say, Hey, I'm

answering from this plugin.

917

And then you can see it's,

it's streaming that information

918

that is coming from our data.

919

So this is not an LLM making anything up.

920

This is not an LLM saying what it

thinks it is, it's give being provided

921

data and regurgitating that and maybe

massaging it based on how the LLM would

922

like to, you know, predict the next word

based on the, the, the neural network.

923

So, and then you have, uh, uh, you

know, links to the source documents.

924

So what's kind of cool here, and what

was really complicated for us to figure

925

out was, well, if there's five answers

to a question or a additional data from

926

these, all these different documents, I.

927

Like, I wanna get as much information

to you to answer your question, but I

928

wanna separate that information out.

929

And how do I, how do I let that user know?

930

So we, we provide all the additional

related links in here that may be relevant

931

to that user's question, but the LLM

has the capability through instruction.

932

It's told it's okay to

add a consolidated answer.

933

So not just one answer rules them

all, and I just pick the first top

934

search result and that's what I use.

935

No, you can make a more concerted

effort to be a bit more cohesive.

936

Right?

937

And a bit more

938

Randall Stevens: I think, uh, I

939

think referencing back to the, the

sources of the information is an important

940

piece of, of building that trust too.

941

It's like in these early stages when

you get an answer back, it's like, well,

942

here's the answer, but here's the source.

943

Right?

944

It's like you're citing the sources.

945

So,

946

Steve Germano: a hundred percent.

947

A hundred percent.

948

So we didn't do this in

the early alpha testing.

949

Um, the first version of this that

saw, you know, public users and

950

that was the first thing they said.

951

Hey, that's cool,

952

but I need to see where it's from and I

need to go validate, you know, trust yet.

953

Verify.

954

Exactly, exactly.

955

And, and it's funny 'cause I've had that

same conversation with other developers

956

who are doing the same thing in the

enterprise for different enterprises,

957

not just AC sector, but others.

958

And that's the first thing they

have to tackle is, hey, people

959

wanna know where it's coming from.

960

They wanna validate that information.

961

And so we

962

Evan Troxel: though, that.

963

that's coming from like huge OneNote

documents or a a SQL database.

964

So when you click on a link, like

where does it actually take you?

965

Steve Germano: Um, they

could be various locations.

966

So like this, this is a SharePoint,

so this is in some SharePoint

967

folders, somewhere that looks

like finance and accounting.

968

Um, so, so these can all be

in different locations across

969

different sectors of the business.

970

Um, but this particular plugin, most

of its data is, is uh, you know,

971

unstructured content and SharePoint.

972

Evan Troxel: So will it take you to a

specific location in a huge OneNote or

973

will it just take you to the, the general

folder and then you have to search

974

Steve Germano: Uh,

975

no, this will actually, we

have it set so it'll actually

976

open the file directly right

977

Evan Troxel: Nice.

978

Randall Stevens: link.

979

You can, you can build like deep links

into these documents and stuff, right?

980

Anchor

981

Steve Germano: yeah.

982

Correct.

983

Yeah.

984

Most of the documents aren't built

like that because they're, they were

985

just word docs that people have made.

986

Um, but you can do that, correct?

987

Evan Troxel: Hmm,

988

Steve Germano: So I'm gonna, I'm gonna

show you a little bit different one.

989

Um, so this one is actually going

to look through structured data.

990

So we just talked about unstructured.

991

Now we're gonna go through structured

structure is a lot tougher to get right.

992

And, um, this one is actually

doing some database searching.

993

It's, you know, AI writing SQL queries

and answering SQL queries and all

994

these really complicated things.

995

That's really, really tough to

get consistently accurate because

996

while AI can code really well,

they don't understand your data

997

structure and there's a lot of tough.

998

Uh, things to kind of, uh, uh,

there's a lot of challenges in

999

there that a lot of developers are

having, um, to really kind of think

00:44:48,850 --> 00:44:50,500

through and how to solve right now.

00:44:50,800 --> 00:44:53,050

And, uh, we are really lucky.

00:44:53,140 --> 00:44:54,730

Like this is a very hard thing.

00:44:54,730 --> 00:44:59,170

If I didn't have the collaboration

with our data team to say, Hey, let's

00:44:59,170 --> 00:45:05,740

massage the data side to work better

with our, our product, it would've

00:45:05,740 --> 00:45:07,270

been almost impossible, right?

00:45:07,390 --> 00:45:10,780

So if somebody said, Hey, just go build

this thing for, uh, you know, Johnson

00:45:10,780 --> 00:45:15,220

Controls and here's their database with,

uh, you know, 5,000 columns of data,

00:45:16,180 --> 00:45:18,010

that's gonna be really difficult, right?

00:45:18,100 --> 00:45:22,480

LMS get confused really easily, especially

when it comes to structured data.

00:45:22,750 --> 00:45:26,260

So we were able to kind of get a good

marriage there to give it just the

00:45:26,260 --> 00:45:27,940

right of data, not too much data.

00:45:28,450 --> 00:45:32,740

And keep it really focused on the

data we want it to answer from.

00:45:32,770 --> 00:45:35,380

So it doesn't even have access

to a whole bunch of other stuff.

00:45:35,440 --> 00:45:35,860

Right.

00:45:36,130 --> 00:45:38,440

And it only has access to

things that we want it to court.

00:45:38,590 --> 00:45:40,210

There's no social

security numbers in here.

00:45:40,210 --> 00:45:41,890

There's none of that kind

of silly stuff, right?

00:45:42,160 --> 00:45:46,450

Um, so it's very, very, um, you know,

restricted as to what it can have and,

00:45:46,455 --> 00:45:51,460

and, you know, you don't want it to have,

uh, to go crazy and delete tables, right?

00:45:51,460 --> 00:45:53,560

So it's where you only access all

those types of protections you

00:45:53,560 --> 00:45:54,820

just gotta kind of think through.

00:45:55,150 --> 00:45:58,210

But you can see here it's, it's

actually doing inquiries to the point

00:45:58,215 --> 00:46:00,430

where it can summarize and quantify.

00:46:00,640 --> 00:46:02,920

So it can tell you, Hey,

we've got 71 people in the

00:46:02,920 --> 00:46:04,900

Las Vegas office today, right?

00:46:05,110 --> 00:46:09,130

Uh, one I really like, uh, we just added

this functionality through analytics.

00:46:09,130 --> 00:46:14,800

We saw people were asking, um,

let's see, how many, uh, how many,

00:46:14,800 --> 00:46:21,155

let's see, licensed mechanical

engineers, uh, are in the state.

00:46:22,810 --> 00:46:24,610

Uh, Nevada.

00:46:25,090 --> 00:46:26,530

Hopefully this works, but should work.

00:46:27,010 --> 00:46:31,960

So this was something we didn't really

have this level of data exposed and

00:46:31,990 --> 00:46:34,960

oh, so you see that one's actually

answering, I think from the wrong plugin.

00:46:34,960 --> 00:46:37,180

It should be answering from

DURs, so I'll do that again.

00:46:37,510 --> 00:46:42,340

Um, but it's, it should be

searching through a new set of

00:46:42,340 --> 00:46:46,030

data that we added in and then

answering and quantifying from that.

00:46:46,330 --> 00:46:50,290

So, so this is one of the challenges

right now is picking the right plugin.

00:46:50,290 --> 00:46:53,590

So you'll see here it's actually

answering from our token library when

00:46:53,590 --> 00:46:56,260

it really shouldn't be, should be

answering from our directory library.

00:46:56,650 --> 00:46:59,740

So since it stops responding, I'll

go ahead and, and ask that again.

00:46:59,960 --> 00:47:03,845

Evan Troxel: While it's thinking about

this, uh, can you talk a little bit about,

00:47:03,845 --> 00:47:08,945

you talked about, uh, IMEG does a lot

of acquisition and so you're acquiring

00:47:08,975 --> 00:47:10,775

these other firms that I'm sure have.

00:47:11,080 --> 00:47:12,280

It's all over the map.

00:47:12,280 --> 00:47:16,930

The level of sophistication of their

databases and where their data is

00:47:16,930 --> 00:47:19,660

stored and how it's stored and if

it's structured or if it's not.

00:47:19,665 --> 00:47:24,040

So is your data team responsible for

ingesting that and figuring out the

00:47:24,040 --> 00:47:25,900

best way to bring all this together?

00:47:26,320 --> 00:47:26,590

How?

00:47:26,590 --> 00:47:29,050

How are you dealing with

that in an ongoing basis?

00:47:29,770 --> 00:47:31,030

Steve Germano: Yeah,

that's a great question.

00:47:31,030 --> 00:47:36,730

So we have a, um, a full team that

does the, um, integration of new

00:47:36,730 --> 00:47:41,350

acquisitions and they have kind of, um,

a graduation process, which can take,

00:47:41,355 --> 00:47:43,060

I believe around a year to two years.

00:47:43,420 --> 00:47:47,770

And so throughout that process, right,

they're transforming or they're getting

00:47:47,770 --> 00:47:51,550

onto our networks, they're getting their

data transformed into our data sets.

00:47:51,790 --> 00:47:54,040

They may have software that's

really useful for us that

00:47:54,045 --> 00:47:55,480

we may take and integrate.

00:47:55,660 --> 00:47:58,030

Um, they're getting trained on our

existing software, so there's a

00:47:58,030 --> 00:47:59,920

full team that does that process.

00:48:00,250 --> 00:48:02,980

And throughout that process is

when we'll do that evaluation.

00:48:03,010 --> 00:48:06,520

And we haven't had to do too much of

that because this is such a new product.

00:48:06,805 --> 00:48:10,975

But as we consume their information,

the data team typically figures

00:48:10,975 --> 00:48:15,505

out how to get them onto our

vantage point, our Salesforce.

00:48:15,505 --> 00:48:19,525

So we shouldn't have to, um, they, they

do have to consume it and ingest their,

00:48:19,525 --> 00:48:23,245

their previous projects and all that data,

but that shouldn't really affect, like,

00:48:23,250 --> 00:48:27,505

I wouldn't have to do any coding changes

on our team side because this knows

00:48:27,505 --> 00:48:29,575

how to talk to that data set already.

00:48:29,875 --> 00:48:32,275

So it becomes a little bit

simpler from that perspective.

00:48:32,425 --> 00:48:37,015

But I will say on the acquisition side,

this is a really valuable tool for that.

00:48:37,465 --> 00:48:42,325

And the reason is, um, we, we've

been seeing people searching just,

00:48:42,565 --> 00:48:45,595

hey, gimme contact information

for this, this, and this person.

00:48:45,595 --> 00:48:47,905

And they don't know where they

have located, but they need that

00:48:47,905 --> 00:48:50,185

information to go put into some

other workflow they're doing.

00:48:50,185 --> 00:48:50,635

And we had this.

00:48:51,150 --> 00:48:53,635

Just this week we had somebody who

was doing this over and over and over.

00:48:53,635 --> 00:48:58,225

So our PO reached out to 'em, Hey, why do

you keep searching for A, B and C people?

00:48:58,435 --> 00:49:01,255

And you know, and well she's like,

oh, I'm doing this and I'm copy

00:49:01,255 --> 00:49:03,505

and pasting it into here 'cause

it has to go in this report.

00:49:03,835 --> 00:49:05,635

And we're like, oh, interesting.

00:49:05,640 --> 00:49:10,525

So these emergent features kind of come

out and we don't know all the workflows

00:49:10,525 --> 00:49:12,115

people are gonna use today, right?

00:49:12,115 --> 00:49:14,005

We don't know all the

questions people are gonna use.

00:49:14,185 --> 00:49:18,805

So having that non-deterministic method

for them to search whatever they want

00:49:19,075 --> 00:49:23,575

becomes a really big value, especially

for new acquisition folks, right?

00:49:23,575 --> 00:49:27,115

Someone comes on the company,

who do I talk to for HR issues?

00:49:27,145 --> 00:49:28,345

Who do I talk to for this?

00:49:28,375 --> 00:49:30,115

Hey, who's the CE in this office?

00:49:30,115 --> 00:49:31,375

Or who's the licensed engineer here?

00:49:31,375 --> 00:49:34,285

'cause I think we got a bid on this

project and I don't even know if we

00:49:34,290 --> 00:49:35,605

have licensed engineers in the state.

00:49:35,935 --> 00:49:40,435

Those are the types of things that we

try to make this system help them with.

00:49:40,975 --> 00:49:45,085

Um, so, so this question finished, which

was not the right plugin, but um, and then

00:49:45,085 --> 00:49:48,685

this one here actually fired the directory

plugin directly and said We have 13, you

00:49:48,685 --> 00:49:50,455

know, engineers in that, in that state.

00:49:50,455 --> 00:49:53,695

So, and I could list them out, but I don't

wanna put people's information out here.

00:49:53,695 --> 00:49:56,245

But, um, so here's another really fun one.

00:49:56,245 --> 00:49:59,575

So let's do, let's do more

engineering question, right?

00:50:00,895 --> 00:50:06,235

So here's one of how do I size

a, a wire for 50 horsepower

00:50:06,235 --> 00:50:08,335

motor, um, on mechanical.

00:50:08,335 --> 00:50:11,875

So I wouldn't know off the top of

my head, but, uh, there's a lot of

00:50:11,875 --> 00:50:13,675

elec electrical engineering data.

00:50:13,675 --> 00:50:15,775

There's some NEC code data.

00:50:15,955 --> 00:50:20,125

So a lot of this information can be kind

of put together in a cohesive answer.

00:50:20,130 --> 00:50:22,735

So you'll see it's Andrew from

a couple different sources.

00:50:22,735 --> 00:50:25,855

Elevator design, guide pump,

you know, fire pump design

00:50:25,855 --> 00:50:28,165

guides, VFD bearing, pitting.

00:50:29,365 --> 00:50:32,785

I don't personally know what all these

documents are, but the information

00:50:32,785 --> 00:50:36,325

lives in that document and it's able

to regurgitate it and find what's

00:50:36,330 --> 00:50:38,755

relevant in all those documents

and bring it to the forefront.

00:50:39,175 --> 00:50:40,570

So it's, it's really cool the way

00:50:40,660 --> 00:50:43,390

Randall Stevens: that kind of makes me

think, Steve, uh, you know, some of the

00:50:43,390 --> 00:50:47,245

first experiments that I was just using

Chet GPT, you know, a year or so ago,

00:50:47,595 --> 00:50:50,045

experimenting with, and I would ask.

00:50:50,665 --> 00:50:53,605

Uh, I would ask, you know, I was

kind of testing what you were doing

00:50:53,605 --> 00:50:57,445

there, some technical questions,

and it would give me answers back.

00:50:57,445 --> 00:51:00,235

And I was like, well, I don't even,

I, I don't even know if that's the

00:51:00,235 --> 00:51:01,615

right answer or not the right answer.

00:51:01,615 --> 00:51:02,575

So it was back to that,

00:51:02,575 --> 00:51:04,105

you know, hallucinations.

00:51:04,705 --> 00:51:05,965

And I, I think you're

00:51:05,965 --> 00:51:06,265

Right,

00:51:06,270 --> 00:51:09,775

about the, you know, if you can,

you can say, just be verbose.

00:51:09,775 --> 00:51:13,525

Don't, you know, give me, give me

just the short version of this, and

00:51:13,525 --> 00:51:16,765

then if you don't know, tell me you

don't know, instead of just making

00:51:16,765 --> 00:51:17,455

something up.

00:51:17,455 --> 00:51:17,905

And, uh,

00:51:18,295 --> 00:51:20,935

some of the experiments that we've been

running, you know, we've been using

00:51:20,940 --> 00:51:25,255

those kinds of approaches to it, and

it's like, just get me, especially

00:51:25,255 --> 00:51:30,055

if you're asking it technical, not

asking you to write a, uh, a flowery,

00:51:30,115 --> 00:51:35,305

uh, uh, you know, cv, you know,

description of me for some, uh, you know,

00:51:36,295 --> 00:51:36,925

for, for some

00:51:36,925 --> 00:51:38,665

description of my past histories.

00:51:39,235 --> 00:51:42,625

So I, I want real data, real answers.

00:51:42,630 --> 00:51:44,275

I don't want any fluff in there.

00:51:44,275 --> 00:51:48,475

And to make sure that these, because

I think this is an audience, you know.

00:51:48,790 --> 00:51:53,680

That, that ultimately, I bet, I bet

within IMEG, a lot of the engineering,

00:51:53,950 --> 00:51:57,370

you know, people with an engineering hat

on, it's like, it's kind of fun to play

00:51:57,370 --> 00:51:59,920

with, but I want real data, real answers.

00:52:00,040 --> 00:52:00,700

No fluff.

00:52:01,210 --> 00:52:01,510

Let's go.

00:52:01,970 --> 00:52:02,250

Steve Germano: percent.

00:52:02,250 --> 00:52:02,930

Hundred percent.

00:52:03,440 --> 00:52:04,290

Like we've already,

00:52:05,190 --> 00:52:05,770

oh, sorry.

00:52:05,770 --> 00:52:06,410

Go ahead, Evan.

00:52:06,475 --> 00:52:08,785

Evan Troxel: I was just gonna say

that the prompt engineering is

00:52:08,785 --> 00:52:10,405

changing all the time as well.

00:52:10,405 --> 00:52:14,515

So if you teach somebody how to do

it today, it's gonna be different a

00:52:14,520 --> 00:52:17,455

year from now or, and probably in a

lot shorter amount of time than that.

00:52:17,455 --> 00:52:22,435

But like for example, mid journey five

to mid journey six changes changed

00:52:22,435 --> 00:52:23,935

prompt structure significantly.

00:52:23,935 --> 00:52:25,225

They made it a lot simpler.

00:52:25,810 --> 00:52:30,730

You can enter a much simpler prompt

and still get really amazing results

00:52:30,730 --> 00:52:32,920

now with, with six versus five.

00:52:33,340 --> 00:52:37,900

And, and so I think that's another

interesting point to make about this,

00:52:37,900 --> 00:52:40,900

is like there might be things you need

to include in your prompts today that

00:52:40,900 --> 00:52:42,670

you maybe won't need to include later.

00:52:42,670 --> 00:52:48,220

And it's just keeping everybody educated

on the best way to prompt these systems

00:52:48,370 --> 00:52:50,320

is a moving target all the time.

00:52:51,250 --> 00:52:51,540

Steve Germano: Yeah.

00:52:51,655 --> 00:52:55,465

And you know, the problem to engineering

happens behind the scenes, right?

00:52:55,465 --> 00:52:57,775

In our plugins and all these things.

00:52:58,015 --> 00:53:02,245

And we noticed, we just went to, um, you

know, the newest version of G PT four

00:53:02,245 --> 00:53:08,485

Turbo and from GT four to GBT four Turbo,

the prompts change from 3.5 to four.

00:53:08,515 --> 00:53:09,355

The prompts change.

00:53:09,360 --> 00:53:10,705

So we were like, all right, we're ready.

00:53:10,705 --> 00:53:11,845

We're gonna go with faster model.

00:53:11,845 --> 00:53:13,180

We upgrade this thing,

and all of a sudden.

00:53:13,750 --> 00:53:15,640

Whoa, it's not responding the right way.

00:53:15,640 --> 00:53:16,900

It was before like, what's going on.

00:53:17,080 --> 00:53:19,930

So you have to expect that

because the inference engines are

00:53:19,930 --> 00:53:21,250

just natively different, right?

00:53:21,250 --> 00:53:22,120

With these lms.

00:53:22,155 --> 00:53:22,375

so

00:53:22,660 --> 00:53:24,610

so that's, you know, a

little bit of a risk.

00:53:24,610 --> 00:53:27,220

It's like, hey, let's go this faster,

better, smarter, cheaper model.

00:53:27,250 --> 00:53:31,000

But you do have to put some investment in

there and gets back to the unit testing.

00:53:31,000 --> 00:53:32,170

Well how do we batch test it?

00:53:32,200 --> 00:53:35,230

How do we, you know, so there,

there is a lot to consider with

00:53:35,230 --> 00:53:37,540

that, um, as the technology changes.

00:53:37,930 --> 00:53:41,200

And also if you wanna start

exploring things like, hey, let me

00:53:41,200 --> 00:53:44,140

go test a open source model, right?

00:53:44,410 --> 00:53:48,130

Hey, I can host my own model and I

can cut out these open AI costs and I

00:53:48,130 --> 00:53:51,280

can go, you know, put a Mistral model

in here, or LAMA two model and fine

00:53:51,280 --> 00:53:52,300

tune or whatever the case may be.

00:53:52,600 --> 00:53:53,650

Yeah, you can do that.

00:53:54,025 --> 00:53:55,525

What do you lose with that process?

00:53:55,585 --> 00:53:55,795

Right?

00:53:55,795 --> 00:53:58,855

You might save some money, but

you lose all those protections.

00:53:58,855 --> 00:54:01,615

You open yourself up a liability

potentially of this thing going off

00:54:01,615 --> 00:54:03,475

the rails and Meg having a bad day.

00:54:03,775 --> 00:54:07,255

Or maybe you get hallucinations now

because it doesn't know how to handle

00:54:07,255 --> 00:54:09,475

the same prompting to keep it in line.