Shownotes
Hello and Welcome to Data Driven.
In this episode, Frank and Andy speak with researcher Matteo Interlandi about project Hummingbird.
Audio file
Transcript
00:00:00 BAILey
Hello and welcome to dated driven.
00:00:02 BAILey
In this episode, Frank and Andy speak with researcher Matteo Interlandi about project Hummingbird.
00:00:09 BAILey
Now on with the show.
00:00:10 Frank
Second, hello and welcome to data driven.
00:00:21 Frank
The podcast where we explore the emerging fields of data science, machine learning and artificial intelligence.
00:00:27 Frank
If you'd like to think of data as the new oil, then you can consider us.
00:00:30 Frank
Car Talk because we focus on where the rubber meets the virtual road and with me on this epic Rd.
00:00:36 Frank
We're on the information superhighway as oh is Andy Leonard.
00:00:39 Frank
How you doing Andy?
00:00:40 Andy
I'm well Frank, how are?
00:00:41 Frank
You I'm doing alright. We're recording this on Wednesday, September 1st, 2021 and the the.
00:00:51 Frank
The the remnants of Hurricane Ida are ripping through the DC area.
00:00:57 Frank
Uh, so if, uh, if I suddenly get dropped, that's because we probably lost power.
00:01:03 Frank
But I do have the backup generator, the one that the professionals installed and my.
00:01:10 Frank
Duct taped together a solar generator so.
00:01:15 Frank
I will be offline.
00:01:17 Frank
For a short.
00:01:18 Frank
Bit and hopefully come back online.
00:01:20 Frank
How how you doing, Eddie.
00:01:23 Andy
I'm doing alright Frank. Well, we are you know I'm about gosh 250 miles South of UM we didn't get near the near the effects of Hurricane Ida as you did.
00:01:34 Andy
We're getting a little bit of rain now.
00:01:36 Andy
We've had some wind.
00:01:37 Andy
Gusts, but it's been really mild, and if you look on the radar.
00:01:41 Andy
Gotta watch it into track and I I do.
00:01:43 Andy
I'm a weather weenie and amateur but it it just kind of went around us to the to the West and it actually started the east when it got a little north of us and aimed right for your house.
00:01:54 Andy
I was looking outside that's where Frank lived, right?
00:01:56 Andy
And look, the eye is coming right for.
00:01:58 Andy
Frank what's left?
00:02:00 Frank
Well, fortunately we're safe.
00:02:02 Frank
There was some kind of flooding in Rockville and the small overnight, and some folks they got up.
00:02:09 Frank
No one, nobody died that I'm.
00:02:10 Frank
Aware of so.
00:02:11
It it says.
00:02:12 Frank
You know we're not.
00:02:13 Frank
Custom the floods or hurricanes or tornadoes up here in DC and and we're more used to the human threats of, you know, little things like terrorism and things.
00:02:25 Frank
Like that, but.
00:02:26 Andy
Yeah yeah, you guys got a little bit more to worry about that than we do here in FarmVille, right?
00:02:32 Andy
But you know these days.
00:02:33 Andy
Who knows?
00:02:35 Andy
The, uh, definitely our thoughts and prayers are with the folks in in Louisiana and Mississippi.
00:02:40 Andy
They were hit very hard.
00:02:42 Andy
I've got got friends in Georgia, Western Georgia were telling me that.
00:02:47 Andy
They they took a beating as well and you know it just it looks horrible I.
00:02:53 Andy
I you know, I've I've been in a few of those places after hurricanes have hit as part of like church efforts to help clean up and stabilize and stuff like that.
00:03:04 Andy
It looks like I don't know.
00:03:06 Andy
They people describe it as like a war.
00:03:09 Andy
I've never been in a war so I don't know.
00:03:10 Andy
I've seen pictures and.
00:03:13 Andy
There's a lot.
00:03:14 Andy
It looks like a lot of stuff is blowing over, and that sort of.
00:03:16 Andy
Stuff, it's just.
00:03:18 Andy
So, and they're talking weeks and weeks before power comes back on.
00:03:22 Frank
That's horrible, that's.
00:03:23 Andy
Similar places, yeah.
00:03:25 Frank
That's that's.
00:03:26 Frank
Probably going to be do more damage from for a lot of things.
00:03:30 Andy
Were you worried?
00:03:30
But on a.
00:03:30 Frank
More positive note, uh, a positive note.
00:03:31 Andy
Yes, on a positive note.
00:03:35 Frank
Uh, we are.
00:03:37 Frank
I am super excited to have a special guest and I say super excited because he's from Microsoft.
00:03:42 Frank
He's a senior scientist in Jelt at Microsoft, working on scalable machine learning systems.
00:03:50 Frank
Before he was at Microsoft, he was a postdoc scholar at the Computer Science department at UCLA, and this he was doing a lot of interesting stuff there.
00:04:03 Frank
He was doing research at Qatar or Qatar.
00:04:05 Frank
I'm not sure how to say that exactly, but he has a PhD in computer science.
00:04:11 Frank
In university.
00:04:12 Frank
Of Modena and or?
00:04:15 Frank
I'm going to botch this.
00:04:15 Frank
Reggio Emilia.
00:04:17 Frank
Welcome to the show, Mateo.
00:04:22 Frank
Awesome, so we are really excited to have you here.
00:04:25 Frank
We actually booked you a whole month in advance.
00:04:27 Frank
I've been looking forward to this.
00:04:29 Frank
Yeah, because you're coming by way of some of the folks at the Mlad conference.
00:04:35 Frank
And for those who don't know, I'm a I've mentioned this.
00:04:37 Frank
Mlad stands for machine learning and data science summit.
00:04:40 Frank
It used to be in person I think now it's entirely virtual for the foreseeable future.
00:04:45 Frank
Uh, but that why I attended M lads in 2016 summer of 2016 and it was uh, it was life altering like I don't say that.
00:04:55 Frank
Lightly so.
00:04:56 Frank
So Microsoft does amazing work in the machine learning and data science space.
00:05:02 Frank
Very much cutting edge stuff very much I.
00:05:06 Frank
I wouldn't say under the radar, but Microsoft does not do a great job putting its own horn, so we're very excited for you to come on Mateo and talk about this little project that you're working on.
00:05:17 Frank
And what is the is it have a code name or what?
00:05:20 Frank
What is it called?
00:05:22 Matteo
Hummingbird should the code name is actually I'm in.
00:05:26 Matteo
Don't have any specific internal names for.
00:05:28 Matteo
This for this.
00:05:28 Frank
OK, what what is GL stand for?
00:05:32 Frank
That was my that was my first question.
00:05:33 Frank
When I saw your bio.
00:05:35 Matteo
Uh is for Gray system lamp and is the after Jim Gray which.
00:05:41
Oh, OK.
00:05:41 Matteo
Is putting award yeah?
00:05:45
OK.
00:05:46 Matteo
So these are the search lab after this name yeah and use within the Azure data organization.
00:05:49
Oh, interesting.
00:05:53 Frank
And uhm, So what?
00:05:56 Frank
What what cool stuff does Hummingbird do?
00:06:00 Matteo
So, Hummingbird, uh?
00:06:03 Matteo
Is a little bit, uh, weird project in the sense that when we started this project we didn't know if it was going to.
00:06:10 Matteo
To be a success or not?
00:06:12 Matteo
Because what we try to do basically is to uhm translate traditional machine learning models and into neural networks.
00:06:22 Matteo
Actually not Internet format into tensor programs such that then we can run over tensor runtime, such as pipers.
00:06:30 Matteo
In terms of.
00:06:32 Matteo
Uhm, so when we started this project actually idea was hey there is a lot of investment in general pulling into this neural network frameworks and.
00:06:45 Matteo
Coming from the Azure data organization, instead, we are more interested in these traditional machine learning methods such as decision trees.
00:06:52 Matteo
Linear models were not encoding all those boring traditional algorithms.
00:07:00 Matteo
And so we look at this.
00:07:01 Matteo
The neural network system and say hey how we can take advantage of all this technology that is built.
00:07:05 Matteo
Into this domain so you can run neural.
00:07:08 Matteo
Network over CPU.
00:07:10 Matteo
Over the GPU, then you can use like fancy compilers to compile to generate the transfer programs.
00:07:16 Matteo
All those sort of techniques and we were.
00:07:19 Matteo
Kind of struggling.
00:07:20 Matteo
To see what we could do with the with this stack and and what we come up with with is this Amber project.
00:07:27 Matteo
So we basically take a.
00:07:32 Matteo
Traditional machine learning pipelines composed right feature iser and machine learning models.
00:07:37 Matteo
After the day trained.
00:07:39 Matteo
So first you need to train it using cycle ornamental net or.
00:07:43 Matteo
Uhm, uhm, one of those traditional machine learning platforms and then once it is trained we basically convert it into a set of tensor operations in.
00:07:54 Matteo
In the current version we use basically PY torch for doing this conversion and then basically you have a pipeline model so you can do whatever you can do with Python.
00:08:03 Matteo
Models so you can deploy it in in it into a PY torch.
00:08:08 Matteo
Uhm, deployments you can run over CPU ran over the GPU or you can do the torch script if you want to get rid of all the Python dependency and just have a C++ program you can.
00:08:19 Matteo
Do all those all those tricks.
00:08:22 Frank
Interesting, does it impact accuracy precision?
00:08:26 Frank
Does it improve it?
00:08:27 Frank
Keep it the same.
00:08:29 Matteo
We tried to keep it the same so we are able to keep.
00:08:33 Matteo
It The same up to floating point numbers roundings?
00:08:36 Matteo
So since we use, you know we use PY torch to run these programs and not like a socket or ornamental net.
00:08:44 Matteo
There are some differences in how they do you know, floating point operations.
00:08:48 Matteo
So the.
00:08:49 Matteo
Accuracy is up to roundings in the Floating Points, which sometimes are actually.
00:08:54 Matteo
It can be quite a bit, but most of the time is really small, almost not noticeable.
00:09:00 Frank
Interesting, interesting, uhm.
00:09:03 Frank
Do you would you know.
00:09:05 Frank
If there was like.
00:09:06 Frank
A discrepancy, or you Dutch as part of testing?
00:09:09 Matteo
It's part of testing.
00:09:10 Frank
Right, all software is tested, right Andy?
00:09:11 Matteo
So we have we have.
00:09:13 Frank
Sometimes intentionally is that the email.
00:09:15 Andy
That's right.
00:09:17 Frank
And he has a saying where all softwares I I forget exactly what it is.
00:09:21 Frank
But what is it?
00:09:23 Andy
Yeah, all software is tested, some intentionally.
00:09:27 Frank
There you go.
00:09:30 Frank
Uhm, so what's the?
00:09:33 Frank
What's the real?
00:09:34 Frank
What are?
00:09:34 Frank
What are the advantages of of of converting kind of a traditional model over to a tensor model?
00:09:41 Frank
Is it?
00:09:41 Frank
Is it portability?
00:09:42 Frank
Is it speed?
00:09:43 Frank
You did mention that you can run it on.
00:09:45 Frank
You could take advantage of GPU as well as CPU.
00:09:51 Matteo
Yes, exactly so you most mostly is related to speed, so you can basically run your socket, learn model on GPU end to end and and this user provides you know a little bit of quite a bit of speed up we for some of our example we even saw like 2 ordinal Magneto speedups.
00:10:11 Matteo
For some of the models.
00:10:13 Matteo
And uhm, and usually we try to show that.
00:10:18 Matteo
If you use GPU.
00:10:19 Matteo
Can be much faster, but on CPU we try to be kind of as close as possible scikit learn or the base or the base or diminished model.
00:10:27 Matteo
Sometimes we can, sometimes we are a little bit slower.
00:10:31 Matteo
Uh, but we.
00:10:32 Matteo
We had some really interesting result.
00:10:34 Matteo
Like for instance, we did some experiment with some.
00:10:39 Matteo
Some folks at the VM and we took some extra boost model and we compiled some training accuracy boost model.
00:10:47 Matteo
Uh, using Hummingbird anti VM into some uh, we basically do code generation and we show that the that model that was compiled to Python was even faster than they quoted the C++ implementation that they're having next used, but those CPU and GPU. Yeah, there was kind of OK. What's going on?
00:11:06 Matteo
This is not.
00:11:08 Matteo
This was not expected.
00:11:08 Frank
Wait, did you say it was faster than a C++ implementation?
00:11:11 Matteo
Yes, I mean if she used.
00:11:13 Matteo
Underneath C++ even scikit learn.
00:11:15 Matteo
You know they use like.
00:11:16 Matteo
From C++ library and yeah, using TVM for doing the code generation, they are able to do like a operator fusion which you don't normally have for like these traditional models.
00:11:28 Matteo
So we told these tricks bigger, basically that are coming from the neural network.
00:11:31 Matteo
Famous we were able to get like this.
00:11:34 Matteo
These surprising numbers.
00:11:36 Frank
Interesting, so that's a real performance boost, and probably if you scale that up into the cloud that probably.
00:11:44 Frank
Means a lot of money saving too in terms of on cloud computing things like, I imagine a company like the size of Microsoft would be very interested in getting better results faster with less cloud compute.
00:11:56 Frank
You did mention an acronym, I just wanna make sure folks know.
00:11:59 Frank
What that is?
00:12:00 Frank
Tyvm what is that?
00:12:03 Matteo
Uh, I don't know what is exactly for, uh, some tensor maybe?
00:12:08 Frank
Andy looks like he knows, but he's on mute.
00:12:10 Andy
I don't, yeah I I don't know.
00:12:13 Frank
OK, I'm just curious.
00:12:13 Andy
I'll go look it up.
00:12:15 Frank
There you go.
00:12:16 Andy
EVM acronym.
00:12:19 Matteo
I think is for tensor virtual machine, but I'm.
00:12:21 Matteo
Not sure if this is approach.
00:12:22 Frank
That sounds about right.
00:12:23 Frank
Tector,...
