Artwork for podcast Android Bytes (powered by Esper)
The hidden challenges in porting AOSP
Episode 29 • 19th October 2022 • Android Bytes (powered by Esper) • Esper.io
00:00:00 00:45:54

Share Episode

Shownotes

It's dangerous to go alone! Take us with you!

In this episode, we're joined by our very own Jon West and Nikhil Punathil at Esper to discuss a key part of what we do here — getting AOSP up and running on ARM and x86 hardware. If you want to port AOSP, it's not as easy as just compiling an image from Google's git repos and slapping it onto a device.

  • 05:59 - What is a device bring-up? What are some of the challenges in doing a bring-up?
  • 10:58 - How do AOSP developers deal with a lack of kernel source code?
  • 18:16 - How did Project Treble affect building AOSP?
  • 26:54 - How does Android on x86 differ from Android on ARM?
  • 29:48 - What problem does the Generic Kernel Image try to solve?
  • 35:22 - How long does Google support a particular AOSP release? What can AOSP developers do once support has ended?

Android Bytes is hosted by Mishaal Rahman, Senior Technical Editor, and David Ruddock, Editor in Chief, of Esper.

Esper enables next-gen device management for company-owned and managed tablets, kiosks, smart phones, IoT edge devices, and more.

For more about Esper:

New from the Esper blog:

Our music is "19" by HOME and is licensed under CC BY 3.0.

Transcripts

David:

Hello and welcome to Android by Its Powered by Asper.

David:

I'm David Ruddick, and each week I'm joined by my co-host Michelle Ramen,

David:

diving deep into the world of Android.

David:

And this time we're diving deep into the world of Asper because I have one

David:

fellow Asper, uh, team member next to me.

David:

And we have another fellow, Asper, right?

David:

Asper, I'm not sure what we call each Asper.

Jon:

Anyon is what we usually.

David:

That's Peren.

David:

Okay.

David:

I like that.

David:

It's kind of fun.

David:

Sounds like an alien from Star Trek.

David:

Michelle, would you like to introduce our guests?

David:

Yeah,

Mishaal:

thanks David.

Mishaal:

So as David mentioned, we have two very special guests and of course every

Mishaal:

guest who comes on the show is a very special guest, but these are even special

Mishaal:

to us because they are colleagues.

Mishaal:

We have Nicki Puntil and John West on the show today.

Mishaal:

I'd like to introduce both of you to Android Bites.

Mishaal:

Thanks for joining.

Nikhil:

So, uh, I'll, I'll start.

Nikhil:

I'm Nick Pun.

Nikhil:

I run Esther's Innovation Lab, so that's where we do all the fun stuff

Nikhil:

and just go nuts with Android and, you know, everything technology.

Jon:

John West, I help run the Sper device targets.

Jon:

So we grab new things coming in and see what changes we need to make and

Jon:

make sure everything's working with the devices and can be provisioned just fine.

Jon:

Thanks for, uh, having

Nikhil:

us.

Nikhil:

Yep.

Nikhil:

Happy to be here.

David:

And so I guess it's fair to say both of you on a regular basis are

David:

answering the question, can we do this?

David:

Is it

Mishaal:

possible

Jon:

And Yes, Yes, we can.

Mishaal:

Can we build it?

Mishaal:

Yes.

Mishaal:

You're all Bob, the builders here.

Mishaal:

And y'all were doing this even before you joined Sper.

Mishaal:

So where did you guys come from?

Mishaal:

What were you doing before Sper that related to Android development?

Mishaal:

Yeah,

Nikhil:

so I started messing with Android really a long time ago.

Nikhil:

Around the time, I think the first version of Android that I played

Nikhil:

around with was, uh, gingerbread with my first Android device.

Nikhil:

It was.

Nikhil:

It was a really crappy Samsung Galaxy MP4 player.

Nikhil:

Like it was one of the, one of the iPod Touch competitors.

Nikhil:

Those it, The Galaxy Player.

Nikhil:

The Galaxy Player 4.2.

Nikhil:

That was my first Android device.

Nikhil:

Seems like ages ago, but also that device.

Nikhil:

Never got any updates.

Nikhil:

It was shunned by the community.

Nikhil:

And that's really what led to me trying to see, okay, I want

Nikhil:

this to be better than it is.

Nikhil:

Cuz getting a new device really at that point wasn't an option.

Nikhil:

So you gotta deal with what you have.

Nikhil:

And that's when I started, just like checking out the community, see what

Nikhil:

was going on, looking at Synogen mod, which was a huge thing back then rooting.

Nikhil:

In fact, one of the weird things that I ended up learning a lot about was smelly

Nikhil:

at its, which is essentially compiled.

Nikhil:

Java code for Android and working with that and like adding mods to your OS

Nikhil:

through smelly mods, it's, it was a pretty interesting time and it grew

Nikhil:

from there to me, building Android from source building, unofficial builds of

Nikhil:

signage mod for the devices I owned.

Nikhil:

To me, joining the Carbon Rom team as a maintainer and eventually a core

Nikhil:

team member made my way through the community and I finally found home at.

Mishaal:

I gotta say, I don't envy the things you guys have

Mishaal:

to do back then to modify apps.

Mishaal:

Small editing small is a pain to read.

Mishaal:

I'm so glad there are tools like J X now that actually convert that to readable

Mishaal:

Java code and like Jeb Decompiler, so much better de compilation, reverse

Mishaal:

engineering tools, these stays.

Mishaal:

Oh, absolutely.

Mishaal:

So what about you John?

Mishaal:

What did you do prior to joining Sper?

Mishaal:

Oh

Jon:

goodness.

Jon:

Uh, you guys are gonna hate me, but probably start way

Jon:

back with Windows Mobile.

Jon:

Moved in to Android when the HTC one came out and the HTC Hero came out after that.

Jon:

Pretty much started on my own for a while there, did a bunch of stuff, and

Jon:

then started on Team Bliss just as that project was being brought to light.

Jon:

And soon after that became interested in Android X 86.

Jon:

So that's building Android for PC hardware.

Jon:

Moving into the realm of taking things where no man has gone before and

Jon:

seeing how far we can get it going.

Jon:

That ended up going on from 2013 to present.

Jon:

It's still going under new management now, new developments.

Jon:

So we have a bunch of developers that are working on that project

Jon:

while I'm working for Sper here.

Jon:

And yeah, that's pretty much my, uh, experience.

Jon:

I specialize.

Jon:

And the X 86 PC hardware stuff, uh, melding the worlds with

Jon:

Linux and Android into one.

Jon:

And that's where I get my kicks.

Jon:

So how I fit into Eser here is we get in all these devices, POS

Jon:

devices, and they might have goals to be on a newer version of Android.

Jon:

We can simply bring them up, get them to work and make them provision

Jon:

fine over ota, using new version Androids, which are always updated

Jon:

thanks to our foundation project.

Jon:

And yeah, making customers happy nicely.

Mishaal:

Yeah, so you both come from very different Android hacking backgrounds.

Mishaal:

Like on the one hand we have.

Mishaal:

Who's delved a lot of work with X 86 hardware, which I think is pretty

Mishaal:

rare in the Android community cuz most Android, A O S P engineers kind of

Mishaal:

deal with arm devices like Nicki did.

Mishaal:

And so like if you wanna bring Android, Android challenge to niche hardware, you

Mishaal:

know, you, you're gonna need a lot of expertise in compiling Android modifying

Mishaal:

Linux, reverse engineering binaries, doing all sorts of just hacky things

Mishaal:

to get Android up and running on a.

Mishaal:

Where you either wasn't meant to run or you're trying to run a vanilla flavor

Mishaal:

of Android on a device that originally shipped with a heavily proprietary version

Mishaal:

with heavy modifications to the kernel.

Mishaal:

So there's all sorts of different considerations you need to make based

Mishaal:

on the base device you're trying to modify and bring Android onto.

Mishaal:

And it's far from easy, and I'm sure like.

Mishaal:

By the end of this episode, you'll appreciate just how much work has

Mishaal:

to go into actually bringing Android onto a device that wasn't originally

Mishaal:

running on, or you know, it was running something else Android based.

Mishaal:

So I kinda wanted to start out, but just like some basic terminology.

Mishaal:

So this episode we mentioned device bring up.

Mishaal:

What does it mean to do a device bring up?

Jon:

Device Bring up pretty much starts with either A, A device, or

Jon:

B, the released source for a device.

Jon:

Unfortunately, in our world, we have to do it both ways because

Jon:

sometimes we'll get sourced without a device attached, or sometimes we'll

Jon:

get device without source attach.

Jon:

So either A, you're pulling the device information and parsing

Jon:

your device source from the device itself, or B, if source is provided,

Jon:

you're using that along with aos P.

Jon:

In order to compile Android with all the device house vendors, info every,

Jon:

all the information for that device in place so that it works out at the

Mishaal:

box.

Mishaal:

All right, so before we dive into the intricacies and the many ways that you

Mishaal:

gotta fill in the gaps for yourself, I kind of wanted to start with a golden

Mishaal:

example of like, what's an example of where everything's done right?

Mishaal:

You basically have to just follow the step by step instructions that are listed.

Mishaal:

You don't have to go digging for something somewhere else.

Mishaal:

And I'm sure obviously the example would be like Pixel, right?

Mishaal:

If you wanna do a ring up on pixel, everything is handed down for you.

Mishaal:

Like, can you mention some of the things that Google does to make

Mishaal:

device bring up significantly easier?

Mishaal:

And then we'll start from there and like talk about, you know, where others

Mishaal:

kind of struggle to meet those needs.

Mishaal:

Can we start

Jon:

with, uh, get Attri.

Jon:

Who adds and uses proper GI attribution.

Jon:

So when you look at the device tree, every change that has potentially

Jon:

been made in the past is still intact within that GI history.

Jon:

This includes things that have been added and taken away, or support that has

Jon:

yet to be added, can be found on their.

Jon:

Or their Garrett.

Jon:

This is a great tool that we don't see too often being used properly across the web.

Jon:

So even in the open source world, everybody has a problem

Jon:

with proper get attribution.

Jon:

If you get a bsp, sometimes they come in a tarball and get

Jon:

attribution is not attached.

Jon:

So we don't know what changes they were made to the device Source

Jon:

street and when they were made.

Jon:

So this is our number one and most important thing that I like to.

Jon:

When I'm doing a device bring up, and Google

Mishaal:

does

Jon:

provide this handover hands better than anybody I've.

David:

Those who are maybe not quite as technical.

David:

Just to give you an anecdote there, you could essentially think of this as

David:

versions of revisions of a document.

David:

And then every change that has been made is documented and assigned to

David:

whoever, whatever, whoever's responsible for that change, why they made it.

David:

Hopefully they have comments explaining why they did those things.

David:

And context is obviously immensely helpful in any kind of

Mishaal:

project.

Mishaal:

Right?

Mishaal:

And Google provides you all of that versus like with any off the shelf.

Mishaal:

You may get the document, but you may not get its revision history, or you

Mishaal:

might not even get the document at all and you have to go digging and

Mishaal:

like begging for the OEM or the ODM who made it, who actually developed

Mishaal:

the software to release it to you.

Mishaal:

Or if they do release it, it's broken and you can't actually compile it or use it.

Mishaal:

What do you do then?

Mishaal:

You just have to keep begging and asking for more releases, so, There's all sorts

Mishaal:

of things where this can go wrong, but like a proper version, not just a tar

Mishaal:

ball, which is basically just an archive of the kernel files, but the actual GI

Mishaal:

repository that they were themselves are probably working on having access to that.

Mishaal:

So you can see everything that David just mentioned is immensely helpful.

Mishaal:

And, uh, just

Jon:

to add context, most of the technology being released is

Jon:

not being followed through with what Google does for good attri.

Jon:

For example, the kernel source, I think it was zing that was having trouble or maybe

Jon:

a different company that was having a hard time producing kernel source for a device.

Jon:

And, uh, one of the Twitter people, uh, Sexy Cyboard made a

Jon:

video of her going into the place.

Jon:

And Rick Westing, the colonel source from the developer,

Jon:

was like a week long ordeal.

Jon:

She compiled into one video, but sometimes it takes all that

Jon:

just to get what we should.

Jon:

On every device

David:

release.

David:

Yeah, and I mean, if you've been involved in the Android community,

David:

obviously Kernel source is a raging war that never ends with, especially

David:

the customization community.

David:

Companies like Motorola are infamous for just lagging behind on things like this.

David:

And I know, Show me, like you mentioned, they just stop releasing kernel

David:

source a lot from what I understand.

Mishaal:

So to be fair, like a lot of OEMs either, as I

Mishaal:

mentioned, don't really sit at all.

Mishaal:

Release it months late or in a half broken state.

Mishaal:

And there are a few golden childs who do everything really

Mishaal:

well, like Google, of course.

Mishaal:

But of course, if, if you're going with off the shelf hardware from a lesser

Mishaal:

known brand, then chances are you might not ever get the kernel source.

Mishaal:

And what happens if you don't have a kernel source?

Mishaal:

What happens if all you have access to is the pre-compiled kernel

Mishaal:

binary that ships on the device?

Mishaal:

What can you do with.

Mishaal:

If anything, you can actually

Jon:

pull that pre-compiled kernel and use it as a, uh, pre-built kernel

Jon:

within the device tree that allows us to deal with the lack of responsibility

Jon:

from OEMs and actually still

Mishaal:

make it work.

Nikhil:

That comes at the cost of not being able to make any

Nikhil:

necessary kernel changes, right?

Nikhil:

So if you're starting with Android 12 and you have an Android 12 kernel from

Nikhil:

the ODM or oem, you're probably fine for Android 12, but as soon as Android

Nikhil:

13 comes along and you wanna do a bring up of that, chances are you need to

Nikhil:

make kernel changes and well, you don't have the source to make those change.

Nikhil:

So now you're stuck.

Nikhil:

And some very smart people in the community have figured out ways to

Nikhil:

essentially reverse engineer and figure out they, they'll take the kernel

Nikhil:

source from a very similar device, probably running the same chip set if

Nikhil:

possible, from the same manufacturer, but that's not always possible.

Nikhil:

And just figure out what needs to be done to add changes on top of

Nikhil:

that and make it boot on a device.

Nikhil:

It always impresses me when I see things like that cuz that's

Nikhil:

just dedication to the craft.

David:

Well it's like kind of reminds me of like it's the Chevy small

David:

block of, basically there had to be devices that were especially popular

David:

to use as basically a kernel source.

David:

Right?

David:

Like over the years.

Nikhil:

Absolutely.

Nikhil:

Yeah.

Nikhil:

Well, and the other part of it is you, you could be working with a

Nikhil:

weird combination of vendors where it could be an unsupportive OEM who

Nikhil:

doesn't wanna release sources, but they could be running a Qualcomm chip

Nikhil:

in there, which Qualcomm is very.

Nikhil:

One of the most open source friendly chip vendors for Android, and so you

Nikhil:

have generic Qualcomm sources that are just out there and available.

Nikhil:

So you have something to start from.

Nikhil:

Right?

Nikhil:

That's usually the best case scenario when it comes to unsupportive

Nikhil:

and no source situations.

Nikhil:

Yeah.

Nikhil:

With

Jon:

those you can just run a diff and grab what you're missing pretty much,

Jon:

and then have that as a handful of one or two commits to add on top of that

Nikhil:

specific device.

Nikhil:

Right.

Nikhil:

So that's exactly what I was saying, which is you have a foundation,

Nikhil:

and that foundation is really what makes all the difference.

Nikhil:

And you'll notice that devices with not much support are typically devices.

Nikhil:

Don't have open source support from the tip vendor themselves.

Nikhil:

Right?

Nikhil:

And so, I mean, you can look at this as a bunch of layers, right?

Nikhil:

Where the real original source comes from, the chip vendor, Qualcomm, or

Nikhil:

Media Tech, or Brock Chip or whoever.

Nikhil:

We will build a new chip and they will have a generic set of what we

Nikhil:

call board support package, right?

Nikhil:

BSP is just a bunch of files that you need to add on top of Android to make

Nikhil:

Android work on that particular chip.

Nikhil:

Now, this is generic Android.

Nikhil:

There's no modifications other than hardware support stuff added.

Nikhil:

And then this is what gets handed out to ODMs OEMs and everyone else

Nikhil:

who uses that chip and they add their own sauce on top of that.

Nikhil:

At one point it becomes very recognizable.

Nikhil:

But you do know that under the hood was that original basic

Nikhil:

bare bones BSP that came from the chip vendor that's being used.

Nikhil:

So if you have that, you have something to work with.

Nikhil:

And I guess

David:

the context here of why this makes things difficult is that you then have

David:

to learn everything and more that the OEM and ODM did as part of their bring up.

David:

So you're not only repeating the process, you're doing it

David:

with less information than they

Nikhil:

had.

Nikhil:

Absolutely.

Nikhil:

And, and a lot of times, like, I mean, I, I say Qualcomm is a good example, right?

Nikhil:

But even Qualcomm, and rightfully so, they don't actually have

Nikhil:

fully open source binaries.

Nikhil:

There are still closed source proprietary binaries where they have

Nikhil:

their own secret sauce and their, I mean, big sense iss P gpu Exactly.

Nikhil:

Things like that.

Nikhil:

Things like that.

Nikhil:

So especially when you're dealing with an end of life device or an end of life chip.

Nikhil:

You don't have updates to those binaries.

Nikhil:

Even with supportive situations like that, you still have to do

Nikhil:

some hacking to get stuff working.

Nikhil:

Yeah.

Nikhil:

And

David:

I think that getting us too far off track, that's part of why we're

David:

seeing Google try to modularize Android more so that we can start updating these

David:

things standalone and stop basically like making everything beholden to BSP

Jon:

version.

Jon:

Absolutely.

Jon:

And that was their goal for project travel or travel support when they

Jon:

introduced that into uh, I think Android.

Jon:

Was the first introduction for it that made it separated, kind of where

Jon:

we had a vendor partition that would contain all the device information, ODM

Jon:

stuff, and that would be static that would only be updated by the vendor

Jon:

and then everything else around it, the system partition, product system

Jon:

extended, et cetera, would all be updated by the OEM or Google the source.

Jon:

For Android.

Jon:

So that made it kind of simpler for some forward parts, but a little bit

Jon:

less simpler cuz we still have to pull that vendor image, parse it,

Jon:

figure out what all is added to it.

Jon:

And if we plan on doing anything with it, it has to be mostly private.

Jon:

Unfortunately,

Nikhil:

and things are moving in a good direction too, right?

Nikhil:

So now Google has the generic kernel image concept.

Nikhil:

Where that would come in handy is in the future where most devices do come

Nikhil:

with gki support, which Google enforces.

Nikhil:

You wouldn't even need the kernel sources, which technically you're supposed to

Nikhil:

get, but a lot of times you don't get, as we talked about earlier, right?

Nikhil:

You could still run a generic kernel.

Nikhil:

Top of your device that's running the OEM specific kernel

Nikhil:

image and still get it working.

Nikhil:

And so now you have the ability to switch out everything on your device

Nikhil:

to generic versions and just swap out the entire operating system basically

Nikhil:

without any help from the OEM or oem.

Nikhil:

I think that's the ideal goal that Google is aspiring towards and

Nikhil:

seems like we're getting there.

Nikhil:

Right,

David:

and you know, just.

David:

Again, is a kind like kind of very high level check.

David:

This also assumes that you can get into the device, which is a big assumption

David:

we have to make for any of this, right?

David:

So you may either need cooperation and existing exploit or something

David:

else, which I don't, I'm not sure what

Nikhil:

other routes there would be.

Nikhil:

Well, there's a few ways, right?

Nikhil:

So at least in a professional space when it comes to Android devices and building

Nikhil:

a O S P, , you have two situations.

Nikhil:

One is where you have the bsp, the whole package that the ODM

Nikhil:

uses themselves or gives out to other people who need it, right?

Nikhil:

In that situation, the images you build and the output really is something

Nikhil:

that you can term as a whole package, meaning like the entire operating

Nikhil:

system files, including stuff outside of Android that is needed by Android,

Nikhil:

but it's technically not Android.

Nikhil:

It's quite a full system image, right?

Nikhil:

Boot loader and other.

Nikhil:

In the device.

Nikhil:

You have all of that and.

Nikhil:

Basically use whatever flashing process that the chip vendor has

Nikhil:

determined for their board to flash it.

Nikhil:

And at that point, you're acting like the odm, right?

Nikhil:

You're doing what they would do in the factory.

Nikhil:

And so that's the more clean process.

Nikhil:

But in the community and as a consumer, you don't really get

Nikhil:

access to that with any odm really.

Nikhil:

Right?

Nikhil:

And so at that point, you're working with a S P and you're working with

Nikhil:

whatever sources they've dropped, and you're just building what you need.

Nikhil:

To swap out and bring it up to a new version of Android or update whatever

Nikhil:

you need to update and leave those other auxiliary partitions just untouched.

Nikhil:

So there's like two levels to this.

Mishaal:

We briefly touched upon project trouble before jumping into generic

Mishaal:

kernel image and then you know this topic.

Mishaal:

But I kind of wanted to step back and focus more on Project Trouble

Mishaal:

because it's such an important.

Mishaal:

Part of what makes device bring up pretre versus post trouble.

Mishaal:

So different.

Mishaal:

So I wanted to ask you, can you explain how a device bring up differs

Mishaal:

before product troubles introduced versus after it was introduced?

Mishaal:

Like what is the fundamental differences?

Mishaal:

Pretre,

Jon:

everything was pretty much built into system is root.

Jon:

So all of your device information, all of the vendor applications were all included

Jon:

in one system, partition on the device.

Jon:

Post travel, all of that is separated into a system, a product, a system

Jon:

extended, and a vendor bar issue where you have access to all the separate

Jon:

parts, but they're mounted within the system a little bit differently and

Jon:

separated on the device themselves.

Jon:

So when we update on a project travel device, we're only

Jon:

updating the system parti.

Jon:

And possibly product, uh, vendor system extended as well from the actual OEM or

Jon:

the vendor that leads it to where they can do security updates much faster

Jon:

without having to rebuild everything they've added previously on top of

Jon:

a brand new security release onp.

Jon:

They can easily add those patches into the system and it's all kept.

Mishaal:

When Google introduced trouble, when they decided to strip the

Mishaal:

vendor specific hardware abstraction layers from the system image into its

Mishaal:

own dedicated vendor partition, they also defined an interface, a standard

Mishaal:

interface between the vendor partition and the Android operating system.

Mishaal:

So to allow the OS to communicate with those hardware abstraction

Mishaal:

layers in a standardized way.

Mishaal:

And they call that the vendor I.

Mishaal:

I wanted to ask you, like, can you talk a bit about this vendor interface and how

Mishaal:

its introduction affected device bring up?

Mishaal:

Well,

Jon:

there were two ways of vendor interface was added.

Jon:

It was, uh, added either a through the vendor image that the hardware vendor

Jon:

would create and that would contain all the added files that they needed.

Jon:

For I D L interfacing, it would also contain some of the information that they

Jon:

need for external apps that they're adding onto the system and services as well.

Jon:

They also introduced the V N D K interface.

Jon:

And that is pretty much where all the hardware interaction goes.

Jon:

So anything that goes through V N DK has to be certified pretty much

Jon:

to work with their device interface.

Jon:

I don't have too much experience working with the dk, Maybe does.

Nikhil:

The BDK is less talked about aspect of travel.

Nikhil:

You know, when we say travel, talk about project travel, typically you

Nikhil:

talk about how things are separated between system and vendor, but.

Nikhil:

Part of that separation was Google defined that interface, like you said.

Nikhil:

Right?

Nikhil:

And VDK is an important part of it.

Nikhil:

Vdk, by the way, stands for Vendor Native Development Kit.

Nikhil:

And what it basically does is it gives these ODMs and chip side vendors is

Nikhil:

standardized way to define their Hals and their hardware support libraries.

Nikhil:

And what it also does is it enforces backwards compatibility, right?

Nikhil:

So if you have a vendor that was for Android nine, Then by Google's

Nikhil:

compatibility definition, you would have to make sure that that

Nikhil:

vendor is or compatible up to three versions I think, of Android.

Nikhil:

So that means you could use that same vendor.

Nikhil:

And a system image from Android 12 say, and it would work perfectly fine,

Nikhil:

and they have to work well together.

Nikhil:

And this is the sort of rules that Google is setting that

Nikhil:

makes device bring up easier.

Nikhil:

So if you have a device that doesn't get updated that often,

Nikhil:

it's running on Android 10 and you wanna run Android 13 on it.

Nikhil:

You don't really have to mess with the vendor layer at all.

Nikhil:

Maybe perhaps slight modifications depending on device specific works,

Nikhil:

but for the most part you can leave that vendor part untouched and work

Nikhil:

on the upper layer, so to speak.

Nikhil:

And, and that really is the beauty of trouble is there's definitions

Nikhil:

now and there's standardization of.

Nikhil:

This communication between system and vendor, and I

David:

guess is a good way to describe that, really, how the hardware

David:

extract itself to software and says, I can do this, I can do that.

David:

And here's how you call that, right?

David:

Essentially like the camera can zoom.

David:

Therefore you should have a hook in here for your vendor, specific camera

David:

implementation if you have a special one.

David:

For Zoom, not to get into Google or Android camera AP guys, cuz

David:

that's a whole other thing.

David:

But in general, conceptually, is that kind of the way that works?

Nikhil:

Right.

Nikhil:

And it doesn't really block manufacturers or Google from implementing new things.

Nikhil:

It just makes sure that when you do implement new things,

Nikhil:

you have to make sure that.

Nikhil:

It doesn't break compatibility with the older versions of the vdk.

Nikhil:

Right.

Nikhil:

So that's why you see, I think if you're in the Android community now,

Nikhil:

new versions of Android, especially like Custom ROMs are dropping much sooner.

Nikhil:

It's because they don't really have to spend, That's really

Nikhil:

where most of the time is, right?

Nikhil:

Because the vendor layer and the Hals and the hardware support

Nikhil:

stuff, that's where all the IP is.

Nikhil:

That's where you have the.

Nikhil:

Source code access, and so that's where you have to do the most.

Nikhil:

Hacks and hacks are usually trial and error and usually take a lot of time, and

Nikhil:

people who do this for free in their free time tend to not have much free time.

Nikhil:

That's really what helped things get better over the years, especially

Nikhil:

with trouble because you don't really have to do much anymore.

Nikhil:

For example, if you're waiting on one plus to drop their Android 13

Nikhil:

builds, you can still get Android 13 as a custom realm on your.

Nikhil:

While you wait for that Android 13, and when it does drop, you can have a

Nikhil:

newer build, which uses that updated vendor that's more suited for Android

Jon:

13.

Jon:

A lot of times that updated vendor will have updated firmware

Jon:

for the device, et cetera.

Jon:

So Nel Image, that type of stuff.

Mishaal:

Technically is a generic system.

Mishaal:

In the one plus example with their recent devices, things are getting

Mishaal:

more complicated with the Google requirements freeze program.

Mishaal:

So a lot of modern flagship devices are now actually shipping older

Mishaal:

vendor software implementations that were built for an older release.

Mishaal:

So for example, a device that launched with Android 12

Mishaal:

that upgrades the Android 13.

Mishaal:

Might still be using the same vendor software implementation that

Mishaal:

was developed for Android 12 when it upgrades, and that's a quirk

Mishaal:

of Google requirements freeze.

Mishaal:

We talked about that before, but I don't wanna get into the exact nitty gritty

Mishaal:

of that aspect right now, but I kind of wanted to touch upon a bit more on

Mishaal:

the genericization of the interface between the hardware extraction layers

Mishaal:

and the OS, and why that's so important.

Mishaal:

So let's say pre-project trouble, if you were to try to just flash an A

Mishaal:

S P system image onto the device and have it booted up, if the OEM of that

Mishaal:

device had some custom hardware and they wrote their hardware abstraction

Mishaal:

layer in a nonstandard way that a S P wouldn't be able to interface with it.

Mishaal:

Then when you boot up a P, that piece of hardware that that abstraction

Mishaal:

layer was written for just might not work or it might just crash.

Mishaal:

So like if you had a camera and you were expecting it to boot up,

Mishaal:

when you open up the a SB camera app, you might just get nothing.

Mishaal:

You might see nothing at all.

Mishaal:

And that's obviously a huge problem because you're dealing with

Mishaal:

fundamental hardware components of a device that you would expect to.

Mishaal:

Out of the box.

Mishaal:

But with project trouble standardizing a lot of these base components and

Mishaal:

Google enforcing this through these vendor test suites, their, their vendor

Mishaal:

interfaces, you know, all these things like the diversions that OEMs to follow.

Mishaal:

Like it ensures that if you take a modern project trouble compatible

Mishaal:

device and you boot AOS P onto it, Chances are that at least almost all

Mishaal:

the basic hardware components will work.

Mishaal:

So like for example, we have a Lenovo tab K 10, that we all frequently use

Mishaal:

for work purposes and that device ship put Android 11 outta the box.

Mishaal:

I've been able to boot Android 1212 L Android, 13 generic system images

Mishaal:

onto it and all the hardware works.

Mishaal:

A s p works perfectly for just fine.

Mishaal:

There's no issues with it at all, and that's thanks to project.

Nikhil:

And we're where to the point where you can't even tell the difference

Nikhil:

between a GSI and a dedicated image.

Nikhil:

Right?

Nikhil:

So unless your device has specific, like when it comes to one plus devices,

Nikhil:

they have that alert slider, or they have the in display fingerprint sensor.

Nikhil:

Those are specific hardware choices that aren't really a part of a s P or gsi.

Nikhil:

But for a generic device like a Lenovo K 10, it's a decent tablet

Nikhil:

that has everything you need.

Nikhil:

If you go to Gs.

Nikhil:

It works.

Nikhil:

Every feature you want to work works.

Nikhil:

And so at this point, I mean the sort of direction we're heading into is

Nikhil:

we will just have a Windows or Linux like situation where you have a build

Nikhil:

of Android that drops that you can just install on any device and it

Nikhil:

more or less behaves the same next step for, you know, hardware specific

Jon:

works.

Jon:

Let's talk about the difference there though with Windows, Linux, and Android.

Jon:

So on Windows, it ships with monolithic kernel a lot like Linux

Jon:

does well, So that contains all the device information, the drivers, the

Jon:

firmware for the drivers, et cetera.

Jon:

And hopefully 90% of that is detected upon a net, or when it first boots

Jon:

up on windows, it detects a list of what's available, downloads what it

Jon:

can, after connection is available, and then, uh, pairs it up based on that.

Jon:

But with Android, it's a.

Jon:

Find a bit different because it relies on what's on the device.

Jon:

None of that extra stuff is ever included on the Linux NEL that's sent

Jon:

with Android, at least for most devices.

Jon:

If you're dealing with an Android X 86 device, we did switch to a

Jon:

monolithic kernel and it kind of pairs up hardware available using mod probe

Jon:

and if probe exists, mod probe this.

Jon:

If found great at it to the list, if not, great, ignore it

Jon:

and then continue booting when.

Jon:

So it'll go through every potential connection in your device until it

Jon:

finds it all just like Linux does and pairs it up with that device.

Jon:

Hardware on Android, especially using Project trl, it's a bit

Jon:

different because you have that product definition already set up.

Jon:

So it's gonna be shipped with that product image, vendor image, and that's gonna

Jon:

contain all the information they need to load the kernel, load the drivers,

Jon:

load any extra blobs, and drive device hardware for those drivers, and then

Jon:

can make the connections and then boot.

Jon:

Probably

David:

philosophically there's some really good reasons that Android ended up this

David:

way, aside from inheriting from Linux.

David:

But number one, Android was always designed to be ultra lightweight.

David:

And so by being able to define the firmer image at a very high level, you can strip

David:

out everything that is unnecessary and have a super lightweight device, which

David:

was very important in early Android, especially . You did not want bloat.

David:

Everybody hated bloat.

David:

But I think the other side is probably.

David:

For Google, you know, this is a better way for them to pull the levers.

David:

They want to controlling the ecosystem.

David:

They can do it very high up as opposed to genericizing, which they have been doing.

David:

Um, that's a harder problem to solve, right?

David:

It's a lot easier to make a rule and say, You vendor can do this.

David:

You vendor can't do that.

David:

Rather than giving them a tool that says, Okay, we can make this

David:

more flexible, adaptable, which.

David:

Really what trouble I think was probably

Mishaal:

about, This actually brings us back to the generic kernel

Mishaal:

image topic and the reason why Google went for that in the first.

Mishaal:

Since, as you mentioned, Android devices, they generally chip with

Mishaal:

their device drivers on the device.

Mishaal:

Not everything is included in the Linux terminal, and if you just boot

Mishaal:

Linux onto an Android device, you can't expect everything to work.

Mishaal:

So the problem is that a lot of open source developers have

Mishaal:

been pushing for Android device makers and vendors for years to.

Mishaal:

Upstream your drivers open source your drivers and submit them to Linux and

Mishaal:

let us take care of maintaining that.

Mishaal:

But that has never happened.

Mishaal:

That pushback.

Mishaal:

It just just didn't happen.

Mishaal:

No, it's, There was no ip.

Jon:

All these device manufacturers want that IP to stay private.

Jon:

So I

Mishaal:

completely understand why.

Mishaal:

Yeah, exactly.

Mishaal:

But like then that becomes a problem because there's so much out of tree

Mishaal:

coat or there used to be so much out of tree code running on, you

Mishaal:

know, in the Linux NEL that ships.

Mishaal:

Your average Android device and, and so if device drivers aren't being

Mishaal:

upstreamed, then it kind of limits the longevity of those devices, the ability

Mishaal:

to upgrade those kernels and implement security patches, because there's so

Mishaal:

much of a difference between the kernel that ships on a device and the upstream

Mishaal:

Linux kernel that a lot of work has to be done to manage those merges.

Mishaal:

So what Google decided to do is, Okay, we'll let you keep your kernel

Mishaal:

drivers closed source, but instead you gotta ship it as a kernel module that

Mishaal:

interacts with the generic kernel image in a standardized way, specifically

Jon:

built kernel module that interfaces a specific way using gki.

Mishaal:

Right?

Mishaal:

So on a modern gki device right now, so like the boot.

Mishaal:

It has the, the generic NEL image.

Mishaal:

So as Naqui mentioned, you can go, literally go and download Google

Mishaal:

signed generic kernel image, and that's going to be the same kernel

Mishaal:

that ships on other gki devices.

Mishaal:

But what actually enables the kernel to talk to the device specific hardware?

Mishaal:

Is those kernel modules that are specific to that device, and

Mishaal:

that's contained on a separate partition from the boot partition.

Jon:

And that's automatically detected and loaded as soon as Android

Jon:

zygote loads on the boot to process.

Nikhil:

It's, It's interesting to see how we got here though, right?

Nikhil:

So you were kind of touching on that, David, but if you look at why

Nikhil:

Android is the way it is, I think fundamentally, The goal with Android

Nikhil:

was to appeal to developers, right?

Nikhil:

So the architecture itself is designed to make sure that

Nikhil:

applications are easy to write, right?

Nikhil:

So for example, instead of having to interface with hardware

Nikhil:

directly like Linux, you have hardware abstraction layer.

Nikhil:

So your application doesn't have to change depending on how the hardware

Nikhil:

features are implemented per device.

Nikhil:

But that means.

Nikhil:

Device vendors have to put more time and effort into defining

Nikhil:

these hardware abstraction layers specific to their devices.

Nikhil:

And when they put in that effort, they don't want to put all that information

Nikhil:

out there for the world to see.

Nikhil:

So now you have people moving a lot of hardware specific code

Nikhil:

from the kernel where it would typically reside in a regular device

Nikhil:

to vendor binaries like blogs.

Nikhil:

Ah,

David:

so it's kinda a shell game thing that

Nikhil:

they're encouraging.

Nikhil:

Yeah, it really is.

Nikhil:

Yeah.

Nikhil:

And so now you.

Nikhil:

The idea of close source as much as possible with like minimal changes

Nikhil:

to the kernel, which makes Android as an OS very highly device specific.

Nikhil:

. And so you don't have the concept of a generic, Well, you didn't have

Nikhil:

a concept of a generic OS image like you do with Windows of Linux.

Jon:

Yeah.

Jon:

There is no one size fits all anymore.

Jon:

And I guess you, but there kinda is.

Jon:

That kind is, and there kind isn't.

Jon:

You can't take a brand new device that has no information provided

Jon:

for it from the manufacturer and get it working like you can with Linux.

Jon:

Linux.

Jon:

You'd be able to take what we worked on this last HP desktop for example.

Jon:

And the Linux kernel that has all that hardware, the monolithic kernel, and

Jon:

it'll automatically mod probe and look for any existing hardware, and that

Jon:

can probably get 90% of it working on a new device with a new chip set.

Jon:

That is not doable with Android anymore though, and that's one of the downsides.

Jon:

I

David:

mean, there's also of the business and partner model of

David:

Android is very different from like Microsoft and Windows, where

David:

Microsoft says, You wanna work on us?

David:

You gotta talk to Windows daddy and submit your driver for

David:

signing . Like there's no other way.

David:

Like that's how you make things work on Windows, Whereas Android.

David:

There are so many vendors at so many different layers, and also you

David:

have vertically integrated vendors who are doing things that multiple

David:

vendors would otherwise be doing.

David:

Whereas with Windows, Microsoft had the clout in the authority to say,

David:

You will build a Windows computer like this, Android and Google.

David:

They've had less authority, I think, until recently, the last five.

David:

But how

Jon:

far innovation's gone by letting the OEMs have that ability?

Jon:

Totally.

Jon:

With Android devices, I mean, we've gone three, four times

Jon:

what Windows was able to.

Jon:

Right.

Jon:

Yeah.

Jon:

And I

David:

think, you know, one of the, where I was going with this, and I will stop

David:

after this, is that smartphones come from a much more dedicated device mindset.

David:

They come from telephones.

David:

Telephones had a very limited number of things.

David:

They just started doing more and more and more of them, and so

David:

the smarter telephones got, you need a firmware on them, right?

David:

They got smart enough that you had to have an operating system under

David:

telephone, and so they come from that.

David:

This is a device with a list of features that does these things.

David:

You rely on it for that.

David:

Windows comes from, this is a computer, you use it to make stuff.

David:

And I think that actually informed probably the way manufacturers

David:

look at firmware for these devices.

David:

They probably looked at, especially early on as fixed in place.

David:

I make a product, it has firmware, it stays stable, which is in our

David:

world something many people expect.

David:

But in the consumer world, especially when it comes to general computing devices, no.

David:

They expect they get better.

David:

And Android has started to do that with smartphone.

David:

Obviously, but that's a pretty big sea

Mishaal:

change.

Mishaal:

I.

Mishaal:

Yeah.

Mishaal:

David, you'd mentioned that you know, it's true that Google has less of an

Mishaal:

iron grip over a s p and Android than Microsoft does over Windows, but they

Mishaal:

still do have a very size of control.

Mishaal:

Yeah.

Mishaal:

Especially when it comes to requirements to getting nude Android versions up and

Mishaal:

running, or certified on an older device.

Mishaal:

So for example, like.

Mishaal:

It's true that Google is genericized Android to the point where you can take

Mishaal:

an A O S P GSI for like Android 14, for example, and you could boot it on a device

Mishaal:

that originally launched with Android 12.

Mishaal:

But can you do that with Android 16?

Mishaal:

Maybe not, because Google tends to only support backward compatibility

Mishaal:

for three letter upgrades.

Mishaal:

So like if you have a device that launches with Android 12, Google will make sure.

Mishaal:

That Android 15 won't include any new requirements that break backward

Mishaal:

compatibility with Android devices all the way back to Android 12.

Mishaal:

But they don't guarantee any background fat ability.

Mishaal:

After that.

Mishaal:

It's generally just three letter upgrades and you can kind of see

Mishaal:

this extended across everywhere.

Mishaal:

Like if you look up, how long does Google provide security Backport to AO s p?

Mishaal:

It's generally only threes versions.

Mishaal:

So like right now, if you look on the security bulletin, they support

Mishaal:

Backporting patches to Android 10, because that's three letter

Mishaal:

upgrades before, but soon that'll stop once they launch Android 14.

Mishaal:

And this also goes into, it's not just becoming difficult

Mishaal:

to port a new Android version.

Mishaal:

Sometimes they'll just.

Mishaal:

Entire features that you might depend on.

Mishaal:

So for example, I think Lineage, when they released their Android 12

Mishaal:

builds, they had to drop a ton of older devices because those devices

Mishaal:

didn't have a Linux kernel version new enough to support EPF F, which is like

Mishaal:

the Berkeley packet filter feature.

Mishaal:

It is like a kernel feature that's usually for like network filtering

Mishaal:

or firewalls and stuff like that.

Mishaal:

And so they had to drop a ton of devices because of that, because

Mishaal:

Android straight up acquired it for Android 12, and those older devices

Mishaal:

with older carnivals just couldn't.

Nikhil:

It's interesting to look at Android as sort of a pre-market, right?

Nikhil:

So you have Google try to enforce it like the government with their GMs

Nikhil:

requirements and the compatibility definition document and whatnot.

Nikhil:

And then you have the whole AOS fee space where there's no laws, It's

Nikhil:

people doing whatever they want.

Nikhil:

ODMs, they don't have to follow any rules, right?

Nikhil:

And so how do you enforce.

Nikhil:

The concept of a standard in such a space where essentially the

Nikhil:

goal is to cut as many corners as possible to a working device.

Nikhil:

And we talked about how Android is more rigid in terms of as an operating system.

Nikhil:

That really helped here, if you think about it, because what's

Nikhil:

the goal of an Android device?

Nikhil:

Right?

Nikhil:

It should run all the Android apps that are available, right?

Nikhil:

Right.

Nikhil:

So if apps don't.

Nikhil:

Then your device doesn't work.

Nikhil:

And so because apps rely on this nice layer of communication, the how and

Nikhil:

the interface between the system and the hardware, you need to write that

Nikhil:

well enough where apps don't break.

Nikhil:

And so it's an interesting problem for Google to solve because.

Nikhil:

They can only do so much in the aosp space and for an open source operating

Nikhil:

system with tons of people using and tons of vendors and tons of manufacturers.

Nikhil:

It seems like it's worked out almost as well as it could.

Nikhil:

And

David:

I think to Google's credit, honestly, the fact that they

David:

seem to have the confidence that leaving AOS Aosp out there is not

David:

something that's going to hurt them.

David:

It's something.

David:

The changes they make in proper Android are going to percolate basically into the

David:

way people use AO s p, because guess what?

David:

The ecosystem is gonna move in that direction.

David:

If you're Qualcomm or media tech or even rock chip, you are going to

David:

encourage then ODMs and OEMs to do things because you have to do them.

David:

So there is definitely rising tides kind of effect sort of

David:

thing, you know, follow the leader.

David:

But yeah, I mean, aos P obviously has led to so many different innovations.

David:

Car OEMs build their infotainment on Aosp and have for years because it was

David:

a much more reliable and lightweight way to do it than building Linux.

David:

Like it was just like, Oh, this is design work with Bluetooth and wifi and all

David:

the other like kind of communication standards you would expect of a modern os.

David:

And that is because the Android was designed to be totally wireless connect

David:

with all standards of humanly can.

David:

Just generally be pretty flexible that way because Google wanted

David:

OEMs to experiment, so I do.

David:

You

Nikhil:

mentioned the app ecosystem, right?

Nikhil:

Right.

Nikhil:

They don't have to build anything.

Nikhil:

There's already a huge, probably the largest app ecosystem out there that

Nikhil:

you don't even have to go through Google certification to access.

Nikhil:

Right.

Nikhil:

You could just side load an apk.

Nikhil:

Right?

Nikhil:

And you have YouTube running on your OSP head unit and your card.

Nikhil:

I think the modularity and the openness of a USP is why it's super popular.

Mishaal:

Well, just a small caveat on that, a lot of apps will require

Mishaal:

your device to have GMs or Google Mobile services in order to access

Mishaal:

the APIs provided through it.

Mishaal:

Our previous episode on Google Play Services covers that in

Mishaal:

detail if you wanna learn more.

Mishaal:

And yeah, I also wanted to kind of caveat something I mentioned before.

Mishaal:

You know, I mentioned that.

Mishaal:

Google makes it difficult to support new Android versions.

Mishaal:

They drop support for older features of, they update their

Mishaal:

requirements after three letter up versions, but it's not impossible.

Mishaal:

Like David mentioned, the Aosp is a wild, wild west.

Mishaal:

You can do whatever you want.

Mishaal:

If you have the technical knowledge and expertise, you can bring up Android.

Mishaal:

For as many versions as you want.

Mishaal:

If you have extensive Linux kernel experience, you could keep porting newer

Mishaal:

versions of Linux onto an older device.

Mishaal:

Like it's not impossible.

Mishaal:

It's just really, really, really freaking hard.

Mishaal:

You can

Jon:

also do both and back Port Linux modularity from a monolithic kernel

Jon:

into a newer Android type build.

Jon:

And this is a lot of what we see happening with our work here at

Jon:

Sper and elsewhere in the community for all like Android X 86.

Jon:

Project Way, droid, those types

Nikhil:

of projects.

Nikhil:

Yeah.

Nikhil:

And with the community, you see these exceptional cases, right?

Nikhil:

Where you have devices that were launched with ice cream sandwich running

Nikhil:

Android 13, or dine running Android 13.

Nikhil:

It's, it's insane.

Nikhil:

And how these developers get Android, like a new version of Android

Nikhil:

came out 10 years after the last update was issued to that device.

Nikhil:

Working as if it was built by the oem.

Nikhil:

That's the beauty of open source.

Nikhil:

Hey man, my

Jon:

HTC Hero was updated for like 10 years, thanks to the

Mishaal:

community.

Mishaal:

Oh, you remember the HTC HD two, that legendary phones that would never with the

Nikhil:

slide.

David:

That was one of those

Jon:

at first came with Windows Mobile, I think.

Nikhil:

Yep.

Nikhil:

The device I've worked with a lot was the Sony Xperia Z one, and that beast was up.

Nikhil:

I think the last update Sony gave it was 5.1, so Snap Track in 8 0 1.

Nikhil:

800.

Nikhil:

800.

David:

Wow.

David:

800,

Nikhil:

okay.

Nikhil:

The first cryo chip.

Nikhil:

Yeah.

Nikhil:

Last I checked it was Android 12.

Nikhil:

Right.

Nikhil:

So from five to 12, this people have been, uh, I think I was involved all the

Nikhil:

way until Android 10 and after that other people have taken over and just, uh,

Nikhil:

kept updating and Android 11, Android 12.

Nikhil:

In fact, the whole Sony platform is one of a great example of the

Nikhil:

communities coming together and just giving life to a legacy device.

David:

And you know this, this kind of gets into the question of like,

David:

why can't we extend Android forever?

David:

And I think one of the reasons is Android, unlike Linux and Windows

David:

does not run on devices that are like easily air gapped from the internet.

David:

Android doesn't really work without the internet.

David:

Let's be real.

David:

An Android device without internet is kind of a brick, unless you wanna use it as a

David:

word processor or calculator or something.

David:

That means though, you have to keep up with the internet, and that's like

David:

Michelle, you bring up that part of Linux NEL for firewall, like packet handling.

David:

That is an example of things that you just like, well,

David:

somebody would have to build it.

David:

And nobody's gonna do it.

David:

And that's just kind of part of the ever evolving nature of

David:

connected and cloud technology.

David:

If your device can't talk to the systems that it needs to to do

David:

those things anymore, you get stuck.

David:

And that's what happened with the HTC Hero, I believe.

David:

Basically, it couldn't talk to the Google servers anymore to like get

David:

like wifi credentials basically, so it wouldn't get certifi.

David:

You'd have to connect it via ethernet cable over USB otg.

David:

And even then, that didn't really work, right?

David:

Like we had Ryan, one of our editors tried to do this at AP and

David:

it was just a nightmare to try to even get this phone to load Gmail.

David:

But yeah, it's different than a Windows computer.

David:

You can buy a 20 year old Windows computer, open up Internet Explorer,

David:

and probably getting parts of the internet to mostly load or

David:

download Chrome, whatever, you know.

David:

Well, I think we are at about time.

David:

This was a wonderful chat about, you know, I mean, I learned a lot in the last hour,

David:

quite frankly, about the practicalities of building A O S P, which is really

David:

at Asper, what we do on a daily basis.

David:

Take things that run what you wouldn't probably Google wouldn't call Android,

David:

but that for everybody else's purposes, Yeah, they run Android and there are

David:

some things that you do have to change.

David:

Now John, um, works on our X 86 foundation product and obviously that

David:

is so unusual in the world of Android.

Jon:

We are doing, it does use a monolithic nel, so that's the project that

Jon:

melds everything together and some mashup.

Jon:

And I had no idea.

Jon:

Best of Linux, best of Android, and we make

David:

it work.

David:

Peanut butter and jelly.

David:

So if you are interested in learning about building from AO s p, and

David:

like what it takes to make a device or to make a device work the way

David:

you want, come talk to usper.io.

Mishaal:

All right, thanks David.

Mishaal:

And thank you John and Nala for joining us and uh, we do this with all our guests.

Mishaal:

We wanted to ask you now, like, where can people follow you if they

Mishaal:

wanna follow your work online or see where you tweet or do whatever?

Jon:

I'm on Twitter for the most part, so Twitter at E L E c T r i k J e s U

Nikhil:

S and not on Twitter.

Nikhil:

So and I'm.

Nikhil:

I'm kind of thinking, where Can't you follow me?

Nikhil:

I don't know.

Nikhil:

GitHub.

Nikhil:

Yeah, I

Mishaal:

was gonna say

Mishaal:

. Jon: There's that too.

Mishaal:

I'm also in GitHub.

Mishaal:

Same place.

Mishaal:

Well, maybe it's good that you're not on Twitter because, uh, it's a

Mishaal:

place where, uh, nuance goes to die.

Mishaal:

All right.

Mishaal:

Well, this show is a place where Nuance doesn't die.

Mishaal:

We love talking to experts every week.

Mishaal:

You know, every other week, kind of a regular schedule right now about Android,

Mishaal:

A O S P, the whole ecosystem and platform.

Mishaal:

So thank you again, both of you, for joining us to talk about,

Mishaal:

you know, device bring up.

Mishaal:

And I hope those of you listening learn a thing or two about this.

Mishaal:

And if this sounds interesting to you, as David mentioned,

Mishaal:

come check us out@esper.io.

Mishaal:

Now show some interest.

Chapters

Video

More from YouTube