Navigating the complex waters of Drupal migration can be daunting, but the latest episode of Tag1 Team Talks demystifies the process, offering invaluable insights for developers and IT professionals. Drupal experts from Tag1 Consulting, including Mike Ryan and Benji Fisher, delve into the transform transformation phase of the ETL (Extract, Transform, Load) process. They compare the unique, row-by-row approach of Drupal's Migrate system with traditional bulk processing, highlighting the flexibility and customization it offers for various data sources.

This podcast series is a deep dive into the mechanics and challenges of Drupal migration, and this episode addresses the crucial role of the transform transformation phase. Here, data is reshaped and prepared for its new home in Drupal. The experts discuss performance considerations, stressing the impact of transformation efficiency on the overall migration timeline. They also explore Drupal-specific nuances, offering practical tips and strategies for a smooth transition.

Welcome to Tag1 Team Talks, brought to you by the Tag1 Consulting.

With Drupal 7 rapidly approaching and Drupal 9 already end of life, we are

hearing people talk about migrating and upgrading more than ever before.

And anyone who's ever been involved with a large scale migration, Migrating

a large site or application from one technology stack to another will

tell you that it's complex, time consuming, and it demands expertise.

That's why we're bringing you this series of talks.

Diving deep into the world of Drupal migrations.

And who better to guide us than Tag1's very own Drupal migration experts.

From the masterminds and maintainers of Drupal's migration tooling to the

individuals behind the most groundbreaking Drupal migrations, we've got an all

star lineup who'll cover everything you need to know about every aspect

of migrating large scale applications.

This team talk is part of the three part series about ETL, extract,

transform, and load process, which is used by many enterprise migration

systems, Drupal's Migrate included.

In today's episode, we're going to talk about how to use Drupal's Migrate

system to transform the data before loading it into the Drupal's database.

Be sure to stick around to the end because we are going to announce

the next few talks in our series.

Let's dive in.

I'm Janez Urevc , senior engineer here at Tag1, and a

longtime contributor to Drupal.

I'm joined today by, well-known top contributors to Drupal, Benji

Fisher, one of the five current Drupal Migrate core system maintainers.

And Mike Ryan, co-creator of Migrate.

Welcome.

Thank you both for joining me.

Thanks for having us.

We're glad to have you.

Before we dive in, I would just like to mention that in case you didn't

already watch or listen to the previous episode in this series about

E, Extract, we'd suggest that you do

in that episode, we, among other things, provided a high level overview

of what ETL stands for, so we'll not repeat that in this episode.

Now, finally, let's dive into today's topic, which is T Transform.

Mike, could you tell us what is being done as part of the transform phase

in general and how Drupal does it?

Is it like similar to how other enterprise systems do it or are

there any specialties to it?

One difference from your classic ETL is the classic ETL usually goes in bulk.

You extract all your data into a big blob.

Then you run it through a transformer, which transforms everything.

And then you run it into a loader, which does a bulk load.

ouR approach is to run through the data one logical row at a time.

We say row because most often we're dealing with databases as our sources.

But.

Technically, it could be anything like any form like a web service or CSV.

Basically, we Use the Drupal plugin system.

And you can, for a given pipeline, each field that is being run through

the pipeline um, can go through any number of transformers because

they're plugins, it's very flexible.

There's a YAML format you can use to write your migrations, which

specifies for each field, what plugins it's going to transform with

and whatever configuration you add.

So it takes the output of the source plugin and all the source plugins,

regardless of the source CSV, et cetera.

produce a common data structure, which feeds into the transformer.

And the transform pipeline will take one row from that.

And it will go through each piece of the process.

The transform step, uh, we call it process in Drupal and apply the transformers.

And each transformer can take actually multiple fields from the

source row, or it can take none.

You might use a processor that simply sets a constant value.

And the transformers can be very flexible and for the most part,

they're not very Drupal dependent.

You'll do a lot of string manipulation, for example.

Let's see, I'm not sure what else there is to say about the general process.

But

One thing I'd like to add at this point is that sometimes we cheat.

We don't strictly follow the ETL paradigm.

But we take a peek at the database.

So for example, we might look for an existing taxonomy term that

has the name dessert, or we might check how the editor is configured.

So that's one way in which it's very Drupal specific and doesn't

strictly follow the ETL paradigm.

You have complete flexibility.

You can do anything you want.

Good or bad.

Like always.

And maybe while we're mentioning bad things to do in these processors.

It should be noted that this pipeline is run for each source row

in your data, and when dealing with multiple value fields, it might run

several times for one source row.

So the processing pipeline is a key place to watch performance.

One, one slow processor will kill the overall migration process.

Makes sense.

Cause it could be run a lot of times and that adds up, right?

Oh, thousands, millions.

Yeah.

And it can take days as we discussed before.

So Benji, I heard you state in the past that the transform stage is the most

interesting part of the migration and I know for a fact that you are probably

the most excited about it in the whole.

ETL migrate world.

Why is that?

Yeah you're right.

And this is something that I decided when I first started working for migrations.

And by the way I am the most junior member of the current maintainers and

And I have a lot less experience than most of them, or than Mike, so I defer

to Mike on questions of experience and performance in large scale migrations.

But I do have some pretty strong opinions about the transform

stage, the process plugins.

The first reason that it's the most interesting is that any migration

project will be broken up into a bunch of different migrations.

And each one of those migrations will have a single source.

and a single destination.

But any one migration has many fields.

So if you have a migration for your article nodes, it'll have a body field.

It'll have a couple of timestamps.

It might have taxonomy and images and so on.

Each one of those fields is going to have at least one process plugin.

One transformer, as Mike described them, and some fields will

have several transformations.

iN that sense, it's it's where the most variety is.

One, each migration again has one source plugin, one destination plugin,

but can have many transformation plugins or process plugins.

The second thing is that the transform stage, the process

plugins are where you have the most opportunity for reasoning your code.

So if you look at the source plugin, it has to understand whatever cruft

is involved in your source data the site you're migrating from.

And the only time you're going to be able to reuse a source plugin is

if you have the same type of source.

So once you've written a source plugin for a WordPress XML file, you can reuse that.

And once you've written a source plugin for Drupal 6 or

Drupal 7, you can reuse that.

The destination plugin, almost always migrating into Drupal entities.

They could be taxonomy terms or nodes menu links are entities,

um, and the core migration system already understands the destination.

So that's already done.

bUt getting from one to the other is in my opinion, the interesting

part and the part that has the most opportunity for reusing code.

So that's why I think that the transform stage is the most interesting.

Yeah it's, it, for most migrations, you'll find that the source and the the extract

and the load phases you simply need to use

core plugins and some configuration.

You don't usually need to do very much PHP coding.

It's the process plugins where you're most likely to need to write your own

plugins, write your own application logic, because that's where,

you're transmogrifying your data.

You can do the new system.

Although there are some people who prefer to do it all in the source plugin.

They'll just write all their custom PHP there and prepare everything

so that it's ready to be imported.

And again, I don't like that approach because it, you can't reuse

the code if you do it that way.

Yeah, it makes it way harder to reuse it.

It's also against the ETL paradigm, because then you're...

Basically throwing away this separation of different phases that

we're trying to introduce here.

So to be a little bit more concrete, what would be the most common

transform operations in a migration?

Like what would we do in transform process plugins?

Yeah, so by far the most common is just a straight copy.

You have a text field, and you pass it over to the new text field,

which often has the same field name.

Sometimes you decide to change that as part of your site redesign.

That's the most common.

And, that's almost not using a transform plugin at all.

It's technically using the get plugin, but it's not doing any transformation.

Another common thing is that your source has a comma separated list of values,

and you split that into pieces, and you convert each word into a taxonomy term ID.

sO that's something that comes up pretty commonly.

aNother really important one is since Drupal deals with structured

data, you might have references to other nodes, other taxonomy terms

identified by their entity IDs.

And if those entity IDs are changing as they often do in a complex

migration, then you have to translate the old entity id, the ID on the

source system to the new entity id.

And that's possible because the migration system keeps track

of the old and new entity IDs.

So that, that's a really important one.

Some other things you might wanna do is, make your site

better as you're transforming it.

So if you see that people are consistently using CSS classes, font dash bold, size

dash large, color dash red well, you can replace that with my theme dash warning.

And suddenly your CSS markup is a lot more semantic and a lot

easier to maintain in the long run.

Another common one is to convert date formats, like maybe they're in year

month day format, and you want to convert it to a timestamp or vice versa.

And then there are a whole bunch of utility operations.

And you wouldn't think of these as the things you want to do to your data, but

they're the things that end up getting used, in the middle of the process.

So flatten an array, combine several arrays into one, filter out empty

values, or apply a callback function.

So those I think are the most commonly used process plugins.

Mike, am I leaving anything out?

Yeah, I think that those are the key ones.

I'm looking now at the list of all the ones that are in core and

maybe we might want to highlight a few other interesting ones.

here?

While you're looking at it, I just wanted to comment.

The callback one is an interesting one because it almost

lets you cheat a little bit.

If you're, if you need to introduce your custom logic, but you don't want to

create a plugin and go through all that.

You can always use a callback existing callback function process plugin,

and then just create a function in PHP, which will be called.

Or use a basic PHP function.

Or use a basic PHP function.

yOu don't need to wrap trim in a plugin.

You simply use callback, specify trim, it's the callback, and boom, you got it.

Maybe this is a good time to mention that our show notes include some

links to the documentation where we list all of the plugins that are in

core, and those will be available on our pages after we publish this talk.

ONe of the interesting ones is static map.

This is...

It basically, it's like translating enums that is if the source field contains a

finite list of distinct strings and those need to be different on the Drupal side,

you use a static map plugin, which says change this string to that string, and

that's a very handy in a lot of cases.

Or if you're dealing with NFL team names and the Redskins are now the

commanders, you can say that this finite list of names has changed and

anything else you pass through unchanged.

Yeah.

And the Cleveland Guardians in baseball.

I'm seeing sub process, and that is one of the more complicated ones that allows

you to do some really complicated things.

When a field consists of a list, an array, and it allows you to basically

have a sub process pipeline for the pieces of this source field.

And this is very complicated and technical.

I'm not going to go through it right now because...

I always have to relearn it

when I need to use it.

Will a subprocess use the same set of plugins as the main migration?

With source and transform and all those things?

Oh, no.

The source, rather than being a row from your source plugin, At

the source is the contents of the field, the extracted field.

So it's used on fields, which themselves have structure.

But it does use or have access to all the same process plugins that

the general transform stage has,

Which obviously is immensely powerful then..

It is.

It is.

In theory, instead of.

Migrating your taxonomy terms that's a bad example because we've got a

shortcut for that, but user accounts, in theory, instead of doing them

in a separate migration from your main content migration, you could

do them dynamically within a sub process within your content migration.

We do not recommend that, like I said, that the process plugins are very

powerful and you can cut yourself.

Um, there, there is a plugin for copying files, which, if you're going

from one system to another, an old version of Drupal to a new one uh,

you want your images, your videos, your documents to come across too.

And the file copy plugin is very flexible because it gives you.

feW different options for doing that and for doing that performantly.

For example, it could simply copy it into Drupal's public files directory.

And it can keep track.

You can set a flag on it.

So that if the file already exists at the destination, you don't overwrite it.

And that's great for your performance when you're rerunning migrations,

especially during development.

You can also,

now I'm trying to remember the other, but of course you could use it to copy

directly into the files directory.

You could copy it to an S3 bucket.

Or some other, storage service, uh,

Or I think that I had a use case in the past where we needed to copy files from

like another website into our local file system as part of the migration.

And I think that file copy was used for that as well, which

is obviously terribly slow.

And a little tidbit is that the file copy relies on the download

plugin if the source is remote.

And that uses Guzzle in a way that's slightly different from anywhere else

it's used in Drupal Core and caused some interesting test failures years ago.

Yes, you have, sometimes you have to be clever to make things

work and work performantly.

Yes.

I also remember...

During one of the migrations I was working on, we used file copy to copy

files straight from NFS probably to another public sites folder, public files

folder and we were actually copying files and that slowed the migration a lot.

And then we figure out that it's better to rsync before running the migration

and then this, check if the file exists kicks in and you don't need to

copy, you just find it there and that sped up the migration significantly.

But we're getting into performance considerations now, which is

another talk in the future we will be doing, what about contrib?

What kind of interesting process plugins can we find in contrib

that are not part of core?

Oh, so many.

So many, I need to jog my memory here and take a look.

So the Migrate Plus module is an add on to the core migration system, and

it has a number of interesting ones.

There are

several several plugins for manipulating and and scanning

a DOM document object model.

So you can scan your HTML or XML and find, extract, the span with the

TextBold class that's in the, underneath a P, for example, if you need to

manipulate that piece of your content.

There is entity lookup entity generate that makes it easy

to find a matching entity.

That's not one necessarily one that you migrated and what you can find via

the map tables that migration provides.

But if you're migrating in the system, you've got a, maybe a

taxonomy there you want to hook up to.

You can use entity lookup.

To find a matching term in that vocabulary and link to it.

And you can also use entity generate, which does the same thing, but also

if it doesn't find the matching term would create it for you.

lEt's see, there's file blob.

If you've got file data in a database blob, you can convert

that to a real file with that.

File blob reminded me of tHe beginning of my career, which predates

Drupal, where I had experience with proprietary CMS that was really into

storing all files in the database.

That was fun.

Yeah.

So those, and those are the ones that pop out to me immediately.

besides migrate plus, which is a grab bag of several different, uh, plugins there

are several other contributed modules.

That have plugins of all sorts.

And before you go writing your own plugins, take a look through the contrib

modules that are available on drupal.

org and you might find someone that's already solved your problem.

Maybe they've got a SOAP plugin.

Actually, I know they do because I wrote it, but whatever your scenario

is, assume you are not that unique.

Until you prove you are.

And Migrate has been around for years, like probably more

than a decade at this point.

And it's migrated many enterprise.

Large scale applications.

So I'd be almost convinced that if there is a use case, it has

probably already been done.

Yeah, it started out as a contributed module.

Mike and Moshe Weitzman developed it in Drupal 5 or Drupal 6?

6.

Okay.

It was 6.

I think we may have started trying it for 5 and we jumped ahead to 6 because 6 was

It's more conducive to what we were doing.

Yeah.

My first experience with it was the Drupal 7 version.

Yeah.

The first big project the first client you would know was the Economist.

Economist.

com way back in the day.

Which I think also sponsored a lot of initial Migrate module work, right?

Yes.

Yes.

After Economist, it was examiner.

com, which all of them are sponsored.

A lot of D7 work, right?

Yes, they sponsored most of our Port D7.

Martha Stewart Living was about that time too.

Speaking about history, what, do you have any anecdotes or any interesting

or unusual process related use cases that you've experienced in the past?

Oh, boy, they're all jumbled together.

One I don't want to remember is the time her client thought we had created

a major security breach because uh, our development migration suddenly started

sending out emails to all their customers.

And this was the migration, and this is something to watch out for.

The migration system actually explicitly disables, uh, the mail system while

running uh, which we thought was safe, but what happened was a module was enabled,

which during entity creation, which happens during migration queues emails

to be sent and it was fine for a while because this was a development system and

that no one saw but a one little ping on port 88 to that system caused Cron to run

it was using the lazy cron or whatever you call it boom Those, yeah, so those emails

started going out and caused quite a stir.

So yes, you this is something you do need to be careful about is um,

effects, whatever effects the ultimate website might have beyond itself.

Be careful that you control them within your development and testing system.

Which is good advice in general, not just for.

Yes.

And this is where I find DDEV, which is one of the projects that we are

really excited about at Tag1 really useful because I believe that DDEV will

reconfigure your development environment to redirect, uh, emails into like this.

MailHog.

MailHog.

It basically redirects everything in there and it stays just in memory even, I think,

so if you are inside DDEV with regards to mails, you can be pretty sure that

no matter what's going on, you're safe.

And it's handy too for testing your outgoing emails, testing

the formatting or whatever.

Yeah, exactly.

That's probably the usual use case when it was created.

Yeah.

But it, you'll see as a side effect, it also provides a layer

of security and peace of mind.

I, there, there are two points here.

I want to make sure we don't lose track of them.

The first is that while you're developing, no emails will be sent out.

But the second one is equally important.

You have to look at the emails that did get captured by MailHog

because those are the ones that will be sent out in real life when

you're on production and not local.

Yeah, make sure your tokens are being substituted, all that stuff.

Yes.

Benji, what about you?

Do you have any interesting stories from the past?

Yeah.

And, I'm really flattered that when Mike was going through the list of plugins in

Migrate Plus the first ones he singled out were the DOM processing plugins,

because that was one of my contributions.

And And let me call out, this was on a project for Pega Systems, and I was

working for Isovera at the time with Marco Villegas, and we developed the

first DOM processing plugins, and both Isovera and Pega were supportive of

contributing that back to Migrate Plus And I guess the original problem I

was trying to solve is that as I said earlier the node IDs were changing.

So if we have separate entity reference fields, Drupal could already handle that

and just using the migrate lookup plugin.

And you could say that the next article used to be node one, two,

three in the migrated system.

It's node four, five, six.

You can do that translation, but what if you have a text field and

inside that text field, there's an anchor link and the anchor href,

goes to node slash one two three.

How do you translate that to node four five six?

And part of the answer was to use proper DOM processing.

And so I realized I guess I, I had the idea on an earlier project

and I didn't get to make it happen until it was actually needed here.

Everyone knows that you shouldn't be processing HTML with regular expressions.

But people do it anyway.

Yes.

People do it anyway because it's the tool they know it's convenient.

And so the first step was to introduce some process plugins to make it

easy to do proper DOM processing so that you have less overhead in

creating that DOM document object and the XPath object and so forth.

Once you've eliminated the overhead, it is often both simpler and more reliable to

do the proper HTML processing rather than to do things with regular expressions.

In fact, if you look at the search API module, um, I just ran into a case where

it's not only simpler and more reliable, it's also more performant to do processing

and there's there's an open issue on the search API module that that handles that.

So anyway that was my original purpose for putting the DOM processing

plugins in on that project with Pega.

Since then, um, some other people have done work on that.

There's a contrib module that builds on the DOM processing plugins, and it

handles if you have the media module on your Drupal 7 module, on your Drupal 7

site rather, and you want to migrate that to the core media module in Drupal 10.

It understands the tokens that the Drupal 7 media module used and and

handles transforming your text fields.

I had a really complicated project where we not only had to migrate

the site Um, from Drupal 7 to Drupal 8 at that point, I think it was.

We also had to import some really complicated XML documents into the site.

And that project gave me a real appreciation for the expressive

power of XPath, because that was the only way to manage these

really complicated XML structures.

And luckily we already had the DOM processing plugins available.

another complicated project I had was that we had these HTML text fields and and

each text field had just some image tags.

So basic HTML markup and we wanted to download the files from those image

tags and save them as files and create media entities out of them and then

just insert the media references.

Into the text field.

I did that with a custom PHP plugin.

And I do want to point out that this is thing where you can

shoot yourself in the foot.

It's not following the ETL paradigm.

You don't have a row creating each one of those media items.

And it does have certain disadvantages because it breaks the ETL paradigm,

but it is a practical way to handle that sort of situation.

Another weird one I had was...

a Single HTML page, um, in the source site was Drupal 7 and you would think looking

at this page, oh, this page is a view.

It's listing the person content type.

But in fact, it was just a basic page and all the markup was

just there in the body field.

And we wanted to pick it apart and create person nodes and then

create a view of the person nodes.

And so luckily the markup was consistent.

It always started with an H3 tag.

It had a title and that was immediately followed by an image tag.

So there was that consistency that I could take advantage of.

I Extracted the title into a text field.

I extracted the image, created a file media entity, and then and then just

stripped those from the body field.

and let an actual view in the Drupal, I think it was Drupal 9 at that point let

that put the pieces back together to make something like the original source site.

And the last one that I noted down was again for Pega and we

were importing documentation from an external XML based system.

So this wasn't a site migration, this was a recurring migration that someone

was writing the documentation in this external system, and we had to import

it into the Drupal site and make it look like it fit the rest of the site.

And.

And that's where I did the sort of thing I talked about before, where you look for

some consistent pattern of CSS classes and say, okay, we're going to replace

that with something more semantic.

And um, and again this used the DOM plugins.

And it also peeked at the current database the destination database to

see how the editor module was configured so that we could pick and choose

the CSS classes that were that the current site editors would naturally

be adding through the user interface.

And we added those same CSS classes programmatically through the migration.

So those are some of the more complicated cases I've had in the transform stage.

Very nice.

I guess that this brings us to the end of today's episode, unless

you have anything more to add.

I'm good..

We have some great talks coming up.

Our goal is to put one per week over the next few months to support the

community in the migration process.

pErformance is something we care deeply about, Tag1, and we did touch

performance in today's episode a little bit because it applies to migrations.

When you're handling really large data sets, um, a full data migration

can take 12 hours or even days.

we'll do a handful of talks on this topic, including how to profile

and tune a migration, and a talk about incremental migrations.

Every project owner wants their migration to be a success.

So we will dedicate an episode to discuss the most important

factors for a successful Drupal 7 to Drupal 10 migration.

oTher topics includes porting custom code the future of migrate tooling,

how to port a theme, and so much more.

We hope that you'll tune in and enjoy our upcoming team talks.

A huge thank you to the Tag1 team, Benji Fisher and Mike Ryan.

Thank you for joining me.

Make sure that you check out the other segments in this series.

There will be links to them in the show notes, along with links to the

modules and documentation and other things that we mentioned today.

If you like this talk, please remember to upvote, subscribe and share it.

Check out our past talks at Tag1.com/ttt.

That's three T's for Tag1 Team Talks.

As always, we'd love your feedback and any topic suggestions.

You can always write to us at ttt@tag1.Com.

Again, that's three T's for Tag1 Team Talks.

One more time, big thank you to our guests and everybody who tuned in.

Thanks for joining us.

Thanks.

Bye.