The pipeline design pattern in software development, particularly within the Laravel framework, offers a flexible and maintainable way to process data through a series of independent, focused steps, enhancing code readability and reusability.
Mind Map
Click to expand
Click to explore the full interactive mind map • Zoom, pan, and navigate
please welcome Bobby [Applause]
[Music] [Applause]
Bowman extra special effect
right good morning Amsterdam well good
morning Amsterdam are we ready LaRon
well as K already mentioned I'm Bobby
Bama it's awesome being here it's
actually my 10th time speaking at
aericon so I'm very excited about that
and uh we get some extra smoke today
it's so good you won't see me that's
better for the
talk can you see the screen now
it's all good now it's all good so I
work as a product engineer at hospitable
uh we build software for vcation rental
owners our goal is to give them as much
time back as possible and we do this by
syncing calendars between platforms like
Airbnb and booking.com we give them one
Unified Inbox we use a lot of AI um and
much more stuff so um but that's enough
about me let's talk about
pipelines so as the name suggests
something is being passed along in a
pipeline um so you put data in it it
process it and then it returns or passes
it through to the next pipe and then
eventually it returns something um and
as you already see there's some pizzas
here we're going to do a lot of pizza
slides today um there won't be any
pizzas until maybe tonight I don't know
um You probably get
hungry so if we have a pizza here every
step is independent every time we add an
ingredient it's a new Step um and every
step can focus on a specific task so it
doesn't really care about the preview
step like it doesn't really matter if
you add the vegetable first or the meat
first you can do it in different
directions and eventually we will end up
with a pizza with multiple ingredients
when we translate this into code it's
very much like adding multiple items to
an array and returning it at the
end and a pipeline can also be a
sequence of actions on a certain object
so for example if you have an order you
have a package it's going from the web
shop to the warehouse then being shipped
so we're taking actions on that specific
package uh this is similar to performing
various actions on an object
and in this case we don't really change
the state of the uh object we just pass
it along maybe we give it an update
status so we can either do both you can
change the status of this object and
pass it along and take a certain action
so the pipeline pattern is very flexible
there so typically a design pattern is
uh a common solution to a very um a
common occurring
problem so the goal of the pipeline
pattern is to make sure that each step
is independent and it focus on those
specific task and does not worry about
the other tasks and what you want to
achieve is maintainability readability and
and
flexibility so you want to maintain your
code and you want to make sure that sure
that every step is well tested and
especially when it becomes more complex
if you just have to take one or two
actions then the pipeline pattern might
not be for you but if you have five or
six actions you have to take it becomes
more complicated and that's where you
want to maybe use this pattern and you
can write tests for each individual
step and what a nice thing about it is
that you also can reuse certain steps I
will give you examples of that
later I first want to give you an a deep
dive of the basics so how the API is
defined in larell and then I will give
you specific use cases around
that so a pipeline is always separated
in three different steps so we first
start with a subject so that's what we
do when we call send we pass in an
object or a collection it can be
anything that's what we call a
subject and for every pipeline that we
edit it can be one or many um for every
pipeline we the it will go through that
subject it will perform something on the
subject and then returns
it and eventually you will have the end
result of your
object and laral supports multiple ways
of passing in a pipeline so for example
you can pass in a closure uh that's what
the documentation for example shows you
as well using that you can just pass in
a class string and L will resolve the
class from the container which is very
helpful or you can just new up your own
class pass it in as well so you can just
pass in an object basically
um and then finally you will return it
so if we look at one of these pipe
classes you can see here that we have a
basic class it doesn't extend anything
it has a handle method the first
argument is the subject that it gets and
the second argument is the closure which
is called next then we perform our
action our on our pipe or in this case
our object we do something there and
then we return next with the object
again basically we're saying here pass
it on to the next
pipe and this approach makes it really
easy to swap the order of pipes because
we're just passing to the next pipe and
we don't really care what the next pipe
is because we assume that every pipe can
handle whatever the object is and do
it so that's the basic setup here um
laral offers a simple way of changing
how this works so as we saw before here
it's using the handle method which is
the default in larl but if you use the
FIA method you can set a different
method for example if you have action
claes which have an execute method by
default then you can tell the pipeline
hey use this method instead and
otherwise it will by default search for
handle or if you use an invocable class
instead and then laral has two options
so we have a den method which will give
you a closure with the end result and
you can do something with it or you can
return the result so then return it's a
very self-explanatory method name um I I
personally like to use then return more
than then but um yeah you have two
options here so it's very
helpful and finally there's an option to
dynamically add more pipes to an
existing pipeline so you can just start
a pipeline with a few things and
depending on your code you can add more
or less so this is
configurable um and larl has this
conditional trade that you can that is
being used on almost all classes within
larl but now it's also used on the
pipeline so with the when method you can
just put in something that returns a
Boolean and then if it's true it will
apply something on the pipeline so
that's very helpful or you can also do
your own if statement and just apply it
on the pipeline it both works it's just
I guess preference of syntax in the end
so this is one way of building up a
pipeline as
well so I think we have a good
understanding of how the API looks in
larl um you can try this out on your own
um you might have already been doing
that uh but today I want to focus mostly
on use cases when can you use this
pattern and when does it really shine
when is it really
helpful and then you can play around see
it as homework you can play around with
it um during the
breaks so laral actually using this
pipeline pattern as well inside the code
base um let's have a look at this so
this is the actual kernel class um and
you can see here it's creates a new
pipeline it passes in the application so
in this case the subject is the the
whole application and it sends the
request uh sorry it's the request that's
being is the is the subject so it sends
the whole request through the pipeline
and then the pipeline the pipes that are
in there are the middlewares so each
middleware that you have to find
wherever that's coming from it's looping
over those middlewares and then passing
on next and you probably have seen it
like a lot of times right you have a
basic middleware class it has a handle
method with a request and a closure so
this is actually the pipeline pattern
it's using it the same way and that's
why you always have to return next
request to make it work otherwise the
pipeline would stop and it wouldn't know
next okay some more
examples add hospitable We sync prices
between different kind of platforms like
Airbnb and booking.com and to do that we
need to make make sure that we have the
right price in the right currency um
with the right markup and with markup I
mean um for example you want to have
your prices on Airbnb a little bit
higher than on booking.com because the
fees on Airbnb are a little bit higher
so you want to have the same revenue
from both platforms so this is where we
need to apply a lot of different things
on a
price so if we look into this as code we
just start with value object we set a
base price and then we apply a pipeline
which will then help us to change the
current price to whatever ever we need
at that point in time so in this case
for example the um we start with $100
which is the flat we turn it into an
integer because it's easier to work with
then we apply the markup and then we
convert it to a different um currency in
this case it might be that it's going
from USD to Euros so it's using the
actual current exchange rate and
translated it to there so this is where
it's really helpful to have a pipeline
because it will just apply the steps and
if we for example in the future decide
that we have another kind of Mark
applying we just add the class there and
it just works you don't have to change
any code you don't have to uh change
anything in there so that makes it really
really
powerful and if we look at one of these
pipe classes uh you can see here we have
the handle method again um in this case
we're extending a base class because we
can reuse some things from there um but
we start with the value object we have
the back there we set the current
working price in this case we're going
from floats to integers so we're setting
that value and what the nice thing is
because you now have a value object or
dto we change something in there but we
still have the original price we just
set the current working price but we
still have the initial price set in
there so the next pipe has all this
information now it doesn't have the
initial price but it also has the new
working price so it becomes more
valuable to have this object so you can
also be smart about the order of pipes
where you just initialize some data on
the object and then start applying
things so that makes it very
helpful and it's very easy to write a
test for this specific case we can just
write one test that changes the price
and we can just check against the value
object if the actual price is set or not
helpful um at hospitable we also love a
lot about AI um and we use all we use it
for all kinds of things um but I'm just
going going to talk about messaging so
whenever we have a message um we have
the whole conversation of someone and to
give a correct answer we need a lot of
extra information and to do that um we
use the pipeline pattern to determine that
that
so this is a little bit of a simplified
version of the uh code otherwise it
wouldn't fit on the screen um but here
we build up all the data for the
pipeline so we start with a back it's
just an empty back with conditions um so
you can use an empty collection or an
empty dto in this case and we can add
all these classes here so we check are
there any pets involved are there any
children involved when is the last
message coming in all that kind of
information can be very helpful for
generating a message to the host or to
the guest
um and as you can see if we just um for
example they're requesting for a parking
spot well if we if we want to add that
to one of the messages we just add one
more class and it just works the prompt
will understand it
now so if we look at one of these
classes it's a very simple class we just
check are there any number of pets and
if that's the case then we add the
condition to the back so we keep track
of a list of met
isn't it annoying that
the the lid is still stuck to the bottle jeez
jeez
man um so let's do one more one more
pipe a little bit more
complicated so you can make these
classes as complicated as you like um it
can even dispatch a job it can do an API
call it can do all of those kind of
things um so I'm not doing too
complicated things otherwise it wouldn't
fit on the screen you wouldn't be able
to read it um but you can go as far as
you want with these kind of things um so
that makes it very helpful and what I
really like about the pipeline pattern
is that you can combine it with other
patterns like for example the if you
have action classes it's very easy to
inject an action class in a pipe and
just call the action and return
there so one more thing where the
pipeline pattern really shines is if you
if you have to do a lot of operations on
a certain set of data so let's say you
have an incoming request that either
creates removes um updates different
kind of things on a model and the model
is connected to multiple other relationships
relationships
uh in that case it's very hard to do
everything in your controller because
it's just a lot of
logic and then the pipeline can help out
there because you can Define each step
in separated actions action Clauses that do
do
something so for example we have this
update commission request so as you can
imagine updating a a commission a
commission always starts at a certain
start date and might have an end date so
there might be for some cases it might
be that it needs to create certain fees
or certain uh commission rates at a
starting point but that also means that
it might delete certain things at
sometimes so for example the end or
delete fees uh class might depending on
the end date delete some stuff or just
set the end date based on
that um so we don't really want all of
this logic inside our controller and
this is where the pipeline pattern
really makes it easy because you just
have input your request Clause with all
the data that you posted um you can even
turn it into a dto if you like that and
then you can just apply all your actions
that you need need to do on the database
to do this and then if you wrap all of
this in a TR catch or transaction it's
way so collections are very easy to use
with pipelines because it's basically
just an array on steroids so you can
keep call a lot of methods on it and
just keep passing on the collection to
the next pipe so usually you can start
with an empty collection like we did
with the prompt stuff we just have an
empty collection or we can start with
initial data uh but since it's Lile we
can easily create our own custom
collections which makes it very easy to
add custom methods which makes it a lot
easier to work with in
pipelines so let's say we have this
controller that Returns the health
checks of our application um in this
case we want to have a pipeline that
sends the health checks collection
through certain pipes and then returns
and in this case we're using the config
because we want this to be dynamic
depending on the environment we want to
do certain checks so this is where you
can combine all kinds of features within
the Pipelines and then uh we have this
result which is the health checks
collection we have a method for it has
failures because it's a custom
collection we can add these custom
methods to make it easier to read the
code and to apply the code as well so
that's when we return the code um and
then you can use um this for external
monitoring tools or use it as a health
check in your kubernetes cluster
anything that works for
you so this is the config for example uh
we have the health checks list and
there's a list here of the checks that
we're doing but you can also combine
this with environment variables or if
statements if you want to so it's very
flexible to do it this
way and then finally we have the custom
collection Clause you can easily spin up
these things right you just create a new
collection Clause extend the base
collection uh which is illuminate
support collection just add your own
methods and it just works and it's very
easy to work with so um yeah use that um
lots more examples to
go so if you are reading about pipelines
or Googling about Pipelines you will
probably see one or two um tutorials
about using the query Builder as an
object because basically it's just an
object you're applying new things to it
um and usually with the pipeline pattern
you're adding extra data to it but in
this case we might be filtering down
data so that's where the query Builder
comes in U but this is very helpful if
you either have a post request or a get
request um the query Builder can help
you very easily start with something and
then build it down so how does that work
so let's say we have a payouts model and
we stuck with a query so you start with
a query Builder that's what we that's
the subject that we're going to send
through and then we have the pipelines
and those are just filter claes so they
just apply a certain condition on the
query Builder and then finally then
return so then return returns to query
Builder so the nice thing about this we
can just keep chaining on it because we
know we get a query Builder back we can
just G paginate on it and it just works
so it's very clean code so if you would
like to have like a controller that
returns a list of for a table this is a
very nice way of doing things and you
can easily extend by adding more filters or
not so if we look at one of these
classes you can see here that we use the
we method again the conditional we
method so if we have a certain condition
met in that case we are applying a
certain condition on the query and since
we return the query again it will be the
turn to the the filtered query will be
passed on to the next query Builder and
so on so you can easily build up these
kind of queries
but this m makes it very flexible
especially when the query becomes more
complicated when you have to do a lot of
joints or if you have to do do something
with relationships or if you have to
fetch some kind of data to make a wear
in this is very helpful to it this way
and it makes it very readable um like if
you go to the overview here um if you
just want to know what the how the table
is filtered you just see the classes and
they are very have a very good name so
it's very easy to understand and if you
want to know more about specific R you
there okay so I I want to talk about
reusing uh pipes because I think that's
a very nice way of reusing it because
eventually you just have an object
that's being passed through a pipeline
and you can do the same one twice if you want
want
to um I can give you the boring example
of doing like a trim on a string um but
uh within hospitable our one of our key
things that we need to take care of is
double booking protection which means
that if we are syncing the calendars for
Airbnb booking.com if a res res a comes
in at the same time we want to notify
the host because they get penalized if
they have a reservation on both uh
platforms at the same time you can only
accommodate one guest at the same time
so we need to protect against that so
for example in this case we whenever we
get a new reservation in we store the
reservation we check against double
booking protection and then we apply
certain availability rules and what this
means is that we block the calendar
around the reservation so you might have
a setting where you say I want to have
one day extra after reservation for
cleaning or in that case we apply those
availability rules to block the calendar
after a
reservation um and then it might be a
parent child setup where different um
properties are connected to each other
and then we want to do the double
booking protection again just to make
sure that we are actually
good um since we are updating the
calendar in those pipe glasses we just
want to make sure that we actually did a
good job so that's why we do it twice
but it's totally fine because we're just
passing through the reservation every
time we can apply the same checks again
I love enms who is using enms in the
codebase almost all of you nice nice um
so I think enms are a a life hack to
pipelines or to reusing pipelines so
earlier we talked about the markup rate
where we applied a bunch of
pipes so we also have the reverse right
we get the the money from or the price
of a of a room rate from Airbnb and we
want to turn it down to the markup rate
or we want to turn it to the base price
that that we show them and that we need
to sync again to booking.com for
example so to do that we need to apply
it both ways um and this is where an
enum is very helpful because we can just
start an enum and we can add a custom
method to it where we Define all the
pipes that are part of the enum so uh we
had the um class where it goes from a
float to an integer well you can apply
it on both uh pipelines in this case
because the the starting result the the
price back is everywhere the same
so in this case the enum creates a great
overview of whatever is possible within
the pipelines and you can just click
through the classes if you want to know
more so for example if we have um an
action class or a place where we handle
this um you can see here that we just
pass in the enum as the as the through
in the FR method and just call pipes on
it and it return just Returns the list
of actionable pipelines in there and
it's a very nice way of coding it's very
readable if you want to know more you
just click through um it's it's very
things so this is the final use case I
want to talk to you about uh which is
called sagas um so one downside of
pipelines is that they run after each
other and there is no way to roll them
back because they're just applying
things and then that's it uh you can
wrap them in a transaction for your
database queries uh to prevent um issues
with your database but what if your one
of your pipelines is doing an API call
to the external world how do you handle
that how do you return from that there's
no way of doing that so this is where
Saga has come into play and I don't want
to go too deep into this matter I'm
going to give you homework again um you
can just look it up yourself but I would
just want to give you a basic example of
how this works and then um you can try
it out
yourself so let's say we have a pipeline
here uh we um we work with smartlocks
for example so whenever you have a
reservation coming in we generate a code
based usually based on the phone number
of a guest so the last four digits of
their phone number that's being sent to
the API of the lock and then it will set
the lock code so whenever the guest
arrives they are able to check in with
the lock code that they
received um so depending on the state of the
the
reservation um we might create the code
or we might update the code or we might
even remove
it but this canol all kinds of
exceptions for example the code already
exists or there might be other things
happening so this is where the s comes
in so this is a very very basic
implementation of a saga so it's just a
collection of events that are happening
and we have this method called add
compensation so whenever something goes
wrong um or we expect something might go
wrong we add a compensation to it which
is eventually just a call back that
handles a certain exception as you can
see on the compensate method which which
we will call on on the catch if you
compensate it will just pass in the
exception and every uh compensation can
use its own um WI way of doing things
exception so if we for example look at
how this works in with a pipeline we
have a new value object we start the
pipeline we send it through the
pipelines and then on the catch we have
this compensate
action where we pass in the exception
and every compensate can decide on its
own exception so there might be a
specific API exception that we are
looking for to compensate on there might
be another exception we don't care about
and then we don't compensate so that's very
very
flexible one important thing here is
that the lock code also needs the Saga
inside the object so if we are if we're
going to send a value object through the
pipeline we need to have access to the
Saga to add a compensation to it so I
usually just create a new value object
at The Saga in the value object and then
pass that along so how does a pipeline
of this looks
like um so here we're doing the external
API call
um and then whenever something might go
wrong we add a compensation to it so we
cue a job that will delete the code on
the lock for
compensation so in this case when
something goes wrong we compensate we
will go back to the original state and
we can combine this with a transaction
well so the pipeline patterns are is all
coming down to making it EAS easier to
understand your code um the options are
endless um I hope you learned something
about how you can apply this to your own
code base you get some ideas from this
that's the whole goal of it um and if
you have any use cases you want to show
me today I will be walking around in the pink
pink
shirt um so finally no pizza images um
this is a stro waffle which is a Dutch
cookie um hospitable is sponsoring the a
fresh Str waffel truck this year I think
it's in your booklet as well
make sure you grab a delicious Dutch
cookie today and maybe we can eat one
together and uh come say hi to us in the
pink shirts thank [Applause]
you oh my
God I have questions you have questions
okay let's I have questions
So uran Divine is
asking can one of the steps invoke an
early return of the pipeline without
throwing an
exception I haven't tried I don't know I
I I think it will break because the
expects to return next always okay you
can just not do anything in the pipeline
just return return next in stad life
finds a way life finds a way yeah yeah
okay it might show an exception I don't
know I haven't tried I I don't write
breakable code
maybe I don't know like my AI helps me
it just doesn't write those kind of
stuff that's always the the right answer Yeah
Yeah
Siri is asking how to
prevent a Pizza Pipeline to accept
pineapple that's a good one you should
you should add a pipeline at the end
that removes any pineapple yeah I think
I think it works yeah and uh
I'm asking what's your thought about p
uh pineapple on Pizza I I don't like it
but my wife eats it so I have to accept
it that it exists either
way you're going to be tricky okay but
don't needed myself
so okay that's all a big Applause for
Click on any text or timestamp to jump to that moment in the video
Share:
Most transcripts ready in under 5 seconds
One-Click Copy125+ LanguagesSearch ContentJump to Timestamps
Paste YouTube URL
Enter any YouTube video link to get the full transcript
Transcript Extraction Form
Most transcripts ready in under 5 seconds
Get Our Chrome Extension
Get transcripts instantly without leaving YouTube. Install our Chrome extension for one-click access to any video's transcript directly on the watch page.
