YouTube Transcript:
Meet BOB: the supply chain provided “bill of behaviour” for anomaly-based runtime security
Skip watching entire videos - get the full transcript, search for keywords, and copy with one click.
Share:
Video Transcript
View:
Yeah. Hey, good morning Munich. Awesome.
And thanks for technology that it's
working. Good morning. This talk is
about cyber resilience. My name is is
Constance. I am a doctor, but you should
not be taking any medical advice.
That'll become clear in a second. And I
am an independent researcher and I'm
really really grateful for the people
that make it possible to do this in this
day and age. Thank you very much. And
with me is
Hello everyone. My name is Vadim. I'm um
one of the maintainers of Harbor and we
do some business around harbor. So we
provide support for harbor and
integration and coding. Um and my part
will be then on the let's say the the
other part that's related to OCI here, right?
right?
Um so who here knows what sbombs are?
Yay. Who uses them actively?
Some. Great. Now I have with me a container
container
and let's read the bill of materials.
All right. So, so 1 g of 1 milligram of
and a whole lot of chemistry. Um well
I'm not so I'm somewhat this is an
allergy medication and I'm somewhat of
an allergy expert but beyond the first
active ingredient I wouldn't be able to
to read this. Now I'm not saying that
this is not useful what is on this for
those who understand but so so what do
you do when you don't know you find some
medication you don't know what to do
with it you don't know what it's going
to do to your body um you don't know if
exactly so but why why do you read this
thing because the pharmaceutical company
that made this right they spent years on
clinical trials experime experimenting
with humans. What these meds do to the
human body, how the medication behaves
in the human body so that you don't have
to. Now
let's talk about different kinds of
containers and this talk is about can we
as software vendors do the same thing
and use that knowledge that we have
about a software because if I if I
develop a software I have the should
should have the knowledge of how that
thing behaves and can I attach that to a
container and transfer it to a user so
that when I as an end user download
that's saying I I get a description of
the benign behavior of the expected
behavior with the software and I can
also do directly anomaly detection based
on that and more than that and that's
actually almost the key point here when
I then get an update of the software
with it comes an updated understanding
of this of the vendor's recommendation
of how that software should act now IRL
like it's not like this thing is 100%
useful, right? In in real life, there's
so many corner cases. Everybody is
different. So, it's the same thing going
to be here. There's so many
like pets out there, right? It's not
expected that everything works. So, we
need to take into account if we make
this work that people need to uh have
their own opinions and override the
settings if they if they are the local
experts. All right, this talk, so my
talk is about the runtime part is about
showing you that this is actually
possible. And here we start with when
you normally are an end user, you
install something. Um, let's say I'm I'm
looking at um Dinatra one agent. So I'm
pulling that down as an end user. I'm
not affiliated with Dinatrace and I'm
installing it. I have an idea what it
should do, but I don't have all of their
expertise of multi-year what that why is
um of what that thing should do.
Why is this? Sorry, one second.
Technical malfunction.
A timer.
Yeah, probably like
it's also showing different things here.
Okay. Anyway, so usually I get the data
out in some form. We call it
observability. And then I do some
analysis either with humans or with
machines of whether that was good or
bad. In order to speed this up, I use
rules. And they can be very simple or
you can have real life thread
intelligence feeds and make it like all
the way complicated, right? That's your
are somewhere in the middle. Now, the
idea here is that it's not me um who
writes the rules, but that I actually
let the vendor write them and attach
them to the to the software. And we call
that Bob. And I really apologize for my
malfunctioning PowerPoint. So currently
it's you. We wanted to be the software
vendor. And the reason for that is that
that will achieve an humongous amount of
scalability. Right? Even if I had all
the brains of all of the dinos people
combined in me, it's still just going to
be one one person that adapts one agent
to their system. But if they do it and
ship it to everyone, then that's real
scale. And that's what we try to to demo
to you here, right? So, I'm not exactly
sure what I'm seeing versus what you're
seeing, but that's the story line. Let's
go into the details. All right. If
you've seen Sbombs, there's a spec
called SPDX. Um, so we can do the same
thing for runtime profiles. I'm not
going to go into details. You can uh
enjoy that yourself. And my reference
implementation since we had the idea in
London is going to be using CubeCape.
CubeCape is an incubating CNCF project
here represented by Mo. That's the
little space panda here. And huge thanks
to the cubescape people. And what it
does, it wraps amongst other things, it
does many things. It wraps inspector
gadget which is an ebpf tool that traces
out um the runtime behavior of an
application. And here you can see the
elements. Maybe you can see them. Um
there's capabilities, there is events,
there's network endpoints, there are sys
calls, um xx and file path file
descriptors in there. So when if I'm the
vendor, I need to trace out my behavior
of my software in this term in these
terms and that's and the way I achieve
it is I take the behavior that I have
like behavioral tests like traffic
generation load tests whatever tests you
do um that's it how you should usually
do your quality assurance I hope um and
run the profiler while that's happening
it might take a long while right and
then you transfer it so I'm giving you a
live lab and I'll switch to a live lab
in a second and it will contain the
vendor aspect. This one takes long and
it's the the complicated part of the of
the thing because you need to actually
understand your your software. You need
to run those tests, right? So in here
there will be a sample PHP application
which is like a oneliner that shows you
how you could do this. But you're very
encouraged to take the 20 minutes and
say, "Well, I I'll do my own app and
I'll profile it and I'll see for myself
if how difficult, how easy it is to do
because the the cubecape configuration
has to be different for the vendor and
for the user." But
PowerPoint works again. Um, now that
that's one thing, but the crucial part
that I was also sweating it out really
until Monday, like Monday night 4 a.m. I
was still panicking. Does it really
work? Anyway, so does it transfer?
That's the big question, right? um and h
and how far does it not transfer? We
said IRL there's always going to be
exceptions, right? And but I wanted to
understand if they're systematic and we
can template template them and
parameterize them away. So source code
is um so I I run the storm center um for
open source threat intelligence and one
of the repos there is called BobCTL and
there you'll find the source code. Now
I'm going to switch to the live lab and
show you how as a user we assume now
that the vendor has done this right they
they created this this profile and we're
on a different infrastructure on a
different kernel and we're going to
bring it in and see if we can do anomaly
detection um and if it works. So now
first question is if this works. Yeah
that's good. That's already really good.
All right. So so this lab is now
listable. If you go labs excemius you'll
get the link later.com under courses
you'll find one of the two community
courses is now the Bob since last night.
So we assume that we have this wonderful
web app here. This is a Helm chart. I've
I've given you different ways of of
helming this in okay so that you can use
your usual tools that you hopefully know
Helm or you've heard of it at least. And
in this web app I've put a bob.l.
So this is this um benign behavior
profile with all the sys calls and
endpoints and whatnot. I've recorded
this profile and now I've cloned it
already. I've already installed it. So
this is a this is a installation takes
about 2 three minutes. So I've already
done that. Um and what it did during the
installation, it configured Cubescape
for me such that it will do this um the
tracing and the anomaly detection and it
deployed the app in its um as the vendor
recommended it to me and I had a values
file to override values if I wanted to.
All right. So now the first thing is I'm
going to in the uh right side check if I
get anything unexpected. So what I'm
seeing here is CubeCape telling me um
yeah so this is theuler that was
happening earlier. So everything um
Cubescape is monitoring now in my entire
system. This is a K3S um what are the
unexpected SIS calls and I will switch
to a new tab here um and I've got this
app installed and now I want to have the
positive test. So that means if I have a
positive test of the expected behavior
and I'm running uh anomaly detection on
the other window. So if I test it now,
what anomaly should I be seeing?
Nothing. Exactly correct. 100% correct.
So theoretically, Helm should do a test.
Yes. So, so it takes a second to to to
verify it. Yes, we we saw nothing.
That's exactly correct. Um, and so if
you did the full four full logs, you'd
see some some some scattering, some uh
it did notice it. Um, and now I'm going
to do the opposite. I'm going to verify
that if I sample attack it, don't worry,
it's not like this is insulated system.
Uh, sample attack it that I get exactly
the anomalies that I expect to get. So
this verifies that my detection is
working correctly. This is goes even
further than runtime rules, right? So
I'm first going to forward my port which
is going to uh do some funkiness. Okay,
fine. So good old almost almost make
forward. So yay. And
And
now this is a injection attack.
So this web app is vulnerable by
default. And yes, so we should see two
things. We and if you look into it,
there was an ls and there was a system
call. Um these long chasing documents.
It takes 5 minutes to elapse. It's free
for 1 hour per day. Try it out yourself.
Highly encouraged. And the last part is
so this was just one. Let's do a suite
of attacks and then count like for
example you can do this in CI/CD. you
can say I think it's 14 of them that
while this you know shit's scrolling by
um while this goes by I can do an assert
are these exactly the uh anomalies that
I was looking for and that's that's
actually pretty powerful because I
cannot just have the rules I can also
check if they're working so that's a lot
a whole load of chain of things that
have to be cor configured correctly in
order for this to work
that worked so yeah that was actually
live if I hope you believe that so for
those of you who are doubtful. Did she
test all the kernel versions, all the
Kubernetes versions, all the OS systems,
all the different whatever not GIPC
version is also important. Um, not yet.
Not yet, but I'm working on it. Okay.
So, KCD Sophia in two months, uh, I'll
update you on the full parameter study.
Currently, you can find here a link in
all the profiles of the differences. So,
if you see, now switching back to
PowerPoint. If you see differences, um,
are they expected? So, is it all going
to match? No, it's not all going to
match. Mostly in the SIS calls you will
see small but systematic differences and
they are mostly related to the chipsy
version that is baked into operating
systems or it is the kernel because
since 4.4 4 to now we are 6.11 maybe um
that's just a huge range and the kernel
did change um so but in general it is
predictable but there are differences
and I will be looking into you know how
to exactly make this nice in the UX but
at this point I hope that I have given
you also a tool to convince yourself
that I'm not lying to you that's
important to me I don't want to you know
um but now the question is so okay I've
I've hacked this up since London right
use cubecape inspect gadget ebpf etc etc
how can we how can we make this into the
OCI standard you've worked with harbor
for a long time
so the the question now that we
basically or constants proof that is
working and now we need to make this
next step that out of the lab in into a
product in a stand into product into a
standard and into the community so that
it can be used right and one of the the
steps now to to make it productized is
to you
from the consumer perspective because
from the from the vendor perspective um
things going to be probably quite
similar as they are currently already.
So the vendor has to do bit of a work
and to to create those profile. But from
the consumer perspective, we of course
want to make it super easy for the
consumer to um to use these profiles and
install them into their clusters and
just use them, right? And um this is
where we follow the same principle or
try to follow the same principle because
the naming is similar sbop sbomb and
also that we want to attach the profiles
uh attach the profiles to the OCI image,
right? And then we need to to have some
um an operator on the cluster. We call
it Bob demon, right? Because uh why not?
And so Bob demon is the the the operator
that will fetch the profile from the
image and it will have an
like in the future. Currently there is
basically just one one solution that is
working with this bit of profile is
cubecape. So currently it's only
possible to use cubescape because
there's just one solution that does
this. But hopefully in the future when
the the profile becomes more common
there will be other vendors to provide
the solution. So the Bob demon is design
is designed in a way that um is vendor
agnostic. So they can create a profile
and then Bob Demon is also the one who
is has an understanding about the
Kubernetes cluster right so he has
understanding about the Kubernetes
cluster the the kernel versions the the
all this aspect that might vary between
between the profiles and it will then be
able to create the profile or adapt the
profile that comes from the vendor to
the specific cluster right so because if
some Kubernetes version does not do this
call we can you know strike it out and
and Bob demon will be able to do this
and it will of course you know watch for
the resources find profile and apply
those manifests right so this is work in
progress and that's why it's called
funding so we need people who can
implement this right so it's an operator
it has some interfaces
and so we're looking for people who will
work on that or fund it in in any way
and this will be then the next step to
bring it out of the lab into real life
or a real product space.
Exactly. So, we need people that that
give us feedback and of course also
vendors that that test it out and I
literally if you even if you have a
small app and just contact us on any uh
you know chat medium or so or PR of
course is always or issues or not you
try it out you trace out your app does
it work for your thing. So I've recorded
in in the sample repo there is like a
simple web application there's reddis
and there's I was even interested so as
a stateful set and I am working on
tetragonon as that was kind of funny to
to with ebpf trace and ebpf agent and
see if I can even transfer that um
because it's really really deep down I
wanted to see what kind of applications
can be covered by this are there you
know one thing is front end like this
web app is like you know has an API but
what what about the the deeper more
nested things that are deep down in
clusters and but of course very few
people cannot cover the parameter space
so it needs to be a community effort
here um and also putting it into CI/CD
and the the labs are really practical
right so the like there are two labs one
is from the consumer perspective and one
is from the vendor perspective they're
really approachable and um like when you
run the lab you get all the results and
then you can study the results and
analyze and see how it is behaving and
you get a better understanding how
things are behaving and and what is the
outcome of this whole thing. Uh yeah,
that's uh I recommend for for the vendor
side and also for the consumer side to
try out the the labs and see what
results you will get because the
profiles are really powerful. So you can
really cover a lot of things with you
can you can really analyze the behavior
of your application uh in in in in quite
granular way.
Exactly. And if you know any templating
language so I used Helm because I did a
survey and it was 80 something% of you
use Helm. So but if you have you know
customize Argo whatever flux what you
know you're not going saying I'm pretty
sure it also works in that language I
just use the maturity language for for
reference and yeah if you don't want to
remember this um this link uh navigate
to the main page and then under
community content in courses you'll find
it um if you are a researcher or student
and you want to do um anything here just
reach out I'm also writing funding
proposals of course for this and at this
point I think we conclude
um because we hope that you have some
some questions and some discussion. Um
yeah, so we call it the uh container
bypuck settle. Um and for those who
don't know how to pronounce that word,
Bob, um
yeah, thank you so much and I hope that
Yeah. So we have quite some time for
questions. So if you have questions
just keep the mic. Okay.
And thanks so much for Lind. Super quick
shout out for the ones that that have
trusted me since the beginning and give
me their medal just to use without any
expectation in return. That's that's a
huge thing. Um yeah go ahead.
Yeah. Uh really cool presentation and
cool topic. I was wondering you
mentioned earlier that probably in some
environments there are special things
which are expected different behavior
and have you already looked into how for
example if I deploy that a bit of
behavior into my own cluster how can I
tell it okay in this environment it's
expected that you call a specific URL or whatever
whatever
yeah so um so you can I'm not sure if
Let's see if we can do a making this
bigger thing
somewhere here. All right. So, yeah, I
have for example here. Um
Um
I think it still shows the presentation.
Thank you very much for pointing that
out to me. Um no, no, literally because
um so let's I hope this is big enough.
So, for example, here you have an
endpoint direction endpoint port 8080.
Do you see this? And then with Helm,
this is like host. Um, I'm inside the
cluster. I'm using the DNS resolution
and I'm trying to make this bigger. One
sec. So you can actually read it. Does
it sort of answer your question while
Yeah. No, that's that's it exactly here. So
So
um you see that this is this you can
basically edit it by hand. Um so so this
is a network endpoint to add you can
also do so cubescape also has different
CRDs where you actually literally have
known network no or known endpoint but
then you use a different CRD. Okay.
Okay.
Yeah. So two two options. Yeah.
The other the other use cases that you
mentioned is that what are the
difference like there are some
differences in the operating systems
right so that some operations do some
sys calls and some other operating
systems don't do some sys calls and you
will see that in the profiles right and
then hopefully in like with the demon it
will be able to figure out which sys are
related to which version and then it can
filter out and create a profile for your
specific environment then so that those
differences can be covered
Right. So yeah, there is in the CI/CD
pipeline there's a large matrix of tests
um and for the sys calls is probably a
superset and we we imagine that um there
is a demons or two-step installation
that queries which yeah all of these
parameters from the cluster and also
gets you maybe the the end points like
which C is your master um your
Kubernetes master installed and and
parses that into the template at
installation time. as a vendor you need
to test your application. I mean you
should anyway right you should anyway
test your application different
environment and yeah so commercial
shape doing this version whatever 31 35 33
33
so it should not be a surprise
and and the other thing is that you can
use globber so here you see this these
are placeholder languages here there's
there's an asterisk and there you see
the three dot um so as a as a vendor you
have to substitute because a lot of
tokens or mount slices, uh, cublet
slices, they have these UU ids,
typically date and some nonsense, um,
that you need to substitute out because
they Yeah, but this is, I think,
relatively, uh, yeah, self-explanatory,
but you have to do it.
How's the notification working? How do
you get notified when something is is blocked?
blocked?
So this is a cube this is the cubecape
implementation. If we go to the first
part of the lab here
um so they have a picture of the
architecture. So this is a part that is
not part of the bob. I'm just um
piggybacking on the shoulders of giants.
So this is node agents architecture
internally. It diffs. So it puts that
CRD into CD and so it diffs each time
the Spectre gadget there's a certain
time frame in which it it reads what are
the actual uh Cisco file scripts etc
versus what is stored and that diffing
goes into alerts which you then can put
into alert manager. So this is a part of
cubecape's um architecture on top of the
tracing itself that we're making use of here.
here.
Okay. So no events for instance like a
cubectl event or something
would be handy maybe.
I don't think it I um
um
no no there there are no events like you
might also be flooding because what has
happened is a lot of duplication
happening and I've done this myself and
you get flooded so quickly. So I'm not
sure if I be careful with going for just interesting.
interesting.
Yeah. Yeah. No. Um, this is also why I
piggybacked on an existing system and
didn't try to implement this myself
because it's hard to do this whole dedu
behind the scenes. Thanks for the
question. Good question. Yeah.
Thanks.
Um there are already so many methods
with which uh let's say Helmchart vendor
could add sec profiles SE Linux and so
on to the app. What makes you optimistic
that vendors will publish a bop instead
of just adding these hard controls?
Well they are different right? Um so the
so if I do a sec comp profile that's
enforcing whereas this is alerting um
and it does anomaly detection so a sec
and I mean the other question is how
many do really publish seccom profiles
um and especially as I have not seen a
single and show me a single SC Linux
profile that comes with a helm chat that
works um sorry I I like SC Linux but I
know how much work it is Um whereas this
the thing that convinced me was um that
I think one week after we had the idea I
had a prototype that was essentially
this and it it just I didn't have to do
much. Okay. There's there's some like
nonsense with template hashes that are
annoying but you know details whereas
yeah see Linux I'd be careful in sec and
enforcing mode.
Yeah especially because the sys calls differers.
differers.
I haven't seen anyone supersetting the
sis calls systematically. Yeah.
But but you still would need to do that
for uh publishing like recommended
profiles, right?
Either you superset them. Yeah. Or you
um uh as as the Bob demon uh basically
read out which kernel version is it and
if that kernel version was uh well in
combination with the chipsy um if you
have that combo already. But pro it's so
it looks like it's already plus minus 10
calls typically that differ mostly. So
but I'll be careful. Thanks.
Thanks.
Yeah. So why nobody is doing it? I think
because at the risk of breaking it and
also because we have no standard. So
lots of vendors have individual tools
that can do this. Lot like almost
everybody can can trace it out. That's
not the magic. The magic is the transfer.
Um great talk. Uh I have another remark.
Um we talk now a lot about the security
part of it which is great of course. Um
what you mentioned was that you have to
profile your app to basically know what
kind of behavior it has. I would even
turn it around because test coverage is
a really hard problem too. So you could
even I won't say abuse it like leverage
it to increase a lot your your test
coverage of your realtime application
because this is a big problem too. So
yeah I really really see a lot of
potential there.
You can also do per performance
profiling. Now, I'm not in the
performance space, but for example, why
does my app not work on my open shift,
but it works on the GKU just fine. And I
I just it's an indicator. Um, and since
yeah, it's it's one step further to a
flame graph, but you at least you see
the objects that are loaded here and
then maybe it's one of them. Gives you
an indication maybe.
Yeah, absolutely. Thanks.
So, you can use it for debugging too
actually. Totally. Yeah, totally did that.
that.
Yeah. Any other questions? More
questions? No. All right. Um, then thank
Click on any text or timestamp to jump to that moment in the video
Share:
Most transcripts ready in under 5 seconds
One-Click Copy125+ LanguagesSearch ContentJump to Timestamps
Paste YouTube URL
Enter any YouTube video link to get the full transcript
Transcript Extraction Form
Most transcripts ready in under 5 seconds
Get Our Chrome Extension
Get transcripts instantly without leaving YouTube. Install our Chrome extension for one-click access to any video's transcript directly on the watch page.
Works with YouTube, Coursera, Udemy and more educational platforms
Get Instant Transcripts: Just Edit the Domain in Your Address Bar!
YouTube
←
→
↻
https://www.youtube.com/watch?v=UF8uR6Z6KLc
YoutubeToText
←
→
↻
https://youtubetotext.net/watch?v=UF8uR6Z6KLc