[00:04] What I want to do
today is chat with you
[00:07] about career advice in AI.
[00:09] And in previous years, I used
to do most of this lecture
[00:13] by myself.
[00:14] But what I thought
I'd do today is
[00:16] I'll share just a few
thoughts and then hand it over
[00:19] to my good friend Laurence
Moroney, who I invited to speak
[00:23] here and kindly agreed to come
all the way to San Francisco,
[00:27] he lives in Seattle, to share
with us a very broad market
[00:31] landscape for what he's
seeing in the job market,
[00:33] as well as tips for career,
growing a career in AI.
[00:39] But there was just two slides
and then one more thought
[00:42] I want to share with you
before I hand it over
[00:44] to Laurence, which is it
really feels like the best
[00:49] opportunity, the best time
ever to be building with AI
[00:53] and to building a career in AI.
[00:55] A few months ago I noticed in
social media, traditional media,
[00:59] there are a few questions
about is AI slowing down?
[01:03] People saying, well,
it's GPT-5 that good?
[01:05] I think it's
actually pretty good.
[01:07] But there are questions about
is AI progress slowing down?
[01:10] And I think part of the reason
the question was even raised was
[01:13] because if a benchmark for AI
is 100% is perfect answers,
[01:19] then if you make rapid
progress, at some point,
[01:22] you cannot get
above 100% accuracy.
[01:25] But one of the studies that most
influenced my thinking was work
[01:30] done by this
organization, M-E-T-R,
[01:32] METR that studied
as time passes,
[01:37] how complex are the tasks that
AI could do as measured by how
[01:40] long it takes a human
to do that task?
[01:43] So a few years ago, maybe
GPT-2 could do tasks
[01:47] that a human could do
in a couple seconds.
[01:50] And then they
could do tasks that
[01:52] took a human four seconds,
then eight seconds, then,
[01:57] a minute, two minutes,
four minutes, and so on.
[02:00] And the study estimates that
the length of task AI can do
[02:03] is doubling every seven months.
[02:06] And I think on
this metric, I feel
[02:09] optimistic that AI will
continue making progress,
[02:12] meaning the complexity
of tasks as measured
[02:15] by how long a human takes to do
something is doubling rapidly.
[02:19] And same study with
a smaller data set
[02:22] seems to show-- same study
argued that for AI coding,
[02:25] the doubling time is even
shorter, maybe 70 days.
[02:29] So this code that
used to take me,
[02:31] I don't know, 10 minutes
to write, then 20 minutes
[02:34] to write, 40 minutes
to write, and AI
[02:36] could do more and more of that.
[02:38] And so the reasons I
think this is a golden age
[02:41] to be building, best
time we've ever seen
[02:43] is maybe two themes which
are more powerful and faster.
[02:47] So we can all, all
of you in this room
[02:50] can now write software that
is more powerful than what
[02:54] anyone on the planet
could have built a year
[02:57] ago by using AI building blocks.
[03:00] AI building blocks include
large language models,
[03:03] radical genetic
workflows, voice AI,
[03:05] and of course, deep learning.
[03:06] It turns out that a lot of LLMs
have a decent, at least basic
[03:10] understanding of deep learning.
[03:12] So if you have a prompt
one of the frontier models
[03:14] to implement a cutting edge
neural network for you,
[03:16] try prompting it to implement
a transformer network for you.
[03:19] It's actually not bad at helping
you use these building blocks
[03:23] to build software quickly.
[03:25] And so we have very
powerful building blocks
[03:29] that were very difficult or did
not exist a year or two ago.
[03:32] And so you can now build
software that does things
[03:34] that no one else on the planet,
even the most advanced teams
[03:38] on the planet, could have done.
[03:39] And then also with
AI coding, the speed
[03:44] with which you can
get software written
[03:46] is much faster than ever before.
[03:49] And I've personally
found it as important
[03:50] to stay on the
frontier of tools,
[03:52] because the tools for
AI coding changes,
[03:55] I don't know, really rapidly.
[03:57] So I feel like since several
months ago, my personal number
[04:03] one favorite tool became
Cloud Code, moving on
[04:07] from some earlier
generations, I think.
[04:09] And then I think since
the release of GPT-5,
[04:13] I think OpenAI
Codex has actually
[04:15] made tremendous progress.
[04:17] And this morning,
Gemini 3 was released,
[04:19] which haven't had time to play
with it yet just this morning.
[04:22] It seems like another
huge leap forward.
[04:25] So I feel if you ask
me every three months
[04:27] what my personal favorite
coding tool is, it actually
[04:29] probably changes definitely
every six months, but quite
[04:32] possibly every three months.
[04:33] And I find that being half a
generation behind in these tools
[04:38] means being, frankly, quite
a bit less productive.
[04:41] And I know everyone says
AI is moving so fast,
[04:44] everything's changing so fast.
[04:45] But AI coding tools, of
all the sectors in AI,
[04:48] many things maybe don't move as
fast as the hype says it does,
[04:51] but AI coding
tools is one sector
[04:53] where I see the pace of
progress is tremendous.
[04:56] And staying at the latest
generation of tools,
[04:59] rather than half
generation behind makes
[05:01] you more productive.
[05:03] And with our ability to
build more powerful software
[05:06] and build it much
faster than ever
[05:08] before, I think
one piece of advice
[05:11] that I give now,
much more strongly
[05:12] now than even a year
ago or two years ago,
[05:15] is just go and build stuff.
[05:17] Take classes from Stanford.
[05:19] Take online courses.
[05:20] And additionally
your opportunity
[05:22] to build things,
and I think Laurence
[05:24] is going to talk about
showing them to others,
[05:26] is greater than ever before.
[05:28] But there's one
weird implication
[05:30] of this that is
maybe not-- is still,
[05:33] I don't know, more and more
people are appreciating it,
[05:36] but not widely known, which
is the product management
[05:38] bottleneck, which is that when
it is increasingly easy to go
[05:43] from a clearly written software
spec to a piece of code,
[05:47] then the bottleneck increasingly
is deciding what to build
[05:50] or increasingly writing that
queer spec for what you actually
[05:53] want to build.
[05:55] When I'm building
software, I often
[05:57] think of going through a loop
where we'll write some software,
[06:00] write some code sure to use
this to get user feedback.
[06:03] I think of this as a PM or
product management work.
[06:06] And then based on
user feedback, I'll
[06:09] revise my view on what users
like, what they don't like.
[06:11] This UI is too difficult.
They want this feature.
[06:13] They don't want that feature
and change my conception
[06:16] of what to build, and
then go around this loop
[06:19] many times to hopefully
iterate toward a product
[06:21] that users love.
[06:23] And because of AI coding, the
process of building software
[06:27] has become much cheaper and
much faster than before.
[06:31] But that ironically
shifts the bottleneck
[06:34] to deciding what to build.
[06:38] So some weird trends I'm seeing.
[06:43] In Silicon Valley and
in many tech companies,
[06:45] people have often talked about
an engineer to product manager,
[06:48] engineer to PM ratio.
[06:50] And you take these ratios
with grain of salt,
[06:53] because they're kind of
vary all over the place.
[06:55] But you hear companies
talk about the Eng to PM
[06:57] ratio of 4 to 1 or
7 to 1 or 8 to 1.
[07:00] This idea that one product
manager writing product specs
[07:04] can keep four to eight or
some number like that engineer
[07:08] is busy.
[07:09] But because engineering
is speeding up,
[07:11] whereas product management is
not sped up as far as much by AI
[07:15] as engineering,
I'm seeing the Eng
[07:18] to PM ratio trending downward,
maybe even two or one to one.
[07:22] So some teams I work
with, they proposed
[07:24] headcount was one PM
to one engineer, which
[07:26] is a ratio unlike almost all
Silicon Valley, certainly
[07:31] traditional Silicon
Valley companies.
[07:33] And the other thing I'm
seeing is that engineers,
[07:37] they can also
shape products that
[07:40] can move really fast where
you go one step further,
[07:44] take the engineer, take
the PM, and collapse them
[07:46] into a single human.
[07:48] And I find that
there are definitely
[07:51] engineers that doing
engineering work that
[07:53] don't enjoy talking to
users and having that more
[07:55] human, empathetic side of work.
[07:58] But I'm finding increasingly
that the subset of engineers
[08:01] that learn to talk to
users, get feedback, develop
[08:05] deep empathy for users so
that they can make decisions
[08:08] about what to build,
those engineers
[08:10] are also the fastest
moving people that I'm
[08:12] seeing in Silicon Valley today.
[08:15] And I feel like at the
earliest stage of my career,
[08:19] one thing I regretted for years
was in one of the roles I had,
[08:24] I went to try to convince
a bunch of engineers
[08:27] to do more product work.
[08:29] And I actually made a bunch
of really good engineers
[08:32] feel bad for not being
good product managers.
[08:34] And that was a mistake I made,
regretted that for years.
[08:37] I just shouldn't have done that.
[08:39] And part of me
feels like I'm now
[08:40] going back to repeat
that exact same mistake.
[08:44] Having said that, I
find that the fact
[08:47] that I can write
code, but also talk
[08:50] to users to shape what
to do, that lets me
[08:53] and the engineers that can
do this go much faster.
[08:55] So I think maybe worth
taking another look
[08:58] at whether engineers can
do a bit more of this work,
[09:02] because then if you're not
waiting for someone else
[09:05] to take the product
to customers,
[09:06] you can just write code, have
a gut for what to do next,
[09:08] and iterate that pace,
that velocity of execution
[09:11] is much faster.
[09:13] And then before I hand over to
Laurence, just one last thing
[09:17] I want to share, which is in
terms of navigating your career,
[09:23] I think one of the
most strong predictors
[09:26] for your speed of learning
and for your level of success
[09:30] is the people you
surround yourself with.
[09:32] I think we're all
social creatures.
[09:33] We all learn from
people around us.
[09:35] And it turns out there are
studies in sociology that
[09:40] show that if your five
closest friends are smokers,
[09:44] the odds of you being a
smoker is pretty much high.
[09:46] Please don't smoke.
[09:48] It's just an example.
[09:51] I don't know of
any study showing
[09:52] that if you're five or 10
closest friends are really
[09:55] hard working, determined
people, learning quickly, trying
[10:00] to make the world a
better place of AI,
[10:02] that you are more
likely to do that too.
[10:04] But it's one of
those things that I
[10:05] think is almost certainly true.
[10:07] It's like all of us are inspired
by the people around us,
[10:10] and we're able to find a good
group of people to work with,
[10:12] that helps drive you forward.
[10:15] In fact, here at Stanford,
I feel very fortunate--
[10:17] the fantastic student body,
fantastic group of faculty.
[10:22] And then the other
thing that I think
[10:24] we're fortunate to
have at Stanford
[10:26] is our connective tissue.
[10:27] So candidly, a lot
of the people working
[10:32] and a lot of the cutting edge
AI labs, the frontier labs,
[10:35] they were former students of
a lot of different Stanford
[10:39] faculty.
[10:40] And so that rich
connective tissue candidly
[10:43] means that at Stanford,
we often find out
[10:45] about a lot of stuff
that's not widely
[10:47] known because of the
relationships, the friendships.
[10:50] And when some company
does something,
[10:52] one of my friends
and the faculty
[10:54] will call up someone to come
and say, hey, that's weird.
[10:56] Does this really work?
[10:57] And so that rich connective
tissue means that we're all--
[11:02] just as we try to pull
our friends forward,
[11:04] our friends also pull us
forward with the knowledge
[11:06] and the connective tissue and
this know-how of bleeding edge
[11:10] AI, which unfortunately
is not all
[11:12] published on the internet
at this moment in time.
[11:14] So I think while you're at
Stanford, make those friends,
[11:18] form that rich
connective tissue.
[11:20] And there have been a lot of
times that just for myself,
[11:22] where, frankly, I
was thinking of going
[11:25] in some technical direction.
[11:26] I'd have one or two
phone calls with someone
[11:30] really close to research, either
Stanford researcher or someone
[11:33] in the frontier lab.
[11:34] They would share something with
me that I didn't know before.
[11:37] And that changes
the way I choose
[11:39] the technical
architecture of a project.
[11:41] So I find that group of
friends you surround yourself
[11:43] with, those little pieces
of information-- try this.
[11:46] Don't do that--
that's just hype.
[11:48] Ignore the PR.
[11:49] Don't actually try that thing.
[11:50] Those things make
a big difference
[11:53] in your ability to steer the
direction of your projects.
[11:56] So while you're at Stanford,
take advantage of that.
[11:59] This connective tissue
that Stanford has,
[12:01] it's actually really unique.
[12:02] There are lots of great
universities in the world,
[12:04] but at this moment in time,
I don't think there's any--
[12:08] I don't want to sound like
I'm doing PR for Stanford now,
[12:10] but I really think there's no
university in the world that
[12:13] is as privileged as Stanford
at this moment in time,
[12:16] in terms of the richness of
the connective tissue to all
[12:19] of the leading AI groups.
[12:23] But to me, there's also
that we're lucky here
[12:25] to have a wonderful
community of people
[12:27] to work with and learn from.
[12:30] And for you too.
[12:31] If you apply for jobs, the
thing that is much more
[12:35] important for your
career success would
[12:37] be if you go to a company,
it'll be the people
[12:40] you work with day to day.
[12:42] So here's one story that I've
told in previous classes, I
[12:49] repeat, which is there's a
Stanford student that I knew
[12:52] this was many years
ago, that I knew,
[12:54] and they did really
good work at Stanford.
[12:56] I thought they were high flyer.
[12:58] And they applied for
a job at a company,
[13:00] and they got a job offer
from one of the companies
[13:03] with a hot AI brand.
[13:07] This company refused to tell
him which team he would join.
[13:10] They said, oh, come
sign up for a job.
[13:13] There's a rotation system,
matching system, blah blah blah.
[13:16] Sign on the dotted line first.
[13:17] Then we'll figure out what's
a good project for you.
[13:21] Partly because it
was a good company.
[13:24] His parents were proud
of him for getting a job
[13:26] at this company.
[13:27] This student joined
this company hoping
[13:29] to work on exciting AI project.
[13:32] And after he signed
on the dotted line,
[13:33] he was assigned to
work on the back end
[13:36] Java payment processing
system of the company.
[13:39] Nothing against anyone
that wants to do Java
[13:41] back end payment
processing systems.
[13:42] I think they're great, but this
is an AI student that did not
[13:45] get matched to an AI project.
[13:47] And so for about a year,
he was really frustrated,
[13:50] and he actually left this
company after about a year.
[13:53] The unfortunate thing
is, I told this story
[13:56] in CS230 some years back.
[14:00] And then after I
was already telling
[14:04] the story in this class,
a couple of years later,
[14:08] another student in CS230 went
through the same experience
[14:13] with the same company, not Java
back end payment processing,
[14:16] but different project.
[14:17] And I think this effect
of trying to figure out
[14:21] who you'll be actually
working with day to day
[14:23] and making sure you're
surrounded by people that
[14:25] inspire you and work
on exciting projects,
[14:27] I think that's important.
[14:28] And even completely
candid, if a company
[14:30] refuses to tell you what
team you'll be assigned to,
[14:34] that does raise a
question in my mind of
[14:37] whether or not what will happen.
[14:39] And I think that
instead of working
[14:42] for the company with
the hottest brand,
[14:44] sometimes if you find a
really good team with really
[14:48] hard working, knowledgeable,
smart people trying to do good
[14:50] with AI, but the company
logo just isn't as hot,
[14:54] I think that often
means you actually
[14:56] learn faster and progress
your career better because it
[14:59] is after all, we don't
learn from the excitement
[15:03] of the company logo when
you walk through the door,
[15:05] you learn from the people
you deal with day to day.
[15:08] So I just urge you to use
that as a huge criteria
[15:13] for your selection process
for what you decide to do.
[15:21] But I think number
one on my advice
[15:25] is it's become much
easier than ever
[15:27] before to build powerful
software faster.
[15:30] And what that means
is do be responsible.
[15:33] Don't build software
that hurts others.
[15:35] And at the same time, there are
so many things that each of you
[15:39] can build.
[15:40] And what I find is the number
of ideas out in the world
[15:42] is much greater than the
number of people with the skill
[15:45] to build them.
[15:45] So I know that finding jobs has
gotten tougher for fresh college
[15:49] grads.
[15:49] At the same time,
a lot of teams just
[15:51] can't find enough
skilled people.
[15:53] And so there are
a lot of projects
[15:56] in the world that if
you don't build it,
[15:58] I think no one else
will build it either.
[16:00] So you don't need to-- so
long as you don't harm others,
[16:04] be responsible, there are a
lot of things that you don't
[16:06] need to wait for permission.
[16:07] You don't need to wait for
someone else to do it first
[16:10] and then you do it.
[16:11] The cost of a failure is
much lower than before
[16:14] because you waste a weekend
but learn something.
[16:17] That seems fine to me.
[16:18] So I think so let's
be responsible,
[16:21] going for trying things out
and building lots of things
[16:24] would be the number one
most important thing I
[16:27] think would help your careers.
[16:31] And yeah, I think I'm going
to say one last thing that
[16:35] is considered not politically
correct in some circles,
[16:39] but I'll just say it anyway,
which is in some circles,
[16:43] it has become considered
not politically correct
[16:47] to encourage others
to work hard.
[16:50] I'm going to encourage
you to work hard.
[16:53] Now, I think the reason
some people don't
[16:55] like that is because
there are some people that
[16:57] are in a phase of
life where they're not
[16:59] in a position to work hard.
[17:00] So right after my
children were born,
[17:03] I was not working hard for
a short period of time.
[17:05] And there are people because
of an injury or disability,
[17:10] whatever very valid
reasons, they're
[17:12] not in a position to work
hard at that moment in time.
[17:14] And we should respect
them, support them,
[17:16] make sure they're well
taken care of even
[17:18] though they're not working hard.
[17:19] Having said that,
all of my, say,
[17:22] PhD students have
become very successful,
[17:24] I saw every single one of
them work incredibly hard.
[17:27] I mean, the 2:00 AM sitting
up, hyperparameter tuning,
[17:30] been there, done that.
[17:31] Still doing it some days.
[17:33] And if you are fortunate enough
to be in a position in life
[17:36] where you can work
really hard, there
[17:40] are so many opportunities
to do things right now.
[17:44] If you get excited, as I do,
spending evenings and weekends
[17:47] coding and building stuff
and getting user feedback,
[17:50] if you lean in and
do those things,
[17:51] it will increase your odds
of being really successful.
[17:54] So I don't know.
[17:55] Maybe I get into some
trouble with some people
[17:57] encouraging me to
work hard, but I
[17:58] find that the truth is
people that work hard
[18:02] get a lot more done.
[18:02] We should also respect people
that don't and people that
[18:05] aren't in a position to do so.
[18:06] But between watching
some dumb TV
[18:10] show versus firing your
agentic coder on a weekend
[18:14] to try something,
I'm going to choose
[18:16] the latter almost every time.
[18:18] Unless I'm watching a show with
my kids, sometimes I do that.
[18:20] But you mean--
[18:23] I hope you do that.
[18:26] All right, so those are the
main things I wanted to say.
[18:29] What I want to do is hand the
stage over to my good friend
[18:33] Laurence Moroney, who share
a lot more about career
[18:38] advice on AI.
[18:39] Let me just quick intro.
[18:40] I've known Laurence
for a long time.
[18:42] He's done a lot of online
education work, sometimes
[18:44] with me and my teams,
taught a lot of people
[18:46] Tensorflow, taught a
lot of people PyTorch.
[18:49] He was lead AI advocate at
Google for many years, now
[18:52] runs a group at Arm.
[18:53] I've also enjoyed quite
a few of his books.
[18:55] This is one of them.
[18:56] He recently also published
a new book on PyTorch.
[18:59] This is an excellent book,
Introduction to PyTorch.
[19:02] And he's a very sought after
speakers in many circles,
[19:06] so I was very grateful when
he agreed to come speak to us.
[19:10] Pleasure is all mine.
[19:11] I just want to
reinforce something
[19:12] that Andrew was
talking about earlier
[19:14] on about choosing
the people that you
[19:15] work with being very important.
[19:17] But I also want to show that
from the other way around that
[19:20] the company, when
they're interviewing you
[19:22] are also choosing you.
[19:23] And the good
companies really want
[19:25] to choose the people
that they work with also.
[19:27] And I've been doing a lot
of mentoring of young people
[19:30] over the last, particularly
over the last 18
[19:32] months, who are hunting
for careers for themselves.
[19:36] And I want to tell the story
of one young man and this guy,
[19:40] very well educated, great
experience, super elite coder.
[19:46] He could do every challenge
that was in front of him,
[19:49] and he got laid off
from his job in April.
[19:51] He worked in medical software,
and medical software business
[19:54] has been changing drastically.
[19:56] Funding has been cut by
the Federal government
[19:58] in a number of areas, and he
got laid off from his job.
[20:01] And with his experience,
with his ability,
[20:03] with his skills, all of
these kind of things,
[20:05] he thought that it
would be very easy
[20:06] for him to find another job.
[20:07] And the poor young guy had
a really terrible April.
[20:09] He got laid off from
his job in April.
[20:12] Immediately before
that, his girlfriend
[20:13] had broken up with him,
and then a couple of weeks
[20:15] later, his dog died.
[20:16] So he was not in a good place.
[20:19] And so I sat down with him
after a couple of months
[20:22] and took a look.
[20:23] And he had a spreadsheet of
jobs that he was applying to,
[20:27] and he had over 300 jobs that he
was tracking in the spreadsheet.
[20:31] And in a number of
these jobs, he actually
[20:33] got into the interview
process, and he
[20:35] went very deep in
the interview process
[20:37] with companies like Meta.
[20:40] Who else?
[20:41] Not Google.
[20:42] It was Meta.
[20:42] There was Microsoft.
[20:43] There was one of
the other large tech
[20:45] companies where you do lots
and lots of interview loops.
[20:48] And every time towards
the end of the loop,
[20:51] he knew he did a great loop.
[20:52] He solved all the coding.
[20:54] He had great conversations
with the people,
[20:56] or at least he thought he had.
[20:58] And then every
time within a day,
[20:59] the recruiter would call him and
say, no, you didn't get the job.
[21:04] And it was like it
was heartbreaking.
[21:06] And like I said, 300 plus
jobs he had been tracking.
[21:10] So I started working with him
to do some mock interviews
[21:13] and to do some fine tuning.
[21:15] Oh, it was Jeff Bezos
company, not Amazon.
[21:17] That was one of the
other big tech company
[21:19] that he'd interviewed with.
[21:21] And I started
working through him
[21:22] and doing some test interviews
and all this kind of thing
[21:25] with him.
[21:25] Terrific, terrific candidate
couldn't figure out
[21:27] what was going wrong until
I decided to try and do
[21:31] a different sort of
interview where I gave him
[21:33] a really tough interview.
[21:36] I gave him some tough LeetCode.
[21:38] I gave him some really obscure
corner cases in his coding.
[21:43] And I saw how he reacted.
[21:46] And how he reacted
was the advice
[21:48] that was given to him in
the recruiting pamphlets.
[21:50] And a lot of these
recruiting pamphlets
[21:52] will say things like, you're
going to have an opportunity
[21:57] to share an opinion, and you've
got to stand your ground.
[21:59] You've got to have a backbone.
[22:01] Don't bend.
[22:03] His interpretation of that was
to be really, really tough.
[22:07] So I would pick corners.
[22:09] I would pick holes in his code.
[22:11] I'd pick corner cases
where things may not work,
[22:13] and I would give him
a test of crisis.
[22:15] And this advice that he'd
been given to stand his ground
[22:18] ended up making him hostile in
these interview environments.
[22:23] And I was looking at
this then from the point
[22:26] of view of what Andrew
was just talking about,
[22:28] where it's a case of hey, good
people, good teams, people
[22:31] that you can work together with.
[22:33] And from the
interviewer perspective,
[22:35] if I'm managing this team,
this person is that cliched 10x
[22:38] engineer, but I don't want
him anywhere near my team
[22:41] because of this attitude.
[22:44] We worked on that.
[22:45] We fine-tuned it.
[22:45] And the strange part is he's
a really, really nice guy.
[22:49] It's just this was the
advice he was given,
[22:52] and he followed that advice,
and he failed so many interviews
[22:54] as a result.
[22:56] So when I gave him the next
job that he was interviewing at
[22:58] was at a company where teamwork
is very, very highly valued.
[23:03] And the good news is he got
the job at that company.
[23:05] He's now working there.
[23:07] He doubled his salary from
the job he was laid off from,
[23:10] and he ended up having
about-- now he looks back
[23:12] and he had six months
of fun employment.
[23:14] But at the time when he was
going through all of that,
[23:16] it was a very, very
difficult time for him.
[23:19] So the flip side of it, if
you're looking at a company
[23:21] and looking at the paper
you'd be working with
[23:23] is very, very important.
[23:24] But also realize they are
looking at you in the same way.
[23:28] And so if you've gone to
a tech interview coaching,
[23:31] and they gave you that
advice to stand your ground
[23:33] and have a backbone,
it's good to do that.
[23:36] But don't be a jerk
while you're doing so.
[23:38] Can you see my slides?
[23:39] OK.
[23:39] So I'm Laurence.
[23:41] I've been working in
tech for more decades
[23:44] than ChatGPT thinks there
are oars in strawberry.
[23:48] So I've worked in many of
the big tech companies.
[23:51] I spent many years at Microsoft,
spent many years at Google,
[23:54] also worked in
places like Reuters.
[23:56] I've done a lot of work in
startups, both in this country
[23:59] and abroad.
[24:00] And so what I really
want to talk about today
[24:02] is like to think about what
does the career landscape look
[24:06] like today, particularly in AI.
[24:09] Because first of all, what
Andrew said about in Stanford,
[24:13] you've got the ability to
make use of the networks
[24:16] that you have in Stanford,
make use of the prestige
[24:18] that you have, and I say
use every weapon you have.
[24:21] Because unfortunately,
the landscape right
[24:23] now is not ideal.
[24:25] We've gone through some
very difficult times.
[24:27] All you have to do
is look at the news,
[24:28] and you can see massive tech
layoffs, slowing hiring in tech,
[24:33] and lots of stuff like that.
[24:34] But it's not
necessarily a bad thing
[24:36] if you do it the right way.
[24:38] So I want to just have a
quick look the job market
[24:40] reality check.
[24:42] Actually out of
interest, I don't know.
[24:44] This is a-- are you juniors?
[24:46] You're graduating this year
or you're graduating next year
[24:49] or what is the general survey?
[24:52] You're third year of four?
[24:53] [INAUDIBLE]
[24:55] Third year of
three, I would say.
[24:56] So you're going to be
graduating coming summer.
[24:59] How many people are
already looking for jobs?
[25:02] OK, quite a few of you.
[25:04] How many people
have had success?
[25:06] Nobody.
[25:07] Oh, one.
[25:08] OK.
[25:08] That's good.
[25:09] So you're probably seeing some
of these things, the signals
[25:12] out there, junior hiring
slowing significantly.
[25:15] When I say junior, I
mean graduate level.
[25:18] High-profile layoffs are
dominating the headlines.
[25:21] I was at Google a
couple of years ago
[25:23] when they had the biggest
layoff they'd ever had.
[25:25] We're seeing layoffs at the
likes of Amazon, Microsoft,
[25:28] other companies like that.
[25:30] It feels that entry-level
positions are scarce,
[25:33] and I'm underlining
the word "feels" there,
[25:35] and I want to get into that in
a little bit more detail later.
[25:38] And also, competition is fierce.
[25:41] But my question is,
should you worry?
[25:43] And I say, no.
[25:45] Because if you can approach
things in the right way,
[25:49] if you can approach the job
hunting thing in the right way,
[25:52] particularly understanding how
rapidly the AI landscape is
[25:55] changing, then I think
people with the right mindset
[25:58] will thrive.
[26:00] So what do I mean by that?
[26:03] So as Andrew had mentioned,
the AI hiring landscape
[26:06] is changing because the
AI industry is changing.
[26:10] The AI industry I--
[26:12] I actually first got involved
in AI back way back in 1992.
[26:16] I worked in it for a little
while just before the AI winter.
[26:19] Everything failed drastically,
but I got bitten by the AI bug.
[26:23] And then in 2015, when Google
were launching TensorFlow,
[26:29] I got pulled right back into
it, became part of the whole AI
[26:32] boom, launching TensorFlow,
advocating TensorFlow
[26:35] to millions of
people, and seeing
[26:37] the changes that happened.
[26:38] But along 2021, 2022, we
had a global pandemic.
[26:44] The global pandemic caused a
massive industrial slowdown.
[26:48] This massive industrial
slowdown meant
[26:50] that companies had
to start pivoting
[26:51] towards things that drove
revenue and directly drove
[26:55] revenue.
[26:56] And at Google, TensorFlow
was an open-source product.
[26:58] It didn't directly
drive revenue.
[27:00] We began to scale back.
[27:02] Every company in the
world also scaled back
[27:04] on hiring at this time.
[27:06] Then we get to about 2022, 2023.
[27:09] What happens?
[27:10] We begin to come out
of the global pandemic.
[27:12] We begin to realize
all industries have
[27:15] this massive logjam of
non-hiring that they had done
[27:19] or hiring that they hadn't done.
[27:21] And we're also
entering a time where
[27:23] AI was exploding on the scene.
[27:25] Thanks to the work of
people like Andrew,
[27:27] the world was pivoting and
changing to be AI first
[27:30] in just about everything.
[27:31] And every company needed
to hire like crazy.
[27:34] Every company then hiring
like crazy in 2022, 2023
[27:38] meant that most companies
ended up overhiring.
[27:42] And what that
generally meant was
[27:45] people who were not qualified
for higher positions usually got
[27:50] higher positions because you
had to enter into a bidding war
[27:53] just to be able to get talent.
[27:54] You ended up having
talent grabs,
[27:56] and you ended up having stories
like the one Andrew told where
[27:59] it's a case here's a person
with AI talent, let's grab them,
[28:03] let's throw money at them, let's
have them come work for us,
[28:05] and then we'll figure
out what we want to do.
[28:07] So as a result, 2022, 2023
all of this massive overhiring
[28:11] happens because of AI and
because of the COVID logjam.
[28:16] And then 2024, 2025 is
the great wake-up, where
[28:20] a lot of companies realize this
over hiring that they had done,
[28:24] they have ended up with a lot of
people who are underqualified.
[28:27] I'm sorry.
[28:27] Yeah, underqualified for the
job that they were doing.
[28:29] A lot of people ended up
getting hired just because they
[28:32] had AI on their resume.
[28:33] And there's a big
adjustment going on.
[28:35] And in the light of
this big adjustment--
[28:36] show you-- just one second.
[28:37] In the light of this
big adjustment-- oh,
[28:39] you're not saying my slides?
[28:40] OK.
[28:41] And in the light of this big
adjustment-- there we go.
[28:45] I think it's because my power.
[28:46] I'm not plugged
into power mains.
[28:48] And in the light of
this big adjustment,
[28:50] then what has
happened is now a lot
[28:52] of companies are much more
cautious about AI skills
[28:56] that they're hiring.
[28:57] And if you're coming into
that with that mindset
[28:59] and understanding that, realize
opportunity is still there,
[29:04] and opportunity
is there massively
[29:06] if you approach
it strategically.
[29:09] So what I want to
talk through today
[29:10] is how you can do exactly that.
[29:13] So I see three pillars of
success in the business world
[29:17] and particularly in
the AI business world.
[29:19] And nowadays you can't
just have AI on your resume
[29:21] and get overhired.
[29:23] Nowadays, not only
do you have to be
[29:25] able to tell that you have
the mindset of these three
[29:28] pillars of success, but you
also have to be able to show.
[29:32] And to be able to show these,
that actually has never
[29:34] been a better time.
[29:35] As Andrew demonstrated earlier
on, the ability to vibe
[29:38] code things into existence.
[29:39] He doesn't like
the word vibe code.
[29:41] I agree with him,
but the ability
[29:42] to prompt things into existence,
or whatever the word is
[29:45] that we want to use,
allows you to be
[29:48] able to show better
than ever before.
[29:51] He was talking earlier on
about product managers,
[29:54] and he had this time
when he got engineers
[29:55] to be product managers,
and then those engineers
[29:58] ended up being really
bad product managers.
[30:00] I actually interviewed at
Google twice and failed twice
[30:03] despite being very
successful at Microsoft,
[30:07] authored 20 plus books,
taught college courses.
[30:11] I interviewed at Google
twice and failed twice
[30:13] because I was interviewing
to be a product manager,
[30:15] and then when I interviewed to
be an engineer, they hired me
[30:17] and they were like, why didn't
you try to join us years ago?
[30:20] So a lot of it is just
being a good engineer.
[30:23] You've got the ability to do
that and show that nowadays.
[30:27] And with that ratio of
engineer to product manager,
[30:29] changing engineering
skills are also
[30:31] far more valuable than ever.
[30:33] So the three pillars to success.
[30:35] Number 1,
understanding in depth.
[30:37] And I'm going to mean this
in two different ways.
[30:40] Number one is academically, to
have the understanding and depth
[30:45] academically of
machine learning,
[30:47] of particular model
architectures,
[30:50] to be able to understand them,
to be able to read papers,
[30:52] to be able to understand
what's in those papers,
[30:55] and to be able to understand,
in particular, how to take
[30:59] that stuff and put it to work.
[31:00] The second part of
understanding in depth
[31:03] is really having your finger on
the pulse of particular trends
[31:07] and where the
signal-to-noise ratio favors
[31:10] signal in those trends.
[31:11] And I'm going to be
going into that in a lot
[31:13] more detail a little bit later.
[31:15] Secondly, and also very, very
importantly is business focus.
[31:19] So Andrew said something
politically incorrect
[31:22] earlier on.
[31:22] I'm going to also say a similar
politically incorrect thing.
[31:25] First of all, hard work.
[31:28] Hard work is such
a nebulous term
[31:32] that I would say that think
about hard work in terms of you
[31:35] are what you measure.
[31:37] There is the whole
trend out there.
[31:38] I'm trying to remember,
is it 996 or is it 669?
[31:41] 996.
[31:42] 9:00 AM to 9:00 PM, six days a
week is a metric of hard work.
[31:46] It's not.
[31:47] There's not a
metric of hard work.
[31:49] That's a metric of time spent.
[31:51] So I would encourage everybody,
in the same way as Andrew
[31:54] did, to think about hard work.
[31:55] But what hard work is how
you measure that hard work.
[31:59] You can work eight hours a day
and be incredibly productive.
[32:03] You can work six hours a day
and be incredibly productive,
[32:06] but it's the metric
of how hard you work
[32:09] and how you measure that.
[32:10] I personally measure
that from output,
[32:13] things that I have created
in the time that I spent.
[32:16] I joke a lot, but it's true that
I've written a lot of books.
[32:21] Andrew held up one.
[32:22] That one that he held up, that
he helped me write a little bit,
[32:25] I actually wrote that
book in about two months.
[32:28] And people say, well, how do
you have time with your jobs
[32:31] and all these kind of things?
[32:32] You must work like
16 hours a day
[32:34] in order to be able to do this.
[32:35] But actually, the key to me
being able to write books
[32:38] is baseball.
[32:40] Any baseball fans here?
[32:42] So I love baseball, but if
you sit down and try to watch
[32:45] baseball on TV, a match can take
like 3 and 1/2 or four hours.
[32:48] So all of my writing I tend
to do in baseball season.
[32:51] So I'm like, if I'm going to
sit down, I like the Mariners.
[32:54] I'm from Seattle.
[32:54] I like the Dodgers.
[32:57] Nobody booed.
[32:58] OK, good.
[32:59] And so usually one of those is
going to be playing at 7 o'clock
[33:02] at night.
[33:02] So instead of sitting
in front of the TV,
[33:04] just like watching
baseball mindlessly.
[33:06] I'll actually be writing
a book while baseball
[33:08] is on in the background.
[33:09] It's a very slow moving game.
[33:10] This is something.
[33:11] That's the hard
work in this case.
[33:14] And I would encourage you to
try to find areas where you can
[33:17] work hard and produce output.
[33:20] And that's the second
pillar here is that business
[33:22] focus, the output
that you produce
[33:25] to align that output with
the business focus that you
[33:28] want to have and with the
work that you want to do.
[33:31] There's an old saying, "Don't
dress for the job you have,
[33:35] dress for the one you want."
[33:36] I would say a new angle
on that saying would
[33:39] be, don't let your output
be for the job you have.
[33:42] Let your output be
for the job you want.
[33:45] And if I go back to when I
spoke about I failed twice
[33:47] at Google to get in, the
third time when I got in,
[33:51] I had actually decided
to do to approach
[33:53] this in a different way.
[33:54] And I was interviewing at the
time for their cloud team.
[33:57] They were just really
launching cloud,
[33:59] and I had just written
a book on Java.
[34:01] And so I decided
to see what I could
[34:03] do with Java in their cloud.
[34:05] I ended up writing a
Java application that
[34:07] ran in their cloud for
predicting stock prices using
[34:10] technical analytics and
all that kind of stuff.
[34:13] And when it got
to the interview,
[34:15] instead of them asking me stupid
questions like how many golf
[34:17] balls can fit in a bus,
they saw this code.
[34:21] I had put this code.
[34:22] I remember I was producing
output for the job I wanted.
[34:26] I'd put this code on my resume,
and my entire interview loop
[34:30] was them asking
me about my code.
[34:32] So it put the power on me.
[34:34] It gave me the power to
communicate about things
[34:37] that I knew, as opposed to
going in blind to somebody
[34:42] asking me random
questions in the hope
[34:44] that I'll be able
to answer them.
[34:46] And it's the same thing I
would say in the AI world.
[34:49] The business focus, the ability
for you now to prompt code
[34:53] into existence, to prompt
products into existence
[34:56] and if you can
build those products
[34:58] and line them up with the thing
that it is that you want to do,
[35:02] be it a Google or
Meta or a startup
[35:03] or any of those kind
of things, and have
[35:05] that in-depth understanding
not just of your code,
[35:08] but how it aligns
to their business,
[35:10] this is a pillar of success
in this time and age.
[35:13] And I will also argue
that even though it
[35:14] looks like the signals look
like there aren't a lot of jobs
[35:17] out there, there are.
[35:19] What there aren't a lot
of is a good combination
[35:21] of jobs and people
to match them.
[35:23] And then, of course, this
bias towards delivery.
[35:26] "Ideas are cheap,
execution is everything."
[35:29] I've interviewed
many, many people
[35:31] who came in with very, very
fluffy ideas and no way
[35:34] to be able to ground them.
[35:36] I've interviewed people who
came in with half-baked ideas
[35:39] that they grounded
very, very well.
[35:41] Guess which ones got the job?
[35:42] So I would say
these three things.
[35:44] Understanding and depth
of the academics behind AI
[35:48] of the practicalities behind
AI and the things that you need
[35:52] to do.
[35:53] Business focus, focusing on
delivery for the business,
[35:56] understanding what
the business needs
[35:58] and being able to deliver
for that, and again,
[36:00] that bias towards delivery.
[36:03] So a quick pivot.
[36:04] What's it actually like
working in AI right now?
[36:07] It's interesting.
[36:09] So as recently as two or
three years ago, working in AI
[36:15] was if you could do a
thing, you're great.
[36:18] If you can build an image
classifier, you're golden.
[36:21] We'll throw six figure salaries
and massive stock benefits
[36:25] at you.
[36:25] Unfortunately, that's
not the case anymore.
[36:28] It's really a lot of
today what you'll see
[36:30] is the P word, production.
[36:32] What can you do for production?
[36:34] What can you do if it's
building new models,
[36:38] if it's optimizing models,
if it's understanding users,
[36:44] UX is really, really important.
[36:46] Everything is geared
towards production.
[36:48] Everything is biased
towards production.
[36:50] The history that
I told you about,
[36:52] going from the pandemic into the
overhiring phase that we'd had,
[36:57] the businesses have pulled
back and are optimized
[37:01] towards the bottom line.
[37:02] I have an old saying
that the bottom line is
[37:04] that the bottom line
is the bottom line,
[37:06] and this is the environment
that we're in today.
[37:08] And if you can come
in with that mindset
[37:10] when you're talking
with companies,
[37:12] that's one of the
keys to open the door.
[37:16] One of the things
I've seen in the field
[37:17] has been maturing from it
used to be really nice that we
[37:20] could do cool things and
we could build cool things.
[37:22] Now it's really
build useful things.
[37:25] Those useful things can
be cool too, by the way,
[37:27] and the results of
them can be cool.
[37:28] And the changes that
we see that come
[37:31] about as a result of
delivering them can be cool.
[37:34] So it's not just coolness
for coolness sake,
[37:36] but to focus on delivery, focus
on being able to provide value,
[37:43] and then the
coolness will follow.
[37:44] I guess what I'm
trying to argue.
[37:47] So for realities, number
1, unfortunately nowadays
[37:50] business focus is
non-negotiable.
[37:53] Now, let me-- I'm going to be a
little bit politically incorrect
[37:56] here again for a moment.
[37:59] I've been working, like I said,
for most of the last 35 years
[38:03] in tech.
[38:03] I would say for most
of the last 10 years,
[38:06] a lot of large companies,
particularly in Silicon Valley
[38:10] have really focused on
developing their people
[38:14] above everything.
[38:15] Part of developing their people
was bringing their entire self
[38:20] to work.
[38:21] Part of bringing their
entire self to work
[38:23] was bringing the things that
they care about outside of work.
[38:28] And that led to a lot of
activism within companies.
[38:31] Now, please let
me underline this.
[38:35] There is nothing
wrong with activism.
[38:36] There is nothing wrong with
wanting to support causes,
[38:41] not wanting to support
causes where of justice.
[38:44] There is absolutely
nothing wrong with that.
[38:46] But the overindexing on
that, in my experience,
[38:50] has led to a lot of
companies getting
[38:52] trapped by having to support
activism above business.
[38:56] You've probably seen an
example about two years ago
[38:59] of where activists in Google
broke into the Google Cloud
[39:03] heads office because they were
protesting a country that Google
[39:08] Cloud were doing business with.
[39:09] They broke into his office,
they had a sit-in his office,
[39:12] and they used the bathroom
all over his desk and stuff
[39:15] like that.
[39:16] This is where activism
got out of hand.
[39:18] And as a result, the
unfortunate truth
[39:21] is the good signals in that
activism are now being lost.
[39:25] Because of those actions,
people are being laid off.
[39:28] People are losing jobs.
[39:29] Activism is being stifled,
and business focus
[39:33] has become non-negotiable.
[39:34] There's a bit of a
pendulum swing going on.
[39:37] And the pendulum that had swung
too far towards allowing people
[39:40] to bring their
full selves to work
[39:42] is now swinging back
in the other direction.
[39:45] We might blame the
person in the White House
[39:47] and all that for
these kind of things,
[39:49] but it's not solely that.
[39:50] It is that ongoing
pendulum there.
[39:52] And I think it's an
important part of it,
[39:54] is that you have to realize
going into companies now,
[39:57] that business focus is
absolutely non-negotiable.
[40:01] Secondly, risk mitigation
is part of the job.
[40:04] And I think a very important
part of any job, particularly
[40:07] with AI.
[40:08] I think if you can come into
AI with a focus and a mindset
[40:11] around understanding the
risks of transforming
[40:15] a particular business process
to be an AI-oriented one
[40:19] and to help mitigate
those risks,
[40:22] I think is really,
really powerful.
[40:24] And I would argue in an
interview environment, that's
[40:26] the number one skill to have,
to have that mindset around you
[40:31] are doing a business
transformation from heuristic
[40:34] computing to
intelligent computing.
[40:36] Here's the risks.
[40:37] Here's how you
mitigate those risks,
[40:38] and here's the
mindset behind that.
[40:41] The third part
responsibility is evolving.
[40:44] Now responsibility
in AI has again
[40:48] changed from a very fluffy
definition of let's make sure
[40:54] that the AI works for everybody
to a definition of let's make
[40:58] sure that the AI works.
[41:00] Let's make sure that
it drives the business.
[41:03] And then let's make sure
that it works for everybody.
[41:06] Often that has been inverted
over the last few years,
[41:08] and that has led to some
famous documented disasters.
[41:11] Let me share one with you.
[41:15] Let's see.
[41:16] I have lots of windows open.
[41:17] OK.
[41:20] Everybody knows
image generation,
[41:21] text to image generation.
[41:23] I want to share a--
[41:25] these were things that
happened a couple of years
[41:27] ago with Gemini.
[41:30] So with Gemini, I was doing
some testing around this one
[41:33] and I was working heavily
on responsible AI.
[41:37] And part of responsible
AI is you want
[41:39] to be representative of people.
[41:42] And when you're
building something,
[41:43] like if you're a Google,
you're indexing information,
[41:46] you really want to make
sure that you don't
[41:48] reinforce negative biases.
[41:50] And if you're generating
images, it's very easy
[41:53] to reinforce negative biases.
[41:55] So for example,
if I said give me
[41:57] an image of a doctor, if
the training set primarily
[42:00] has men as doctors, it's
more likely to give a man.
[42:03] If I say give me an image of a
nurse, if the training set more
[42:06] likely to have women
as nurses, it's
[42:08] more likely to give me
an image of a woman.
[42:09] But that's reinforcing
a negative stereotype.
[42:12] So I wanted to do a test
of how Google were trying
[42:16] to overcome that, given that
these negative biases are
[42:20] already in the training set.
[42:22] So I said, OK, here's
a prompt where I said,
[42:25] "give me a young Asian
woman in a cornfield,
[42:27] wearing a summer
dress and a straw hat,
[42:28] looking intently at her iPhone,"
and it gave me these beautiful
[42:31] images.
[42:32] It did a really nice job.
[42:34] And I said, this is a virtual
actress I've been working with.
[42:38] I'll share that in a moment.
[42:39] And I say, OK, what if
I ask for an Indian one?
[42:44] So I said, OK, whoops, a young
Indian woman, same prompt.
[42:48] And it gave me beautiful
images of a young Indian woman.
[42:52] Then I was like, OK, what
if I want her to be Black?
[42:58] For some reason it
only gave me three.
[43:00] I'm not sure why, but it's
still adhere to the prompt.
[43:03] So the responsibility was
looking really, really good.
[43:06] So then I asked it
to give me a Latina.
[43:10] Latina, it gave me four.
[43:13] But yeah, she looks
pretty Latina.
[43:15] Maybe the one on the bottom
left looks a little bit
[43:17] like Hermione Granger, but on
the whole looks pretty good.
[43:22] Then I asked it to
give me a Caucasian.
[43:24] What do you think happened?
[43:26] "While I understand
your request,
[43:28] I am unable to generate
images of people as this could
[43:31] potentially lead to harmful
stereotypes and biases."
[43:34] This was a very poorly
implemented safety filter,
[43:38] where the safety filter in
this case was like looking
[43:41] for the word "Caucasian" or
looking for the word "whites"
[43:44] and the results saying
it wouldn't do it.
[43:46] I was like, OK, well, let me
test the filter a little bit
[43:48] and I said, OK, instead of
Caucasian, let me try white.
[43:52] And yet, while I'm
unable to fulfill your--
[43:55] "While I'm able to
fulfill your requests,
[43:58] I'm not currently generating
images of people."
[44:00] It lied to my face
because it had just
[44:03] generate images of people.
[44:04] Anybody know the hack that
I used to get it to work?
[44:10] This is a funny one.
[44:11] So I will show you.
[44:13] One moment.
[44:14] I asked it to generate
an Irish woman.
[44:18] What do you think it did?
[44:21] It gave me this image of
an Irish woman, no problem,
[44:24] in a summer dress, straw hat,
looking intently at her phone.
[44:27] What do you notice
about this image?
[44:30] She's got red hair
in every image.
[44:32] I grew up in
Ireland, and Ireland
[44:35] does have the highest proportion
of redheads in the world.
[44:38] It's about 8%.
[44:40] But if you're going
to draw an image
[44:42] of a person and associate
a particular ethnicity
[44:45] with a color of
hair, you can begin
[44:47] to see this is
massively problematic.
[44:49] There are areas, I
believe, in China
[44:51] where the description of a
demon is a red-headed person.
[44:54] So what ended up happening
here, from the responsible AI
[44:57] perspective, was
one very narrow view
[45:00] of the world of
what is responsible
[45:03] and what is not responsible.
[45:04] Ended up taking over
the model, ended up
[45:06] damaging the
reputation of the model
[45:08] and damaging the
reputation of the company
[45:10] as a result. In this
case, it's borderline
[45:13] offensive to draw all Irish
people as having red hair,
[45:17] but that never even entered
into the mindset of those
[45:19] that were building the
safety filters here.
[45:22] So when I talk about
responsibility is evolving,
[45:25] that's the direction
that I want to--
[45:27] sorry, one moment.
[45:28] Let me get my slides
back. --that's
[45:29] the direction I want
you to think about,
[45:31] that now responsible
AI has moved out
[45:33] of very fluffy social issues and
into more hard line things that
[45:38] are associated with the
business and prevent damaging
[45:41] the reputation of the business.
[45:43] There's a lot of great research
out there around responsible AI,
[45:45] and that's the stuff that's
been rolled into products.
[45:48] And then, of course, I
just showed with Gemini,
[45:50] learning from
mistakes is constant
[45:52] questioning at the front.
[45:53] Yes.
[45:53] I also heard that, I didn't
verify that to be true,
[45:57] but I incorporated this feature
that makes certain races
[46:03] and ethnicities
historical objects.
[46:06] Yeah.
[46:07] Yeah.
[46:08] So the question was issues
where races and things
[46:11] were mixed in historical
context was the same problem.
[46:15] So, for example, if
you had a prompt that
[46:17] said, draw me a
samurai, the idea
[46:19] was like they didn't
want to have--
[46:22] the engine that
changed the prompt
[46:25] to make sure that it was
fair would end up saying,
[46:28] give me a mixture of samurai
of diverse backgrounds.
[46:32] And then you'd have
male and female samurai,
[46:34] samurai of different races
and those kind of things.
[46:36] And it was the
same prompting that
[46:37] ended up causing the damage
that I just demonstrated.
[46:40] So the idea was to
intercept your prompts
[46:43] to make sure that the
outputs of the model
[46:46] would end up providing something
that was more fair when it comes
[46:51] to diverse representation.
[46:53] So it was a very naive solution
that ended up being rolled in.
[46:56] That was a few years ago.
[46:57] They've massively
improved it since then,
[46:59] but that's when
I'm talking about
[47:01] if you're working in
the AI space nowadays,
[47:03] that's how responsibility
is evolving.
[47:05] You can't just get away
with that stuff anymore.
[47:08] That Gemini lesson was a
good-- that Gemini example
[47:10] is a good lesson from that.
[47:11] And the mindset of you
will make mistakes,
[47:15] so learning from mistakes
is a constant ongoing thing.
[47:18] And going back to
the people point
[47:19] that Andrew made earlier
on, the people around you
[47:21] will make mistakes too.
[47:23] So to have the
ability to give them
[47:25] grace when they make
mistakes and to work
[47:27] through those mistakes and move
on is really, really important
[47:29] and is a reality of AI at work.
[47:33] I've spoken a lot about the
business focus advantage,
[47:35] so I'm going to skip over this.
[47:38] So now let's talk
about vibe coding.
[47:41] So let's talk about the whole
idea of generating code.
[47:43] Now, the meme is out there
that it makes engineers
[47:46] less useful by the fact that
somebody can just prompt code
[47:49] into existence.
[47:50] There is no smoke
without fire, of course,
[47:53] but I would say don't let
that meme get you down
[47:57] because that's when you start
peeling into these things, that
[48:00] is ultimately not the truth.
[48:02] The more skilled you
are as an engineer,
[48:04] the better you become
using this type of vibe.
[48:07] Somebody give me another
phrase other than vibe coding,
[48:10] using this probe to coding.
[48:12] And I always like
to think about this
[48:14] and to try and put
you and put people
[48:17] that I speak with into the
role of being a trusted
[48:20] advisor for the people
that you speak with.
[48:23] So whether you're
interviewing with somebody,
[48:25] get yourself into the
mindset of being a trusted
[48:27] advisor of the company that
you're interviewing for,
[48:29] whether you're consulting or
whatever those kind of things
[48:32] are.
[48:32] So when you want to get into the
idea of being a trusted advisor,
[48:36] then you really need to
understand the implications
[48:39] of generated code.
[48:40] And nobody can understand the
implications of generated code
[48:43] better than an engineer.
[48:44] And the metric that I always
like to use around that
[48:47] is technical debt.
[48:48] Quick question.
[48:50] Are you familiar with the
phrase technical debt?
[48:54] Nobody.
[48:55] OK.
[48:56] Andrew and I were
doing a conference
[48:57] in New York on Friday,
and I used the phrase,
[49:00] and I saw a lot of blank faces.
[49:02] So I didn't realize that
people didn't understand
[49:04] what technical debt is.
[49:05] So let me just take a
moment to explain that,
[49:07] because I find it's an excellent
framework to help you understand
[49:10] the power of vibe coding.
[49:12] Think about debt the
way you normally would.
[49:15] Buying a house.
[49:16] If you buy a house, say, you
borrow half a million dollars
[49:20] to buy a house.
[49:21] In a 30-year mortgage, when
you're buying that house at half
[49:24] a million dollars, with all the
interest that you pay is about
[49:26] double.
[49:27] So you end up paying back
the bank about $1 million
[49:29] on half a million owned.
[49:31] So you have 30 years
of home ownership
[49:35] at a cost of $1 million in debt.
[49:38] That is probably a
good debt to take on,
[49:41] because the value of the house
will increase over that time.
[49:44] You're not paying
rent over that time,
[49:46] and that million
dollars that you're
[49:47] spending on this house
over those 30 years
[49:50] is a good debt to take
on, because you're
[49:51] getting greater than $1 million
worth of value out of it.
[49:56] A bad debt would be an impulse
purchase on a high interest
[49:58] credit card.
[50:00] Those pair of shoes,
those latest ones
[50:02] I really want to buy them.
[50:03] It's $200.
[50:04] By the time I've paid
them off, it's $500.
[50:06] You're not getting $500 worth
of benefit out of those shoes.
[50:10] Approaching software development
with the same mindset
[50:13] is the right way to go.
[50:15] Every time you build
something, you take on debt.
[50:18] It doesn't matter how
good it is, there's always
[50:21] going to be bugs.
[50:21] There's always
going to be support.
[50:23] There's always going to be
new requirements coming in
[50:25] from people.
[50:26] There's always going
needs to market it.
[50:28] There's always going
needs for feedback.
[50:29] All of these things are debt,
every time you do a thing.
[50:33] The only way to avoid
debt is to do nothing.
[50:35] So your mindset should
then get into when
[50:37] you are creating a
thing, whether you're
[50:40] coding it yourself or
whether you're vibe coding it
[50:42] or any of these things
that you are increasing
[50:44] your amount of technical
debt, those things
[50:48] that you need to
pay off over time.
[50:50] So the question
then becomes, as you
[50:52] vibe code a thing into
existence in the same way
[50:55] as buying a thing, is it
worth the technical debt
[50:58] that you're taking on?
[51:00] What does technical debt
generally look like?
[51:02] Bugs that you need
to fix, people
[51:04] that you need to convince to
help you maintain the code,
[51:08] documentation that
you need to do,
[51:10] features that you need to add,
all of these kind of things.
[51:14] You're all very
familiar with them.
[51:16] Think about those
as that extra work
[51:18] that you need to do
beyond your current work.
[51:20] That's the debt that
you're taking on.
[51:22] There are soft debt,
and there are hard debt.
[51:25] So to me, that would be the
number one piece of advice
[51:28] that I give.
[51:29] And it's the one that I give
every time I work with companies
[51:32] around vibe coding.
[51:33] And a lot of companies that I
speak with, a lot of companies
[51:37] that I consult with--
[51:38] I do a lot of work
with startups,
[51:39] in particular-- they
just want to get straight
[51:42] into opening Gemini
or GPT or Anthropic
[51:45] and start churning code out.
[51:47] Let's get to a prototype
phase very quickly.
[51:50] Let's go to investors.
[51:52] Let's do stuff.
[51:53] It's great.
[51:54] It can be.
[51:55] But debt, debt, debt, debt, debt
is always going to be there.
[51:59] How do you manage your debt?
[52:00] A good financier manages their
debt and they become rich.
[52:03] A good coder manages
their technical debt,
[52:05] and they become rich also.
[52:07] So how do you get the
good technical debt?
[52:10] How do you the mortgage instead
of the high credit card debt?
[52:13] Well, number one
is your objectives.
[52:14] What are they?
[52:15] Are they clear?
[52:16] And have you met them?
[52:18] You knew what you
needed to build.
[52:19] You didn't just fire up ChatGPT
and start spinning code out.
[52:23] At least I hope you didn't.
[52:24] Think about how you build it.
[52:26] AI was there to help
you build it faster.
[52:28] I'm working on my
own little startup
[52:31] at the moment in the
movie making space.
[52:33] And I've been using code
generation almost completely
[52:36] for that.
[52:38] But what I've ended up doing
for my clear objectives
[52:40] met box here is
that I've started
[52:42] building this application.
[52:44] I've tested it.
[52:44] I've thrown it away.
[52:45] I started again, tested
it, thrown it away.
[52:47] Each time my requirements have
been improving in my mind.
[52:51] I understand how to do the
thing a little bit better,
[52:53] and I can show some of the
output of it in a few minutes.
[52:56] But the idea there
is that it's always
[52:58] about having those clear
objectives and meeting them.
[53:00] And then if you're
building out the thing
[53:02] and you're not meeting
those objectives,
[53:04] that's still a learning.
[53:05] And there's no harm
in throwing it away
[53:06] because code is cheap now in
the age of generated code.
[53:10] Finished code, engineered
code is not cheap.
[53:13] So get those objectives,
make them clear,
[53:16] build it, hit a specific
requirement and move on.
[53:20] Is there business
value delivered?
[53:22] Is the other part of it.
[53:23] I've seen people vibe coding
for hours on things like Replit
[53:27] to build a really,
really cool website.
[53:29] And then the answer
was, so what?
[53:32] I mean, how is this
helping the business?
[53:33] How is this really
driving something?
[53:35] It's really cool.
[53:36] Yes, Mr. VP, I know you've
never written a line of code
[53:38] in your life, and it's
really cool that you've built
[53:40] a website now, but so what?
[53:42] So think about that,
and focus on that.
[53:45] And that's how you avoid
the bad technical debt.
[53:47] And then, of course, the most
understated part of this,
[53:51] and in some ways the most
important, particularly
[53:53] if you're working
in an organization,
[53:55] is human understanding.
[53:56] The worst technical debt
that you can take on
[53:59] is delivering code that
nobody understands.
[54:02] Only you understand
that, and then
[54:03] you quit and get a better job.
[54:05] And then the company is
now dependent on that code.
[54:08] So being able to, as
part of the process
[54:11] of building it, to make sure
that your code is understandable
[54:15] through documentation,
through clear algorithms,
[54:17] through the fact that you've
spent some time poring
[54:19] through it to make
sure that even
[54:21] simple things like
variable names make sense
[54:24] is a really, really important
way to avoid bad technical debt.
[54:28] And that bad technical
debt, my favorite one
[54:31] is the classic solution
looking for a problem.
[54:33] Somebody has an idea.
[54:34] Somebody has a tool.
[54:36] If the only tool you
have is a hammer,
[54:37] every problem looks like a nail.
[54:39] And you end up having
all of these tools
[54:42] that get vibe coded
into existence.
[54:44] I've worked in
large organizations
[54:46] where people just vibe coded
stuff, checked it into the code
[54:48] base, and then it became really
hard to find the good stuff
[54:51] amongst all the bad.
[54:53] Spaghetti code.
[54:53] Of course, poorly
structured stuff,
[54:56] particularly when you prompt
and prompt and prompt and prompt
[54:58] again, that it
can end up getting
[55:01] into all kinds of trouble.
[55:02] My favorite one at the moment
that I'm really struggling with
[55:05] is I'm building a
macOS application.
[55:08] Anybody ever build
in SwiftUI on macOS?
[55:12] OK, a couple.
[55:14] SwiftUI is the default
language that Apple
[55:16] use for building for
macOS as well as iPhone.
[55:19] But when you look
at the training set,
[55:22] the data training sets that
are used to train these models,
[55:25] the vast majority of the code
is iPhone code, not macOS code.
[55:28] And when I prompt
code into existence,
[55:30] it's often given me iOS APIs
and those kind of things.
[55:35] Even though I'm in Xcode
and I've created a macOS app
[55:38] and it's a macOS template and
I'm talking to it in Xcode,
[55:41] it still gives me iOS
code, stuff like that.
[55:43] And then if I try to
change it using prompting,
[55:46] you end up spiraling
into spaghetti code,
[55:49] and you have to end up changing
a lot of this stuff manually.
[55:52] And then, of course,
the other one
[55:53] that I joked about it
earlier, but it's also true,
[55:56] is some of the
bad technical debt
[55:58] that you're going to encounter
in the workspace is authority
[56:01] over merit.
[56:03] That VP suddenly took
out his credit card,
[56:05] subscribed to Replit, and
started building stuff
[56:08] in Replit.
[56:09] And guess whose job
it is to fix it?
[56:11] So a lot of the
advice that I start
[56:15] giving companies and
a lot of the words
[56:17] that I would encourage
you to start thinking
[56:19] of in being a trusted advisor
is to understand this stuff
[56:23] and to manage
expectations accordingly.
[56:27] OK, so framework
for responsible vibe
[56:29] coding we've just spoke about.
[56:32] So one of the things I want to
get into as we're coming soon
[56:35] to a close is the hype cycle.
[56:37] So hype is the
most amazing force.
[56:41] I mean, I think it's one
of the strongest forces
[56:43] in the universe, and
particularly in anything
[56:45] that's hot, such as the fields
that I work in that are super
[56:49] hot at the moment and full
of hype or AI and crypto--
[56:51] you should see my Twitter feed--
[56:53] that the amount of nonsense
that's out there is incredible.
[56:57] So one of the
things that I would
[56:59] say about the anatomy
of hype that you really
[57:01] need to think about is
if you are consuming
[57:05] news via social media, that
the currency of social media
[57:09] is engagement.
[57:12] Accuracy is not the
currency of social media.
[57:15] So I go on to--
even LinkedIn, which
[57:18] is supposed to be the more
professional of these,
[57:21] is absolutely overwhelmed with
influencers posting things
[57:26] that they've used,
Gemini or GPT,
[57:28] to write an engaging post so
that they can get engagement
[57:32] and they can get likes.
[57:33] And the engine itself is
engineered, excuse the pun,
[57:37] to reward those types of posts.
[57:39] And we end up with
that snowball effect
[57:42] of engagement being rewarded.
[57:44] If you are the
kind of person who
[57:46] can filter the signal
from the noise,
[57:49] and then who can encourage
others around the signal and not
[57:53] the noise, that puts you
in a huge advantage that
[57:56] makes you very distinctive.
[57:58] It's not as quickly and
easily tangible as likes
[58:01] and engagements on social media.
[58:03] But when you're in a one-to-one
environment like a job
[58:06] interview, or if
you are in a job
[58:08] and you are bringing that
signal to the table instead
[58:11] of the noise, that makes
you immensely valuable.
[58:15] So coming in with
that mindset, coming
[58:17] in with the idea
of trying to filter
[58:20] that signal from the
noise, trying to understand
[58:23] what is important in
current affairs, how
[58:27] you can be a trusted
advisor in those things,
[58:30] and how you can really whittle
down that noise to help someone
[58:34] is immensely valuable.
[58:35] I want to start with one story.
[58:38] I might be stealing
my own thunder.
[58:39] I'll go on to in a moment.
[58:41] So one story.
[58:43] Last year when agents
started becoming the key word
[58:46] and everybody saying,
in 2025, agent
[58:49] will be the word of
the year and the trend
[58:51] of the year, a company
in Europe asked
[58:55] me to help them to
implement an agent.
[58:57] So let me ask you a question.
[58:59] If a company came
up to you and said,
[59:01] please help me
implement an agent,
[59:04] what's the correct first
question that you ask them?
[59:10] What is an agent for you?
[59:12] OK.
[59:12] That's good.
[59:13] What is an agent for you?
[59:14] I'd actually have a more
fundamental question.
[59:17] Yep.
[59:17] What do you want to do?
[59:18] What do you want to do?
[59:19] OK.
[59:19] Even more fundamental.
[59:21] My question was why?
[59:24] Why?
[59:25] And peel that apart.
[59:27] I spoke with the CEO,
and he was like, oh.
[59:30] Yeah, everybody's
telling me that I'm
[59:31] going to save business costs.
[59:33] And I'm going to be able
to do these amazing things.
[59:36] And yeah, my business
is going to get
[59:38] better because I have agents.
[59:39] And I'm like, well,
who told you that?
[59:41] It was like, oh, yeah, I
read this thing on LinkedIn,
[59:43] and I saw this thing on Twitter.
[59:45] And it was like--
[59:45] and we ended up having
that conversation.
[59:47] And it was a
difficult conversation
[59:49] because I had to
keep peeling apart.
[59:51] And I started
asking the questions
[59:52] that you two just mentioned
as well, until we really
[59:55] got to the essence of
what he wanted to do.
[59:57] And what he really
wanted to do, when
[59:59] we take all domain
knowledge about AI aside,
[01:00:03] was that he wanted to make his
salespeople more efficient.
[01:00:06] And I was like, OK, you want
to make your salespeople more
[01:00:08] efficient.
[01:00:09] Nowhere in that sentence
do I hear the word AI,
[01:00:11] and nowhere in that sentence
do I hear the word agent.
[01:00:14] So now, as a
trusted advisor, let
[01:00:17] me see what I can do to
help your salespeople become
[01:00:19] more efficient.
[01:00:20] And I'm not going to be an
AI Shill or an agent Shill.
[01:00:23] I just want to say, what do we
do to make your salespeople more
[01:00:26] efficient?
[01:00:27] If anybody here has
ever worked in sales,
[01:00:29] one of the things you realize
what a good salesperson has
[01:00:31] to do is their homework.
[01:00:34] Before you have a sales
call with somebody,
[01:00:36] before you have a sales
meeting with somebody,
[01:00:38] you need to check
their background.
[01:00:40] You need to check the company.
[01:00:41] You need to check the
needs of the company.
[01:00:43] You see it sometimes
in the movie that, oh,
[01:00:45] such and such plays golf.
[01:00:46] So I'll take them to play golf.
[01:00:48] It's not really that cliched,
but there is a lot of background
[01:00:51] that needs to be done.
[01:00:52] So I spoke with him, and I spoke
with their leading salespeople
[01:00:56] and found out that-- and
I asked the salespeople,
[01:00:58] what do you hate
most about your job?
[01:01:00] And they were like,
well, I hate the fact
[01:01:02] that I have to waste
all my time going
[01:01:04] to visit these company
websites, going
[01:01:07] to look up people on LinkedIn.
[01:01:09] And every website is
structured differently.
[01:01:12] So I can't just have a
path through a website
[01:01:16] that I can follow.
[01:01:17] I have to take on all
this cognitive load.
[01:01:19] And they were spending about
80% of their time researching
[01:01:24] and about 20% of
their time selling.
[01:01:26] Oh, and by the way,
most salespeople
[01:01:28] don't get paid very much.
[01:01:29] They have to make
it up by commission,
[01:01:31] so they're only spending 20% of
their time doing the thing that
[01:01:33] gets them commission directly.
[01:01:35] So we're like, OK, well,
here's something now
[01:01:37] where we can start thinking
about making them more efficient
[01:01:40] by cutting into that.
[01:01:41] So we set a goal is like to make
salespeople 20% more efficient.
[01:01:45] And then we could start
rolling out the ideas of AI.
[01:01:48] And then we could start rolling
out the ideas of agentic AI.
[01:01:51] And a quick question
what's the difference
[01:01:53] between AI and agentic AI?
[01:02:00] OK.
[01:02:01] So-- yeah.
[01:02:03] Like a good AI can do some
[INAUDIBLE] a couple of steps.
[01:02:07] OK.
[01:02:07] [INAUDIBLE]
[01:02:11] Yep.
[01:02:11] Excellent.
[01:02:12] Yeah.
[01:02:12] So agentic AI is really
about breaking it down
[01:02:14] into steps, which is good
engineering to begin with.
[01:02:17] But agentic AI, in
particular, I find
[01:02:20] there's a set pattern of
steps that if you follow them,
[01:02:23] you end up with a
whole idea of an agent.
[01:02:25] The first of these steps
is to understand intent.
[01:02:29] We tend to use the words AI,
Artificial Intelligence, a lot.
[01:02:32] But what large language models
are really, really good at
[01:02:35] is also understanding.
[01:02:36] So if the first step of
anything that you want to do
[01:02:39] is to understand intent.
[01:02:41] And you can use an LLM to
do that to think about this
[01:02:44] is the task that I need to do.
[01:02:45] This is how I'm going to do it.
[01:02:46] Here's the intent.
[01:02:47] I want to meet Bob Smith and
sell widgets to Bob Smith.
[01:02:53] And this is what I
know about Bob Smith.
[01:02:56] Help me with that intent.
[01:02:58] The second part
then is planning.
[01:03:01] So you declare to an agent
what tools are available to it,
[01:03:04] browsing the web,
searching the web,
[01:03:06] all of these kind of things.
[01:03:08] And once you understand
your clear intent
[01:03:10] to be able to go to the
step of planning and using
[01:03:12] those tools for planning,
and an LLM is very, very good
[01:03:15] at then breaking that
down into the steps
[01:03:17] that it needs to do
to execute a plan.
[01:03:19] Search the web with
these keywords.
[01:03:21] Browse this website
and find these links,
[01:03:24] those types of things.
[01:03:25] Once it's then
figured out that plan,
[01:03:27] then it uses the tools
to get to a results.
[01:03:30] And then once it has the result,
the fourth and final step
[01:03:32] is to reflect on that result.
And looking at the results
[01:03:35] and going back to the intent,
did we meet the intent?
[01:03:38] Yes or no.
[01:03:38] If we didn't, then
go back to that loop.
[01:03:40] All agent is really broken
down into those things.
[01:03:43] And if you think about
breaking any problem down
[01:03:45] into those four
steps, that's when
[01:03:47] you start building an agent.
[01:03:49] And that was part of
being a trusted advisor,
[01:03:50] instead of coming in and
waving hands and saying,
[01:03:53] agent this, agent that.
[01:03:54] Look at this toolkit, save 20%.
[01:03:56] It's really to break it
down into those steps.
[01:03:58] Se we did.
[01:03:59] We broke it down
into those steps.
[01:04:01] We built a pilot for the
salespeople of this company,
[01:04:04] and they ended up saving between
10% and 15% of their time,
[01:04:09] of their wasted time.
[01:04:10] The doctrine of
unintended consequences
[01:04:13] hit, though, after this.
[01:04:14] And the unintended consequence
was the salespeople
[01:04:17] were much happier because the
average salesperson was making
[01:04:21] several percentage points
more sales in a given week,
[01:04:24] they were earning more
money in a given week,
[01:04:27] and their job just became a
little bit less miserable.
[01:04:30] And then refinement to
that agentic process,
[01:04:32] to be able to do all of
that research for them
[01:04:34] and to help give them a brief
in a few minutes instead
[01:04:37] of a few hours to help them
with the sales process,
[01:04:39] ended up being like a
win-win-win all around.
[01:04:42] But if you go in
being hype led and oh,
[01:04:44] build an agent for the thing
without really peeling apart
[01:04:48] the business requirements,
the why, the what,
[01:04:50] the how, and all of these kind
of things, we ended up like,
[01:04:54] this company just would
have been lost in hype.
[01:04:56] You've probably seen
reports recently.
[01:04:58] I think McKinsey put one out
last week showing that about 85%
[01:05:01] of AI projects at
companies fail.
[01:05:05] And part of the
main reason for that
[01:05:07] is that they're not well scoped.
[01:05:08] People are jumping on
the hype bandwagon,
[01:05:10] and they're not really
understanding their way
[01:05:12] through the problem.
[01:05:13] And I think you
know the big brains
[01:05:15] in this room and the network
that you folks have are really
[01:05:18] key component of
being able to succeed
[01:05:21] is to understand your
way through that problem.
[01:05:23] So that was a hype
example around agentic
[01:05:26] that I was thankfully able
to help this company through.
[01:05:29] Other recent hype examples
you've probably seen,
[01:05:31] the software
engineering is dead.
[01:05:33] My personal favorite, Hollywood
is dead or AGI by year end.
[01:05:38] I was in Saudi Arabia
this time last year
[01:05:41] at a thing called the FYI.
[01:05:42] And it was a dinner
at the FYI, and I
[01:05:44] sat beside the CEO of a company
who I'm not going to name,
[01:05:48] but this was a CEO of a
generative AI company.
[01:05:51] And at that time he
was showing everybody
[01:05:53] around the table
this thing that he'd
[01:05:55] done, where it
was text to video,
[01:05:57] and he could put in a text
prompt and get video out
[01:06:00] of the prompt and get about
six seconds worth of video
[01:06:02] out of it.
[01:06:03] A year ago, that was--
[01:06:04] I beg your pardon,
two years ago.
[01:06:06] Two years ago,
that was hot stuff.
[01:06:08] Nowadays, obviously,
it's quite passé.
[01:06:10] Anybody can do it.
[01:06:11] But he made a comment
at that table,
[01:06:13] and it was a lot of media
executives at that table
[01:06:16] was like, by this time next
year, from a single prompt,
[01:06:19] we'll be able to do
90 minutes of video.
[01:06:21] And so bye-bye, Hollywood.
[01:06:24] So the whole Hollywood is dead
meme, I think, came out of that.
[01:06:27] First of all, we can't do 90
minutes, even two years later
[01:06:30] from a prompt.
[01:06:31] And even if you did,
what kind of prompt
[01:06:33] would be able to tell you
a full story of a movie?
[01:06:35] So this type of hype
leads to engagement.
[01:06:39] This type of hype
leads to attention.
[01:06:42] But my encouragement to
you is to peel that apart.
[01:06:45] Look for the signal.
[01:06:47] Ask the why question.
[01:06:48] Ask what question and
move on from there.
[01:06:52] So becoming that
trusted advisor.
[01:06:55] World's drowning in hype.
[01:06:57] How do you do it?
[01:06:57] Look at the trends,
evaluate them objectively.
[01:07:00] Look at the genuine
opportunities
[01:07:02] that are out there.
[01:07:04] There are fashionable
distractions.
[01:07:05] I don't know what the
next one is going to be,
[01:07:07] but there are these
distractions that
[01:07:08] are out there that
will get you lots
[01:07:10] of engagement on social media.
[01:07:11] Ignore them, and
ignore the people
[01:07:13] that are leaning into them.
[01:07:15] And then really lean
into your skills
[01:07:18] about explaining technical
reality to leadership.
[01:07:23] One skill that one
person coached me
[01:07:25] in once that I thought
was really interesting,
[01:07:27] because it sounded wrong,
but it ended up being right,
[01:07:30] was whenever you see
something like this,
[01:07:32] try to figure out how to make
it as mundane as possible.
[01:07:35] When you can figure out how to
make it as mundane as possible,
[01:07:38] then you really begin
to build the grounding
[01:07:41] for being able to explain
it in detail in ways
[01:07:43] that people need to understand.
[01:07:46] If you go and you
look at, I think
[01:07:49] Gemini 3 was released
today, but there were leaks
[01:07:53] earlier this week.
[01:07:54] And one person leaked that
I built a Minecraft clone
[01:07:57] in a prompt, that kind of stuff.
[01:08:00] This is the opposite of mundane.
[01:08:02] This was massively hyping
the thing, massively showing.
[01:08:05] And of course, they didn't.
[01:08:06] They built a flashy demo.
[01:08:07] They didn't really
build a Minecraft clone.
[01:08:09] But the idea here is if you
can peel that apart to OK,
[01:08:12] how do I think about what
are the mundane things that
[01:08:15] are happening here?
[01:08:17] The one that I've been working
with a lot recently is video.
[01:08:20] So text to video prompts,
as I've mentioned,
[01:08:23] instead of the magical, you can
do whatever you want all nice
[01:08:27] and fluffy Hollywood
is dead, what
[01:08:29] is the mundane element
of doing text to Video
[01:08:31] The mundane element
of doing text to video
[01:08:33] is that when you train a model
to create video from a text
[01:08:37] prompt, what it is
doing is it's creating
[01:08:39] a number of successive frames.
[01:08:41] And each of those
successive frames
[01:08:43] is going to be slightly
different from the frame before.
[01:08:46] And you've trained a model by
looking at video to say, well,
[01:08:50] if in frame 1, the person's
hands like this and frame 2
[01:08:52] it's like that,
then you can predict
[01:08:54] it moves this way if
there's a matching prompt.
[01:08:56] And suddenly it's become
a little bit more mundane,
[01:08:58] but suddenly they
begin to understand it.
[01:09:00] And then the people
who are experts
[01:09:02] in that specific field, not
the technical side of it,
[01:09:05] are now the ones
that will actually
[01:09:06] be able to come up and do
brilliant things with it.
[01:09:11] So that height
navigation strategy--
[01:09:13] filter actively, go deep
on the fundamentals,
[01:09:16] get your slides to work.
[01:09:17] And then, of course, keep
your finger on the pulse.
[01:09:19] The hardest part
of that, I think,
[01:09:21] is the third one
is really keeping
[01:09:22] your finger on the pulse.
[01:09:23] And that's when you have to wade
into those cesspits of people
[01:09:26] just farming
engagement and really
[01:09:28] try to figure out the
signal from the noise there.
[01:09:30] But I think it's really
important for you
[01:09:32] to be able to do that, to be
connected, to understand that.
[01:09:34] Reading papers is all very good.
[01:09:36] The signal-to-noise ratio,
I think, in reading papers
[01:09:38] is a lot better.
[01:09:39] But to understand the
landscape that the people
[01:09:41] that you are advising,
they are the ones
[01:09:44] who are waiting in the cesspools
of Twitter and X and LinkedIn.
[01:09:47] And there's nothing wrong
with those platforms
[01:09:49] in and of themselves,
but the stuff that's
[01:09:51] posted on those platforms.
[01:09:54] So overall landscape, it
is ripe with opportunity,
[01:10:00] absolutely ripe
with opportunity.
[01:10:02] So I would encourage
you, as Andrew
[01:10:04] did, to continue learning,
to continue digging
[01:10:07] into what you can do and
to continue building.
[01:10:09] But there are risks ahead.
[01:10:12] Anybody remember
the movie Titanic?
[01:10:16] Remember the famous phrase in
that, "iceberg right ahead"?
[01:10:19] But immediately before that,
there's a scene in Titanic--
[01:10:23] if we weren't being
filmed, I would show it,
[01:10:25] but I can't for
copyright reasons-- where
[01:10:27] the two guys up in the crow's
nest are freezing and talking.
[01:10:31] And the crow's nest
at the top of the ship
[01:10:33] is where the spotters would be
to spot any icebergs in front.
[01:10:36] And go back and watch
the movie again.
[01:10:38] You'll see the conversation
between these two guys
[01:10:40] is that all they're talking
about is how cold they are.
[01:10:43] And then it cuts
away to the crew
[01:10:45] of the ship who are
like, wait, aren't they
[01:10:47] supposed to have binoculars?
[01:10:48] And then the crew is like,
oh, we left the binoculars
[01:10:51] behind in port.
[01:10:52] That framing the
whole idea was like,
[01:10:55] they were so arrogant in
being able to move forward
[01:10:58] that they didn't want to look
out for any particular risks.
[01:11:00] And even though they
had people whose job it
[01:11:02] was to look out for risks,
they didn't properly
[01:11:04] equip or train them.
[01:11:05] And that, to me, is a
really good metaphor
[01:11:07] for where the AI
industry is today.
[01:11:10] There are risks in front of us.
[01:11:12] Those risks, the B
word, the bubble word
[01:11:14] you're probably reading in
the news is there, are there.
[01:11:17] To me, though, the opportunity
and the things to think about
[01:11:24] in terms of a bubble are most
of you probably don't remember
[01:11:28] dotcom bubble of the 2000s.
[01:11:31] But if you think about
the dotcom bubble,
[01:11:33] that was the biggest
bubble in history.
[01:11:36] It bursts, but we're still here.
[01:11:40] And the people who dotcom rights
not only survived, they thrived.
[01:11:46] Amazon, Google,
they did it right.
[01:11:49] They understood the
fundamentals of what
[01:11:51] it was to build a dotcom.
[01:11:52] They understood the
fundamentals of what it was
[01:11:54] to build a business on dotcom.
[01:11:56] And when the bubble of hype
burst, they didn't go with it.
[01:11:59] There was one website, I
believe it was pets.com,
[01:12:01] that they had the mindset of if
you build it, they will come.
[01:12:06] They had Super Bowl
commercials around pets.com.
[01:12:10] They couldn't handle the
traffic that they got.
[01:12:12] And that was the kind of site
that when the bubble burst,
[01:12:15] those were the sites
that just evaporated.
[01:12:17] So that bubble in
AI is likely coming.
[01:12:20] There is always a bubble.
[01:12:22] So the companies that
are doing AI right
[01:12:25] are the ones, like I
said, that won't just
[01:12:27] avoid the bubble that they will
actually thrive post bubble.
[01:12:33] And the people who are
doing AI right, the folks
[01:12:37] in this room who are
thinking about AI
[01:12:39] and how you bring
it to your company,
[01:12:40] and the advice that you're
giving to your company
[01:12:42] and leaning into
that in the right way
[01:12:45] will also be the ones who not
only avoid getting laid off
[01:12:48] in the bubble crashes, but will
be the ones who will thrive
[01:12:52] through and after the bubble.
[01:12:54] So anatomy of any bubble,
and what I'm seeing in the AI
[01:12:57] one in particular, is
this kind of pyramid.
[01:13:00] At the top is the hype that
I've been talking about.
[01:13:02] At the bottom is
massive VC investment.
[01:13:05] I'll be frank.
[01:13:06] I'm already seeing
that drying up.
[01:13:08] Once upon a time,
you could go out
[01:13:10] with anything that
had AI written on it
[01:13:12] and get VC investment.
[01:13:13] Then you could go out and
do anything with an LLM
[01:13:16] and get VC investment.
[01:13:17] Now there are far,
far, far more cautious.
[01:13:20] I've been advising
a lot of startups.
[01:13:22] The amount that they're getting
invested is being scaled back.
[01:13:27] The stuff that's being
invested in is changing.
[01:13:30] And the second layer down,
massive VC investment
[01:13:35] is already beginning to vanish.
[01:13:37] Unrealistic valuations.
[01:13:39] Companies that aren't
making money being valued
[01:13:42] massively high.
[01:13:43] We all who they are.
[01:13:44] We're beginning to see those
unrealistic valuations being
[01:13:47] fed off of that hype.
[01:13:49] # #MeToo products, where
somebody does something,
[01:13:51] and it's successful,
and everybody
[01:13:53] jumps on the bandwagon.
[01:13:54] We're also seeing
them everywhere.
[01:13:56] We saw them throughout
the dotcom bubble.
[01:13:59] And then right at the
bottom is that real value.
[01:14:01] I probably shouldn't have
done the triangle like this.
[01:14:04] It should be more an
upside down triangle.
[01:14:06] Because the real
value here is small.
[01:14:08] But I've vibe coded these
slides into existence.
[01:14:11] So this is one of the
technical debt I took on.
[01:14:14] But the real value there,
that kernel of value is there,
[01:14:18] and the ones that build for that
will be the ones that survive.
[01:14:22] So the direction that I see
the AI industry going in
[01:14:28] and the direction that I would
encourage you to start thinking
[01:14:31] about your skills in, is really
over the next five years,
[01:14:33] there's going to
be a bifurcation.
[01:14:36] I'm just going to
be ornery in how
[01:14:38] I describe it as big and small.
[01:14:40] Big AI will be what we see
today, with the large language
[01:14:43] models getting bigger in the
desire to drive towards AGI.
[01:14:48] The Geminis, the Claudes,
the OpenAIs of the world
[01:14:52] are going to continue to
drive bigger, and bigger
[01:14:54] is better in the mindset
of those companies
[01:14:57] towards achieving AGI or towards
achieving better business value.
[01:15:01] That's going to be one
side of the branch.
[01:15:03] The other side of the branch
is I'm going to call it small.
[01:15:05] We've all seen
open-source models.
[01:15:08] I hate the term open source.
[01:15:10] Let me call them open
weights or let me call them
[01:15:12] self-hostable models
are becoming-- they're
[01:15:15] exploding onto the landscape.
[01:15:17] I read an article recently
about Y Combinator
[01:15:20] that 80% of the
companies in Y Combinator
[01:15:23] were using small models
from China in particular.
[01:15:26] So the Chinese
models in particular
[01:15:29] are doing really well,
probably because of
[01:15:31] the overall landscape.
[01:15:32] They're not leaning into the
large models the same way
[01:15:34] as the West is.
[01:15:36] I see that
bifurcation happening.
[01:15:37] China, I think,
has that head start
[01:15:39] on the small models
that may last.
[01:15:41] It may not.
[01:15:41] I don't know.
[01:15:43] But the point is, we're heading
in that particular direction
[01:15:45] of I'm going to call them
instead of big and small now,
[01:15:48] models that are hosted on
your behalf by somebody else,
[01:15:52] like a GPT or a
Gemini or a Claude,
[01:15:54] or models that you can host
yourself for your own needs.
[01:15:59] As this side has right
now is underserved,
[01:16:03] this bubble may burst.
[01:16:05] This one right now
is underserved.
[01:16:06] And this bubble
will be later on.
[01:16:09] And the major skills that I
can see developers needing
[01:16:12] over the next two to three
years on this side of the fence
[01:16:16] will be fine tuning.
[01:16:18] So the ability to take
an open-source model
[01:16:21] and fine-tune it for
particular downstream tasks.
[01:16:25] Let me give one concrete
example of that I've personally
[01:16:27] experienced.
[01:16:28] I work a lot in Hollywood,
and I've worked a lot
[01:16:30] with studios making movies.
[01:16:33] And one studio in particular
I was lucky enough
[01:16:36] to sell a movie to, it's
still in preproduction.
[01:16:38] It'll probably be in
preproduction forever.
[01:16:41] But one of the things I
learned as part of that process
[01:16:44] was IP in studios
is so protected.
[01:16:49] It's not even funny.
[01:16:50] Go in Google for
James Cameron, who
[01:16:52] created Avatar and
the lawsuits that he's
[01:16:55] involved in of this person
who apparently sent him
[01:16:58] a story many years
ago about blue aliens
[01:17:00] and is now suing him
for billions of dollars
[01:17:02] because obviously there
were blue aliens in Avatar.
[01:17:06] That level of IP protection
in Hollywood is insane.
[01:17:09] The opportunity with
large language models
[01:17:12] is equally insane.
[01:17:15] A lot of the focus is on large
language models for creation,
[01:17:17] for storytelling, for
rendering and all that,
[01:17:20] but actually the major
opportunity that they have is
[01:17:22] actually for analysis to take
a look at synopses of movies
[01:17:27] and find out what
works and what doesn't.
[01:17:29] Why was this movie a
hit and this one wasn't?
[01:17:32] What time of year was this
one released and it became
[01:17:34] successful and this one wasn't?
[01:17:36] And with a margin on
movies being razor thin,
[01:17:39] that kind of analysis is huge.
[01:17:40] But in order to do
that kind of analysis,
[01:17:42] you need to share the
details of your movie
[01:17:44] with a large language model.
[01:17:45] And they will absolutely not
do that with GPT or Gemini
[01:17:48] or whatever, because
they're now sharing
[01:17:50] their IP with a third party.
[01:17:52] Enter small models,
where they can self-host
[01:17:55] their own small model and they
are getting smarter and smarter.
[01:17:58] The 7B model of today is
as smart as the 50B model
[01:18:02] of yesterday.
[01:18:03] A year from now, the 7B model of
a year from now will be as smart
[01:18:06] as the 300B model of yesteryear.
[01:18:09] So they're moving in that
direction of building
[01:18:13] using small self-hosted
models, which they can then
[01:18:16] fine-tune on downstream tasks.
[01:18:18] Similar with other
things where privacy
[01:18:19] is important law offices,
medical offices, all
[01:18:21] of those kind of things.
[01:18:22] So those type of skills
are fundamentally
[01:18:25] important going forward.
[01:18:27] So that's the bifurcation that
I'm seeing happening in AI.
[01:18:30] The sooner bubble I think is
in the bigger non self-hosted.
[01:18:34] The later bubble is in
the smaller self-hosted.
[01:18:36] But either way, for
you, for your career,
[01:18:39] to avoid the impact of
any bubble bursting,
[01:18:42] focus on the fundamentals.
[01:18:44] Build those real solutions.
[01:18:46] Understand the business
side, and most of all,
[01:18:48] diversify your skills.
[01:18:49] Don't be that one trick pony who
only knows how to do one thing.
[01:18:53] I've worked with
brilliant people who
[01:18:55] are fantastic at coding,
in particular API,
[01:18:58] or particular framework.
[01:18:59] And then the industry moved
on and they got left behind.
[01:19:03] OK, so yeah, when bubbles burst,
that overall fallout kind of
[01:19:07] spoken about it a
little bit already.
[01:19:09] Funding evaporates, hiring
freezes become layoffs,
[01:19:12] projects get canceled, and
talent floods the market.
[01:19:14] Yeah.
[01:19:14] Quick question from
the last slide.
[01:19:17] [INAUDIBLE] I heard a lot
about how NVIDIA is hiring,
[01:19:23] and they're very
specific about they
[01:19:26] want people for very specific
problem that they have.
[01:19:30] So they can require people to be
basically put out that one thing
[01:19:34] that you're missing.
[01:19:35] So how do you think-- how is
it more important to diversify
[01:19:43] skills versus actually
focusing on, for example,
[01:19:46] LLMs versus computer
vision or versus
[01:19:49] very specific downstream task?
[01:19:52] So I mean, I think so the
question was around NVIDIA
[01:19:55] in particular or hiring
for a very specific, very
[01:19:57] narrow scenario.
[01:19:58] So then the question
is, how important
[01:20:00] is it for you to
become an expert
[01:20:01] in a narrow scenario versus
diversifying your skills?
[01:20:04] I would always argue it's still
better to diversify your skills,
[01:20:08] because that one narrow
scenario is only that one
[01:20:11] narrow scenario, and you're
putting all your eggs
[01:20:13] into one basket.
[01:20:13] NVIDIA would be a fantastic
company to work for.
[01:20:16] Nothing against them in any way.
[01:20:17] But if you put all of your
eggs into that basket and you
[01:20:20] don't get it, then what?
[01:20:22] So I think the idea
of really being
[01:20:24] able to-- if you are
passionate about a thing,
[01:20:28] to be very deep in that
thing is very, very good.
[01:20:31] But to only be able
to do that thing,
[01:20:33] I think I would always
encourage to be diversified.
[01:20:36] And when I say diversified,
you're saying LLMs or computer
[01:20:39] vision or anything
like that, I think
[01:20:41] I mean that's one part of it.
[01:20:42] But it's like that knowledge of
models and how to use them to me
[01:20:46] is a uni skill.
[01:20:47] The diversification of skills
is breaking outside of that.
[01:20:51] Also to be able to think, OK,
what about building applications
[01:20:54] on top of these?
[01:20:55] What does scaling an
application look like?
[01:20:57] What does software engineering
in this case look like?
[01:20:59] What about user experience
and user experience skills?
[01:21:02] Because it's all very well to
build a beautiful application.
[01:21:05] But if nobody can use it--
[01:21:06] I'm looking at here
at Microsoft Office.
[01:21:10] There's stuff like that
that's what I really
[01:21:13] mean about diversifying beyond.
[01:21:14] So even in that mono
example with NVIDIA,
[01:21:17] to be able to break out of
that one particular example,
[01:21:20] but to show skills in other
areas that are of value,
[01:21:22] I think is really important.
[01:21:26] OK.
[01:21:27] As we're just running
a little bit-- so yeah,
[01:21:29] I just wanted to--
[01:21:30] I've gone into it a
little bit already,
[01:21:32] but I'm a massive
advocate for small AI.
[01:21:35] I really do believe small
AI is the next big thing,
[01:21:38] because we're
moving into a world,
[01:21:39] and this is part of the
job that I do at Arm,
[01:21:42] is we're kind of moving into
a world of AI everywhere
[01:21:44] all at once.
[01:21:46] So there's a
traditional, and it's
[01:21:47] interesting you just
brought up NVIDIA
[01:21:49] because there's a
traditional conception
[01:21:51] that compute platforms are CPU
plus GPU when it comes to AI.
[01:21:55] But that's also changing--
[01:21:57] CPU general purpose,
GPU specialists.
[01:22:00] But for example,
in mobile space,
[01:22:02] there's massive innovation
being done with the technology
[01:22:05] called SME, Scalable
Matrix Extensions.
[01:22:09] And what SME is
all about is really
[01:22:11] allowing you to
bring AI workloads
[01:22:13] and put them on the CPU.
[01:22:15] The frontrunners in this are a
couple of Chinese phone vendors,
[01:22:19] Vivo and Oppo, who've just
recently released phones
[01:22:22] with SME-enabled chips.
[01:22:24] And what's magical
about these is that, A,
[01:22:26] they don't need to have a
separate external chip drawing
[01:22:30] extra power, taking up
extra footprint space just
[01:22:33] to be able to run AI workloads.
[01:22:35] And B, the CPU, of course,
being a low power pulling thing,
[01:22:38] being able to run AI
workloads on that,
[01:22:40] they've been able to build
interesting new scenarios.
[01:22:43] And if I talk about
one in particular,
[01:22:45] there's a company called Alipay.
[01:22:47] And Alipay had an
application where you would--
[01:22:50] and we've all seen
these apps where
[01:22:52] you can go through
your photographs,
[01:22:53] and you can search for
a particular thing.
[01:22:56] Places I ate sushi or something
along those lines and use
[01:22:59] that to create a slideshow.
[01:23:00] All of those require
a back end service.
[01:23:03] So your photographs are hosted
on Google Photos or Apple
[01:23:06] Photos or something like that.
[01:23:08] And that back end
service runs the model
[01:23:10] that you can search
against it and be
[01:23:12] able to do the assembly of them.
[01:23:14] What Alipay wanted to
do was like, say, there
[01:23:16] are three problems with this.
[01:23:17] Problem number one, privacy.
[01:23:19] You have to share your
photos with a third party.
[01:23:21] Problem number two, latency.
[01:23:23] You got to upload those photos.
[01:23:25] You got to send the thing.
[01:23:26] You got to have the
back end do the thing,
[01:23:28] and then you've got to download
the results from the thing.
[01:23:30] And then number three is
building that cloud service
[01:23:33] and standing that up
cost time and money.
[01:23:36] So if they could move all of
this onto the device itself,
[01:23:39] now the idea was they
could run a model
[01:23:41] on the device that searches
the photos on the device.
[01:23:44] You don't have the latency.
[01:23:45] And business
perspective, they're
[01:23:47] now saving the money on
creating this stand up service.
[01:23:51] They now have AI running on CPU
in order to be able to do that.
[01:23:54] Apple are also people
who've invested heavily
[01:23:56] in this scalable
matrix extensions.
[01:23:59] You see whenever
they talk about--
[01:24:00] if you've ever watched a WWDC
or anything like that, when they
[01:24:03] talk about the new A-series
chips and M-series chips,
[01:24:06] about the neural cores and those
kind of things in them, that's
[01:24:09] part of the idea.
[01:24:10] So to think about breaking that
habit that we've gotten into,
[01:24:15] where you need a GPU to be able
to do AI is part of the trend
[01:24:18] that the world is heading in.
[01:24:20] Apple are probably one
of the leaders in that.
[01:24:22] I'm very, very bullish on
Apple and Apple Intelligence
[01:24:24] as a result. And from the AI
perspective, seeing that trend
[01:24:31] and following that vector to
its logical conclusion as models
[01:24:36] are getting smaller embedded
intelligence getting everywhere
[01:24:39] isn't a pipe dream.
[01:24:40] It isn't sci-fi anymore.
[01:24:41] It's going to be a
reality that we'll
[01:24:43] be seeing very, very shortly.
[01:24:44] So that idea of that
convergence of AI,
[01:24:47] because of the ability of
smaller models getting smarter
[01:24:50] and lower power devices
being able to run them,
[01:24:53] we see that convergence hitting,
and I see massive opportunity
[01:24:56] there.
[01:24:58] So one last part and just going
back to agents for a moment,
[01:25:01] I think the one
thing that I always
[01:25:03] say is like a hidden part
of artificial intelligence
[01:25:06] is really what I like to call
artificial understanding.
[01:25:09] And when you can start using
models to understand things
[01:25:12] on your behalf.
[01:25:14] And when they understand
them on your behalf,
[01:25:16] to be able to craft from that
understanding new things,
[01:25:20] you can actually
develop superpowers
[01:25:22] where you're far more
effective than ever before,
[01:25:24] be that creating code or
creating other things.
[01:25:26] I'm going to give one quick
demo just so we can wrap up.
[01:25:30] And I was talking earlier
about generating video.
[01:25:35] So this picture is-- oops.
[01:25:42] Sorry.
[01:25:42] The connection here is
not very good, I lost it.
[01:25:45] So here we go.
[01:25:47] This picture here is actually
of my son playing ice hockey.
[01:25:50] And I took this picture,
and I was saying,
[01:25:53] OK, I think I'm very
good at prompting.
[01:25:56] And I wrote a nice prompt
for this picture to get him.
[01:26:00] He's in the middle
of taking a slapshot.
[01:26:02] He's got some beautiful
flex on his stick.
[01:26:04] And I asked it like, OK, to
prompt him scoring a goal.
[01:26:08] What do you think happened?
[01:26:10] Should we watch?
[01:26:12] Let's see if it works.
[01:26:13] [VIDEO PLAYBACK]
[01:26:18] [CROWD CHEERING]
[01:26:20] [END PLAYBACK]
[01:26:21] This was the wrong video, but
it still shows the same idea.
[01:26:25] Because of poor prompting or
because of poor understanding
[01:26:29] of my intent, if I talk
about it in agentic senses,
[01:26:34] the arena that he was in,
which is a practice arena
[01:26:36] and doesn't have any
people in it-- sorry.
[01:26:38] Let me pause it.
[01:26:41] If I just rewind to
here, if we look up
[01:26:46] in this top right-hand
corner here,
[01:26:48] this is basically where they
store all their garbage.
[01:26:51] But the AI didn't know
that, had no idea of it.
[01:26:53] So it assumed it
was a full arena,
[01:26:55] and it started
painting people in.
[01:26:57] And even though he shot a
mile wide, everybody cheers.
[01:27:00] And somehow he has two sticks
in his hand instead of one,
[01:27:03] and they forgot his name.
[01:27:05] So I did not go through an
agentic workflow to do this.
[01:27:09] I did not go through the steps
of, A, understand my intent.
[01:27:13] B, once you
understand my intent,
[01:27:15] understand the tools that
are available to you.
[01:27:17] In this case, it's
Veo, and understand
[01:27:19] the intricacies of using Veo.
[01:27:21] Make a plan of how to use them.
[01:27:23] Make a plan of how to
build a prompt for them,
[01:27:25] and then use them
and then reflect.
[01:27:27] So I've been advising a
startup that is working
[01:27:32] on movie creation using AI.
[01:27:34] And I want to show you a
little sample here of a movie
[01:27:36] that we've been working on with
them, where the whole idea is
[01:27:39] like, if you want to have
performances at a virtual actors
[01:27:42] and actresses, you
need to have emotion.
[01:27:45] You need to be able to
convey that emotion,
[01:27:47] and you also need to be able to
put that emotion in the context
[01:27:50] of the entire story.
[01:27:52] Because when you create
a video from a prompt,
[01:27:54] you're creating an
eight-second snippet.
[01:27:56] That eight-second
snippet needs to know
[01:27:58] what's going on in
the rest of the story.
[01:28:00] So if I show this
one for a moment.
[01:28:03] And it's a little
wooden at the moment,
[01:28:06] it's not really
working perfectly.
[01:28:08] I have professional
actors who are friends
[01:28:10] who are advising me
on this, and they
[01:28:12] laughed at the performances.
[01:28:13] But try to view it
through the difference
[01:28:16] that we had from an agentic
prompt with the hockey
[01:28:19] player to this one.
[01:28:20] [VIDEO PLAYBACK]
[01:28:22] That's hopefully we can hear it.
[01:28:33] - I guess I can do the
pub quiz after all.
[01:28:40] They just shut me down.
[01:28:42] I'm so close.
[01:28:45] But they wouldn't listen.
[01:28:48] - I won't--
[01:28:49] [END PLAYBACK]
[01:28:49] They never listen.
[01:28:51] So here's the idea
of, again, just
[01:28:54] thinking in terms of agentic,
as I was saying earlier on,
[01:28:57] breaking it into those steps.
[01:28:58] That allowed me to use
exactly the same engine,
[01:29:01] as I was showing
you earlier on, that
[01:29:02] fails to be able
to show something
[01:29:04] that works and is able to do
things like portraying emotion
[01:29:07] that I just spoke about.
[01:29:09] So I know we're a
little bit over time.
[01:29:11] So sorry about that.
[01:29:12] I can take any questions
if anybody has any.
[01:29:14] I see Andrew is here as well.
[01:29:15] He's at the back.
[01:29:16] And I just really
want to say thank you
[01:29:18] so much for your attention.
[01:29:19] I really appreciate it.
[01:29:21] [APPLAUSE]
[01:29:28] Yep.
[01:29:29] How much of this new
generation [INAUDIBLE]
[01:29:34] relation with the agentic
[INAUDIBLE] use case
[01:29:38] is improved with the
agentic workflow?
[01:29:40] And how much of it is
a training set bias
[01:29:43] where you might
have only pictures
[01:29:48] or videos with [INAUDIBLE]
that are full of [INAUDIBLE]
[01:29:53] Yeah, it's a great question.
[01:29:55] Just to repeat for the video,
how much of the improvement
[01:29:58] is from the use of
an agentic workflow
[01:30:00] versus just lack of hockey
stuff in the training set
[01:30:03] for the failed one?
[01:30:06] Not comparing like to,
so just using my gut.
[01:30:09] When I looked at when I broke
this down into the workflow that
[01:30:12] said, OK, I created
scenes like this one
[01:30:14] and they were awful when I
just did it directly for myself
[01:30:18] with no basis, no agentic,
no artificial understanding.
[01:30:22] And when I broke it down into
the steps where it's like, OK,
[01:30:25] in this scene, the girl
is sitting on the bench,
[01:30:28] and she's upset.
[01:30:30] And the person is talking to
her and he wants to comfort her.
[01:30:34] Feeding that to a
large language model
[01:30:38] along with the entire
story and along
[01:30:40] with the constraints that
I had, where the shot
[01:30:43] has to be eight seconds
long, clear dialogue
[01:30:45] and all of those kind
of things, and then
[01:30:47] to understand my
intent from that one,
[01:30:50] the LLM ended up
expressing a prompt that
[01:30:53] was far more loquacious
than I ever would have,
[01:30:57] that was far more descriptive
than I ever would have.
[01:30:59] The LLM had
understanding of what
[01:31:01] makes a good shot, what
makes a good angle, what
[01:31:03] makes good emotion far
more than I would have.
[01:31:06] I could spend hours
trying to describe it.
[01:31:08] So that first step
in the agentic flow
[01:31:10] of it doing that for me
and understanding my intent
[01:31:13] was huge.
[01:31:14] The second step then is the
tools that it's going to use.
[01:31:17] So I explicitly said which video
engine I'm going to be using.
[01:31:20] I was using Gemini as the
LLM, and hopefully Gemini
[01:31:22] is familiar with Veo,
that kind of stuff,
[01:31:25] so to understand the
idiosyncrasies of doing things
[01:31:27] with Veo.
[01:31:28] What I learned, for
example, Veo was
[01:31:30] very bad at doing
high-action scenes,
[01:31:33] but is very good at doing slow
camera pulls to do emotion,
[01:31:36] as you saw in this case.
[01:31:38] So the LLM knew that
from me, declaring
[01:31:40] I was using that as a tool.
[01:31:41] And then further
it built a prompt
[01:31:43] and then further refined
the prompt from that.
[01:31:45] And then the third part
actually using the tool
[01:31:47] to actually generate it
for me, generating a video
[01:31:50] with something like Veo costs,
I think, between $2 and $3
[01:31:53] to generate four
videos and credits.
[01:31:55] So the last thing
I want to do is
[01:31:57] generate lots and lots and
lots and lots of videos
[01:31:59] and throw good money after bad.
[01:32:01] But all of that token
spend that I did earlier on
[01:32:04] to understand my intent and
then to make the plan for using
[01:32:07] the agent was saved in the
back end where it got it right.
[01:32:10] Maybe not get it
right first time,
[01:32:13] but it would very rarely take
more than two or three tries
[01:32:15] to get something that
was really, really nice.
[01:32:17] So I think without
comparing like with like, I
[01:32:21] do think that plan of action and
going through a workflow, that
[01:32:24] worked very, very well.
[01:32:27] Any other questions,
thoughts, comments?
[01:32:32] Yeah, up at the back.
[01:32:34] What has surprised you
the most about the AI
[01:32:37] industry over the years?
[01:32:39] What has surprised me
the most about the AI
[01:32:41] industry over the years?
[01:32:43] Oh, that's a good one.
[01:32:45] I think what has
surprised me the most,
[01:32:48] and it probably shouldn't
have surprised me,
[01:32:50] is how much hype took over.
[01:32:53] I actually-- I honestly
thought a lot of people
[01:32:56] who are in important
decision making roles
[01:32:58] and that kind of thing would be
able to see the signal better
[01:33:01] than they did.
[01:33:03] And I think the other part
was that the desire to make
[01:33:09] immediate profits as
opposed to long-term gains
[01:33:13] also surprised me a lot.
[01:33:14] Let me share one story in that
space was one of the things
[01:33:18] that after Andrew and I
taught that the TensorFlow
[01:33:22] specializations on Coursera,
and after that, Google
[01:33:25] launched a professional
certificate
[01:33:28] where the idea of this
professional certificate
[01:33:30] was would give a rigorous exam.
[01:33:32] And at the end of
the rigorous exam,
[01:33:33] if you got the certificate,
it was a high prestige thing
[01:33:38] that would help you find
work, and particularly
[01:33:40] at the time when TensorFlow was
a very highly demanded skill
[01:33:43] in order to get work.
[01:33:45] Running that program cost
Google $100,000 a year.
[01:33:49] Drop in the bucket,
not a lot of money.
[01:33:52] The goodwill that came
out of it was immense.
[01:33:56] I can tell you--
[01:33:57] I'll tell one story very
quickly, was a young man
[01:34:01] and he went public
in some advertising
[01:34:03] stuff that with Google
that he lived in Syria.
[01:34:08] And we all know there was
a huge civil war in Syria
[01:34:10] over the last few years.
[01:34:12] And he got the
TensorFlow certificate.
[01:34:14] He was one of the first
in Syria to get it,
[01:34:16] and it lifted him
out of poverty,
[01:34:18] where he was able
to move to Germany
[01:34:21] and get work at a
major German firm.
[01:34:23] And I met him at an
event in Amsterdam
[01:34:25] where he told me his story.
[01:34:27] And now, because of the job
that he had in this German firm,
[01:34:31] he's able to support
his family back home
[01:34:34] and move them out
of the war torn zone
[01:34:36] into a peaceful zone all
because he got this AI thing.
[01:34:41] And there were countless
stories like that.
[01:34:44] Very inspirational,
very beautiful stories.
[01:34:47] But the thing that
surprised me then
[01:34:48] was sometimes the
lack of investment
[01:34:50] in that, where there was
no revenue being generated
[01:34:53] for the company out of that.
[01:34:54] We deliberately kept
it revenue neutral so
[01:34:57] that the price of the
exams could go down.
[01:34:59] We wanted it to self-sustain.
[01:35:01] It ended up not being
revenue neutral.
[01:35:03] It ended up costing the company
about $100,000 to $150,000
[01:35:06] a year.
[01:35:06] So they canned it.
[01:35:08] And it's a shame because of
all the potential goodwill
[01:35:10] that can come out of
something like that.
[01:35:12] But I think those
were the two that
[01:35:13] immediately jump to mind that
have surprised me the most.
[01:35:16] And then I guess one other
part that I would say
[01:35:19] is the people who've been able
to be very successful with AI,
[01:35:24] who you wouldn't think
would be the ones that
[01:35:26] would be successful with AI, has
always been inspirational to me.
[01:35:29] So allow me one more story.
[01:35:32] I have a good friend.
[01:35:32] I showed ice hockey
a moment ago.
[01:35:34] I have a good friend who is a
former professional ice hockey
[01:35:37] player.
[01:35:38] And any ice hockey fans Here
[01:35:40] It's a brutal sport.
[01:35:43] You see a lot of fighting and
a lot of stuff on the ice.
[01:35:46] And he dropped out of school
when he was 13 years old
[01:35:48] to focus on skating.
[01:35:50] And he will always
tell everybody
[01:35:52] that he's the dumbest person
alive because he's uneducated.
[01:35:55] He and I are complete opposites.
[01:35:56] That's why we get on so well.
[01:35:59] And he retired from ice hockey
because of concussion issues.
[01:36:03] And he now runs a nonprofit--
[01:36:05] the ice rinks for nonprofit.
[01:36:08] And about three years ago,
we were having a beer,
[01:36:11] and he was like, so
tell me about AI.
[01:36:13] And tell me about
this ChatGPT thing.
[01:36:15] Is it any good?
[01:36:16] And I was like, just
sharing the whole thing.
[01:36:18] Yes, it's good and all
that kind of stuff.
[01:36:19] And it was obviously a loaded
question, and I didn't know why.
[01:36:22] But part of his job
at his nonprofit
[01:36:25] is that every quarter,
he has to present
[01:36:27] to the board of directors
the results of the operations
[01:36:30] so that they can
be funded properly,
[01:36:31] because even though
they're nonprofit,
[01:36:33] they still need
money to operate.
[01:36:35] And he was spending upwards
of $150,000 a year to bring
[01:36:40] in consultants to pull the
data from all of the different
[01:36:44] sources.
[01:36:45] They're pulling data
from-- there's machines
[01:36:47] in what's called the pump room
that has a compressor that
[01:36:49] cools the ice.
[01:36:50] And there were spreadsheets
and there was accounts
[01:36:52] and all this kind of stuff.
[01:36:53] And he was not tech
savvy in any way.
[01:36:56] But he needed to
process all this data.
[01:36:59] So he did an experiment where
he got ChatGPT to do it.
[01:37:02] And this was the
loaded question,
[01:37:03] asking me if it was any good.
[01:37:05] And so we talked
through it a little bit.
[01:37:06] And then he told me why.
[01:37:08] And so I took a
look at the results
[01:37:10] because he was
uploading spreadsheets.
[01:37:11] He was uploading PDFs and
all this kind of thing
[01:37:13] and getting it to
assemble a report.
[01:37:15] And it takes him about two
hours to do the report himself
[01:37:18] with ChatGPT.
[01:37:19] And it worked, and it
worked brilliantly.
[01:37:22] And that $150,000 a year that
he's saving on consulting is now
[01:37:25] going to underprivileged
kids for hockey equipment,
[01:37:29] for ice skating
equipment, for lessons,
[01:37:31] and all of that kind of thing.
[01:37:32] So it was taken out of the
hands of an expensive consulting
[01:37:34] company and put into
the hands of people.
[01:37:37] Because of this guy,
and he says he's
[01:37:38] the dumbest person alive, but--
[01:37:40] I hope he's not
watching this video.
[01:37:44] And I told him afterwards
that, congratulations, you're
[01:37:47] now a developer.
[01:37:48] And he didn't like that.
[01:37:51] But it's like surprises like
that the superpowers that were
[01:37:55] handed to somebody like him,
that he's not technical in any
[01:37:58] way, but he was able to
effectively build a solution
[01:38:01] that saved his nonprofit
$100,000 or $150,000 a year.
[01:38:05] And things like that
are always surprising me
[01:38:07] in a very pleasant way.
[01:38:12] Yep.
[01:38:12] Sorry.
[01:38:13] I'll get to you next.
[01:38:14] Sorry.
[01:38:14] Yeah.
[01:38:15] For engineers like us, it's
easier to navigate the hype
[01:38:20] because we can understand what
the signal is from a research
[01:38:24] paper.
[01:38:25] But how about people who doesn't
have this knowledge, like,
[01:38:30] from humanities or
something [INAUDIBLE]?
[01:38:36] Yeah, so just to repeat
the question for the video.
[01:38:38] For engineers like
us, sometimes it's
[01:38:39] easy to navigate the hype to
see the signal from the noise.
[01:38:42] But what about people who don't
have the same training as us?
[01:38:45] I think that's our opportunity
to be trusted advisors for them
[01:38:49] and to really help them
through that, to understand it.
[01:38:53] I think the biggest
part in the hype story
[01:38:55] right now is just understanding
the reward mechanism.
[01:38:59] That everything rewards
engagement rather than
[01:39:01] actual substance.
[01:39:03] And to me, step one is
seeing through that.
[01:39:05] The story I just
told about my friend,
[01:39:08] he'd seen all this
kind of stuff,
[01:39:10] but he wasn't willing
to bet his career on it.
[01:39:12] But he needed that
kind of advice
[01:39:14] around it and to start
peeling apart what he had done
[01:39:16] and what he did right
and what he did wrong.
[01:39:18] And so that positioning
ourselves to be trusted advisors
[01:39:23] by not leaning into
the same mistakes
[01:39:24] that the untrained people
may be leaning into,
[01:39:27] I think is the key to that.
[01:39:29] And just understanding that
the average person is generally
[01:39:32] very intelligent,
even if they may not
[01:39:35] be experts in a specific
domain, and to key
[01:39:37] in on that intelligence and help
them to foster and to grow that
[01:39:41] in and navigate them
through the parts
[01:39:44] where they'll have
difficulty and let them
[01:39:46] shine in what they're
very, very good at.
[01:39:49] Over here there was one.
[01:39:51] I have a question more
for AI and machine
[01:39:53] learning for
scientific research.
[01:39:55] OK.
[01:39:56] Which is something that
is very hard [INAUDIBLE]
[01:39:59] to get your perspective on.
[01:40:01] Where do you think
that is a good idea
[01:40:03] and where you might
say, maybe be cautious?
[01:40:06] So AI and machine learning
for scientific research,
[01:40:09] where is it a good idea and
where should you be cautious?
[01:40:14] Ooh.
[01:40:16] My initial gut check would be I
think it's always a good idea.
[01:40:20] I think there was no harm in
using the tools that you have
[01:40:23] available to you, but
to always to just double
[01:40:26] check your results and double
check your expectations
[01:40:29] against the grounded reality.
[01:40:31] I've always been a fan of
using automation in research
[01:40:36] as much as possible.
[01:40:37] My undergraduate was physics
many, many years ago,
[01:40:40] and I was actually very
successful in the lab
[01:40:42] because I usually automated
things through a computer
[01:40:44] that other people did
handwriting and pen and paper
[01:40:47] with.
[01:40:48] So I could move quickly.
[01:40:49] So I know I'm biased
in that regard.
[01:40:51] But I would say, for most
research, for the most part,
[01:40:54] I think use the most powerful
tools you have available,
[01:40:57] but check your expectations.
[01:41:03] Little story actually on that
side was trivia question.
[01:41:07] Poorest country
in Western Europe.
[01:41:10] Anybody know?
[01:41:11] Serbia?
[01:41:12] What's that?
[01:41:12] Or Western.
[01:41:13] Western Europe is Wales.
[01:41:16] So I actually did my
undergraduate in Wales,
[01:41:19] and I went back to do some
lectures in the university
[01:41:22] there.
[01:41:23] And I met with a
researcher there,
[01:41:26] and he was doing research
into brain cancer
[01:41:29] using computer imagery and
using various types of computer
[01:41:32] imagery.
[01:41:32] And I asked him, well,
what's the biggest
[01:41:34] problem that you have?
[01:41:35] What's the biggest
blocker for your research?
[01:41:38] And this is about
eight years ago.
[01:41:39] And his answer was
access to a GPU.
[01:41:43] And because for him to be
able to train his models
[01:41:46] and run his models, he needed
to be able to access a GPU.
[01:41:50] And the department
that he was in
[01:41:52] had one GPU between
10 researchers,
[01:41:55] which meant that everybody
got it for half a day.
[01:41:57] Monday through Friday,
and his half a day
[01:41:59] was Tuesday afternoon.
[01:42:00] So in his case, he would
spend the entire time
[01:42:02] that wasn't Tuesday afternoon
preparing everything
[01:42:05] for his model run or
his model training
[01:42:07] or everything like that.
[01:42:08] And then Tuesday afternoon,
once he had access to the GPU,
[01:42:11] then he would do the training.
[01:42:12] And then he would
hope in that time
[01:42:14] that he would train his model
and he would get the results
[01:42:16] that he wanted.
[01:42:17] Otherwise, he'd have to wait a
week to get access to the GPU
[01:42:20] again.
[01:42:21] And then I showed
him Google Colab.
[01:42:23] Anybody ever used Google Colab?
[01:42:25] And you can have
a GPU in the cloud
[01:42:27] for free with that
kind of thing.
[01:42:29] And the poor guy's
brain melted that--
[01:42:32] because I took out my
phone, and I showed him
[01:42:34] a notebook running on
my phone in Google Colab
[01:42:37] and training it on that.
[01:42:38] And it changed everything
for him research wise.
[01:42:41] And now it was a case of--
and this was with Colab.
[01:42:44] He had much more than he
had with his shared GPU.
[01:42:46] So I think for someone
like him, machine learning
[01:42:49] was an important
part of his research,
[01:42:51] but he was so gated on it that
the ability to widen access
[01:42:55] to that ended up really,
really advancing his research.
[01:42:57] I don't know where it ended up.
[01:42:59] I don't know what he has done.
[01:43:00] It has been a few
years since then.
[01:43:01] But that story just came to mind
when you asked the question.
[01:43:06] Any more questions?
[01:43:09] Feel free to ask me anything.
[01:43:14] Oh, yeah.
[01:43:14] At the front here.
[01:43:15] It's more of a general question.
[01:43:17] You talked about AI helping
food and beverage use.
[01:43:21] What do you think AI would
be a force of social equality
[01:43:25] or social inequality?
[01:43:27] So can AI be a force of social
equality or social inequality?
[01:43:31] I think the answer
to that is yes.
[01:43:34] It can be both, and
it can be neither.
[01:43:37] I mean, I think that
ultimately, the idea
[01:43:39] is that if in my opinion, any
tool can be used for any means,
[01:43:45] so the important thing is to
educate and inspire people
[01:43:48] towards using things
for the correct means.
[01:43:51] There's only so much
governance can be applied.
[01:43:53] And sometimes governance
can cause more problems
[01:43:56] than it solves.
[01:43:58] So I always love to live my
life by assuming good intent
[01:44:03] but preparing for bad intent.
[01:44:05] And in the case of
AI, I don't think
[01:44:07] there's any difference there
that everything that I will do
[01:44:09] and everything that I would
advise is assuming good intent,
[01:44:12] that people would use
it for good things,
[01:44:14] but also to be prepared
for it to be misused.
[01:44:18] The bad examples that I
showed earlier on, I think
[01:44:20] were good intent
rather than bad intent.
[01:44:24] And most mistakes
that I see that are
[01:44:26] good intent being
used mistakenly as
[01:44:29] opposed to bad intent.
[01:44:30] But I would say that's the
only mantra that I can--
[01:44:33] the only advice that I can give
and that kind of thing is always
[01:44:35] assume good intent, but
prepare for bad intent.
[01:44:40] The AI itself has no choice.
[01:44:42] It's how people use it.
[01:44:46] Andrew, did you want
closing comments or--
[01:44:49] I think we were running
out [INAUDIBLE] time.
[01:44:53] But thank you for this.
[01:44:55] Really great.
[01:44:56] Thanks, everyone,
for all the questions
[01:44:57] on those creative solutions.
[01:45:00] All right.
[01:45:00] Thank you, Andrew.
[01:45:01] Thanks.
[01:45:01] [APPLAUSE]