What I want to do today is chat with you
about career advice in AI.
And in previous years, I used to do most of this lecture
by myself.
But what I thought I'd do today is
I'll share just a few thoughts and then hand it over
to my good friend Laurence Moroney, who I invited to speak
here and kindly agreed to come all the way to San Francisco,
he lives in Seattle, to share with us a very broad market
landscape for what he's seeing in the job market,
as well as tips for career, growing a career in AI.
But there was just two slides and then one more thought
I want to share with you before I hand it over
to Laurence, which is it really feels like the best
opportunity, the best time ever to be building with AI
and to building a career in AI.
A few months ago I noticed in social media, traditional media,
there are a few questions about is AI slowing down?
People saying, well, it's GPT-5 that good?
I think it's actually pretty good.
But there are questions about is AI progress slowing down?
And I think part of the reason the question was even raised was
because if a benchmark for AI is 100% is perfect answers,
then if you make rapid progress, at some point,
you cannot get above 100% accuracy.
But one of the studies that most influenced my thinking was work
done by this organization, M-E-T-R,
METR that studied as time passes,
how complex are the tasks that AI could do as measured by how
long it takes a human to do that task?
So a few years ago, maybe GPT-2 could do tasks
that a human could do in a couple seconds.
And then they could do tasks that
took a human four seconds, then eight seconds, then,
a minute, two minutes, four minutes, and so on.
And the study estimates that the length of task AI can do
is doubling every seven months.
And I think on this metric, I feel
optimistic that AI will continue making progress,
meaning the complexity of tasks as measured
by how long a human takes to do something is doubling rapidly.
And same study with a smaller data set
seems to show-- same study argued that for AI coding,
the doubling time is even shorter, maybe 70 days.
So this code that used to take me,
I don't know, 10 minutes to write, then 20 minutes
to write, 40 minutes to write, and AI
could do more and more of that.
And so the reasons I think this is a golden age
to be building, best time we've ever seen
is maybe two themes which are more powerful and faster.
So we can all, all of you in this room
can now write software that is more powerful than what
anyone on the planet could have built a year
ago by using AI building blocks.
AI building blocks include large language models,
radical genetic workflows, voice AI,
and of course, deep learning.
It turns out that a lot of LLMs have a decent, at least basic
understanding of deep learning.
So if you have a prompt one of the frontier models
to implement a cutting edge neural network for you,
try prompting it to implement a transformer network for you.
It's actually not bad at helping you use these building blocks
to build software quickly.
And so we have very powerful building blocks
that were very difficult or did not exist a year or two ago.
And so you can now build software that does things
that no one else on the planet, even the most advanced teams
on the planet, could have done.
And then also with AI coding, the speed
with which you can get software written
is much faster than ever before.
And I've personally found it as important
to stay on the frontier of tools,
because the tools for AI coding changes,
I don't know, really rapidly.
So I feel like since several months ago, my personal number
one favorite tool became Cloud Code, moving on
from some earlier generations, I think.
And then I think since the release of GPT-5,
I think OpenAI Codex has actually
made tremendous progress.
And this morning, Gemini 3 was released,
which haven't had time to play with it yet just this morning.
It seems like another huge leap forward.
So I feel if you ask me every three months
what my personal favorite coding tool is, it actually
probably changes definitely every six months, but quite
possibly every three months.
And I find that being half a generation behind in these tools
means being, frankly, quite a bit less productive.
And I know everyone says AI is moving so fast,
everything's changing so fast.
But AI coding tools, of all the sectors in AI,
many things maybe don't move as fast as the hype says it does,
but AI coding tools is one sector
where I see the pace of progress is tremendous.
And staying at the latest generation of tools,
rather than half generation behind makes
you more productive.
And with our ability to build more powerful software
and build it much faster than ever
before, I think one piece of advice
that I give now, much more strongly
now than even a year ago or two years ago,
is just go and build stuff.
Take classes from Stanford.
Take online courses.
And additionally your opportunity
to build things, and I think Laurence
is going to talk about showing them to others,
is greater than ever before.
But there's one weird implication
of this that is maybe not-- is still,
I don't know, more and more people are appreciating it,
but not widely known, which is the product management
bottleneck, which is that when it is increasingly easy to go
from a clearly written software spec to a piece of code,
then the bottleneck increasingly is deciding what to build
or increasingly writing that queer spec for what you actually
want to build.
When I'm building software, I often
think of going through a loop where we'll write some software,
write some code sure to use this to get user feedback.
I think of this as a PM or product management work.
And then based on user feedback, I'll
revise my view on what users like, what they don't like.
This UI is too difficult. They want this feature.
They don't want that feature and change my conception
of what to build, and then go around this loop
many times to hopefully iterate toward a product
that users love.
And because of AI coding, the process of building software
has become much cheaper and much faster than before.
But that ironically shifts the bottleneck
to deciding what to build.
So some weird trends I'm seeing.
In Silicon Valley and in many tech companies,
people have often talked about an engineer to product manager,
engineer to PM ratio.
And you take these ratios with grain of salt,
because they're kind of vary all over the place.
But you hear companies talk about the Eng to PM
ratio of 4 to 1 or 7 to 1 or 8 to 1.
This idea that one product manager writing product specs
can keep four to eight or some number like that engineer
is busy.
But because engineering is speeding up,
whereas product management is not sped up as far as much by AI
as engineering, I'm seeing the Eng
to PM ratio trending downward, maybe even two or one to one.
So some teams I work with, they proposed
headcount was one PM to one engineer, which
is a ratio unlike almost all Silicon Valley, certainly
traditional Silicon Valley companies.
And the other thing I'm seeing is that engineers,
they can also shape products that
can move really fast where you go one step further,
take the engineer, take the PM, and collapse them
into a single human.
And I find that there are definitely
engineers that doing engineering work that
don't enjoy talking to users and having that more
human, empathetic side of work.
But I'm finding increasingly that the subset of engineers
that learn to talk to users, get feedback, develop
deep empathy for users so that they can make decisions
about what to build, those engineers
are also the fastest moving people that I'm
seeing in Silicon Valley today.
And I feel like at the earliest stage of my career,
one thing I regretted for years was in one of the roles I had,
I went to try to convince a bunch of engineers
to do more product work.
And I actually made a bunch of really good engineers
feel bad for not being good product managers.
And that was a mistake I made, regretted that for years.
I just shouldn't have done that.
And part of me feels like I'm now
going back to repeat that exact same mistake.
Having said that, I find that the fact
that I can write code, but also talk
to users to shape what to do, that lets me
and the engineers that can do this go much faster.
So I think maybe worth taking another look
at whether engineers can do a bit more of this work,
because then if you're not waiting for someone else
to take the product to customers,
you can just write code, have a gut for what to do next,
and iterate that pace, that velocity of execution
is much faster.
And then before I hand over to Laurence, just one last thing
I want to share, which is in terms of navigating your career,
I think one of the most strong predictors
for your speed of learning and for your level of success
is the people you surround yourself with.
I think we're all social creatures.
We all learn from people around us.
And it turns out there are studies in sociology that
show that if your five closest friends are smokers,
the odds of you being a smoker is pretty much high.
Please don't smoke.
It's just an example.
I don't know of any study showing
that if you're five or 10 closest friends are really
hard working, determined people, learning quickly, trying
to make the world a better place of AI,
that you are more likely to do that too.
But it's one of those things that I
think is almost certainly true.
It's like all of us are inspired by the people around us,
and we're able to find a good group of people to work with,
that helps drive you forward.
In fact, here at Stanford, I feel very fortunate--
the fantastic student body, fantastic group of faculty.
And then the other thing that I think
we're fortunate to have at Stanford
is our connective tissue.
So candidly, a lot of the people working
and a lot of the cutting edge AI labs, the frontier labs,
they were former students of a lot of different Stanford
faculty.
And so that rich connective tissue candidly
means that at Stanford, we often find out
about a lot of stuff that's not widely
known because of the relationships, the friendships.
And when some company does something,
one of my friends and the faculty
will call up someone to come and say, hey, that's weird.
Does this really work?
And so that rich connective tissue means that we're all--
just as we try to pull our friends forward,
our friends also pull us forward with the knowledge
and the connective tissue and this know-how of bleeding edge
AI, which unfortunately is not all
published on the internet at this moment in time.
So I think while you're at Stanford, make those friends,
form that rich connective tissue.
And there have been a lot of times that just for myself,
where, frankly, I was thinking of going
in some technical direction.
I'd have one or two phone calls with someone
really close to research, either Stanford researcher or someone
in the frontier lab.
They would share something with me that I didn't know before.
And that changes the way I choose
the technical architecture of a project.
So I find that group of friends you surround yourself
with, those little pieces of information-- try this.
Don't do that-- that's just hype.
Ignore the PR.
Don't actually try that thing.
Those things make a big difference
in your ability to steer the direction of your projects.
So while you're at Stanford, take advantage of that.
This connective tissue that Stanford has,
it's actually really unique.
There are lots of great universities in the world,
but at this moment in time, I don't think there's any--
I don't want to sound like I'm doing PR for Stanford now,
but I really think there's no university in the world that
is as privileged as Stanford at this moment in time,
in terms of the richness of the connective tissue to all
of the leading AI groups.
But to me, there's also that we're lucky here
to have a wonderful community of people
to work with and learn from.
And for you too.
If you apply for jobs, the thing that is much more
important for your career success would
be if you go to a company, it'll be the people
you work with day to day.
So here's one story that I've told in previous classes, I
repeat, which is there's a Stanford student that I knew
this was many years ago, that I knew,
and they did really good work at Stanford.
I thought they were high flyer.
And they applied for a job at a company,
and they got a job offer from one of the companies
with a hot AI brand.
This company refused to tell him which team he would join.
They said, oh, come sign up for a job.
There's a rotation system, matching system, blah blah blah.
Sign on the dotted line first.
Then we'll figure out what's a good project for you.
Partly because it was a good company.
His parents were proud of him for getting a job
at this company.
This student joined this company hoping
to work on exciting AI project.
And after he signed on the dotted line,
he was assigned to work on the back end
Java payment processing system of the company.
Nothing against anyone that wants to do Java
back end payment processing systems.
I think they're great, but this is an AI student that did not
get matched to an AI project.
And so for about a year, he was really frustrated,
and he actually left this company after about a year.
The unfortunate thing is, I told this story
in CS230 some years back.
And then after I was already telling
the story in this class, a couple of years later,
another student in CS230 went through the same experience
with the same company, not Java back end payment processing,
but different project.
And I think this effect of trying to figure out
who you'll be actually working with day to day
and making sure you're surrounded by people that
inspire you and work on exciting projects,
I think that's important.
And even completely candid, if a company
refuses to tell you what team you'll be assigned to,
that does raise a question in my mind of
whether or not what will happen.
And I think that instead of working
for the company with the hottest brand,
sometimes if you find a really good team with really
hard working, knowledgeable, smart people trying to do good
with AI, but the company logo just isn't as hot,
I think that often means you actually
learn faster and progress your career better because it
is after all, we don't learn from the excitement
of the company logo when you walk through the door,
you learn from the people you deal with day to day.
So I just urge you to use that as a huge criteria
for your selection process for what you decide to do.
But I think number one on my advice
is it's become much easier than ever
before to build powerful software faster.
And what that means is do be responsible.
Don't build software that hurts others.
And at the same time, there are so many things that each of you
can build.
And what I find is the number of ideas out in the world
is much greater than the number of people with the skill
to build them.
So I know that finding jobs has gotten tougher for fresh college
grads.
At the same time, a lot of teams just
can't find enough skilled people.
And so there are a lot of projects
in the world that if you don't build it,
I think no one else will build it either.
So you don't need to-- so long as you don't harm others,
be responsible, there are a lot of things that you don't
need to wait for permission.
You don't need to wait for someone else to do it first
and then you do it.
The cost of a failure is much lower than before
because you waste a weekend but learn something.
That seems fine to me.
So I think so let's be responsible,
going for trying things out and building lots of things
would be the number one most important thing I
think would help your careers.
And yeah, I think I'm going to say one last thing that
is considered not politically correct in some circles,
but I'll just say it anyway, which is in some circles,
it has become considered not politically correct
to encourage others to work hard.
I'm going to encourage you to work hard.
Now, I think the reason some people don't
like that is because there are some people that
are in a phase of life where they're not
in a position to work hard.
So right after my children were born,
I was not working hard for a short period of time.
And there are people because of an injury or disability,
whatever very valid reasons, they're
not in a position to work hard at that moment in time.
And we should respect them, support them,
make sure they're well taken care of even
though they're not working hard.
Having said that, all of my, say,
PhD students have become very successful,
I saw every single one of them work incredibly hard.
I mean, the 2:00 AM sitting up, hyperparameter tuning,
been there, done that.
Still doing it some days.
And if you are fortunate enough to be in a position in life
where you can work really hard, there
are so many opportunities to do things right now.
If you get excited, as I do, spending evenings and weekends
coding and building stuff and getting user feedback,
if you lean in and do those things,
it will increase your odds of being really successful.
So I don't know.
Maybe I get into some trouble with some people
encouraging me to work hard, but I
find that the truth is people that work hard
get a lot more done.
We should also respect people that don't and people that
aren't in a position to do so.
But between watching some dumb TV
show versus firing your agentic coder on a weekend
to try something, I'm going to choose
the latter almost every time.
Unless I'm watching a show with my kids, sometimes I do that.
But you mean--
I hope you do that.
All right, so those are the main things I wanted to say.
What I want to do is hand the stage over to my good friend
Laurence Moroney, who share a lot more about career
advice on AI.
Let me just quick intro.
I've known Laurence for a long time.
He's done a lot of online education work, sometimes
with me and my teams, taught a lot of people
Tensorflow, taught a lot of people PyTorch.
He was lead AI advocate at Google for many years, now
runs a group at Arm.
I've also enjoyed quite a few of his books.
This is one of them.
He recently also published a new book on PyTorch.
This is an excellent book, Introduction to PyTorch.
And he's a very sought after speakers in many circles,
so I was very grateful when he agreed to come speak to us.
Pleasure is all mine.
I just want to reinforce something
that Andrew was talking about earlier
on about choosing the people that you
work with being very important.
But I also want to show that from the other way around that
the company, when they're interviewing you
are also choosing you.
And the good companies really want
to choose the people that they work with also.
And I've been doing a lot of mentoring of young people
over the last, particularly over the last 18
months, who are hunting for careers for themselves.
And I want to tell the story of one young man and this guy,
very well educated, great experience, super elite coder.
He could do every challenge that was in front of him,
and he got laid off from his job in April.
He worked in medical software, and medical software business
has been changing drastically.
Funding has been cut by the Federal government
in a number of areas, and he got laid off from his job.
And with his experience, with his ability,
with his skills, all of these kind of things,
he thought that it would be very easy
for him to find another job.
And the poor young guy had a really terrible April.
He got laid off from his job in April.
Immediately before that, his girlfriend
had broken up with him, and then a couple of weeks
later, his dog died.
So he was not in a good place.
And so I sat down with him after a couple of months
and took a look.
And he had a spreadsheet of jobs that he was applying to,
and he had over 300 jobs that he was tracking in the spreadsheet.
And in a number of these jobs, he actually
got into the interview process, and he
went very deep in the interview process
with companies like Meta.
Who else?
Not Google.
It was Meta.
There was Microsoft.
There was one of the other large tech
companies where you do lots and lots of interview loops.
And every time towards the end of the loop,
he knew he did a great loop.
He solved all the coding.
He had great conversations with the people,
or at least he thought he had.
And then every time within a day,
the recruiter would call him and say, no, you didn't get the job.
And it was like it was heartbreaking.
And like I said, 300 plus jobs he had been tracking.
So I started working with him to do some mock interviews
and to do some fine tuning.
Oh, it was Jeff Bezos company, not Amazon.
That was one of the other big tech company
that he'd interviewed with.
And I started working through him
and doing some test interviews and all this kind of thing
with him.
Terrific, terrific candidate couldn't figure out
what was going wrong until I decided to try and do
a different sort of interview where I gave him
a really tough interview.
I gave him some tough LeetCode.
I gave him some really obscure corner cases in his coding.
And I saw how he reacted.
And how he reacted was the advice
that was given to him in the recruiting pamphlets.
And a lot of these recruiting pamphlets
will say things like, you're going to have an opportunity
to share an opinion, and you've got to stand your ground.
You've got to have a backbone.
Don't bend.
His interpretation of that was to be really, really tough.
So I would pick corners.
I would pick holes in his code.
I'd pick corner cases where things may not work,
and I would give him a test of crisis.
And this advice that he'd been given to stand his ground
ended up making him hostile in these interview environments.
And I was looking at this then from the point
of view of what Andrew was just talking about,
where it's a case of hey, good people, good teams, people
that you can work together with.
And from the interviewer perspective,
if I'm managing this team, this person is that cliched 10x
engineer, but I don't want him anywhere near my team
because of this attitude.
We worked on that.
We fine-tuned it.
And the strange part is he's a really, really nice guy.
It's just this was the advice he was given,
and he followed that advice, and he failed so many interviews
as a result.
So when I gave him the next job that he was interviewing at
was at a company where teamwork is very, very highly valued.
And the good news is he got the job at that company.
He's now working there.
He doubled his salary from the job he was laid off from,
and he ended up having about-- now he looks back
and he had six months of fun employment.
But at the time when he was going through all of that,
it was a very, very difficult time for him.
So the flip side of it, if you're looking at a company
and looking at the paper you'd be working with
is very, very important.
But also realize they are looking at you in the same way.
And so if you've gone to a tech interview coaching,
and they gave you that advice to stand your ground
and have a backbone, it's good to do that.
But don't be a jerk while you're doing so.
Can you see my slides?
OK.
So I'm Laurence.
I've been working in tech for more decades
than ChatGPT thinks there are oars in strawberry.
So I've worked in many of the big tech companies.
I spent many years at Microsoft, spent many years at Google,
also worked in places like Reuters.
I've done a lot of work in startups, both in this country
and abroad.
And so what I really want to talk about today
is like to think about what does the career landscape look
like today, particularly in AI.
Because first of all, what Andrew said about in Stanford,
you've got the ability to make use of the networks
that you have in Stanford, make use of the prestige
that you have, and I say use every weapon you have.
Because unfortunately, the landscape right
now is not ideal.
We've gone through some very difficult times.
All you have to do is look at the news,
and you can see massive tech layoffs, slowing hiring in tech,
and lots of stuff like that.
But it's not necessarily a bad thing
if you do it the right way.
So I want to just have a quick look the job market
reality check.
Actually out of interest, I don't know.
This is a-- are you juniors?
You're graduating this year or you're graduating next year
or what is the general survey?
You're third year of four?
[INAUDIBLE]
Third year of three, I would say.
So you're going to be graduating coming summer.
How many people are already looking for jobs?
OK, quite a few of you.
How many people have had success?
Nobody.
Oh, one.
OK.
That's good.
So you're probably seeing some of these things, the signals
out there, junior hiring slowing significantly.
When I say junior, I mean graduate level.
High-profile layoffs are dominating the headlines.
I was at Google a couple of years ago
when they had the biggest layoff they'd ever had.
We're seeing layoffs at the likes of Amazon, Microsoft,
other companies like that.
It feels that entry-level positions are scarce,
and I'm underlining the word "feels" there,
and I want to get into that in a little bit more detail later.
And also, competition is fierce.
But my question is, should you worry?
And I say, no.
Because if you can approach things in the right way,
if you can approach the job hunting thing in the right way,
particularly understanding how rapidly the AI landscape is
changing, then I think people with the right mindset
will thrive.
So what do I mean by that?
So as Andrew had mentioned, the AI hiring landscape
is changing because the AI industry is changing.
The AI industry I--
I actually first got involved in AI back way back in 1992.
I worked in it for a little while just before the AI winter.
Everything failed drastically, but I got bitten by the AI bug.
And then in 2015, when Google were launching TensorFlow,
I got pulled right back into it, became part of the whole AI
boom, launching TensorFlow, advocating TensorFlow
to millions of people, and seeing
the changes that happened.
But along 2021, 2022, we had a global pandemic.
The global pandemic caused a massive industrial slowdown.
This massive industrial slowdown meant
that companies had to start pivoting
towards things that drove revenue and directly drove
revenue.
And at Google, TensorFlow was an open-source product.
It didn't directly drive revenue.
We began to scale back.
Every company in the world also scaled back
on hiring at this time.
Then we get to about 2022, 2023.
What happens?
We begin to come out of the global pandemic.
We begin to realize all industries have
this massive logjam of non-hiring that they had done
or hiring that they hadn't done.
And we're also entering a time where
AI was exploding on the scene.
Thanks to the work of people like Andrew,
the world was pivoting and changing to be AI first
in just about everything.
And every company needed to hire like crazy.
Every company then hiring like crazy in 2022, 2023
meant that most companies ended up overhiring.
And what that generally meant was
people who were not qualified for higher positions usually got
higher positions because you had to enter into a bidding war
just to be able to get talent.
You ended up having talent grabs,
and you ended up having stories like the one Andrew told where
it's a case here's a person with AI talent, let's grab them,
let's throw money at them, let's have them come work for us,
and then we'll figure out what we want to do.
So as a result, 2022, 2023 all of this massive overhiring
happens because of AI and because of the COVID logjam.
And then 2024, 2025 is the great wake-up, where
a lot of companies realize this over hiring that they had done,
they have ended up with a lot of people who are underqualified.
I'm sorry.
Yeah, underqualified for the job that they were doing.
A lot of people ended up getting hired just because they
had AI on their resume.
And there's a big adjustment going on.
And in the light of this big adjustment--
show you-- just one second.
In the light of this big adjustment-- oh,
you're not saying my slides?
OK.
And in the light of this big adjustment-- there we go.
I think it's because my power.
I'm not plugged into power mains.
And in the light of this big adjustment,
then what has happened is now a lot
of companies are much more cautious about AI skills
that they're hiring.
And if you're coming into that with that mindset
and understanding that, realize opportunity is still there,
and opportunity is there massively
if you approach it strategically.
So what I want to talk through today
is how you can do exactly that.
So I see three pillars of success in the business world
and particularly in the AI business world.
And nowadays you can't just have AI on your resume
and get overhired.
Nowadays, not only do you have to be
able to tell that you have the mindset of these three
pillars of success, but you also have to be able to show.
And to be able to show these, that actually has never
been a better time.
As Andrew demonstrated earlier on, the ability to vibe
code things into existence.
He doesn't like the word vibe code.
I agree with him, but the ability
to prompt things into existence, or whatever the word is
that we want to use, allows you to be
able to show better than ever before.
He was talking earlier on about product managers,
and he had this time when he got engineers
to be product managers, and then those engineers
ended up being really bad product managers.
I actually interviewed at Google twice and failed twice
despite being very successful at Microsoft,
authored 20 plus books, taught college courses.
I interviewed at Google twice and failed twice
because I was interviewing to be a product manager,
and then when I interviewed to be an engineer, they hired me
and they were like, why didn't you try to join us years ago?
So a lot of it is just being a good engineer.
You've got the ability to do that and show that nowadays.
And with that ratio of engineer to product manager,
changing engineering skills are also
far more valuable than ever.
So the three pillars to success.
Number 1, understanding in depth.
And I'm going to mean this in two different ways.
Number one is academically, to have the understanding and depth
academically of machine learning,
of particular model architectures,
to be able to understand them, to be able to read papers,
to be able to understand what's in those papers,
and to be able to understand, in particular, how to take
that stuff and put it to work.
The second part of understanding in depth
is really having your finger on the pulse of particular trends
and where the signal-to-noise ratio favors
signal in those trends.
And I'm going to be going into that in a lot
more detail a little bit later.
Secondly, and also very, very importantly is business focus.
So Andrew said something politically incorrect
earlier on.
I'm going to also say a similar politically incorrect thing.
First of all, hard work.
Hard work is such a nebulous term
that I would say that think about hard work in terms of you
are what you measure.
There is the whole trend out there.
I'm trying to remember, is it 996 or is it 669?
996.
9:00 AM to 9:00 PM, six days a week is a metric of hard work.
It's not.
There's not a metric of hard work.
That's a metric of time spent.
So I would encourage everybody, in the same way as Andrew
did, to think about hard work.
But what hard work is how you measure that hard work.
You can work eight hours a day and be incredibly productive.
You can work six hours a day and be incredibly productive,
but it's the metric of how hard you work
and how you measure that.
I personally measure that from output,
things that I have created in the time that I spent.
I joke a lot, but it's true that I've written a lot of books.
Andrew held up one.
That one that he held up, that he helped me write a little bit,
I actually wrote that book in about two months.
And people say, well, how do you have time with your jobs
and all these kind of things?
You must work like 16 hours a day
in order to be able to do this.
But actually, the key to me being able to write books
is baseball.
Any baseball fans here?
So I love baseball, but if you sit down and try to watch
baseball on TV, a match can take like 3 and 1/2 or four hours.
So all of my writing I tend to do in baseball season.
So I'm like, if I'm going to sit down, I like the Mariners.
I'm from Seattle.
I like the Dodgers.
Nobody booed.
OK, good.
And so usually one of those is going to be playing at 7 o'clock
at night.
So instead of sitting in front of the TV,
just like watching baseball mindlessly.
I'll actually be writing a book while baseball
is on in the background.
It's a very slow moving game.
This is something.
That's the hard work in this case.
And I would encourage you to try to find areas where you can
work hard and produce output.
And that's the second pillar here is that business
focus, the output that you produce
to align that output with the business focus that you
want to have and with the work that you want to do.
There's an old saying, "Don't dress for the job you have,
dress for the one you want."
I would say a new angle on that saying would
be, don't let your output be for the job you have.
Let your output be for the job you want.
And if I go back to when I spoke about I failed twice
at Google to get in, the third time when I got in,
I had actually decided to do to approach
this in a different way.
And I was interviewing at the time for their cloud team.
They were just really launching cloud,
and I had just written a book on Java.
And so I decided to see what I could
do with Java in their cloud.
I ended up writing a Java application that
ran in their cloud for predicting stock prices using
technical analytics and all that kind of stuff.
And when it got to the interview,
instead of them asking me stupid questions like how many golf
balls can fit in a bus, they saw this code.
I had put this code.
I remember I was producing output for the job I wanted.
I'd put this code on my resume, and my entire interview loop
was them asking me about my code.
So it put the power on me.
It gave me the power to communicate about things
that I knew, as opposed to going in blind to somebody
asking me random questions in the hope
that I'll be able to answer them.
And it's the same thing I would say in the AI world.
The business focus, the ability for you now to prompt code
into existence, to prompt products into existence
and if you can build those products
and line them up with the thing that it is that you want to do,
be it a Google or Meta or a startup
or any of those kind of things, and have
that in-depth understanding not just of your code,
but how it aligns to their business,
this is a pillar of success in this time and age.
And I will also argue that even though it
looks like the signals look like there aren't a lot of jobs
out there, there are.
What there aren't a lot of is a good combination
of jobs and people to match them.
And then, of course, this bias towards delivery.
"Ideas are cheap, execution is everything."
I've interviewed many, many people
who came in with very, very fluffy ideas and no way
to be able to ground them.
I've interviewed people who came in with half-baked ideas
that they grounded very, very well.
Guess which ones got the job?
So I would say these three things.
Understanding and depth of the academics behind AI
of the practicalities behind AI and the things that you need
to do.
Business focus, focusing on delivery for the business,
understanding what the business needs
and being able to deliver for that, and again,
that bias towards delivery.
So a quick pivot.
What's it actually like working in AI right now?
It's interesting.
So as recently as two or three years ago, working in AI
was if you could do a thing, you're great.
If you can build an image classifier, you're golden.
We'll throw six figure salaries and massive stock benefits
at you.
Unfortunately, that's not the case anymore.
It's really a lot of today what you'll see
is the P word, production.
What can you do for production?
What can you do if it's building new models,
if it's optimizing models, if it's understanding users,
UX is really, really important.
Everything is geared towards production.
Everything is biased towards production.
The history that I told you about,
going from the pandemic into the overhiring phase that we'd had,
the businesses have pulled back and are optimized
towards the bottom line.
I have an old saying that the bottom line is
that the bottom line is the bottom line,
and this is the environment that we're in today.
And if you can come in with that mindset
when you're talking with companies,
that's one of the keys to open the door.
One of the things I've seen in the field
has been maturing from it used to be really nice that we
could do cool things and we could build cool things.
Now it's really build useful things.
Those useful things can be cool too, by the way,
and the results of them can be cool.
And the changes that we see that come
about as a result of delivering them can be cool.
So it's not just coolness for coolness sake,
but to focus on delivery, focus on being able to provide value,
and then the coolness will follow.
I guess what I'm trying to argue.
So for realities, number 1, unfortunately nowadays
business focus is non-negotiable.
Now, let me-- I'm going to be a little bit politically incorrect
here again for a moment.
I've been working, like I said, for most of the last 35 years
in tech.
I would say for most of the last 10 years,
a lot of large companies, particularly in Silicon Valley
have really focused on developing their people
above everything.
Part of developing their people was bringing their entire self
to work.
Part of bringing their entire self to work
was bringing the things that they care about outside of work.
And that led to a lot of activism within companies.
Now, please let me underline this.
There is nothing wrong with activism.
There is nothing wrong with wanting to support causes,
not wanting to support causes where of justice.
There is absolutely nothing wrong with that.
But the overindexing on that, in my experience,
has led to a lot of companies getting
trapped by having to support activism above business.
You've probably seen an example about two years ago
of where activists in Google broke into the Google Cloud
heads office because they were protesting a country that Google
Cloud were doing business with.
They broke into his office, they had a sit-in his office,
and they used the bathroom all over his desk and stuff
like that.
This is where activism got out of hand.
And as a result, the unfortunate truth
is the good signals in that activism are now being lost.
Because of those actions, people are being laid off.
People are losing jobs.
Activism is being stifled, and business focus
has become non-negotiable.
There's a bit of a pendulum swing going on.
And the pendulum that had swung too far towards allowing people
to bring their full selves to work
is now swinging back in the other direction.
We might blame the person in the White House
and all that for these kind of things,
but it's not solely that.
It is that ongoing pendulum there.
And I think it's an important part of it,
is that you have to realize going into companies now,
that business focus is absolutely non-negotiable.
Secondly, risk mitigation is part of the job.
And I think a very important part of any job, particularly
with AI.
I think if you can come into AI with a focus and a mindset
around understanding the risks of transforming
a particular business process to be an AI-oriented one
and to help mitigate those risks,
I think is really, really powerful.
And I would argue in an interview environment, that's
the number one skill to have, to have that mindset around you
are doing a business transformation from heuristic
computing to intelligent computing.
Here's the risks.
Here's how you mitigate those risks,
and here's the mindset behind that.
The third part responsibility is evolving.
Now responsibility in AI has again
changed from a very fluffy definition of let's make sure
that the AI works for everybody to a definition of let's make
sure that the AI works.
Let's make sure that it drives the business.
And then let's make sure that it works for everybody.
Often that has been inverted over the last few years,
and that has led to some famous documented disasters.
Let me share one with you.
Let's see.
I have lots of windows open.
OK.
Everybody knows image generation,
text to image generation.
I want to share a--
these were things that happened a couple of years
ago with Gemini.
So with Gemini, I was doing some testing around this one
and I was working heavily on responsible AI.
And part of responsible AI is you want
to be representative of people.
And when you're building something,
like if you're a Google, you're indexing information,
you really want to make sure that you don't
reinforce negative biases.
And if you're generating images, it's very easy
to reinforce negative biases.
So for example, if I said give me
an image of a doctor, if the training set primarily
has men as doctors, it's more likely to give a man.
If I say give me an image of a nurse, if the training set more
likely to have women as nurses, it's
more likely to give me an image of a woman.
But that's reinforcing a negative stereotype.
So I wanted to do a test of how Google were trying
to overcome that, given that these negative biases are
already in the training set.
So I said, OK, here's a prompt where I said,
"give me a young Asian woman in a cornfield,
wearing a summer dress and a straw hat,
looking intently at her iPhone," and it gave me these beautiful
images.
It did a really nice job.
And I said, this is a virtual actress I've been working with.
I'll share that in a moment.
And I say, OK, what if I ask for an Indian one?
So I said, OK, whoops, a young Indian woman, same prompt.
And it gave me beautiful images of a young Indian woman.
Then I was like, OK, what if I want her to be Black?
For some reason it only gave me three.
I'm not sure why, but it's still adhere to the prompt.
So the responsibility was looking really, really good.
So then I asked it to give me a Latina.
Latina, it gave me four.
But yeah, she looks pretty Latina.
Maybe the one on the bottom left looks a little bit
like Hermione Granger, but on the whole looks pretty good.
Then I asked it to give me a Caucasian.
What do you think happened?
"While I understand your request,
I am unable to generate images of people as this could
potentially lead to harmful stereotypes and biases."
This was a very poorly implemented safety filter,
where the safety filter in this case was like looking
for the word "Caucasian" or looking for the word "whites"
and the results saying it wouldn't do it.
I was like, OK, well, let me test the filter a little bit
and I said, OK, instead of Caucasian, let me try white.
And yet, while I'm unable to fulfill your--
"While I'm able to fulfill your requests,
I'm not currently generating images of people."
It lied to my face because it had just
generate images of people.
Anybody know the hack that I used to get it to work?
This is a funny one.
So I will show you.
One moment.
I asked it to generate an Irish woman.
What do you think it did?
It gave me this image of an Irish woman, no problem,
in a summer dress, straw hat, looking intently at her phone.
What do you notice about this image?
She's got red hair in every image.
I grew up in Ireland, and Ireland
does have the highest proportion of redheads in the world.
It's about 8%.
But if you're going to draw an image
of a person and associate a particular ethnicity
with a color of hair, you can begin
to see this is massively problematic.
There are areas, I believe, in China
where the description of a demon is a red-headed person.
So what ended up happening here, from the responsible AI
perspective, was one very narrow view
of the world of what is responsible
and what is not responsible.
Ended up taking over the model, ended up
damaging the reputation of the model
and damaging the reputation of the company
as a result. In this case, it's borderline
offensive to draw all Irish people as having red hair,
but that never even entered into the mindset of those
that were building the safety filters here.
So when I talk about responsibility is evolving,
that's the direction that I want to--
sorry, one moment.
Let me get my slides back. --that's
the direction I want you to think about,
that now responsible AI has moved out
of very fluffy social issues and into more hard line things that
are associated with the business and prevent damaging
the reputation of the business.
There's a lot of great research out there around responsible AI,
and that's the stuff that's been rolled into products.
And then, of course, I just showed with Gemini,
learning from mistakes is constant
questioning at the front.
Yes.
I also heard that, I didn't verify that to be true,
but I incorporated this feature that makes certain races
and ethnicities historical objects.
Yeah.
So the question was issues where races and things
were mixed in historical context was the same problem.
So, for example, if you had a prompt that
said, draw me a samurai, the idea
was like they didn't want to have--
the engine that changed the prompt
to make sure that it was fair would end up saying,
give me a mixture of samurai of diverse backgrounds.
And then you'd have male and female samurai,
samurai of different races and those kind of things.
And it was the same prompting that
ended up causing the damage that I just demonstrated.
So the idea was to intercept your prompts
to make sure that the outputs of the model
would end up providing something that was more fair when it comes
to diverse representation.
So it was a very naive solution that ended up being rolled in.
That was a few years ago.
They've massively improved it since then,
but that's when I'm talking about
if you're working in the AI space nowadays,
that's how responsibility is evolving.
You can't just get away with that stuff anymore.
That Gemini lesson was a good-- that Gemini example
is a good lesson from that.
And the mindset of you will make mistakes,
so learning from mistakes is a constant ongoing thing.
And going back to the people point
that Andrew made earlier on, the people around you
will make mistakes too.
So to have the ability to give them
grace when they make mistakes and to work
through those mistakes and move on is really, really important
and is a reality of AI at work.
I've spoken a lot about the business focus advantage,
so I'm going to skip over this.
So now let's talk about vibe coding.
So let's talk about the whole idea of generating code.
Now, the meme is out there that it makes engineers
less useful by the fact that somebody can just prompt code
into existence.
There is no smoke without fire, of course,
but I would say don't let that meme get you down
because that's when you start peeling into these things, that
is ultimately not the truth.
The more skilled you are as an engineer,
the better you become using this type of vibe.
Somebody give me another phrase other than vibe coding,
using this probe to coding.
And I always like to think about this
and to try and put you and put people
that I speak with into the role of being a trusted
advisor for the people that you speak with.
So whether you're interviewing with somebody,
get yourself into the mindset of being a trusted
advisor of the company that you're interviewing for,
whether you're consulting or whatever those kind of things
are.
So when you want to get into the idea of being a trusted advisor,
then you really need to understand the implications
of generated code.
And nobody can understand the implications of generated code
better than an engineer.
And the metric that I always like to use around that
is technical debt.
Quick question.
Are you familiar with the phrase technical debt?
Nobody.
OK.
Andrew and I were doing a conference
in New York on Friday, and I used the phrase,
and I saw a lot of blank faces.
So I didn't realize that people didn't understand
what technical debt is.
So let me just take a moment to explain that,
because I find it's an excellent framework to help you understand
the power of vibe coding.
Think about debt the way you normally would.
Buying a house.
If you buy a house, say, you borrow half a million dollars
to buy a house.
In a 30-year mortgage, when you're buying that house at half
a million dollars, with all the interest that you pay is about
double.
So you end up paying back the bank about $1 million
on half a million owned.
So you have 30 years of home ownership
at a cost of $1 million in debt.
That is probably a good debt to take on,
because the value of the house will increase over that time.
You're not paying rent over that time,
and that million dollars that you're
spending on this house over those 30 years
is a good debt to take on, because you're
getting greater than $1 million worth of value out of it.
A bad debt would be an impulse purchase on a high interest
credit card.
Those pair of shoes, those latest ones
I really want to buy them.
It's $200.
By the time I've paid them off, it's $500.
You're not getting $500 worth of benefit out of those shoes.
Approaching software development with the same mindset
is the right way to go.
Every time you build something, you take on debt.
It doesn't matter how good it is, there's always
going to be bugs.
There's always going to be support.
There's always going to be new requirements coming in
from people.
There's always going needs to market it.
There's always going needs for feedback.
All of these things are debt, every time you do a thing.
The only way to avoid debt is to do nothing.
So your mindset should then get into when
you are creating a thing, whether you're
coding it yourself or whether you're vibe coding it
or any of these things that you are increasing
your amount of technical debt, those things
that you need to pay off over time.
So the question then becomes, as you
vibe code a thing into existence in the same way
as buying a thing, is it worth the technical debt
that you're taking on?
What does technical debt generally look like?
Bugs that you need to fix, people
that you need to convince to help you maintain the code,
documentation that you need to do,
features that you need to add, all of these kind of things.
You're all very familiar with them.
Think about those as that extra work
that you need to do beyond your current work.
That's the debt that you're taking on.
There are soft debt, and there are hard debt.
So to me, that would be the number one piece of advice
that I give.
And it's the one that I give every time I work with companies
around vibe coding.
And a lot of companies that I speak with, a lot of companies
that I consult with--
I do a lot of work with startups,
in particular-- they just want to get straight
into opening Gemini or GPT or Anthropic
and start churning code out.
Let's get to a prototype phase very quickly.
Let's go to investors.
Let's do stuff.
It's great.
It can be.
But debt, debt, debt, debt, debt is always going to be there.
How do you manage your debt?
A good financier manages their debt and they become rich.
A good coder manages their technical debt,
and they become rich also.
So how do you get the good technical debt?
How do you the mortgage instead of the high credit card debt?
Well, number one is your objectives.
What are they?
Are they clear?
And have you met them?
You knew what you needed to build.
You didn't just fire up ChatGPT and start spinning code out.
At least I hope you didn't.
Think about how you build it.
AI was there to help you build it faster.
I'm working on my own little startup
at the moment in the movie making space.
And I've been using code generation almost completely
for that.
But what I've ended up doing for my clear objectives
met box here is that I've started
building this application.
I've tested it.
I've thrown it away.
I started again, tested it, thrown it away.
Each time my requirements have been improving in my mind.
I understand how to do the thing a little bit better,
and I can show some of the output of it in a few minutes.
But the idea there is that it's always
about having those clear objectives and meeting them.
And then if you're building out the thing
and you're not meeting those objectives,
that's still a learning.
And there's no harm in throwing it away
because code is cheap now in the age of generated code.
Finished code, engineered code is not cheap.
So get those objectives, make them clear,
build it, hit a specific requirement and move on.
Is there business value delivered?
Is the other part of it.
I've seen people vibe coding for hours on things like Replit
to build a really, really cool website.
And then the answer was, so what?
I mean, how is this helping the business?
How is this really driving something?
It's really cool.
Yes, Mr. VP, I know you've never written a line of code
in your life, and it's really cool that you've built
a website now, but so what?
So think about that, and focus on that.
And that's how you avoid the bad technical debt.
And then, of course, the most understated part of this,
and in some ways the most important, particularly
if you're working in an organization,
is human understanding.
The worst technical debt that you can take on
is delivering code that nobody understands.
Only you understand that, and then
you quit and get a better job.
And then the company is now dependent on that code.
So being able to, as part of the process
of building it, to make sure that your code is understandable
through documentation, through clear algorithms,
through the fact that you've spent some time poring
through it to make sure that even
simple things like variable names make sense
is a really, really important way to avoid bad technical debt.
And that bad technical debt, my favorite one
is the classic solution looking for a problem.
Somebody has an idea.
Somebody has a tool.
If the only tool you have is a hammer,
every problem looks like a nail.
And you end up having all of these tools
that get vibe coded into existence.
I've worked in large organizations
where people just vibe coded stuff, checked it into the code
base, and then it became really hard to find the good stuff
amongst all the bad.
Spaghetti code.
Of course, poorly structured stuff,
particularly when you prompt and prompt and prompt and prompt
again, that it can end up getting
into all kinds of trouble.
My favorite one at the moment that I'm really struggling with
is I'm building a macOS application.
Anybody ever build in SwiftUI on macOS?
OK, a couple.
SwiftUI is the default language that Apple
use for building for macOS as well as iPhone.
But when you look at the training set,
the data training sets that are used to train these models,
the vast majority of the code is iPhone code, not macOS code.
And when I prompt code into existence,
it's often given me iOS APIs and those kind of things.
Even though I'm in Xcode and I've created a macOS app
and it's a macOS template and I'm talking to it in Xcode,
it still gives me iOS code, stuff like that.
And then if I try to change it using prompting,
you end up spiraling into spaghetti code,
and you have to end up changing a lot of this stuff manually.
And then, of course, the other one
that I joked about it earlier, but it's also true,
is some of the bad technical debt
that you're going to encounter in the workspace is authority
over merit.
That VP suddenly took out his credit card,
subscribed to Replit, and started building stuff
in Replit.
And guess whose job it is to fix it?
So a lot of the advice that I start
giving companies and a lot of the words
that I would encourage you to start thinking
of in being a trusted advisor is to understand this stuff
and to manage expectations accordingly.
OK, so framework for responsible vibe
coding we've just spoke about.
So one of the things I want to get into as we're coming soon
to a close is the hype cycle.
So hype is the most amazing force.
I mean, I think it's one of the strongest forces
in the universe, and particularly in anything
that's hot, such as the fields that I work in that are super
hot at the moment and full of hype or AI and crypto--
you should see my Twitter feed--
that the amount of nonsense that's out there is incredible.
So one of the things that I would
say about the anatomy of hype that you really
need to think about is if you are consuming
news via social media, that the currency of social media
is engagement.
Accuracy is not the currency of social media.
So I go on to-- even LinkedIn, which
is supposed to be the more professional of these,
is absolutely overwhelmed with influencers posting things
that they've used, Gemini or GPT,
to write an engaging post so that they can get engagement
and they can get likes.
And the engine itself is engineered, excuse the pun,
to reward those types of posts.
And we end up with that snowball effect
of engagement being rewarded.
If you are the kind of person who
can filter the signal from the noise,
and then who can encourage others around the signal and not
the noise, that puts you in a huge advantage that
makes you very distinctive.
It's not as quickly and easily tangible as likes
and engagements on social media.
But when you're in a one-to-one environment like a job
interview, or if you are in a job
and you are bringing that signal to the table instead
of the noise, that makes you immensely valuable.
So coming in with that mindset, coming
in with the idea of trying to filter
that signal from the noise, trying to understand
what is important in current affairs, how
you can be a trusted advisor in those things,
and how you can really whittle down that noise to help someone
is immensely valuable.
I want to start with one story.
I might be stealing my own thunder.
I'll go on to in a moment.
So one story.
Last year when agents started becoming the key word
and everybody saying, in 2025, agent
will be the word of the year and the trend
of the year, a company in Europe asked
me to help them to implement an agent.
So let me ask you a question.
If a company came up to you and said,
please help me implement an agent,
what's the correct first question that you ask them?
What is an agent for you?
OK.
That's good.
What is an agent for you?
I'd actually have a more fundamental question.
Yep.
What do you want to do?
OK.
Even more fundamental.
My question was why?
Why?
And peel that apart.
I spoke with the CEO, and he was like, oh.
Yeah, everybody's telling me that I'm
going to save business costs.
And I'm going to be able to do these amazing things.
And yeah, my business is going to get
better because I have agents.
And I'm like, well, who told you that?
It was like, oh, yeah, I read this thing on LinkedIn,
and I saw this thing on Twitter.
And it was like--
and we ended up having that conversation.
And it was a difficult conversation
because I had to keep peeling apart.
And I started asking the questions
that you two just mentioned as well, until we really
got to the essence of what he wanted to do.
And what he really wanted to do, when
we take all domain knowledge about AI aside,
was that he wanted to make his salespeople more efficient.
And I was like, OK, you want to make your salespeople more
efficient.
Nowhere in that sentence do I hear the word AI,
and nowhere in that sentence do I hear the word agent.
So now, as a trusted advisor, let
me see what I can do to help your salespeople become
more efficient.
And I'm not going to be an AI Shill or an agent Shill.
I just want to say, what do we do to make your salespeople more
efficient?
If anybody here has ever worked in sales,
one of the things you realize what a good salesperson has
to do is their homework.
Before you have a sales call with somebody,
before you have a sales meeting with somebody,
you need to check their background.
You need to check the company.
You need to check the needs of the company.
You see it sometimes in the movie that, oh,
such and such plays golf.
So I'll take them to play golf.
It's not really that cliched, but there is a lot of background
that needs to be done.
So I spoke with him, and I spoke with their leading salespeople
and found out that-- and I asked the salespeople,
what do you hate most about your job?
And they were like, well, I hate the fact
that I have to waste all my time going
to visit these company websites, going
to look up people on LinkedIn.
And every website is structured differently.
So I can't just have a path through a website
that I can follow.
I have to take on all this cognitive load.
And they were spending about 80% of their time researching
and about 20% of their time selling.
Oh, and by the way, most salespeople
don't get paid very much.
They have to make it up by commission,
so they're only spending 20% of their time doing the thing that
gets them commission directly.
So we're like, OK, well, here's something now
where we can start thinking about making them more efficient
by cutting into that.
So we set a goal is like to make salespeople 20% more efficient.
And then we could start rolling out the ideas of AI.
And then we could start rolling out the ideas of agentic AI.
And a quick question what's the difference
between AI and agentic AI?
OK.
So-- yeah.
Like a good AI can do some [INAUDIBLE] a couple of steps.
OK.
[INAUDIBLE]
Yep.
Excellent.
Yeah.
So agentic AI is really about breaking it down
into steps, which is good engineering to begin with.
But agentic AI, in particular, I find
there's a set pattern of steps that if you follow them,
you end up with a whole idea of an agent.
The first of these steps is to understand intent.
We tend to use the words AI, Artificial Intelligence, a lot.
But what large language models are really, really good at
is also understanding.
So if the first step of anything that you want to do
is to understand intent.
And you can use an LLM to do that to think about this
is the task that I need to do.
This is how I'm going to do it.
Here's the intent.
I want to meet Bob Smith and sell widgets to Bob Smith.
And this is what I know about Bob Smith.
Help me with that intent.
The second part then is planning.
So you declare to an agent what tools are available to it,
browsing the web, searching the web,
all of these kind of things.
And once you understand your clear intent
to be able to go to the step of planning and using
those tools for planning, and an LLM is very, very good
at then breaking that down into the steps
that it needs to do to execute a plan.
Search the web with these keywords.
Browse this website and find these links,
those types of things.
Once it's then figured out that plan,
then it uses the tools to get to a results.
And then once it has the result, the fourth and final step
is to reflect on that result. And looking at the results
and going back to the intent, did we meet the intent?
Yes or no.
If we didn't, then go back to that loop.
All agent is really broken down into those things.
And if you think about breaking any problem down
into those four steps, that's when
you start building an agent.
And that was part of being a trusted advisor,
instead of coming in and waving hands and saying,
agent this, agent that.
Look at this toolkit, save 20%.
It's really to break it down into those steps.
Se we did.
We broke it down into those steps.
We built a pilot for the salespeople of this company,
and they ended up saving between 10% and 15% of their time,
of their wasted time.
The doctrine of unintended consequences
hit, though, after this.
And the unintended consequence was the salespeople
were much happier because the average salesperson was making
several percentage points more sales in a given week,
they were earning more money in a given week,
and their job just became a little bit less miserable.
And then refinement to that agentic process,
to be able to do all of that research for them
and to help give them a brief in a few minutes instead
of a few hours to help them with the sales process,
ended up being like a win-win-win all around.
But if you go in being hype led and oh,
build an agent for the thing without really peeling apart
the business requirements, the why, the what,
the how, and all of these kind of things, we ended up like,
this company just would have been lost in hype.
You've probably seen reports recently.
I think McKinsey put one out last week showing that about 85%
of AI projects at companies fail.
And part of the main reason for that
is that they're not well scoped.
People are jumping on the hype bandwagon,
and they're not really understanding their way
through the problem.
And I think you know the big brains
in this room and the network that you folks have are really
key component of being able to succeed
is to understand your way through that problem.
So that was a hype example around agentic
that I was thankfully able to help this company through.
Other recent hype examples you've probably seen,
the software engineering is dead.
My personal favorite, Hollywood is dead or AGI by year end.
I was in Saudi Arabia this time last year
at a thing called the FYI.
And it was a dinner at the FYI, and I
sat beside the CEO of a company who I'm not going to name,
but this was a CEO of a generative AI company.
And at that time he was showing everybody
around the table this thing that he'd
done, where it was text to video,
and he could put in a text prompt and get video out
of the prompt and get about six seconds worth of video
out of it.
A year ago, that was--
I beg your pardon, two years ago.
Two years ago, that was hot stuff.
Nowadays, obviously, it's quite passé.
Anybody can do it.
But he made a comment at that table,
and it was a lot of media executives at that table
was like, by this time next year, from a single prompt,
we'll be able to do 90 minutes of video.
And so bye-bye, Hollywood.
So the whole Hollywood is dead meme, I think, came out of that.
First of all, we can't do 90 minutes, even two years later
from a prompt.
And even if you did, what kind of prompt
would be able to tell you a full story of a movie?
So this type of hype leads to engagement.
This type of hype leads to attention.
But my encouragement to you is to peel that apart.
Look for the signal.
Ask the why question.
Ask what question and move on from there.
So becoming that trusted advisor.
World's drowning in hype.
How do you do it?
Look at the trends, evaluate them objectively.
Look at the genuine opportunities
that are out there.
There are fashionable distractions.
I don't know what the next one is going to be,
but there are these distractions that
are out there that will get you lots
of engagement on social media.
Ignore them, and ignore the people
that are leaning into them.
And then really lean into your skills
about explaining technical reality to leadership.
One skill that one person coached me
in once that I thought was really interesting,
because it sounded wrong, but it ended up being right,
was whenever you see something like this,
try to figure out how to make it as mundane as possible.
When you can figure out how to make it as mundane as possible,
then you really begin to build the grounding
for being able to explain it in detail in ways
that people need to understand.
If you go and you look at, I think
Gemini 3 was released today, but there were leaks
earlier this week.
And one person leaked that I built a Minecraft clone
in a prompt, that kind of stuff.
This is the opposite of mundane.
This was massively hyping the thing, massively showing.
And of course, they didn't.
They built a flashy demo.
They didn't really build a Minecraft clone.
But the idea here is if you can peel that apart to OK,
how do I think about what are the mundane things that
are happening here?
The one that I've been working with a lot recently is video.
So text to video prompts, as I've mentioned,
instead of the magical, you can do whatever you want all nice
and fluffy Hollywood is dead, what
is the mundane element of doing text to Video
The mundane element of doing text to video
is that when you train a model to create video from a text
prompt, what it is doing is it's creating
a number of successive frames.
And each of those successive frames
is going to be slightly different from the frame before.
And you've trained a model by looking at video to say, well,
if in frame 1, the person's hands like this and frame 2
it's like that, then you can predict
it moves this way if there's a matching prompt.
And suddenly it's become a little bit more mundane,
but suddenly they begin to understand it.
And then the people who are experts
in that specific field, not the technical side of it,
are now the ones that will actually
be able to come up and do brilliant things with it.
So that height navigation strategy--
filter actively, go deep on the fundamentals,
get your slides to work.
And then, of course, keep your finger on the pulse.
The hardest part of that, I think,
is the third one is really keeping
your finger on the pulse.
And that's when you have to wade into those cesspits of people
just farming engagement and really
try to figure out the signal from the noise there.
But I think it's really important for you
to be able to do that, to be connected, to understand that.
Reading papers is all very good.
The signal-to-noise ratio, I think, in reading papers
is a lot better.
But to understand the landscape that the people
that you are advising, they are the ones
who are waiting in the cesspools of Twitter and X and LinkedIn.
And there's nothing wrong with those platforms
in and of themselves, but the stuff that's
posted on those platforms.
So overall landscape, it is ripe with opportunity,
absolutely ripe with opportunity.
So I would encourage you, as Andrew
did, to continue learning, to continue digging
into what you can do and to continue building.
But there are risks ahead.
Anybody remember the movie Titanic?
Remember the famous phrase in that, "iceberg right ahead"?
But immediately before that, there's a scene in Titanic--
if we weren't being filmed, I would show it,
but I can't for copyright reasons-- where
the two guys up in the crow's nest are freezing and talking.
And the crow's nest at the top of the ship
is where the spotters would be to spot any icebergs in front.
And go back and watch the movie again.
You'll see the conversation between these two guys
is that all they're talking about is how cold they are.
And then it cuts away to the crew
of the ship who are like, wait, aren't they
supposed to have binoculars?
And then the crew is like, oh, we left the binoculars
behind in port.
That framing the whole idea was like,
they were so arrogant in being able to move forward
that they didn't want to look out for any particular risks.
And even though they had people whose job it
was to look out for risks, they didn't properly
equip or train them.
And that, to me, is a really good metaphor
for where the AI industry is today.
There are risks in front of us.
Those risks, the B word, the bubble word
you're probably reading in the news is there, are there.
To me, though, the opportunity and the things to think about
in terms of a bubble are most of you probably don't remember
dotcom bubble of the 2000s.
But if you think about the dotcom bubble,
that was the biggest bubble in history.
It bursts, but we're still here.
And the people who dotcom rights not only survived, they thrived.
Amazon, Google, they did it right.
They understood the fundamentals of what
it was to build a dotcom.
They understood the fundamentals of what it was
to build a business on dotcom.
And when the bubble of hype burst, they didn't go with it.
There was one website, I believe it was pets.com,
that they had the mindset of if you build it, they will come.
They had Super Bowl commercials around pets.com.
They couldn't handle the traffic that they got.
And that was the kind of site that when the bubble burst,
those were the sites that just evaporated.
So that bubble in AI is likely coming.
There is always a bubble.
So the companies that are doing AI right
are the ones, like I said, that won't just
avoid the bubble that they will actually thrive post bubble.
And the people who are doing AI right, the folks
in this room who are thinking about AI
and how you bring it to your company,
and the advice that you're giving to your company
and leaning into that in the right way
will also be the ones who not only avoid getting laid off
in the bubble crashes, but will be the ones who will thrive
through and after the bubble.
So anatomy of any bubble, and what I'm seeing in the AI
one in particular, is this kind of pyramid.
At the top is the hype that I've been talking about.
At the bottom is massive VC investment.
I'll be frank.
I'm already seeing that drying up.
Once upon a time, you could go out
with anything that had AI written on it
and get VC investment.
Then you could go out and do anything with an LLM
and get VC investment.
Now there are far, far, far more cautious.
I've been advising a lot of startups.
The amount that they're getting invested is being scaled back.
The stuff that's being invested in is changing.
And the second layer down, massive VC investment
is already beginning to vanish.
Unrealistic valuations.
Companies that aren't making money being valued
massively high.
We all who they are.
We're beginning to see those unrealistic valuations being
fed off of that hype.
# #MeToo products, where somebody does something,
and it's successful, and everybody
jumps on the bandwagon.
We're also seeing them everywhere.
We saw them throughout the dotcom bubble.
And then right at the bottom is that real value.
I probably shouldn't have done the triangle like this.
It should be more an upside down triangle.
Because the real value here is small.
But I've vibe coded these slides into existence.
So this is one of the technical debt I took on.
But the real value there, that kernel of value is there,
and the ones that build for that will be the ones that survive.
So the direction that I see the AI industry going in
and the direction that I would encourage you to start thinking
about your skills in, is really over the next five years,
there's going to be a bifurcation.
I'm just going to be ornery in how
I describe it as big and small.
Big AI will be what we see today, with the large language
models getting bigger in the desire to drive towards AGI.
The Geminis, the Claudes, the OpenAIs of the world
are going to continue to drive bigger, and bigger
is better in the mindset of those companies
towards achieving AGI or towards achieving better business value.
That's going to be one side of the branch.
The other side of the branch is I'm going to call it small.
We've all seen open-source models.
I hate the term open source.
Let me call them open weights or let me call them
self-hostable models are becoming-- they're
exploding onto the landscape.
I read an article recently about Y Combinator
that 80% of the companies in Y Combinator
were using small models from China in particular.
So the Chinese models in particular
are doing really well, probably because of
the overall landscape.
They're not leaning into the large models the same way
as the West is.
I see that bifurcation happening.
China, I think, has that head start
on the small models that may last.
It may not.
I don't know.
But the point is, we're heading in that particular direction
of I'm going to call them instead of big and small now,
models that are hosted on your behalf by somebody else,
like a GPT or a Gemini or a Claude,
or models that you can host yourself for your own needs.
As this side has right now is underserved,
this bubble may burst.
This one right now is underserved.
And this bubble will be later on.
And the major skills that I can see developers needing
over the next two to three years on this side of the fence
will be fine tuning.
So the ability to take an open-source model
and fine-tune it for particular downstream tasks.
Let me give one concrete example of that I've personally
experienced.
I work a lot in Hollywood, and I've worked a lot
with studios making movies.
And one studio in particular I was lucky enough
to sell a movie to, it's still in preproduction.
It'll probably be in preproduction forever.
But one of the things I learned as part of that process
was IP in studios is so protected.
It's not even funny.
Go in Google for James Cameron, who
created Avatar and the lawsuits that he's
involved in of this person who apparently sent him
a story many years ago about blue aliens
and is now suing him for billions of dollars
because obviously there were blue aliens in Avatar.
That level of IP protection in Hollywood is insane.
The opportunity with large language models
is equally insane.
A lot of the focus is on large language models for creation,
for storytelling, for rendering and all that,
but actually the major opportunity that they have is
actually for analysis to take a look at synopses of movies
and find out what works and what doesn't.
Why was this movie a hit and this one wasn't?
What time of year was this one released and it became
successful and this one wasn't?
And with a margin on movies being razor thin,
that kind of analysis is huge.
But in order to do that kind of analysis,
you need to share the details of your movie
with a large language model.
And they will absolutely not do that with GPT or Gemini
or whatever, because they're now sharing
their IP with a third party.
Enter small models, where they can self-host
their own small model and they are getting smarter and smarter.
The 7B model of today is as smart as the 50B model
of yesterday.
A year from now, the 7B model of a year from now will be as smart
as the 300B model of yesteryear.
So they're moving in that direction of building
using small self-hosted models, which they can then
fine-tune on downstream tasks.
Similar with other things where privacy
is important law offices, medical offices, all
of those kind of things.
So those type of skills are fundamentally
important going forward.
So that's the bifurcation that I'm seeing happening in AI.
The sooner bubble I think is in the bigger non self-hosted.
The later bubble is in the smaller self-hosted.
But either way, for you, for your career,
to avoid the impact of any bubble bursting,
focus on the fundamentals.
Build those real solutions.
Understand the business side, and most of all,
diversify your skills.
Don't be that one trick pony who only knows how to do one thing.
I've worked with brilliant people who
are fantastic at coding, in particular API,
or particular framework.
And then the industry moved on and they got left behind.
OK, so yeah, when bubbles burst, that overall fallout kind of
spoken about it a little bit already.
Funding evaporates, hiring freezes become layoffs,
projects get canceled, and talent floods the market.
Yeah.
Quick question from the last slide.
[INAUDIBLE] I heard a lot about how NVIDIA is hiring,
and they're very specific about they
want people for very specific problem that they have.
So they can require people to be basically put out that one thing
that you're missing.
So how do you think-- how is it more important to diversify
skills versus actually focusing on, for example,
LLMs versus computer vision or versus
very specific downstream task?
So I mean, I think so the question was around NVIDIA
in particular or hiring for a very specific, very
narrow scenario.
So then the question is, how important
is it for you to become an expert
in a narrow scenario versus diversifying your skills?
I would always argue it's still better to diversify your skills,
because that one narrow scenario is only that one
narrow scenario, and you're putting all your eggs
into one basket.
NVIDIA would be a fantastic company to work for.
Nothing against them in any way.
But if you put all of your eggs into that basket and you
don't get it, then what?
So I think the idea of really being
able to-- if you are passionate about a thing,
to be very deep in that thing is very, very good.
But to only be able to do that thing,
I think I would always encourage to be diversified.
And when I say diversified, you're saying LLMs or computer
vision or anything like that, I think
I mean that's one part of it.
But it's like that knowledge of models and how to use them to me
is a uni skill.
The diversification of skills is breaking outside of that.
Also to be able to think, OK, what about building applications
on top of these?
What does scaling an application look like?
What does software engineering in this case look like?
What about user experience and user experience skills?
Because it's all very well to build a beautiful application.
But if nobody can use it--
I'm looking at here at Microsoft Office.
There's stuff like that that's what I really
mean about diversifying beyond.
So even in that mono example with NVIDIA,
to be able to break out of that one particular example,
but to show skills in other areas that are of value,
I think is really important.
OK.
As we're just running a little bit-- so yeah,
I just wanted to--
I've gone into it a little bit already,
but I'm a massive advocate for small AI.
I really do believe small AI is the next big thing,
because we're moving into a world,
and this is part of the job that I do at Arm,
is we're kind of moving into a world of AI everywhere
all at once.
So there's a traditional, and it's
interesting you just brought up NVIDIA
because there's a traditional conception
that compute platforms are CPU plus GPU when it comes to AI.
But that's also changing--
CPU general purpose, GPU specialists.
But for example, in mobile space,
there's massive innovation being done with the technology
called SME, Scalable Matrix Extensions.
And what SME is all about is really
allowing you to bring AI workloads
and put them on the CPU.
The frontrunners in this are a couple of Chinese phone vendors,
Vivo and Oppo, who've just recently released phones
with SME-enabled chips.
And what's magical about these is that, A,
they don't need to have a separate external chip drawing
extra power, taking up extra footprint space just
to be able to run AI workloads.
And B, the CPU, of course, being a low power pulling thing,
being able to run AI workloads on that,
they've been able to build interesting new scenarios.
And if I talk about one in particular,
there's a company called Alipay.
And Alipay had an application where you would--
and we've all seen these apps where
you can go through your photographs,
and you can search for a particular thing.
Places I ate sushi or something along those lines and use
that to create a slideshow.
All of those require a back end service.
So your photographs are hosted on Google Photos or Apple
Photos or something like that.
And that back end service runs the model
that you can search against it and be
able to do the assembly of them.
What Alipay wanted to do was like, say, there
are three problems with this.
Problem number one, privacy.
You have to share your photos with a third party.
Problem number two, latency.
You got to upload those photos.
You got to send the thing.
You got to have the back end do the thing,
and then you've got to download the results from the thing.
And then number three is building that cloud service
and standing that up cost time and money.
So if they could move all of this onto the device itself,
now the idea was they could run a model
on the device that searches the photos on the device.
You don't have the latency.
And business perspective, they're
now saving the money on creating this stand up service.
They now have AI running on CPU in order to be able to do that.
Apple are also people who've invested heavily
in this scalable matrix extensions.
You see whenever they talk about--
if you've ever watched a WWDC or anything like that, when they
talk about the new A-series chips and M-series chips,
about the neural cores and those kind of things in them, that's
part of the idea.
So to think about breaking that habit that we've gotten into,
where you need a GPU to be able to do AI is part of the trend
that the world is heading in.
Apple are probably one of the leaders in that.
I'm very, very bullish on Apple and Apple Intelligence
as a result. And from the AI perspective, seeing that trend
and following that vector to its logical conclusion as models
are getting smaller embedded intelligence getting everywhere
isn't a pipe dream.
It isn't sci-fi anymore.
It's going to be a reality that we'll
be seeing very, very shortly.
So that idea of that convergence of AI,
because of the ability of smaller models getting smarter
and lower power devices being able to run them,
we see that convergence hitting, and I see massive opportunity
there.
So one last part and just going back to agents for a moment,
I think the one thing that I always
say is like a hidden part of artificial intelligence
is really what I like to call artificial understanding.
And when you can start using models to understand things
on your behalf.
And when they understand them on your behalf,
to be able to craft from that understanding new things,
you can actually develop superpowers
where you're far more effective than ever before,
be that creating code or creating other things.
I'm going to give one quick demo just so we can wrap up.
And I was talking earlier about generating video.
So this picture is-- oops.
Sorry.
The connection here is not very good, I lost it.
So here we go.
This picture here is actually of my son playing ice hockey.
And I took this picture, and I was saying,
OK, I think I'm very good at prompting.
And I wrote a nice prompt for this picture to get him.
He's in the middle of taking a slapshot.
He's got some beautiful flex on his stick.
And I asked it like, OK, to prompt him scoring a goal.
What do you think happened?
Should we watch?
Let's see if it works.
[VIDEO PLAYBACK]
[CROWD CHEERING]
[END PLAYBACK]
This was the wrong video, but it still shows the same idea.
Because of poor prompting or because of poor understanding
of my intent, if I talk about it in agentic senses,
the arena that he was in, which is a practice arena
and doesn't have any people in it-- sorry.
Let me pause it.
If I just rewind to here, if we look up
in this top right-hand corner here,
this is basically where they store all their garbage.
But the AI didn't know that, had no idea of it.
So it assumed it was a full arena,
and it started painting people in.
And even though he shot a mile wide, everybody cheers.
And somehow he has two sticks in his hand instead of one,
and they forgot his name.
So I did not go through an agentic workflow to do this.
I did not go through the steps of, A, understand my intent.
B, once you understand my intent,
understand the tools that are available to you.
In this case, it's Veo, and understand
the intricacies of using Veo.
Make a plan of how to use them.
Make a plan of how to build a prompt for them,
and then use them and then reflect.
So I've been advising a startup that is working
on movie creation using AI.
And I want to show you a little sample here of a movie
that we've been working on with them, where the whole idea is
like, if you want to have performances at a virtual actors
and actresses, you need to have emotion.
You need to be able to convey that emotion,
and you also need to be able to put that emotion in the context
of the entire story.
Because when you create a video from a prompt,
you're creating an eight-second snippet.
That eight-second snippet needs to know
what's going on in the rest of the story.
So if I show this one for a moment.
And it's a little wooden at the moment,
it's not really working perfectly.
I have professional actors who are friends
who are advising me on this, and they
laughed at the performances.
But try to view it through the difference
that we had from an agentic prompt with the hockey
player to this one.
[VIDEO PLAYBACK]
That's hopefully we can hear it.
- I guess I can do the pub quiz after all.
They just shut me down.
I'm so close.
But they wouldn't listen.
- I won't--
[END PLAYBACK]
They never listen.
So here's the idea of, again, just
thinking in terms of agentic, as I was saying earlier on,
breaking it into those steps.
That allowed me to use exactly the same engine,
as I was showing you earlier on, that
fails to be able to show something
that works and is able to do things like portraying emotion
that I just spoke about.
So I know we're a little bit over time.
So sorry about that.
I can take any questions if anybody has any.
I see Andrew is here as well.
He's at the back.
And I just really want to say thank you
so much for your attention.
I really appreciate it.
[APPLAUSE]
Yep.
How much of this new generation [INAUDIBLE]
relation with the agentic [INAUDIBLE] use case
is improved with the agentic workflow?
And how much of it is a training set bias
where you might have only pictures
or videos with [INAUDIBLE] that are full of [INAUDIBLE]
Yeah, it's a great question.
Just to repeat for the video, how much of the improvement
is from the use of an agentic workflow
versus just lack of hockey stuff in the training set
for the failed one?
Not comparing like to, so just using my gut.
When I looked at when I broke this down into the workflow that
said, OK, I created scenes like this one
and they were awful when I just did it directly for myself
with no basis, no agentic, no artificial understanding.
And when I broke it down into the steps where it's like, OK,
in this scene, the girl is sitting on the bench,
and she's upset.
And the person is talking to her and he wants to comfort her.
Feeding that to a large language model
along with the entire story and along
with the constraints that I had, where the shot
has to be eight seconds long, clear dialogue
and all of those kind of things, and then
to understand my intent from that one,
the LLM ended up expressing a prompt that
was far more loquacious than I ever would have,
that was far more descriptive than I ever would have.
The LLM had understanding of what
makes a good shot, what makes a good angle, what
makes good emotion far more than I would have.
I could spend hours trying to describe it.
So that first step in the agentic flow
of it doing that for me and understanding my intent
was huge.
The second step then is the tools that it's going to use.
So I explicitly said which video engine I'm going to be using.
I was using Gemini as the LLM, and hopefully Gemini
is familiar with Veo, that kind of stuff,
so to understand the idiosyncrasies of doing things
with Veo.
What I learned, for example, Veo was
very bad at doing high-action scenes,
but is very good at doing slow camera pulls to do emotion,
as you saw in this case.
So the LLM knew that from me, declaring
I was using that as a tool.
And then further it built a prompt
and then further refined the prompt from that.
And then the third part actually using the tool
to actually generate it for me, generating a video
with something like Veo costs, I think, between $2 and $3
to generate four videos and credits.
So the last thing I want to do is
generate lots and lots and lots and lots of videos
and throw good money after bad.
But all of that token spend that I did earlier on
to understand my intent and then to make the plan for using
the agent was saved in the back end where it got it right.
Maybe not get it right first time,
but it would very rarely take more than two or three tries
to get something that was really, really nice.
So I think without comparing like with like, I
do think that plan of action and going through a workflow, that
worked very, very well.
Any other questions, thoughts, comments?
Yeah, up at the back.
What has surprised you the most about the AI
industry over the years?
What has surprised me the most about the AI
industry over the years?
Oh, that's a good one.
I think what has surprised me the most,
and it probably shouldn't have surprised me,
is how much hype took over.
I actually-- I honestly thought a lot of people
who are in important decision making roles
and that kind of thing would be able to see the signal better
than they did.
And I think the other part was that the desire to make
immediate profits as opposed to long-term gains
also surprised me a lot.
Let me share one story in that space was one of the things
that after Andrew and I taught that the TensorFlow
specializations on Coursera, and after that, Google
launched a professional certificate
where the idea of this professional certificate
was would give a rigorous exam.
And at the end of the rigorous exam,
if you got the certificate, it was a high prestige thing
that would help you find work, and particularly
at the time when TensorFlow was a very highly demanded skill
in order to get work.
Running that program cost Google $100,000 a year.
Drop in the bucket, not a lot of money.
The goodwill that came out of it was immense.
I can tell you--
I'll tell one story very quickly, was a young man
and he went public in some advertising
stuff that with Google that he lived in Syria.
And we all know there was a huge civil war in Syria
over the last few years.
And he got the TensorFlow certificate.
He was one of the first in Syria to get it,
and it lifted him out of poverty,
where he was able to move to Germany
and get work at a major German firm.
And I met him at an event in Amsterdam
where he told me his story.
And now, because of the job that he had in this German firm,
he's able to support his family back home
and move them out of the war torn zone
into a peaceful zone all because he got this AI thing.
And there were countless stories like that.
Very inspirational, very beautiful stories.
But the thing that surprised me then
was sometimes the lack of investment
in that, where there was no revenue being generated
for the company out of that.
We deliberately kept it revenue neutral so
that the price of the exams could go down.
We wanted it to self-sustain.
It ended up not being revenue neutral.
It ended up costing the company about $100,000 to $150,000
a year.
So they canned it.
And it's a shame because of all the potential goodwill
that can come out of something like that.
But I think those were the two that
immediately jump to mind that have surprised me the most.
And then I guess one other part that I would say
is the people who've been able to be very successful with AI,
who you wouldn't think would be the ones that
would be successful with AI, has always been inspirational to me.
So allow me one more story.
I have a good friend.
I showed ice hockey a moment ago.
I have a good friend who is a former professional ice hockey
player.
And any ice hockey fans Here
It's a brutal sport.
You see a lot of fighting and a lot of stuff on the ice.
And he dropped out of school when he was 13 years old
to focus on skating.
And he will always tell everybody
that he's the dumbest person alive because he's uneducated.
He and I are complete opposites.
That's why we get on so well.
And he retired from ice hockey because of concussion issues.
And he now runs a nonprofit--
the ice rinks for nonprofit.
And about three years ago, we were having a beer,
and he was like, so tell me about AI.
And tell me about this ChatGPT thing.
Is it any good?
And I was like, just sharing the whole thing.
Yes, it's good and all that kind of stuff.
And it was obviously a loaded question, and I didn't know why.
But part of his job at his nonprofit
is that every quarter, he has to present
to the board of directors the results of the operations
so that they can be funded properly,
because even though they're nonprofit,
they still need money to operate.
And he was spending upwards of $150,000 a year to bring
in consultants to pull the data from all of the different
sources.
They're pulling data from-- there's machines
in what's called the pump room that has a compressor that
cools the ice.
And there were spreadsheets and there was accounts
and all this kind of stuff.
And he was not tech savvy in any way.
But he needed to process all this data.
So he did an experiment where he got ChatGPT to do it.
And this was the loaded question,
asking me if it was any good.
And so we talked through it a little bit.
And then he told me why.
And so I took a look at the results
because he was uploading spreadsheets.
He was uploading PDFs and all this kind of thing
and getting it to assemble a report.
And it takes him about two hours to do the report himself
with ChatGPT.
And it worked, and it worked brilliantly.
And that $150,000 a year that he's saving on consulting is now
going to underprivileged kids for hockey equipment,
for ice skating equipment, for lessons,
and all of that kind of thing.
So it was taken out of the hands of an expensive consulting
company and put into the hands of people.
Because of this guy, and he says he's
the dumbest person alive, but--
I hope he's not watching this video.
And I told him afterwards that, congratulations, you're
now a developer.
And he didn't like that.
But it's like surprises like that the superpowers that were
handed to somebody like him, that he's not technical in any
way, but he was able to effectively build a solution
that saved his nonprofit $100,000 or $150,000 a year.
And things like that are always surprising me
in a very pleasant way.
Yep.
Sorry.
I'll get to you next.
Sorry.
Yeah.
For engineers like us, it's easier to navigate the hype
because we can understand what the signal is from a research
paper.
But how about people who doesn't have this knowledge, like,
from humanities or something [INAUDIBLE]?
Yeah, so just to repeat the question for the video.
For engineers like us, sometimes it's
easy to navigate the hype to see the signal from the noise.
But what about people who don't have the same training as us?
I think that's our opportunity to be trusted advisors for them
and to really help them through that, to understand it.
I think the biggest part in the hype story
right now is just understanding the reward mechanism.
That everything rewards engagement rather than
actual substance.
And to me, step one is seeing through that.
The story I just told about my friend,
he'd seen all this kind of stuff,
but he wasn't willing to bet his career on it.
But he needed that kind of advice
around it and to start peeling apart what he had done
and what he did right and what he did wrong.
And so that positioning ourselves to be trusted advisors
by not leaning into the same mistakes
that the untrained people may be leaning into,
I think is the key to that.
And just understanding that the average person is generally
very intelligent, even if they may not
be experts in a specific domain, and to key
in on that intelligence and help them to foster and to grow that
in and navigate them through the parts
where they'll have difficulty and let them
shine in what they're very, very good at.
Over here there was one.
I have a question more for AI and machine
learning for scientific research.
OK.
Which is something that is very hard [INAUDIBLE]
to get your perspective on.
Where do you think that is a good idea
and where you might say, maybe be cautious?
So AI and machine learning for scientific research,
where is it a good idea and where should you be cautious?
Ooh.
My initial gut check would be I think it's always a good idea.
I think there was no harm in using the tools that you have
available to you, but to always to just double
check your results and double check your expectations
against the grounded reality.
I've always been a fan of using automation in research
as much as possible.
My undergraduate was physics many, many years ago,
and I was actually very successful in the lab
because I usually automated things through a computer
that other people did handwriting and pen and paper
with.
So I could move quickly.
So I know I'm biased in that regard.
But I would say, for most research, for the most part,
I think use the most powerful tools you have available,
but check your expectations.
Little story actually on that side was trivia question.
Poorest country in Western Europe.
Anybody know?
Serbia?
What's that?
Or Western.
Western Europe is Wales.
So I actually did my undergraduate in Wales,
and I went back to do some lectures in the university
there.
And I met with a researcher there,
and he was doing research into brain cancer
using computer imagery and using various types of computer
imagery.
And I asked him, well, what's the biggest
problem that you have?
What's the biggest blocker for your research?
And this is about eight years ago.
And his answer was access to a GPU.
And because for him to be able to train his models
and run his models, he needed to be able to access a GPU.
And the department that he was in
had one GPU between 10 researchers,
which meant that everybody got it for half a day.
Monday through Friday, and his half a day
was Tuesday afternoon.
So in his case, he would spend the entire time
that wasn't Tuesday afternoon preparing everything
for his model run or his model training
or everything like that.
And then Tuesday afternoon, once he had access to the GPU,
then he would do the training.
And then he would hope in that time
that he would train his model and he would get the results
that he wanted.
Otherwise, he'd have to wait a week to get access to the GPU
again.
And then I showed him Google Colab.
Anybody ever used Google Colab?
And you can have a GPU in the cloud
for free with that kind of thing.
And the poor guy's brain melted that--
because I took out my phone, and I showed him
a notebook running on my phone in Google Colab
and training it on that.
And it changed everything for him research wise.
And now it was a case of-- and this was with Colab.
He had much more than he had with his shared GPU.
So I think for someone like him, machine learning
was an important part of his research,
but he was so gated on it that the ability to widen access
to that ended up really, really advancing his research.
I don't know where it ended up.
I don't know what he has done.
It has been a few years since then.
But that story just came to mind when you asked the question.
Any more questions?
Feel free to ask me anything.
Oh, yeah.
At the front here.
It's more of a general question.
You talked about AI helping food and beverage use.
What do you think AI would be a force of social equality
or social inequality?
So can AI be a force of social equality or social inequality?
I think the answer to that is yes.
It can be both, and it can be neither.
I mean, I think that ultimately, the idea
is that if in my opinion, any tool can be used for any means,
so the important thing is to educate and inspire people
towards using things for the correct means.
There's only so much governance can be applied.
And sometimes governance can cause more problems
than it solves.
So I always love to live my life by assuming good intent
but preparing for bad intent.
And in the case of AI, I don't think
there's any difference there that everything that I will do
and everything that I would advise is assuming good intent,
that people would use it for good things,
but also to be prepared for it to be misused.
The bad examples that I showed earlier on, I think
were good intent rather than bad intent.
And most mistakes that I see that are
good intent being used mistakenly as
opposed to bad intent.
But I would say that's the only mantra that I can--
the only advice that I can give and that kind of thing is always
assume good intent, but prepare for bad intent.
The AI itself has no choice.
It's how people use it.
Andrew, did you want closing comments or--
I think we were running out [INAUDIBLE] time.
But thank you for this.
Really great.
Thanks, everyone, for all the questions
on those creative solutions.
All right.
Thank you, Andrew.
Thanks.
[APPLAUSE]