Player is loading...

Embed

Embed code

Transcriptions

Note: this content has been automatically generated.
00:00:00
After lunch and I would like to welcome
00:00:03
all of you to our then with Jim pollen
00:00:07
today I think lots of you know GM
00:00:10
already and I'm really excited to be
00:00:12
able to introducing today because one
00:00:15
is maybe my second year on the faculty
00:00:19
at Stanford to did is sabbatical what
00:00:22
everybody got nine you know in our
00:00:24
group and Gina sat in on my graduate
00:00:27
class which was a real honour and a
00:00:30
little bit daunting any came up to me
00:00:32
after class one day. And he said why
00:00:36
you in a class is really great but I
00:00:38
have I have one small piece of advice
00:00:40
if you're if you're interested as I get
00:00:41
you said when you ask a question you
00:00:45
always immediately start talking right
00:00:48
afterwards and use like if you wait
00:00:50
just a couple milliseconds you'll be
00:00:52
amazed by if you if you let that
00:00:54
silence hang in the air how much more
00:00:57
interesting the discussion but columns.
00:01:00
And I think that that's one thing that
00:01:02
we can learn from from James career in
00:01:04
general is that I think he's one of the
00:01:06
people who is the best one of the best
00:01:08
listeners and our field and so we're
00:01:11
here today to listen to GM in part
00:01:15
because he's the one who taught us how
00:01:18
to listen and sure enough in class one
00:01:20
of the biggest changes I made in
00:01:21
teaching the line out sell any
00:01:24
colleague to listen is after you ask a
00:01:26
question take a moment and just let the
00:01:29
silence wagon and see what people have
00:01:31
the side for those of you who are
00:01:35
walking in or new the car I wanted to
00:01:38
tell you just a very few things about
00:01:41
what you is done nobody tell most of
00:01:42
the story. Um jealous then is that a
00:01:46
number of things over his career
00:01:48
including ah he was one of the people
00:01:51
who is in cognitive science very early
00:01:53
on at UC San Diego over his life
00:01:57
actually has been a department chair.
00:01:59
Here's wrong to research lab many his
00:02:03
former students and colleagues are in
00:02:05
the audience here at back at San Diego
00:02:08
with at Hutchins team created one of
00:02:10
the first big undergraduate
00:02:13
specialisations in human computer
00:02:14
interaction was a very early very
00:02:16
influential per program. And the kind
00:02:19
of science department. And I think that
00:02:21
that that curriculum it just doesn't
00:02:24
amazing job with with marrying the ate
00:02:27
the C in the eye together I I think
00:02:29
it's it's a real model for that also
00:02:32
Jim is without question one wanna HCI
00:02:35
is most affective undergraduate mentors
00:02:39
after I joined the cognitive science
00:02:41
department at San Diego I am amazed by
00:02:44
how passionate and dedicated and
00:02:47
excited the undergraduate alumni lower
00:02:50
will come back to an extremely engage
00:02:52
bunch the gym has several back to speak
00:02:54
in his classes they come back to have
00:02:57
dinner with us to give us advice on the
00:02:58
program it's a really it's a really
00:03:00
awesome group. And lastly as I think
00:03:05
that I hope my opening story share it
00:03:09
the genus then one of the most powerful
00:03:12
mentors in the field of human computer
00:03:14
interaction I think we're gonna really
00:03:16
lucky field if you look at how massive
00:03:19
this event has grown. And and it hasn't
00:03:24
been all that long. And I think they
00:03:27
rapid growth in the success and in
00:03:29
particular the the you know actually
00:03:32
actually medical nature of car is due
00:03:34
to the the leadership and mentor ship
00:03:37
of people like to him and so let's
00:03:40
welcome him to stay oh Okay I I see if
00:03:59
I can be here here since I'm a listener
00:04:03
try to speak. Um it's a know incredibly
00:04:07
great honour to receive this award and
00:04:12
probably as much as anything else I owe
00:04:15
any success I've had to selecting
00:04:18
really and collaborators and wonderful
00:04:20
students sciences surely a social
00:04:23
collaborative activity and and and I've
00:04:27
been incredibly fortunate in that
00:04:29
regard. And to honour them I like to
00:04:33
sort of share this award with them I
00:04:36
have room for everyone in fact is
00:04:39
walking around the conference I'm
00:04:41
already embarrassed of people that I
00:04:43
left off of this list but but anything
00:04:49
I've done has really been done in
00:04:50
collaboration with with other folks and
00:04:53
so I really like to sure they work with
00:04:56
them it was simple to get introduced by
00:04:59
scott. Um line calling now and and it
00:05:05
was fantastic to sort of attract a and
00:05:09
equally stellar other aboard that ski
00:05:13
to UCSD and and so we had a series of
00:05:18
really interesting events of late this
00:05:21
past summer were able to attract don
00:05:23
Norman to boomerang back to UCSD don
00:05:28
was there and when I first got there
00:05:31
and been there for quite a while before
00:05:33
and together with Scott don and I are
00:05:37
starting a new design lab at uses you
00:05:40
know mention it briefly a little bit
00:05:42
later. Um and and personally for me all
00:05:47
of this came at a really good time my
00:05:50
colleague of over thirty years and
00:05:53
Hutchins from whom I've learned
00:05:56
probably more than anyone else about
00:06:00
this area about people and about
00:06:03
distributed cognition decided to retire
00:06:08
last summer and I thought oh god I
00:06:11
share the lab with and for all this
00:06:13
time what am I gonna do now and so just
00:06:17
then Scott shows up and don shows up
00:06:19
and we were starting a new lab so it's
00:06:21
really great and also really happy that
00:06:25
that at is following it dawns footsteps
00:06:28
don is probably one of the most
00:06:31
successful failures at retirement that
00:06:34
ever happened he's retired from many
00:06:37
places now and now it's back I'm
00:06:40
retired at UCSD and add only lasted the
00:06:44
summer and now he's back with this
00:06:47
again so I'm I'm really I'm really
00:06:50
happy for that. So I was trying to
00:06:55
think a lot about this well yeah really
00:06:57
procrastinating last night making this
00:06:59
list of names of all the folks and
00:07:03
fearful that I would be people out
00:07:05
which I have but I wanna do three
00:07:08
things I wanna sort of share a little
00:07:10
bit what my excitement is about the
00:07:14
field and and why I think it's such a
00:07:17
crucially important time for eight CI
00:07:22
it was one thing in the early days one
00:07:24
each of us at what little mac on our
00:07:26
desks unconnected do anything else. Uh
00:07:29
and try to design systems and it was it
00:07:31
was hard enough to do that but now
00:07:34
we're faced with a very different world
00:07:36
and were crucially part of just every
00:07:40
system that's being built so I'll talk
00:07:43
a little bit about that maybe a little
00:07:44
bit too much but but I think I'm
00:07:46
envious of all the young folks here
00:07:49
coming into feel like exactly the right
00:07:51
time well if I could've chosen went
00:07:54
into the field it would be now and and
00:07:58
I'm actually tell you a little brief
00:08:00
story about my path and and and some of
00:08:03
the lessons I've learned it's been it's
00:08:06
been quite a pleasure to sort of
00:08:08
reflect back sort of a thirty or more
00:08:13
years of working in the field. And then
00:08:16
I spent a little bit of time talking
00:08:18
about challenges and opportunities and
00:08:22
try to throw in a little advice along
00:08:23
the way I think it's really all about
00:08:26
stories I've always been very taken by
00:08:30
the Nobel laureate peer metal war when
00:08:33
he talks about science as says so it's
00:08:36
begins as a story about a possible
00:08:38
world. Uh a story which we invent and
00:08:41
criticise and modify as we go along so
00:08:45
that ends up being is nearly as sick as
00:08:48
we can make it the story about real
00:08:50
life. Uh and I think that that's what
00:08:52
science is really about but I think we
00:08:56
have in our discipline we have an
00:08:58
incredible advantage because we can we
00:09:02
can sort of make at least some of our
00:09:03
stories true or writing code in
00:09:06
building systems and and I think that
00:09:10
that's a tremendous advantage that we
00:09:12
have and so anyway so exciting times I
00:09:20
think probably the most positive thing
00:09:23
that I see in the field is that we're
00:09:26
moving away I think for way too long
00:09:29
we've conceived of thinking is
00:09:31
something that happens exclusively in
00:09:33
the green in the head and when I start
00:09:37
off actually thought about thinking
00:09:39
very much that way but you know I think
00:09:43
we all sort of realise now that
00:09:46
thinking happiness in the world. Uh as
00:09:48
well as in the hand we think with with
00:09:50
things with our bodies with marks on
00:09:54
paper in the say and and with other
00:09:57
people and that thinking is sort of a
00:10:01
distributed social activity other
00:10:04
exploits these wonderful facilities of
00:10:06
language all the kinds of
00:10:09
representational media that we have and
00:10:13
are embodied interaction with the world
00:10:15
with other people and and today we we
00:10:19
sort of increasingly think with
00:10:21
computers and and I think that that's
00:10:25
one of the reasons that this is such an
00:10:27
important field because we want we want
00:10:32
to build kinds of systems that are
00:10:34
really allows the think thoughts that
00:10:36
we couldn't think without computers and
00:10:39
other people and and but it's it's not
00:10:43
the computer we grew up with this
00:10:45
surely not the computer I kind of thing
00:10:49
when I first got this UCSD there was a
00:10:52
big brothers sixty five hundred in a
00:10:56
room with a lot of air conditioning and
00:10:59
glass window so you can sort of looking
00:11:02
at that and you could put you know you
00:11:04
give your cards to someone to read into
00:11:07
the computer and a day later you come
00:11:09
and get a a printout of that little
00:11:11
syntax error you may in your program
00:11:14
that monolithic computer that we grew
00:11:17
up is is really coming apart and it's
00:11:20
being reassembled and myriad you forms
00:11:23
and the boundary sort of between people
00:11:27
and computers in between physical and
00:11:30
digital world is becoming increasingly
00:11:33
more and permeable things are moving
00:11:36
back and forth well along that thing we
00:11:40
talked today a a being on the threshold
00:11:42
of what people call the age of the
00:11:46
internet of things and I think that's
00:11:49
not quite right kind of thing it's
00:11:53
really more than things and people
00:11:55
always have to be in the picture. Uh so
00:11:59
I think we're really moving towards I
00:12:01
in internet everyone and everything
00:12:04
kind of thing and that's again one of
00:12:07
the reasons that this field is so
00:12:09
crucially important every side of sort
00:12:13
of then thinking about why movement is
00:12:16
happening. So quickly apart Abbott is
00:12:18
more long part of it is all the sensors
00:12:20
and and and part of it is you know the
00:12:23
world getting network where I can we
00:12:25
connected everywhere but but more and
00:12:28
more I'm sort of thinking it's also
00:12:29
this changing cost structure. We can do
00:12:32
things that we would never ever even
00:12:35
conceived up for where the the the
00:12:37
price of of using a thousand computers
00:12:42
is the same as using one computer for
00:12:45
the same period of time kind of thing
00:12:48
and that's that's radically changing
00:12:50
things in what we can do in the way
00:12:53
that we can we can do it. Uh
00:12:56
fundamentally changing and and
00:13:01
interaction is is not sort of movie but
00:13:05
it's it's rushing towards away from the
00:13:09
desktop and and I recently helped
00:13:14
organise actual conference in Germany
00:13:18
on interaction be on the desktop and
00:13:20
and just realises this drawing
00:13:23
zeitgeist of the importance of moving
00:13:25
beyond the desktop is shared
00:13:28
internationally with with folks. But
00:13:31
the reason I think that it's an
00:13:33
exciting times is computation is really
00:13:37
ling Kay have this wonderful expression
00:13:41
the computation was the first that'd be
00:13:44
a medium where we can build other
00:13:46
medium. And for me the thing that's
00:13:49
appealing and exciting and attractive
00:13:51
about computation is is is incredibly
00:13:54
plastic media that we can you know
00:13:57
fashion representations you kinda
00:14:00
action. And in ways of communicating
00:14:03
that we all experience everyday. But we
00:14:06
can also memory well older me. So we
00:14:09
can make I had to look like the look
00:14:11
and act like about change pages like
00:14:14
about but we can also create these
00:14:17
really wonderful new kinds of
00:14:19
representations that are sort of
00:14:22
unconstrained by the constraints of
00:14:24
paper and other media and you know the
00:14:29
models I mean you know I can now was I
00:14:35
stopped in Hawaii on the way here to go
00:14:37
surfing and the models are such that I
00:14:41
could know what the server is going to
00:14:43
be like on the north shore of a while
00:14:45
in San diego. And really deciding
00:14:50
whether I should get on the plane kind
00:14:52
of thing you know we can build these
00:14:56
wonderful virtual worlds which are you
00:14:58
know where which are now long past the
00:15:01
sort of metaphorical desktop grew up on
00:15:04
the alto kinds of things suddenly these
00:15:07
world can be you know how all the
00:15:10
wonderful techniques of film and
00:15:12
animation kinds of of things. And and I
00:15:16
think we're you know and probably see
00:15:19
some of this of the conference sort of
00:15:21
just going to our route these areas of
00:15:24
virtual reality and augmented reality
00:15:26
kinds of technologies are just coming
00:15:28
to fruition and they're going to be I
00:15:32
think quite major for a few and and we
00:15:36
can also combined the real and the
00:15:38
virtual and things like computer
00:15:42
augmented surgery. So you know here's
00:15:45
eventually like surgery system the the
00:15:51
surgeon is sitting at the console
00:15:53
manipulating kinds of things you really
00:15:55
dealing with a virtual which impacts
00:15:59
the robot the this world allows him to
00:16:03
take sort of tremors and things out of
00:16:07
his hands it allows him to see for
00:16:10
example doing a might roll valve
00:16:13
replacement in your heart which used to
00:16:14
be this open really crack open the
00:16:17
whole channels. Now it's done with
00:16:19
these are just copy kind of body things
00:16:22
that he can be in their time knots
00:16:24
inside of someone's heart of that can
00:16:27
be in large you could be aided in doing
00:16:30
that that kind of virtual world. Uh as
00:16:32
exciting what is connected with the
00:16:35
real world and there's like you know
00:16:37
some problems with the line stuff
00:16:39
there's there's no in principle reason
00:16:42
that the surgeon with special skills
00:16:45
could be operating on someone on the
00:16:47
other side of the work the thing I mean
00:16:50
that's an exciting kind of thing
00:16:52
exciting times but for most of us lots
00:16:56
of excitement is come with the movement
00:16:58
onto the off the desktop and and into
00:17:01
smart phones and here's Steve jobs in
00:17:04
reducing the the I phone. And
00:17:06
increasingly we carry many of these you
00:17:09
know I sort of you know use like travel
00:17:11
with the phone and then I had a laptop
00:17:14
and kind of thing and some of them we
00:17:17
don't even think about as being
00:17:18
computers although they are incredibly
00:17:21
powerful computers and part of the
00:17:24
power of the connected to computers and
00:17:27
sensors and people so I can sit here
00:17:31
chat with my wife before giving a talk
00:17:35
here in Korea and you know just all
00:17:39
over activities are increasingly
00:17:41
mediated by computers and that's
00:17:44
changing just almost every aspect of
00:17:47
more professional and personal lives
00:17:50
and I think often for good. Um but also
00:17:53
often for sale my students Kevin lee
00:17:56
came up with this great that picture
00:17:58
number of years ago which I can't stop
00:18:01
using and and I think this is for you
00:18:06
know I think these young folks are
00:18:08
missing something and I had shown this
00:18:11
a couple of times when I realised. They
00:18:14
could be texting each other so it could
00:18:18
be even worse than I think and and
00:18:23
we're saying you know this movement off
00:18:25
the desktop and into the cars in fact I
00:18:27
was surprised to see there was voters
00:18:30
show here that I would been interesting
00:18:32
coming to just before just before I've
00:18:35
seen in a Mercedes have this a
00:18:38
prototype car states recently we know
00:18:42
about the goals of driving or kinds of
00:18:45
things. And we become interested in the
00:18:49
lab and communication between cars and
00:18:53
pedestrians how was the car going to
00:18:56
let you know when you're crossing a
00:18:58
street at that see you know how are you
00:19:02
gonna communicate well with that kind
00:19:05
of thing there's lots of really
00:19:07
challenging issues as we get or semi
00:19:09
autonomous and timeless cars and and
00:19:13
and most of them are you know a CI
00:19:16
really how to relate that impresses you
00:19:19
this great video play some of the tools
00:19:22
which just too good to give you a feel
00:19:26
for for you know if you're looking it's
00:19:29
a an intersection in your city MMM in a
00:20:08
I oh I say SMCI someone. MMMFC but the
00:20:55
exciting times also brought why are all
00:20:57
of these kinds of new device. We've
00:20:59
done a lot of work with another digital
00:21:01
pens and suddenly you know hands can
00:21:04
report you're writing report what's
00:21:07
being said while you're writing put on
00:21:10
a glass and you can record what you're
00:21:13
seeing and tell you about what you see
00:21:15
there's like field cameras come you can
00:21:20
take a picture only today to refocus
00:21:25
after you take a picture there's you
00:21:29
know cars are just becoming computing
00:21:32
platforms all kinds of devices heads up
00:21:35
displays all of these kinds of things
00:21:37
all all needing a help of for this one
00:21:41
today everyone's excited about watch
00:21:44
out to be available to try you think a
00:21:49
million were ordered on the first day
00:21:52
kind of thing I think a million more at
00:21:56
least has been ordered sense then we're
00:21:59
excited about this where ability have a
00:22:04
watch is one of the biggest get starter
00:22:07
kinds of things over ten million
00:22:09
dollars people that but it's I don't
00:22:21
that's good so the couple walks to log
00:22:31
just ah sorry. I'm unless there is not
00:22:34
set so so and I thinking about this and
00:22:38
you know one of the first watches not
00:22:41
not a watch person. But I was excited
00:22:43
when this watch came out and I don't
00:22:47
know if if you miss are old enough to
00:22:50
sort of remember this watch that was a
00:22:54
commodore watch from the seventies some
00:22:59
of you might still remember the
00:23:02
commodore sixty four microcomputer
00:23:05
utterance of the commodore was watch
00:23:07
company before it became a computing
00:23:10
company. That's sort of interesting to
00:23:12
see companies may be starting to go the
00:23:14
other direction kind of thing one of my
00:23:18
students brought me this wonderful
00:23:20
cartoon I need to share with you. Uh
00:23:24
about this reach glorious period when
00:23:28
we were or risks were free kind of
00:23:31
thing and and I think both and thinking
00:23:35
of the commodore really computer that I
00:23:39
guess. I think one so have gonna
00:23:41
through in my first piece of
00:23:42
unsolicited advice and I think they're
00:23:46
field needs to spend more time knowing
00:23:49
the history and the literature of er
00:23:53
feel I think ideas have histories and
00:23:57
can really only be understood fully in
00:24:00
the context of those histories and
00:24:03
that's a that's a message that bill
00:24:05
Buxton has been delivering to us for
00:24:09
quite some time and and I think both he
00:24:13
and terry or not only sources of very
00:24:16
council and wisdom but also provided
00:24:19
really valuable online resources I
00:24:21
think bill's thirty years of examples
00:24:26
of interaction techniques and
00:24:27
interaction devices that is incredibly
00:24:30
valuable tool to look at and and I
00:24:33
think we all owe terry an incredible
00:24:36
amount of praise for starting the the
00:24:42
the people computer computing and and
00:24:44
design seminars at Stanford and now
00:24:46
there's is personally twenty five years
00:24:49
of very influential talks so one
00:24:52
personal biases to go back and and and
00:24:55
look a little bit of the history of the
00:24:57
ideas now the reason for exciting times
00:25:00
is I think we're in the midst of a so
00:25:03
the data revolution not just the big
00:25:07
data that that people are talking about
00:25:09
what sort of more thick data data of
00:25:12
people really interacting us and
00:25:15
expensive digital recording devices and
00:25:18
video of revolutionise a behavioural
00:25:21
science so suddenly we can we can look
00:25:24
at things in ways that we never could
00:25:26
before we can capture this is my
00:25:29
colleague and Hutchins when he had more
00:25:33
here and when he was young and this was
00:25:37
when he was doing his PHD of fieldwork
00:25:39
it's a cognitive anthropologist and use
00:25:42
to his PH if you work the new guinea
00:25:44
looking at land litigation when someone
00:25:48
dies and how things are are passed down
00:25:50
and people get together in the village
00:25:53
and the square and and discuss this
00:25:55
kind of thing. And and wanted to
00:25:58
understand that and line this new to
00:26:00
the language well he's new to the
00:26:02
culture but this you could almost see
00:26:05
it here what he's he's got this I think
00:26:07
around his neck and it's just great big
00:26:09
square thing. It's a battery powered
00:26:11
tape recorder. So for the first time
00:26:14
you could take that into the field and
00:26:17
record what people really said and if
00:26:19
you've ever listen to what people
00:26:21
really said say you know even yourself
00:26:24
on some very startling kind of
00:26:27
experience because it's very different
00:26:29
than what we hear kind of thing and and
00:26:33
so and was able to study that and I
00:26:34
think we're we're sort of at a period
00:26:36
were were we're able to do that in ways
00:26:40
that we're just unimaginable just a few
00:26:43
years ago I'm very taken by the work of
00:26:46
of shock and candy good one we're Ucla
00:26:50
currently and and they looked at all
00:26:54
kinds of of of really interesting
00:26:56
activities so like children playing
00:26:58
hopscotch all the complex things that
00:27:01
are going on in the way people like
00:27:03
could use their bodies and their facial
00:27:05
expressions and things to to
00:27:08
communicate their intentions and that
00:27:11
the elderly gentleman in the bottom
00:27:13
picture is is trucks father and he had
00:27:18
a stroke in left hemisphere and I can
00:27:23
only say three words after the stroke
00:27:27
it's a yes no and the word and and that
00:27:33
was his vocabulary. And what shot was
00:27:36
able to document and begin to
00:27:38
understand this how is father was still
00:27:41
and incredibly active participant in
00:27:44
conversation was able to get his
00:27:46
intentions over to people was able to
00:27:48
accomplish what you wanted to do by
00:27:52
facial expressions by movement like
00:27:54
pointing by all of this kind of thing.
00:27:57
So I think where the you know a really
00:27:59
interesting time of for looking at what
00:28:03
people really do and I I think that we
00:28:05
don't do enough of that I I think we we
00:28:09
don't know enough about even what we do
00:28:12
kind of things is and we have an
00:28:15
opportunity now to do that. Okay so
00:28:17
talk a little bit about sort of my
00:28:20
research staff and sort of my story
00:28:26
kind of thing say young kid I was about
00:28:31
as nerdy as they get a and you know
00:28:35
fortunately I was an only child so I
00:28:37
could be even dirtier kind of thing.
00:28:41
And I got involved in now. Um act was
00:28:46
sort of like thing and I particularly
00:28:49
got involved in studying mathematicians
00:28:53
lives a and Reading novels and not
00:28:56
novels and stories about meditation one
00:28:59
famous one I remember ah who the gods
00:29:02
loved which was written by two
00:29:04
mathematicians about a rusty gal wow
00:29:08
who invented group theory who died in
00:29:11
the dual as a young man who couldn't
00:29:14
get in the cold polytechnic because you
00:29:18
were supposed to solve the classic
00:29:21
problems in the classic ways and gal a
00:29:24
came up with the new proof every time
00:29:27
and was not allowed the and and so
00:29:31
mathematicians we really interesting to
00:29:33
me because they were they were weird
00:29:35
and they we're doing cool things and I
00:29:37
was I was totally taken by things like
00:29:41
Anders diagonal proof a word first I
00:29:44
can sort of see that you know that I
00:29:47
could understand something about
00:29:49
uncountable infinite kinds of things
00:29:52
and so I thought and I'm gonna be a
00:29:54
mathematician and I will I scored high
00:29:56
mismatch Olympia and everyone pay maybe
00:29:58
with the map brush and so off to
00:30:02
college and and I I unfortunately got
00:30:09
bitten by the what was then called a
00:30:12
four colour conjecture and if you don't
00:30:15
know this this thing it's the
00:30:18
conjecture was that it only takes for
00:30:21
colours to colour any planar map. So
00:30:24
that the process any border it's
00:30:25
different colours. And and this is a a
00:30:29
longstanding problem which was
00:30:32
sometimes called a four colour disease.
00:30:35
Because it it'd been passed down from
00:30:37
one generation to the next where the
00:30:40
father or mother worked on the problem
00:30:42
for a period of time and then the the
00:30:44
their the son or daughter worked on the
00:30:48
problem. So I was I was you know it's
00:30:52
very easy to map to send the graph
00:30:55
theoretic kinds of stuff and I got
00:30:57
really interested in graph theory and I
00:31:00
became obsessed I was I was totally
00:31:02
convinced that there was this monster
00:31:05
counter example that that's the reason
00:31:07
that no one had ever proven that's and
00:31:10
I was going to find it and the
00:31:14
unfortunately a few years later it was
00:31:18
proven to be true but what I really
00:31:21
found was computers and programming and
00:31:24
and I used to massive amounts of of
00:31:26
time on the computing systems looking
00:31:29
for counter examples and needlessly. Um
00:31:34
and then I came across a block by
00:31:38
George Paul you how to solve it and and
00:31:42
it really change my orientation
00:31:45
suddenly. I became much more interested
00:31:49
in how people do mathematics and how
00:31:52
you come up with conjectures in the
00:31:55
whole cognitive side of things. Uh and
00:32:00
and you know and it it sort of resulted
00:32:04
in and we wanted to go to became a
00:32:06
double major in math and psychology.
00:32:09
And almost no one in psychology wanted
00:32:11
to talk about the things I wanted to
00:32:13
talk about in those days I I went to
00:32:17
graduate school that married is
00:32:19
undergrad to a woman who I wanna go to
00:32:23
med school shoes all we admitted to to
00:32:25
one medical school in fact it was the
00:32:28
sort of the last year that it was
00:32:30
really difficult for women to get into
00:32:32
med school there were there were four
00:32:33
women in her med school class kind of
00:32:37
thing and so we went a place which
00:32:39
actually was accepted I didn't think it
00:32:41
was a good place for me because no one
00:32:43
was interested in the kinds of things I
00:32:44
was interested in but it turned out
00:32:47
that that was probably the best
00:32:48
possible place to go because I was sort
00:32:51
of on my own and I had to to sort of
00:32:55
make my own sort of graduate experience
00:32:59
and then I was you know very fortunate
00:33:01
to be accepted a postdoc with absolute
00:33:04
rupees it's standard in the institute
00:33:06
for mathematical studies and social
00:33:08
sciences but that wasn't really the
00:33:11
right place for me either I thought it
00:33:13
was but I started hanging out of the I
00:33:16
lab I and xerox park and that sort of
00:33:20
totally change my life because I found
00:33:22
the alto what I found list machines
00:33:24
which was love at first sight and and
00:33:30
you know and I at the time I didn't I
00:33:33
didn't know what I was going to do but
00:33:34
I knew what was going to do with
00:33:36
machines like this. Uh that they were
00:33:38
just so interesting in the researcher
00:33:40
such possibilities representational
00:33:44
like with them twenty so the phases of
00:33:48
my life these are these are not all
00:33:50
decades the two in the middle or half
00:33:53
decades kind of thing in fact a unit of
00:33:56
time I think about or try to think
00:33:58
about is decades now kind of of thing
00:34:02
and I spent the first decade at UCSD
00:34:06
and also because my my wife by this
00:34:09
time was resident surgery I had plenty
00:34:13
of time so I had two jobs and still
00:34:15
felt like I was loving and and I we
00:34:20
built some systems I'll tell you a
00:34:22
little bit about them in a minute in
00:34:25
that I left to go to to head up the
00:34:29
human interface lab at MCC we've got
00:34:31
some more systems and and did some
00:34:33
other work. And then I left there to go
00:34:35
to start a group at Bellcore and left
00:34:39
their to feature of the computer
00:34:42
science that you wish to Mexico and
00:34:44
then finally boomerang back to UCSD so
00:34:48
the early research at UCSD again it was
00:34:52
the two body problem the really led me
00:34:54
to UCSDC only only place my wife was
00:34:58
accepted into resonance Ian surgery and
00:35:02
but it turned out to be a really really
00:35:05
amazing place in those days I mean it
00:35:07
was I was so lucky and so fortunate not
00:35:13
the least of which I started to learn
00:35:15
to surf which was also incredibly
00:35:18
important a vent but it was a
00:35:22
intellectually it was the most
00:35:24
productive interesting environment I
00:35:27
think that I've ever been that was as
00:35:28
great collaboration between don Norman
00:35:31
and Dave promo for and and then J
00:35:34
Mcclellan joint and we were sort of
00:35:37
transitioning to form the first
00:35:40
cognitive science laboratory there it
00:35:44
was really the beginnings of cognitive
00:35:46
science the about time I got there was
00:35:50
the first cognitive science conference
00:35:52
which was held at UCST it was the first
00:35:56
mental models conference was held there
00:36:00
at UCSP we had this wonderful Sloan
00:36:05
foundation postdoc program where we
00:36:11
were able to attract this whole host of
00:36:14
just incredibly enlightened postdoc
00:36:19
some aimed and Hutchins was one of the
00:36:22
just handing was another one and it
00:36:25
went on and on we had this amazing set
00:36:27
of graduate students every cell and
00:36:30
build behaviour bob clash go Jonathan
00:36:34
routine Michael Jordan Mike knows your
00:36:36
and on a on it was the beginnings of
00:36:40
this whole connection is movement and a
00:36:43
very different view of cognition the
00:36:47
the original parallel distributed
00:36:49
processing books we're being written
00:36:51
and and Francis crick the Nobel
00:36:55
laureate used to sit in our meetings
00:36:58
and berate everyone about how little
00:37:01
they knew about the brain and how
00:37:03
foolish other models for but it was
00:37:07
just an intellectually incredibly
00:37:09
stimulating kind of environment. And
00:37:12
the same time I was able to maintain
00:37:15
ties to park and what was going on
00:37:18
there and and be a consultant and I
00:37:20
convinced the navy to give us a bunch
00:37:22
of money to buy these really expensive
00:37:24
list machines and and we build a number
00:37:28
of systems well one of them was
00:37:31
navigational training system and and
00:37:35
what we're really interested in is I
00:37:37
think in the the slider so the the
00:37:39
buzzword of the slogan of the day was
00:37:42
conceptual vitality rather than
00:37:43
physical vitality. So it was a time of
00:37:46
mental models there was a time of
00:37:48
concern for how people thought about
00:37:50
these things. And could we build
00:37:52
systems that aided the the the building
00:37:55
of those models but not the models that
00:37:58
you get in physics textbooks the models
00:38:00
use you think with and so we build a
00:38:03
series of the of navigational training
00:38:06
systems and and the some of them on a
00:38:11
little computer called a Tara
00:38:15
microcomputer reasonable Ellis about
00:38:17
living computer built in Arizona that
00:38:21
was one of the first that that really
00:38:23
inexpensive displays we also I was
00:38:26
involved what can balls and we bought
00:38:28
you see a C Pascal this really
00:38:29
interesting us no system we had some of
00:38:34
the first per computers which you
00:38:37
probably never heard of the other done
00:38:39
by a spinoff of CMU and it informally
00:38:44
called Pascal toes because they were
00:38:46
like altos but where and Pascal and
00:38:50
these are pretty successful our train
00:38:53
systems on the little micro with on
00:38:56
every ship in the navy and I had we had
00:39:00
a I had you know she should taught
00:39:04
these kind of navigational things for
00:39:06
fifteen or twenty years have lightbulb
00:39:09
experiences "'cause" they had taught
00:39:11
these procedures by rote. And suddenly
00:39:14
they could interpret lots of things we
00:39:16
did was mapping between
00:39:17
representational system so kind of
00:39:19
relative system that you get on a radar
00:39:22
with the geographic kind of map based
00:39:23
system kind of thing the per thing we
00:39:27
built a bunch of tutoring systems on
00:39:31
and I installed one of these on a on a
00:39:35
new nuclear carrier that was just
00:39:38
getting commissioned other cars vents
00:39:40
and with I still remember captain
00:39:43
martin who was the commissioning
00:39:44
officer for that ship it was very
00:39:49
strange to install it on that so we we
00:39:52
don't use systems that we're really
00:39:55
primarily cognitively motivated
00:39:57
motivated by trying to to help people
00:40:00
build the kind of models they needed
00:40:02
treason about things. Uh we also build
00:40:05
a a system called steamer which so do
00:40:10
we whites when when the guys who worked
00:40:11
with this this is on a you might use
00:40:14
this machine a symbolic celeste machine
00:40:16
the the display on the the left is sort
00:40:21
of a dynamic graphical view of this
00:40:24
complex dynamic system there are lots
00:40:27
of really good ideas we could see the
00:40:29
it show the flow in the system you can
00:40:31
see the causal topology you could could
00:40:34
you would many different levels we
00:40:37
built some of the the the first things
00:40:40
of qualitative graphical components so
00:40:44
lots of responses and systems like this
00:40:47
respond to rates of change to to the
00:40:48
derivatives of things so we built these
00:40:51
derivative icons were just arrows
00:40:53
support it up or down and the size of
00:40:55
the all said how fast they were
00:40:56
changing and you can step through the
00:40:58
simulation. And have this graphical
00:41:00
instantiation of what seemed very close
00:41:03
to what experts would discuss in the
00:41:07
way they would describe the system so
00:41:10
we we we have a lot of fun and and
00:41:14
about the time we we sort of step back
00:41:17
from this which I think is a a really
00:41:19
good thing and and add and don and I
00:41:22
wrote this regional direct
00:41:24
manipulations of paper we we were
00:41:27
responding a little bit too been
00:41:29
Schneider men's direct manipulation
00:41:31
paper which was very influential but we
00:41:35
saw it more is a checklist kind of fury
00:41:38
and we wanted a much more cognitive
00:41:41
theory of what was going on and so we
00:41:44
tried to to do that and and first
00:41:47
instance so what was in this user
00:41:49
centred system design but which was
00:41:52
named that because we could get the
00:41:54
UCSCD acronym and and I was a very I
00:41:59
think influential book and the whole
00:42:02
movement to sort of user centred design
00:42:04
I think was influenced by so the and
00:42:07
the lesson sort of learn from that face
00:42:10
is you know that I'm just as more and
00:42:17
more convinced that computation assist
00:42:19
plastic representational system and and
00:42:24
and the work there is a convinced me
00:42:28
even more I was sort of also very taken
00:42:31
by how interesting and fun it was to
00:42:36
build systems and then transition them
00:42:38
into use and and that that was both you
00:42:43
know incredibly challenging but but but
00:42:47
also I think it it leads to better
00:42:49
science if you're able to do that I
00:42:52
mean you have to you have to deal with
00:42:54
issues that you might avoid otherwise
00:42:57
and and the and I think it continues is
00:43:04
is sort of a this kind of relationship
00:43:09
do you know that got introduced to me
00:43:11
by ad Hutchins were and spend a lot of
00:43:14
time in the balance of navy ships
00:43:18
looking and observing how people really
00:43:21
operate "'em" And just how valuable
00:43:25
that was two systems builders kind of
00:43:28
thing it's really one of the big
00:43:30
lessons of my career and yeah I thing
00:43:36
we built this this graphics editor one
00:43:39
of the first object oriented graphics
00:43:41
editors for building interfaces
00:43:45
interactive dynamic interfaces. And I
00:43:49
was incredibly struck by how powerful
00:43:53
building tools to build interfaces all
00:43:55
were and and how you could leverage
00:43:58
that in fact at the time we've thought
00:44:03
for a momentarily of doing a startup
00:44:06
before that kind of thing was really
00:44:08
problem and and and the scary thing is
00:44:12
the company was willing to give us a
00:44:14
bunch of money to do that in the idea
00:44:18
was to use this graphics editor to to
00:44:21
build instances of these kinds of
00:44:23
interfaces for all kinds of things. You
00:44:25
know nuclear power other kinds of
00:44:27
complex dynamic systems but we we
00:44:30
quickly realise that we would move from
00:44:32
a research career to a different career
00:44:34
and and ran quickly in the other
00:44:37
directors the the other thing was see
00:44:41
the importance of of oh I'm sort of
00:44:47
really abstracting out the cognitive
00:44:51
design principles I think this direct
00:44:53
manipulation paper was something that
00:44:55
so that taught me that you need to do
00:44:58
this kind of building kind of stuff but
00:45:00
then you have to step back. And really
00:45:02
abstract out what you've learned from
00:45:04
but and and then the other thing that
00:45:07
was flight reflecting back on this that
00:45:10
we were just so incredibly naive about
00:45:13
the scale of the project that we took
00:45:16
on I mean this you know huge you know
00:45:19
the first thing we're gonna build a
00:45:20
graphical dynamic interface to is a
00:45:22
team propulsion plant you know just
00:45:25
this very complex kind of thing. And
00:45:28
sometimes and I think in this case it
00:45:31
did that lead you to do more important
00:45:35
things and had we started much smaller
00:45:37
so anyway so I left UCSD about this
00:45:41
time. Um due to listening to the advice
00:45:45
of don norman. Um not always the best
00:45:48
thing to do a and I had been consulting
00:45:52
at MCC for a while and it was a very
00:45:55
interesting is research thing really
00:45:58
wanting to do long range research a a
00:46:00
lot of money a lot of good people and
00:46:04
and as so and they offered me the job
00:46:09
would being director this really large
00:46:11
three forty five PHD lab which I didn't
00:46:16
wanna do it anyway and and so but I
00:46:20
really wanted to keep consulting
00:46:21
"'cause" it was really interesting that
00:46:23
lotta list machines and they were doing
00:46:24
interesting things lasted on a well
00:46:27
what should I do how can I say no no
00:46:29
way that to keep me consulting swell
00:46:32
simple gem just ask for a lot of stuff
00:46:37
ah and and that sounded like recent. So
00:46:41
I asked for all these sort of pretty
00:46:43
outlandish stuff in my view. And they
00:46:45
said sure and and so I was like I
00:46:50
didn't know what to do and and it
00:46:51
seemed like an opportunity then that it
00:46:53
didn't seem like before. And so I went
00:46:56
off to MCC and and we started building
00:47:00
sort of building on what we've done
00:47:04
with this graphics editor and but
00:47:07
extending it to be tools to build
00:47:09
multimodal interfaces long before many
00:47:12
people were ever thinking about
00:47:13
multimodal interfaces and so we're
00:47:17
building tools a two construction of in
00:47:20
a natural language interfaces and
00:47:22
graphics and gesture and intelligent
00:47:25
systems that we give advice and and it
00:47:30
was sort of you know that I'm that he
00:47:32
did a I and we were not unnecessarily
00:47:35
well convinced that that was the right
00:47:37
way to go but we really an average of
00:47:39
representational languages and so we
00:47:43
get a lot of interesting things of
00:47:45
representing the whole system in this
00:47:47
representational language so only
00:47:49
interfaces but the tools themselves had
00:47:51
representation and and the main idea
00:47:55
that was sort of driving me and driving
00:47:57
the development was an idea that I
00:48:00
think is still has a lot of of credence
00:48:03
and the idea was that somehow when
00:48:06
you're building systems you have to get
00:48:08
from this bit shuffling lateral level
00:48:09
of computing two levels that people
00:48:12
really care about and most of the time
00:48:15
we think about doing that and maybe one
00:48:17
step you know we build an editor or we
00:48:20
build one kind of development
00:48:21
environment and stuff. And I think the
00:48:23
normal idea we had was that it's really
00:48:26
going to take a series of tools. And
00:48:30
that the and that the important thing
00:48:32
was that those tools were all connected
00:48:34
to each other. And sort of live. So
00:48:37
that you could go in and make the
00:48:39
changes and appropriate level for the
00:48:43
thing it would be for the whole system
00:48:46
so as you moved up the tool chain you
00:48:48
would empower people with different
00:48:50
levels of expertise and but you we keep
00:48:53
the whole thing tied and I still think
00:48:55
that a really interesting and valuable
00:49:00
idea. And I think we get a well I was
00:49:03
at TMCC is my car broke down as scene
00:49:08
always have cars involved and and I'd
00:49:13
been out to get our Marvin Minsk keys
00:49:15
to come down to this it Dublin at at it
00:49:19
MCC and we would sometimes go out to
00:49:22
dinner like Travis it is mexican place
00:49:26
there and and sort of argue well I mean
00:49:29
both Marvin and don't love to argue and
00:49:32
and and and you know is bringing Marvin
00:49:37
back one late night from there and and
00:49:40
we broke down out on this you know just
00:49:42
the most deserted road you can ever
00:49:44
imagine only Texas can you so the
00:49:47
imagine these kind of really deserted
00:49:49
roads and anyway long story I I ended
00:49:52
up in this auto repair shop yeah the
00:49:55
next day getting my car fixed this all
00:49:58
Portia I folks in the lab had give me
00:50:01
this bumper sticker my other cars a
00:50:04
piece of junk to and which I didn't
00:50:08
appreciate and I was there are like a
00:50:12
looking at and the thing that
00:50:13
fascinated me I just became enraptured
00:50:16
by these manuals and the thing that was
00:50:20
interesting about them is mechanics
00:50:23
said dirty hands. And when they when
00:50:26
they deal with these manuals the the
00:50:28
places that they deal with the most get
00:50:30
the dirtiest. And and and I I became
00:50:33
really taken by that notion that people
00:50:36
do things. And and it has these kind of
00:50:39
representational consequences and then
00:50:42
you sort of start to see it everywhere
00:50:44
you see past for more people just walk
00:50:47
and it leaves these kind of things that
00:50:49
you look start looking at your
00:50:50
paperback books and they they sort of
00:50:52
open to the place you last read if you
00:50:54
sort of break the spy and and so we we
00:50:58
we did a lot of work we got sort of
00:51:00
taken by this notion we do a lot of
00:51:01
work on what we called in the day and
00:51:04
it where read where kind of thing and
00:51:08
and or more generally sort of history
00:51:11
enrich digital objects. And we've got
00:51:13
some interesting I think well ways of
00:51:16
visualising that we we build is what we
00:51:19
call attribute map scrollbars so that
00:51:22
we could represent that history of who
00:51:25
edited maybe how long they took and
00:51:27
today the co editing of stuff whether
00:51:29
it was you know as long before the days
00:51:32
of you know track changes in Microsoft
00:51:34
word we modify the old version of Emacs
00:51:37
to to do all of this kind of recording
00:51:40
and and was really I think really nice
00:51:42
to make it in the scroll bar because
00:51:45
you used real estate that was already
00:51:47
there and it you just touch the place
00:51:50
you wanted to look at and you were
00:51:51
taking their so it was really good and
00:51:54
we we got really involved with that so
00:51:58
we had this idea of menu where so you
00:52:01
know the history of your use of various
00:52:03
features in the system should should
00:52:06
show their we had buffer where so how
00:52:10
much time you spend on buffers well we
00:52:12
had are read where in our email readers
00:52:16
so you could tell us how long so so one
00:52:19
of things we played with was this kind
00:52:21
of messages that might go to everyone
00:52:23
in a research group then your way
00:52:25
you're coming back when it be nice to
00:52:27
see how long an average people in your
00:52:30
group spent Reading those so you might
00:52:32
prioritise things in terms of the
00:52:34
things that people spend a lot of time
00:52:36
really today we got really an hammered
00:52:39
by the notion and and I still think
00:52:42
this sort of general notion of of in a
00:52:47
computational medium where you can
00:52:49
capture kind of activity and and then
00:52:52
re represent that in ways that informed
00:52:54
tasks and do things so you're not
00:52:56
asking people to do anything so that
00:53:00
the lessons learned is first lesson I
00:53:02
learned MCC was I had no skill and be
00:53:06
able to manage forty personal out my my
00:53:11
whole management philosophy up at that
00:53:13
time was jump in the car go to lunch
00:53:16
and talk about things. Uh and and you
00:53:19
know walk around. And chat with people.
00:53:22
And suddenly a sort of realise that no
00:53:24
that doesn't scale and that was a real
00:53:27
a lesson learned a lot for me again I
00:53:31
was sort of the lesson was that the
00:53:33
building a system is really a great way
00:53:36
to integrate research. And integrate
00:53:38
people's activity and keep you on on
00:53:41
course and it focused things in a way
00:53:44
that I think most of the the successful
00:53:47
research projects I've been involved in
00:53:49
the bin tightly tied to building
00:53:52
something building the system. And
00:53:53
using that as an integrating kind of
00:53:55
thing I think this tool chain idea is
00:53:58
still worthy of a lot of consideration
00:54:01
that we're still trying to jump these
00:54:04
massive gaps in building systems and we
00:54:08
we try to do with the programming
00:54:09
language or we try to do with one tool.
00:54:12
And I and I think in tool we have a
00:54:14
series of tools like that we won't be
00:54:17
able to do it well and this chi idea
00:54:21
about a history enrich digital objects
00:54:23
that that that you just let people do
00:54:26
what they normally do. But but because
00:54:28
there do you know a computational
00:54:30
environment you can you can re
00:54:33
represent that in ways that will inform
00:54:35
I think that's a very very powerful
00:54:37
kind of idea and again I'm further and
00:54:41
further convince that computation just
00:54:43
plastic medium. So I left MCC in austin
00:54:48
Texas reluctantly in some ways you know
00:54:53
in CC was a fairly successful but as we
00:54:56
see all too frequently and long range
00:54:59
research places they seem to slide
00:55:02
backwards into being concerned was
00:55:05
shorter and shorter time frames and and
00:55:09
and the CC the CI lab were starting to
00:55:12
do that and I wasn't interested in that
00:55:15
so I left and I I formed a a group at
00:55:18
Bellcore colour computer graphics and
00:55:21
and interactive media group. And all
00:55:24
speed up a little bit here so it
00:55:27
finished on time it was right at the
00:55:29
time where we were really interested
00:55:32
these dynamic graphical kinds of
00:55:34
interfaces it was really clear to
00:55:36
everyone that inexpensive graphic cards
00:55:40
were on the horizon and suddenly every
00:55:43
PC was going to have the kind of
00:55:45
graphics that we only had on high an
00:55:48
SGI machines a list machines and very
00:55:51
very expensive machines. So many many
00:55:53
people got very involved this was a
00:55:56
time is week card we're starting up
00:55:58
work at park on the information
00:56:00
visualiser well all this concern with
00:56:03
with with with how we can so the
00:56:05
building dynamic interactive three D
00:56:08
kind of interfaces and we we started
00:56:10
building such a system ourselves that
00:56:12
we had originally called art and the
00:56:15
lawyers got involved every time the
00:56:17
lawyers get involved is bad and and
00:56:20
they similar commercial systems call
00:56:22
that you can't call it art and I said
00:56:25
well okay well we'll call it a are
00:56:27
three T and you know "'cause" three D
00:56:30
three T and then we'll tell everyone
00:56:32
that the three silent and and it's
00:56:36
pronounced are kind of thing and and so
00:56:39
we're building this kind of very
00:56:41
interesting system and visualising big
00:56:44
software systems and and kind of things
00:56:48
and then I sort of went up with can
00:56:50
Perl and who's a professor at NYU and
00:56:54
can have just put together this sort of
00:56:57
very kludge together and then L of a
00:57:01
sort of multi scale environment to
00:57:03
tinkle pat and and I thought it was
00:57:08
just exactly on the right course that
00:57:12
is very hard to navigate in three D
00:57:15
easy to get lost a three D leads you to
00:57:18
this kind of imitation kind of you
00:57:21
trying to make it like the real world.
00:57:23
Uh in two D sort of multi scale was
00:57:27
looming seemed like a a really
00:57:29
interesting thing. And and so to us in
00:57:34
the group a lot of the group are still
00:57:35
very interested in three D we did some
00:57:37
stuff to make every polygon sort of
00:57:40
zoom mobile and and the two D kind of
00:57:42
surface but I just hired a a young guy
00:57:46
from and why you a guy doing a robotic
00:57:50
vision name bin peterson. And and we
00:57:54
convince them that he should get away
00:57:56
from that and become an interface guy
00:57:58
and and then I became totally
00:58:02
enraptured by the zoom a kind of of
00:58:05
metaphor. And so we started building
00:58:07
this had plus plus system sort of it in
00:58:11
homage to to to cans system and and
00:58:14
about the time I I decided that the
00:58:17
billboard again was maybe not looking
00:58:19
for had enough and that kind of
00:58:21
research they were funding and was
00:58:23
maybe time to go back to the academic
00:58:26
environment so I moved to the version
00:58:27
Mexico and and then came with me there
00:58:31
and and and then I convinced darpa to
00:58:34
give this a bunch of money and we
00:58:36
started building this possible system
00:58:38
which I think was really influential to
00:58:43
one of the first sort of zooming
00:58:45
interface there are lots of interesting
00:58:47
ideas about it that notions of semantic
00:58:51
zooming. So instead of just doing some
00:58:53
geometric scaling we did semantic kinds
00:58:56
of things you got closer to you might
00:58:57
get a very different oh kind if you we
00:59:00
get a lot of work with the lenses sort
00:59:04
of influence by some of the park work
00:59:06
and but also you know the the idea had
00:59:11
a postdoc came to study would maybe one
00:59:14
of actually cans can students did this
00:59:18
really interesting thing on sort of a
00:59:20
principal way of of composing lenses
00:59:23
and you know one way of thinking about
00:59:25
lenses is doing the retrieval is very
00:59:27
natural way of thinking about it. And
00:59:30
and we also thought about using lenses
00:59:33
or something you reached room. And and
00:59:35
when you reach through them you can do
00:59:37
various things. And being able to move
00:59:39
those ones around with the non dominant
00:59:41
hand and reached through an effect with
00:59:43
the dominant hand was very powerful
00:59:45
uses bins file system and the day so we
00:59:49
consume in I actually yeah I don't have
00:59:53
many good images from those days but
00:59:55
that's the best I could do okay so we
00:59:57
did a lot of of that and I when I moved
01:00:01
to you and am I got this big contract
01:00:04
from darpa and I made some contract to
01:00:05
sort of support my group back at
01:00:07
Bellcore which George furnishing taken
01:00:09
over and and also can and just rooted
01:00:13
NYU and then George move to Michigan in
01:00:16
so we did a contract with with George
01:00:18
there another thing we did was this
01:00:24
beyond being their work and and this we
01:00:28
just hired bulk or as this young
01:00:31
physicist from Stanford us concern that
01:00:34
and and Scott nice really hit it off
01:00:39
and and and and we both like going
01:00:43
around seeing all the stuff going on at
01:00:45
Bellcore today and we just a lot of
01:00:47
work on video teleconferencing I mean
01:00:51
the I think the phone company sort of
01:00:53
foolishly thought that the way the
01:00:55
future was moving gets a you know kind
01:00:59
of thing and and and what we're both
01:01:02
struck by is is is how much the focus
01:01:06
was on imitation you know the the the
01:01:09
the goal was really being there you
01:01:11
know that that you know that the way
01:01:15
the telecommunication problem was going
01:01:16
to be solved by was is was it was
01:01:19
establishing the sense of being there
01:01:22
and and and it was it pretty clear that
01:01:26
you could throw more more money at it
01:01:27
and get higher resolution graphics and
01:01:30
you could full duplex audio and you can
01:01:32
get closer and closer to being there.
01:01:35
But what struck Scott my bosses that
01:01:37
that maybe that was just fundamentally
01:01:40
wrong that you know you were trying to
01:01:42
imitate face to face kind of physical
01:01:46
proximate reality in this computational
01:01:48
medium and and and and the focus
01:01:52
usually imitation it takes you away
01:01:55
from doing anything that is distinctive
01:01:57
and that you can only do in the
01:01:59
computational medium and so we we
01:02:02
started this beyond being their project
01:02:05
which we had originally called better
01:02:09
than reality but that was little too
01:02:13
much even for us and and and and looked
01:02:17
at that and and and and lots of
01:02:19
different ways and I think learned a
01:02:22
lot from what the the lessons really
01:02:24
from Bellcore whatever reflect back on
01:02:25
is is that this idea of imitating one
01:02:29
medium and another is just you know not
01:02:31
the best research approach I think we
01:02:34
found that both and had plus plus and
01:02:37
also and beyond being there and and I
01:02:42
think for me one of the things that
01:02:44
comes out of it is a kind of research
01:02:47
strategies I recommend for students is
01:02:50
is if you spend a lot of time when
01:02:52
you're Reading papers or when you're
01:02:54
considering projects really focus on
01:02:57
the things that people are saying you
01:02:59
know one of the presuppositions what
01:03:02
are sort of wonder people sort of
01:03:04
presuppose a is known controversy le
01:03:07
something you don't even have to say
01:03:10
kind of thing and I think that that's a
01:03:12
a very powerful research strategy. Um
01:03:15
you know beyond being their lead to
01:03:20
lots of interesting things maybe we did
01:03:22
this kind of work on ephemeral interest
01:03:24
groups and and some asynchronous
01:03:26
discussions and we did some of the
01:03:28
original collaborative filtering kind
01:03:31
of work in those days and really
01:03:33
recommend or systems and and and second
01:03:36
I did some follow on work that we
01:03:37
called and negotiated access which I I
01:03:41
still think is a is a more and more
01:03:44
important topic that we try to get we
01:03:47
we got a patent assigned to a a
01:03:49
nonprofit so that someone else could
01:03:53
take the idea and and you know it's
01:03:57
always further and further contents
01:03:58
that this plastic representational
01:04:00
medium is that so anyway so I I I hate
01:04:05
back after being at Bellcore in and you
01:04:07
and am to UCSD and and I start a lot of
01:04:15
projects I can't really talk about some
01:04:17
here. Um but what I mentioned a few of
01:04:21
them but mean wanna say that we and
01:04:23
then I started this this unique
01:04:26
cognition an HCI lab which we wanted to
01:04:30
promote this shift in cognitive science
01:04:34
toward this broader view of cognition
01:04:36
is really really a property of systems
01:04:39
and and and that kind of functional
01:04:44
approach I think is really really
01:04:46
valuable extending the boundaries of
01:04:48
cognition beyond the cranium to include
01:04:51
other people artifacts and and and all
01:04:54
those as and and sort of the the the
01:04:57
central image for all of our work is
01:04:59
really no that we want to build
01:05:02
environments in which people pursue
01:05:04
their activities and collaboration with
01:05:07
the elements of the material world are
01:05:09
kind of thing to to look at that and to
01:05:12
do that you really have to know what
01:05:14
people do. Um how they coordinator
01:05:17
activity and and and really to be
01:05:20
really thoughtful about what role
01:05:21
technology should play in doing that.
01:05:25
And so one of the first projects I got
01:05:27
involved in was my my daughter just had
01:05:30
a automobile accident she she fell
01:05:33
asleep coming home from our boyfriends
01:05:36
late at night and cross the median had
01:05:39
a head on collision fortunately she's
01:05:41
okay fortunately everyone was okay from
01:05:44
but but sort of got me thinking about
01:05:46
cars and driving and and and centre
01:05:49
realising the cars are becoming
01:05:51
computer platforms and and why think
01:05:53
them often attracted to a sort of times
01:05:56
when you know new interfaces are
01:05:59
forming in the technology so attracted
01:06:01
to digital pens and those kind of
01:06:03
things "'cause" there isn't this
01:06:04
establish basin there's you can do new
01:06:06
things and in cars seem to be like that
01:06:09
that you the other lots of new
01:06:12
interfaces happening so I linked up
01:06:15
with the guy intellectual computer
01:06:16
engineering Hunter valley and we were
01:06:18
to speak proposal. And we started
01:06:21
looking at driving so we we bought a
01:06:25
car we instrumented with ten cameras so
01:06:29
we could look at what people really do
01:06:32
we had cameras on people's heads we had
01:06:35
cameras looking at them we had a media
01:06:37
cameras on the top we had the the first
01:06:41
ever foot cam so we could watch what
01:06:44
people were doing with their feet we
01:06:46
also we're interface to the bus of the
01:06:48
car so we hear all of the details out
01:06:51
of the car in terms of what it was
01:06:53
doing and the data sort of look like
01:06:55
this. Uh kind of thing where we were
01:06:59
actually digitising all this
01:07:01
information and the trunk of the car
01:07:05
and you know here you at the top left
01:07:08
you see this very first person view
01:07:10
from a camera on the forces had if you
01:07:12
look down we have other cameras this is
01:07:15
the first time this guys ever driven
01:07:16
this car you'll note that he has one
01:07:19
finger on the steering wheel a little
01:07:21
bit later probably won't get to it that
01:07:24
he almost gets in an accident you
01:07:25
briefly touches the steering wheel was
01:07:28
other hand and then starts joking about
01:07:31
this is I'm probably the first person
01:07:32
that almost wrecked your car kind of
01:07:34
thing. And we got interested in
01:07:36
interfaces and force feedback
01:07:38
accelerator panels and and using
01:07:41
dramatic things in the C provide
01:07:43
information but what I really got
01:07:45
interested in is is I I think I saw a
01:07:48
bit of the future here that that it's
01:07:52
going to be this kind of data that's
01:07:53
gonna drive design a kind of thing and
01:07:56
so I really got interested in building
01:07:59
a system to to eighty that kind of
01:08:03
analysis and so one of my students adam
01:08:05
files me and a bunch of other people
01:08:08
build a system called product is which
01:08:12
is really designed to eighteen
01:08:16
fieldwork eight ethnographic kinds
01:08:18
investigations eighteen things where
01:08:20
people have rich time based data be at
01:08:23
video be a simulator data kinds of
01:08:27
things and so this is an image from
01:08:29
some work well with and was doing with
01:08:32
bowling so we have a mobile video
01:08:36
streams this is from a high fidelity a
01:08:39
simulator and so we're watching pilots
01:08:42
in various tasks there we have multiple
01:08:45
sources of video we provide really
01:08:48
interesting facilities for annotating
01:08:51
that kind of thing you're trying to
01:08:52
understand it you have to support how
01:08:54
those how those kind of conceptual
01:08:56
stations are ways of annotating it
01:08:58
growing change over time we in this
01:09:03
case I can link it up to geographic and
01:09:06
spatial data we have transcript data
01:09:09
that being like and we have a notable
01:09:13
pen digital don't that people are
01:09:16
taking and on this on this is sort of
01:09:20
time stamp so we can we can link it
01:09:22
together and and so we can sort of go
01:09:28
from the you know for example from your
01:09:32
digital and yeah can note to video so
01:09:35
you can actually touch your paper with
01:09:37
the envelope in have a drive the video
01:09:40
to what was going on at the time you
01:09:41
took that what we find for it's not the
01:09:45
first is this kind of facilities change
01:09:47
the kind of notes that people take that
01:09:50
suddenly you can do things like just
01:09:52
say interesting you know we'll go back
01:09:55
to or that kind of thing and rather
01:09:58
than have you putting will get other
01:10:00
folks looking at other kinds of things
01:10:02
can record really detailed timing of
01:10:05
those events just by putting dots on
01:10:07
the paper kind of thing. And sort of
01:10:10
running on a little bit short time I
01:10:12
was gonna show you a a bit about video
01:10:15
where we've implemented I tracking in
01:10:20
this kind of thing again it's another
01:10:21
timestamp kind of data. So we can do
01:10:24
things for example like look at all the
01:10:27
fixations on an instrument. And then
01:10:29
map that back to the actual segments in
01:10:32
which people were doing that kind of
01:10:36
that kind of thing I think I will skip
01:10:38
over that we and you know we can is and
01:10:43
all kinds of things other people using
01:10:45
it it's it's open source it's available
01:10:47
yet rather biz dot com and and early
01:10:51
we're using it or looking at
01:10:53
interactions in a medical clinic on
01:10:55
south in San Diego and so really
01:10:58
complex kind of a situation where we
01:11:01
have a physician and patient. And often
01:11:04
interpreter because of language kinds
01:11:07
of issues we put a couple of connects
01:11:10
in the environment and we're doing sort
01:11:14
of automatic annotation of some of that
01:11:17
kind of data who's talking who's
01:11:19
pointing when they're when they're
01:11:21
using electronic medical records but
01:11:23
all the goal to sort of ratchet up the
01:11:27
level at which someone trying to
01:11:29
understand this kind of complex
01:11:31
phenomena is sorta placed and so anyway
01:11:36
so we do a lot of that kind of thing
01:11:38
with the curve is and and we're right
01:11:41
now filming pedestrians interacting
01:11:43
with cars and we'll be doing analysis
01:11:46
that we've also done a fair amount of
01:11:48
stuff with the note of digital pens
01:11:50
very much inspired by France walls were
01:11:54
on paper augmented digital documents we
01:11:58
collaborated some work with one of the
01:12:01
students and and I and it's a very very
01:12:07
interesting kind of technology for us
01:12:09
because it links up this kind of paper
01:12:12
based world with the digital world. Uh
01:12:14
and we've done quite a few different
01:12:18
kinds of things and then environment
01:12:20
some work in a a children strongly unit
01:12:26
DC we're we're looking at taking I
01:12:30
usually in the trauma bay there were
01:12:32
sort of code is saying some could come
01:12:34
to some awful kind of think for rush
01:12:37
that trauma bay there's a recording
01:12:39
nurse is taking kinds of information we
01:12:43
thought it would be really interesting
01:12:45
to to to let that person do exactly
01:12:48
what they did so we may we made that
01:12:50
but those things out on another paper
01:12:52
or we can record that we're looking at
01:12:55
projecting that onto displays so that
01:12:58
had a coming in later doesn't have to
01:13:01
interrupt to check on these kind of
01:13:02
things in again it's sort of this
01:13:05
wonderful thing of letting people do
01:13:06
what they normally do in the way they
01:13:08
normally do it. But because they do it
01:13:10
in a computational environment getting
01:13:12
these these added benefits once we've
01:13:14
done a lot of of work in that area with
01:13:16
a lot of what would your white ball
01:13:19
came to to oppose talk with me I was on
01:13:21
the speech D committee your and but my
01:13:26
favourite one of all time to some work
01:13:28
of and re Piper and and Marie was the
01:13:33
the easy student to advise of all time
01:13:37
you mainly just had to stay under her
01:13:40
way and she was always like a couple of
01:13:43
steps ahead of you. You know I would
01:13:46
say I you know I think we should really
01:13:47
do oh yeah I did that last weekend. And
01:13:49
I have those results and oh yeah I
01:13:51
already have a draft of that paper it's
01:13:53
it's three months from now but I have a
01:13:55
draft and and Andreas was also really
01:14:00
good at at speech choosing double
01:14:04
research projects that were nicely
01:14:06
sculpted which is really hard for kind
01:14:09
of thing and so Emery was going off to
01:14:12
a birthday party for great grandmother
01:14:17
who was celebrating a hundred and fifty
01:14:20
per day and so in those are good genes
01:14:25
to have and and so we we thought of
01:14:30
this project we could print out a bunch
01:14:32
of images of another paper should take
01:14:36
a bunch of cans with your and and and
01:14:40
people could point and talk about this
01:14:43
kind of things and know get recording
01:14:45
so that her great grandmother could
01:14:47
experience this mainly interested in
01:14:50
family at that point you had a lots of
01:14:52
images honour walls of family but it
01:14:54
was too far away to really see suddenly
01:14:57
if you could open up these kind of
01:14:58
things and interact with this pen. And
01:15:00
it was incredibly touching have these
01:15:03
recordings of one of her eighty year
01:15:06
old daughters talking about an image
01:15:10
and pointy things and and saying how
01:15:12
she remembers that day and remember
01:15:14
what we did after that and and it's in
01:15:17
the voice of of the daughter and so
01:15:22
again a really interesting kind of
01:15:25
environment for for interfaces lately
01:15:30
for the last two years ago and looking
01:15:32
at I what I'm sort of calling activity
01:15:35
enrich computing your again sort of
01:15:37
capturing and mining activity history
01:15:39
some sort of returning to and have
01:15:42
student guess stone county Otto who
01:15:45
enforcing died of cancer right is is
01:15:49
finishing his PHDO dissertation and get
01:15:53
started on this incredibly interesting
01:15:57
study and in a law office immigration
01:16:00
law office and is sort of thought he
01:16:02
was going to do a project about us
01:16:05
software gate collaboration in this
01:16:08
kind of of office environment. And when
01:16:11
I think C started doing just recording
01:16:13
a bunch of screen captures what people
01:16:15
doing see good sort of the movie of
01:16:17
directivity. And he started replaying
01:16:20
those movies for folks and just asking
01:16:23
to talk about what they were doing and
01:16:26
we were both like incredibly struck by
01:16:29
how provocative that seem to be in
01:16:32
terms of people's what they remember
01:16:35
they seem to remember all kinds of
01:16:37
things of what they were doing at the
01:16:39
time someone came into their office I
01:16:41
meet a phone call about directly to
01:16:43
follow up on that and so we both got
01:16:45
enamoured of maybe using that kind of
01:16:49
thing to help people reinstate context.
01:16:52
And so guess to built a system that
01:16:54
well we collectively trails which was
01:16:57
sort of an episodic retrieval system so
01:17:01
you capture all the text on the screen
01:17:03
and everything everyone type. And and
01:17:06
since real task mean we have pretty
01:17:08
good history facilities within
01:17:10
applications but real tasks are spread
01:17:12
across applications and so it would do
01:17:15
this retrieval and and bring you back
01:17:17
episodes you get little thumbnails of
01:17:19
that and then you could replace sort of
01:17:21
the movie of what was going on. And I'm
01:17:24
still we just got some innocent funding
01:17:27
to look at this. This more of this kind
01:17:30
of context reinstatement and I'm I I
01:17:33
think it's a very interesting
01:17:35
conjecture because so so many things
01:17:37
now our computer mediated or so many of
01:17:40
interruptions in there so often
01:17:42
difficult to get back into various
01:17:44
activities and lots of other folks of
01:17:46
of done work in this area but but but I
01:17:48
think it may require less than one
01:17:50
imagines in that kind of thing and the
01:17:53
in the the metaphor I have very much is
01:17:56
that you know if you come to my office
01:17:58
and look at my desk what you see is a
01:18:00
messy desk. And and I probably if I go
01:18:04
to your office I I see a messy desk.
01:18:06
But when I look at it I I see projects
01:18:09
and I see the things I'm gonna take a
01:18:10
when I see okay "'cause" it's my
01:18:12
activity it's my guess kind of thing
01:18:14
and I think that looking at recordings
01:18:17
of your activity is similar to that
01:18:20
when I look at my recordings yeah gotta
01:18:22
running out of time and you know we
01:18:26
need to I I'm sort of thinking of my
01:18:31
intentions what I was doing and why I
01:18:33
was doing it kind of thing and they and
01:18:35
I I think there's this bill would you
01:18:36
do exploit a visual memory. So finish
01:18:39
up not really quickly but I what's a
01:18:41
little bit about the study "'cause" it
01:18:42
really since recognition memory study
01:18:45
it's you know very classic kind of
01:18:47
study that you knew you showing a
01:18:49
series of of images and then you're
01:18:52
asked to do a forced choice a kind of
01:18:54
thing but you've seen something for not
01:18:56
and that they did these really to other
01:19:00
interesting vision but what you know we
01:19:01
just incredibly good that we really
01:19:03
good at visual kinds of things like
01:19:05
that. But they get these two other
01:19:07
conditions one of them was a condition
01:19:09
where it was it was all objects but
01:19:13
you're shown a pair of objects that
01:19:15
were from the same category when you'd
01:19:17
seen before when you hand that kind of
01:19:20
thing it's still people are like you
01:19:21
know eighty ninety percent good at
01:19:24
doing that. But in a sort of killer
01:19:26
condition for me was you show the same
01:19:29
object in a different state and still
01:19:32
people are like at eighty seven percent
01:19:35
kind of of things so you know seeing
01:19:39
use a picture abacus and then you see
01:19:41
the abacus with the balls in a
01:19:43
different position or that kind of
01:19:45
thing. But I think what that leads me
01:19:47
this is it you know or visual memory is
01:19:49
incredibly good and if we can exploit
01:19:51
data retrieval okay so I'm just gonna
01:19:53
end up with some future challenges and
01:19:56
opportunities and and you know again I
01:20:02
think the opportunity all comes from
01:20:03
this plastic medium kind of thing I
01:20:06
think actually are greatest challenge
01:20:08
is is being locked into the current way
01:20:12
of doing things we surely saw that it
01:20:15
had plus plus would just very hard for
01:20:18
students to think of things other than
01:20:20
in terms of menus and icons and and
01:20:23
that kind of thing the image I often
01:20:26
use of this is is what I actually the
01:20:28
same age it just looks like a I don't
01:20:32
know what but if I outline it. Um to
01:20:35
show you a cow. And I go back and forth
01:20:38
a couple times then you can see this
01:20:42
without seeing the cal. And I think
01:20:44
that we you know it's equivalent to
01:20:47
lots of kinds of things for interfaces
01:20:48
we just lot into those kind of notions
01:20:52
so the you know the I I you know in one
01:20:58
sense you know really excited about
01:20:59
things but then once it's it's sort of
01:21:01
sad to see how little progress in some
01:21:04
sense it's been made you know almost
01:21:07
everything we do with computers is
01:21:09
difficult and awkward and frustrating
01:21:12
you know even after like what it's been
01:21:15
six decades of evolution. There's just
01:21:18
little of the naturalness of and the
01:21:21
contextual sensitivity of of working
01:21:24
with other activities and you know
01:21:27
there's not this can the reality that
01:21:29
we have with working with paper another
01:21:32
kind of tangible mediums the crucial I
01:21:37
think in this is the unit of analysis
01:21:40
that we don't I think you know and
01:21:42
science the universe analysis is
01:21:44
crucially important. And I think one of
01:21:46
the key inside to distribute condition
01:21:49
is to be flexible about in you know of
01:21:52
analysis you know sometimes the
01:21:54
individual is the right unit of
01:21:55
analysis but often. It's not and and I
01:22:00
think for maybe too long I think
01:22:02
israel's very much changing today our
01:22:05
focus has been on the individual the
01:22:08
other thing I think very much is that
01:22:10
we still you know this presuppositions
01:22:13
we have about information about
01:22:16
applications. I think I think we have
01:22:19
some very unfortunate presuppositions
01:22:21
about that I think that we're sort of
01:22:26
trapped in this spiral currently of
01:22:28
apps we're we're just building more and
01:22:32
more perhaps for things. Uh and that
01:22:36
and I don't think that that's going to
01:22:38
be the way to go to support real
01:22:41
convince real friendly kind of
01:22:43
environments were were sort of left to
01:22:45
break everything into the right units
01:22:47
of apps and then re place things back
01:22:51
and and I think that we're also one of
01:22:54
the other presuppositions is is that
01:22:57
information is basically get bits I
01:22:59
mean there's I think there's some work
01:23:01
I mean you know go now kind of things
01:23:04
where things are sort of context
01:23:05
sensitive and stuff. But but if I had
01:23:08
to put my money and we're to invest. I
01:23:10
would invest in building a a priori
01:23:13
different informational infrastructure
01:23:16
you know that I want to think of this
01:23:18
being sort of a cognitively inspired
01:23:21
physics for information that that that
01:23:24
physics it is such that the information
01:23:26
is more active. And and context
01:23:29
sensitive and that and that the we
01:23:31
build things on top of that kind of
01:23:34
thing. So anyway. So just wanna end up
01:23:37
and you know we're interest in doing
01:23:39
some of those things at the new design
01:23:42
lab at San Diego and many other kinds
01:23:45
of things. And and and I just wanna
01:23:49
think you know just all of my
01:23:51
collaborators and students and and
01:23:54
people have been involved and and one
01:23:56
thank you for your attention and it's a
01:23:58
true honour yeah oh and we are going to
01:24:20
the straight into the break and now but
01:24:23
will it will be outside if anybody

Share this talk: 


Conference Program

SIGCHI Lifetime Research Award Talk
James Hollan
21 April 2015 · 2:35 p.m.

Recommended talks

Small Devices for Big Impact
Nuria Oliver, PhD, Scientific Director Telefonica Research
19 Sept. 2013 · 3:05 p.m.
The Experience when business meets design
Brian Solis, Award-winning Author, Prominent Blogger/Writer, Principal Analyst at Altimeter Group
11 Feb. 2016 · 2:06 p.m.
149 views