SIGCHI Lifetime Research Award Talk

Player is loading...

Embed

Copy embed code

Transcriptions

Note: this content has been automatically generated.

00:00:00

After lunch and I would like to welcome

00:00:03

all of you to our then with Jim pollen

00:00:07

today I think lots of you know GM

00:00:10

already and I'm really excited to be

00:00:12

able to introducing today because one

00:00:15

is maybe my second year on the faculty

00:00:19

at Stanford to did is sabbatical what

00:00:22

everybody got nine you know in our

00:00:24

group and Gina sat in on my graduate

00:00:27

class which was a real honour and a

00:00:30

little bit daunting any came up to me

00:00:32

after class one day. And he said why

00:00:36

you in a class is really great but I

00:00:38

have I have one small piece of advice

00:00:40

if you're if you're interested as I get

00:00:41

you said when you ask a question you

00:00:45

always immediately start talking right

00:00:48

afterwards and use like if you wait

00:00:50

just a couple milliseconds you'll be

00:00:52

amazed by if you if you let that

00:00:54

silence hang in the air how much more

00:00:57

interesting the discussion but columns.

00:01:00

And I think that that's one thing that

00:01:02

we can learn from from James career in

00:01:04

general is that I think he's one of the

00:01:06

people who is the best one of the best

00:01:08

listeners and our field and so we're

00:01:11

here today to listen to GM in part

00:01:15

because he's the one who taught us how

00:01:18

to listen and sure enough in class one

00:01:20

of the biggest changes I made in

00:01:21

teaching the line out sell any

00:01:24

colleague to listen is after you ask a

00:01:26

question take a moment and just let the

00:01:29

silence wagon and see what people have

00:01:31

the side for those of you who are

00:01:35

walking in or new the car I wanted to

00:01:38

tell you just a very few things about

00:01:41

what you is done nobody tell most of

00:01:42

the story. Um jealous then is that a

00:01:46

number of things over his career

00:01:48

including ah he was one of the people

00:01:51

who is in cognitive science very early

00:01:53

on at UC San Diego over his life

00:01:57

actually has been a department chair.

00:01:59

Here's wrong to research lab many his

00:02:03

former students and colleagues are in

00:02:05

the audience here at back at San Diego

00:02:08

with at Hutchins team created one of

00:02:10

the first big undergraduate

00:02:13

specialisations in human computer

00:02:14

interaction was a very early very

00:02:16

influential per program. And the kind

00:02:19

of science department. And I think that

00:02:21

that that curriculum it just doesn't

00:02:24

amazing job with with marrying the ate

00:02:27

the C in the eye together I I think

00:02:29

it's it's a real model for that also

00:02:32

Jim is without question one wanna HCI

00:02:35

is most affective undergraduate mentors

00:02:39

after I joined the cognitive science

00:02:41

department at San Diego I am amazed by

00:02:44

how passionate and dedicated and

00:02:47

excited the undergraduate alumni lower

00:02:50

will come back to an extremely engage

00:02:52

bunch the gym has several back to speak

00:02:54

in his classes they come back to have

00:02:57

dinner with us to give us advice on the

00:02:58

program it's a really it's a really

00:03:00

awesome group. And lastly as I think

00:03:05

that I hope my opening story share it

00:03:09

the genus then one of the most powerful

00:03:12

mentors in the field of human computer

00:03:14

interaction I think we're gonna really

00:03:16

lucky field if you look at how massive

00:03:19

this event has grown. And and it hasn't

00:03:24

been all that long. And I think they

00:03:27

rapid growth in the success and in

00:03:29

particular the the you know actually

00:03:32

actually medical nature of car is due

00:03:34

to the the leadership and mentor ship

00:03:37

of people like to him and so let's

00:03:40

welcome him to stay oh Okay I I see if

00:03:59

I can be here here since I'm a listener

00:04:03

try to speak. Um it's a know incredibly

00:04:07

great honour to receive this award and

00:04:12

probably as much as anything else I owe

00:04:15

any success I've had to selecting

00:04:18

really and collaborators and wonderful

00:04:20

students sciences surely a social

00:04:23

collaborative activity and and and I've

00:04:27

been incredibly fortunate in that

00:04:29

regard. And to honour them I like to

00:04:33

sort of share this award with them I

00:04:36

have room for everyone in fact is

00:04:39

walking around the conference I'm

00:04:41

already embarrassed of people that I

00:04:43

left off of this list but but anything

00:04:49

I've done has really been done in

00:04:50

collaboration with with other folks and

00:04:53

so I really like to sure they work with

00:04:56

them it was simple to get introduced by

00:04:59

scott. Um line calling now and and it

00:05:05

was fantastic to sort of attract a and

00:05:09

equally stellar other aboard that ski

00:05:13

to UCSD and and so we had a series of

00:05:18

really interesting events of late this

00:05:21

past summer were able to attract don

00:05:23

Norman to boomerang back to UCSD don

00:05:28

was there and when I first got there

00:05:31

and been there for quite a while before

00:05:33

and together with Scott don and I are

00:05:37

starting a new design lab at uses you

00:05:40

know mention it briefly a little bit

00:05:42

later. Um and and personally for me all

00:05:47

of this came at a really good time my

00:05:50

colleague of over thirty years and

00:05:53

Hutchins from whom I've learned

00:05:56

probably more than anyone else about

00:06:00

this area about people and about

00:06:03

distributed cognition decided to retire

00:06:08

last summer and I thought oh god I

00:06:11

share the lab with and for all this

00:06:13

time what am I gonna do now and so just

00:06:17

then Scott shows up and don shows up

00:06:19

and we were starting a new lab so it's

00:06:21

really great and also really happy that

00:06:25

that at is following it dawns footsteps

00:06:28

don is probably one of the most

00:06:31

successful failures at retirement that

00:06:34

ever happened he's retired from many

00:06:37

places now and now it's back I'm

00:06:40

retired at UCSD and add only lasted the

00:06:44

summer and now he's back with this

00:06:47

again so I'm I'm really I'm really

00:06:50

happy for that. So I was trying to

00:06:55

think a lot about this well yeah really

00:06:57

procrastinating last night making this

00:06:59

list of names of all the folks and

00:07:03

fearful that I would be people out

00:07:05

which I have but I wanna do three

00:07:08

things I wanna sort of share a little

00:07:10

bit what my excitement is about the

00:07:14

field and and why I think it's such a

00:07:17

crucially important time for eight CI

00:07:22

it was one thing in the early days one

00:07:24

each of us at what little mac on our

00:07:26

desks unconnected do anything else. Uh

00:07:29

and try to design systems and it was it

00:07:31

was hard enough to do that but now

00:07:34

we're faced with a very different world

00:07:36

and were crucially part of just every

00:07:40

system that's being built so I'll talk

00:07:43

a little bit about that maybe a little

00:07:44

bit too much but but I think I'm

00:07:46

envious of all the young folks here

00:07:49

coming into feel like exactly the right

00:07:51

time well if I could've chosen went

00:07:54

into the field it would be now and and

00:07:58

I'm actually tell you a little brief

00:08:00

story about my path and and and some of

00:08:03

the lessons I've learned it's been it's

00:08:06

been quite a pleasure to sort of

00:08:08

reflect back sort of a thirty or more

00:08:13

years of working in the field. And then

00:08:16

I spent a little bit of time talking

00:08:18

about challenges and opportunities and

00:08:22

try to throw in a little advice along

00:08:23

the way I think it's really all about

00:08:26

stories I've always been very taken by

00:08:30

the Nobel laureate peer metal war when

00:08:33

he talks about science as says so it's

00:08:36

begins as a story about a possible

00:08:38

world. Uh a story which we invent and

00:08:41

criticise and modify as we go along so

00:08:45

that ends up being is nearly as sick as

00:08:48

we can make it the story about real

00:08:50

life. Uh and I think that that's what

00:08:52

science is really about but I think we

00:08:56

have in our discipline we have an

00:08:58

incredible advantage because we can we

00:09:02

can sort of make at least some of our

00:09:03

stories true or writing code in

00:09:06

building systems and and I think that

00:09:10

that's a tremendous advantage that we

00:09:12

have and so anyway so exciting times I

00:09:20

think probably the most positive thing

00:09:23

that I see in the field is that we're

00:09:26

moving away I think for way too long

00:09:29

we've conceived of thinking is

00:09:31

something that happens exclusively in

00:09:33

the green in the head and when I start

00:09:37

off actually thought about thinking

00:09:39

very much that way but you know I think

00:09:43

we all sort of realise now that

00:09:46

thinking happiness in the world. Uh as

00:09:48

well as in the hand we think with with

00:09:50

things with our bodies with marks on

00:09:54

paper in the say and and with other

00:09:57

people and that thinking is sort of a

00:10:01

distributed social activity other

00:10:04

exploits these wonderful facilities of

00:10:06

language all the kinds of

00:10:09

representational media that we have and

00:10:13

are embodied interaction with the world

00:10:15

with other people and and today we we

00:10:19

sort of increasingly think with

00:10:21

computers and and I think that that's

00:10:25

one of the reasons that this is such an

00:10:27

important field because we want we want

00:10:32

to build kinds of systems that are

00:10:34

really allows the think thoughts that

00:10:36

we couldn't think without computers and

00:10:39

other people and and but it's it's not

00:10:43

the computer we grew up with this

00:10:45

surely not the computer I kind of thing

00:10:49

when I first got this UCSD there was a

00:10:52

big brothers sixty five hundred in a

00:10:56

room with a lot of air conditioning and

00:10:59

glass window so you can sort of looking

00:11:02

at that and you could put you know you

00:11:04

give your cards to someone to read into

00:11:07

the computer and a day later you come

00:11:09

and get a a printout of that little

00:11:11

syntax error you may in your program

00:11:14

that monolithic computer that we grew

00:11:17

up is is really coming apart and it's

00:11:20

being reassembled and myriad you forms

00:11:23

and the boundary sort of between people

00:11:27

and computers in between physical and

00:11:30

digital world is becoming increasingly

00:11:33

more and permeable things are moving

00:11:36

back and forth well along that thing we

00:11:40

talked today a a being on the threshold

00:11:42

of what people call the age of the

00:11:46

internet of things and I think that's

00:11:49

not quite right kind of thing it's

00:11:53

really more than things and people

00:11:55

always have to be in the picture. Uh so

00:11:59

I think we're really moving towards I

00:12:01

in internet everyone and everything

00:12:04

kind of thing and that's again one of

00:12:07

the reasons that this field is so

00:12:09

crucially important every side of sort

00:12:13

of then thinking about why movement is

00:12:16

happening. So quickly apart Abbott is

00:12:18

more long part of it is all the sensors

00:12:20

and and and part of it is you know the

00:12:23

world getting network where I can we

00:12:25

connected everywhere but but more and

00:12:28

more I'm sort of thinking it's also

00:12:29

this changing cost structure. We can do

00:12:32

things that we would never ever even

00:12:35

conceived up for where the the the

00:12:37

price of of using a thousand computers

00:12:42

is the same as using one computer for

00:12:45

the same period of time kind of thing

00:12:48

and that's that's radically changing

00:12:50

things in what we can do in the way

00:12:53

that we can we can do it. Uh

00:12:56

fundamentally changing and and

00:13:01

interaction is is not sort of movie but

00:13:05

it's it's rushing towards away from the

00:13:09

desktop and and I recently helped

00:13:14

organise actual conference in Germany

00:13:18

on interaction be on the desktop and

00:13:20

and just realises this drawing

00:13:23

zeitgeist of the importance of moving

00:13:25

beyond the desktop is shared

00:13:28

internationally with with folks. But

00:13:31

the reason I think that it's an

00:13:33

exciting times is computation is really

00:13:37

ling Kay have this wonderful expression

00:13:41

the computation was the first that'd be

00:13:44

a medium where we can build other

00:13:46

medium. And for me the thing that's

00:13:49

appealing and exciting and attractive

00:13:51

about computation is is is incredibly

00:13:54

plastic media that we can you know

00:13:57

fashion representations you kinda

00:14:00

action. And in ways of communicating

00:14:03

that we all experience everyday. But we

00:14:06

can also memory well older me. So we

00:14:09

can make I had to look like the look

00:14:11

and act like about change pages like

00:14:14

about but we can also create these

00:14:17

really wonderful new kinds of

00:14:19

representations that are sort of

00:14:22

unconstrained by the constraints of

00:14:24

paper and other media and you know the

00:14:29

models I mean you know I can now was I

00:14:35

stopped in Hawaii on the way here to go

00:14:37

surfing and the models are such that I

00:14:41

could know what the server is going to

00:14:43

be like on the north shore of a while

00:14:45

in San diego. And really deciding

00:14:50

whether I should get on the plane kind

00:14:52

of thing you know we can build these

00:14:56

wonderful virtual worlds which are you

00:14:58

know where which are now long past the

00:15:01

sort of metaphorical desktop grew up on

00:15:04

the alto kinds of things suddenly these

00:15:07

world can be you know how all the

00:15:10

wonderful techniques of film and

00:15:12

animation kinds of of things. And and I

00:15:16

think we're you know and probably see

00:15:19

some of this of the conference sort of

00:15:21

just going to our route these areas of

00:15:24

virtual reality and augmented reality

00:15:26

kinds of technologies are just coming

00:15:28

to fruition and they're going to be I

00:15:32

think quite major for a few and and we

00:15:36

can also combined the real and the

00:15:38

virtual and things like computer

00:15:42

augmented surgery. So you know here's

00:15:45

eventually like surgery system the the

00:15:51

surgeon is sitting at the console

00:15:53

manipulating kinds of things you really

00:15:55

dealing with a virtual which impacts

00:15:59

the robot the this world allows him to

00:16:03

take sort of tremors and things out of

00:16:07

his hands it allows him to see for

00:16:10

example doing a might roll valve

00:16:13

replacement in your heart which used to

00:16:14

be this open really crack open the

00:16:17

whole channels. Now it's done with

00:16:19

these are just copy kind of body things

00:16:22

that he can be in their time knots

00:16:24

inside of someone's heart of that can

00:16:27

be in large you could be aided in doing

00:16:30

that that kind of virtual world. Uh as

00:16:32

exciting what is connected with the

00:16:35

real world and there's like you know

00:16:37

some problems with the line stuff

00:16:39

there's there's no in principle reason

00:16:42

that the surgeon with special skills

00:16:45

could be operating on someone on the

00:16:47

other side of the work the thing I mean

00:16:50

that's an exciting kind of thing

00:16:52

exciting times but for most of us lots

00:16:56

of excitement is come with the movement

00:16:58

onto the off the desktop and and into

00:17:01

smart phones and here's Steve jobs in

00:17:04

reducing the the I phone. And

00:17:06

increasingly we carry many of these you

00:17:09

know I sort of you know use like travel

00:17:11

with the phone and then I had a laptop

00:17:14

and kind of thing and some of them we

00:17:17

don't even think about as being

00:17:18

computers although they are incredibly

00:17:21

powerful computers and part of the

00:17:24

power of the connected to computers and

00:17:27

sensors and people so I can sit here

00:17:31

chat with my wife before giving a talk

00:17:35

here in Korea and you know just all

00:17:39

over activities are increasingly

00:17:41

mediated by computers and that's

00:17:44

changing just almost every aspect of

00:17:47

more professional and personal lives

00:17:50

and I think often for good. Um but also

00:17:53

often for sale my students Kevin lee

00:17:56

came up with this great that picture

00:17:58

number of years ago which I can't stop

00:18:01

using and and I think this is for you

00:18:06

know I think these young folks are

00:18:08

missing something and I had shown this

00:18:11

a couple of times when I realised. They

00:18:14

could be texting each other so it could

00:18:18

be even worse than I think and and

00:18:23

we're saying you know this movement off

00:18:25

the desktop and into the cars in fact I

00:18:27

was surprised to see there was voters

00:18:30

show here that I would been interesting

00:18:32

coming to just before just before I've

00:18:35

seen in a Mercedes have this a

00:18:38

prototype car states recently we know

00:18:42

about the goals of driving or kinds of

00:18:45

things. And we become interested in the

00:18:49

lab and communication between cars and

00:18:53

pedestrians how was the car going to

00:18:56

let you know when you're crossing a

00:18:58

street at that see you know how are you

00:19:02

gonna communicate well with that kind

00:19:05

of thing there's lots of really

00:19:07

challenging issues as we get or semi

00:19:09

autonomous and timeless cars and and

00:19:13

and most of them are you know a CI

00:19:16

really how to relate that impresses you

00:19:19

this great video play some of the tools

00:19:22

which just too good to give you a feel

00:19:26

for for you know if you're looking it's

00:19:29

a an intersection in your city MMM in a

00:20:08

I oh I say SMCI someone. MMMFC but the

00:20:55

exciting times also brought why are all

00:20:57

of these kinds of new device. We've

00:20:59

done a lot of work with another digital

00:21:01

pens and suddenly you know hands can

00:21:04

report you're writing report what's

00:21:07

being said while you're writing put on

00:21:10

a glass and you can record what you're

00:21:13

seeing and tell you about what you see

00:21:15

there's like field cameras come you can

00:21:20

take a picture only today to refocus

00:21:25

after you take a picture there's you

00:21:29

know cars are just becoming computing

00:21:32

platforms all kinds of devices heads up

00:21:35

displays all of these kinds of things

00:21:37

all all needing a help of for this one

00:21:41

today everyone's excited about watch

00:21:44

out to be available to try you think a

00:21:49

million were ordered on the first day

00:21:52

kind of thing I think a million more at

00:21:56

least has been ordered sense then we're

00:21:59

excited about this where ability have a

00:22:04

watch is one of the biggest get starter

00:22:07

kinds of things over ten million

00:22:09

dollars people that but it's I don't

00:22:21

that's good so the couple walks to log

00:22:31

just ah sorry. I'm unless there is not

00:22:34

set so so and I thinking about this and

00:22:38

you know one of the first watches not

00:22:41

not a watch person. But I was excited

00:22:43

when this watch came out and I don't

00:22:47

know if if you miss are old enough to

00:22:50

sort of remember this watch that was a

00:22:54

commodore watch from the seventies some

00:22:59

of you might still remember the

00:23:02

commodore sixty four microcomputer

00:23:05

utterance of the commodore was watch

00:23:07

company before it became a computing

00:23:10

company. That's sort of interesting to

00:23:12

see companies may be starting to go the

00:23:14

other direction kind of thing one of my

00:23:18

students brought me this wonderful

00:23:20

cartoon I need to share with you. Uh

00:23:24

about this reach glorious period when

00:23:28

we were or risks were free kind of

00:23:31

thing and and I think both and thinking

00:23:35

of the commodore really computer that I

00:23:39

guess. I think one so have gonna

00:23:41

through in my first piece of

00:23:42

unsolicited advice and I think they're

00:23:46

field needs to spend more time knowing

00:23:49

the history and the literature of er

00:23:53

feel I think ideas have histories and

00:23:57

can really only be understood fully in

00:24:00

the context of those histories and

00:24:03

that's a that's a message that bill

00:24:05

Buxton has been delivering to us for

00:24:09

quite some time and and I think both he

00:24:13

and terry or not only sources of very

00:24:16

council and wisdom but also provided

00:24:19

really valuable online resources I

00:24:21

think bill's thirty years of examples

00:24:26

of interaction techniques and

00:24:27

interaction devices that is incredibly

00:24:30

valuable tool to look at and and I

00:24:33

think we all owe terry an incredible

00:24:36

amount of praise for starting the the

00:24:42

the people computer computing and and

00:24:44

design seminars at Stanford and now

00:24:46

there's is personally twenty five years

00:24:49

of very influential talks so one

00:24:52

personal biases to go back and and and

00:24:55

look a little bit of the history of the

00:24:57

ideas now the reason for exciting times

00:25:00

is I think we're in the midst of a so

00:25:03

the data revolution not just the big

00:25:07

data that that people are talking about

00:25:09

what sort of more thick data data of

00:25:12

people really interacting us and

00:25:15

expensive digital recording devices and

00:25:18

video of revolutionise a behavioural

00:25:21

science so suddenly we can we can look

00:25:24

at things in ways that we never could

00:25:26

before we can capture this is my

00:25:29

colleague and Hutchins when he had more

00:25:33

here and when he was young and this was

00:25:37

when he was doing his PHD of fieldwork

00:25:39

it's a cognitive anthropologist and use

00:25:42

to his PH if you work the new guinea

00:25:44

looking at land litigation when someone

00:25:48

dies and how things are are passed down

00:25:50

and people get together in the village

00:25:53

and the square and and discuss this

00:25:55

kind of thing. And and wanted to

00:25:58

understand that and line this new to

00:26:00

the language well he's new to the

00:26:02

culture but this you could almost see

00:26:05

it here what he's he's got this I think

00:26:07

around his neck and it's just great big

00:26:09

square thing. It's a battery powered

00:26:11

tape recorder. So for the first time

00:26:14

you could take that into the field and

00:26:17

record what people really said and if

00:26:19

you've ever listen to what people

00:26:21

really said say you know even yourself

00:26:24

on some very startling kind of

00:26:27

experience because it's very different

00:26:29

than what we hear kind of thing and and

00:26:33

so and was able to study that and I

00:26:34

think we're we're sort of at a period

00:26:36

were were we're able to do that in ways

00:26:40

that we're just unimaginable just a few

00:26:43

years ago I'm very taken by the work of

00:26:46

of shock and candy good one we're Ucla

00:26:50

currently and and they looked at all

00:26:54

kinds of of of really interesting

00:26:56

activities so like children playing

00:26:58

hopscotch all the complex things that

00:27:01

are going on in the way people like

00:27:03

could use their bodies and their facial

00:27:05

expressions and things to to

00:27:08

communicate their intentions and that

00:27:11

the elderly gentleman in the bottom

00:27:13

picture is is trucks father and he had

00:27:18

a stroke in left hemisphere and I can

00:27:23

only say three words after the stroke

00:27:27

it's a yes no and the word and and that

00:27:33

was his vocabulary. And what shot was

00:27:36

able to document and begin to

00:27:38

understand this how is father was still

00:27:41

and incredibly active participant in

00:27:44

conversation was able to get his

00:27:46

intentions over to people was able to

00:27:48

accomplish what you wanted to do by

00:27:52

facial expressions by movement like

00:27:54

pointing by all of this kind of thing.

00:27:57

So I think where the you know a really

00:27:59

interesting time of for looking at what

00:28:03

people really do and I I think that we

00:28:05

don't do enough of that I I think we we

00:28:09

don't know enough about even what we do

00:28:12

kind of things is and we have an

00:28:15

opportunity now to do that. Okay so

00:28:17

talk a little bit about sort of my

00:28:20

research staff and sort of my story

00:28:26

kind of thing say young kid I was about

00:28:31

as nerdy as they get a and you know

00:28:35

fortunately I was an only child so I

00:28:37

could be even dirtier kind of thing.

00:28:41

And I got involved in now. Um act was

00:28:46

sort of like thing and I particularly

00:28:49

got involved in studying mathematicians

00:28:53

lives a and Reading novels and not

00:28:56

novels and stories about meditation one

00:28:59

famous one I remember ah who the gods

00:29:02

loved which was written by two

00:29:04

mathematicians about a rusty gal wow

00:29:08

who invented group theory who died in

00:29:11

the dual as a young man who couldn't

00:29:14

get in the cold polytechnic because you

00:29:18

were supposed to solve the classic

00:29:21

problems in the classic ways and gal a

00:29:24

came up with the new proof every time

00:29:27

and was not allowed the and and so

00:29:31

mathematicians we really interesting to

00:29:33

me because they were they were weird

00:29:35

and they we're doing cool things and I

00:29:37

was I was totally taken by things like

00:29:41

Anders diagonal proof a word first I

00:29:44

can sort of see that you know that I

00:29:47

could understand something about

00:29:49

uncountable infinite kinds of things

00:29:52

and so I thought and I'm gonna be a

00:29:54

mathematician and I will I scored high

00:29:56

mismatch Olympia and everyone pay maybe

00:29:58

with the map brush and so off to

00:30:02

college and and I I unfortunately got

00:30:09

bitten by the what was then called a

00:30:12

four colour conjecture and if you don't

00:30:15

know this this thing it's the

00:30:18

conjecture was that it only takes for

00:30:21

colours to colour any planar map. So

00:30:24

that the process any border it's

00:30:25

different colours. And and this is a a

00:30:29

longstanding problem which was

00:30:32

sometimes called a four colour disease.

00:30:35

Because it it'd been passed down from

00:30:37

one generation to the next where the

00:30:40

father or mother worked on the problem

00:30:42

for a period of time and then the the

00:30:44

their the son or daughter worked on the

00:30:48

problem. So I was I was you know it's

00:30:52

very easy to map to send the graph

00:30:55

theoretic kinds of stuff and I got

00:30:57

really interested in graph theory and I

00:31:00

became obsessed I was I was totally

00:31:02

convinced that there was this monster

00:31:05

counter example that that's the reason

00:31:07

that no one had ever proven that's and

00:31:10

I was going to find it and the

00:31:14

unfortunately a few years later it was

00:31:18

proven to be true but what I really

00:31:21

found was computers and programming and

00:31:24

and I used to massive amounts of of

00:31:26

time on the computing systems looking

00:31:29

for counter examples and needlessly. Um

00:31:34

and then I came across a block by

00:31:38

George Paul you how to solve it and and

00:31:42

it really change my orientation

00:31:45

suddenly. I became much more interested

00:31:49

in how people do mathematics and how

00:31:52

you come up with conjectures in the

00:31:55

whole cognitive side of things. Uh and

00:32:00

and you know and it it sort of resulted

00:32:04

in and we wanted to go to became a

00:32:06

double major in math and psychology.

00:32:09

And almost no one in psychology wanted

00:32:11

to talk about the things I wanted to

00:32:13

talk about in those days I I went to

00:32:17

graduate school that married is

00:32:19

undergrad to a woman who I wanna go to

00:32:23

med school shoes all we admitted to to

00:32:25

one medical school in fact it was the

00:32:28

sort of the last year that it was

00:32:30

really difficult for women to get into

00:32:32

med school there were there were four

00:32:33

women in her med school class kind of

00:32:37

thing and so we went a place which

00:32:39

actually was accepted I didn't think it

00:32:41

was a good place for me because no one

00:32:43

was interested in the kinds of things I

00:32:44

was interested in but it turned out

00:32:47

that that was probably the best

00:32:48

possible place to go because I was sort

00:32:51

of on my own and I had to to sort of

00:32:55

make my own sort of graduate experience

00:32:59

and then I was you know very fortunate

00:33:01

to be accepted a postdoc with absolute

00:33:04

rupees it's standard in the institute

00:33:06

for mathematical studies and social

00:33:08

sciences but that wasn't really the

00:33:11

right place for me either I thought it

00:33:13

was but I started hanging out of the I

00:33:16

lab I and xerox park and that sort of

00:33:20

totally change my life because I found

00:33:22

the alto what I found list machines

00:33:24

which was love at first sight and and

00:33:30

you know and I at the time I didn't I

00:33:33

didn't know what I was going to do but

00:33:34

I knew what was going to do with

00:33:36

machines like this. Uh that they were

00:33:38

just so interesting in the researcher

00:33:40

such possibilities representational

00:33:44

like with them twenty so the phases of

00:33:48

my life these are these are not all

00:33:50

decades the two in the middle or half

00:33:53

decades kind of thing in fact a unit of

00:33:56

time I think about or try to think

00:33:58

about is decades now kind of of thing

00:34:02

and I spent the first decade at UCSD

00:34:06

and also because my my wife by this

00:34:09

time was resident surgery I had plenty

00:34:13

of time so I had two jobs and still

00:34:15

felt like I was loving and and I we

00:34:20

built some systems I'll tell you a

00:34:22

little bit about them in a minute in

00:34:25

that I left to go to to head up the

00:34:29

human interface lab at MCC we've got

00:34:31

some more systems and and did some

00:34:33

other work. And then I left there to go

00:34:35

to start a group at Bellcore and left

00:34:39

their to feature of the computer

00:34:42

science that you wish to Mexico and

00:34:44

then finally boomerang back to UCSD so

00:34:48

the early research at UCSD again it was

00:34:52

the two body problem the really led me

00:34:54

to UCSDC only only place my wife was

00:34:58

accepted into resonance Ian surgery and

00:35:02

but it turned out to be a really really

00:35:05

amazing place in those days I mean it

00:35:07

was I was so lucky and so fortunate not

00:35:13

the least of which I started to learn

00:35:15

to surf which was also incredibly

00:35:18

important a vent but it was a

00:35:22

intellectually it was the most

00:35:24

productive interesting environment I

00:35:27

think that I've ever been that was as

00:35:28

great collaboration between don Norman

00:35:31

and Dave promo for and and then J

00:35:34

Mcclellan joint and we were sort of

00:35:37

transitioning to form the first

00:35:40

cognitive science laboratory there it

00:35:44

was really the beginnings of cognitive

00:35:46

science the about time I got there was

00:35:50

the first cognitive science conference

00:35:52

which was held at UCST it was the first

00:35:56

mental models conference was held there

00:36:00

at UCSP we had this wonderful Sloan

00:36:05

foundation postdoc program where we

00:36:11

were able to attract this whole host of

00:36:14

just incredibly enlightened postdoc

00:36:19

some aimed and Hutchins was one of the

00:36:22

just handing was another one and it

00:36:25

went on and on we had this amazing set

00:36:27

of graduate students every cell and

00:36:30

build behaviour bob clash go Jonathan

00:36:34

routine Michael Jordan Mike knows your

00:36:36

and on a on it was the beginnings of

00:36:40

this whole connection is movement and a

00:36:43

very different view of cognition the

00:36:47

the original parallel distributed

00:36:49

processing books we're being written

00:36:51

and and Francis crick the Nobel

00:36:55

laureate used to sit in our meetings

00:36:58

and berate everyone about how little

00:37:01

they knew about the brain and how

00:37:03

foolish other models for but it was

00:37:07

just an intellectually incredibly

00:37:09

stimulating kind of environment. And

00:37:12

the same time I was able to maintain

00:37:15

ties to park and what was going on

00:37:18

there and and be a consultant and I

00:37:20

convinced the navy to give us a bunch

00:37:22

of money to buy these really expensive

00:37:24

list machines and and we build a number

00:37:28

of systems well one of them was

00:37:31

navigational training system and and

00:37:35

what we're really interested in is I

00:37:37

think in the the slider so the the

00:37:39

buzzword of the slogan of the day was

00:37:42

conceptual vitality rather than

00:37:43

physical vitality. So it was a time of

00:37:46

mental models there was a time of

00:37:48

concern for how people thought about

00:37:50

these things. And could we build

00:37:52

systems that aided the the the building

00:37:55

of those models but not the models that

00:37:58

you get in physics textbooks the models

00:38:00

use you think with and so we build a

00:38:03

series of the of navigational training

00:38:06

systems and and the some of them on a

00:38:11

little computer called a Tara

00:38:15

microcomputer reasonable Ellis about

00:38:17

living computer built in Arizona that

00:38:21

was one of the first that that really

00:38:23

inexpensive displays we also I was

00:38:26

involved what can balls and we bought

00:38:28

you see a C Pascal this really

00:38:29

interesting us no system we had some of

00:38:34

the first per computers which you

00:38:37

probably never heard of the other done

00:38:39

by a spinoff of CMU and it informally

00:38:44

called Pascal toes because they were

00:38:46

like altos but where and Pascal and

00:38:50

these are pretty successful our train

00:38:53

systems on the little micro with on

00:38:56

every ship in the navy and I had we had

00:39:00

a I had you know she should taught

00:39:04

these kind of navigational things for

00:39:06

fifteen or twenty years have lightbulb

00:39:09

experiences "'cause" they had taught

00:39:11

these procedures by rote. And suddenly

00:39:14

they could interpret lots of things we

00:39:16

did was mapping between

00:39:17

representational system so kind of

00:39:19

relative system that you get on a radar

00:39:22

with the geographic kind of map based

00:39:23

system kind of thing the per thing we

00:39:27

built a bunch of tutoring systems on

00:39:31

and I installed one of these on a on a

00:39:35

new nuclear carrier that was just

00:39:38

getting commissioned other cars vents

00:39:40

and with I still remember captain

00:39:43

martin who was the commissioning

00:39:44

officer for that ship it was very

00:39:49

strange to install it on that so we we

00:39:52

don't use systems that we're really

00:39:55

primarily cognitively motivated

00:39:57

motivated by trying to to help people

00:40:00

build the kind of models they needed

00:40:02

treason about things. Uh we also build

00:40:05

a a system called steamer which so do

00:40:10

we whites when when the guys who worked

00:40:11

with this this is on a you might use

00:40:14

this machine a symbolic celeste machine

00:40:16

the the display on the the left is sort

00:40:21

of a dynamic graphical view of this

00:40:24

complex dynamic system there are lots

00:40:27

of really good ideas we could see the

00:40:29

it show the flow in the system you can

00:40:31

see the causal topology you could could

00:40:34

you would many different levels we

00:40:37

built some of the the the first things

00:40:40

of qualitative graphical components so

00:40:44

lots of responses and systems like this

00:40:47

respond to rates of change to to the

00:40:48

derivatives of things so we built these

00:40:51

derivative icons were just arrows

00:40:53

support it up or down and the size of

00:40:55

the all said how fast they were

00:40:56

changing and you can step through the

00:40:58

simulation. And have this graphical

00:41:00

instantiation of what seemed very close

00:41:03

to what experts would discuss in the

00:41:07

way they would describe the system so

00:41:10

we we we have a lot of fun and and

00:41:14

about the time we we sort of step back

00:41:17

from this which I think is a a really

00:41:19

good thing and and add and don and I

00:41:22

wrote this regional direct

00:41:24

manipulations of paper we we were

00:41:27

responding a little bit too been

00:41:29

Schneider men's direct manipulation

00:41:31

paper which was very influential but we

00:41:35

saw it more is a checklist kind of fury

00:41:38

and we wanted a much more cognitive

00:41:41

theory of what was going on and so we

00:41:44

tried to to do that and and first

00:41:47

instance so what was in this user

00:41:49

centred system design but which was

00:41:52

named that because we could get the

00:41:54

UCSCD acronym and and I was a very I

00:41:59

think influential book and the whole

00:42:02

movement to sort of user centred design

00:42:04

I think was influenced by so the and

00:42:07

the lesson sort of learn from that face

00:42:10

is you know that I'm just as more and

00:42:17

more convinced that computation assist

00:42:19

plastic representational system and and

00:42:24

and the work there is a convinced me

00:42:28

even more I was sort of also very taken

00:42:31

by how interesting and fun it was to

00:42:36

build systems and then transition them

00:42:38

into use and and that that was both you

00:42:43

know incredibly challenging but but but

00:42:47

also I think it it leads to better

00:42:49

science if you're able to do that I

00:42:52

mean you have to you have to deal with

00:42:54

issues that you might avoid otherwise

00:42:57

and and the and I think it continues is

00:43:04

is sort of a this kind of relationship

00:43:09

do you know that got introduced to me

00:43:11

by ad Hutchins were and spend a lot of

00:43:14

time in the balance of navy ships

00:43:18

looking and observing how people really

00:43:21

operate "'em" And just how valuable

00:43:25

that was two systems builders kind of

00:43:28

thing it's really one of the big

00:43:30

lessons of my career and yeah I thing

00:43:36

we built this this graphics editor one

00:43:39

of the first object oriented graphics

00:43:41

editors for building interfaces

00:43:45

interactive dynamic interfaces. And I

00:43:49

was incredibly struck by how powerful

00:43:53

building tools to build interfaces all

00:43:55

were and and how you could leverage

00:43:58

that in fact at the time we've thought

00:44:03

for a momentarily of doing a startup

00:44:06

before that kind of thing was really

00:44:08

problem and and and the scary thing is

00:44:12

the company was willing to give us a

00:44:14

bunch of money to do that in the idea

00:44:18

was to use this graphics editor to to

00:44:21

build instances of these kinds of

00:44:23

interfaces for all kinds of things. You

00:44:25

know nuclear power other kinds of

00:44:27

complex dynamic systems but we we

00:44:30

quickly realise that we would move from

00:44:32

a research career to a different career

00:44:34

and and ran quickly in the other

00:44:37

directors the the other thing was see

00:44:41

the importance of of oh I'm sort of

00:44:47

really abstracting out the cognitive

00:44:51

design principles I think this direct

00:44:53

manipulation paper was something that

00:44:55

so that taught me that you need to do

00:44:58

this kind of building kind of stuff but

00:45:00

then you have to step back. And really

00:45:02

abstract out what you've learned from

00:45:04

but and and then the other thing that

00:45:07

was flight reflecting back on this that

00:45:10

we were just so incredibly naive about

00:45:13

the scale of the project that we took

00:45:16

on I mean this you know huge you know

00:45:19

the first thing we're gonna build a

00:45:20

graphical dynamic interface to is a

00:45:22

team propulsion plant you know just

00:45:25

this very complex kind of thing. And

00:45:28

sometimes and I think in this case it

00:45:31

did that lead you to do more important

00:45:35

things and had we started much smaller

00:45:37

so anyway so I left UCSD about this

00:45:41

time. Um due to listening to the advice

00:45:45

of don norman. Um not always the best

00:45:48

thing to do a and I had been consulting

00:45:52

at MCC for a while and it was a very

00:45:55

interesting is research thing really

00:45:58

wanting to do long range research a a

00:46:00

lot of money a lot of good people and

00:46:04

and as so and they offered me the job

00:46:09

would being director this really large

00:46:11

three forty five PHD lab which I didn't

00:46:16

wanna do it anyway and and so but I

00:46:20

really wanted to keep consulting

00:46:21

"'cause" it was really interesting that

00:46:23

lotta list machines and they were doing

00:46:24

interesting things lasted on a well

00:46:27

what should I do how can I say no no

00:46:29

way that to keep me consulting swell

00:46:32

simple gem just ask for a lot of stuff

00:46:37

ah and and that sounded like recent. So

00:46:41

I asked for all these sort of pretty

00:46:43

outlandish stuff in my view. And they

00:46:45

said sure and and so I was like I

00:46:50

didn't know what to do and and it

00:46:51

seemed like an opportunity then that it

00:46:53

didn't seem like before. And so I went

00:46:56

off to MCC and and we started building

00:47:00

sort of building on what we've done

00:47:04

with this graphics editor and but

00:47:07

extending it to be tools to build

00:47:09

multimodal interfaces long before many

00:47:12

people were ever thinking about

00:47:13

multimodal interfaces and so we're

00:47:17

building tools a two construction of in

00:47:20

a natural language interfaces and

00:47:22

graphics and gesture and intelligent

00:47:25

systems that we give advice and and it

00:47:30

was sort of you know that I'm that he

00:47:32

did a I and we were not unnecessarily

00:47:35

well convinced that that was the right

00:47:37

way to go but we really an average of

00:47:39

representational languages and so we

00:47:43

get a lot of interesting things of

00:47:45

representing the whole system in this

00:47:47

representational language so only

00:47:49

interfaces but the tools themselves had

00:47:51

representation and and the main idea

00:47:55

that was sort of driving me and driving

00:47:57

the development was an idea that I

00:48:00

think is still has a lot of of credence

00:48:03

and the idea was that somehow when

00:48:06

you're building systems you have to get

00:48:08

from this bit shuffling lateral level

00:48:09

of computing two levels that people

00:48:12

really care about and most of the time

00:48:15

we think about doing that and maybe one

00:48:17

step you know we build an editor or we

00:48:20

build one kind of development

00:48:21

environment and stuff. And I think the

00:48:23

normal idea we had was that it's really

00:48:26

going to take a series of tools. And

00:48:30

that the and that the important thing

00:48:32

was that those tools were all connected

00:48:34

to each other. And sort of live. So

00:48:37

that you could go in and make the

00:48:39

changes and appropriate level for the

00:48:43

thing it would be for the whole system

00:48:46

so as you moved up the tool chain you

00:48:48

would empower people with different

00:48:50

levels of expertise and but you we keep

00:48:53

the whole thing tied and I still think

00:48:55

that a really interesting and valuable

00:49:00

idea. And I think we get a well I was

00:49:03

at TMCC is my car broke down as scene

00:49:08

always have cars involved and and I'd

00:49:13

been out to get our Marvin Minsk keys

00:49:15

to come down to this it Dublin at at it

00:49:19

MCC and we would sometimes go out to

00:49:22

dinner like Travis it is mexican place

00:49:26

there and and sort of argue well I mean

00:49:29

both Marvin and don't love to argue and

00:49:32

and and and you know is bringing Marvin

00:49:37

back one late night from there and and

00:49:40

we broke down out on this you know just

00:49:42

the most deserted road you can ever

00:49:44

imagine only Texas can you so the

00:49:47

imagine these kind of really deserted

00:49:49

roads and anyway long story I I ended

00:49:52

up in this auto repair shop yeah the

00:49:55

next day getting my car fixed this all

00:49:58

Portia I folks in the lab had give me

00:50:01

this bumper sticker my other cars a

00:50:04

piece of junk to and which I didn't

00:50:08

appreciate and I was there are like a

00:50:12

looking at and the thing that

00:50:13

fascinated me I just became enraptured

00:50:16

by these manuals and the thing that was

00:50:20

interesting about them is mechanics

00:50:23

said dirty hands. And when they when

00:50:26

they deal with these manuals the the

00:50:28

places that they deal with the most get

00:50:30

the dirtiest. And and and I I became

00:50:33

really taken by that notion that people

00:50:36

do things. And and it has these kind of

00:50:39

representational consequences and then

00:50:42

you sort of start to see it everywhere

00:50:44

you see past for more people just walk

00:50:47

and it leaves these kind of things that

00:50:49

you look start looking at your

00:50:50

paperback books and they they sort of

00:50:52

open to the place you last read if you

00:50:54

sort of break the spy and and so we we

00:50:58

we did a lot of work we got sort of

00:51:00

taken by this notion we do a lot of

00:51:01

work on what we called in the day and

00:51:04

it where read where kind of thing and

00:51:08

and or more generally sort of history

00:51:11

enrich digital objects. And we've got

00:51:13

some interesting I think well ways of

00:51:16

visualising that we we build is what we

00:51:19

call attribute map scrollbars so that

00:51:22

we could represent that history of who

00:51:25

edited maybe how long they took and

00:51:27

today the co editing of stuff whether

00:51:29

it was you know as long before the days

00:51:32

of you know track changes in Microsoft

00:51:34

word we modify the old version of Emacs

00:51:37

to to do all of this kind of recording

00:51:40

and and was really I think really nice

00:51:42

to make it in the scroll bar because

00:51:45

you used real estate that was already

00:51:47

there and it you just touch the place

00:51:50

you wanted to look at and you were

00:51:51

taking their so it was really good and

00:51:54

we we got really involved with that so

00:51:58

we had this idea of menu where so you

00:52:01

know the history of your use of various

00:52:03

features in the system should should

00:52:06

show their we had buffer where so how

00:52:10

much time you spend on buffers well we

00:52:12

had are read where in our email readers

00:52:16

so you could tell us how long so so one

00:52:19

of things we played with was this kind

00:52:21

of messages that might go to everyone

00:52:23

in a research group then your way

00:52:25

you're coming back when it be nice to

00:52:27

see how long an average people in your

00:52:30

group spent Reading those so you might

00:52:32

prioritise things in terms of the

00:52:34

things that people spend a lot of time

00:52:36

really today we got really an hammered

00:52:39

by the notion and and I still think

00:52:42

this sort of general notion of of in a

00:52:47

computational medium where you can

00:52:49

capture kind of activity and and then

00:52:52

re represent that in ways that informed

00:52:54

tasks and do things so you're not

00:52:56

asking people to do anything so that

00:53:00

the lessons learned is first lesson I

00:53:02

learned MCC was I had no skill and be

00:53:06

able to manage forty personal out my my

00:53:11

whole management philosophy up at that

00:53:13

time was jump in the car go to lunch

00:53:16

and talk about things. Uh and and you

00:53:19

know walk around. And chat with people.

00:53:22

And suddenly a sort of realise that no

00:53:24

that doesn't scale and that was a real

00:53:27

a lesson learned a lot for me again I

00:53:31

was sort of the lesson was that the

00:53:33

building a system is really a great way

00:53:36

to integrate research. And integrate

00:53:38

people's activity and keep you on on

00:53:41

course and it focused things in a way

00:53:44

that I think most of the the successful

00:53:47

research projects I've been involved in

00:53:49

the bin tightly tied to building

00:53:52

something building the system. And

00:53:53

using that as an integrating kind of

00:53:55

thing I think this tool chain idea is

00:53:58

still worthy of a lot of consideration

00:54:01

that we're still trying to jump these

00:54:04

massive gaps in building systems and we

00:54:08

we try to do with the programming

00:54:09

language or we try to do with one tool.

00:54:12

And I and I think in tool we have a

00:54:14

series of tools like that we won't be

00:54:17

able to do it well and this chi idea

00:54:21

about a history enrich digital objects

00:54:23

that that that you just let people do

00:54:26

what they normally do. But but because

00:54:28

there do you know a computational

00:54:30

environment you can you can re

00:54:33

represent that in ways that will inform

00:54:35

I think that's a very very powerful

00:54:37

kind of idea and again I'm further and

00:54:41

further convince that computation just

00:54:43

plastic medium. So I left MCC in austin

00:54:48

Texas reluctantly in some ways you know

00:54:53

in CC was a fairly successful but as we

00:54:56

see all too frequently and long range

00:54:59

research places they seem to slide

00:55:02

backwards into being concerned was

00:55:05

shorter and shorter time frames and and

00:55:09

and the CC the CI lab were starting to

00:55:12

do that and I wasn't interested in that

00:55:15

so I left and I I formed a a group at

00:55:18

Bellcore colour computer graphics and

00:55:21

and interactive media group. And all

00:55:24

speed up a little bit here so it

00:55:27

finished on time it was right at the

00:55:29

time where we were really interested

00:55:32

these dynamic graphical kinds of

00:55:34

interfaces it was really clear to

00:55:36

everyone that inexpensive graphic cards

00:55:40

were on the horizon and suddenly every

00:55:43

PC was going to have the kind of

00:55:45

graphics that we only had on high an

00:55:48

SGI machines a list machines and very

00:55:51

very expensive machines. So many many

00:55:53

people got very involved this was a

00:55:56

time is week card we're starting up

00:55:58

work at park on the information

00:56:00

visualiser well all this concern with

00:56:03

with with with how we can so the

00:56:05

building dynamic interactive three D

00:56:08

kind of interfaces and we we started

00:56:10

building such a system ourselves that

00:56:12

we had originally called art and the

00:56:15

lawyers got involved every time the

00:56:17

lawyers get involved is bad and and

00:56:20

they similar commercial systems call

00:56:22

that you can't call it art and I said

00:56:25

well okay well we'll call it a are

00:56:27

three T and you know "'cause" three D

00:56:30

three T and then we'll tell everyone

00:56:32

that the three silent and and it's

00:56:36

pronounced are kind of thing and and so

00:56:39

we're building this kind of very

00:56:41

interesting system and visualising big

00:56:44

software systems and and kind of things

00:56:48

and then I sort of went up with can

00:56:50

Perl and who's a professor at NYU and

00:56:54

can have just put together this sort of

00:56:57

very kludge together and then L of a

00:57:01

sort of multi scale environment to

00:57:03

tinkle pat and and I thought it was

00:57:08

just exactly on the right course that

00:57:12

is very hard to navigate in three D

00:57:15

easy to get lost a three D leads you to

00:57:18

this kind of imitation kind of you

00:57:21

trying to make it like the real world.

00:57:23

Uh in two D sort of multi scale was

00:57:27

looming seemed like a a really

00:57:29

interesting thing. And and so to us in

00:57:34

the group a lot of the group are still

00:57:35

very interested in three D we did some

00:57:37

stuff to make every polygon sort of

00:57:40

zoom mobile and and the two D kind of

00:57:42

surface but I just hired a a young guy

00:57:46

from and why you a guy doing a robotic

00:57:50

vision name bin peterson. And and we

00:57:54

convince them that he should get away

00:57:56

from that and become an interface guy

00:57:58

and and then I became totally

00:58:02

enraptured by the zoom a kind of of

00:58:05

metaphor. And so we started building

00:58:07

this had plus plus system sort of it in

00:58:11

homage to to to cans system and and

00:58:14

about the time I I decided that the

00:58:17

billboard again was maybe not looking

00:58:19

for had enough and that kind of

00:58:21

research they were funding and was

00:58:23

maybe time to go back to the academic

00:58:26

environment so I moved to the version

00:58:27

Mexico and and then came with me there

00:58:31

and and and then I convinced darpa to

00:58:34

give this a bunch of money and we

00:58:36

started building this possible system

00:58:38

which I think was really influential to

00:58:43

one of the first sort of zooming

00:58:45

interface there are lots of interesting

00:58:47

ideas about it that notions of semantic

00:58:51

zooming. So instead of just doing some

00:58:53

geometric scaling we did semantic kinds

00:58:56

of things you got closer to you might

00:58:57

get a very different oh kind if you we

00:59:00

get a lot of work with the lenses sort

00:59:04

of influence by some of the park work

00:59:06

and but also you know the the idea had

00:59:11

a postdoc came to study would maybe one

00:59:14

of actually cans can students did this

00:59:18

really interesting thing on sort of a

00:59:20

principal way of of composing lenses

00:59:23

and you know one way of thinking about

00:59:25

lenses is doing the retrieval is very

00:59:27

natural way of thinking about it. And

00:59:30

and we also thought about using lenses

00:59:33

or something you reached room. And and

00:59:35

when you reach through them you can do

00:59:37

various things. And being able to move

00:59:39

those ones around with the non dominant

00:59:41

hand and reached through an effect with

00:59:43

the dominant hand was very powerful

00:59:45

uses bins file system and the day so we

00:59:49

consume in I actually yeah I don't have

00:59:53

many good images from those days but

00:59:55

that's the best I could do okay so we

00:59:57

did a lot of of that and I when I moved

01:00:01

to you and am I got this big contract

01:00:04

from darpa and I made some contract to

01:00:05

sort of support my group back at

01:00:07

Bellcore which George furnishing taken

01:00:09

over and and also can and just rooted

01:00:13

NYU and then George move to Michigan in

01:00:16

so we did a contract with with George

01:00:18

there another thing we did was this

01:00:24

beyond being their work and and this we

01:00:28

just hired bulk or as this young

01:00:31

physicist from Stanford us concern that

01:00:34

and and Scott nice really hit it off

01:00:39

and and and and we both like going

01:00:43

around seeing all the stuff going on at

01:00:45

Bellcore today and we just a lot of

01:00:47

work on video teleconferencing I mean

01:00:51

the I think the phone company sort of

01:00:53

foolishly thought that the way the

01:00:55

future was moving gets a you know kind

01:00:59

of thing and and and what we're both

01:01:02

struck by is is is how much the focus

01:01:06

was on imitation you know the the the

01:01:09

the goal was really being there you

01:01:11

know that that you know that the way

01:01:15

the telecommunication problem was going

01:01:16

to be solved by was is was it was

01:01:19

establishing the sense of being there

01:01:22

and and and it was it pretty clear that

01:01:26

you could throw more more money at it

01:01:27

and get higher resolution graphics and

01:01:30

you could full duplex audio and you can

01:01:32

get closer and closer to being there.

01:01:35

But what struck Scott my bosses that

01:01:37

that maybe that was just fundamentally

01:01:40

wrong that you know you were trying to

01:01:42

imitate face to face kind of physical

01:01:46

proximate reality in this computational

01:01:48

medium and and and and the focus

01:01:52

usually imitation it takes you away

01:01:55

from doing anything that is distinctive

01:01:57

and that you can only do in the

01:01:59

computational medium and so we we

01:02:02

started this beyond being their project

01:02:05

which we had originally called better

01:02:09

than reality but that was little too

01:02:13

much even for us and and and and looked

01:02:17

at that and and and and lots of

01:02:19

different ways and I think learned a

01:02:22

lot from what the the lessons really

01:02:24

from Bellcore whatever reflect back on

01:02:25

is is that this idea of imitating one

01:02:29

medium and another is just you know not

01:02:31

the best research approach I think we

01:02:34

found that both and had plus plus and

01:02:37

also and beyond being there and and I

01:02:42

think for me one of the things that

01:02:44

comes out of it is a kind of research

01:02:47

strategies I recommend for students is

01:02:50

is if you spend a lot of time when

01:02:52

you're Reading papers or when you're

01:02:54

considering projects really focus on

01:02:57

the things that people are saying you

01:02:59

know one of the presuppositions what

01:03:02

are sort of wonder people sort of

01:03:04

presuppose a is known controversy le

01:03:07

something you don't even have to say

01:03:10

kind of thing and I think that that's a

01:03:12

a very powerful research strategy. Um

01:03:15

you know beyond being their lead to

01:03:20

lots of interesting things maybe we did

01:03:22

this kind of work on ephemeral interest

01:03:24

groups and and some asynchronous

01:03:26

discussions and we did some of the

01:03:28

original collaborative filtering kind

01:03:31

of work in those days and really

01:03:33

recommend or systems and and and second

01:03:36

I did some follow on work that we

01:03:37

called and negotiated access which I I

01:03:41

still think is a is a more and more

01:03:44

important topic that we try to get we

01:03:47

we got a patent assigned to a a

01:03:49

nonprofit so that someone else could

01:03:53

take the idea and and you know it's

01:03:57

always further and further contents

01:03:58

that this plastic representational

01:04:00

medium is that so anyway so I I I hate

01:04:05

back after being at Bellcore in and you

01:04:07

and am to UCSD and and I start a lot of

01:04:15

projects I can't really talk about some

01:04:17

here. Um but what I mentioned a few of

01:04:21

them but mean wanna say that we and

01:04:23

then I started this this unique

01:04:26

cognition an HCI lab which we wanted to

01:04:30

promote this shift in cognitive science

01:04:34

toward this broader view of cognition

01:04:36

is really really a property of systems

01:04:39

and and and that kind of functional

01:04:44

approach I think is really really

01:04:46

valuable extending the boundaries of

01:04:48

cognition beyond the cranium to include

01:04:51

other people artifacts and and and all

01:04:54

those as and and sort of the the the

01:04:57

central image for all of our work is

01:04:59

really no that we want to build

01:05:02

environments in which people pursue

01:05:04

their activities and collaboration with

01:05:07

the elements of the material world are

01:05:09

kind of thing to to look at that and to

01:05:12

do that you really have to know what

01:05:14

people do. Um how they coordinator

01:05:17

activity and and and really to be

01:05:20

really thoughtful about what role

01:05:21

technology should play in doing that.

01:05:25

And so one of the first projects I got

01:05:27

involved in was my my daughter just had

01:05:30

a automobile accident she she fell

01:05:33

asleep coming home from our boyfriends

01:05:36

late at night and cross the median had

01:05:39

a head on collision fortunately she's

01:05:41

okay fortunately everyone was okay from

01:05:44

but but sort of got me thinking about

01:05:46

cars and driving and and and centre

01:05:49

realising the cars are becoming

01:05:51

computer platforms and and why think

01:05:53

them often attracted to a sort of times

01:05:56

when you know new interfaces are

01:05:59

forming in the technology so attracted

01:06:01

to digital pens and those kind of

01:06:03

things "'cause" there isn't this

01:06:04

establish basin there's you can do new

01:06:06

things and in cars seem to be like that

01:06:09

that you the other lots of new

01:06:12

interfaces happening so I linked up

01:06:15

with the guy intellectual computer

01:06:16

engineering Hunter valley and we were

01:06:18

to speak proposal. And we started

01:06:21

looking at driving so we we bought a

01:06:25

car we instrumented with ten cameras so

01:06:29

we could look at what people really do

01:06:32

we had cameras on people's heads we had

01:06:35

cameras looking at them we had a media

01:06:37

cameras on the top we had the the first

01:06:41

ever foot cam so we could watch what

01:06:44

people were doing with their feet we

01:06:46

also we're interface to the bus of the

01:06:48

car so we hear all of the details out

01:06:51

of the car in terms of what it was

01:06:53

doing and the data sort of look like

01:06:55

this. Uh kind of thing where we were

01:06:59

actually digitising all this

01:07:01

information and the trunk of the car

01:07:05

and you know here you at the top left

01:07:08

you see this very first person view

01:07:10

from a camera on the forces had if you

01:07:12

look down we have other cameras this is

01:07:15

the first time this guys ever driven

01:07:16

this car you'll note that he has one

01:07:19

finger on the steering wheel a little

01:07:21

bit later probably won't get to it that

01:07:24

he almost gets in an accident you

01:07:25

briefly touches the steering wheel was

01:07:28

other hand and then starts joking about

01:07:31

this is I'm probably the first person

01:07:32

that almost wrecked your car kind of

01:07:34

thing. And we got interested in

01:07:36

interfaces and force feedback

01:07:38

accelerator panels and and using

01:07:41

dramatic things in the C provide

01:07:43

information but what I really got

01:07:45

interested in is is I I think I saw a

01:07:48

bit of the future here that that it's

01:07:52

going to be this kind of data that's

01:07:53

gonna drive design a kind of thing and

01:07:56

so I really got interested in building

01:07:59

a system to to eighty that kind of

01:08:03

analysis and so one of my students adam

01:08:05

files me and a bunch of other people

01:08:08

build a system called product is which

01:08:12

is really designed to eighteen

01:08:16

fieldwork eight ethnographic kinds

01:08:18

investigations eighteen things where

01:08:20

people have rich time based data be at

01:08:23

video be a simulator data kinds of

01:08:27

things and so this is an image from

01:08:29

some work well with and was doing with

01:08:32

bowling so we have a mobile video

01:08:36

streams this is from a high fidelity a

01:08:39

simulator and so we're watching pilots

01:08:42

in various tasks there we have multiple

01:08:45

sources of video we provide really

01:08:48

interesting facilities for annotating

01:08:51

that kind of thing you're trying to

01:08:52

understand it you have to support how

01:08:54

those how those kind of conceptual

01:08:56

stations are ways of annotating it

01:08:58

growing change over time we in this

01:09:03

case I can link it up to geographic and

01:09:06

spatial data we have transcript data

01:09:09

that being like and we have a notable

01:09:13

pen digital don't that people are

01:09:16

taking and on this on this is sort of

01:09:20

time stamp so we can we can link it

01:09:22

together and and so we can sort of go

01:09:28

from the you know for example from your

01:09:32

digital and yeah can note to video so

01:09:35

you can actually touch your paper with

01:09:37

the envelope in have a drive the video

01:09:40

to what was going on at the time you

01:09:41

took that what we find for it's not the

01:09:45

first is this kind of facilities change

01:09:47

the kind of notes that people take that

01:09:50

suddenly you can do things like just

01:09:52

say interesting you know we'll go back

01:09:55

to or that kind of thing and rather

01:09:58

than have you putting will get other

01:10:00

folks looking at other kinds of things

01:10:02

can record really detailed timing of

01:10:05

those events just by putting dots on

01:10:07

the paper kind of thing. And sort of

01:10:10

running on a little bit short time I

01:10:12

was gonna show you a a bit about video

01:10:15

where we've implemented I tracking in

01:10:20

this kind of thing again it's another

01:10:21

timestamp kind of data. So we can do

01:10:24

things for example like look at all the

01:10:27

fixations on an instrument. And then

01:10:29

map that back to the actual segments in

01:10:32

which people were doing that kind of

01:10:36

that kind of thing I think I will skip

01:10:38

over that we and you know we can is and

01:10:43

all kinds of things other people using

01:10:45

it it's it's open source it's available

01:10:47

yet rather biz dot com and and early

01:10:51

we're using it or looking at

01:10:53

interactions in a medical clinic on

01:10:55

south in San Diego and so really

01:10:58

complex kind of a situation where we

01:11:01

have a physician and patient. And often

01:11:04

interpreter because of language kinds

01:11:07

of issues we put a couple of connects

01:11:10

in the environment and we're doing sort

01:11:14

of automatic annotation of some of that

01:11:17

kind of data who's talking who's

01:11:19

pointing when they're when they're

01:11:21

using electronic medical records but

01:11:23

all the goal to sort of ratchet up the

01:11:27

level at which someone trying to

01:11:29

understand this kind of complex

01:11:31

phenomena is sorta placed and so anyway

01:11:36

so we do a lot of that kind of thing

01:11:38

with the curve is and and we're right

01:11:41

now filming pedestrians interacting

01:11:43

with cars and we'll be doing analysis

01:11:46

that we've also done a fair amount of

01:11:48

stuff with the note of digital pens

01:11:50

very much inspired by France walls were

01:11:54

on paper augmented digital documents we

01:11:58

collaborated some work with one of the

01:12:01

students and and I and it's a very very

01:12:07

interesting kind of technology for us

01:12:09

because it links up this kind of paper

01:12:12

based world with the digital world. Uh

01:12:14

and we've done quite a few different

01:12:18

kinds of things and then environment

01:12:20

some work in a a children strongly unit

01:12:26

DC we're we're looking at taking I

01:12:30

usually in the trauma bay there were

01:12:32

sort of code is saying some could come

01:12:34

to some awful kind of think for rush

01:12:37

that trauma bay there's a recording

01:12:39

nurse is taking kinds of information we

01:12:43

thought it would be really interesting

01:12:45

to to to let that person do exactly

01:12:48

what they did so we may we made that

01:12:50

but those things out on another paper

01:12:52

or we can record that we're looking at

01:12:55

projecting that onto displays so that

01:12:58

had a coming in later doesn't have to

01:13:01

interrupt to check on these kind of

01:13:02

things in again it's sort of this

01:13:05

wonderful thing of letting people do

01:13:06

what they normally do in the way they

01:13:08

normally do it. But because they do it

01:13:10

in a computational environment getting

01:13:12

these these added benefits once we've

01:13:14

done a lot of of work in that area with

01:13:16

a lot of what would your white ball

01:13:19

came to to oppose talk with me I was on

01:13:21

the speech D committee your and but my

01:13:26

favourite one of all time to some work

01:13:28

of and re Piper and and Marie was the

01:13:33

the easy student to advise of all time

01:13:37

you mainly just had to stay under her

01:13:40

way and she was always like a couple of

01:13:43

steps ahead of you. You know I would

01:13:46

say I you know I think we should really

01:13:47

do oh yeah I did that last weekend. And

01:13:49

I have those results and oh yeah I

01:13:51

already have a draft of that paper it's

01:13:53

it's three months from now but I have a

01:13:55

draft and and Andreas was also really

01:14:00

good at at speech choosing double

01:14:04

research projects that were nicely

01:14:06

sculpted which is really hard for kind

01:14:09

of thing and so Emery was going off to

01:14:12

a birthday party for great grandmother

01:14:17

who was celebrating a hundred and fifty

01:14:20

per day and so in those are good genes

01:14:25

to have and and so we we thought of

01:14:30

this project we could print out a bunch

01:14:32

of images of another paper should take

01:14:36

a bunch of cans with your and and and

01:14:40

people could point and talk about this

01:14:43

kind of things and know get recording

01:14:45

so that her great grandmother could

01:14:47

experience this mainly interested in

01:14:50

family at that point you had a lots of

01:14:52

images honour walls of family but it

01:14:54

was too far away to really see suddenly

01:14:57

if you could open up these kind of

01:14:58

things and interact with this pen. And

01:15:00

it was incredibly touching have these

01:15:03

recordings of one of her eighty year

01:15:06

old daughters talking about an image

01:15:10

and pointy things and and saying how

01:15:12

she remembers that day and remember

01:15:14

what we did after that and and it's in

01:15:17

the voice of of the daughter and so

01:15:22

again a really interesting kind of

01:15:25

environment for for interfaces lately

01:15:30

for the last two years ago and looking

01:15:32

at I what I'm sort of calling activity

01:15:35

enrich computing your again sort of

01:15:37

capturing and mining activity history

01:15:39

some sort of returning to and have

01:15:42

student guess stone county Otto who

01:15:45

enforcing died of cancer right is is

01:15:49

finishing his PHDO dissertation and get

01:15:53

started on this incredibly interesting

01:15:57

study and in a law office immigration

01:16:00

law office and is sort of thought he

01:16:02

was going to do a project about us

01:16:05

software gate collaboration in this

01:16:08

kind of of office environment. And when

01:16:11

I think C started doing just recording

01:16:13

a bunch of screen captures what people

01:16:15

doing see good sort of the movie of

01:16:17

directivity. And he started replaying

01:16:20

those movies for folks and just asking

01:16:23

to talk about what they were doing and

01:16:26

we were both like incredibly struck by

01:16:29

how provocative that seem to be in

01:16:32

terms of people's what they remember

01:16:35

they seem to remember all kinds of

01:16:37

things of what they were doing at the

01:16:39

time someone came into their office I

01:16:41

meet a phone call about directly to

01:16:43

follow up on that and so we both got

01:16:45

enamoured of maybe using that kind of

01:16:49

thing to help people reinstate context.

01:16:52

And so guess to built a system that

01:16:54

well we collectively trails which was

01:16:57

sort of an episodic retrieval system so

01:17:01

you capture all the text on the screen

01:17:03

and everything everyone type. And and

01:17:06

since real task mean we have pretty

01:17:08

good history facilities within

01:17:10

applications but real tasks are spread

01:17:12

across applications and so it would do

01:17:15

this retrieval and and bring you back

01:17:17

episodes you get little thumbnails of

01:17:19

that and then you could replace sort of

01:17:21

the movie of what was going on. And I'm

01:17:24

still we just got some innocent funding

01:17:27

to look at this. This more of this kind

01:17:30

of context reinstatement and I'm I I

01:17:33

think it's a very interesting

01:17:35

conjecture because so so many things

01:17:37

now our computer mediated or so many of

01:17:40

interruptions in there so often

01:17:42

difficult to get back into various

01:17:44

activities and lots of other folks of

01:17:46

of done work in this area but but but I

01:17:48

think it may require less than one

01:17:50

imagines in that kind of thing and the

01:17:53

in the the metaphor I have very much is

01:17:56

that you know if you come to my office

01:17:58

and look at my desk what you see is a

01:18:00

messy desk. And and I probably if I go

01:18:04

to your office I I see a messy desk.

01:18:06

But when I look at it I I see projects

01:18:09

and I see the things I'm gonna take a

01:18:10

when I see okay "'cause" it's my

01:18:12

activity it's my guess kind of thing

01:18:14

and I think that looking at recordings

01:18:17

of your activity is similar to that

01:18:20

when I look at my recordings yeah gotta

01:18:22

running out of time and you know we

01:18:26

need to I I'm sort of thinking of my

01:18:31

intentions what I was doing and why I

01:18:33

was doing it kind of thing and they and

01:18:35

I I think there's this bill would you

01:18:36

do exploit a visual memory. So finish

01:18:39

up not really quickly but I what's a

01:18:41

little bit about the study "'cause" it

01:18:42

really since recognition memory study

01:18:45

it's you know very classic kind of

01:18:47

study that you knew you showing a

01:18:49

series of of images and then you're

01:18:52

asked to do a forced choice a kind of

01:18:54

thing but you've seen something for not

01:18:56

and that they did these really to other

01:19:00

interesting vision but what you know we

01:19:01

just incredibly good that we really

01:19:03

good at visual kinds of things like

01:19:05

that. But they get these two other

01:19:07

conditions one of them was a condition

01:19:09

where it was it was all objects but

01:19:13

you're shown a pair of objects that

01:19:15

were from the same category when you'd

01:19:17

seen before when you hand that kind of

01:19:20

thing it's still people are like you

01:19:21

know eighty ninety percent good at

01:19:24

doing that. But in a sort of killer

01:19:26

condition for me was you show the same

01:19:29

object in a different state and still

01:19:32

people are like at eighty seven percent

01:19:35

kind of of things so you know seeing

01:19:39

use a picture abacus and then you see

01:19:41

the abacus with the balls in a

01:19:43

different position or that kind of

01:19:45

thing. But I think what that leads me

01:19:47

this is it you know or visual memory is

01:19:49

incredibly good and if we can exploit

01:19:51

data retrieval okay so I'm just gonna

01:19:53

end up with some future challenges and

01:19:56

opportunities and and you know again I

01:20:02

think the opportunity all comes from

01:20:03

this plastic medium kind of thing I

01:20:06

think actually are greatest challenge

01:20:08

is is being locked into the current way

01:20:12

of doing things we surely saw that it

01:20:15

had plus plus would just very hard for

01:20:18

students to think of things other than

01:20:20

in terms of menus and icons and and

01:20:23

that kind of thing the image I often

01:20:26

use of this is is what I actually the

01:20:28

same age it just looks like a I don't

01:20:32

know what but if I outline it. Um to

01:20:35

show you a cow. And I go back and forth

01:20:38

a couple times then you can see this

01:20:42

without seeing the cal. And I think

01:20:44

that we you know it's equivalent to

01:20:47

lots of kinds of things for interfaces

01:20:48

we just lot into those kind of notions

01:20:52

so the you know the I I you know in one

01:20:58

sense you know really excited about

01:20:59

things but then once it's it's sort of

01:21:01

sad to see how little progress in some

01:21:04

sense it's been made you know almost

01:21:07

everything we do with computers is

01:21:09

difficult and awkward and frustrating

01:21:12

you know even after like what it's been

01:21:15

six decades of evolution. There's just

01:21:18

little of the naturalness of and the

01:21:21

contextual sensitivity of of working

01:21:24

with other activities and you know

01:21:27

there's not this can the reality that

01:21:29

we have with working with paper another

01:21:32

kind of tangible mediums the crucial I

01:21:37

think in this is the unit of analysis

01:21:40

that we don't I think you know and

01:21:42

science the universe analysis is

01:21:44

crucially important. And I think one of

01:21:46

the key inside to distribute condition

01:21:49

is to be flexible about in you know of

01:21:52

analysis you know sometimes the

01:21:54

individual is the right unit of

01:21:55

analysis but often. It's not and and I

01:22:00

think for maybe too long I think

01:22:02

israel's very much changing today our

01:22:05

focus has been on the individual the

01:22:08

other thing I think very much is that

01:22:10

we still you know this presuppositions

01:22:13

we have about information about

01:22:16

applications. I think I think we have

01:22:19

some very unfortunate presuppositions

01:22:21

about that I think that we're sort of

01:22:26

trapped in this spiral currently of

01:22:28

apps we're we're just building more and

01:22:32

more perhaps for things. Uh and that

01:22:36

and I don't think that that's going to

01:22:38

be the way to go to support real

01:22:41

convince real friendly kind of

01:22:43

environments were were sort of left to

01:22:45

break everything into the right units

01:22:47

of apps and then re place things back

01:22:51

and and I think that we're also one of

01:22:54

the other presuppositions is is that

01:22:57

information is basically get bits I

01:22:59

mean there's I think there's some work

01:23:01

I mean you know go now kind of things

01:23:04

where things are sort of context

01:23:05

sensitive and stuff. But but if I had

01:23:08

to put my money and we're to invest. I

01:23:10

would invest in building a a priori

01:23:13

different informational infrastructure

01:23:16

you know that I want to think of this

01:23:18

being sort of a cognitively inspired

01:23:21

physics for information that that that

01:23:24

physics it is such that the information

01:23:26

is more active. And and context

01:23:29

sensitive and that and that the we

01:23:31

build things on top of that kind of

01:23:34

thing. So anyway. So just wanna end up

01:23:37

and you know we're interest in doing

01:23:39

some of those things at the new design

01:23:42

lab at San Diego and many other kinds

01:23:45

of things. And and and I just wanna

01:23:49

think you know just all of my

01:23:51

collaborators and students and and

01:23:54

people have been involved and and one

01:23:56

thank you for your attention and it's a

01:23:58

true honour yeah oh and we are going to

01:24:20

the straight into the break and now but

01:24:23

will it will be outside if anybody

Share this talk:

Recommended talks

01:17:41

Crossing: HCI, Design and Sustainability
Lou Yongqi, Dean, College of Design and Innovation, Tongji University
April 20, 2015 · 8:45 a.m.

408 views

01:45:09

Embodiment and Language
Francis Quek
Oct. 31, 2013 · 11:08 a.m.

106 views

SIGCHI Lifetime Research Award Talk
James Hollan

Embed

Transcriptions

Recommended talks

Crossing: HCI, Design and Sustainability
Lou Yongqi, Dean, College of Design and Innovation, Tongji University
April 20, 2015 · 8:45 a.m.

Embodiment and Language
Francis Quek
Oct. 31, 2013 · 11:08 a.m.

Klewel SA

What is Klewel?

Follow Us

Contact Us

SIGCHI Lifetime Research Award Talk James Hollan

Embed

Transcriptions

Recommended talks

Crossing: HCI, Design and Sustainability Lou Yongqi, Dean, College of Design and Innovation, Tongji University April 20, 2015 · 8:45 a.m.

Embodiment and Language Francis Quek Oct. 31, 2013 · 11:08 a.m.

Klewel SA

What is Klewel?

Follow Us

Contact Us

SIGCHI Lifetime Research Award Talk
James Hollan

Crossing: HCI, Design and Sustainability
Lou Yongqi, Dean, College of Design and Innovation, Tongji University
April 20, 2015 · 8:45 a.m.

Embodiment and Language
Francis Quek
Oct. 31, 2013 · 11:08 a.m.