Player is loading...

Embed

Copy embed code

Transcriptions

Note: this content has been automatically generated.
00:00:00
Oh it is almost it is I think it is the
00:00:04
last the session oh you event but yeah
00:00:07
I hope that thank you very much for
00:00:09
coming out and then helped injury our
00:00:12
talks okay I I'll mean our conversation
00:00:17
my name is finally I'm I'm working in
00:00:19
commissioner university we we will as
00:00:25
I'll talk about it some some motivation
00:00:27
and then we'll talk about the stand
00:00:31
that it enabler Eugene opportunity by
00:00:33
new traffic and X puppies do really
00:00:36
gonna explode them at icon. And be in a
00:00:39
sense a procession by B to and crossed
00:00:42
and and what what can yeah there is
00:00:45
kind of Intel API ten operations by L
00:00:50
on and then we will have a some kind of
00:00:52
panel discussion on the last part of
00:00:55
our station. K but general motivation
00:01:00
came minus twenty so maybe yeah so you
00:01:04
are you X should be in in a kind kinda
00:01:09
a cushy send some I mean our tight is
00:01:13
battery be says for example I if is not
00:01:17
to and evolution icing the UJUJ
00:01:20
interaction you experience but you
00:01:22
really yeah the revolution should lead
00:01:25
pollution it we can define the it this
00:01:28
leverage and then it should be affect
00:01:31
it issued a changes so many people and
00:01:34
you should be effective so many people.
00:01:36
So vivid that this to be we making this
00:01:41
kind of us circular acquisition simple
00:01:44
examples them if they already so
00:01:45
usually experience then that so it
00:01:49
makes a mac Mckinney then we'll go to
00:01:54
to make bigger market bigger market do
00:01:56
we should have a some kind of standard
00:01:58
API was standardisation what's is
00:02:01
supply. And then it'll be connected to
00:02:05
slick and so we so many companies will
00:02:09
so initially commanders will produce
00:02:11
and actually initially can and then
00:02:13
it'll be the assembled into device
00:02:17
mobile phone and have a smart one and
00:02:21
then the many company will develop
00:02:23
contentions obvious. And then it each
00:02:26
generate on I did the user experience.
00:02:29
And and then and then so on and so so
00:02:32
we believe that standard API can
00:02:39
generate a has big more in in this a
00:02:44
consistent because standard a pay can
00:02:47
provide the populace features and
00:02:49
efficiencies really hype up ones and
00:02:52
and low power consumption and then is
00:02:56
the menu people to walk in to get it to
00:02:59
optimise the features and then into
00:03:02
into one device and so so markedly
00:03:06
those participating in this is ten
00:03:07
valuation activities so it makes little
00:03:11
P confusion number production is
00:03:13
possible so so you billion need till
00:03:17
device on the market and that kind is a
00:03:20
big change it make some another big
00:03:23
market an put related businesses so you
00:03:27
did of course another revolution and so
00:03:31
did this kind of a big market scale
00:03:36
reuse cost endoscope investment for
00:03:38
example you play make some again it
00:03:41
there it get in market there are so
00:03:43
many devices income one device inc one
00:03:46
people on environment then I can lead
00:03:49
use our mildly scanned some kind of
00:03:52
opportunity because I'll a mile
00:03:54
opportunity okay today especially is oh
00:03:58
well we'll talk about some kind of a
00:04:00
foundation that those posting extend
00:04:04
that they PI second thing is the device
00:04:06
is and then and and the so some hard
00:04:11
that imitation econ chips we believe
00:04:14
that these the plantation WJ experience
00:04:17
if you don't have a silly constellation
00:04:19
then it would be we cannot chip that
00:04:21
hype up once user interface all user
00:04:23
interaction is not possible so we'll
00:04:26
talk about this. So I hope that and the
00:04:31
you can get in spite of for you you are
00:04:35
you X using you know pitch it means I I
00:04:38
mean I it is now at the prediction is
00:04:40
near future do we read the members of
00:04:44
course we will produce it's cheap. But
00:04:47
after one O one and happy as of blade
00:04:49
of we will produce that chip you be
00:04:52
installed eighteen euros smart one so I
00:04:55
porno Underwood one will adapt that
00:04:57
technology. So if you if you know what
00:05:00
that some kind all yeah up to to use
00:05:03
this technology will be editing smart
00:05:05
on K active to that this kind things or
00:05:07
something. So please get inspired and
00:05:10
be on a yeah we have one other all that
00:05:13
if you have a very good idea for user
00:05:15
experience please P back to us and we
00:05:17
can that we cannot up that technology
00:05:20
all new ideas to our used and so thank
00:05:24
you very much. But I put the next talks
00:05:27
the extra little buttons standard API I
00:05:29
need to have it I spreading dental in
00:05:32
BT and preaching on top can cross blue
00:05:34
okay Thank you. Thank you want. so I
00:05:44
think we we are in a revolution in user
00:05:48
interface and that revolution is being
00:05:50
driven by a mobile devices and we think
00:05:54
for the human interface community there
00:05:57
many emerging opportunities for
00:06:01
advanced ways users could interact with
00:06:04
the devices and their environment. And
00:06:07
it's because mobile devices are getting
00:06:09
two things because they getting more
00:06:10
sensors to understand their
00:06:12
environments and more processing power
00:06:15
to actually process the data that's
00:06:17
coming from that sensors. So in the two
00:06:20
speakers we have here the next our we
00:06:22
going to look at what all the sensors
00:06:25
what processing do we have how can you
00:06:28
program the senses and use the senses
00:06:30
in applications. And as ones has
00:06:32
hopefully that inspire you to generates
00:06:35
really great user interfaces. It's it's
00:06:38
kind of hard to understate the impact
00:06:41
in society of the mobile revolution in
00:06:45
the last few years the way we interact
00:06:48
with technology with each other. And
00:06:51
are environment has fundamentally
00:06:54
change the mobile phone market is now
00:06:57
easily the largest market ever in
00:07:00
history for a manufacturer device. And
00:07:03
that market volume means that it makes
00:07:06
sense for many companies in the
00:07:08
ecosystem to invest significant amounts
00:07:11
of dollars into research. And
00:07:13
development and so we now have mobile
00:07:15
devices that have the capabilities that
00:07:18
you could only dream of a few years ago
00:07:20
but even if you paid hundreds of
00:07:22
thousands of dollars could not get a
00:07:24
device that has the capability of the
00:07:27
phone that's in your pocket right now.
00:07:29
So that's that's numbers and some
00:07:34
context. So mention two things
00:07:36
processing power. And sensors. So this
00:07:40
is a typical road map this is my own
00:07:41
company but many other soc vendors have
00:07:44
similar road maps the in the last four
00:07:46
years we have increased the amount of
00:07:50
processing power in a typical mobile
00:07:52
and see by a factor of a hundred and
00:07:56
we're now shipping make a flops is
00:07:59
actually now reach the terrible bar in
00:08:03
a mobile device that can ship it in a
00:08:05
tablet for car for phone. And to put
00:08:08
that in the context just in two
00:08:10
thousand the fastest supercomputer in
00:08:13
the world was a terrible. So in just
00:08:17
fifteen years we come from the fastest
00:08:19
supercomputer in the world to the
00:08:21
processing power not in your mobile
00:08:24
device. Imagine what that come
00:08:27
processing power would unleash once we
00:08:29
learn factually do really cool stuff
00:08:32
with it and sensors the other part of
00:08:37
the equation how we can to gather data
00:08:40
or into these mobile devices. So we can
00:08:42
begin to enable the user to interact
00:08:45
with the device. And environment in
00:08:48
include ways. So how we senses do we
00:08:51
have in the smart from today we kinda
00:08:53
counted up every few months county
00:08:55
where twenty two ambient like centres
00:08:58
out sensors pressure sensors the men's
00:09:02
sensors pick several major the gyro
00:09:04
many if you're probably familiar with
00:09:06
but one of the really interesting areas
00:09:08
of innovation I think it's about to
00:09:09
explode in its own little why gives a
00:09:12
conference a cumbersome traditionally
00:09:15
been used for photography capturing
00:09:18
beautiful images they're about to be
00:09:20
useful much more about to be used for
00:09:22
vision processing acquiring data from
00:09:26
the environment cursing. And enabling U
00:09:29
interaction models. So it's not just
00:09:32
pictures anymore it's it's data from
00:09:34
the environment including that later
00:09:36
understanding not just what the like
00:09:39
happen it's but how far away things are
00:09:42
in environment gives the mobile device
00:09:45
much more understanding of what's going
00:09:47
around around it I think this mobile
00:09:51
acceleration of vision it's going to an
00:09:55
able ongoing way so you application and
00:10:00
interaction opportunities you will
00:10:02
probably already taken hang around was
00:10:05
using computational problems we
00:10:07
probably have some some of them even
00:10:09
waved and gestured on mobile devices to
00:10:12
interact with that soon note amount of
00:10:14
processing power and sensor
00:10:16
capabilities going to allow us to to
00:10:18
scan three D models all devices objects
00:10:22
once this room ugly people sitting here
00:10:25
and that's going to take all kinds of
00:10:29
applications and business
00:10:30
opportunities. And then the buzz term
00:10:33
augmented reality something that we
00:10:35
will be maybe for for a long time
00:10:36
understanding the environment and then
00:10:39
super imposing a data on top or
00:10:42
plantations on top the environment
00:10:45
useful wise how people want to have our
00:10:48
man display well we're not quite there
00:10:51
yet but we are making good progress one
00:10:55
of the the key challenges revision
00:10:57
process you know I mean they have a
00:10:58
terrible well no problem solved tunable
00:11:01
continue people the terrible well the
00:11:03
problem is you have to do it in a
00:11:04
mobile device so actually power
00:11:07
consumption is the new a limit to
00:11:10
performance you you can always turn on
00:11:13
the whole terror for the whole time.
00:11:16
And expect to battery to last very long
00:11:19
or a mobile device to get too warm. So
00:11:23
the big challenge now for the silicon
00:11:25
puma T is to be able to to deliver
00:11:27
these levels of performance in a very
00:11:30
small power and blow tablets can have
00:11:34
more power than phones but we going to
00:11:35
go to your augmented reality classes
00:11:38
this is gonna be a real challenge the
00:11:40
fit processors take almost no power out
00:11:44
to fit in this kind of form factor how
00:11:47
we're gonna be using a whole range of
00:11:48
different architectures we have C pews
00:11:52
jeep use for much type kinds of vision
00:11:55
processing could be more power
00:11:57
efficient because they're optimised for
00:11:59
doing parallel pixel operations I think
00:12:03
you begin to see also dedicated
00:12:05
hardware blocks for doing a vision
00:12:08
processing. We can fit so many
00:12:10
transistors on a mobile soc today we
00:12:14
can't turn them all on the same time
00:12:16
within the power and brought up that we
00:12:18
have in a mobile device. So you have to
00:12:21
turn it so the chip on at various
00:12:23
times. So we actually have we what we
00:12:25
called dark silicon. And it means we
00:12:28
have effectively in many cases every so
00:12:31
the chip we could put dedicated
00:12:33
processes into effectively for free.
00:12:36
But we have to figure out how to use
00:12:38
them in a smart way and use them the
00:12:40
appropriate time. But this is gonna
00:12:42
crack a real issue for the developer
00:12:45
community software community will
00:12:47
continue based all these different
00:12:49
types of processors pop up box how are
00:12:52
you going to program them. And software
00:12:56
needs to be a vital part of this power
00:12:59
optimised a solution you won't you
00:13:02
won't be able to conceivable future
00:13:05
Turner can on in a mobile device we
00:13:08
you're what three wearable displays I
00:13:10
just leave it on twenty four hours it's
00:13:13
gonna take too much power battery will
00:13:15
go down too fast. So we need the
00:13:17
software community to help. And smart
00:13:20
applications so if one mobile devices
00:13:23
sitting on my my desk no one's touching
00:13:25
it you don't need to have a camera
00:13:27
going there's no one's looking. So you
00:13:29
need you can use the mens sensors very
00:13:31
low power detect when when the devices
00:13:34
actually be use then you can start a
00:13:37
camera perhaps in low resolution mode
00:13:39
scanning for interesting events but in
00:13:42
a in a low power white only when you
00:13:44
find something interesting turn on the
00:13:46
full power of the prosody have full
00:13:49
resolution of sensors and begin to to
00:13:52
know the full application experience
00:13:57
but again this is gonna take a lot of
00:13:59
ingenuity from the developers if we're
00:14:02
not careful they up to synchronise and
00:14:04
use what his mobile sensors gonna have
00:14:07
a lot of different that income or
00:14:10
technologies to use it to leave rich
00:14:13
right processes at the right time
00:14:15
report everything has to be maintained
00:14:16
coat portable portable across multiple
00:14:20
operating systems or running sixty
00:14:22
hertz. And this is where standards come
00:14:25
in if we don't have standards it's
00:14:28
going to be such a chaotic landscape of
00:14:31
programming interfaces that no one will
00:14:33
be able to make we'll applications. And
00:14:36
the colonel group. Um is an open
00:14:39
standards organisation we're committed
00:14:41
to open royalty free standards we
00:14:44
specialise in the interface between
00:14:47
hardware and software. So when silicon
00:14:50
do cool stuff we define a few eyes. So
00:14:54
developers can get to that acceleration
00:14:57
capability you have a a hundred thirty
00:15:01
members everyone from people apple and
00:15:04
Intel all the way down some some all
00:15:07
the way down to individual developers.
00:15:10
And kernels is the place where all
00:15:13
these people that often compete in the
00:15:15
marketplace income and safely cool
00:15:18
right together to build the market and
00:15:21
and make the world a better place. And
00:15:24
we have about twelve different
00:15:26
standards that are currently being
00:15:28
developed I have we got to have a short
00:15:31
video about a minute. And a kind of
00:15:33
gives you a little glimpse into kind of
00:15:35
user interface scenarios where matching
00:15:38
with community without status because
00:15:41
quickly up like that before kind time
00:15:43
yeah oh oh so apparently only screen
00:16:09
graphics a guide experienced See yes
00:16:13
sorry CLSUGLC yes or things like
00:16:23
actually and that's other a all sending
00:16:33
australians that's between insistence
00:16:38
GL yes that's not that's not under
00:16:40
remotes everybody whites technically
00:16:45
capable it's not just in mobile phones
00:16:48
budget yes is also used in order about
00:16:51
another that it display yeah yes gee
00:16:58
guess if they're not trying to get
00:17:00
this. So just one but so yeah have
00:17:09
children suspended that's you and I
00:17:14
missed that how do I know it's not
00:17:18
something that's something that's great
00:17:21
function chips in hungary. Um BM treaty
00:17:31
process yeah wait a standard hello
00:17:43
vision remove the execution eh you I
00:17:48
take the sixers system wide yes you
00:18:02
sure so and I is actually it's not a
00:18:11
question from every it which is yeah
00:18:18
yes I was wrong so talk things. Yeah
00:18:36
yeah let's see we have to use and maybe
00:18:40
on spec six yeah most of us what you
00:18:49
know we interactive to wear I see oh
00:18:57
see why that's a or I don't know here.
00:19:12
So I anything in sight. So actually
00:19:17
much. So next up is Tom Tom is on track
00:19:24
to graphics research arm and also chair
00:19:27
of both the open GL yes on the bottom
00:19:28
working group so if you need anything
00:19:30
about three feet onto mine Hmmm thank
00:19:35
you know Um so the title of this panel
00:19:41
is how mobile devices are
00:19:43
revolutionising user interaction of
00:19:45
course we could just as well have
00:19:47
called it how user interaction is
00:19:49
revolutionary as plan set at the
00:19:52
beginning it's a loop. Um the
00:19:54
capabilities of the devices and the
00:19:56
capabilities that we expose three the
00:19:58
API is the corners develops define what
00:20:01
is possible is the applications and the
00:20:05
user interfaces and the interaction
00:20:07
techniques that people build on top of
00:20:09
those that makes the devices useful and
00:20:14
in an ideal world beautiful of
00:20:16
satisfying and fun to interact with so
00:20:25
the the story of a mobile graphics for
00:20:30
past decade or so through the
00:20:34
revolution the mail talked about has
00:20:36
been largely the story of open GL yes.
00:20:39
Um this was a project original that
00:20:44
start thousand two to develop a version
00:20:49
of desktop open GL very well
00:20:51
established well known but relative
00:20:53
right to find a version of that that
00:20:56
was appropriate to run on the much time
00:20:59
memory bandwidth and CP budget of the
00:21:04
mobile device. Um we began early with
00:21:09
one point zero series which were based
00:21:13
on the technology that I'm strictly
00:21:15
fixed function API is the model for the
00:21:18
programmer was set state the state
00:21:21
defines completely what happens data
00:21:23
flows through it produces images are we
00:21:26
came up with their first version in two
00:21:27
thousand one that was sized to just
00:21:31
multiple on a mobile CPO but there were
00:21:34
also mobile GP use coming out at the
00:21:36
time and by the time those jeep abuse
00:21:39
at the market tape or about the time
00:21:42
the standard at the market these cheap
00:21:45
use had developed more K expose. So we
00:21:50
did a very quick turn in two thousand
00:21:52
four produced yes one not one to expose
00:21:55
the capability of those devices that
00:21:57
was essentially as a software release a
00:22:01
week can began working on the next
00:22:03
revolution in a mobile hardware which
00:22:06
was the transition from fixed function
00:22:09
graphics hardware to programmable
00:22:11
instead of setting state to flow you
00:22:16
write code that runs on the GPO that
00:22:19
performs the transformations on
00:22:21
vertices and on pixels that you were
00:22:23
and we that was a big job it required
00:22:28
released in two thousand seven
00:22:32
coincidental I found that the market
00:22:36
with yes one by one and the rate of
00:22:39
adoption of open GLE es just exploded
00:22:42
and right two thousand eight with the
00:22:45
same capability set we let the market
00:22:50
set for a while we had a new standard
00:22:51
yes too available. But all of the
00:22:54
growth was happy my phone in the S one
00:22:57
dot one so we allowed a number of years
00:23:00
ago by before we read standard yeah
00:23:02
three don't zero coming out with a very
00:23:06
wide range of these I won't go through
00:23:12
them in detail it's along less that
00:23:15
took us to twenty twelve once again by
00:23:18
the time we released the standard we
00:23:19
discovered that a great deal of the
00:23:21
mobile hardware on the market had
00:23:23
capabilities that we we're not exposing
00:23:25
and three to zero. So once again we did
00:23:27
a quick turn project leading is to our
00:23:30
current standard yes three that one so
00:23:45
that you can do operations which are
00:23:47
not just graphics this is similar to
00:23:50
what open CL provides frankly less
00:23:53
flexible more but it's more try X has
00:23:57
advantages if you're trying to do very
00:24:00
low latency interaction between and
00:24:03
graphics and computing another feature
00:24:08
we don't one provides is indirect
00:24:11
drawing that is that the GP you can
00:24:14
execute drawing commands that are
00:24:15
stored in its own memory rather than
00:24:17
being submitted by the CPU that have
00:24:20
that capability by itself is boring.
00:24:22
But combine it with computer shaders
00:24:24
it's huge because now that you you can
00:24:28
run a computer shader which defines for
00:24:29
example geometry to be drawn. And also
00:24:32
the drawing commands me this D couples
00:24:36
CPU work from cheap you work and allows
00:24:38
you to have much higher throughput on
00:24:40
side a final feature three that one
00:24:45
aimed at the devices that were current
00:24:48
that is three hardware defines the
00:24:50
capabilities set for three readout one
00:24:53
it turned out as we discovered through
00:24:56
actions that all of the three dot zero
00:24:59
well what's Perth the capable of doing
00:25:02
computer. So one a three about one is a
00:25:07
officially exposed an android L as of
00:25:09
early this year and starting to ship
00:25:11
insubstantial fine. So that's where we
00:25:15
are today. Um what that's lettuce to is
00:25:19
a world where we're now shipping on
00:25:22
close to two billion devices here a
00:25:25
truly staggering number I remember
00:25:27
being very proud it's a graph two
00:25:28
thousand four or so to say that my
00:25:31
company had delivered ten million open
00:25:33
GLE es devices we're now at the rate of
00:25:36
about one point. Um open GL remains
00:25:40
very active and shipping and much less
00:25:43
impressive volumes but remember that
00:25:45
it's a different kind of device
00:25:46
continues to pioneer new technology and
00:25:49
and to drive the industry and then as
00:25:52
we all said the latest development not
00:25:54
controlled by any the committee's I
00:25:56
work on is the move to the web a
00:25:59
completely different execution
00:26:01
environment a different model of what
00:26:02
an application is and also requiring a
00:26:05
different kind of graphics API what you
00:26:08
always in fact a javascript rubber over
00:26:11
open GLE es two point zero to first
00:26:14
order. But with me stations to work in
00:26:18
an environment of untrusted content.
00:26:20
And haitian on the other side of the
00:26:23
film type see from the display device.
00:26:31
So what comes next. Well one thing that
00:26:33
will certainly happen is that open GL
00:26:35
yes well and open GL will continue to
00:26:38
evolve very useful they're excellent
00:26:41
for today's applications certainly the
00:26:44
less demanding ones they're easy to use
00:26:46
well understood and as I said shipping
00:26:51
in huge volumes. Um however the kind of
00:26:55
advanced future applications that males
00:26:57
talking about make demands on the
00:27:00
system that are very difficult to me
00:27:02
through open GL yes. And for that
00:27:05
reason prone to start oh to define a
00:27:08
new API originally called GL next
00:27:12
because it was based on open GL but we
00:27:14
architect in in fundamental ways and
00:27:17
released earlier this year sorry the
00:27:19
standard not released but we disclosed
00:27:21
the project and its basic technical
00:27:23
features earlier this year under the
00:27:25
name of welcome. So why do we need this
00:27:31
why isn't open GL yes good enough. Um
00:27:34
there are a number of reasons. Um they
00:27:38
largely well down to the fact that
00:27:40
these modern applications are extremely
00:27:42
demanding in terms of latency for
00:27:44
example if you're doing a less than a
00:27:48
frame time of latency otherwise your
00:27:50
users get sick. Um they need extremely
00:27:53
predictable latency it's not not for to
00:27:55
be short you'd have to be can't have
00:27:58
variance otherwise you get the
00:28:00
appearance of starter objects that
00:28:02
should be solid and physical suddenly
00:28:04
start to feel mushy to the user it's
00:28:06
not acceptable that so that's a demand
00:28:11
that for various reasons open GLE es
00:28:13
finds difficult to me. Why is that well
00:28:17
when open GL was new twenty years ago
00:28:19
it was actually a very thin abstraction
00:28:22
over exactly but the hardware is about
00:28:26
enormously in twenty years. And it's
00:28:28
now the case that open GL is a very
00:28:30
high level abstraction hiding enormous
00:28:33
driver come cleverness which causes the
00:28:37
underlying as if it was this a nineteen
00:28:41
ninety zero very sequential machine so
00:28:47
that's a problem all devices where
00:28:51
lectures the majority of architectures
00:28:54
shipping or tile based which means they
00:28:57
do things in a completely different
00:28:58
order from what the application
00:29:00
specifies them when it's using an open
00:29:02
GL style if you die CPU intensive
00:29:06
driver does a great deal of work all
00:29:09
errors are checked out all errors are
00:29:11
defined to turn into no ops. So the
00:29:13
system stays very robust and say that
00:29:16
was necessary twenty years ago when we
00:29:18
have relatives traction in you know
00:29:20
operating systems those days are gone
00:29:23
out objection anymore probably the
00:29:26
biggest problem open GL is the model is
00:29:29
almost entirely sequential inherently
00:29:33
and in the days now when an entry level
00:29:36
phone has for CPU chords and for
00:29:38
graphics course that's just not
00:29:40
acceptable we just all can locking is
00:29:43
intended to solve all of these problems
00:29:46
it is based conceptually on the same
00:29:48
hardware model there's not a not a
00:29:51
mistake that we call the GR next
00:29:53
originally but it is in combat designed
00:29:56
from first principles. I won't go into
00:29:59
detail about makes it work. Um need
00:30:19
that if you're going to have
00:30:21
predictable latency and predictable
00:30:22
performance. So how do we get to vulcan
00:30:27
how's it different from open GL first
00:30:30
thing we do is where GL is based on a
00:30:32
monolithic context where there's an
00:30:34
execution Q and all work is done well
00:30:38
what you is inherently a sequential
00:30:39
serial construct. So you can't use
00:30:42
multiple cores in that case we divided
00:30:45
so that there's a submission queue
00:30:47
which is sequential drawing this issue
00:30:49
but all the other work is done
00:30:51
separately invisibly audible threads.
00:30:55
Um we we do see the overhead largely by
00:30:58
pushing work back on the application
00:31:00
star we do much less error checking by
00:31:03
default we do not dependency tracking
00:31:05
by default. We do validation layers
00:31:10
which can be enabled in the device
00:31:11
their specified in the back which will
00:31:14
do some of those things for you during
00:31:15
development we encourage you to turn
00:31:17
those off when you're trying to do
00:31:19
something different status we began
00:31:24
this project last July it was a
00:31:29
wonderful moment from this business for
00:31:32
so many years to see the whole industry
00:31:35
realise the need and come together
00:31:37
extremely rapidly. I we had
00:31:39
particularly participation from many
00:31:42
high end game developers of blizzard oh
00:31:46
we have participation from production
00:31:50
people like outlook as film pixar and
00:31:53
the new applications are certainly
00:31:55
represented calculus PR is there to
00:31:57
make sure that vulcan meets their needs
00:31:59
as so it's very exciting times we did a
00:32:03
deep technical disclosure a GDC twenty
00:32:06
fifteen you can find videos of that up
00:32:08
on our you to channel without also
00:32:12
exposes some of that stuff or if you
00:32:15
have access to GDC four you can see we
00:32:19
are on track to release a at least a
00:32:21
provisional specification later this
00:32:23
year. Um we're targeting hardware
00:32:25
starting with that I spend mobile
00:32:27
devices and go as for so that's where
00:32:32
we are I look forward to talking to you
00:32:35
later about any questions you have like
00:32:38
to introduce a vector or through every
00:32:41
you more I the best I can do a chairman
00:32:45
of the open PX chair it sees I think it
00:32:54
all computer vision during lost ten
00:32:59
years has changed from they're radical
00:33:02
discipline of computer science into
00:33:05
into something that's actually driving
00:33:07
the industry change there are lots of
00:33:10
scenarios where computer vision changes
00:33:15
all lights mobile phone takes pictures
00:33:19
sophisticated algorithms do someone
00:33:24
which processing to prove the the
00:33:26
quality of the picture under the hood.
00:33:28
Um this snare and particularly excited
00:33:31
about is that can be division is coming
00:33:33
into cars detecting pedestrians
00:33:36
preventing collisions saving a lot of
00:33:39
lice. Um that there are a lot of
00:33:42
algorithms that are being developed
00:33:47
right now to enable always ornament at
00:33:50
reality something that working your
00:33:53
last or what whatever wearable devices
00:33:59
are there in in the next several years
00:34:02
at the same time all of these
00:34:06
computations that are done like
00:34:09
division albertson's they shift from
00:34:11
desktop to more while architectures.
00:34:14
And and there's a pretty compute
00:34:17
intensive algorithms it's that and what
00:34:20
is special about computer vision
00:34:23
running on mobile devices all these use
00:34:26
cases they require real time processing
00:34:29
just like in graphics you you have to
00:34:31
do a certain frames per second other
00:34:34
otherwise it doesn't make sense in the
00:34:37
card if you don't detect a pedestrian
00:34:39
you're in real time it doesn't make any
00:34:42
sense either you you have to do it fast
00:34:45
enough. And a typical to build white
00:34:51
plano what computer vision algorithm is
00:34:54
you you take you take it in image or
00:34:56
you give stream from a camera. And and
00:34:59
then then you put that in which really
00:35:02
do stream to process it on a on a
00:35:05
mobile architecture. And and then you
00:35:08
you do some high level decisions based
00:35:11
based on the information extracted from
00:35:15
the images and also information that
00:35:18
you extracted from all the sensors and
00:35:23
all all of these parts they they're
00:35:26
developing at the same time so that the
00:35:29
hardware ease these kind of diverse
00:35:31
there are lots of different cameras if
00:35:35
you if you have a program that can be
00:35:37
division algorithm on for now and right
00:35:41
for you you may have noticed that the
00:35:44
the resolution is different if you go
00:35:46
from one form to another the the
00:35:50
orientation of the camera maybe
00:35:51
different and then if you go from from
00:35:55
one platform to another like and Rory
00:35:59
to buy that's the way you get images is
00:36:02
different. Um but moving to actually
00:36:07
executing computer vision lots of
00:36:11
various card in mobile devices that you
00:36:14
can use to program to to run efficient
00:36:17
computer vision algorithms but there is
00:36:21
there is no one way if you if you have
00:36:23
to move from one platform to another
00:36:26
like again you you create an algorithm
00:36:28
for an android folder lots of them that
00:36:30
different how how do you make sure that
00:36:33
you use the the hardware in the most
00:36:37
efficient way. And the the same way
00:36:41
about the fusing that without the
00:36:43
sensor information different platforms
00:36:46
have different interfaces to to get the
00:36:51
information from from various sensors
00:36:53
that are available there. So so I'm
00:36:58
gonna I'm gonna focus for now on the
00:37:02
central part of this like the the us
00:37:05
open VX standard we which is about how
00:37:09
how are you program can be division
00:37:12
algorithms one mobile architectures and
00:37:15
then later I'm I'm gonna be talking
00:37:18
about the the left or which is open
00:37:22
cake and the the interface for
00:37:24
interacting with camera and and the
00:37:26
right part the stream input the
00:37:28
interface for interacting with other
00:37:30
sensors. So open PXEZ hardware
00:37:34
abstraction which is standardised by
00:37:37
chronos for power efficient
00:37:40
acceleration of convolutional
00:37:41
algorithms I it enables low or real
00:37:46
time applications or on mobile devices
00:37:50
so all the scenarios that I was
00:37:54
mentioning computational photography
00:37:56
automotive safety. And many many others
00:38:01
all of them that it's critical for them
00:38:04
to to be executed in real time on a
00:38:07
mobile. And that it low power devices.
00:38:11
That's exactly what happened be access
00:38:13
for it's designed to be portable across
00:38:17
a very wide variety of hardware that is
00:38:21
used in the in this kind of
00:38:23
architectures CP use and that it should
00:38:28
be used DS P.s multi course abuse SP
00:38:34
G.'s all kinds of accelerators you can
00:38:37
think of including the chips that run
00:38:42
specific computer vision functions that
00:38:45
like dedicated computer vision
00:38:46
accelerators and and as a hardware
00:38:51
abstraction layer the way a developer
00:38:55
an application developer interacts with
00:38:57
open VX is that you you you called an
00:39:00
open the X function and and the the
00:39:04
that function is has different
00:39:08
implementations on different hardware
00:39:10
architectures so so you write your
00:39:13
application just once and then he it's
00:39:17
it's executed unit the formants
00:39:20
efficient and power efficient matter on
00:39:23
a mobile device. So how do we do that
00:39:29
mobile devices are pretty challenging
00:39:32
to programme right because the PCB use
00:39:35
low power there's dedicated
00:39:36
accelerators that you can use. But all
00:39:39
of them are different some of them have
00:39:40
their own memories some don't so our
00:39:43
way to address this is something that
00:39:47
we call it gruffly PI we can see here
00:39:50
use is a gruff with where where each
00:39:54
node corresponds to to a computer
00:39:56
vision function and the connection
00:39:59
between the notes indicate the data
00:40:02
flow. And the the the standard also
00:40:07
defines the in which container as being
00:40:11
a peak which means you cannot access
00:40:14
data all along that which may seem
00:40:17
restrictive but in fact this allows
00:40:19
very efficient optimisation series
00:40:21
because now in the image does not
00:40:23
necessarily have to reside in in the
00:40:26
memory of of the main processor you can
00:40:29
actually be replicated by the
00:40:32
implementation of open VX two GPU
00:40:35
memory what do use P or any any other
00:40:39
accelerator that the system things will
00:40:41
run this specific L within the most
00:40:43
efficient manner up that there are
00:40:48
other optimisation strategies that the
00:40:51
gruff allows you to handle because you
00:40:54
have to specify the the growth in
00:40:57
advance before actually execute the L
00:40:59
great the the implementation of looping
00:41:03
VX can use all kinds of strategies to
00:41:07
fuse notes together so that for example
00:41:09
if you have several filters that are
00:41:11
run on the same in HE which
00:41:14
sequentially you you can do filter's
00:41:17
taking and combine several graft nose
00:41:20
into one running just once through the
00:41:24
image and and that would be extremely
00:41:28
efficient on on systems that have
00:41:30
limited memory bandwidth in in terms of
00:41:38
functionality open VX one point zero
00:41:41
contains. Um low level operations that
00:41:46
that are absolutely essential for any
00:41:48
kind of scenarios I've mentioned and
00:41:52
that there's also some higher level
00:41:56
operations like a calculating histogram
00:42:00
edge detectors a corner detectors
00:42:03
sparse optical flow functionality that
00:42:07
enables you to do high level algorithms
00:42:11
such as object tracking at the the way
00:42:17
the standard will evolve he's that now
00:42:21
the framework and basic functionalities
00:42:24
to find then there's who implement the
00:42:28
standard will introduce vendor
00:42:29
extensions. And then these enter
00:42:33
extensions will become chronos
00:42:36
extensions and then that will be
00:42:37
incorporated in in the standard in the
00:42:40
future versions. This is an example I
00:42:46
of all the computer vision algorithm
00:42:48
implemented with the graph the API so
00:42:53
so here you you can see S computation
00:42:57
of stereo if you have two cameras you
00:43:00
can you can calculate the
00:43:03
correspondence between the pixels and
00:43:05
and trying you late find to the other
00:43:09
three D coordinates for for each of the
00:43:11
pixels. So here you can see that there
00:43:13
are two cameras supplying images into
00:43:17
open VX and there is rectification
00:43:22
which is the first step and this is the
00:43:24
part that can be done with open BX and
00:43:27
then you competed that map and that's
00:43:29
map computation is not a portable open
00:43:31
VX but we have user notes that that
00:43:36
allow a user to write custom graph
00:43:41
nodes and and that this into the growth
00:43:44
in which your mate is a part of the
00:43:48
open VX and then optical flow is also
00:43:52
and know that you can use to do object
00:43:56
tracking and and then detecting and
00:44:00
tracking objects is another using no
00:44:02
there is there is a defacto standard in
00:44:10
in computer vision community called
00:44:13
open CV this open source computer
00:44:15
vision library. And the other than the
00:44:17
main difference behind be between open
00:44:21
C and opened the X is that open VX
00:44:25
defines I it's it's created as an
00:44:29
agreement between the numbers of
00:44:31
chronos. And which are like all
00:44:35
hardware companies who create hardware
00:44:38
to run a computer vision algorithms at
00:44:40
and as a part of the standard we define
00:44:45
conformance tests that make sure the
00:44:50
application you create a gives you the
00:44:53
same results on different platforms and
00:45:01
the the open CV you is is great for
00:45:07
creating a prototype algorithms and
00:45:10
then when you want to get a real time
00:45:13
power efficient implementation on a
00:45:15
mobile device you you can port your L
00:45:19
Grayson from open C implementation to
00:45:22
open the ex you can use different
00:45:32
computer eighty eyes to to accelerate
00:45:35
tuition both to implement open the X
00:45:38
nodes and to implement user notes which
00:45:41
are parts of the graph you user open CL
00:45:44
you can use open GL compute shaders oh
00:45:47
the state has open the X one to zero
00:45:54
specification was really eased off
00:45:57
fullest here. It's available online on
00:46:02
colours to work and you can see here
00:46:04
the companies which are members of the
00:46:07
open VX working group looking on the
00:46:10
standard and I will briefly mention the
00:46:17
generate a PI which chronos is working
00:46:20
on and defines the way an application
00:46:24
interacts with camera how exactly is
00:46:27
the application is getting images from
00:46:30
the cameras it sequentially is that
00:46:32
concurrent with the the processing of
00:46:35
the previous frames. Um how do
00:46:39
synchronise Cameron sensors how do you
00:46:43
control frequency and and many other
00:46:47
topics. And open cake and standard use
00:46:52
now gathering requirements and then in
00:46:57
order to combine the high level
00:46:59
information that you extracted from the
00:47:02
image will be deal with open the ex
00:47:05
with other sensors that are available
00:47:09
on your mobile platform there's other
00:47:11
the stream input API that allows you to
00:47:16
with with the same API get get censored
00:47:20
data from from multiple platforms and
00:47:23
begin will without worrying about
00:47:25
synchronisation and other issues and
00:47:28
you can use low level API to to get a
00:47:32
raw data like what anyone's or you can
00:47:34
use higher level API to to get a no
00:47:42
some information little like always my
00:47:44
phone oriented and and answer questions
00:47:47
like that. And thank you very much and
00:47:52
now I'd like to invite well not a lot
00:47:56
who is with Samsung and who is the
00:47:59
chair of you looking group and he'll be
00:48:01
talking about how well all corner
00:48:04
standards are coming together with the
00:48:07
same API okay Q vector I know I know my
00:48:18
email address isn't on the but so this
00:48:21
Goodman opting to firstly to try
00:48:22
pronounce my name correctly answer it
00:48:25
you can always email me if you have any
00:48:26
questions asked a longer but all back
00:48:29
but without the hyphen and on what I'm
00:48:32
alone all back one word but you may get
00:48:34
quite a few pictures of cats and the
00:48:36
like one that toward. Um right that's
00:48:40
good. So as Neil outlined of the stock
00:48:46
user interfaces are massively
00:48:48
transformed by and that all these
00:48:51
wasting. So also sees these days have
00:48:54
evolved very rapidly in the past few
00:48:57
years increasing capability within a
00:49:00
very limited power budget all mobile
00:49:03
where balls in particular mark my area
00:49:07
of expertise comes and you use have
00:49:09
really helped us enable beautiful you
00:49:11
wise wishing high and high resolution
00:49:14
screens and technology like mobile we
00:49:16
are actually hitting commercial
00:49:19
products and really immersive graphics
00:49:22
enabling these user interfaces. And is
00:49:26
also well capabilities and devices. So
00:49:29
this high end cameras coming out with a
00:49:31
range of new senses. So if you look at
00:49:34
my mind no tangible yeah you go hell
00:49:37
senses on then addition as a as a new
00:49:40
technology coming out on Morton devices
00:49:43
and this also input method to be on the
00:49:45
keypad. So if you look at how give ER
00:49:48
works with with with sensor technology
00:49:51
if you look at the noted again with
00:49:53
with with this pen other the other ways
00:49:56
of inputting when you've got where
00:49:57
apple's coming into into the fray with
00:49:59
watches and and all the bits of
00:50:01
technology really expand which so one
00:50:06
of the key things is bring together a
00:50:09
graphics with complete. So the in terms
00:50:14
of what we can do with all the
00:50:17
technology a because the the what it's
00:50:20
mixed reality augmented reality bring
00:50:24
together three D graphics with reality
00:50:27
as you can see in this look in this
00:50:29
demo and this is one the GP enabled
00:50:32
laptop so this is still being finesse
00:50:36
this still a lot at a way to go before
00:50:38
becomes as a massive as three D
00:50:40
graphics on mobile but we are getting
00:50:42
the so one of the final bits of "'em"
00:50:48
possible to cover is can compute
00:50:51
technology. So open CL is crevices
00:50:55
landed API for parallel programming and
00:50:57
covers a wealth of devices from high
00:51:00
performance computing to well it's a
00:51:03
heterogeneous standard that enables
00:51:05
targeting all the different components
00:51:09
that exists whether you're talking CP
00:51:11
use what we talking GPDS P.s and even
00:51:15
the hardware specialised hardware
00:51:17
blocks the the advantage to conceal
00:51:20
brings is having a a clean clean
00:51:23
framework that you can use kernels that
00:51:26
you can transfer between these devices
00:51:28
as well as if you really want to get
00:51:31
the best performance specialise for
00:51:34
each architecture of device. Um CL has
00:51:39
evolved to a lot since its first
00:51:41
release and and most recently GDC and
00:51:45
the open still to do one the
00:51:47
provisional spec work was published I
00:51:51
think I'm you C post plus colour
00:51:53
language so this massive improving
00:51:57
programmer productivity improve
00:51:59
performance. And one of the things that
00:52:01
but for us is so important is the
00:52:03
announcement of speed V so spiffy is
00:52:07
the new evolution and then intermediate
00:52:10
representation but kernels defines and
00:52:14
that will go into that more in a moment
00:52:17
but it it is able to take C plus plus
00:52:22
front end and it's a common languages
00:52:26
as well as all all those that will go
00:52:29
into I still to the one runs on any
00:52:32
zero to point capable Broadway so and
00:52:35
not not seen on mobile yes but look
00:52:39
forward to so spiffy. So cross have
00:52:44
been working on this this side of
00:52:46
portable be presentation standard for
00:52:49
while spiffy is that is it is a big
00:52:52
step forwards to define across event
00:52:55
intermediate presentation. So I'm I'm
00:52:59
gonna move on to this light explain how
00:53:02
spiffy fits together around all the
00:53:04
standards. So it's a first to bitch
00:53:07
what's stream that extend the
00:53:09
extensible and easily passed. So you
00:53:13
have a high level couple languages like
00:53:17
open CLC that we already have it in CO
00:53:19
one dot axial two point oh and open CLC
00:53:22
plus plus and seal to do one they get
00:53:27
compile down to speed V that is that
00:53:31
the the format for distribution which
00:53:33
is very well standardise and then back
00:53:36
and run on different sealed drivers the
00:53:38
the whole point is it is that it is a
00:53:41
core across this across industry
00:53:44
guaranteed. And to work but then you've
00:53:47
got and this is what it comes really
00:53:49
interesting for us I am in in mobile
00:53:52
you've got graphics coming in at
00:53:55
suspended B is more more just for
00:53:57
computer graphics I'm compute having in
00:54:00
to meet your presentation about that is
00:54:03
combining two technologies. So the the
00:54:07
vulcan standard that Tom spoke about us
00:54:11
PL P.s at the hospital that is the way
00:54:14
you write your shaders so you you you
00:54:16
use I would jealous L or new colour or
00:54:20
shading lounges to to put together your
00:54:22
shade is they get you compiled into
00:54:25
spiffy and that's how you is to be them
00:54:29
and or other intermediate
00:54:31
representations and the LBM community
00:54:34
are very important to us. So we we are
00:54:36
engaged very closely with LBM the key
00:54:40
thing for us why spiffy was the final
00:54:43
by criminals as the importance of
00:54:45
having a stable standard because then
00:54:48
the guarantee that will work and
00:54:50
consistent of all as we progress
00:54:52
devices. And then if for for all the
00:54:58
languages so I'm a bit beyond criminals
00:55:01
and all the high level languages spiffy
00:55:04
retains all the information so you can
00:55:07
go back to high level language from
00:55:10
spiffy should you wish to one of the
00:55:14
key things occurrences considering is
00:55:16
developing open source implementations
00:55:18
of these translators to help weaker
00:55:20
system of all and adopts a bit spiffy
00:55:25
and the the key advantages couple of
00:55:27
mentioned already at one of the the
00:55:31
main ones that motivated us is
00:55:33
eliminating I'm cross spend the
00:55:35
portability issues. So for for this
00:55:38
especially important for us and some so
00:55:40
that we use a very wide range of GP use
00:55:43
across a device range. So I'm being
00:55:46
able to guarantee that actually the
00:55:48
works right across all the different
00:55:50
architectures is is so crucial and so
00:55:53
crucial for the industry is also as a
00:55:55
benefit improve performance of reduces
00:55:58
the runtime compilation time because
00:56:00
you ready I'm pretty compile it into
00:56:03
into me to pursue representation it
00:56:07
also helps with IP protection so it's
00:56:09
it's not doesn't have any absolute
00:56:11
guarantees but you you can it's fall
00:56:14
far harder to and get psyche out of a
00:56:18
spear Connell then actual first source
00:56:21
distributed shade of colour and it
00:56:25
improves the simplicity of how drivers
00:56:29
interpret the the colonels or shaders.
00:56:32
So there's a lot of discrepancy as I
00:56:35
mentioned I'm fourteen between
00:56:37
different compiler stacks and this this
00:56:40
is that so I on to some exciting stuff
00:56:45
that's always technology enables. So
00:56:48
why mention terms of the the big
00:56:50
pollution was seeing in the power of
00:56:53
and capabilities of soc is when I was
00:56:58
saying the possibility of getting
00:56:59
decent frame rates of facial
00:57:02
recognition technology combined with
00:57:05
with high end graphics of this is the
00:57:06
this them oh yeah shows how using open
00:57:10
CL you can I mean I'm had enable real
00:57:13
time face detection on all I am tablet
00:57:17
device and then went on to three D
00:57:20
graphics now one of the the key no item
00:57:28
as I can put it that way from my
00:57:31
experience is is how to get things to
00:57:35
work together. So I'm I'm I'm mania
00:57:38
graphics guy I try intro to multimedia
00:57:41
guys at remote me guys don't often
00:57:43
understand mean and I think the the
00:57:45
feeling is mutual. And it's a real
00:57:49
challenge to try and get different bits
00:57:51
of technology to talk the same language
00:57:55
and and one of the key things is the
00:57:57
data interchange between them. So
00:58:00
different different specialise comply
00:58:04
units take different formats all all
00:58:08
for different formats. So whether in
00:58:11
graphics which where you we where using
00:58:13
I'm ought to be A as a main format
00:58:17
buses cover IS P.s that I put YTV
00:58:21
versus comer senses would bear formats
00:58:23
each scroll G domain as it spy has
00:58:26
developed their own specials formats
00:58:28
but within that there's also a real
00:58:30
before of different exactly what you
00:58:35
mean by YTV so if if I say use for TV
00:58:38
that's not a question it this a there's
00:58:41
a lot of detail behind that question to
00:58:43
be also and you can see and don't just
00:58:45
how many things that we need to to
00:58:47
really specifying pinned down when with
00:58:49
two where where we we can about the
00:58:51
details of format. So one of the things
00:58:55
that we are really keen to do and I'm
00:58:57
I'm really glad that crosses scar
00:58:58
ensuring interesting is developing a
00:59:01
data format standard. So this is very
00:59:03
much a work in progress and we really
00:59:06
keen for anyone who's interested in
00:59:07
this to get in touch to help us I'm
00:59:11
shape it. So the the main reason for
00:59:15
for eight is to be moving beyond what
00:59:19
for CC provides a full CC has a very
00:59:21
wide range of of descriptions of
00:59:24
formats but then not machine readable
00:59:26
the very descriptive in terms of
00:59:29
explanation of what they do but the not
00:59:32
the the they can't just be passed but
00:59:34
carmel. So what we are off to is is is
00:59:40
a standard that can be used with the
00:59:43
base real economical and has all the
00:59:45
information you need should you need it
00:59:48
a key thing is that it's that good idea
00:59:51
behind it is to be extensible. So it's
00:59:54
not I'm gonna be fixed forever there's
00:59:55
gonna be different standards the gonna
00:59:57
be propriety times we know that what
00:59:59
what we realistic about it bites about
01:00:01
trying to provide some transparency to
01:00:03
the industry in this here. So but the
01:00:07
the cool thing in terms of how we're
01:00:08
thinking of developing it is if one API
01:00:12
has some some concepts of of of formats
01:00:15
adjust and you know I'm saying I'm
01:00:17
using ideology beyond in saying I'm
01:00:19
while TV all this is a great format the
01:00:22
the the the the the descriptive behind
01:00:25
that actually explains it in details
01:00:28
what do you mean by backed by that you
01:00:30
know and then when you go to another
01:00:32
API it can understand exactly what you
01:00:35
mean by back behind the scenes. So
01:00:37
examples here when you said why TV
01:00:40
really what you mean was and then be
01:00:42
twelve standard and and and that's all
01:00:44
included in the data. So this one that
01:00:50
does all this technology bring
01:00:51
together. Um they lot this example here
01:00:56
of the Ikea furniture catalogue is the
01:01:01
probably best example of how
01:01:03
complicated all the different bits of
01:01:05
technology ought to bring together so
01:01:08
what you're taking is an input from a
01:01:11
camera that input from other senses and
01:01:14
trying to overlay that onto onto the
01:01:17
cover data graphics the ride as
01:01:20
received okay all the positioning
01:01:22
information I try to combine the frames
01:01:24
together. So timing is important the
01:01:27
data format important to bring all
01:01:29
those the there's a PR is to get them
01:01:32
out all the information is actually
01:01:34
crucial. And a cold kind of when you
01:01:37
still in experimental stage is using a
01:01:41
occlusion with they are so that it it's
01:01:44
yeah involves a lot more computation.
01:01:47
So still still not out in the
01:01:49
commercial devices but you can see how
01:01:53
how impressive it looks to be able to
01:01:56
to to really embedded something that's
01:02:01
augmented on to reality with a clue. So
01:02:06
as I said I what what I do is big these
01:02:10
all how that small just work together
01:02:13
some of these are as I was like all
01:02:14
aspiration allow rooms. And they
01:02:18
there's a lot of the we have developed
01:02:20
a a and we have seen real advances. So
01:02:24
open open CL two graphics inter
01:02:27
operation has really improved over the
01:02:29
years and and as trump alluded and the
01:02:33
next the next big step with vulcan we
01:02:35
see really helps bring these together.
01:02:38
But what was what was showing here is
01:02:41
how the all the different kernel sleepy
01:02:44
eyes which can work together. So taking
01:02:47
input from members senses and from
01:02:51
camera other streaming backed into
01:02:54
other standards like open the X and
01:02:56
stream input using that information to
01:02:58
get the what one by the application and
01:03:02
then I'm putting a by all do and also
01:03:05
for us in the bar bright graphics up
01:03:09
onto devices like we will pass another
01:03:13
omens relative devices. So so this
01:03:15
really shows the challenge we face in
01:03:18
bring everything together we have made
01:03:20
a lot of programs that but what once we
01:03:23
make more these arrows from morality
01:03:27
and with the data format standard in
01:03:30
other ways of making sure that you have
01:03:32
maximum efficiency you avoid copies you
01:03:35
avoid ad formats the translations when
01:03:38
when the necessary that is when you can
01:03:41
really bring performance into a men's
01:03:44
reality you wise. So we have anything
01:03:47
further I think let's bring the panel
01:03:49
together so now we get to hear your
01:03:53
questions and how we can to take the
01:03:56
industry forward you oh make it it just
01:04:09
time to guess in in answers if you have
01:04:12
any question please come to this use my
01:04:16
company in in creative. So it is
01:04:20
addition what when you prepare question
01:04:23
I I have connected to the panel yeah as
01:04:27
I I think it is very tightly question
01:04:30
but so we can have a believable. So
01:04:32
what do you think that next next killer
01:04:36
application of smart one was my bicycle
01:04:41
but yeah if you know that's the you you
01:04:44
study you've it is but it I what I want
01:04:46
the U prediction I think could the
01:04:51
scanning capability that's the mixture
01:04:53
of or stuff we've been talking about is
01:04:56
going to be the moment where three D
01:04:59
goes social. So we have pictures go
01:05:03
social it no textbook we've had videos
01:05:08
go social with you too. But up to now
01:05:12
people have not been able to capture
01:05:14
three D models and creating two all
01:05:18
three is very hard. So soon as we can
01:05:21
enable a user to do that and within a
01:05:25
few seconds capture of two three D
01:05:27
model of any object and share it on
01:05:30
place but it's going to an enable a
01:05:34
whole new class of social media
01:05:37
applications. And even not just
01:05:41
displaying three D but searching for
01:05:43
products that look like. This thing I'm
01:05:46
scanning. So that the E commerce sites
01:05:49
are gonna what this yeah I agree I
01:05:55
think for but for me can the the all
01:05:57
the big area which we problem is is is
01:06:00
how much vision increasing so we're
01:06:03
seeing how come or data can be used for
01:06:07
it's for some mission implication the
01:06:09
moment you see some cool I don't
01:06:11
usually like like all men to being able
01:06:13
to show you augmented reality can which
01:06:16
is for media is still early days and
01:06:19
you if you give I can imagine the city
01:06:21
news or mentor out the apps but they
01:06:23
are quite cool but the the the gonna be
01:06:26
a breakthrough my opinion when you
01:06:28
really see the data you have services
01:06:32
like Google maps and so on really come
01:06:35
in as an augmented so that's that's the
01:06:38
the the big thing for me coming up next
01:06:40
but yeah oh sorry I hope should I start
01:07:22
over I know thanks they have as much to
01:07:27
learn as as the rest of us do about how
01:07:29
to exploit for example camera input how
01:07:32
to design a game for a mobile device
01:07:35
but these are without doubt the
01:07:38
smartest people I've ever very timidly
01:07:41
sat down to lunch with and I anticipate
01:07:44
amazing things from them as we go
01:07:46
forward in as walking hits the market
01:07:48
oh well I I think a lot of computer
01:07:57
vision is gonna change the world the
01:08:00
way that we we interact with mobile
01:08:02
devices but yeah I can't agree more
01:08:06
with a new the saying that three D the
01:08:10
scanning capabilities coming to mobile
01:08:13
devices you said that if we knew what
01:08:16
what the next thing is will start a
01:08:18
business so yes that's what it sees
01:08:20
doing we're one of the leaders and what
01:08:22
colour scanning okay thank you. And so
01:08:50
just for for three years okay start
01:09:20
yeah so yes not video super important
01:09:24
we actually to have a a standard that
01:09:26
is for video streaming in processing is
01:09:28
called up my "'cause" it's not quite as
01:09:31
well known as open GLE es but it is
01:09:33
shipping on a number keypad forms
01:09:36
including and right but I I think it's
01:09:40
to say we haven't been quite so
01:09:44
successful in battling fragmentation in
01:09:46
the video spaces we have in the in the
01:09:48
graphics makes but any help appreciated
01:09:51
the but the things that along is
01:09:54
talking about them mean the video
01:09:57
formats it is the key parts of being
01:10:00
able to enable the graphics world in
01:10:01
the video world to talk together so do
01:10:04
to to to what this on the sake well one
01:10:07
things were really keen with the bring
01:10:10
in a data format about is not I'm not
01:10:13
just across and it's or whatever scroll
01:10:16
stands a possible but if you using a
01:10:18
different standard at least you can
01:10:20
interface with the prevalent and
01:10:23
standards we're right yeah so that that
01:10:25
if you have been using G stream on all
01:10:27
the method instead of max which is a
01:10:29
shame at least then you can interact
01:10:33
with that with the GP use effectively
01:10:35
by integrating into opinion about that.
01:10:37
Um just where we're talking a little
01:10:42
bit about the past rather than the
01:10:43
future but the current standards do
01:10:45
have facilities so and for example that
01:10:48
you GL group the lawn chairs and the
01:10:50
open GLE es committee co defined a
01:10:53
functionality called external image
01:10:55
which is the ability for a graphics API
01:10:58
open GLE es in particular to import an
01:11:01
opaque external type the problem it
01:11:04
doesn't solve is that the
01:11:06
implementation of the for example the
01:11:08
decoder or the streamer and the and the
01:11:11
graphics driver have to understand each
01:11:13
other. So there's no abstract
01:11:15
definition of the interface between
01:11:18
it's not a perfect solution it is what
01:11:20
people are using now mostly that sounds
01:11:23
fine so just sort of part of why I was
01:11:27
asking is that was one well she is hot
01:11:35
my textures some stand he's a
01:11:43
processing all done this could run on
01:11:48
any machine. So maybe that's the answer
01:11:51
but is there more to have your actions
01:11:53
might effect Sir right. So well I I
01:12:03
will so so for now open the excess
01:12:05
focused on the on that right running on
01:12:08
on the mobile device that so so you I I
01:12:15
don't know that there's a convenient
01:12:17
way with the current standard to
01:12:18
address you know the problem when
01:12:21
multiple nodes are running in different
01:12:24
like like set a separated over the
01:12:28
network so that we address the the
01:12:30
latency issues and the the real time
01:12:35
make sure the real time performance I
01:12:37
think once always one augmented reality
01:12:41
with wearable devices makes its way
01:12:44
into the industry there's gonna be more
01:12:46
and more use cases like this and the
01:12:49
architecture of the standard is
01:12:51
definitely the defined so that these
01:12:54
issues can be addressed so it's all
01:12:56
possible that one of the future
01:12:57
versions of the standard solves this
01:12:59
problem. You can also three yeah this
01:13:03
problem like you you have your video
01:13:06
stream from the wearable device and you
01:13:08
you have your processing in in your
01:13:10
server farm. And and this is connected
01:13:13
by the network so so you're running
01:13:15
open the X in the server and and you
01:13:18
have some video source which maybe
01:13:21
again defined by open cake and it's a
01:13:25
really important use case I and you
01:13:29
you're probably gonna see this happen
01:13:31
before wearable went to reality I mean
01:13:33
to because it's already happening
01:13:35
right. So I guess today if you wanted
01:13:38
to vision that in the cloud you would
01:13:42
use a standard like open CL but that's
01:13:45
the reality today respect to sense the
01:13:48
can be expanded it. It's twenties over
01:13:51
a specific problem in its version one
01:13:53
point O state which is low power on
01:13:54
mobile. I also can be X games some
01:13:58
industry traction and people invest now
01:14:00
buttons that would be X is no reason
01:14:02
why either you couldn't one open be
01:14:03
Exxon in in the campuses easily as on a
01:14:07
mobile devices no market actually
01:14:08
somewhere you couldn't do that. It'll
01:14:11
clearly be driven by the trade off
01:14:13
between the cost of bandwidth. And you
01:14:16
know high bandwidth low latency
01:14:18
communication for gonna send raw image
01:14:20
data that could be a a really good
01:14:24
solution for for example object
01:14:26
recognition and tagging maybe not a
01:14:28
great solution for optic flow so I
01:14:32
expect to actually see hybrids where
01:14:34
there's quite a bit of image
01:14:36
stabilisation and things happening in
01:14:37
the camera. But certainly if you're
01:14:39
trying to recognise automobiles out of
01:14:43
the entire database of all the
01:14:44
automobiles and the world. You won't do
01:14:46
that on the device just because of the
01:14:48
data storage requirement yes yes no I
01:14:52
mean if you haven't checked out neural
01:14:54
nets in the last six months you need to
01:14:56
because it's got about to change the
01:14:57
world. Uh computers are now officially
01:14:59
better than humans are recognising
01:15:01
things. And because of you wanna
01:15:04
technology. And all the training and
01:15:06
unit certainly can run in the camp and
01:15:07
the training certainly once in the in
01:15:09
the clear but the mobile presses again
01:15:12
that's not you can one I usefully large
01:15:15
trained dataset on device so yeah you
01:15:19
should look people it it's actually and
01:15:21
it's I'm amazed it's all happening in
01:15:23
the last six to twelve months yes
01:15:24
concretely. I guess another another
01:15:27
aspect well like course collecting what
01:15:30
Tom said open the X can also be used
01:15:34
right now for this kind of scenario to
01:15:37
that there there many applications
01:15:38
where you would do you would run some
01:15:40
of the operations locally on the device
01:15:43
that's near the camera you would
01:15:45
extract some high level information
01:15:47
from the image and then you would send
01:15:48
it to over the network to use a reform
01:15:51
instead of sending media stream which
01:15:53
requires much higher bandwidth yeah
01:15:59
well I just I mean for me mobiles is
01:16:03
use of CL and is formal I think we need
01:16:08
to real time but as as big said you
01:16:10
could using more with a reduced. So
01:16:14
with voice such that some stuff is numb
01:16:18
back to do on the device a Macleod that
01:16:21
that that was the changing it could
01:16:27
situation one okay one of the table
01:16:30
right now is competitions but
01:16:32
specialist I understand correctly
01:16:35
therefore the question I would be
01:16:37
asking maybe not from your domain but
01:16:39
it's directly related to what you
01:16:42
present ah says the infrastructure
01:16:45
around the particular sense you find as
01:16:49
I I remember correctly vitro nailed
01:16:52
that one about twenty two sensors
01:16:54
listed and we don't like to be there by
01:16:59
formal process at this point
01:17:02
manufacturer got the attention from the
01:17:05
developers from yeah and there's as the
01:17:11
numbers oh and you can an example three
01:17:18
microphones. But at least it can do I
01:17:22
don't know what the introduction not
01:17:26
maybe so the screen but for the other
01:17:29
direction for users of them moment
01:17:31
noises there is no interface there's no
01:17:34
standard that will and wow this twenty
01:17:37
two this substrate three Michael once
01:17:40
to be available for developers use the
01:17:43
any chat and no I'm asking primarily
01:17:46
people was working yeah it was the
01:17:48
vendor's hardware that these standards
01:17:52
similar shape just promise will be
01:17:55
available yes. I I hope so so as well
01:18:02
and I I totally agree with you today
01:18:05
it's it's a real issue because the the
01:18:09
senses are coming into the into the
01:18:11
devices before the software stack has
01:18:14
had a chance to figure out how to
01:18:16
expose them to developers. So and at
01:18:20
the moment you you're right it's in too
01:18:22
many of the censors it scales you know
01:18:24
every device manufacturer has its own
01:18:28
proprietary API which means it's very
01:18:30
difficult was some on but that's why we
01:18:32
need standards now the at the at the
01:18:36
height of the screen input standard
01:18:38
that we talked about is the first line
01:18:42
of defence because any successful
01:18:47
standard it must enable those sensor
01:18:51
vendors to still to compete and to
01:18:53
innovate. Nobody mustn't. Um prescribe
01:18:57
how the audio processing is going to be
01:18:59
done and that will not create a
01:19:01
successful standard we just need a high
01:19:03
enough level standard saying this is
01:19:05
the result that we need to present to
01:19:07
the application developer and the
01:19:08
easily understood impossible way. And
01:19:11
then let the many different audio
01:19:14
microphone vendors compete in a healthy
01:19:17
way to to produce the best quality data
01:19:20
stream. And the stream input some
01:19:23
standard is intended to be no high
01:19:25
enough that people can innovate but we
01:19:28
can put a layer of portability over not
01:19:32
the chaos which is the multiple sensors
01:19:34
and devices. Now. It's screaming but
01:19:38
that was not done yet but we're still
01:19:40
gathering requirements on that too so
01:19:41
it's very early days but so if you have
01:19:44
ideas not we would love to to the of
01:19:48
course the there is a terrible chicken
01:19:50
and egg problem that we have platform
01:19:55
adoption is is ultimately the big issue
01:19:58
that if if you don't have a big enough
01:20:00
market too if if you want to write an
01:20:03
application which is commercially
01:20:05
viable and which makes use of all these
01:20:07
sensors you have to have an on off
01:20:09
target devices out there that you can
01:20:11
recover your investment in reference.
01:20:13
Um as I say as I said open GLE es was
01:20:18
growing up it was moderately successful
01:20:21
in a small way but it was adoption for
01:20:23
IRS and and write the cost to to
01:20:25
explode the same thing really needs to
01:20:28
happen. And and I think we need to work
01:20:32
hard to show what's possible to
01:20:34
encourage the platform of vendors to
01:20:38
make a commitment to some kind of
01:20:40
standard it if it's ours great but if
01:20:42
it's someone else's fine as long as
01:20:44
there is one that will be happy. But
01:20:47
showing what's possible I think which
01:20:49
is sort of a joint project of ourselves
01:20:51
and and the hardware vendors. And
01:20:53
application writers that I think we
01:20:55
have to start there yeah I'm used all
01:20:59
my point on I hope so as well like Mia
01:21:03
said but it is is a very much to the
01:21:05
platforms at this so right do you don't
01:21:08
and you stand will do we expose this to
01:21:10
develop those or do we then decide
01:21:13
ourselves how this is showing I was six
01:21:17
those so I think one of the key things
01:21:19
that's useful is to get your pressure
01:21:21
again you're two and a different
01:21:24
platforms that that's say we really
01:21:26
there we want these fit functionality
01:21:28
exposed by the platform a PI whether
01:21:31
it's by a standard APR as a lot the
01:21:33
whys why it's important. So I'm glad
01:21:36
think in the graphics industry and we
01:21:39
have got to open GLE es and you know
01:21:42
people have vulcan as a standardised
01:21:44
API that it really gives that power to
01:21:46
do output is we haven't quite seen that
01:21:49
happen all the functionality that's the
01:21:53
these wonderful devices can are right.
01:21:55
But please do you shout loudly for the
01:21:59
functionality what twelve take a
01:22:00
feedback back that within samson. Thank
01:22:03
you I'm slowly but the our time isn't
01:22:06
on this so please mail persisted this
01:22:08
to many people want to participate in
01:22:12
as a member mental some more as a yes a
01:22:17
please please it was quickly we want to
01:22:19
participate in LA briefly I mean yeah
01:22:22
anyone spoken to use corner standards
01:22:24
they're available royalty free but if
01:22:26
if this is a particular this whole
01:22:27
areas particular interest your company
01:22:30
and you would like a voice and a vote
01:22:32
in how these standards of of any
01:22:34
company is more than welcome to join a
01:22:36
kernels and and get involved in
01:22:38
creating these standards and we'd love
01:22:40
to talk to if that's of interest. yeah
01:22:43
also so there are so many a catering
01:22:45
members in our society so so if you
01:22:48
want something to ask the please use
01:22:51
the the intimation that the coffee
01:22:52
break in outside think it very much

Share this talk: 


Recommended talks

ecoRobotix with Steve Tanner
Steve Tanner, ecoRobotix
April 19, 2018 · 1:35 p.m.
479 views