WEBVTT

1
00:00:02.765 --> 00:00:05.385
<00:00:02.765>So, <00:00:02.920>yeah, <00:00:03.076>like <00:00:03.231>I <00:00:03.387>said, <00:00:03.542>today <00:00:03.698>we'll <00:00:03.853>talk <00:00:04.009>about, <00:00:04.165>um,

2
00:00:05.455 --> 00:00:08.505
<00:00:05.455>uncertain <00:00:05.735>estimation <00:00:06.015>in <00:00:06.295>neural <00:00:06.575>networks, <00:00:06.855>um,

3
00:00:08.725 --> 00:00:10.665
<00:00:08.725>by <00:00:09.004>b <00:00:09.285>uh, <00:00:09.565>Niku <00:00:09.844>and <00:00:10.125>Favian.

4
00:00:11.885 --> 00:00:14.105
<00:00:11.885>Um, <00:00:12.118>our, <00:00:12.351>uh, <00:00:12.584>agenda <00:00:12.818>for <00:00:13.051>today <00:00:13.285>is,

5
00:00:14.105 --> 00:00:15.665
<00:00:14.105>first <00:00:14.338>I <00:00:14.571>will <00:00:14.805>give <00:00:15.038>you <00:00:15.271>an <00:00:15.505>introduction

6
00:00:15.665 --> 00:00:17.025
<00:00:15.665>to <00:00:15.970>uncertainty <00:00:16.276>in <00:00:16.583>neural <00:00:16.889>networks.

7
00:00:17.025 --> 00:00:18.025
<00:00:17.025>Generally, <00:00:17.325>what <00:00:17.625>is <00:00:17.924>it?

8
00:00:18.645 --> 00:00:22.865
<00:00:18.645>Um, <00:00:18.820>and <00:00:18.995>I'll <00:00:19.169>introduce, <00:00:19.345>uh, <00:00:19.520>some <00:00:19.695>estimation <00:00:19.870>methods, <00:00:20.044>um,

9
00:00:23.175 --> 00:00:25.785
<00:00:23.175>that <00:00:23.375>can <00:00:23.575>be <00:00:23.775>used <00:00:23.975>to <00:00:24.175>estimate <00:00:24.375>the <00:00:24.575>uncertainty

10
00:00:25.805 --> 00:00:27.385
<00:00:25.805>of <00:00:26.038>a <00:00:26.271>prediction <00:00:26.505>in <00:00:26.738>the <00:00:26.971>neural <00:00:27.205>network.

11
00:00:28.285 --> 00:00:31.385
<00:00:28.285>Um, <00:00:28.440>and <00:00:28.596>then <00:00:28.751>we'll <00:00:28.907>have <00:00:29.062>a <00:00:29.218>first, <00:00:29.373>um, <00:00:29.529>interactive <00:00:29.685>session,

12
00:00:31.885 --> 00:00:36.745
<00:00:31.885>um, <00:00:32.012>where <00:00:32.139>we <00:00:32.266>would <00:00:32.394>like <00:00:32.521>to, <00:00:32.648>um, <00:00:32.775>have, <00:00:32.903>uh, <00:00:33.030>um, <00:00:33.157>you, <00:00:33.285>uh,

13
00:00:36.875 --> 00:00:39.865
<00:00:36.875>think <00:00:37.075>about, <00:00:37.275>uh, <00:00:37.475>what, <00:00:37.675>what <00:00:37.875>applications <00:00:38.075>are <00:00:38.275>there

14
00:00:39.885 --> 00:00:40.945
<00:00:39.885>for <00:00:40.839>uncertainty

15
00:00:40.965 --> 00:00:44.065
<00:00:40.965>and <00:00:41.165>neural <00:00:41.365>networks, <00:00:41.565>um, <00:00:41.765>specifically <00:00:41.965>maybe <00:00:42.165>in <00:00:42.365>automotive.

16
00:00:44.285 --> 00:00:46.585
<00:00:44.285>And <00:00:44.459>we'll <00:00:44.635>create <00:00:44.809>a, <00:00:44.985>a <00:00:45.160>word <00:00:45.334>cloud <00:00:45.510>and, <00:00:45.684>and,

17
00:00:46.845 --> 00:00:50.985
<00:00:46.845>and <00:00:46.972>yeah, <00:00:47.099>so <00:00:47.226>we <00:00:47.354>want <00:00:47.481>you <00:00:47.608>to <00:00:47.735>yeah, <00:00:47.863>get <00:00:47.990>active, <00:00:48.117>um, <00:00:48.245>there.

18
00:00:51.645 --> 00:00:53.945
<00:00:51.645>Uh, <00:00:51.845>and <00:00:52.045>then <00:00:52.245>we'll <00:00:52.445>ask <00:00:52.645>the <00:00:52.845>question, <00:00:53.045>um,

19
00:00:54.405 --> 00:00:56.945
<00:00:54.405>is <00:00:54.871>uncertainty <00:00:55.338>estimation <00:00:55.805>required?

20
00:00:57.085 --> 00:00:59.625
<00:00:57.085>And <00:00:57.212>this <00:00:57.339>is <00:00:57.466>kind <00:00:57.594>of, <00:00:57.721>we <00:00:57.848>want <00:00:57.975>you <00:00:58.103>to <00:00:58.230>think <00:00:58.357>about <00:00:58.485>it.

21
00:00:59.725 --> 00:01:03.265
<00:00:59.725>Um, <00:00:59.841>and <00:00:59.958>then <00:01:00.075>this <00:01:00.191>kind <00:01:00.308>of <00:01:00.425>leads <00:01:00.541>to <00:01:00.658>our, <00:01:00.775>our <00:01:00.891>next <00:01:01.008>topics, <00:01:01.125>um,

22
00:01:03.355 --> 00:01:06.465
<00:01:03.355>where, <00:01:03.555>uh, <00:01:03.754>Nico <00:01:03.955>and <00:01:04.155>Fabian <00:01:04.354>were <00:01:04.554>introduce <00:01:04.754>two

23
00:01:07.305 --> 00:01:10.305
<00:01:07.305>specific <00:01:07.771>applications, <00:01:08.238>um, <00:01:08.705>uh,

24
00:01:10.365 --> 00:01:12.145
<00:01:10.365>for <00:01:10.714>uncertainty <00:01:11.065>in <00:01:11.414>neural <00:01:11.765>networks,

25
00:01:12.325 --> 00:01:13.945
<00:01:12.325>and <00:01:12.525>they'll <00:01:12.725>give <00:01:12.925>an <00:01:13.125>introduction <00:01:13.325>to <00:01:13.525>the <00:01:13.725>topic

26
00:01:13.965 --> 00:01:15.065
<00:01:13.965>and <00:01:14.212>explain <00:01:14.460>what <00:01:14.707>it <00:01:14.955>is.

27
00:01:15.405 --> 00:01:17.985
<00:01:15.405>Um, <00:01:15.560>and <00:01:15.716>then <00:01:15.871>for <00:01:16.027>both <00:01:16.182>parts, <00:01:16.338>we <00:01:16.493>also <00:01:16.649>have <00:01:16.805>a,

28
00:01:18.105 --> 00:01:19.385
<00:01:18.105>a <00:01:18.489>little <00:01:18.873>interactive <00:01:19.257>session.

29
00:01:21.145 --> 00:01:24.155
<00:01:21.145>Okay. <00:01:21.320>So, <00:01:21.494>um, <00:01:21.670>first <00:01:21.845>for <00:01:22.020>the <00:01:22.195>interactive <00:01:22.369>sessions, <00:01:22.545>um,

30
00:01:24.275 --> 00:01:27.275
<00:01:24.275>I <00:01:24.402>posted <00:01:24.529>the <00:01:24.656>link, <00:01:24.784>I <00:01:24.911>hope, <00:01:25.038>um, <00:01:25.165>you <00:01:25.293>got <00:01:25.420>the <00:01:25.547>right <00:01:25.675>one.

31
00:01:27.855 --> 00:01:30.315
<00:01:27.855>Um, <00:01:28.055>and, <00:01:28.255>uh, <00:01:28.455>we <00:01:28.655>have <00:01:28.855>a, <00:01:29.055>a <00:01:29.255>pin,

32
00:01:30.455 --> 00:01:31.675
<00:01:30.455>um, <00:01:30.638>that <00:01:30.821>you <00:01:31.003>see <00:01:31.187>on <00:01:31.370>the <00:01:31.553>screen.

33
00:01:31.715 --> 00:01:35.955
<00:01:31.715>I <00:01:31.842>also <00:01:31.969>posted <00:01:32.096>it <00:01:32.224>in <00:01:32.351>the, <00:01:32.478>um, <00:01:32.605>swap <00:01:32.733>card, <00:01:32.860>uh, <00:01:32.987>session, <00:01:33.115>um,

34
00:01:36.255 --> 00:01:37.795
<00:01:36.255>for, <00:01:36.601>for <00:01:36.948>the <00:01:37.294>interactive <00:01:37.641>tool.

35
00:01:38.815 --> 00:01:41.795
<00:01:38.815>Um, <00:01:40.215>yes.

36
00:01:42.015 --> 00:01:46.755
<00:01:42.015>Um, <00:01:42.142>and <00:01:42.269>so, <00:01:42.396>uh, <00:01:42.524>uh, <00:01:42.651>we <00:01:42.778>have, <00:01:42.905>uh, <00:01:43.033>um, <00:01:43.160>I <00:01:43.287>would <00:01:43.415>like

37
00:01:46.755 --> 00:01:47.795
<00:01:46.755>to <00:01:47.067>ask <00:01:47.379>two <00:01:47.691>questions

38
00:01:47.855 --> 00:01:51.715
<00:01:47.854>before, <00:01:47.982>um, <00:01:48.109>also <00:01:48.236>to <00:01:48.364>get <00:01:48.491>everybody <00:01:48.618>to, <00:01:48.745>to, <00:01:48.873>to <00:01:49.000>use <00:01:49.127>the <00:01:49.255>tool.

39
00:01:52.335 --> 00:01:56.755
<00:01:52.335>Um, <00:01:52.510>so <00:01:52.685>let <00:01:52.860>me <00:01:53.035>put <00:01:53.210>that <00:01:53.385>up <00:01:53.560>really <00:01:53.735>quick.

40
00:01:57.535 --> 00:02:02.295
<00:01:57.535>Um, <00:01:58.001>so <00:01:58.468>first <00:01:58.935>question.

41
00:02:02.635 --> 00:02:05.295
<00:02:02.635>Um, <00:02:02.790>I <00:02:02.946>hope, <00:02:03.101>uh, <00:02:03.257>people <00:02:03.412>are <00:02:03.568>tricking <00:02:03.723>into <00:02:03.879>our <00:02:04.035>tool.

42
00:02:05.755 --> 00:02:08.975
<00:02:05.755>Um, <00:02:05.895>I <00:02:06.035>see <00:02:06.175>there's <00:02:06.315>also <00:02:06.455>some, <00:02:06.595>already <00:02:06.735>some <00:02:06.875>voting <00:02:07.015>going <00:02:07.155>on.

43
00:02:09.275 --> 00:02:11.135
<00:02:09.275>So, <00:02:09.555>um, <00:02:09.835>how <00:02:10.115>familiar <00:02:10.395>are <00:02:10.675>you

44
00:02:11.135 --> 00:02:13.015
<00:02:11.135>with <00:02:11.484>general <00:02:11.834>machine <00:02:12.185>learning <00:02:12.535>concepts?

45
00:02:13.835 --> 00:02:16.045
<00:02:13.835>Okay, <00:02:14.035>and <00:02:14.235>let's <00:02:14.435>see <00:02:14.635>how <00:02:14.835>the <00:02:15.035>voting <00:02:15.235>goes.

46
00:02:16.145 --> 00:02:18.805
<00:02:16.145>We <00:02:16.300>see, <00:02:16.456>we <00:02:16.611>have <00:02:16.767>some <00:02:16.922>people <00:02:17.078>that, <00:02:17.233>uh, <00:02:17.389>that <00:02:17.545>know

47
00:02:19.315 --> 00:02:22.485
<00:02:19.315>some <00:02:19.515>stuff <00:02:19.715>about <00:02:19.915>machine <00:02:20.115>learning, <00:02:20.315>some <00:02:20.515>experts, <00:02:20.715>um,

48
00:02:22.555 --> 00:02:24.205
<00:02:22.555>some <00:02:22.835>in <00:02:23.115>the <00:02:23.395>middle, <00:02:23.675>and <00:02:23.955>yeah,

49
00:02:24.205 --> 00:02:25.725
<00:02:24.204>also <00:02:24.478>some <00:02:24.752>people <00:02:25.025>that <00:02:25.299>don't <00:02:25.572>know.

50
00:02:25.785 --> 00:02:30.635
<00:02:25.785>So <00:02:25.960>a <00:02:26.135>good <00:02:26.310>distribution, <00:02:26.484>I <00:02:26.660>would <00:02:26.835>say. <00:02:27.010>Um, <00:02:27.185>yeah.

51
00:02:33.375 --> 00:02:35.265
<00:02:33.375>Okay. <00:02:33.550>Good <00:02:33.725>to <00:02:33.900>see <00:02:34.075>that <00:02:34.250>the <00:02:34.425>tool <00:02:34.600>is <00:02:34.775>working.

52
00:02:39.175 --> 00:02:42.745
<00:02:39.175>Okay, <00:02:39.330>then <00:02:39.486>I <00:02:39.641>think <00:02:39.797>this <00:02:39.952>is <00:02:40.108>just <00:02:40.263>the <00:02:40.419>introductory <00:02:40.575>questions.

53
00:02:42.865 --> 00:02:46.785
<00:02:42.865>I <00:02:42.992>think <00:02:43.119>I <00:02:43.246>will <00:02:43.374>close <00:02:43.501>the <00:02:43.628>poll <00:02:43.755>in <00:02:43.883>a <00:02:44.010>few <00:02:44.137>seconds, <00:02:44.265>uh,

54
00:02:46.805 --> 00:02:47.985
<00:02:46.805>to <00:02:47.017>go <00:02:47.229>to <00:02:47.442>the <00:02:47.654>next <00:02:47.867>one.

55
00:02:50.245 --> 00:02:51.615
<00:02:50.245>Okay, <00:02:50.491>going <00:02:50.738>to <00:02:50.984>the <00:02:51.231>next <00:02:51.478>one.

56
00:02:52.035 --> 00:02:53.895
<00:02:52.035>Um, <00:02:52.210>and <00:02:52.385>this <00:02:52.560>is <00:02:52.734>just <00:02:52.910>a <00:02:53.085>little <00:02:53.260>follow <00:02:53.435>up.

57
00:02:53.895 --> 00:02:56.095
<00:02:53.895>So <00:02:54.070>how <00:02:54.245>familiar <00:02:54.420>are, <00:02:54.595>are <00:02:54.770>you <00:02:54.945>with <00:02:55.120>dual <00:02:55.295>network?

58
00:02:56.275 --> 00:02:59.685
<00:02:56.275>So <00:02:56.402>I, <00:02:56.529>I <00:02:56.656>think <00:02:56.784>so, <00:02:56.911>maybe <00:02:57.038>some <00:02:57.165>people <00:02:57.293>said <00:02:57.420>they <00:02:57.547>are <00:02:57.675>familiar

59
00:02:59.685 --> 00:03:00.725
<00:02:59.685>with <00:03:00.153>machine <00:03:00.620>learning.

60
00:03:01.225 --> 00:03:04.525
<00:03:01.225>Um, <00:03:01.505>maybe <00:03:01.785>just <00:03:02.065>a <00:03:02.345>few, <00:03:02.625>uh,

61
00:03:04.765 --> 00:03:06.525
<00:03:04.765>a <00:03:04.998>little <00:03:05.231>bit <00:03:05.464>fewer <00:03:05.698>people <00:03:05.931>are <00:03:06.165>familiar

62
00:03:06.525 --> 00:03:08.325
<00:03:06.525>with <00:03:06.991>Neur <00:03:07.458>network <00:03:07.925>concepts.

63
00:03:18.655 --> 00:03:22.265
<00:03:18.655>Okay, <00:03:18.762>great. <00:03:18.870>I <00:03:18.978>think <00:03:19.085>we <00:03:19.193>have <00:03:19.301>a, <00:03:19.408>a, <00:03:19.516>a <00:03:19.624>wide, <00:03:19.731>uh, <00:03:19.839>wide, <00:03:19.947>uh, <00:03:20.055>um,

64
00:03:22.315 --> 00:03:26.265
<00:03:22.315>range <00:03:22.490>of, <00:03:22.665>of, <00:03:22.840>um, <00:03:23.015>uh, <00:03:23.190>knowledge <00:03:23.365>about <00:03:23.540>the <00:03:23.715>topic.

65
00:03:26.805 --> 00:03:30.025
<00:03:26.805>Um, <00:03:26.932>but <00:03:27.059>I <00:03:27.186>think, <00:03:27.314>I <00:03:27.441>hope <00:03:27.568>our <00:03:27.695>presentation <00:03:27.823>is, <00:03:27.950>is, <00:03:28.077>um, <00:03:28.205>we,

66
00:03:30.045 --> 00:03:32.145
<00:03:30.045>we <00:03:30.185>try <00:03:30.325>to <00:03:30.465>structure <00:03:30.605>it <00:03:30.745>in <00:03:30.885>a <00:03:31.025>way <00:03:31.165>that <00:03:31.305>it's, <00:03:31.445>uh,

67
00:03:32.145 --> 00:03:34.505
<00:03:32.145>everybody <00:03:32.320>can <00:03:32.495>take <00:03:32.670>away <00:03:32.845>something <00:03:33.020>from <00:03:33.195>it, <00:03:33.370>of <00:03:33.545>course,

68
00:03:34.505 --> 00:03:36.025
<00:03:34.505>even <00:03:34.733>though <00:03:34.961>un <00:03:35.189>uncertainty <00:03:35.417>is <00:03:35.645>of <00:03:35.873>course,

69
00:03:36.145 --> 00:03:37.545
<00:03:36.145>a <00:03:36.397>little <00:03:36.649>bit <00:03:36.901>an <00:03:37.153>advanced <00:03:37.405>topic.

70
00:03:41.225 --> 00:03:44.035
<00:03:41.225>Okay. <00:03:41.400>I'll <00:03:41.575>leave <00:03:41.750>that <00:03:41.924>poll <00:03:42.100>open <00:03:42.275>in <00:03:42.450>the <00:03:42.625>background.

71
00:03:44.215 --> 00:03:48.235
<00:03:44.215>Um, <00:03:44.415>and <00:03:44.615>I'll <00:03:44.815>start <00:03:45.015>with <00:03:45.215>the <00:03:45.415>presentation. <00:03:45.615>Okay.

72
00:03:48.335 --> 00:03:50.755
<00:03:48.335>So <00:03:48.568>for, <00:03:48.801>in <00:03:49.035>an <00:03:49.268>introduction <00:03:49.501>to <00:03:49.735>uncertainty.

73
00:03:51.215 --> 00:03:53.235
<00:03:51.215>So <00:03:51.495>let's <00:03:51.775>start <00:03:52.055>with <00:03:52.335>some <00:03:52.615>examples.

74
00:03:53.855 --> 00:03:57.355
<00:03:53.855>Um, <00:03:53.994>so <00:03:54.135>here <00:03:54.274>you <00:03:54.415>can <00:03:54.554>see <00:03:54.695>just <00:03:54.834>a, <00:03:54.975>a <00:03:55.114>few <00:03:55.255>pictures

75
00:03:57.545 --> 00:03:59.955
<00:03:57.545>that <00:03:58.011>show <00:03:58.478>rare <00:03:58.945>events.

76
00:04:00.335 --> 00:04:01.755
<00:04:00.335>And <00:04:00.761>why <00:04:01.187>are <00:04:01.613>rare?

77
00:04:01.815 --> 00:04:03.315
<00:04:01.815>Why, <00:04:02.007>why <00:04:02.200>would <00:04:02.393>you <00:04:02.586>say, <00:04:02.779>would <00:04:02.972>we <00:04:03.165>say

78
00:04:03.315 --> 00:04:05.155
<00:04:03.315>that <00:04:03.548>rare <00:04:03.781>events <00:04:04.015>have <00:04:04.248>something <00:04:04.481>to <00:04:04.715>do

79
00:04:05.155 --> 00:04:07.315
<00:04:05.155>with <00:04:05.505>uncertainty <00:04:05.855>in <00:04:06.205>neural <00:04:06.555>networks?

80
00:04:08.095 --> 00:04:10.155
<00:04:08.095>Um, <00:04:08.561>well, <00:04:09.028>basically <00:04:09.495>it's

81
00:04:10.155 --> 00:04:14.905
<00:04:10.155>because <00:04:10.388>neural <00:04:10.621>networks, <00:04:10.855>um, <00:04:11.088>uh, <00:04:11.321>perform <00:04:11.555>worse

82
00:04:15.085 --> 00:04:16.225
<00:04:15.085>or <00:04:15.427>are <00:04:15.769>more <00:04:16.111>uncertain,

83
00:04:16.645 --> 00:04:18.985
<00:04:16.644>or <00:04:16.800>we <00:04:16.956>would <00:04:17.111>think <00:04:17.267>that <00:04:17.422>neural <00:04:17.578>networks <00:04:17.733>are <00:04:17.889>more <00:04:18.044>uncertain

84
00:04:19.125 --> 00:04:20.745
<00:04:19.125>in, <00:04:19.475>in <00:04:19.825>rare <00:04:20.175>events, <00:04:20.524>mostly

85
00:04:20.745 --> 00:04:23.665
<00:04:20.745>because <00:04:21.095>those <00:04:21.445>events <00:04:21.795>should <00:04:22.145>be

86
00:04:24.175 --> 00:04:26.705
<00:04:24.175>less <00:04:24.375>common <00:04:24.575>in <00:04:24.775>our <00:04:24.975>training <00:04:25.175>data <00:04:25.375>set, <00:04:25.575>right?

87
00:04:27.165 --> 00:04:29.865
<00:04:27.165>Um, <00:04:27.320>and <00:04:27.476>so <00:04:27.631>the <00:04:27.787>neural <00:04:27.942>network <00:04:28.098>has <00:04:28.253>seen <00:04:28.409>those <00:04:28.565>kind

88
00:04:29.865 --> 00:04:32.785
<00:04:29.865>of <00:04:30.040>situations, <00:04:30.215>uh, <00:04:30.390>less <00:04:30.565>or <00:04:30.740>maybe <00:04:30.915>not <00:04:31.090>at <00:04:31.265>all.

89
00:04:33.325 --> 00:04:36.265
<00:04:33.325>And <00:04:33.500>so <00:04:33.675>it <00:04:33.849>would <00:04:34.025>be <00:04:34.200>uncertain <00:04:34.375>about <00:04:34.550>those <00:04:34.724>situations.

90
00:04:36.525 --> 00:04:39.825
<00:04:36.525>So <00:04:36.664>for <00:04:36.804>example, <00:04:36.945>uh, <00:04:37.085>the <00:04:37.224>truck <00:04:37.364>pulling <00:04:37.505>the, <00:04:37.645>the <00:04:37.784>little <00:04:37.924>house,

91
00:04:40.365 --> 00:04:42.365
<00:04:40.365>um, <00:04:41.765>it,

92
00:04:42.675 --> 00:04:46.365
<00:04:42.675>it's <00:04:42.802>maybe <00:04:42.929>if <00:04:43.056>we <00:04:43.184>tried <00:04:43.311>a <00:04:43.438>trained <00:04:43.565>in <00:04:43.693>neur <00:04:43.820>network <00:04:43.947>to <00:04:44.075>classify

93
00:04:46.545 --> 00:04:49.605
<00:04:46.545>or <00:04:46.672>to <00:04:46.799>identify <00:04:46.926>houses, <00:04:47.054>then <00:04:47.181>it <00:04:47.308>would <00:04:47.435>identify <00:04:47.563>it <00:04:47.690>as <00:04:47.817>a <00:04:47.945>house.

94
00:04:49.665 --> 00:04:53.125
<00:04:49.665>But <00:04:49.781>in <00:04:49.898>the <00:04:50.015>concept <00:04:50.131>of, <00:04:50.248>of <00:04:50.365>driving <00:04:50.481>on <00:04:50.598>a <00:04:50.715>road, <00:04:50.831>this <00:04:50.948>is <00:04:51.065>not

95
00:04:53.185 --> 00:04:55.765
<00:04:53.185>as <00:04:53.325>much <00:04:53.465>a <00:04:53.605>house <00:04:53.745>as <00:04:53.885>it <00:04:54.025>is, <00:04:54.165>as <00:04:54.305>it's <00:04:54.445>a <00:04:54.585>trailer,

96
00:04:55.905 --> 00:04:57.325
<00:04:55.905>for <00:04:56.544>example, <00:04:57.183>right?

97
00:04:57.545 --> 00:05:01.445
<00:04:57.545>And <00:04:57.720>so <00:04:57.895>the <00:04:58.070>question <00:04:58.245>is <00:04:58.420>how <00:04:58.595>can <00:04:58.770>we, <00:04:58.945>um,

98
00:05:02.285 --> 00:05:03.885
<00:05:02.285>classify <00:05:02.635>those <00:05:02.985>kind <00:05:03.335>of <00:05:03.685>uncertainties

99
00:05:04.345 --> 00:05:05.605
<00:05:04.345>and, <00:05:04.571>and <00:05:04.798>where <00:05:05.025>they <00:05:05.252>come <00:05:05.479>from?

100
00:05:05.665 --> 00:05:07.605
<00:05:05.665>And <00:05:05.898>in, <00:05:06.131>in <00:05:06.365>this <00:05:06.598>kind <00:05:06.831>of <00:05:07.065>uncertainty?

101
00:05:07.705 --> 00:05:10.685
<00:05:07.705>You'll <00:05:07.860>see <00:05:08.016>in <00:05:08.171>on <00:05:08.327>the <00:05:08.482>next <00:05:08.638>slide, <00:05:08.793>I'll <00:05:08.949>explain, <00:05:09.104>um,

102
00:05:10.905 --> 00:05:14.525
<00:05:10.905>arises <00:05:11.184>from <00:05:11.465>uncertainty <00:05:11.744>about, <00:05:12.025>um, <00:05:12.304>data

103
00:05:14.555 --> 00:05:16.765
<00:05:14.555>that <00:05:14.682>I <00:05:14.809>have <00:05:14.936>not <00:05:15.064>seen, <00:05:15.191>but <00:05:15.318>this <00:05:15.445>is <00:05:15.573>not <00:05:15.700>the <00:05:15.827>only <00:05:15.955>kind

104
00:05:16.765 --> 00:05:18.005
<00:05:16.765>of <00:05:17.044>uncertainty <00:05:17.323>that <00:05:17.602>we <00:05:17.881>have.

105
00:05:19.975 --> 00:05:22.145
<00:05:19.975>Okay? <00:05:20.675>Um, <00:05:21.375>so

106
00:05:22.415 --> 00:05:24.105
<00:05:22.415>what <00:05:22.615>you <00:05:22.815>can <00:05:23.015>do <00:05:23.215>is <00:05:23.415>you <00:05:23.615>can <00:05:23.815>classify

107
00:05:24.105 --> 00:05:25.665
<00:05:24.105>uncertainty <00:05:24.571>into <00:05:25.038>two <00:05:25.505>types.

108
00:05:26.325 --> 00:05:29.105
<00:05:26.325>Um, <00:05:26.525>one <00:05:26.724>type <00:05:26.925>is <00:05:27.125>the <00:05:27.325>lea <00:05:27.525>toric <00:05:27.724>uncertainty,

109
00:05:29.445 --> 00:05:31.865
<00:05:29.445>and <00:05:29.645>the <00:05:29.844>second <00:05:30.045>one <00:05:30.245>is <00:05:30.445>the <00:05:30.645>epistemic <00:05:30.844>uncertainty.

110
00:05:32.605 --> 00:05:36.065
<00:05:32.605>Um, <00:05:32.838>and <00:05:33.071>the <00:05:33.305>epistemic <00:05:33.538>uncertainty <00:05:33.771>is <00:05:34.005>related

111
00:05:36.365 --> 00:05:37.465
<00:05:36.365>or <00:05:36.695>is <00:05:37.025>the <00:05:37.355>uncertainty

112
00:05:37.465 --> 00:05:39.265
<00:05:37.465>that <00:05:37.698>you've <00:05:37.931>seen <00:05:38.165>in <00:05:38.398>the <00:05:38.631>pictures <00:05:38.865>before.

113
00:05:39.615 --> 00:05:42.225
<00:05:39.615>It's <00:05:39.848>basically <00:05:40.081>situations <00:05:40.315>that <00:05:40.548>are <00:05:40.781>very <00:05:41.015>rare

114
00:05:42.485 --> 00:05:44.985
<00:05:42.485>or <00:05:42.718>just <00:05:42.951>situations <00:05:43.185>that <00:05:43.418>you <00:05:43.651>haven't <00:05:43.885>sampled

115
00:05:44.985 --> 00:05:47.345
<00:05:44.985>where <00:05:45.185>you <00:05:45.385>just <00:05:45.585>haven't <00:05:45.785>collected <00:05:45.985>any <00:05:46.185>training <00:05:46.385>data.

116
00:05:47.685 --> 00:05:51.545
<00:05:47.685>So <00:05:47.812>basically <00:05:47.939>the, <00:05:48.066>the <00:05:48.194>part <00:05:48.321>on <00:05:48.448>the <00:05:48.575>right <00:05:48.703>side, <00:05:48.830>um, <00:05:48.957>uh, <00:05:49.085>on,

117
00:05:51.565 --> 00:05:55.015
<00:05:51.565>on <00:05:51.692>this <00:05:51.819>graph <00:05:51.946>where <00:05:52.074>we <00:05:52.201>don't <00:05:52.328>have <00:05:52.455>any <00:05:52.583>green <00:05:52.710>data <00:05:52.837>points, <00:05:52.965>um,

118
00:05:55.195 --> 00:05:58.285
<00:05:55.195>and <00:05:55.350>this <00:05:55.506>can <00:05:55.661>be <00:05:55.817>reduced <00:05:55.972>by <00:05:56.128>collecting <00:05:56.283>more <00:05:56.439>training <00:05:56.594>data,

119
00:05:58.585 --> 00:05:59.645
<00:05:58.585>of <00:05:58.823>course, <00:05:59.061>if <00:05:59.300>you <00:05:59.539>know

120
00:05:59.815 --> 00:06:03.615
<00:05:59.815>where <00:06:00.015>you <00:06:00.215>don't <00:06:00.415>have <00:06:00.615>data <00:06:00.815>training <00:06:01.015>data, <00:06:01.215>um,

121
00:06:03.715 --> 00:06:05.815
<00:06:03.715>and <00:06:03.994>then <00:06:04.275>you <00:06:04.554>have <00:06:04.835>the <00:06:05.114>uncertainty,

122
00:06:06.025 --> 00:06:09.655
<00:06:06.025>which <00:06:06.180>is <00:06:06.336>basically <00:06:06.491>just <00:06:06.647>the <00:06:06.802>noise <00:06:06.958>in <00:06:07.113>my <00:06:07.269>data <00:06:07.424>points.

123
00:06:10.355 --> 00:06:12.375
<00:06:10.355>Um, <00:06:10.588>so <00:06:10.821>if <00:06:11.055>I <00:06:11.288>measure <00:06:11.521>at, <00:06:11.755>at,

124
00:06:12.435 --> 00:06:13.935
<00:06:12.435>say <00:06:12.660>this <00:06:12.885>point <00:06:13.110>here <00:06:13.335>in <00:06:13.560>the <00:06:13.785>middle,

125
00:06:14.365 --> 00:06:16.295
<00:06:14.365>then <00:06:14.598>if <00:06:14.831>I <00:06:15.065>measure <00:06:15.298>it <00:06:15.531>10 <00:06:15.765>times

126
00:06:16.515 --> 00:06:18.935
<00:06:16.515>or <00:06:16.748>three <00:06:16.981>times, <00:06:17.215>I <00:06:17.448>will <00:06:17.681>get <00:06:17.914>slightly

127
00:06:18.935 --> 00:06:20.255
<00:06:18.935>different <00:06:19.331>measurements <00:06:19.727>each <00:06:20.123>time.

128
00:06:20.355 --> 00:06:23.055
<00:06:20.355>And <00:06:20.530>this <00:06:20.705>uncertainty <00:06:20.880>in <00:06:21.055>my <00:06:21.230>measurement <00:06:21.405>is <00:06:21.580>called <00:06:21.755>the

129
00:06:23.305 --> 00:06:24.375
<00:06:23.305>atory <00:06:24.268>uncertainty.

130
00:06:25.155 --> 00:06:29.735
<00:06:25.155>And <00:06:25.310>actually <00:06:25.466>you <00:06:25.621>can, <00:06:25.777>um, <00:06:25.932>separate <00:06:26.088>out <00:06:26.243>these <00:06:26.399>two <00:06:26.554>types

131
00:06:29.735 --> 00:06:31.295
<00:06:29.735>of <00:06:30.435>uncertainty, <00:06:31.135>um,

132
00:06:31.295 --> 00:06:33.775
<00:06:31.295>when <00:06:31.495>calculating <00:06:31.695>the <00:06:31.895>uncertainty <00:06:32.095>of <00:06:32.295>a <00:06:32.495>neural <00:06:32.695>network.

133
00:06:33.875 --> 00:06:36.415
<00:06:33.875>And <00:06:34.002>you <00:06:34.129>can <00:06:34.256>see <00:06:34.384>this <00:06:34.511>in <00:06:34.638>the <00:06:34.765>example <00:06:34.893>images <00:06:35.020>on <00:06:35.147>the <00:06:35.275>bottom

134
00:06:36.805 --> 00:06:39.815
<00:06:36.805>side <00:06:36.980>here, <00:06:37.155>um, <00:06:37.330>where <00:06:37.505>you <00:06:37.680>can <00:06:37.855>see <00:06:38.030>the <00:06:38.205>first

135
00:06:40.405 --> 00:06:42.015
<00:06:40.405>blue <00:06:40.871>bluish <00:06:41.338>greenish <00:06:41.804>image.

136
00:06:42.035 --> 00:06:43.375
<00:06:42.034>And, <00:06:42.436>uh, <00:06:42.839>um, <00:06:43.241>uh,

137
00:06:43.395 --> 00:06:46.055
<00:06:43.395>on <00:06:43.570>the <00:06:43.745>right <00:06:43.919>side <00:06:44.094>is <00:06:44.270>the <00:06:44.445>al <00:06:44.620>Toric <00:06:44.794>uncertainty.

138
00:06:46.995 --> 00:06:48.335
<00:06:46.995>And <00:06:47.236>what <00:06:47.477>you <00:06:47.718>will <00:06:47.959>see <00:06:48.200>is

139
00:06:48.405 --> 00:06:52.895
<00:06:48.405>that <00:06:48.580>the <00:06:48.755>image <00:06:48.929>has <00:06:49.104>higher <00:06:49.280>uncertainty <00:06:49.455>around <00:06:49.630>the <00:06:49.804>edges.

140
00:06:53.755 --> 00:06:55.375
<00:06:53.755>And <00:06:54.105>this <00:06:54.455>makes <00:06:54.805>intuitive <00:06:55.155>defense

141
00:06:55.375 --> 00:07:00.015
<00:06:55.375>because <00:06:55.575>the, <00:06:55.775>um, <00:06:55.975>uh, <00:06:56.175>the, <00:06:56.375>the <00:06:56.575>ator <00:06:56.775>uncertainty,

142
00:07:00.015 --> 00:07:02.135
<00:07:00.015>as <00:07:00.215>I <00:07:00.414>said, <00:07:00.615>is <00:07:00.815>about <00:07:01.015>the <00:07:01.215>measurement <00:07:01.414>noise.

143
00:07:02.435 --> 00:07:03.935
<00:07:02.435>And <00:07:02.772>basically <00:07:03.110>here, <00:07:03.447>what, <00:07:03.785>where,

144
00:07:03.935 --> 00:07:06.015
<00:07:03.935>where <00:07:04.075>the <00:07:04.215>measurement <00:07:04.355>noise <00:07:04.495>comes <00:07:04.635>from <00:07:04.775>is <00:07:04.915>that <00:07:05.055>if <00:07:05.195>you <00:07:05.335>have,

145
00:07:06.395 --> 00:07:07.455
<00:07:06.395>uh, <00:07:06.871>sharp <00:07:07.349>edges,

146
00:07:07.645 --> 00:07:10.855
<00:07:07.645>then <00:07:07.800>if <00:07:07.956>I <00:07:08.111>take <00:07:08.267>the <00:07:08.422>same <00:07:08.578>photo <00:07:08.733>multiple <00:07:08.889>times, <00:07:09.044>um,

147
00:07:10.855 --> 00:07:14.925
<00:07:10.855>because <00:07:11.030>of <00:07:11.205>slight <00:07:11.380>positioning <00:07:11.555>change <00:07:11.730>in <00:07:11.905>the <00:07:12.080>camera <00:07:12.255>or,

148
00:07:15.025 --> 00:07:18.485
<00:07:15.025>or <00:07:15.180>slight, <00:07:15.336>um, <00:07:15.491>change <00:07:15.647>in <00:07:15.802>the <00:07:15.958>exposure <00:07:16.113>of <00:07:16.269>the <00:07:16.424>camera,

149
00:07:18.785 --> 00:07:20.565
<00:07:18.785>you'll <00:07:18.985>have <00:07:19.185>the <00:07:19.385>edge <00:07:19.585>on <00:07:19.785>a <00:07:19.985>different <00:07:20.185>pixel.

150
00:07:20.785 --> 00:07:22.925
<00:07:20.785>And <00:07:21.018>so <00:07:21.251>there's <00:07:21.485>an <00:07:21.718>uncertainty <00:07:21.951>as <00:07:22.185>to

151
00:07:22.925 --> 00:07:24.285
<00:07:22.925>where <00:07:23.231>exactly <00:07:23.537>this <00:07:23.843>edge <00:07:24.149>is.

152
00:07:24.625 --> 00:07:27.405
<00:07:24.625>And <00:07:24.825>compare <00:07:25.025>that <00:07:25.225>to <00:07:25.425>the <00:07:25.625>epistemic <00:07:25.825>uncertainty, <00:07:26.025>uncertainty.

153
00:07:27.405 --> 00:07:29.725
<00:07:27.405>You <00:07:27.604>can <00:07:27.804>see <00:07:28.005>that <00:07:28.205>it's <00:07:28.405>very <00:07:28.604>high, <00:07:28.804>um,

154
00:07:30.225 --> 00:07:31.645
<00:07:30.225>around <00:07:30.651>this, <00:07:31.077>um, <00:07:31.503>sidewalk.

155
00:07:32.065 --> 00:07:34.645
<00:07:32.065>And <00:07:32.205>if <00:07:32.344>you <00:07:32.485>look <00:07:32.625>at <00:07:32.765>the <00:07:32.905>prediction <00:07:33.045>by <00:07:33.185>the <00:07:33.325>neural <00:07:33.465>network,

156
00:07:34.905 --> 00:07:36.685
<00:07:34.905>you <00:07:35.104>see <00:07:35.304>that <00:07:35.505>the <00:07:35.705>sidewalk, <00:07:35.905>if <00:07:36.104>you <00:07:36.304>compare

157
00:07:36.685 --> 00:07:39.325
<00:07:36.685>to <00:07:36.860>the <00:07:37.035>ground <00:07:37.210>truth, <00:07:37.385>is <00:07:37.560>not <00:07:37.735>predicted <00:07:37.910>very <00:07:38.085>well.

158
00:07:39.425 --> 00:07:43.845
<00:07:39.425>And <00:07:39.600>accordingly, <00:07:39.775>the <00:07:39.950>epistemic <00:07:40.125>uncertainty, <00:07:40.300>um, <00:07:40.475>is, <00:07:40.650>is <00:07:40.825>high.

159
00:07:44.065 --> 00:07:47.085
<00:07:44.065>And <00:07:44.205>this <00:07:44.344>could <00:07:44.485>be <00:07:44.625>because <00:07:44.765>we <00:07:44.905>don't <00:07:45.045>have <00:07:45.185>enough <00:07:45.325>data <00:07:45.465>points

160
00:07:47.545 --> 00:07:49.285
<00:07:47.545>for <00:07:47.825>this <00:07:48.105>kind <00:07:48.385>of, <00:07:48.665>of <00:07:48.945>sidewalk.

161
00:07:52.655 --> 00:07:55.245
<00:07:52.655>Okay, <00:07:54.054>next.

162
00:07:55.465 --> 00:07:57.045
<00:07:55.465>So, <00:07:55.931>um, <00:07:56.398>getting <00:07:56.865>into

163
00:07:57.145 --> 00:07:59.845
<00:07:57.145>how <00:07:57.320>can <00:07:57.495>we <00:07:57.669>estimate <00:07:57.844>the <00:07:58.020>uncertainty <00:07:58.195>of <00:07:58.370>neural <00:07:58.544>networks?

164
00:08:00.225 --> 00:08:04.085
<00:08:00.225>Um, <00:08:00.365>one <00:08:00.505>obvious <00:08:00.645>thing <00:08:00.785>that, <00:08:00.925>that <00:08:01.065>a <00:08:01.205>lot <00:08:01.345>of <00:08:01.485>people <00:08:01.625>would,

165
00:08:04.135 --> 00:08:05.925
<00:08:04.135>would <00:08:04.485>think <00:08:04.835>could <00:08:05.185>help, <00:08:05.534>uh,

166
00:08:05.925 --> 00:08:07.645
<00:08:05.925>or <00:08:06.125>could <00:08:06.325>be <00:08:06.525>a <00:08:06.725>a <00:08:06.925>a <00:08:07.125>starting <00:08:07.325>point

167
00:08:07.645 --> 00:08:10.565
<00:08:07.645>for <00:08:07.800>getting <00:08:07.956>an <00:08:08.111>uncertainty <00:08:08.267>estimate <00:08:08.422>is <00:08:08.578>just <00:08:08.733>looking <00:08:08.889>at <00:08:09.044>the,

168
00:08:11.065 --> 00:08:14.525
<00:08:11.065>um, <00:08:11.531>classification <00:08:11.998>scores, <00:08:12.465>um,

169
00:08:14.785 --> 00:08:16.645
<00:08:14.785>of <00:08:14.985>the <00:08:15.185>neural <00:08:15.385>network <00:08:15.585>for <00:08:15.785>the <00:08:15.985>different <00:08:16.185>classes.

170
00:08:16.905 --> 00:08:20.685
<00:08:16.905>So <00:08:17.012>in <00:08:17.120>this <00:08:17.228>example, <00:08:17.335>you <00:08:17.443>see, <00:08:17.551>uh, <00:08:17.658>a <00:08:17.766>image <00:08:17.874>of <00:08:17.981>a, <00:08:18.089>of <00:08:18.197>a <00:08:18.304>plain,

171
00:08:21.025 --> 00:08:24.805
<00:08:21.025>uh, <00:08:21.164>wing, <00:08:21.304>um, <00:08:21.445>and <00:08:21.585>then, <00:08:21.724>then <00:08:21.864>we <00:08:22.005>have <00:08:22.145>some <00:08:22.284>neural <00:08:22.424>network,

172
00:08:25.105 --> 00:08:27.445
<00:08:25.105>and <00:08:25.385>it <00:08:25.665>classifies, <00:08:25.945>um, <00:08:26.225>this <00:08:26.505>image

173
00:08:27.785 --> 00:08:31.285
<00:08:27.785>and <00:08:27.925>it <00:08:28.065>says, <00:08:28.205>okay, <00:08:28.345>it <00:08:28.485>could <00:08:28.625>be, <00:08:28.765>um, <00:08:28.905>with <00:08:29.045>our, <00:08:29.185>um,

174
00:08:31.925 --> 00:08:36.785
<00:08:31.925>software <00:08:32.100>score <00:08:32.275>could <00:08:32.450>be <00:08:32.625>16.3% <00:08:32.799>wing, <00:08:32.975>or <00:08:33.150>13.3% <00:08:33.325>plane,

175
00:08:37.685 --> 00:08:40.265
<00:08:37.684>and <00:08:37.859>then <00:08:38.034>a <00:08:38.209>10% <00:08:38.385>city, <00:08:38.559>which <00:08:38.734>all <00:08:38.909>makes <00:08:39.084>sense

176
00:08:40.265 --> 00:08:42.385
<00:08:40.265>because <00:08:40.381>you <00:08:40.498>see <00:08:40.615>a <00:08:40.731>little <00:08:40.848>bit <00:08:40.965>of <00:08:41.081>a <00:08:41.198>city, <00:08:41.314>you <00:08:41.431>see <00:08:41.548>a <00:08:41.664>wing

177
00:08:42.385 --> 00:08:43.545
<00:08:42.385>and <00:08:42.558>wing <00:08:42.732>is <00:08:42.906>part <00:08:43.081>of <00:08:43.255>a <00:08:43.429>plane.

178
00:08:43.545 --> 00:08:45.145
<00:08:43.544>It <00:08:43.895>all <00:08:44.245>makes <00:08:44.594>sense. <00:08:44.944>Um,

179
00:08:45.525 --> 00:08:49.945
<00:08:45.525>but <00:08:45.680>if <00:08:45.836>you <00:08:45.991>look <00:08:46.147>at, <00:08:46.302>um, <00:08:46.458>those <00:08:46.613>scores, <00:08:46.769>um, <00:08:46.925>um,

180
00:08:50.165 --> 00:08:51.505
<00:08:50.165>in <00:08:50.466>detail, <00:08:50.768>you <00:08:51.069>will <00:08:51.371>find

181
00:08:51.505 --> 00:08:53.945
<00:08:51.505>that <00:08:51.738>they <00:08:51.971>are <00:08:52.205>not <00:08:52.438>very <00:08:52.671>good <00:08:52.905>predictors

182
00:08:53.965 --> 00:08:55.685
<00:08:53.965>of <00:08:54.315>uncertainty <00:08:54.665>of <00:08:55.015>neural <00:08:55.365>networks.

183
00:08:56.345 --> 00:08:57.725
<00:08:56.345>Um, <00:08:56.759>and <00:08:57.173>this <00:08:57.587>is

184
00:08:57.725 --> 00:09:01.085
<00:08:57.725>because <00:08:57.900>neural <00:08:58.075>networks <00:08:58.250>are <00:08:58.425>not <00:08:58.600>trained <00:08:58.775>to <00:08:58.950>assess, <00:08:59.125>um,

185
00:09:01.545 --> 00:09:03.245
<00:09:01.545>how <00:09:01.824>likely <00:09:02.104>they <00:09:02.385>are <00:09:02.665>to <00:09:02.944>fail

186
00:09:03.385 --> 00:09:05.285
<00:09:03.385>or <00:09:03.585>how <00:09:03.785>likely <00:09:03.985>they <00:09:04.185>are <00:09:04.385>to <00:09:04.585>be <00:09:04.785>correct.

187
00:09:06.105 --> 00:09:08.925
<00:09:06.105>Um, <00:09:06.385>and <00:09:06.665>I'll <00:09:06.945>show <00:09:07.225>you <00:09:07.505>how

188
00:09:08.995 --> 00:09:10.765
<00:09:08.995>that <00:09:09.195>can <00:09:09.395>be <00:09:09.595>measured <00:09:09.795>in <00:09:09.995>the <00:09:10.195>next <00:09:10.395>slide.

189
00:09:11.265 --> 00:09:14.775
<00:09:11.265>So <00:09:11.465>what <00:09:11.665>you <00:09:11.865>want <00:09:12.064>when <00:09:12.265>you, <00:09:12.465>when <00:09:12.665>you

190
00:09:14.775 --> 00:09:19.415
<00:09:14.775>what <00:09:14.891>you <00:09:15.008>want <00:09:15.125>in <00:09:15.241>a <00:09:15.358>good <00:09:15.475>uncertainty <00:09:15.591>score <00:09:15.708>is <00:09:15.824>that <00:09:15.941>if <00:09:16.058>I <00:09:16.175>say

191
00:09:20.175 --> 00:09:24.895
<00:09:20.175>I <00:09:20.314>am <00:09:20.454>90% <00:09:20.594>sure, <00:09:20.734>then <00:09:20.875>if <00:09:21.015>I <00:09:21.155>repeat <00:09:21.295>the <00:09:21.435>same, <00:09:21.574>uh,

192
00:09:24.985 --> 00:09:26.455
<00:09:24.985>prediction <00:09:25.426>for <00:09:25.867>the <00:09:26.308>same

193
00:09:26.475 --> 00:09:31.215
<00:09:26.475>or <00:09:26.615>similar, <00:09:26.755>um, <00:09:26.895>sample, <00:09:27.035>uh, <00:09:27.175>a <00:09:27.315>hundred <00:09:27.455>times, <00:09:27.595>then <00:09:27.735>I <00:09:27.875>want

194
00:09:31.215 --> 00:09:35.375
<00:09:31.215>that <00:09:31.370>the, <00:09:31.526>that <00:09:31.681>this <00:09:31.837>prediction <00:09:31.992>is, <00:09:32.148>uh, <00:09:32.303>90 <00:09:32.459>times <00:09:32.615>correct

195
00:09:35.675 --> 00:09:37.455
<00:09:35.675>and <00:09:36.141>10 <00:09:36.608>times <00:09:37.074>wrong.

196
00:09:38.115 --> 00:09:42.215
<00:09:38.115>But, <00:09:38.290>um, <00:09:38.465>if <00:09:38.640>you <00:09:38.815>plot <00:09:38.990>this, <00:09:39.165>um, <00:09:39.340>uh, <00:09:39.515>for,

197
00:09:42.355 --> 00:09:44.855
<00:09:42.355>for <00:09:42.555>some <00:09:42.755>data, <00:09:42.955>um, <00:09:43.155>what <00:09:43.355>you <00:09:43.555>will <00:09:43.755>find

198
00:09:44.885 --> 00:09:47.335
<00:09:44.885>that <00:09:45.040>most <00:09:45.196>neural <00:09:45.351>networks, <00:09:45.507>this <00:09:45.662>is <00:09:45.818>a <00:09:45.973>very <00:09:46.129>common <00:09:46.285>plot.

199
00:09:47.875 --> 00:09:52.015
<00:09:47.875>Um, <00:09:47.991>if <00:09:48.108>you <00:09:48.225>look <00:09:48.341>at <00:09:48.458>the <00:09:48.575>red <00:09:48.691>line <00:09:48.808>for, <00:09:48.925>for <00:09:49.041>90%, <00:09:49.158>um, <00:09:49.275>uh,

200
00:09:52.105 --> 00:09:55.175
<00:09:52.105>confidence <00:09:52.385>score, <00:09:52.665>you <00:09:52.945>would <00:09:53.225>actually <00:09:53.505>find

201
00:09:55.175 --> 00:09:58.575
<00:09:55.175>that <00:09:55.330>you <00:09:55.486>have <00:09:55.641>something <00:09:55.797>like, <00:09:55.952>in <00:09:56.108>this <00:09:56.263>case, <00:09:56.419>70% <00:09:56.574>only,

202
00:09:58.575 --> 00:09:59.855
<00:09:58.575>correct, <00:09:59.727>meaning

203
00:09:59.855 --> 00:10:02.095
<00:09:59.855>that <00:10:00.088>the <00:10:00.321>neural <00:10:00.555>network <00:10:00.788>is <00:10:01.021>actually <00:10:01.255>overconfident.

204
00:10:02.555 --> 00:10:03.655
<00:10:02.555>Um, <00:10:02.752>and <00:10:02.950>on <00:10:03.149>the <00:10:03.347>other <00:10:03.545>hand,

205
00:10:03.715 --> 00:10:07.095
<00:10:03.715>if <00:10:03.855>you <00:10:03.995>look <00:10:04.135>at <00:10:04.275>like <00:10:04.415>10% <00:10:04.555>confidence, <00:10:04.695>then <00:10:04.835>you <00:10:04.975>will <00:10:05.115>see

206
00:10:07.095 --> 00:10:09.535
<00:10:07.095>that <00:10:07.445>actually <00:10:07.795>30% <00:10:08.145>of <00:10:08.495>the,

207
00:10:09.595 --> 00:10:11.895
<00:10:09.595>of <00:10:09.875>the <00:10:10.155>samples <00:10:10.435>predicted <00:10:10.715>was <00:10:10.995>10%

208
00:10:11.895 --> 00:10:13.535
<00:10:11.895>confidence <00:10:12.361>would <00:10:12.828>be <00:10:13.295>correct.

209
00:10:13.675 --> 00:10:16.615
<00:10:13.675>So <00:10:13.814>actually <00:10:13.954>in, <00:10:14.094>in <00:10:14.234>on <00:10:14.375>the, <00:10:14.515>on <00:10:14.655>the <00:10:14.795>lower <00:10:14.935>end, <00:10:15.074>um,

210
00:10:16.615 --> 00:10:18.655
<00:10:16.615>it's <00:10:17.081>under <00:10:17.548>confident, <00:10:18.015>um,

211
00:10:18.675 --> 00:10:20.975
<00:10:18.675>and <00:10:18.849>you <00:10:19.025>get <00:10:19.199>this <00:10:19.375>kind <00:10:19.550>of <00:10:19.724>s-shaped, <00:10:19.900>uh, <00:10:20.074>curve.

212
00:10:21.235 --> 00:10:23.935
<00:10:21.235>And <00:10:21.375>what <00:10:21.515>you <00:10:21.655>ideally <00:10:21.795>want <00:10:21.935>is <00:10:22.075>you <00:10:22.215>want <00:10:22.355>an <00:10:22.495>even <00:10:22.635>distribution

213
00:10:23.935 --> 00:10:27.335
<00:10:23.935>of <00:10:24.074>the <00:10:24.214>scores <00:10:24.354>so <00:10:24.494>that <00:10:24.635>when <00:10:24.775>I <00:10:24.915>say <00:10:25.055>I'm <00:10:25.194>30% <00:10:25.334>confident,

214
00:10:27.615 --> 00:10:30.335
<00:10:27.615>actually <00:10:27.848>30% <00:10:28.081>of <00:10:28.315>the, <00:10:28.548>um, <00:10:28.781>empirical <00:10:29.015>samples

215
00:10:30.365 --> 00:10:33.375
<00:10:30.365>that <00:10:30.520>I <00:10:30.676>feed <00:10:30.831>into <00:10:30.987>the <00:10:31.142>network <00:10:31.298>are <00:10:31.453>actually, <00:10:31.609>uh, <00:10:31.765>correct

216
00:10:33.555 --> 00:10:34.815
<00:10:33.555>and <00:10:34.122>70% <00:10:34.689>wrong.

217
00:10:35.115 --> 00:10:37.615
<00:10:35.115>And <00:10:35.255>so <00:10:35.395>you <00:10:35.535>want <00:10:35.675>this, <00:10:35.815>uh, <00:10:35.955>this <00:10:36.095>line, <00:10:36.235>this <00:10:36.375>blue <00:10:36.515>line

218
00:10:37.675 --> 00:10:40.695
<00:10:37.675>to <00:10:37.875>be <00:10:38.074>this <00:10:38.275>black <00:10:38.474>dotted, <00:10:38.675>uh, <00:10:38.875>diagonal <00:10:39.074>line.

219
00:10:40.995 --> 00:10:44.655
<00:10:40.995>And <00:10:41.122>what <00:10:41.249>you <00:10:41.376>can <00:10:41.504>do <00:10:41.631>is, <00:10:41.758>uh, <00:10:41.885>you <00:10:42.013>can <00:10:42.140>calibrate <00:10:42.267>your <00:10:42.395>output,

220
00:10:45.275 --> 00:10:47.655
<00:10:45.275>um, <00:10:45.625>with <00:10:45.975>some <00:10:46.324>methods, <00:10:46.675>um,

221
00:10:47.715 --> 00:10:51.615
<00:10:47.715>or <00:10:47.890>a <00:10:48.065>better <00:10:48.240>uncertainty <00:10:48.415>estimation, <00:10:48.590>um, <00:10:48.765>to <00:10:48.940>be <00:10:49.115>more

222
00:10:52.245 --> 00:10:53.855
<00:10:52.245>like <00:10:52.478>this, <00:10:52.711>and <00:10:52.945>you <00:10:53.178>have <00:10:53.411>a <00:10:53.645>better

223
00:10:54.135 --> 00:10:55.175
<00:10:54.135>estimate <00:10:54.447>of <00:10:54.759>your <00:10:55.070>uncertainty.

224
00:10:57.925 --> 00:11:01.135
<00:10:57.925>Okay, <00:10:58.099>so <00:10:58.275>let's <00:10:58.449>start <00:10:58.625>with <00:10:58.800>one <00:10:58.974>simple <00:10:59.150>method, <00:10:59.324>um,

225
00:11:01.725 --> 00:11:03.975
<00:11:01.725>that <00:11:01.958>has <00:11:02.191>some <00:11:02.425>good <00:11:02.658>advantages <00:11:02.891>and, <00:11:03.125>and,

226
00:11:04.235 --> 00:11:08.015
<00:11:04.235>but <00:11:04.468>also <00:11:04.701>some <00:11:04.935>disadvantages <00:11:05.168>about <00:11:05.401>uncertainty <00:11:05.635>estimation.

227
00:11:08.155 --> 00:11:11.335
<00:11:08.155>Um, <00:11:08.388>this <00:11:08.621>approach <00:11:08.855>is <00:11:09.088>called <00:11:09.321>alert, <00:11:09.555>um,

228
00:11:11.915 --> 00:11:13.855
<00:11:11.915>and <00:11:12.265>it <00:11:12.615>doesn't <00:11:12.964>actually, <00:11:13.314>um,

229
00:11:14.375 --> 00:11:16.255
<00:11:14.375>estimate <00:11:14.608>the <00:11:14.841>uncertainty <00:11:15.075>of <00:11:15.308>the <00:11:15.541>neural <00:11:15.775>network.

230
00:11:16.795 --> 00:11:19.735
<00:11:16.795>Um, <00:11:16.935>but <00:11:17.074>what <00:11:17.214>it <00:11:17.354>rather <00:11:17.495>does, <00:11:17.635>so <00:11:17.775>you <00:11:17.915>see, <00:11:18.055>um, <00:11:18.194>with,

231
00:11:19.795 --> 00:11:23.295
<00:11:19.795>you <00:11:19.935>have <00:11:20.074>your <00:11:20.214>base <00:11:20.354>system <00:11:20.495>or <00:11:20.635>your <00:11:20.775>base <00:11:20.915>model <00:11:21.055>trained, <00:11:21.194>um,

232
00:11:23.675 --> 00:11:27.055
<00:11:23.675>and <00:11:23.802>this <00:11:23.929>base <00:11:24.056>model <00:11:24.184>has <00:11:24.311>some <00:11:24.438>error <00:11:24.565>on, <00:11:24.693>on, <00:11:24.820>on <00:11:24.947>the <00:11:25.074>data,

233
00:11:27.355 --> 00:11:28.895
<00:11:27.355>on <00:11:27.586>your, <00:11:27.817>say, <00:11:28.048>on <00:11:28.279>your <00:11:28.510>test <00:11:28.741>data.

234
00:11:29.475 --> 00:11:30.655
<00:11:29.475>Um, <00:11:30.006>so <00:11:30.537>it'll

235
00:11:30.655 --> 00:11:33.175
<00:11:30.655>of <00:11:30.795>course <00:11:30.935>not <00:11:31.074>be <00:11:31.214>a <00:11:31.355>hundred <00:11:31.495>percent <00:11:31.635>right <00:11:31.775>of <00:11:31.915>the <00:11:32.055>time.

236
00:11:33.675 --> 00:11:36.815
<00:11:33.675>Um, <00:11:33.802>and <00:11:33.929>what <00:11:34.056>the <00:11:34.184>alert <00:11:34.311>system <00:11:34.438>does <00:11:34.565>is <00:11:34.693>you <00:11:34.820>train <00:11:34.947>a <00:11:35.074>second

237
00:11:36.825 --> 00:11:37.935
<00:11:36.825>model, <00:11:37.324>doesn't <00:11:37.824>have

238
00:11:37.935 --> 00:11:40.015
<00:11:37.935>to <00:11:38.109>be <00:11:38.285>the <00:11:38.459>same <00:11:38.635>model, <00:11:38.810>can <00:11:38.984>be <00:11:39.160>completely <00:11:39.334>different.

239
00:11:40.015 --> 00:11:43.655
<00:11:40.015>One <00:11:40.155>can <00:11:40.295>be <00:11:40.435>a <00:11:40.574>simpler, <00:11:40.715>much <00:11:40.855>simpler <00:11:40.995>model, <00:11:41.135>in <00:11:41.275>fact, <00:11:41.415>um,

240
00:11:44.035 --> 00:11:45.455
<00:11:44.035>and <00:11:44.354>you <00:11:44.674>train <00:11:44.993>it, <00:11:45.313>um,

241
00:11:45.555 --> 00:11:47.975
<00:11:45.555>to <00:11:45.788>predict <00:11:46.021>when <00:11:46.255>your <00:11:46.488>neural <00:11:46.721>network <00:11:46.954>fails.

242
00:11:48.715 --> 00:11:53.365
<00:11:48.715>Um, <00:11:48.948>and <00:11:49.181>this <00:11:49.415>can <00:11:49.648>give <00:11:49.881>you, <00:11:50.115>um,

243
00:11:53.685 --> 00:11:57.285
<00:11:53.685>a <00:11:53.885>very <00:11:54.084>cheap, <00:11:54.285>um, <00:11:54.484>way <00:11:54.685>to <00:11:54.885>estimate, <00:11:55.084>to,

244
00:11:57.345 --> 00:12:00.925
<00:11:57.345>to <00:11:57.485>tell <00:11:57.625>when <00:11:57.765>your <00:11:57.905>neural <00:11:58.045>network, <00:11:58.185>uh, <00:11:58.325>will, <00:11:58.465>will <00:11:58.605>be <00:11:58.745>wrong.

245
00:12:01.505 --> 00:12:03.605
<00:12:01.505>Um, <00:12:01.660>and <00:12:01.816>the <00:12:01.971>cool <00:12:02.127>thing <00:12:02.282>about <00:12:02.438>this <00:12:02.593>is <00:12:02.749>you <00:12:02.905>can,

246
00:12:03.625 --> 00:12:05.925
<00:12:03.625>you <00:12:03.780>don't <00:12:03.936>even <00:12:04.091>have <00:12:04.247>to <00:12:04.402>run <00:12:04.558>your <00:12:04.713>base <00:12:04.869>system, <00:12:05.025>you

247
00:12:05.945 --> 00:12:08.645
<00:12:05.945>or <00:12:06.085>your <00:12:06.225>base <00:12:06.365>model, <00:12:06.505>you <00:12:06.645>can <00:12:06.785>just <00:12:06.925>run <00:12:07.065>the <00:12:07.205>alert <00:12:07.345>system

248
00:12:08.745 --> 00:12:12.765
<00:12:08.745>to <00:12:08.885>say, <00:12:09.025>I <00:12:09.165>am, <00:12:09.305>uh, <00:12:09.445>I, <00:12:09.585>I <00:12:09.725>think <00:12:09.865>in <00:12:10.005>this <00:12:10.145>sample,

249
00:12:12.945 --> 00:12:14.685
<00:12:12.945>our <00:12:13.225>neural <00:12:13.505>network <00:12:13.785>would <00:12:14.065>perform <00:12:14.345>poorly.

250
00:12:15.585 --> 00:12:19.645
<00:12:15.585>Um, <00:12:15.740>the <00:12:15.896>problem <00:12:16.051>is, <00:12:16.207>um, <00:12:16.362>with <00:12:16.518>this, <00:12:16.673>um, <00:12:16.829>approach <00:12:16.985>is

251
00:12:19.645 --> 00:12:22.805
<00:12:19.645>that <00:12:19.819>it <00:12:19.995>only <00:12:20.170>works <00:12:20.345>if, <00:12:20.520>uh, <00:12:20.694>for, <00:12:20.870>for <00:12:21.045>data

252
00:12:23.135 --> 00:12:24.885
<00:12:23.135>where <00:12:23.415>we <00:12:23.694>have <00:12:23.975>seen <00:12:24.255>relevant <00:12:24.535>failure.

253
00:12:24.985 --> 00:12:27.005
<00:12:24.985>So <00:12:25.140>where, <00:12:25.296>so <00:12:25.451>where <00:12:25.607>we <00:12:25.762>have <00:12:25.918>seen <00:12:26.073>on <00:12:26.229>our <00:12:26.385>data

254
00:12:27.005 --> 00:12:30.365
<00:12:27.005>that <00:12:27.160>the <00:12:27.316>neural <00:12:27.471>network <00:12:27.627>actually <00:12:27.782>fails, <00:12:27.938>it <00:12:28.093>doesn't <00:12:28.249>work <00:12:28.405>for,

255
00:12:30.825 --> 00:12:33.525
<00:12:30.825>for, <00:12:30.980>um, <00:12:31.136>or <00:12:31.291>it's <00:12:31.447>not <00:12:31.602>guaranteed <00:12:31.758>to <00:12:31.913>work <00:12:32.069>for <00:12:32.225>data

256
00:12:33.525 --> 00:12:34.525
<00:12:33.525>that <00:12:33.750>we <00:12:33.975>have <00:12:34.199>never <00:12:34.425>seen.

257
00:12:37.925 --> 00:12:42.525
<00:12:37.925>Okay? <00:12:39.324>So

258
00:12:42.635 --> 00:12:45.845
<00:12:42.635>then <00:12:42.790>we'll <00:12:42.946>move <00:12:43.101>on <00:12:43.257>to <00:12:43.412>a <00:12:43.568>much <00:12:43.723>more <00:12:43.879>advanced, <00:12:44.035>um,

259
00:12:46.445 --> 00:12:48.205
<00:12:46.445>I <00:12:46.678>would <00:12:46.911>say <00:12:47.145>much <00:12:47.378>more <00:12:47.611>advanced, <00:12:47.845>um,

260
00:12:48.485 --> 00:12:50.205
<00:12:48.485>approach <00:12:48.835>called <00:12:49.185>the <00:12:49.535>gall <00:12:49.885>approach,

261
00:12:50.375 --> 00:12:54.245
<00:12:50.375>which <00:12:50.515>is <00:12:50.655>based <00:12:50.795>on <00:12:50.935>dropout, <00:12:51.075>um, <00:12:51.215>sampling <00:12:51.355>of <00:12:51.495>a <00:12:51.635>neural <00:12:51.775>network.

262
00:12:54.745 --> 00:12:57.445
<00:12:54.745>And <00:12:54.885>before <00:12:55.025>I <00:12:55.165>get <00:12:55.305>into <00:12:55.445>the <00:12:55.585>details <00:12:55.725>of, <00:12:55.865>of <00:12:56.005>the <00:12:56.145>method,

263
00:12:57.545 --> 00:12:59.045
<00:12:57.545>let <00:12:57.814>me <00:12:58.084>explain <00:12:58.354>what <00:12:58.625>dropout <00:12:58.895>is

264
00:12:59.045 --> 00:13:01.285
<00:12:59.045>because <00:12:59.245>we <00:12:59.444>have <00:12:59.645>some <00:12:59.844>people <00:13:00.045>in <00:13:00.245>the <00:13:00.444>car,

265
00:13:01.285 --> 00:13:02.605
<00:13:01.285>of <00:13:01.522>course, <00:13:01.760>we <00:13:01.997>might <00:13:02.235>not <00:13:02.473>know.

266
00:13:03.105 --> 00:13:05.885
<00:13:03.105>Um, <00:13:03.221>so <00:13:03.338>if <00:13:03.455>you <00:13:03.571>look <00:13:03.688>at <00:13:03.805>the <00:13:03.921>right <00:13:04.038>side <00:13:04.155>here <00:13:04.271>on <00:13:04.388>the <00:13:04.505>little

267
00:13:05.885 --> 00:13:09.405
<00:13:05.885>example, <00:13:06.040>neural <00:13:06.196>network, <00:13:06.351>um, <00:13:06.507>you <00:13:06.662>see <00:13:06.818>on <00:13:06.973>the <00:13:07.129>bottom, <00:13:07.285>uh,

268
00:13:09.405 --> 00:13:12.005
<00:13:09.405>the <00:13:09.560>input <00:13:09.716>neuros, <00:13:09.871>and <00:13:10.027>then <00:13:10.182>we <00:13:10.338>have <00:13:10.493>two <00:13:10.649>hidden <00:13:10.805>layers

269
00:13:12.065 --> 00:13:13.845
<00:13:12.065>of <00:13:12.415>neurons, <00:13:12.765>uh, <00:13:13.115>fully <00:13:13.465>connected

270
00:13:13.845 --> 00:13:16.925
<00:13:13.845>and <00:13:14.078>then <00:13:14.311>output <00:13:14.545>layers, <00:13:14.778>uh, <00:13:15.011>output <00:13:15.245>neuron.

271
00:13:17.425 --> 00:13:21.205
<00:13:17.425>Um, <00:13:17.552>and <00:13:17.679>what <00:13:17.806>dropout <00:13:17.934>does <00:13:18.061>is <00:13:18.188>we <00:13:18.315>ca <00:13:18.443>we <00:13:18.570>just <00:13:18.697>delete <00:13:18.824>basically,

272
00:13:21.345 --> 00:13:24.525
<00:13:21.345>um, <00:13:21.520>for <00:13:21.695>temporarily, <00:13:21.870>um, <00:13:22.045>some <00:13:22.220>of <00:13:22.395>the <00:13:22.570>neurons <00:13:22.745>in,

273
00:13:24.545 --> 00:13:25.925
<00:13:24.545>in <00:13:24.722>this <00:13:24.899>case, <00:13:25.077>in <00:13:25.254>all <00:13:25.432>of <00:13:25.609>the <00:13:25.786>layers,

274
00:13:26.265 --> 00:13:30.085
<00:13:26.265>and <00:13:26.392>run <00:13:26.519>the <00:13:26.646>sample <00:13:26.774>through <00:13:26.901>from <00:13:27.028>the <00:13:27.155>bottom <00:13:27.283>to <00:13:27.410>the <00:13:27.537>top, <00:13:27.665>um,

275
00:13:30.165 --> 00:13:33.005
<00:13:30.165>through <00:13:30.365>the <00:13:30.564>network, <00:13:30.765>um, <00:13:30.964>and <00:13:31.165>get <00:13:31.365>a <00:13:31.564>prediction.

276
00:13:33.305 --> 00:13:36.605
<00:13:33.305>And <00:13:33.460>importantly, <00:13:33.616>um, <00:13:33.771>if <00:13:33.927>you <00:13:34.082>want <00:13:34.238>a <00:13:34.393>dropout <00:13:34.549>neural <00:13:34.704>network,

277
00:13:36.865 --> 00:13:39.245
<00:13:36.865>you <00:13:37.065>should <00:13:37.265>train <00:13:37.465>the <00:13:37.665>network <00:13:37.865>also <00:13:38.065>with <00:13:38.265>dropout.

278
00:13:39.245 --> 00:13:41.285
<00:13:39.245>So <00:13:39.400>it's <00:13:39.556>not <00:13:39.711>you <00:13:39.867>train <00:13:40.022>the <00:13:40.178>neural <00:13:40.333>network <00:13:40.489>like <00:13:40.645>this,

279
00:13:41.465 --> 00:13:42.725
<00:13:41.465>you <00:13:41.748>train <00:13:42.032>it <00:13:42.315>with <00:13:42.599>dropout.

280
00:13:42.745 --> 00:13:45.885
<00:13:42.745>And <00:13:42.920>then, <00:13:43.095>um, <00:13:43.270>we <00:13:43.445>can <00:13:43.620>do <00:13:43.795>at <00:13:43.970>inference <00:13:44.145>time,

281
00:13:46.185 --> 00:13:49.445
<00:13:46.185>we <00:13:46.340>can <00:13:46.496>run <00:13:46.651>the <00:13:46.807>sample <00:13:46.962>the <00:13:47.118>same <00:13:47.273>sample <00:13:47.429>multiple <00:13:47.584>times

282
00:13:49.475 --> 00:13:50.965
<00:13:49.475>through <00:13:49.922>the <00:13:50.369>network <00:13:50.816>with,

283
00:13:51.065 --> 00:13:54.205
<00:13:51.065>by <00:13:51.345>disabling <00:13:51.625>different <00:13:51.905>sets <00:13:52.185>of <00:13:52.465>neurons.

284
00:13:54.205 --> 00:13:56.165
<00:13:54.205>So, <00:13:54.438>so <00:13:54.671>we <00:13:54.905>would <00:13:55.138>disable <00:13:55.371>this <00:13:55.605>set,

285
00:13:56.165 --> 00:13:57.845
<00:13:56.165>and <00:13:56.339>then <00:13:56.515>in <00:13:56.689>the <00:13:56.865>next <00:13:57.040>one <00:13:57.214>we <00:13:57.390>would, <00:13:57.564>uh,

286
00:13:58.115 --> 00:14:01.125
<00:13:58.115>disable <00:13:58.315>a <00:13:58.515>different <00:13:58.715>set <00:13:58.915>to <00:13:59.115>get <00:13:59.315>different, <00:13:59.515>uh,

287
00:14:01.285 --> 00:14:03.925
<00:14:01.285>slightly <00:14:01.485>different <00:14:01.685>predictions, <00:14:01.885>um, <00:14:02.084>for <00:14:02.285>the <00:14:02.485>same <00:14:02.685>sample.

288
00:14:04.345 --> 00:14:06.725
<00:14:04.345>And <00:14:04.520>it <00:14:04.695>turns <00:14:04.870>out <00:14:05.045>that <00:14:05.220>in <00:14:05.395>this <00:14:05.570>paper <00:14:05.745>dropout

289
00:14:06.725 --> 00:14:10.245
<00:14:06.725>as <00:14:06.880>a <00:14:07.036>patient <00:14:07.191>approximation, <00:14:07.347>um, <00:14:07.502>from <00:14:07.658>the <00:14:07.813>people <00:14:07.969>by <00:14:08.125>the,

290
00:14:10.275 --> 00:14:13.405
<00:14:10.275>from <00:14:10.449>the <00:14:10.625>Cambridge <00:14:10.800>University, <00:14:10.975>they <00:14:11.150>showed <00:14:11.324>that <00:14:11.500>this, <00:14:11.675>um,

291
00:14:13.855 --> 00:14:16.445
<00:14:13.855>those <00:14:14.055>samples <00:14:14.255>that <00:14:14.455>we <00:14:14.655>draw <00:14:14.855>from <00:14:15.055>different <00:14:15.255>runs

292
00:14:16.445 --> 00:14:21.045
<00:14:16.445>through <00:14:16.572>a <00:14:16.699>dropout <00:14:16.826>network <00:14:16.954>can <00:14:17.081>be <00:14:17.208>used <00:14:17.335>as <00:14:17.463>a, <00:14:17.590>as <00:14:17.717>an <00:14:17.845>estimator

293
00:14:21.185 --> 00:14:23.725
<00:14:21.185>for, <00:14:21.535>uh, <00:14:21.885>neural <00:14:22.234>network <00:14:22.584>uncertainty.

294
00:14:25.095 --> 00:14:26.275
<00:14:25.095>Um, <00:14:25.449>so <00:14:25.803>yeah, <00:14:26.156>we,

295
00:14:26.295 --> 00:14:28.355
<00:14:26.295>and <00:14:26.435>you <00:14:26.574>can <00:14:26.714>see <00:14:26.854>that <00:14:26.995>on <00:14:27.135>the <00:14:27.275>right <00:14:27.415>side <00:14:27.555>at <00:14:27.694>the

296
00:14:28.355 --> 00:14:29.435
<00:14:28.355>bottom <00:14:28.549>right <00:14:28.743>side <00:14:28.938>in <00:14:29.132>this <00:14:29.327>image.

297
00:14:29.435 --> 00:14:30.675
<00:14:29.435>If <00:14:29.594>we <00:14:29.753>don't <00:14:29.913>do <00:14:30.072>it, <00:14:30.232>we <00:14:30.391>just <00:14:30.550>get

298
00:14:30.735 --> 00:14:32.235
<00:14:30.735>for <00:14:31.005>the <00:14:31.275>prediction <00:14:31.545>here, <00:14:31.815>straight <00:14:32.085>line.

299
00:14:32.575 --> 00:14:35.595
<00:14:32.575>And <00:14:32.702>if <00:14:32.829>we <00:14:32.956>do <00:14:33.084>it <00:14:33.211>with <00:14:33.338>a, <00:14:33.465>um, <00:14:33.593>a <00:14:33.720>dropout <00:14:33.847>sampling, <00:14:33.975>we,

300
00:14:35.595 --> 00:14:39.315
<00:14:35.595>we <00:14:35.722>get <00:14:35.849>a <00:14:35.976>nice <00:14:36.104>distribution <00:14:36.231>that <00:14:36.358>gets <00:14:36.485>wider <00:14:36.613>in <00:14:36.740>this <00:14:36.867>case, <00:14:36.995>um,

301
00:14:39.495 --> 00:14:42.115
<00:14:39.495>as <00:14:39.695>we <00:14:39.895>move <00:14:40.095>further <00:14:40.295>into <00:14:40.495>the <00:14:40.695>prediction <00:14:40.895>horizon.

302
00:14:45.835 --> 00:14:48.205
<00:14:45.835>Okay. <00:14:46.035>What <00:14:46.235>are <00:14:46.435>some <00:14:46.635>properties <00:14:46.835>of <00:14:47.035>this <00:14:47.235>approach?

303
00:14:48.825 --> 00:14:53.785
<00:14:48.825>So, <00:14:48.952>um, <00:14:49.079>the, <00:14:49.206>the <00:14:49.334>first <00:14:49.461>one <00:14:49.588>is <00:14:49.715>that <00:14:49.843>ideally <00:14:49.970>you <00:14:50.097>would <00:14:50.225>want

304
00:14:53.785 --> 00:14:55.385
<00:14:53.785>to <00:14:54.251>have <00:14:54.718>dropout <00:14:55.185>everywhere.

305
00:14:55.565 --> 00:14:58.865
<00:14:55.565>So <00:14:55.692>if <00:14:55.819>I <00:14:55.946>go <00:14:56.074>one <00:14:56.201>slide <00:14:56.328>back, <00:14:56.455>uh, <00:14:56.583>on <00:14:56.710>the <00:14:56.837>right <00:14:56.965>here,

306
00:14:58.885 --> 00:15:00.785
<00:14:58.885>you <00:14:59.060>can <00:14:59.235>see <00:14:59.410>we <00:14:59.585>do <00:14:59.760>dropout <00:14:59.935>in <00:15:00.110>every <00:15:00.285>layer,

307
00:15:01.285 --> 00:15:03.265
<00:15:01.285>but, <00:15:01.751>um, <00:15:02.218>this <00:15:02.685>expensive.

308
00:15:03.485 --> 00:15:06.985
<00:15:03.485>So, <00:15:03.612>uh, <00:15:03.739>especially <00:15:03.866>if <00:15:03.994>I <00:15:04.121>only <00:15:04.248>do <00:15:04.375>dropout <00:15:04.503>in <00:15:04.630>the <00:15:04.757>last <00:15:04.885>layer

309
00:15:07.085 --> 00:15:10.385
<00:15:07.085>and <00:15:07.225>keep <00:15:07.365>all <00:15:07.505>of <00:15:07.645>the <00:15:07.785>neurons <00:15:07.925>in <00:15:08.065>the <00:15:08.205>previous <00:15:08.345>layers, <00:15:08.485>um,

310
00:15:10.445 --> 00:15:13.225
<00:15:10.445>to <00:15:10.645>draw <00:15:10.845>multiple <00:15:11.045>samples <00:15:11.245>for <00:15:11.445>the <00:15:11.645>same <00:15:11.845>prediction,

311
00:15:13.345 --> 00:15:16.505
<00:15:13.345>I <00:15:13.500>just <00:15:13.656>have <00:15:13.811>to <00:15:13.967>recalculate <00:15:14.122>the <00:15:14.278>last <00:15:14.433>layer, <00:15:14.589>which <00:15:14.745>is

312
00:15:16.505 --> 00:15:19.705
<00:15:16.505>of <00:15:16.680>course, <00:15:16.855>much, <00:15:17.030>much, <00:15:17.205>much <00:15:17.380>more <00:15:17.555>cheap <00:15:17.730>in <00:15:17.905>terms

313
00:15:19.725 --> 00:15:21.625
<00:15:19.725>of <00:15:20.075>processing <00:15:20.425>power <00:15:20.775>than <00:15:21.125>going

314
00:15:21.625 --> 00:15:23.265
<00:15:21.625>through <00:15:21.905>the <00:15:22.185>whole <00:15:22.465>network <00:15:22.745>multiple <00:15:23.025>times.

315
00:15:24.005 --> 00:15:26.465
<00:15:24.005>So <00:15:24.180>what <00:15:24.355>we <00:15:24.530>are <00:15:24.705>interested <00:15:24.880>in, <00:15:25.055>in <00:15:25.230>is, <00:15:25.405>um,

316
00:15:26.535 --> 00:15:28.945
<00:15:26.535>whether <00:15:26.885>the <00:15:27.235>neural <00:15:27.584>network, <00:15:27.935>uh,

317
00:15:28.945 --> 00:15:31.785
<00:15:28.945>whether <00:15:29.100>it <00:15:29.256>makes <00:15:29.411>a <00:15:29.567>difference, <00:15:29.722>uh, <00:15:29.878>where <00:15:30.033>I <00:15:30.189>do <00:15:30.345>dropout.

318
00:15:31.845 --> 00:15:35.785
<00:15:31.845>And <00:15:31.961>as <00:15:32.078>you <00:15:32.195>can <00:15:32.311>see <00:15:32.428>in <00:15:32.545>this <00:15:32.661>graph, <00:15:32.778>um, <00:15:32.895>it <00:15:33.011>turns <00:15:33.128>out <00:15:33.245>no.

319
00:15:35.845 --> 00:15:38.785
<00:15:35.845>So <00:15:36.020>we <00:15:36.195>can <00:15:36.370>actually <00:15:36.545>just <00:15:36.720>do <00:15:36.895>dropout <00:15:37.070>in <00:15:37.245>the,

320
00:15:38.805 --> 00:15:41.345
<00:15:38.805>in <00:15:39.084>the <00:15:39.364>last <00:15:39.645>convolutional <00:15:39.925>layer, <00:15:40.204>um,

321
00:15:41.765 --> 00:15:44.105
<00:15:41.765>and <00:15:42.045>it <00:15:42.324>performs, <00:15:42.605>um, <00:15:42.885>as <00:15:43.165>well

322
00:15:44.165 --> 00:15:46.265
<00:15:44.165>as <00:15:44.444>doing <00:15:44.724>dropout <00:15:45.005>in <00:15:45.285>earlier <00:15:45.564>layers.

323
00:15:46.685 --> 00:15:48.305
<00:15:46.685>So <00:15:47.035>this <00:15:47.385>is, <00:15:47.734>uh, <00:15:48.084>good.

324
00:15:49.495 --> 00:15:52.315
<00:15:49.495>Um, <00:15:49.635>but <00:15:49.775>still, <00:15:49.915>what <00:15:50.055>are <00:15:50.195>some <00:15:50.335>of <00:15:50.475>the <00:15:50.615>problems <00:15:50.755>with, <00:15:50.895>uh,

325
00:15:52.345 --> 00:15:54.515
<00:15:52.345>dropout <00:15:52.695>sampling <00:15:53.045>for, <00:15:53.395>for <00:15:53.745>application?

326
00:15:54.775 --> 00:15:57.795
<00:15:54.775>Um, <00:15:54.930>so <00:15:55.086>we <00:15:55.241>still <00:15:55.397>have <00:15:55.552>to <00:15:55.708>run <00:15:55.863>it <00:15:56.019>multiple <00:15:56.175>times.

327
00:15:58.495 --> 00:16:00.835
<00:15:58.495>Um, <00:15:58.775>and <00:15:59.055>the <00:15:59.335>question <00:15:59.615>is, <00:15:59.895>um,

328
00:16:00.895 --> 00:16:02.635
<00:16:00.895>how <00:16:01.175>fast <00:16:01.454>can <00:16:01.735>this <00:16:02.015>be <00:16:02.295>implemented?

329
00:16:03.375 --> 00:16:06.715
<00:16:03.375>And <00:16:03.530>of <00:16:03.686>course, <00:16:03.841>taking <00:16:03.997>more <00:16:04.152>samples <00:16:04.308>tends <00:16:04.463>to <00:16:04.619>be <00:16:04.775>better.

330
00:16:06.825 --> 00:16:08.195
<00:16:06.825>I'll <00:16:07.030>show <00:16:07.236>that <00:16:07.441>on <00:16:07.647>the <00:16:07.852>next <00:16:08.058>slide.

331
00:16:08.895 --> 00:16:10.195
<00:16:08.895>Um, <00:16:09.187>and <00:16:09.480>the <00:16:09.772>question <00:16:10.065>is,

332
00:16:10.255 --> 00:16:12.915
<00:16:10.255>can <00:16:10.430>we <00:16:10.605>accelerate <00:16:10.780>our <00:16:10.955>neural <00:16:11.130>network <00:16:11.305>fast <00:16:11.480>enough <00:16:11.655>so

333
00:16:12.915 --> 00:16:15.995
<00:16:12.915>that <00:16:13.089>we <00:16:13.265>can <00:16:13.439>actually <00:16:13.615>afford <00:16:13.790>to <00:16:13.964>run <00:16:14.140>10 <00:16:14.314>samples

334
00:16:16.255 --> 00:16:18.715
<00:16:16.255>or <00:16:16.455>a <00:16:16.655>hundred <00:16:16.855>samples <00:16:17.055>to <00:16:17.255>get <00:16:17.455>a <00:16:17.655>good,

335
00:16:18.735 --> 00:16:19.915
<00:16:18.735>um, <00:16:19.266>uncertainty <00:16:19.797>estimate?

336
00:16:20.295 --> 00:16:21.795
<00:16:20.295>And <00:16:20.969>finally, <00:16:21.645>um,

337
00:16:22.225 --> 00:16:24.395
<00:16:22.225>what <00:16:22.458>is <00:16:22.691>actually <00:16:22.925>a <00:16:23.158>good <00:16:23.391>uncertainty <00:16:23.625>estimate?

338
00:16:24.495 --> 00:16:26.155
<00:16:24.495>And <00:16:24.670>I'll <00:16:24.845>go <00:16:25.020>into <00:16:25.195>that <00:16:25.370>in <00:16:25.545>the <00:16:25.720>next <00:16:25.895>slide.

339
00:16:28.785 --> 00:16:30.645
<00:16:28.785>So, <00:16:29.135>um, <00:16:29.485>here <00:16:29.834>it's <00:16:30.185>important

340
00:16:30.705 --> 00:16:33.485
<00:16:30.705>to <00:16:30.860>understand <00:16:31.016>the <00:16:31.171>graph <00:16:31.327>on <00:16:31.482>the, <00:16:31.638>on <00:16:31.793>the <00:16:31.949>left <00:16:32.105>side.

341
00:16:33.865 --> 00:16:36.445
<00:16:33.865>So <00:16:34.020>what, <00:16:34.176>what <00:16:34.331>you <00:16:34.487>see <00:16:34.642>on <00:16:34.798>the <00:16:34.953>graph <00:16:35.109>is, <00:16:35.265>um,

342
00:16:36.635 --> 00:16:40.525
<00:16:36.635>that <00:16:36.790>on <00:16:36.946>the <00:16:37.101>x <00:16:37.257>axis, <00:16:37.412>the <00:16:37.568>uncertainty <00:16:37.723>is <00:16:37.879>plotted, <00:16:38.035>um,

343
00:16:40.625 --> 00:16:43.165
<00:16:40.625>as <00:16:40.800>estimated <00:16:40.975>by <00:16:41.150>the <00:16:41.325>neural <00:16:41.500>network, <00:16:41.675>by <00:16:41.850>the <00:16:42.025>method.

344
00:16:43.225 --> 00:16:44.925
<00:16:43.225>In <00:16:43.425>this <00:16:43.625>case, <00:16:43.825>it <00:16:44.025>would <00:16:44.225>be <00:16:44.425>drop <00:16:44.625>out.

345
00:16:45.705 --> 00:16:49.245
<00:16:45.705>Um, <00:16:45.832>and <00:16:45.959>then <00:16:46.086>you <00:16:46.214>have <00:16:46.341>the <00:16:46.468>gray <00:16:46.595>points <00:16:46.723>here <00:16:46.850>on <00:16:46.977>the <00:16:47.105>bottom

346
00:16:49.785 --> 00:16:51.405
<00:16:49.785>are <00:16:50.251>the <00:16:50.718>correct <00:16:51.185>samples,

347
00:16:51.745 --> 00:16:54.685
<00:16:51.745>and <00:16:51.900>the, <00:16:52.056>the <00:16:52.211>yellow <00:16:52.367>ones <00:16:52.522>are <00:16:52.678>the <00:16:52.833>other <00:16:52.989>class, <00:16:53.145>um,

348
00:16:54.685 --> 00:16:55.805
<00:16:54.685>that <00:16:54.936>we <00:16:55.189>want <00:16:55.440>to <00:16:55.693>distinguish.

349
00:16:55.825 --> 00:16:57.445
<00:16:55.825>And <00:16:56.025>as <00:16:56.225>you <00:16:56.425>can <00:16:56.625>see, <00:16:56.825>there's <00:16:57.025>an <00:16:57.225>overlap.

350
00:16:57.825 --> 00:17:02.445
<00:16:57.825>Um, <00:16:57.952>and <00:16:58.079>what <00:16:58.206>you <00:16:58.334>see <00:16:58.461>plotted <00:16:58.588>is <00:16:58.715>the, <00:16:58.843>um, <00:16:58.970>the <00:16:59.097>frequency, <00:16:59.225>um,

351
00:17:02.635 --> 00:17:05.685
<00:17:02.635>that <00:17:02.751>you <00:17:02.868>see <00:17:02.985>also <00:17:03.101>in <00:17:03.218>the <00:17:03.335>points, <00:17:03.451>um, <00:17:03.568>is <00:17:03.685>plot <00:17:03.801>here <00:17:03.918>for <00:17:04.035>the,

352
00:17:05.825 --> 00:17:10.365
<00:17:05.825>in, <00:17:05.941>in <00:17:06.058>the, <00:17:06.175>uh, <00:17:06.291>yeah, <00:17:06.408>in <00:17:06.525>the <00:17:06.641>solid <00:17:06.758>line <00:17:06.875>for <00:17:06.991>the <00:17:07.108>gray <00:17:07.225>samples

353
00:17:10.465 --> 00:17:13.765
<00:17:10.464>and <00:17:10.592>the <00:17:10.719>dotted <00:17:10.846>line <00:17:10.974>for <00:17:11.101>the, <00:17:11.228>uh, <00:17:11.355>for <00:17:11.483>the <00:17:11.610>yellow, <00:17:11.737>uh, <00:17:11.865>points.

354
00:17:14.545 --> 00:17:18.445
<00:17:14.545>Um, <00:17:14.685>and <00:17:14.825>I <00:17:14.965>have <00:17:15.105>to <00:17:15.245>put <00:17:15.385>some <00:17:15.525>threshold <00:17:15.665>in <00:17:15.805>to <00:17:15.945>say,

355
00:17:18.795 --> 00:17:21.445
<00:17:18.795>when <00:17:18.950>do <00:17:19.106>I <00:17:19.261>think <00:17:19.417>is <00:17:19.572>my <00:17:19.728>neural <00:17:19.883>network <00:17:20.039>is <00:17:20.195>wrong?

356
00:17:22.105 --> 00:17:25.525
<00:17:22.105>Um, <00:17:22.232>and <00:17:22.359>I <00:17:22.486>can <00:17:22.614>slide <00:17:22.741>the <00:17:22.868>screen <00:17:22.995>line, <00:17:23.123>uh, <00:17:23.250>from <00:17:23.377>left, <00:17:23.505>right,

357
00:17:25.585 --> 00:17:27.205
<00:17:25.585>and <00:17:26.051>select <00:17:26.518>some <00:17:26.985>threshold,

358
00:17:27.205 --> 00:17:29.285
<00:17:27.204>but <00:17:27.405>none <00:17:27.605>of <00:17:27.804>the <00:17:28.004>thresholds <00:17:28.204>will <00:17:28.405>be <00:17:28.605>optima

359
00:17:29.315 --> 00:17:30.965
<00:17:29.315>because <00:17:29.665>I <00:17:30.015>have <00:17:30.365>this <00:17:30.715>overlap.

360
00:17:31.105 --> 00:17:35.725
<00:17:31.105>And <00:17:31.232>so <00:17:31.359>the <00:17:31.486>question <00:17:31.614>is, <00:17:31.741>um, <00:17:31.868>uh, <00:17:31.995>yeah, <00:17:32.123>where <00:17:32.250>is <00:17:32.377>the <00:17:32.505>Optima,

361
00:17:35.785 --> 00:17:38.485
<00:17:35.785>um, <00:17:36.485>uh, <00:17:37.185>line,

362
00:17:38.825 --> 00:17:42.685
<00:17:38.825>but <00:17:38.952>in <00:17:39.079>fact <00:17:39.206>what, <00:17:39.334>what <00:17:39.461>data <00:17:39.588>scientists <00:17:39.715>do <00:17:39.843>is <00:17:39.970>you <00:17:40.097>don't <00:17:40.225>just

363
00:17:42.685 --> 00:17:43.685
<00:17:42.685>take <00:17:43.135>one <00:17:43.585>threshold,

364
00:17:43.685 --> 00:17:46.605
<00:17:43.685>but <00:17:43.860>to, <00:17:44.034>to <00:17:44.210>assess <00:17:44.385>how <00:17:44.560>good <00:17:44.735>a <00:17:44.909>method <00:17:45.085>is,

365
00:17:46.785 --> 00:17:50.925
<00:17:46.785>you <00:17:46.940>slide <00:17:47.096>this <00:17:47.251>line <00:17:47.407>from <00:17:47.562>left <00:17:47.718>to <00:17:47.873>right, <00:17:48.029>um, <00:17:48.185>uh,

366
00:17:51.065 --> 00:17:55.845
<00:17:51.065>and <00:17:51.192>plot, <00:17:51.319>um, <00:17:51.446>for <00:17:51.574>each <00:17:51.701>point <00:17:51.828>plot <00:17:51.955>how, <00:17:52.083>how <00:17:52.210>good <00:17:52.337>the, <00:17:52.465>the,

367
00:17:55.945 --> 00:17:58.845
<00:17:55.945>um, <00:17:56.120>network <00:17:56.294>is <00:17:56.470>in <00:17:56.645>terms <00:17:56.820>of <00:17:56.995>true <00:17:57.169>positive <00:17:57.345>rate

368
00:17:58.865 --> 00:18:00.125
<00:17:58.865>and <00:17:59.243>false <00:17:59.621>positive <00:17:59.999>rate.

369
00:18:00.385 --> 00:18:01.925
<00:18:00.385>And <00:18:00.539>this <00:18:00.693>is <00:18:00.847>what <00:18:01.001>you <00:18:01.155>see <00:18:01.309>on <00:18:01.463>the <00:18:01.617>right <00:18:01.771>side.

370
00:18:02.545 --> 00:18:06.125
<00:18:02.545>Um, <00:18:02.685>and <00:18:02.825>here <00:18:02.965>you <00:18:03.105>see <00:18:03.245>it, <00:18:03.385>um, <00:18:03.525>plotted <00:18:03.665>for <00:18:03.805>different <00:18:03.945>numbers

371
00:18:06.305 --> 00:18:07.405
<00:18:06.305>of <00:18:06.800>dropout <00:18:07.295>samples

372
00:18:07.675 --> 00:18:10.765
<00:18:07.675>that <00:18:07.830>we <00:18:07.986>draw <00:18:08.141>from <00:18:08.297>the <00:18:08.452>neural <00:18:08.608>network <00:18:08.763>going <00:18:08.919>from <00:18:09.075>two

373
00:18:11.345 --> 00:18:13.485
<00:18:11.345>to <00:18:11.811>256 <00:18:12.278>samples <00:18:12.745>drawn.

374
00:18:13.825 --> 00:18:17.445
<00:18:13.825>And <00:18:13.941>what <00:18:14.058>you <00:18:14.175>can <00:18:14.291>see <00:18:14.408>is <00:18:14.525>the <00:18:14.641>more <00:18:14.758>samples <00:18:14.875>I <00:18:14.991>draw <00:18:15.108>from <00:18:15.225>my

375
00:18:17.475 --> 00:18:20.805
<00:18:17.475>dropout <00:18:17.754>sample <00:18:18.034>distribution, <00:18:18.314>the <00:18:18.594>better <00:18:18.875>the,

376
00:18:20.945 --> 00:18:22.725
<00:18:20.945>the <00:18:21.294>estimate <00:18:21.645>gets <00:18:21.995>so <00:18:22.345>better.

377
00:18:22.785 --> 00:18:26.765
<00:18:22.785>In <00:18:22.912>this <00:18:23.039>case, <00:18:23.166>in <00:18:23.294>the, <00:18:23.421>um, <00:18:23.548>ROC <00:18:23.675>curve <00:18:23.803>is <00:18:23.930>always <00:18:24.057>if <00:18:24.185>the,

378
00:18:26.785 --> 00:18:30.525
<00:18:26.785>if <00:18:26.925>the <00:18:27.065>line <00:18:27.205>is <00:18:27.345>as <00:18:27.485>close <00:18:27.625>to, <00:18:27.765>to <00:18:27.905>the <00:18:28.045>horizontal, <00:18:28.185>uh,

379
00:18:30.525 --> 00:18:33.445
<00:18:30.525>to <00:18:30.680>the <00:18:30.836>vertical <00:18:30.991>line <00:18:31.147>here, <00:18:31.302>to <00:18:31.458>the <00:18:31.613>ver <00:18:31.769>vertical <00:18:31.925>axis,

380
00:18:33.545 --> 00:18:36.685
<00:18:33.545>and <00:18:33.720>then <00:18:33.895>goes <00:18:34.070>as <00:18:34.245>straight, <00:18:34.420>uh, <00:18:34.595>over, <00:18:34.770>um, <00:18:34.945>to,

381
00:18:36.705 --> 00:18:38.845
<00:18:36.705>to <00:18:36.905>the <00:18:37.105>bo <00:18:37.304>top <00:18:37.504>right <00:18:37.705>corner <00:18:37.905>as <00:18:38.105>possible.

382
00:18:38.945 --> 00:18:40.925
<00:18:38.945>And <00:18:39.100>so <00:18:39.256>you <00:18:39.411>can <00:18:39.567>see <00:18:39.722>from <00:18:39.878>the <00:18:40.033>blue <00:18:40.189>first <00:18:40.345>one,

383
00:18:40.925 --> 00:18:45.365
<00:18:40.925>which <00:18:41.125>two <00:18:41.325>dropout, <00:18:41.524>uh, <00:18:41.725>samples <00:18:41.925>it's <00:18:42.125>doing <00:18:42.325>okay,

384
00:18:45.725 --> 00:18:47.285
<00:18:45.725>I <00:18:45.925>would <00:18:46.125>say, <00:18:46.324>but <00:18:46.524>it's <00:18:46.725>doing <00:18:46.925>much <00:18:47.125>better,

385
00:18:47.285 --> 00:18:49.445
<00:18:47.285>better <00:18:47.565>if <00:18:47.845>I <00:18:48.125>do <00:18:48.405>256 <00:18:48.685>samples.

386
00:18:53.425 --> 00:18:57.325
<00:18:53.425>So <00:18:53.580>can <00:18:53.736>we <00:18:53.891>do <00:18:54.047>something <00:18:54.202>to <00:18:54.358>make <00:18:54.513>this <00:18:54.669>approach <00:18:54.825>faster?

387
00:18:57.985 --> 00:18:59.925
<00:18:57.985>Um, <00:18:58.264>yes. <00:18:58.544>So <00:18:58.824>there's <00:18:59.104>one <00:18:59.385>paper

388
00:18:59.945 --> 00:19:02.085
<00:18:59.945>and <00:19:00.145>I <00:19:00.345>go <00:19:00.544>just <00:19:00.745>very <00:19:00.945>quickly <00:19:01.145>into <00:19:01.345>this.

389
00:19:02.425 --> 00:19:05.325
<00:19:02.425>You <00:19:02.658>can <00:19:02.891>use <00:19:03.125>the <00:19:03.358>patient, <00:19:03.591>um, <00:19:03.825>uh,

390
00:19:05.325 --> 00:19:07.925
<00:19:05.325>neural <00:19:05.500>network <00:19:05.675>in <00:19:05.850>this <00:19:06.025>case <00:19:06.200>case <00:19:06.375>dropout, <00:19:06.550>for <00:19:06.725>example,

391
00:19:08.505 --> 00:19:10.205
<00:19:08.505>to <00:19:08.855>train <00:19:09.205>a <00:19:09.555>student <00:19:09.905>network.

392
00:19:10.305 --> 00:19:13.565
<00:19:10.305>And <00:19:10.460>this <00:19:10.616>is <00:19:10.771>then <00:19:10.927>kind <00:19:11.082>of <00:19:11.238>knowledge <00:19:11.393>distillation <00:19:11.549>to, <00:19:11.705>uh,

393
00:19:13.585 --> 00:19:17.605
<00:19:13.585>to <00:19:13.725>directly <00:19:13.865>predict <00:19:14.005>in <00:19:14.145>a <00:19:14.285>one <00:19:14.425>shot <00:19:14.565>way, <00:19:14.705>um, <00:19:14.845>the <00:19:14.985>uncertainty.

394
00:19:17.905 --> 00:19:20.845
<00:19:17.905>And <00:19:18.021>this, <00:19:18.138>as <00:19:18.254>you <00:19:18.371>can <00:19:18.488>see <00:19:18.605>in <00:19:18.721>this <00:19:18.838>plot, <00:19:18.955>uh, <00:19:19.071>on <00:19:19.188>the, <00:19:19.305>yeah,

395
00:19:21.025 --> 00:19:26.005
<00:19:21.025>top <00:19:21.152>left <00:19:21.279>plot <00:19:21.406>here, <00:19:21.534>um, <00:19:21.661>while <00:19:21.788>with <00:19:21.915>dropout, <00:19:22.043>uh, <00:19:22.170>my, <00:19:22.297>my, <00:19:22.425>um,

396
00:19:26.315 --> 00:19:29.805
<00:19:26.315>inference <00:19:26.548>time <00:19:26.781>increases <00:19:27.015>linearly <00:19:27.248>with <00:19:27.481>the <00:19:27.715>number

397
00:19:29.805 --> 00:19:33.045
<00:19:29.805>of <00:19:29.960>dropout <00:19:30.116>samples <00:19:30.271>like <00:19:30.427>I <00:19:30.582>sample, <00:19:30.738>um, <00:19:30.893>with <00:19:31.049>a <00:19:31.205>student

398
00:19:33.605 --> 00:19:36.525
<00:19:33.605>approach <00:19:33.732>where <00:19:33.859>I <00:19:33.986>do <00:19:34.114>a <00:19:34.241>one <00:19:34.368>shot <00:19:34.495>where <00:19:34.623>I <00:19:34.750>directly <00:19:34.877>learn <00:19:35.005>this.

399
00:19:36.865 --> 00:19:38.965
<00:19:36.865>Um, <00:19:37.065>at <00:19:37.265>inference <00:19:37.465>time, <00:19:37.665>I <00:19:37.865>have <00:19:38.065>constant <00:19:38.265>time.

400
00:19:39.825 --> 00:19:41.325
<00:19:39.825>And <00:19:39.993>as <00:19:40.162>you <00:19:40.331>can <00:19:40.500>see <00:19:40.668>in <00:19:40.837>the <00:19:41.006>other, <00:19:41.175>plus,

401
00:19:41.345 --> 00:19:44.285
<00:19:41.345>it <00:19:41.520>performs <00:19:41.695>really <00:19:41.870>well, <00:19:42.045>um, <00:19:42.220>very <00:19:42.395>similar <00:19:42.570>to <00:19:42.745>the,

402
00:19:44.305 --> 00:19:46.285
<00:19:44.305>to <00:19:44.538>the <00:19:44.771>teacher, <00:19:45.005>which <00:19:45.238>the <00:19:45.471>dropout <00:19:45.705>sample.

403
00:19:48.815 --> 00:19:52.225
<00:19:48.815>Okay. <00:19:48.970>I <00:19:49.126>hope <00:19:49.281>this <00:19:49.437>was <00:19:49.592>a <00:19:49.748>good <00:19:49.903>introduction <00:19:50.059>to <00:19:50.215>some

404
00:19:52.225 --> 00:19:55.185
<00:19:52.225>of <00:19:52.352>the <00:19:52.479>methods, <00:19:52.606>um, <00:19:52.734>uh, <00:19:52.861>that <00:19:52.988>you <00:19:53.115>can <00:19:53.243>use <00:19:53.370>to <00:19:53.497>have, <00:19:53.625>uh,

405
00:19:55.245 --> 00:19:57.665
<00:19:55.245>to <00:19:55.478>have <00:19:55.711>uncertainty <00:19:55.945>estimates <00:19:56.178>from <00:19:56.411>your <00:19:56.645>networks.

406
00:19:57.805 --> 00:20:00.705
<00:19:57.805>And <00:19:57.960>then <00:19:58.116>we <00:19:58.271>would <00:19:58.427>go <00:19:58.582>to <00:19:58.738>our <00:19:58.893>first <00:19:59.049>interactive <00:19:59.205>session.

407
00:20:02.905 --> 00:20:06.965
<00:20:02.905>Um, <00:20:03.105>and <00:20:03.305>I <00:20:03.504>will <00:20:03.705>start, <00:20:03.905>um, <00:20:04.105>the <00:20:04.305>next,

408
00:20:07.305 --> 00:20:08.525
<00:20:07.305>um, <00:20:08.403>question.

409
00:20:08.825 --> 00:20:10.005
<00:20:08.825>Um, <00:20:09.887>uh,

410
00:20:10.065 --> 00:20:14.165
<00:20:10.065>and <00:20:10.240>the <00:20:10.415>question <00:20:10.590>is, <00:20:10.765>where <00:20:10.940>could <00:20:11.115>neural <00:20:11.290>network <00:20:11.465>uncertainty

411
00:20:14.185 --> 00:20:16.205
<00:20:14.185>be <00:20:14.534>useful <00:20:14.885>in <00:20:15.235>automotive <00:20:15.585>applications?

412
00:20:16.225 --> 00:20:19.645
<00:20:16.225>And <00:20:16.352>I, <00:20:16.479>I <00:20:16.606>think <00:20:16.734>I <00:20:16.861>would <00:20:16.988>just <00:20:17.115>take <00:20:17.243>maybe <00:20:17.370>two <00:20:17.497>minutes, <00:20:17.625>um,

413
00:20:19.705 --> 00:20:23.965
<00:20:19.705>or <00:20:19.845>three <00:20:19.985>minutes <00:20:20.125>and <00:20:20.264>everyone, <00:20:20.405>uh, <00:20:20.544>put <00:20:20.685>in <00:20:20.824>some, <00:20:20.965>some <00:20:21.105>words,

414
00:20:24.145 --> 00:20:25.405
<00:20:24.145>uh, <00:20:24.523>some, <00:20:24.901>some <00:20:25.279>applications,

415
00:20:25.585 --> 00:20:27.285
<00:20:25.585>and, <00:20:25.818>uh, <00:20:26.051>just <00:20:26.285>be <00:20:26.518>interested <00:20:26.751>to <00:20:26.985>see

416
00:20:27.285 --> 00:20:30.645
<00:20:27.285>where <00:20:27.485>you <00:20:27.685>think <00:20:27.885>it <00:20:28.085>could <00:20:28.285>be <00:20:28.485>used <00:20:28.685>already.

417
00:20:30.665 --> 00:20:31.925
<00:20:30.665>See <00:20:31.043>security <00:20:31.421>coming <00:20:31.799>up.

418
00:21:57.245 --> 00:22:02.175
<00:21:57.245>Okay. <00:21:57.372>I <00:21:57.499>think <00:21:57.626>it's <00:21:57.754>not, <00:21:57.881>not <00:21:58.008>too <00:21:58.135>many <00:21:58.263>new <00:21:58.390>votes <00:21:58.517>coming <00:21:58.645>in.

419
00:22:02.445 --> 00:22:04.095
<00:22:02.445>Okay, <00:22:02.725>some <00:22:03.004>more <00:22:03.284>votes <00:22:03.564>coming <00:22:03.845>in.

420
00:22:27.825 --> 00:22:30.795
<00:22:27.825>Okay. <00:22:27.965>Some <00:22:28.105>really <00:22:28.245>good <00:22:28.385>points <00:22:28.525>here. <00:22:28.665>I, <00:22:28.805>I <00:22:28.945>have <00:22:29.085>to <00:22:29.225>say.

421
00:22:30.815 --> 00:22:33.395
<00:22:30.815>So <00:22:30.990>this <00:22:31.165>is <00:22:31.340>really <00:22:31.515>good. <00:22:31.690>Um, <00:22:31.865>so <00:22:32.040>I <00:22:32.215>think

422
00:22:34.545 --> 00:22:38.715
<00:22:34.545>what <00:22:34.661>we <00:22:34.778>see <00:22:34.895>already <00:22:35.011>is <00:22:35.128>that <00:22:35.245>a <00:22:35.361>lot <00:22:35.478>of <00:22:35.595>you <00:22:35.711>put <00:22:35.828>in, <00:22:35.945>um,

423
00:22:40.055 --> 00:22:42.395
<00:22:40.055>tar <00:22:40.255>or <00:22:40.455>use <00:22:40.655>cases <00:22:40.855>that <00:22:41.055>are <00:22:41.255>related <00:22:41.455>to

424
00:22:43.705 --> 00:22:45.755
<00:22:43.705>auto <00:22:44.171>automatic <00:22:44.638>driving <00:22:45.105>assistance.

425
00:22:45.815 --> 00:22:50.395
<00:22:45.815>So <00:22:46.015>for <00:22:46.215>example, <00:22:46.415>um, <00:22:46.615>sign <00:22:46.815>detection, <00:22:47.015>road <00:22:47.215>safety,

426
00:22:50.815 --> 00:22:55.715
<00:22:50.815>we <00:22:50.990>see, <00:22:51.165>um, <00:22:51.340>uh, <00:22:51.515>we <00:22:51.690>see, <00:22:51.865>what <00:22:52.040>was <00:22:52.215>it,

427
00:22:55.995 --> 00:22:58.795
<00:22:55.995>a <00:22:56.195>braking <00:22:56.395>mechanism, <00:22:56.594>um, <00:22:56.794>those <00:22:56.995>kind <00:22:57.195>of <00:22:57.395>things.

428
00:22:58.795 --> 00:23:01.275
<00:22:58.795>Vision, <00:22:59.028>um, <00:22:59.261>all <00:22:59.495>very <00:22:59.728>good <00:22:59.961>points <00:23:00.195>related

429
00:23:01.295 --> 00:23:02.315
<00:23:01.295>to <00:23:01.754>autonomous <00:23:02.213>driving.

430
00:23:03.175 --> 00:23:06.675
<00:23:03.175>Um, <00:23:03.350>and <00:23:03.524>we <00:23:03.700>see <00:23:03.875>also <00:23:04.050>some <00:23:04.225>interesting <00:23:04.399>other, <00:23:04.575>uh,

431
00:23:06.775 --> 00:23:07.835
<00:23:06.775>inputs <00:23:07.252>like <00:23:07.729>medicine.

432
00:23:08.025 --> 00:23:10.835
<00:23:08.025>This, <00:23:08.152>I <00:23:08.279>think <00:23:08.406>could <00:23:08.534>be <00:23:08.661>go, <00:23:08.788>could <00:23:08.915>go <00:23:09.043>into <00:23:09.170>the <00:23:09.297>direction <00:23:09.425>of,

433
00:23:11.015 --> 00:23:13.195
<00:23:11.015>uh, <00:23:11.295>driver <00:23:11.575>health <00:23:11.855>monitoring, <00:23:12.135>for <00:23:12.415>example.

434
00:23:13.215 --> 00:23:17.275
<00:23:13.215>That's <00:23:13.415>a <00:23:13.615>very <00:23:13.814>good <00:23:14.014>point. <00:23:14.215>Um, <00:23:14.415>alerting <00:23:14.615>users.

435
00:23:17.545 --> 00:23:19.155
<00:23:17.545>This <00:23:17.778>could <00:23:18.011>also <00:23:18.245>be, <00:23:18.478>uh, <00:23:18.711>very <00:23:18.945>good.

436
00:23:19.235 --> 00:23:21.955
<00:23:19.235>I <00:23:19.409>think <00:23:19.584>it's <00:23:19.760>also <00:23:19.935>related <00:23:20.110>probably <00:23:20.284>to <00:23:20.459>autonomous <00:23:20.635>driving.

437
00:23:22.895 --> 00:23:25.395
<00:23:22.895>Um, <00:23:23.128>security <00:23:23.361>is <00:23:23.595>actually <00:23:23.828>a <00:23:24.061>good <00:23:24.295>point.

438
00:23:25.435 --> 00:23:27.435
<00:23:25.435>I <00:23:25.590>don't <00:23:25.746>know <00:23:25.901>if, <00:23:26.057>if <00:23:26.212>you <00:23:26.368>meant <00:23:26.523>security <00:23:26.679>or <00:23:26.835>safety,

439
00:23:27.695 --> 00:23:30.155
<00:23:27.695>but <00:23:27.975>if, <00:23:28.254>if <00:23:28.534>security <00:23:28.814>could <00:23:29.095>also

440
00:23:30.595 --> 00:23:31.675
<00:23:30.595>actually <00:23:30.789>be <00:23:30.983>a <00:23:31.178>very <00:23:31.372>good <00:23:31.567>point.

441
00:23:31.815 --> 00:23:33.515
<00:23:31.815>Um, <00:23:32.165>for <00:23:32.515>example, <00:23:32.865>network <00:23:33.215>security,

442
00:23:33.625 --> 00:23:35.395
<00:23:33.625>intrusion <00:23:33.905>detection, <00:23:34.185>this <00:23:34.465>kind <00:23:34.745>of <00:23:35.025>thing.

443
00:23:36.015 --> 00:23:37.275
<00:23:36.015>So <00:23:36.298>yeah, <00:23:36.582>very <00:23:36.865>good <00:23:37.149>points.

444
00:23:38.415 --> 00:23:41.725
<00:23:38.415>Um, <00:23:38.881>then, <00:23:39.348>um, <00:23:39.815>so,

445
00:23:41.945 --> 00:23:44.285
<00:23:41.945>but <00:23:42.085>I <00:23:42.225>think <00:23:42.365>we <00:23:42.504>can <00:23:42.645>agree <00:23:42.784>that <00:23:42.925>for, <00:23:43.064>for <00:23:43.205>autonomous <00:23:43.345>driving,

446
00:23:44.315 --> 00:23:47.285
<00:23:44.315>it's, <00:23:44.548>it's, <00:23:44.781>um, <00:23:45.015>um, <00:23:45.248>uh, <00:23:45.481>very <00:23:45.715>important.

447
00:23:47.425 --> 00:23:50.285
<00:23:47.425>So <00:23:47.580>I <00:23:47.736>think <00:23:47.891>I <00:23:48.047>will <00:23:48.202>close <00:23:48.358>the <00:23:48.513>po <00:23:48.669>uh, <00:23:48.825>poll

448
00:23:50.665 --> 00:23:52.645
<00:23:50.665>and, <00:23:52.065>um,

449
00:23:55.635 --> 00:23:59.255
<00:23:55.635>go <00:23:55.790>to <00:23:55.946>the <00:23:56.101>next <00:23:56.257>one, <00:23:56.412>um, <00:23:56.568>and <00:23:56.723>ask <00:23:56.879>the <00:23:57.035>question.

450
00:23:59.535 --> 00:24:02.895
<00:23:59.535>A <00:23:59.675>quick <00:23:59.815>poll. <00:23:59.955>Um, <00:24:00.095>do <00:24:00.235>you <00:24:00.375>think <00:24:00.515>that <00:24:00.655>uncertainty <00:24:00.795>is <00:24:00.935>required

451
00:24:03.235 --> 00:24:05.415
<00:24:03.235>to <00:24:03.514>implement <00:24:03.794>a <00:24:04.074>driver <00:24:04.354>assistance <00:24:04.635>system?

452
00:24:38.925 --> 00:24:43.815
<00:24:38.925>Okay, <00:24:39.100>I <00:24:39.274>think <00:24:39.450>a <00:24:39.625>lot, <00:24:39.800>uh, <00:24:39.975>majority <00:24:40.149>say <00:24:40.325>yes,

453
00:24:44.645 --> 00:24:46.535
<00:24:44.645>it's <00:24:44.878>needed, <00:24:45.111>but <00:24:45.345>also <00:24:45.578>some <00:24:45.811>people <00:24:46.045>say

454
00:24:46.785 --> 00:24:48.615
<00:24:46.785>maybe <00:24:47.135>it's <00:24:47.485>not <00:24:47.835>required. <00:24:48.185>Um,

455
00:24:49.005 --> 00:24:50.975
<00:24:49.005>Yeah, <00:24:49.355>maybe <00:24:49.705>it's <00:24:50.055>uh, <00:24:50.405>right,

456
00:24:50.975 --> 00:24:52.615
<00:24:50.975>because <00:24:51.254>with <00:24:51.534>the <00:24:51.814>driver <00:24:52.094>assistance <00:24:52.375>system,

457
00:24:52.635 --> 00:24:54.055
<00:24:52.635>the <00:24:52.817>driver <00:24:53.000>is <00:24:53.182>still <00:24:53.365>in <00:24:53.547>the <00:24:53.730>loop, <00:24:53.913>right?

458
00:24:54.195 --> 00:24:58.935
<00:24:54.195>So, <00:24:54.335>um, <00:24:54.475>I <00:24:54.615>can <00:24:54.754>imagine <00:24:54.895>that <00:24:55.034>you <00:24:55.175>can <00:24:55.314>build <00:24:55.455>systems <00:24:55.595>without,

459
00:24:59.035 --> 00:25:02.295
<00:24:59.035>yes, <00:24:59.235>it's <00:24:59.435>not, <00:24:59.635>not, <00:24:59.835>not <00:25:00.035>autonomous <00:25:00.235>driving, <00:25:00.435>right?

460
00:25:03.715 --> 00:25:06.095
<00:25:03.715>Yes, <00:25:04.064>yes. <00:25:04.415>Right. <00:25:04.764>Um, <00:25:05.115>yeah,

461
00:25:06.095 --> 00:25:08.255
<00:25:06.095>and <00:25:06.270>also <00:25:06.445>in <00:25:06.620>the <00:25:06.795>point <00:25:06.970>in <00:25:07.145>the <00:25:07.320>word <00:25:07.495>cloud

462
00:25:08.255 --> 00:25:10.375
<00:25:08.255>before <00:25:08.455>you <00:25:08.655>said <00:25:08.855>alerting <00:25:09.055>the <00:25:09.255>user <00:25:09.455>when <00:25:09.655>the

463
00:25:10.375 --> 00:25:11.455
<00:25:10.375>system <00:25:10.618>is, <00:25:10.860>is <00:25:11.104>not <00:25:11.347>right.

464
00:25:11.475 --> 00:25:13.735
<00:25:11.475>So, <00:25:11.615>but <00:25:11.754>this <00:25:11.895>would, <00:25:12.034>would <00:25:12.175>go <00:25:12.314>into <00:25:12.455>the <00:25:12.594>direction <00:25:12.735>of <00:25:12.875>saying,

465
00:25:13.735 --> 00:25:15.575
<00:25:13.735>yes, <00:25:14.014>maybe <00:25:14.294>we <00:25:14.574>want <00:25:14.854>uncertainty <00:25:15.135>and,

466
00:25:15.595 --> 00:25:17.135
<00:25:15.595>and <00:25:15.872>a <00:25:16.149>driver <00:25:16.426>assistance <00:25:16.703>system <00:25:16.981>really.

467
00:25:17.795 --> 00:25:19.695
<00:25:17.795>So <00:25:18.145>that's <00:25:18.495>very <00:25:18.845>interesting, <00:25:19.195>uh,

468
00:25:20.005 --> 00:25:21.895
<00:25:20.005>because <00:25:20.205>this <00:25:20.405>is <00:25:20.605>kind <00:25:20.805>of <00:25:21.005>a <00:25:21.205>trick <00:25:21.405>question

469
00:25:21.895 --> 00:25:23.895
<00:25:21.895>because <00:25:22.128>I'll <00:25:22.361>move <00:25:22.595>to <00:25:22.828>the <00:25:23.061>next <00:25:23.295>question.

470
00:25:24.315 --> 00:25:27.255
<00:25:24.315>And <00:25:24.515>this <00:25:24.715>one <00:25:24.915>is, <00:25:25.115>do <00:25:25.315>we <00:25:25.515>need, <00:25:25.715>uh,

471
00:25:27.255 --> 00:25:29.215
<00:25:27.255>uncertainty <00:25:27.535>estimates <00:25:27.815>in <00:25:28.095>neural <00:25:28.375>networks <00:25:28.655>for,

472
00:25:29.315 --> 00:25:30.535
<00:25:29.315>for <00:25:29.864>autonomous <00:25:30.413>driving?

473
00:25:30.635 --> 00:25:32.415
<00:25:30.635>So <00:25:30.915>going <00:25:31.195>from <00:25:31.475>driver <00:25:31.754>assistance <00:25:32.035>system

474
00:25:32.835 --> 00:25:34.015
<00:25:32.835>to <00:25:33.366>autonomous <00:25:33.897>driving,

475
00:26:01.235 --> 00:26:02.795
<00:26:01.235>I <00:26:01.435>guess <00:26:01.635>the <00:26:01.834>result <00:26:02.034>is <00:26:02.235>quite <00:26:02.435>clear <00:26:02.635>too

476
00:26:02.795 --> 00:26:04.595
<00:26:02.795>Much. <00:26:04.195>Hmm,

477
00:26:05.355 --> 00:26:07.635
<00:26:05.355>I <00:26:05.555>guess <00:26:05.755>the <00:26:05.955>result <00:26:06.155>is <00:26:06.355>quite, <00:26:06.555>quite <00:26:06.755>definite.

478
00:26:07.865 --> 00:26:09.275
<00:26:07.865>Yeah, <00:26:08.182>yeah, <00:26:08.499>it's <00:26:08.816>quite <00:26:09.134>definite.

479
00:26:09.275 --> 00:26:11.915
<00:26:09.275>So <00:26:09.450>yes, <00:26:09.625>probably <00:26:09.800>we <00:26:09.975>need, <00:26:10.150>uh, <00:26:10.325>uncertainty <00:26:10.500>for <00:26:10.675>that.

480
00:26:12.425 --> 00:26:16.915
<00:26:12.425>Okay, <00:26:12.580>this <00:26:12.736>is <00:26:12.891>great. <00:26:13.047>Um, <00:26:13.202>then <00:26:13.358>let's <00:26:13.513>my <00:26:13.669>mouse, <00:26:13.825>uh,

481
00:26:17.315 --> 00:26:19.675
<00:26:17.315>I <00:26:17.490>close <00:26:17.665>it <00:26:17.840>on <00:26:18.015>my <00:26:18.190>screen <00:26:18.365>and <00:26:18.540>I'll <00:26:18.715>continue.

482
00:26:20.415 --> 00:26:25.315
<00:26:20.415>So <00:26:20.542>yeah, <00:26:20.669>the <00:26:20.796>question <00:26:20.924>is <00:26:21.051>just <00:26:21.178>as <00:26:21.305>a <00:26:21.433>little, <00:26:21.560>um, <00:26:21.687>going <00:26:21.815>into

483
00:26:26.115 --> 00:26:27.355
<00:26:26.115>a <00:26:26.274>lot <00:26:26.433>of <00:26:26.593>the <00:26:26.752>points <00:26:26.912>that <00:26:27.071>you <00:26:27.231>already

484
00:26:27.355 --> 00:26:28.435
<00:26:27.355>mentioned <00:26:27.598>in <00:26:27.841>the <00:26:28.084>word <00:26:28.327>cloud.

485
00:26:28.495 --> 00:26:30.235
<00:26:28.495>So <00:26:28.728>the <00:26:28.961>question <00:26:29.195>is <00:26:29.428>with, <00:26:29.661>with, <00:26:29.895>um,

486
00:26:30.255 --> 00:26:33.115
<00:26:30.255>driver <00:26:30.410>assistance <00:26:30.566>systems, <00:26:30.721>at <00:26:30.877>least <00:26:31.032>as <00:26:31.188>when <00:26:31.343>do <00:26:31.499>we <00:26:31.655>detect,

487
00:26:33.335 --> 00:26:34.795
<00:26:33.335>um, <00:26:34.649>insufficiencies?

488
00:26:35.375 --> 00:26:39.115
<00:26:35.375>Um, <00:26:35.530>and <00:26:35.686>here <00:26:35.841>you <00:26:35.997>just <00:26:36.152>see, <00:26:36.308>um, <00:26:36.463>well, <00:26:36.619>yeah, <00:26:36.775>like

489
00:26:39.135 --> 00:26:42.515
<00:26:39.135>how <00:26:39.262>early <00:26:39.389>can <00:26:39.516>I <00:26:39.644>say <00:26:39.771>that <00:26:39.898>my <00:26:40.025>system <00:26:40.153>is <00:26:40.280>certain <00:26:40.407>or <00:26:40.535>uncertain,

490
00:26:42.615 --> 00:26:44.845
<00:26:42.615>and <00:26:42.815>how <00:26:43.015>does <00:26:43.215>it <00:26:43.415>influence <00:26:43.615>my <00:26:43.815>decision <00:26:44.015>making?

491
00:26:45.745 --> 00:26:49.925
<00:26:45.745>Um, <00:26:45.919>and, <00:26:46.094>uh, <00:26:46.270>yeah, <00:26:46.445>this, <00:26:46.620>just <00:26:46.794>showing <00:26:46.969>that, <00:26:47.145>yeah,

492
00:26:49.945 --> 00:26:52.445
<00:26:49.945>for <00:26:50.100>example, <00:26:50.256>if <00:26:50.411>I <00:26:50.567>have <00:26:50.722>a <00:26:50.878>bicycle <00:26:51.033>here, <00:26:51.189>how <00:26:51.345>early,

493
00:26:52.545 --> 00:26:53.845
<00:26:52.545>how <00:26:52.740>close <00:26:52.935>do <00:26:53.130>I <00:26:53.325>have <00:26:53.520>to <00:26:53.715>be?

494
00:26:53.845 --> 00:26:56.125
<00:26:53.845>Or <00:26:53.961>how <00:26:54.078>long <00:26:54.195>do <00:26:54.311>I <00:26:54.428>have <00:26:54.545>to <00:26:54.661>see <00:26:54.778>the <00:26:54.895>bicycle <00:26:55.011>to, <00:26:55.128>to <00:26:55.245>save

495
00:26:56.125 --> 00:26:57.445
<00:26:56.125>with <00:26:56.323>certainty <00:26:56.521>that <00:26:56.719>I <00:26:56.917>see <00:26:57.115>a <00:26:57.312>bicycle?

496
00:26:58.425 --> 00:27:02.085
<00:26:58.425>Um, <00:26:58.565>and <00:26:58.705>this <00:26:58.845>goes <00:26:58.985>a <00:26:59.125>little <00:26:59.264>bit <00:26:59.405>into, <00:26:59.544>into <00:26:59.685>system <00:26:59.825>design.

497
00:27:02.665 --> 00:27:05.605
<00:27:02.665>Um, <00:27:02.792>so <00:27:02.919>for <00:27:03.046>example, <00:27:03.174>if <00:27:03.301>I <00:27:03.428>say <00:27:03.555>I <00:27:03.683>have <00:27:03.810>a <00:27:03.937>redundant <00:27:04.065>system

498
00:27:05.605 --> 00:27:08.045
<00:27:05.605>with <00:27:05.955>multiple <00:27:06.305>perception <00:27:06.655>modules, <00:27:07.005>um,

499
00:27:10.425 --> 00:27:14.765
<00:27:10.425>um, <00:27:10.580>then, <00:27:10.736>uh, <00:27:10.891>if <00:27:11.047>I, <00:27:11.202>if <00:27:11.358>my <00:27:11.513>perception <00:27:11.669>modules <00:27:11.825>disagree,

500
00:27:14.765 --> 00:27:15.965
<00:27:14.765>which <00:27:15.035>module <00:27:15.305>do <00:27:15.575>I <00:27:15.845>trust?

501
00:27:16.105 --> 00:27:19.285
<00:27:16.105>Do <00:27:16.221>I <00:27:16.338>say <00:27:16.455>I <00:27:16.571>trust, <00:27:16.688>I <00:27:16.805>have <00:27:16.921>one <00:27:17.038>primary <00:27:17.155>module <00:27:17.271>that <00:27:17.388>I <00:27:17.505>trust?

502
00:27:19.345 --> 00:27:21.845
<00:27:19.345>Do <00:27:19.500>I <00:27:19.656>say <00:27:19.811>I <00:27:19.967>wrote, <00:27:20.122>do <00:27:20.278>I <00:27:20.433>take <00:27:20.589>the <00:27:20.745>uncertainty?

503
00:27:22.185 --> 00:27:24.805
<00:27:22.185>So <00:27:22.340>it <00:27:22.496>might <00:27:22.651>be <00:27:22.807>very <00:27:22.962>helpful, <00:27:23.118>uh, <00:27:23.273>in <00:27:23.429>that <00:27:23.585>case.

504
00:27:26.405 --> 00:27:29.425
<00:27:26.405>And <00:27:26.532>I <00:27:26.659>think <00:27:26.786>I'll <00:27:26.914>jump <00:27:27.041>with <00:27:27.168>regards <00:27:27.295>to <00:27:27.423>looking <00:27:27.550>at <00:27:27.677>the <00:27:27.805>time.

505
00:27:29.455 --> 00:27:30.705
<00:27:29.455>I'll <00:27:29.736>jump <00:27:30.017>over <00:27:30.298>it. <00:27:30.580>Um,

506
00:27:30.965 --> 00:27:34.065
<00:27:30.965>and, <00:27:31.139>uh, <00:27:31.314>say <00:27:31.490>we, <00:27:31.665>I <00:27:31.840>hand <00:27:32.014>over <00:27:32.189>to <00:27:32.365>Nico

507
00:27:34.245 --> 00:27:36.545
<00:27:34.245>for <00:27:34.445>our <00:27:34.645>first <00:27:34.844>session <00:27:35.044>on, <00:27:35.245>uh, <00:27:35.445>one, <00:27:35.645>um,

508
00:27:36.945 --> 00:27:39.745
<00:27:36.945>interesting <00:27:37.145>application, <00:27:37.345>which <00:27:37.544>I <00:27:37.745>think <00:27:37.945>wasn't <00:27:38.145>mentioned, <00:27:38.345>um,

509
00:27:40.325 --> 00:27:42.945
<00:27:40.325>uh, <00:27:40.500>in <00:27:40.675>the <00:27:40.850>word <00:27:41.025>cloud, <00:27:41.200>which <00:27:41.375>I <00:27:41.550>expected <00:27:41.725>actually,

510
00:27:43.005 --> 00:27:44.305
<00:27:43.005>so <00:27:43.297>will <00:27:43.590>be <00:27:43.882>very <00:27:44.175>interesting.

511
00:27:47.945 --> 00:27:50.755
<00:27:47.945>Alright, <00:27:48.225>thanks <00:27:48.504>Tim. <00:27:48.784>Um, <00:27:49.064>yeah, <00:27:49.345>okay.

512
00:27:50.755 --> 00:27:53.395
<00:27:50.755>So <00:27:50.910>let's, <00:27:51.066>um, <00:27:51.221>have <00:27:51.377>a <00:27:51.532>look <00:27:51.688>at <00:27:51.843>the <00:27:51.999>use <00:27:52.155>case

513
00:27:53.525 --> 00:27:56.115
<00:27:53.525>where <00:27:53.805>we <00:27:54.085>use <00:27:54.365>uncertainty <00:27:54.645>for, <00:27:54.925>um,

514
00:27:56.415 --> 00:27:58.235
<00:27:56.415>for <00:27:56.764>corner <00:27:57.115>case <00:27:57.465>detection <00:27:57.815>framework.

515
00:27:58.775 --> 00:28:01.035
<00:27:58.775>Um, <00:27:59.475>lemme <00:28:00.175>check.

516
00:28:01.875 --> 00:28:04.455
<00:28:01.875>So <00:28:02.002>basically <00:28:02.129>we <00:28:02.256>use <00:28:02.384>this <00:28:02.511>in <00:28:02.638>the <00:28:02.765>context <00:28:02.893>of <00:28:03.020>the <00:28:03.147>big <00:28:03.275>loop.

517
00:28:04.555 --> 00:28:07.855
<00:28:04.555>Uh, <00:28:04.671>what <00:28:04.788>we <00:28:04.905>mean <00:28:05.021>by <00:28:05.138>this <00:28:05.255>is, <00:28:05.371>um, <00:28:05.488>we <00:28:05.605>want <00:28:05.721>to <00:28:05.838>be <00:28:05.955>able

518
00:28:07.875 --> 00:28:11.375
<00:28:07.875>to <00:28:08.015>record <00:28:08.155>from <00:28:08.295>the, <00:28:08.435>from <00:28:08.575>the <00:28:08.715>cars, <00:28:08.855>uh, <00:28:08.995>the <00:28:09.135>sensor <00:28:09.275>data

519
00:28:11.795 --> 00:28:13.695
<00:28:11.795>and <00:28:11.970>upload <00:28:12.145>it <00:28:12.320>to <00:28:12.495>some <00:28:12.670>kind <00:28:12.845>of <00:28:13.020>data <00:28:13.195>lake

520
00:28:14.195 --> 00:28:18.775
<00:28:14.195>and, <00:28:14.350>um, <00:28:14.506>use <00:28:14.661>this <00:28:14.817>data <00:28:14.972>for <00:28:15.128>yeah, <00:28:15.283>doing <00:28:15.439>software <00:28:15.595>development,

521
00:28:18.885 --> 00:28:21.855
<00:28:18.885>machine <00:28:19.118>learning, <00:28:19.351>model <00:28:19.585>training <00:28:19.818>and <00:28:20.051>other <00:28:20.285>software,

522
00:28:21.855 --> 00:28:25.535
<00:28:21.855>and <00:28:22.010>deploy <00:28:22.166>this, <00:28:22.321>uh, <00:28:22.477>software <00:28:22.632>over <00:28:22.788>the <00:28:22.943>air, <00:28:23.099>um, <00:28:23.255>back

523
00:28:25.535 --> 00:28:29.015
<00:28:25.535>to <00:28:25.690>the <00:28:25.846>cars, <00:28:26.001>uh, <00:28:26.157>and <00:28:26.312>update <00:28:26.468>the <00:28:26.623>existing <00:28:26.779>software <00:28:26.935>there.

524
00:28:29.715 --> 00:28:34.455
<00:28:29.715>And, <00:28:29.842>um, <00:28:29.969>so <00:28:30.096>with, <00:28:30.224>with <00:28:30.351>this, <00:28:30.478>um, <00:28:30.605>send <00:28:30.733>us, <00:28:30.860>we <00:28:30.987>have <00:28:31.115>currently,

525
00:28:34.915 --> 00:28:37.895
<00:28:34.915>uh, <00:28:35.195>we <00:28:35.475>can <00:28:35.754>really, <00:28:36.034>um, <00:28:36.315>record

526
00:28:38.075 --> 00:28:41.895
<00:28:38.075>or <00:28:38.215>generate <00:28:38.355>a <00:28:38.495>huge <00:28:38.635>bunch <00:28:38.775>of, <00:28:38.915>um, <00:28:39.055>data, <00:28:39.195>a <00:28:39.335>huge <00:28:39.475>amount

527
00:28:41.895 --> 00:28:45.455
<00:28:41.895>of <00:28:42.070>data, <00:28:42.245>and <00:28:42.420>it's <00:28:42.595>maybe <00:28:42.770>not <00:28:42.945>feasible <00:28:43.120>to, <00:28:43.295>uh,

528
00:28:45.515 --> 00:28:49.655
<00:28:45.515>or <00:28:45.622>not, <00:28:45.730>not <00:28:45.838>a <00:28:45.945>good <00:28:46.053>idea <00:28:46.161>to <00:28:46.268>upload <00:28:46.376>all <00:28:46.484>of <00:28:46.591>this <00:28:46.699>data <00:28:46.807>into <00:28:46.915>our,

529
00:28:49.755 --> 00:28:53.095
<00:28:49.755>uh, <00:28:49.910>um, <00:28:50.066>the <00:28:50.221>data <00:28:50.377>lake, <00:28:50.532>uh, <00:28:50.688>wise, <00:28:50.843>that, <00:28:50.999>so <00:28:51.155>the,

530
00:28:53.155 --> 00:28:54.415
<00:28:53.155>the <00:28:53.533>reason <00:28:53.911>is, <00:28:54.289>uh,

531
00:28:54.415 --> 00:28:57.375
<00:28:54.415>that <00:28:54.648>our <00:28:54.881>data <00:28:55.115>is <00:28:55.348>not <00:28:55.581>uniformly <00:28:55.815>distributed.

532
00:28:57.475 --> 00:29:00.695
<00:28:57.475>If <00:28:57.591>you <00:28:57.708>look <00:28:57.824>at, <00:28:57.941>uh, <00:28:58.058>the <00:28:58.175>type <00:28:58.291>of <00:28:58.408>scenarios <00:28:58.524>we, <00:28:58.641>we <00:28:58.758>have, <00:28:58.875>um,

533
00:29:01.035 --> 00:29:05.175
<00:29:01.035>so <00:29:01.175>there <00:29:01.315>are, <00:29:01.455>there <00:29:01.595>are <00:29:01.735>very <00:29:01.875>many <00:29:02.015>common <00:29:02.155>situations <00:29:02.295>that <00:29:02.435>we,

534
00:29:05.355 --> 00:29:07.055
<00:29:05.355>uh, <00:29:05.821>encounter <00:29:06.288>quite <00:29:06.755>frequently.

535
00:29:07.555 --> 00:29:09.135
<00:29:07.555>Uh, <00:29:07.905>so <00:29:08.255>normal <00:29:08.605>driving, <00:29:08.955>uh,

536
00:29:09.135 --> 00:29:12.455
<00:29:09.135>along <00:29:09.275>the <00:29:09.415>road <00:29:09.555>on <00:29:09.695>the <00:29:09.835>highway <00:29:09.975>may, <00:29:10.115>maybe <00:29:10.254>it's <00:29:10.395>rather <00:29:10.535>boring

537
00:29:12.595 --> 00:29:15.295
<00:29:12.595>for <00:29:12.795>our <00:29:12.995>network <00:29:13.195>because <00:29:13.395>it's, <00:29:13.595>um, <00:29:13.795>maybe <00:29:13.995>the,

538
00:29:15.355 --> 00:29:17.655
<00:29:15.355>the <00:29:15.588>weather <00:29:15.821>is <00:29:16.055>kind <00:29:16.288>of, <00:29:16.521>um, <00:29:16.755>nice

539
00:29:17.755 --> 00:29:20.535
<00:29:17.755>and <00:29:17.930>we, <00:29:18.105>we <00:29:18.280>can <00:29:18.455>easily <00:29:18.630>detect <00:29:18.805>everything <00:29:18.980>around <00:29:19.155>us.

540
00:29:21.035 --> 00:29:22.895
<00:29:21.035>Um, <00:29:21.315>but <00:29:21.595>there <00:29:21.875>are <00:29:22.155>certain <00:29:22.435>situations

541
00:29:22.895 --> 00:29:27.095
<00:29:22.895>that <00:29:23.050>are <00:29:23.206>more <00:29:23.361>difficult, <00:29:23.517>uh, <00:29:23.672>more <00:29:23.828>safety <00:29:23.983>critical <00:29:24.139>also, <00:29:24.295>uh,

542
00:29:27.095 --> 00:29:29.775
<00:29:27.095>and <00:29:27.328>is <00:29:27.561>exactly <00:29:27.795>those <00:29:28.028>situations <00:29:28.261>are, <00:29:28.495>um,

543
00:29:29.885 --> 00:29:31.135
<00:29:29.885>occurring <00:29:30.447>very <00:29:31.010>rarely.

544
00:29:31.195 --> 00:29:34.655
<00:29:31.195>And <00:29:31.350>we <00:29:31.506>need <00:29:31.661>to <00:29:31.817>detect <00:29:31.972>those <00:29:32.128>situations, <00:29:32.283>uh, <00:29:32.439>in <00:29:32.595>order

545
00:29:34.655 --> 00:29:37.775
<00:29:34.655>to <00:29:34.782>make <00:29:34.909>sure <00:29:35.036>that <00:29:35.164>our <00:29:35.291>models <00:29:35.418>we <00:29:35.545>train <00:29:35.673>also <00:29:35.800>work <00:29:35.927>on <00:29:36.055>those,

546
00:29:38.075 --> 00:29:40.215
<00:29:38.075>um, <00:29:38.541>uh, <00:29:39.008>rare <00:29:39.475>situations.

547
00:29:40.875 --> 00:29:45.045
<00:29:40.875>So <00:29:41.002>this <00:29:41.129>is <00:29:41.256>why, <00:29:41.384>uh, <00:29:41.511>we, <00:29:41.638>we, <00:29:41.765>um, <00:29:41.893>um, <00:29:42.020>implement <00:29:42.147>an, <00:29:42.275>uh,

548
00:29:45.045 --> 00:29:47.965
<00:29:45.045>in <00:29:45.220>situ <00:29:45.395>intelligent <00:29:45.570>data <00:29:45.745>collector, <00:29:45.920>we <00:29:46.095>call <00:29:46.270>it <00:29:46.445>instinct.

549
00:29:47.995 --> 00:29:51.525
<00:29:47.995>It's <00:29:48.150>a <00:29:48.306>basically <00:29:48.461>a <00:29:48.617>chronic <00:29:48.772>case <00:29:48.928>detection <00:29:49.083>filter <00:29:49.239>that, <00:29:49.395>um,

550
00:29:51.875 --> 00:29:55.765
<00:29:51.875>that <00:29:52.030>decides <00:29:52.186>on <00:29:52.341>the, <00:29:52.497>on <00:29:52.652>the, <00:29:52.808>um, <00:29:52.963>incoming <00:29:53.119>data, <00:29:53.275>whether

551
00:29:56.275 --> 00:30:00.005
<00:29:56.275>this <00:29:56.450>is, <00:29:56.625>um, <00:29:56.800>giving <00:29:56.975>us <00:29:57.150>good, <00:29:57.325>um, <00:29:57.500>good <00:29:57.675>quality,

552
00:30:00.425 --> 00:30:03.045
<00:30:00.425>and <00:30:00.565>it <00:30:00.705>makes <00:30:00.845>sense <00:30:00.985>to <00:30:01.125>upload <00:30:01.264>it <00:30:01.405>and <00:30:01.544>get <00:30:01.685>it <00:30:01.825>labeled

553
00:30:03.225 --> 00:30:04.565
<00:30:03.225>and <00:30:03.466>train <00:30:03.707>our <00:30:03.948>models <00:30:04.189>with <00:30:04.431>that.

554
00:30:05.455 --> 00:30:10.435
<00:30:05.455>So, <00:30:05.629>um, <00:30:05.804>the, <00:30:05.980>the <00:30:06.155>whole, <00:30:06.330>um, <00:30:06.504>data <00:30:06.679>selection <00:30:06.855>mechanism,

555
00:30:11.015 --> 00:30:12.555
<00:30:11.015>uh, <00:30:11.292>should <00:30:11.569>work <00:30:11.846>as <00:30:12.123>as <00:30:12.401>follows.

556
00:30:12.555 --> 00:30:16.555
<00:30:12.555>So <00:30:12.695>we <00:30:12.835>start <00:30:12.975>with <00:30:13.115>getting <00:30:13.255>our <00:30:13.395>sensors, <00:30:13.535>uh, <00:30:13.675>sensor <00:30:13.815>data, <00:30:13.955>uh,

557
00:30:16.705 --> 00:30:19.915
<00:30:16.705>from, <00:30:16.845>from <00:30:16.985>the <00:30:17.125>camera <00:30:17.264>in, <00:30:17.405>in <00:30:17.544>our <00:30:17.685>case <00:30:17.824>now, <00:30:17.965>uh, <00:30:18.105>uh,

558
00:30:20.335 --> 00:30:23.555
<00:30:20.335>we <00:30:20.510>put <00:30:20.685>our <00:30:20.860>focus, <00:30:21.035>we <00:30:21.210>do <00:30:21.385>some, <00:30:21.560>uh, <00:30:21.735>pre-processing

559
00:30:23.655 --> 00:30:25.955
<00:30:23.655>and, <00:30:23.888>uh, <00:30:24.121>put <00:30:24.355>it <00:30:24.588>into <00:30:24.821>our <00:30:25.055>model

560
00:30:25.975 --> 00:30:27.355
<00:30:25.975>and <00:30:26.182>do <00:30:26.389>some <00:30:26.596>inference <00:30:26.803>with <00:30:27.010>our <00:30:27.217>model.

561
00:30:27.455 --> 00:30:30.595
<00:30:27.455>It <00:30:27.655>might <00:30:27.855>be <00:30:28.054>perception <00:30:28.254>task, <00:30:28.455>like <00:30:28.655>semantic <00:30:28.855>segmentation,

562
00:30:30.695 --> 00:30:32.195
<00:30:30.695>object <00:30:31.032>detection, <00:30:31.370>something <00:30:31.707>like <00:30:32.044>this.

563
00:30:33.055 --> 00:30:36.035
<00:30:33.055>And <00:30:33.335>then, <00:30:33.615>uh, <00:30:33.895>we, <00:30:34.175>we <00:30:34.455>build

564
00:30:36.705 --> 00:30:40.155
<00:30:36.705>several <00:30:36.905>trigger <00:30:37.105>function, <00:30:37.304>one <00:30:37.504>or <00:30:37.705>multiple <00:30:37.905>trigger <00:30:38.105>function.

565
00:30:40.155 --> 00:30:41.195
<00:30:40.155>We <00:30:40.389>call <00:30:40.623>them <00:30:40.857>trigger <00:30:41.091>function

566
00:30:41.195 --> 00:30:45.035
<00:30:41.195>because <00:30:41.428>they <00:30:41.661>basically <00:30:41.895>score <00:30:42.128>the <00:30:42.361>data <00:30:42.595>sample

567
00:30:45.775 --> 00:30:50.195
<00:30:45.775>and, <00:30:45.930>um, <00:30:46.086>and <00:30:46.241>classify <00:30:46.397>them <00:30:46.552>as <00:30:46.708>being <00:30:46.863>worth, <00:30:47.019>uh, <00:30:47.175>to,

568
00:30:50.195 --> 00:30:51.315
<00:30:50.195>to <00:30:50.447>be <00:30:50.699>recorded <00:30:50.951>or <00:30:51.203>not.

569
00:30:51.495 --> 00:30:53.675
<00:30:51.495>So, <00:30:51.695>so <00:30:51.895>we <00:30:52.094>classify <00:30:52.294>them <00:30:52.495>as <00:30:52.695>corner <00:30:52.895>cases.

570
00:30:54.295 --> 00:30:58.275
<00:30:54.295>Um, <00:30:54.402>if <00:30:54.510>we, <00:30:54.618>if <00:30:54.725>this <00:30:54.833>is <00:30:54.941>a <00:30:55.048>corner <00:30:55.156>case, <00:30:55.264>we, <00:30:55.371>we <00:30:55.479>upload <00:30:55.587>them, <00:30:55.695>uh,

571
00:30:58.655 --> 00:31:01.795
<00:30:58.655>and <00:30:58.795>then, <00:30:58.935>um, <00:30:59.075>in, <00:30:59.215>in <00:30:59.355>the <00:30:59.495>backend <00:30:59.635>in <00:30:59.774>the <00:30:59.915>cloud, <00:31:00.055>we,

572
00:31:01.935 --> 00:31:05.435
<00:31:01.935>we <00:31:02.090>can <00:31:02.246>post-process <00:31:02.401>them, <00:31:02.557>meaning <00:31:02.712>we <00:31:02.868>need <00:31:03.023>to <00:31:03.179>label <00:31:03.335>it,

573
00:31:05.435 --> 00:31:08.435
<00:31:05.435>for <00:31:05.610>example, <00:31:05.784>in <00:31:05.960>order <00:31:06.135>to <00:31:06.310>retrain <00:31:06.485>our <00:31:06.659>model, <00:31:06.835>uh,

574
00:31:08.435 --> 00:31:10.395
<00:31:08.435>or <00:31:08.668>fine <00:31:08.901>tune <00:31:09.135>our <00:31:09.368>model, <00:31:09.601>whatever <00:31:09.835>works.

575
00:31:10.655 --> 00:31:14.835
<00:31:10.655>And, <00:31:10.771>um, <00:31:10.888>if <00:31:11.004>we <00:31:11.121>then <00:31:11.238>have <00:31:11.355>a, <00:31:11.471>a <00:31:11.588>new <00:31:11.705>model, <00:31:11.821>a <00:31:11.938>better <00:31:12.055>model,

576
00:31:15.215 --> 00:31:19.875
<00:31:15.215>new <00:31:15.389>model <00:31:15.564>generation, <00:31:15.740>we, <00:31:15.915>um, <00:31:16.090>we, <00:31:16.264>uh, <00:31:16.439>yeah, <00:31:16.615>uh,

577
00:31:20.095 --> 00:31:23.235
<00:31:20.095>upload <00:31:20.235>it <00:31:20.375>back <00:31:20.515>to <00:31:20.655>the <00:31:20.795>car <00:31:20.935>or <00:31:21.075>to <00:31:21.215>the <00:31:21.355>fleet <00:31:21.495>and,

578
00:31:23.455 --> 00:31:26.875
<00:31:23.455>and <00:31:23.655>have <00:31:23.855>this, <00:31:24.054>uh, <00:31:24.254>cyclic, <00:31:24.455>uh, <00:31:24.655>improvement <00:31:24.855>of,

579
00:31:26.895 --> 00:31:29.475
<00:31:26.895>of <00:31:27.128>our <00:31:27.361>models, <00:31:27.595>um, <00:31:27.828>uh, <00:31:28.061>over <00:31:28.295>time.

580
00:31:30.415 --> 00:31:32.475
<00:31:30.415>So <00:31:30.648>now <00:31:30.881>how <00:31:31.115>can <00:31:31.348>we <00:31:31.581>do <00:31:31.815>those?

581
00:31:32.615 --> 00:31:35.355
<00:31:32.615>Uh, <00:31:32.790>how, <00:31:32.965>how <00:31:33.140>can <00:31:33.315>we <00:31:33.490>implement <00:31:33.665>those <00:31:33.840>trigger <00:31:34.015>functions?

582
00:31:35.355 --> 00:31:36.635
<00:31:35.355>How <00:31:35.585>do <00:31:35.815>we <00:31:36.046>select <00:31:36.276>the <00:31:36.507>data?

583
00:31:37.215 --> 00:31:40.155
<00:31:37.215>Uh, <00:31:37.355>there, <00:31:37.495>there <00:31:37.635>are <00:31:37.774>several <00:31:37.915>ways <00:31:38.054>on <00:31:38.195>how <00:31:38.334>to <00:31:38.475>do <00:31:38.615>this.

584
00:31:40.775 --> 00:31:45.475
<00:31:40.775>Um, <00:31:40.975>here's <00:31:41.175>one <00:31:41.375>basically <00:31:41.575>very <00:31:41.775>naive <00:31:41.975>example, <00:31:42.175>the,

585
00:31:45.495 --> 00:31:46.715
<00:31:45.495>the <00:31:45.860>max <00:31:46.226>class <00:31:46.592>probability.

586
00:31:47.335 --> 00:31:49.595
<00:31:47.335>Um, <00:31:47.568>Tim <00:31:47.801>talked <00:31:48.035>about <00:31:48.268>this <00:31:48.501>earlier <00:31:48.735>on.

587
00:31:49.595 --> 00:31:52.075
<00:31:49.595>What <00:31:49.795>are <00:31:49.995>the <00:31:50.195>difficulties, <00:31:50.395>uh, <00:31:50.595>with <00:31:50.795>this <00:31:50.995>approach?

588
00:31:52.455 --> 00:31:55.435
<00:31:52.455>Um, <00:31:52.610>is <00:31:52.766>it <00:31:52.921>really <00:31:53.077>a <00:31:53.232>good <00:31:53.388>uncertainty <00:31:53.543>measure <00:31:53.699>or <00:31:53.855>not?

589
00:31:55.535 --> 00:31:59.195
<00:31:55.535>So <00:31:55.735>there <00:31:55.935>are <00:31:56.135>more <00:31:56.335>advanced <00:31:56.535>approaches <00:31:56.735>like <00:31:56.935>Montecarlo

590
00:31:59.195 --> 00:32:02.315
<00:31:59.195>dropout <00:31:59.544>or <00:31:59.895>ensemble <00:32:00.245>methods, <00:32:00.595>uh,

591
00:32:02.315 --> 00:32:05.675
<00:32:02.315>that <00:32:02.490>might <00:32:02.665>then <00:32:02.840>in <00:32:03.015>turn <00:32:03.190>need <00:32:03.365>more <00:32:03.540>computational <00:32:03.715>power.

592
00:32:05.855 --> 00:32:10.155
<00:32:05.855>So <00:32:06.055>we, <00:32:06.255>we, <00:32:06.455>the <00:32:06.655>computational, <00:32:06.855>uh, <00:32:07.055>complexity <00:32:07.255>increases

593
00:32:10.155 --> 00:32:12.395
<00:32:10.155>and <00:32:10.310>we <00:32:10.466>need <00:32:10.621>more <00:32:10.777>powerful <00:32:10.932>hardware <00:32:11.088>maybe <00:32:11.243>in <00:32:11.399>the <00:32:11.555>car.

594
00:32:13.055 --> 00:32:15.715
<00:32:13.055>Um, <00:32:13.230>we <00:32:13.405>can <00:32:13.580>also <00:32:13.755>look <00:32:13.930>at <00:32:14.105>temporal <00:32:14.280>methods <00:32:14.455>that,

595
00:32:15.715 --> 00:32:20.355
<00:32:15.715>that <00:32:15.889>look <00:32:16.064>at <00:32:16.240>the <00:32:16.415>variability <00:32:16.590>of <00:32:16.764>the <00:32:16.939>inference, <00:32:17.115>uh,

596
00:32:20.355 --> 00:32:24.075
<00:32:20.355>results <00:32:20.530>over <00:32:20.705>time <00:32:20.880>and <00:32:21.055>this <00:32:21.230>way, <00:32:21.405>uh, <00:32:21.580>estimating <00:32:21.755>uncertainty.

597
00:32:24.775 --> 00:32:27.995
<00:32:24.775>Um, <00:32:24.950>and <00:32:25.125>there <00:32:25.300>are, <00:32:25.475>um, <00:32:25.650>other <00:32:25.825>methods <00:32:26.000>like <00:32:26.175>metastatic

598
00:32:28.145 --> 00:32:30.395
<00:32:28.145>that <00:32:28.425>predict, <00:32:28.705>uh, <00:32:28.985>the, <00:32:29.264>the <00:32:29.545>model

599
00:32:31.215 --> 00:32:34.235
<00:32:31.215>and <00:32:31.448>performance <00:32:31.681>basically, <00:32:31.915>uh, <00:32:32.148>on <00:32:32.381>certain <00:32:32.615>samples.

600
00:32:34.375 --> 00:32:36.155
<00:32:34.375>And <00:32:34.550>we <00:32:34.725>can <00:32:34.900>use <00:32:35.075>this <00:32:35.250>as <00:32:35.425>a <00:32:35.600>signal <00:32:35.775>for,

601
00:32:36.415 --> 00:32:38.315
<00:32:36.415>for <00:32:36.881>triggering <00:32:37.348>our <00:32:37.815>recording.

602
00:32:39.385 --> 00:32:41.805
<00:32:39.385>So <00:32:39.618>this <00:32:39.851>whole <00:32:40.085>framework <00:32:40.318>is <00:32:40.551>known <00:32:40.785>actually

603
00:32:41.865 --> 00:32:43.845
<00:32:41.865>as <00:32:42.215>an <00:32:42.565>active <00:32:42.915>learning <00:32:43.265>framework.

604
00:32:44.505 --> 00:32:46.245
<00:32:44.505>So <00:32:44.738>this <00:32:44.971>means, <00:32:45.205>this <00:32:45.438>is <00:32:45.671>a <00:32:45.905>framework

605
00:32:46.245 --> 00:32:48.845
<00:32:46.245>where <00:32:46.445>a <00:32:46.645>system <00:32:46.844>could <00:32:47.044>learn <00:32:47.245>from <00:32:47.445>small <00:32:47.645>amounts

606
00:32:48.845 --> 00:32:50.605
<00:32:48.845>of <00:32:49.195>data <00:32:49.545>initially, <00:32:49.895>small <00:32:50.245>amounts,

607
00:32:50.945 --> 00:32:53.445
<00:32:50.945>and <00:32:51.178>then <00:32:51.411>decide <00:32:51.645>by <00:32:51.878>itself <00:32:52.111>which <00:32:52.345>samples

608
00:32:53.545 --> 00:32:54.565
<00:32:53.545>are <00:32:54.004>most <00:32:54.463>informative.

609
00:32:55.305 --> 00:32:59.645
<00:32:55.305>Uh, <00:32:55.445>by <00:32:55.585>informative <00:32:55.725>we <00:32:55.865>mean <00:32:56.005>always, <00:32:56.145>like, <00:32:56.285>do <00:32:56.425>they <00:32:56.565>give <00:32:56.705>us,

610
00:32:59.905 --> 00:33:03.405
<00:32:59.905>uh, <00:33:00.080>an, <00:33:00.254>an <00:33:00.430>advantage <00:33:00.605>if <00:33:00.780>we <00:33:00.955>use <00:33:01.129>them, <00:33:01.305>uh,

611
00:33:03.665 --> 00:33:04.845
<00:33:03.665>for <00:33:03.930>if <00:33:04.196>we <00:33:04.461>label <00:33:04.727>them

612
00:33:04.905 --> 00:33:07.925
<00:33:04.905>and <00:33:05.080>then, <00:33:05.254>uh, <00:33:05.430>use <00:33:05.605>them <00:33:05.780>for, <00:33:05.955>for <00:33:06.129>further <00:33:06.305>training?

613
00:33:08.145 --> 00:33:11.205
<00:33:08.145>So <00:33:08.320>do <00:33:08.495>they <00:33:08.670>add, <00:33:08.845>uh, <00:33:09.020>performance <00:33:09.195>improvements, <00:33:09.370>for <00:33:09.545>example,

614
00:33:11.665 --> 00:33:14.765
<00:33:11.665>or <00:33:11.898>robustness <00:33:12.131>improvements, <00:33:12.365>um, <00:33:12.598>for <00:33:12.831>our <00:33:13.065>model,

615
00:33:14.785 --> 00:33:17.205
<00:33:14.785>and <00:33:14.985>thereby <00:33:15.185>reducing <00:33:15.385>the <00:33:15.585>cost <00:33:15.785>of <00:33:15.985>human <00:33:16.185>labeling

616
00:33:17.205 --> 00:33:20.445
<00:33:17.205>because <00:33:17.438>we <00:33:17.671>don't <00:33:17.905>just <00:33:18.138>randomly <00:33:18.371>select <00:33:18.605>data,

617
00:33:20.505 --> 00:33:22.965
<00:33:20.505>and <00:33:20.621>then <00:33:20.738>there <00:33:20.855>might <00:33:20.971>be <00:33:21.088>a <00:33:21.205>lot <00:33:21.321>of <00:33:21.438>data <00:33:21.555>that <00:33:21.671>is <00:33:21.788>not <00:33:21.905>so

618
00:33:23.695 --> 00:33:25.925
<00:33:23.695>maybe <00:33:23.975>redundant <00:33:24.254>or <00:33:24.534>not <00:33:24.814>so <00:33:25.095>informative.

619
00:33:26.545 --> 00:33:28.165
<00:33:26.545>And <00:33:26.825>that's <00:33:27.105>the <00:33:27.385>active <00:33:27.665>learning <00:33:27.945>framework.

620
00:33:28.165 --> 00:33:30.925
<00:33:28.165>So <00:33:28.305>we <00:33:28.445>start <00:33:28.585>with <00:33:28.725>the <00:33:28.865>label <00:33:29.004>training <00:33:29.145>set <00:33:29.284>and <00:33:29.425>initial <00:33:29.565>set,

621
00:33:30.925 --> 00:33:33.205
<00:33:30.925>and <00:33:31.100>then <00:33:31.274>we, <00:33:31.450>uh, <00:33:31.625>train <00:33:31.800>our <00:33:31.975>model <00:33:32.149>on <00:33:32.325>that.

622
00:33:34.025 --> 00:33:35.085
<00:33:34.025>Um, <00:33:34.979>and,

623
00:33:35.145 --> 00:33:39.325
<00:33:35.145>and <00:33:35.272>we <00:33:35.399>apply <00:33:35.526>this <00:33:35.654>model <00:33:35.781>to <00:33:35.908>a <00:33:36.035>bunch <00:33:36.163>of <00:33:36.290>unlabeled, <00:33:36.417>uh, <00:33:36.545>data

624
00:33:39.865 --> 00:33:42.565
<00:33:39.865>select <00:33:40.020>set <00:33:40.176>in, <00:33:40.331>in <00:33:40.487>our <00:33:40.642>case, <00:33:40.798>uh, <00:33:40.953>with <00:33:41.109>a <00:33:41.265>car,

625
00:33:42.595 --> 00:33:45.645
<00:33:42.595>it's <00:33:42.795>just <00:33:42.995>driving <00:33:43.195>around <00:33:43.395>in <00:33:43.595>wild <00:33:43.795>And, <00:33:43.995>uh,

626
00:33:45.905 --> 00:33:49.085
<00:33:45.905>and <00:33:46.045>applying <00:33:46.185>this <00:33:46.325>model <00:33:46.465>to, <00:33:46.605>to <00:33:46.745>the <00:33:46.885>situations <00:33:47.024>we <00:33:47.165>occur, <00:33:47.305>uh,

627
00:33:49.145 --> 00:33:50.205
<00:33:49.145>we, <00:33:49.621>we <00:33:50.099>encounter.

628
00:33:51.025 --> 00:33:53.685
<00:33:51.025>And <00:33:51.258>then <00:33:51.491>using <00:33:51.725>our <00:33:51.958>smart <00:33:52.191>triggers, <00:33:52.425>uh,

629
00:33:53.685 --> 00:33:55.045
<00:33:53.685>the <00:33:54.093>acquisition <00:33:54.501>function, <00:33:54.909>we,

630
00:33:55.185 --> 00:33:57.725
<00:33:55.185>we <00:33:55.418>basically <00:33:55.651>select <00:33:55.885>the <00:33:56.118>good <00:33:56.351>ones, <00:33:56.585>the,

631
00:33:57.745 --> 00:33:59.205
<00:33:57.745>the <00:33:58.183>most <00:33:58.620>informative <00:33:59.059>samples,

632
00:33:59.425 --> 00:34:02.205
<00:33:59.425>and, <00:33:59.565>uh, <00:33:59.705>we <00:33:59.845>call <00:33:59.985>them, <00:34:00.125>give <00:34:00.264>them <00:34:00.405>to <00:34:00.544>a <00:34:00.685>human <00:34:00.825>annotator

633
00:34:02.545 --> 00:34:05.125
<00:34:02.545>to <00:34:02.778>get <00:34:03.011>them <00:34:03.245>labeled, <00:34:03.478>and <00:34:03.711>this <00:34:03.945>way,

634
00:34:05.125 --> 00:34:07.885
<00:34:05.125>increasing <00:34:05.300>our <00:34:05.475>labeled <00:34:05.650>training <00:34:05.825>set <00:34:06.000>and <00:34:06.175>close <00:34:06.350>the <00:34:06.525>loop

635
00:34:07.905 --> 00:34:12.285
<00:34:07.905>and <00:34:08.080>do <00:34:08.255>this <00:34:08.429>iteratively <00:34:08.605>in <00:34:08.779>order <00:34:08.955>to, <00:34:09.130>um, <00:34:09.304>yeah,

636
00:34:12.595 --> 00:34:15.365
<00:34:12.594>achieve <00:34:12.875>a <00:34:13.154>good, <00:34:13.435>good <00:34:13.714>and <00:34:13.995>good

637
00:34:15.425 --> 00:34:19.405
<00:34:15.425>and <00:34:15.600>cost <00:34:15.775>efficient, <00:34:15.950>um, <00:34:16.125>uh, <00:34:16.300>model <00:34:16.475>in <00:34:16.650>the <00:34:16.825>end.

638
00:34:20.645 --> 00:34:22.105
<00:34:20.645>So <00:34:20.864>how, <00:34:21.083>how <00:34:21.302>are <00:34:21.521>I <00:34:21.739>doing <00:34:21.958>this?

639
00:34:22.105 --> 00:34:24.985
<00:34:22.105>In, <00:34:22.260>in <00:34:22.416>our <00:34:22.571>example <00:34:22.727>study <00:34:22.882>here, <00:34:23.038>we <00:34:23.193>are <00:34:23.349>using, <00:34:23.505>uh,

640
00:34:25.165 --> 00:34:26.425
<00:34:25.165>the <00:34:25.391>A <00:34:25.618>2D <00:34:25.845>two <00:34:26.072>data <00:34:26.299>set.

641
00:34:26.885 --> 00:34:29.905
<00:34:26.885>Um, <00:34:27.025>you <00:34:27.165>might <00:34:27.305>note <00:34:27.445>it's <00:34:27.585>a <00:34:27.725>publicly <00:34:27.865>available <00:34:28.005>data <00:34:28.145>set, <00:34:28.285>uh,

642
00:34:30.265 --> 00:34:32.705
<00:34:30.264>released <00:34:30.498>by <00:34:30.731>Audi, <00:34:30.964>um, <00:34:31.198>where <00:34:31.431>you <00:34:31.665>have

643
00:34:32.705 --> 00:34:34.985
<00:34:32.705>around <00:34:33.171>40,000 <00:34:33.638>labeled <00:34:34.105>images.

644
00:34:35.405 --> 00:34:36.945
<00:34:35.405>So, <00:34:35.751>uh, <00:34:36.098>we <00:34:36.444>have <00:34:36.791>segmentation,

645
00:34:37.185 --> 00:34:38.945
<00:34:37.185>semantic <00:34:37.534>segmentation <00:34:37.884>labels <00:34:38.235>for <00:34:38.585>those.

646
00:34:39.645 --> 00:34:42.545
<00:34:39.645>And, <00:34:39.820>uh, <00:34:39.995>we <00:34:40.170>use <00:34:40.344>this <00:34:40.520>for <00:34:40.695>active <00:34:40.870>learning <00:34:41.045>experiments.

647
00:34:43.005 --> 00:34:45.385
<00:34:43.005>Um, <00:34:43.160>first <00:34:43.316>we <00:34:43.471>split <00:34:43.627>this <00:34:43.782>data <00:34:43.938>set <00:34:44.093>up <00:34:44.249>into <00:34:44.405>several,

648
00:34:45.725 --> 00:34:46.785
<00:34:45.725>uh, <00:34:46.201>different <00:34:46.679>splits.

649
00:34:47.285 --> 00:34:49.065
<00:34:47.284>Uh, <00:34:47.518>we <00:34:47.751>have <00:34:47.984>a <00:34:48.218>training <00:34:48.451>set, <00:34:48.685>uh,

650
00:34:49.145 --> 00:34:51.145
<00:34:49.145>and <00:34:49.344>validation <00:34:49.545>set <00:34:49.745>we <00:34:49.945>use <00:34:50.145>for <00:34:50.344>initial <00:34:50.545>training.

651
00:34:51.605 --> 00:34:54.265
<00:34:51.605>And <00:34:51.760>then <00:34:51.916>we <00:34:52.071>have <00:34:52.227>a <00:34:52.382>select <00:34:52.538>set <00:34:52.693>which <00:34:52.849>we, <00:34:53.005>um,

652
00:34:54.475 --> 00:34:58.225
<00:34:54.475>which <00:34:54.650>simulates <00:34:54.824>our, <00:34:55.000>um, <00:34:55.174>driving <00:34:55.350>around <00:34:55.525>in <00:34:55.699>the <00:34:55.875>wild.

653
00:34:58.845 --> 00:35:02.305
<00:34:58.845>Um, <00:34:58.961>so <00:34:59.078>the, <00:34:59.194>the <00:34:59.311>data <00:34:59.428>set <00:34:59.544>we <00:34:59.661>want <00:34:59.778>to <00:34:59.895>select <00:35:00.011>from <00:35:00.128>in <00:35:00.245>order

654
00:35:02.305 --> 00:35:05.025
<00:35:02.305>to <00:35:02.460>improve <00:35:02.616>our <00:35:02.771>model, <00:35:02.927>and <00:35:03.082>then <00:35:03.238>we <00:35:03.393>have, <00:35:03.549>um, <00:35:03.705>to

655
00:35:05.545 --> 00:35:08.265
<00:35:05.545>evaluate <00:35:05.672>the <00:35:05.799>performance <00:35:05.926>in <00:35:06.054>the <00:35:06.181>end, <00:35:06.308>we <00:35:06.435>have <00:35:06.563>a, <00:35:06.690>a <00:35:06.817>test <00:35:06.945>set,

656
00:35:08.285 --> 00:35:11.225
<00:35:08.285>and <00:35:08.460>we, <00:35:08.634>we <00:35:08.810>don't <00:35:08.984>only <00:35:09.160>take <00:35:09.335>one <00:35:09.509>test <00:35:09.685>set

657
00:35:11.225 --> 00:35:12.465
<00:35:11.225>that <00:35:11.596>is <00:35:11.969>randomly <00:35:12.341>sampled,

658
00:35:12.465 --> 00:35:15.705
<00:35:12.465>but <00:35:12.620>we <00:35:12.776>also <00:35:12.931>have <00:35:13.087>a <00:35:13.242>second <00:35:13.398>test <00:35:13.553>set <00:35:13.709>that, <00:35:13.865>um,

659
00:35:16.145 --> 00:35:18.185
<00:35:16.145>specifically <00:35:16.495>contains <00:35:16.845>a <00:35:17.195>corner <00:35:17.545>cases.

660
00:35:18.365 --> 00:35:19.705
<00:35:18.365>So <00:35:18.606>where, <00:35:18.847>where <00:35:19.088>the <00:35:19.329>corner <00:35:19.571>case,

661
00:35:19.765 --> 00:35:22.785
<00:35:19.765>the <00:35:19.964>difficult <00:35:20.165>cases <00:35:20.365>are <00:35:20.565>over <00:35:20.765>sampled, <00:35:20.964>let's <00:35:21.165>say,

662
00:35:23.555 --> 00:35:24.845
<00:35:23.555>okay, <00:35:23.748>how, <00:35:23.942>how <00:35:24.135>do <00:35:24.328>we <00:35:24.522>do <00:35:24.716>this?

663
00:35:24.905 --> 00:35:27.325
<00:35:24.905>Uh, <00:35:25.045>we, <00:35:25.185>we, <00:35:25.325>on <00:35:25.465>the <00:35:25.605>initial <00:35:25.745>set, <00:35:25.885>we <00:35:26.025>train <00:35:26.165>our <00:35:26.305>model,

664
00:35:27.465 --> 00:35:30.125
<00:35:27.465>we <00:35:27.640>get <00:35:27.815>a <00:35:27.990>baseline <00:35:28.165>model, <00:35:28.340>we <00:35:28.515>apply <00:35:28.690>this <00:35:28.865>model

665
00:35:30.145 --> 00:35:31.245
<00:35:30.145>to <00:35:30.475>our <00:35:30.805>select <00:35:31.134>set,

666
00:35:31.245 --> 00:35:33.885
<00:35:31.245>and <00:35:31.400>then <00:35:31.556>use <00:35:31.711>the <00:35:31.867>trigger <00:35:32.022>function <00:35:32.178>to <00:35:32.333>evaluate <00:35:32.489>the <00:35:32.645>outcome

667
00:35:34.465 --> 00:35:38.885
<00:35:34.465>and <00:35:34.620>to <00:35:34.776>score <00:35:34.931>the <00:35:35.087>outcome <00:35:35.242>and <00:35:35.398>score <00:35:35.553>each <00:35:35.709>image, <00:35:35.865>um,

668
00:35:39.025 --> 00:35:40.285
<00:35:39.025>and <00:35:39.403>to <00:35:39.781>decide <00:35:40.158>whether

669
00:35:40.345 --> 00:35:42.045
<00:35:40.345>or <00:35:40.625>not <00:35:40.904>to, <00:35:41.185>to <00:35:41.464>select, <00:35:41.745>um,

670
00:35:42.225 --> 00:35:44.165
<00:35:42.225>images <00:35:42.458>from <00:35:42.691>this <00:35:42.924>and, <00:35:43.158>and <00:35:43.391>how <00:35:43.625>many.

671
00:35:44.705 --> 00:35:47.605
<00:35:44.705>And <00:35:44.832>then <00:35:44.959>this <00:35:45.086>way, <00:35:45.214>we <00:35:45.341>generate <00:35:45.468>a <00:35:45.595>new <00:35:45.723>training <00:35:45.850>set, <00:35:45.977>which <00:35:46.105>we,

672
00:35:47.615 --> 00:35:49.805
<00:35:47.615>which <00:35:47.895>we <00:35:48.174>then <00:35:48.455>use <00:35:48.734>for <00:35:49.015>retraining.

673
00:35:50.905 --> 00:35:52.685
<00:35:50.905>We <00:35:51.105>get <00:35:51.305>a <00:35:51.505>better <00:35:51.705>model <00:35:51.905>and <00:35:52.105>improve <00:35:52.305>model,

674
00:35:52.945 --> 00:35:54.845
<00:35:52.945>and <00:35:53.120>we <00:35:53.295>apply <00:35:53.470>this <00:35:53.645>model <00:35:53.820>to <00:35:53.995>our <00:35:54.170>test <00:35:54.345>sets

675
00:35:54.845 --> 00:35:55.885
<00:35:54.845>to <00:35:55.157>get <00:35:55.469>some <00:35:55.781>results.

676
00:35:56.705 --> 00:35:59.925
<00:35:56.705>Um, <00:35:56.845>and <00:35:56.985>of <00:35:57.125>course, <00:35:57.265>this <00:35:57.404>can <00:35:57.545>be <00:35:57.685>training <00:35:57.825>from <00:35:57.965>scratch <00:35:58.105>or,

677
00:36:00.185 --> 00:36:02.925
<00:36:00.185>or <00:36:00.384>fine <00:36:00.585>tuning <00:36:00.785>the, <00:36:00.985>the <00:36:01.185>existing <00:36:01.384>model, <00:36:01.585>uh,

678
00:36:02.925 --> 00:36:04.245
<00:36:02.925>different <00:36:03.519>variations <00:36:04.113>possible.

679
00:36:05.105 --> 00:36:08.885
<00:36:05.105>Uh, <00:36:05.280>so <00:36:05.455>now <00:36:05.630>with <00:36:05.805>those <00:36:05.980>results, <00:36:06.155>um, <00:36:06.330>we, <00:36:06.505>we,

680
00:36:08.885 --> 00:36:11.085
<00:36:08.885>we <00:36:09.060>only <00:36:09.235>get <00:36:09.410>one <00:36:09.585>side <00:36:09.760>of <00:36:09.935>the, <00:36:10.110>the <00:36:10.285>story,

681
00:36:11.105 --> 00:36:15.285
<00:36:11.105>but <00:36:11.245>we <00:36:11.385>want <00:36:11.525>to <00:36:11.665>kind <00:36:11.805>of, <00:36:11.945>um, <00:36:12.085>compare <00:36:12.225>this <00:36:12.365>to, <00:36:12.505>to

682
00:36:15.805 --> 00:36:19.205
<00:36:15.805>existing <00:36:15.980>methods <00:36:16.154>or <00:36:16.330>to <00:36:16.504>a <00:36:16.680>baseline <00:36:16.855>method, <00:36:17.029>um, <00:36:17.205>to,

683
00:36:19.305 --> 00:36:21.045
<00:36:19.305>in <00:36:19.585>this <00:36:19.865>case, <00:36:20.145>random <00:36:20.424>data <00:36:20.705>selection.

684
00:36:21.105 --> 00:36:22.925
<00:36:21.105>So <00:36:21.305>we <00:36:21.505>do <00:36:21.705>the <00:36:21.905>same <00:36:22.105>story <00:36:22.305>again <00:36:22.505>for,

685
00:36:22.925 --> 00:36:24.205
<00:36:22.925>for <00:36:23.213>the <00:36:23.501>random <00:36:23.789>data <00:36:24.077>selection.

686
00:36:24.205 --> 00:36:27.205
<00:36:24.205>We <00:36:24.360>also <00:36:24.516>select <00:36:24.671>a <00:36:24.827>certain <00:36:24.982>number, <00:36:25.138>same <00:36:25.293>amount <00:36:25.449>of <00:36:25.605>data,

687
00:36:27.315 --> 00:36:29.165
<00:36:27.315>like <00:36:27.595>in <00:36:27.875>the, <00:36:28.155>uh, <00:36:28.435>intelligent <00:36:28.715>trigger.

688
00:36:29.625 --> 00:36:32.565
<00:36:29.625>We <00:36:29.800>select <00:36:29.975>the <00:36:30.150>same <00:36:30.325>amount <00:36:30.500>of <00:36:30.675>data <00:36:30.850>randomly, <00:36:31.025>uh,

689
00:36:32.585 --> 00:36:35.445
<00:36:32.585>by <00:36:32.740>just <00:36:32.896>flipping <00:36:33.051>a <00:36:33.207>coin, <00:36:33.362>um, <00:36:33.518>or <00:36:33.673>a <00:36:33.829>biased <00:36:33.985>coin.

690
00:36:35.905 --> 00:36:38.565
<00:36:35.905>And <00:36:36.060>then, <00:36:36.216>uh, <00:36:36.371>using <00:36:36.527>this <00:36:36.682>data <00:36:36.838>to <00:36:36.993>retrain <00:36:37.149>our <00:36:37.305>model

691
00:36:38.585 --> 00:36:42.285
<00:36:38.585>and <00:36:38.740>apply <00:36:38.896>the <00:36:39.051>same <00:36:39.207>evaluations <00:36:39.362>and <00:36:39.518>compare <00:36:39.673>the, <00:36:39.829>the <00:36:39.985>two.

692
00:36:43.265 --> 00:36:47.325
<00:36:43.265>So <00:36:43.440>in <00:36:43.615>our, <00:36:43.790>um, <00:36:43.964>study <00:36:44.140>I <00:36:44.315>present <00:36:44.490>here, <00:36:44.665>we,

693
00:36:47.385 --> 00:36:51.205
<00:36:47.385>we <00:36:47.512>have <00:36:47.639>a <00:36:47.766>specific <00:36:47.894>use <00:36:48.021>case, <00:36:48.148>uh, <00:36:48.275>simply <00:36:48.403>one <00:36:48.530>we <00:36:48.657>use, <00:36:48.785>uh,

694
00:36:51.265 --> 00:36:54.885
<00:36:51.265>not <00:36:51.381>the <00:36:51.498>full <00:36:51.615>class <00:36:51.731>set <00:36:51.848>of <00:36:51.964>the <00:36:52.081>A <00:36:52.198>2D <00:36:52.315>two <00:36:52.431>data <00:36:52.548>set, <00:36:52.665>uh,

695
00:36:54.885 --> 00:36:56.325
<00:36:54.885>with <00:36:55.209>its <00:36:55.533>30 <00:36:55.857>plus <00:36:56.181>classes,

696
00:36:56.465 --> 00:36:58.325
<00:36:56.465>but <00:36:56.698>we <00:36:56.931>use <00:36:57.165>only <00:36:57.398>two <00:36:57.631>namely, <00:36:57.865>um,

697
00:36:58.345 --> 00:36:59.965
<00:36:58.345>the <00:36:58.625>traffic <00:36:58.904>light <00:36:59.185>and <00:36:59.464>everything <00:36:59.745>else.

698
00:37:00.425 --> 00:37:04.245
<00:37:00.425>So <00:37:00.565>we <00:37:00.705>do <00:37:00.845>a <00:37:00.985>two <00:37:01.125>class <00:37:01.265>segmentation, <00:37:01.405>um, <00:37:01.545>uh, <00:37:01.685>task <00:37:01.825>here,

699
00:37:04.525 --> 00:37:06.685
<00:37:04.525>a <00:37:04.805>two <00:37:05.085>class <00:37:05.365>segmentation <00:37:05.645>task, <00:37:05.925>uh,

700
00:37:06.815 --> 00:37:10.365
<00:37:06.815>where <00:37:07.015>we <00:37:07.215>have <00:37:07.415>a <00:37:07.615>segmentation, <00:37:07.815>uh, <00:37:08.015>neur <00:37:08.215>network,

701
00:37:10.525 --> 00:37:14.125
<00:37:10.525>a <00:37:10.700>condition <00:37:10.875>neur <00:37:11.050>network, <00:37:11.225>um, <00:37:11.400>getting <00:37:11.575>the <00:37:11.750>camera <00:37:11.925>image

702
00:37:14.125 --> 00:37:16.925
<00:37:14.125>as <00:37:14.325>an <00:37:14.525>input <00:37:14.725>and <00:37:14.925>then <00:37:15.125>classifying <00:37:15.325>each <00:37:15.525>pixel,

703
00:37:17.075 --> 00:37:19.605
<00:37:17.075>whether <00:37:17.250>this <00:37:17.424>is <00:37:17.600>a <00:37:17.774>traffic <00:37:17.950>light <00:37:18.125>or <00:37:18.299>something <00:37:18.475>else.

704
00:37:20.465 --> 00:37:22.925
<00:37:20.465>And <00:37:20.620>so <00:37:20.776>we <00:37:20.931>end <00:37:21.087>up <00:37:21.242>with <00:37:21.398>a <00:37:21.553>pixel <00:37:21.709>wise <00:37:21.865>classification.

705
00:37:23.105 --> 00:37:27.005
<00:37:23.105>Now <00:37:23.280>we <00:37:23.455>group <00:37:23.630>those <00:37:23.805>pixels <00:37:23.980>that <00:37:24.155>are <00:37:24.330>nearby, <00:37:24.505>uh,

706
00:37:27.005 --> 00:37:30.405
<00:37:27.005>using <00:37:27.285>connected <00:37:27.565>components <00:37:27.845>analysis, <00:37:28.125>uh, <00:37:28.405>into

707
00:37:30.755 --> 00:37:32.965
<00:37:30.755>what <00:37:30.988>we <00:37:31.221>think <00:37:31.455>are <00:37:31.688>traffic <00:37:31.921>light <00:37:32.155>patches.

708
00:37:33.545 --> 00:37:37.125
<00:37:33.545>And <00:37:33.745>those, <00:37:33.945>we, <00:37:34.145>um, <00:37:34.345>then <00:37:34.545>rank, <00:37:34.745>uh, <00:37:34.945>uh,

709
00:37:37.215 --> 00:37:38.965
<00:37:37.215>using <00:37:37.681>a <00:37:38.148>uncertainty <00:37:38.615>calculation.

710
00:37:39.345 --> 00:37:41.445
<00:37:39.345>So <00:37:39.578>for <00:37:39.811>each <00:37:40.044>component, <00:37:40.278>we <00:37:40.511>now, <00:37:40.745>uh,

711
00:37:41.465 --> 00:37:44.645
<00:37:41.465>or <00:37:41.665>we <00:37:41.865>assign <00:37:42.065>a <00:37:42.265>com, <00:37:42.465>uh, <00:37:42.665>an <00:37:42.865>uncertainty

712
00:37:44.665 --> 00:37:45.965
<00:37:44.665>to <00:37:44.957>each <00:37:45.250>of <00:37:45.542>those <00:37:45.835>components.

713
00:37:46.625 --> 00:37:49.685
<00:37:46.625>Uh, <00:37:46.825>for <00:37:47.025>example, <00:37:47.225>using <00:37:47.425>the <00:37:47.625>max <00:37:47.825>class <00:37:48.025>probability.

714
00:37:50.425 --> 00:37:52.925
<00:37:50.425>Uh, <00:37:50.625>and <00:37:50.825>this <00:37:51.025>could <00:37:51.225>be, <00:37:51.425>for <00:37:51.625>example, <00:37:51.825>taking

715
00:37:52.985 --> 00:37:56.685
<00:37:52.985>of <00:37:53.125>all <00:37:53.265>the <00:37:53.405>pixels <00:37:53.545>in <00:37:53.685>one <00:37:53.825>component, <00:37:53.965>taking <00:37:54.105>the <00:37:54.245>median, <00:37:54.385>um,

716
00:37:57.145 --> 00:37:58.685
<00:37:57.145>uh, <00:37:57.607>soft <00:37:58.069>max <00:37:58.531>probability

717
00:37:59.305 --> 00:38:02.845
<00:37:59.305>and <00:37:59.444>say, <00:37:59.585>okay, <00:37:59.725>this <00:37:59.865>is <00:38:00.004>our, <00:38:00.145>um, <00:38:00.285>class <00:38:00.424>probability <00:38:00.565>of <00:38:00.705>this,

718
00:38:03.505 --> 00:38:04.885
<00:38:03.505>um, <00:38:03.919>of <00:38:04.333>this <00:38:04.747>component,

719
00:38:04.945 --> 00:38:06.885
<00:38:04.945>or <00:38:05.145>the <00:38:05.345>other <00:38:05.545>way <00:38:05.745>around <00:38:05.945>the, <00:38:06.145>the <00:38:06.345>uncertainty.

720
00:38:07.425 --> 00:38:10.565
<00:38:07.425>And, <00:38:07.580>um, <00:38:07.736>the <00:38:07.891>same <00:38:08.047>we <00:38:08.202>can <00:38:08.358>do <00:38:08.513>with <00:38:08.669>temporary <00:38:08.825>consistency

721
00:38:10.565 --> 00:38:13.885
<00:38:10.565>where <00:38:10.705>we <00:38:10.845>say <00:38:10.985>we, <00:38:11.125>we <00:38:11.265>look <00:38:11.405>at <00:38:11.545>the <00:38:11.685>component <00:38:11.825>over <00:38:11.965>time,

722
00:38:13.915 --> 00:38:17.005
<00:38:13.915>over <00:38:14.042>frame, <00:38:14.169>so <00:38:14.296>we <00:38:14.424>find <00:38:14.551>the <00:38:14.678>same <00:38:14.805>component <00:38:14.933>in <00:38:15.060>the <00:38:15.187>next <00:38:15.315>frame,

723
00:38:17.625 --> 00:38:22.365
<00:38:17.625>and <00:38:17.765>then <00:38:17.905>see <00:38:18.045>how <00:38:18.185>the <00:38:18.325>network <00:38:18.465>decision <00:38:18.605>has <00:38:18.745>changed <00:38:18.885>over <00:38:19.025>time

724
00:38:22.545 --> 00:38:25.765
<00:38:22.545>or <00:38:22.700>the <00:38:22.856>soft <00:38:23.011>max, <00:38:23.167>um, <00:38:23.322>score <00:38:23.478>has <00:38:23.633>changed <00:38:23.789>over <00:38:23.945>time.

725
00:38:26.745 --> 00:38:28.085
<00:38:26.745>And, <00:38:27.046>and <00:38:27.348>then <00:38:27.649>we, <00:38:27.951>um,

726
00:38:28.395 --> 00:38:31.245
<00:38:28.395>aggregate <00:38:28.628>this <00:38:28.861>information <00:38:29.095>over <00:38:29.328>the <00:38:29.561>whole <00:38:29.795>image

727
00:38:31.245 --> 00:38:33.725
<00:38:31.245>and <00:38:31.420>say, <00:38:31.595>okay, <00:38:31.770>if <00:38:31.944>you <00:38:32.120>have <00:38:32.295>enough, <00:38:32.470>um, <00:38:32.645>components

728
00:38:33.945 --> 00:38:37.285
<00:38:33.945>or <00:38:34.178>patches <00:38:34.411>that, <00:38:34.645>that <00:38:34.878>are <00:38:35.111>uncertain, <00:38:35.345>um,

729
00:38:37.635 --> 00:38:39.245
<00:38:37.635>then <00:38:37.915>we <00:38:38.195>have <00:38:38.475>a <00:38:38.755>certain <00:38:39.035>threshold

730
00:38:39.245 --> 00:38:42.885
<00:38:39.245>and <00:38:39.372>we <00:38:39.499>say, <00:38:39.626>okay, <00:38:39.754>we, <00:38:39.881>we <00:38:40.008>want <00:38:40.135>to, <00:38:40.263>um, <00:38:40.390>record <00:38:40.517>this <00:38:40.645>image

731
00:38:42.905 --> 00:38:45.405
<00:38:42.905>and <00:38:43.105>have <00:38:43.305>this <00:38:43.505>binary <00:38:43.705>trigger <00:38:43.905>signal <00:38:44.105>for <00:38:44.305>recording.

732
00:38:46.785 --> 00:38:49.565
<00:38:46.785>Um, <00:38:46.912>so <00:38:47.039>let's <00:38:47.166>look <00:38:47.294>an, <00:38:47.421>an <00:38:47.548>example <00:38:47.675>of <00:38:47.803>this <00:38:47.930>could <00:38:48.057>look <00:38:48.185>like,

733
00:38:49.565 --> 00:38:51.805
<00:38:49.565>so <00:38:49.765>in <00:38:49.965>this <00:38:50.165>image, <00:38:50.365>uh, <00:38:50.565>or <00:38:50.765>model, <00:38:50.965>uh,

734
00:38:51.805 --> 00:38:54.765
<00:38:51.805>predicted <00:38:52.085>several <00:38:52.365>patches <00:38:52.645>as <00:38:52.924>traffic <00:38:53.205>lights

735
00:38:54.905 --> 00:38:59.085
<00:38:54.905>and, <00:38:55.032>um, <00:38:55.159>we <00:38:55.286>color <00:38:55.414>coded <00:38:55.541>here <00:38:55.668>the, <00:38:55.795>um, <00:38:55.923>soft <00:38:56.050>max <00:38:56.177>scores <00:38:56.305>by,

736
00:38:59.505 --> 00:39:02.485
<00:38:59.505>or <00:38:59.705>the, <00:38:59.905>our <00:39:00.105>component <00:39:00.305>wise <00:39:00.505>uncertainty <00:39:00.705>score, <00:39:00.905>um,

737
00:39:02.625 --> 00:39:05.925
<00:39:02.625>by <00:39:02.825>having <00:39:03.025>the <00:39:03.225>more <00:39:03.425>certain <00:39:03.625>patches <00:39:03.825>in <00:39:04.025>green

738
00:39:05.925 --> 00:39:07.325
<00:39:05.925>and <00:39:06.240>the <00:39:06.555>uncertain <00:39:06.870>in <00:39:07.185>red.

739
00:39:07.345 --> 00:39:10.685
<00:39:07.345>So <00:39:07.500>you <00:39:07.656>see <00:39:07.811>those <00:39:07.967>traffic <00:39:08.122>lights <00:39:08.278>appear <00:39:08.433>were <00:39:08.589>very <00:39:08.745>well

740
00:39:10.965 --> 00:39:13.165
<00:39:10.965>detected <00:39:11.198>with <00:39:11.431>the <00:39:11.665>high <00:39:11.898>confidence, <00:39:12.131>high <00:39:12.365>confidence.

741
00:39:13.165 --> 00:39:15.605
<00:39:13.165>Uh, <00:39:13.320>so <00:39:13.476>the <00:39:13.631>model <00:39:13.787>is <00:39:13.942>very, <00:39:14.098>um, <00:39:14.253>fine <00:39:14.409>about <00:39:14.565>this,

742
00:39:16.025 --> 00:39:19.325
<00:39:16.025>but <00:39:16.152>here, <00:39:16.279>this <00:39:16.406>one <00:39:16.534>on <00:39:16.661>the <00:39:16.788>side <00:39:16.915>is <00:39:17.043>actually, <00:39:17.170>it <00:39:17.297>is <00:39:17.425>detected

743
00:39:19.385 --> 00:39:22.485
<00:39:19.385>as <00:39:19.512>a <00:39:19.639>traffic <00:39:19.766>light, <00:39:19.894>but <00:39:20.021>it's, <00:39:20.148>um, <00:39:20.275>not <00:39:20.403>so <00:39:20.530>certain <00:39:20.657>about <00:39:20.785>it.

744
00:39:22.485 --> 00:39:24.245
<00:39:22.485>There's <00:39:22.718>some <00:39:22.951>high <00:39:23.185>uncertainty <00:39:23.418>in <00:39:23.651>this <00:39:23.885>component.

745
00:39:24.945 --> 00:39:27.325
<00:39:24.945>Um, <00:39:25.120>and <00:39:25.295>even <00:39:25.470>here <00:39:25.645>we <00:39:25.820>have <00:39:25.995>a, <00:39:26.170>a <00:39:26.345>rear

746
00:39:27.425 --> 00:39:31.165
<00:39:27.425>or <00:39:27.552>tail, <00:39:27.679>tail <00:39:27.806>light <00:39:27.934>of <00:39:28.061>a <00:39:28.188>car, <00:39:28.315>which <00:39:28.443>is <00:39:28.570>a <00:39:28.697>false <00:39:28.825>positive.

747
00:39:31.185 --> 00:39:34.405
<00:39:31.185>So <00:39:31.360>this <00:39:31.535>is <00:39:31.710>one, <00:39:31.884>um, <00:39:32.060>tra <00:39:32.235>uh, <00:39:32.410>uh, <00:39:32.585>classified

748
00:39:34.425 --> 00:39:37.845
<00:39:34.425>as <00:39:34.600>a <00:39:34.775>traffic <00:39:34.950>light, <00:39:35.125>but <00:39:35.300>within <00:39:35.475>high <00:39:35.650>uncertainty <00:39:35.825>still.

749
00:39:38.025 --> 00:39:40.085
<00:39:38.025>So <00:39:38.225>the <00:39:38.425>model <00:39:38.625>wasn't <00:39:38.825>so <00:39:39.025>sure <00:39:39.225>about <00:39:39.425>it.

750
00:39:40.275 --> 00:39:42.245
<00:39:40.275>Here <00:39:40.450>we <00:39:40.625>have <00:39:40.800>similar <00:39:40.975>examples <00:39:41.150>with <00:39:41.325>a <00:39:41.500>traffic <00:39:41.675>light,

751
00:39:42.245 --> 00:39:45.045
<00:39:42.245>it's <00:39:42.400>very <00:39:42.556>far <00:39:42.711>away, <00:39:42.867>and <00:39:43.022>some <00:39:43.178>structures <00:39:43.333>on <00:39:43.489>the <00:39:43.645>side

752
00:39:45.045 --> 00:39:46.965
<00:39:45.045>of <00:39:45.245>the <00:39:45.445>street <00:39:45.645>where <00:39:45.845>the <00:39:46.045>model <00:39:46.245>thinks <00:39:46.445>this

753
00:39:46.965 --> 00:39:48.165
<00:39:46.965>could <00:39:47.235>be <00:39:47.505>a <00:39:47.775>traffic <00:39:48.045>light.

754
00:39:48.545 --> 00:39:51.405
<00:39:48.545>So <00:39:48.672>this, <00:39:48.799>these <00:39:48.926>will <00:39:49.054>be <00:39:49.181>examples <00:39:49.308>that <00:39:49.435>we <00:39:49.563>can <00:39:49.690>now <00:39:49.817>take, <00:39:49.945>uh,

755
00:39:51.555 --> 00:39:52.845
<00:39:51.555>into <00:39:52.716>select

756
00:39:52.845 --> 00:39:55.565
<00:39:52.845>and <00:39:53.000>take <00:39:53.156>into <00:39:53.311>our <00:39:53.467>training <00:39:53.622>set <00:39:53.778>to <00:39:53.933>improve <00:39:54.089>our <00:39:54.245>model.

757
00:39:58.785 --> 00:40:01.845
<00:39:58.785>And, <00:39:58.924>uh, <00:39:59.065>what's <00:39:59.205>coming <00:39:59.345>out <00:39:59.484>of <00:39:59.625>this <00:39:59.765>is, <00:39:59.904>uh, <00:40:00.045>this <00:40:00.185>result.

758
00:40:01.945 --> 00:40:05.205
<00:40:01.945>So, <00:40:02.085>um, <00:40:02.225>just <00:40:02.365>as <00:40:02.505>a <00:40:02.645>reminder, <00:40:02.785>we <00:40:02.925>have <00:40:03.065>two <00:40:03.205>different <00:40:03.345>models.

759
00:40:05.425 --> 00:40:08.525
<00:40:05.425>One, <00:40:05.625>we <00:40:05.825>trained <00:40:06.025>on <00:40:06.225>completely <00:40:06.425>random <00:40:06.625>data <00:40:06.825>selected,

760
00:40:08.705 --> 00:40:09.805
<00:40:08.705>uh, <00:40:09.200>data <00:40:09.694>selection.

761
00:40:10.145 --> 00:40:12.765
<00:40:10.145>And <00:40:10.378>one <00:40:10.611>model <00:40:10.845>we <00:40:11.078>improved <00:40:11.311>with, <00:40:11.545>um,

762
00:40:13.025 --> 00:40:14.725
<00:40:13.025>an <00:40:13.375>uncertainty <00:40:13.725>based <00:40:14.075>data <00:40:14.425>selection.

763
00:40:15.185 --> 00:40:17.045
<00:40:15.185>And <00:40:15.360>then <00:40:15.535>we <00:40:15.710>have <00:40:15.884>two, <00:40:16.060>two <00:40:16.235>test <00:40:16.410>sets, <00:40:16.585>right?

764
00:40:17.045 --> 00:40:18.365
<00:40:17.045>The, <00:40:17.342>the <00:40:17.639>random <00:40:17.935>test <00:40:18.232>set

765
00:40:18.825 --> 00:40:21.645
<00:40:18.825>and <00:40:19.000>the <00:40:19.174>test <00:40:19.350>set <00:40:19.524>that <00:40:19.700>contains, <00:40:19.875>um, <00:40:20.049>more <00:40:20.225>cases

766
00:40:21.705 --> 00:40:22.925
<00:40:21.705>or <00:40:21.979>almost <00:40:22.254>only <00:40:22.528>chronic <00:40:22.803>cases.

767
00:40:23.545 --> 00:40:26.685
<00:40:23.545>And <00:40:23.685>we <00:40:23.825>see <00:40:23.965>that <00:40:24.105>the <00:40:24.245>random <00:40:24.385>model <00:40:24.525>is, <00:40:24.665>um, <00:40:24.805>very <00:40:24.945>good,

768
00:40:26.985 --> 00:40:30.885
<00:40:26.985>or <00:40:27.140>both <00:40:27.296>models <00:40:27.451>actually <00:40:27.607>are <00:40:27.762>quite, <00:40:27.918>uh, <00:40:28.073>similar <00:40:28.229>actually <00:40:28.385>on,

769
00:40:31.025 --> 00:40:33.565
<00:40:31.025>on <00:40:31.305>those, <00:40:31.585>um, <00:40:31.865>random <00:40:32.145>test <00:40:32.425>sets.

770
00:40:33.705 --> 00:40:37.885
<00:40:33.705>So <00:40:33.821>in, <00:40:33.938>in <00:40:34.055>the, <00:40:34.171>uh, <00:40:34.288>in <00:40:34.404>the, <00:40:34.521>in <00:40:34.638>the <00:40:34.755>common <00:40:34.871>cases, <00:40:34.988>let's <00:40:35.105>say,

771
00:40:38.305 --> 00:40:39.485
<00:40:38.305>uh, <00:40:38.570>or, <00:40:38.836>or <00:40:39.101>on <00:40:39.367>average.

772
00:40:40.465 --> 00:40:44.365
<00:40:40.465>But, <00:40:40.620>uh, <00:40:40.776>we <00:40:40.931>also <00:40:41.087>see <00:40:41.242>that <00:40:41.398>our <00:40:41.553>uncertainty <00:40:41.709>based, <00:40:41.865>um,

773
00:40:44.875 --> 00:40:48.485
<00:40:44.875>data <00:40:45.050>selection <00:40:45.225>strategy <00:40:45.400>has <00:40:45.575>improved <00:40:45.750>the <00:40:45.925>model <00:40:46.100>on <00:40:46.275>exactly

774
00:40:48.535 --> 00:40:50.565
<00:40:48.535>those <00:40:48.768>difficult <00:40:49.001>cases <00:40:49.234>on <00:40:49.468>the <00:40:49.701>corner <00:40:49.935>cases

775
00:40:50.575 --> 00:40:53.685
<00:40:50.575>where <00:40:50.730>they <00:40:50.886>have <00:40:51.041>a <00:40:51.197>similar <00:40:51.352>performance <00:40:51.508>than <00:40:51.663>on <00:40:51.819>normal <00:40:51.975>cases.

776
00:40:54.465 --> 00:40:57.845
<00:40:54.465>Uh, <00:40:54.620>whereas <00:40:54.776>the, <00:40:54.931>the <00:40:55.087>random, <00:40:55.242>uh, <00:40:55.398>lead <00:40:55.553>trained <00:40:55.709>model <00:40:55.865>has,

777
00:40:57.945 --> 00:41:01.445
<00:40:57.945>has, <00:40:58.145>uh, <00:40:58.345>failed <00:40:58.545>on <00:40:58.745>those, <00:40:58.945>um, <00:40:59.145>images <00:40:59.345>basically

778
00:41:01.825 --> 00:41:03.125
<00:41:01.825>to <00:41:02.117>detect <00:41:02.410>the <00:41:02.702>traffic <00:41:02.995>guide.

779
00:41:04.425 --> 00:41:07.085
<00:41:04.425>And <00:41:04.580>the <00:41:04.736>same <00:41:04.891>you <00:41:05.047>see <00:41:05.202>here, <00:41:05.358>basically <00:41:05.513>in <00:41:05.669>the, <00:41:05.825>uh,

780
00:41:07.085 --> 00:41:08.445
<00:41:07.085>precision <00:41:07.697>recall <00:41:08.309>curves.

781
00:41:08.585 --> 00:41:11.605
<00:41:08.585>Um, <00:41:08.785>the <00:41:08.985>three <00:41:09.185>are <00:41:09.385>kind <00:41:09.585>of <00:41:09.785>similar <00:41:09.985>and,

782
00:41:11.625 --> 00:41:15.205
<00:41:11.625>and <00:41:11.858>then <00:41:12.091>this, <00:41:12.325>this, <00:41:12.558>um, <00:41:12.791>randomly, <00:41:13.025>uh,

783
00:41:15.525 --> 00:41:18.445
<00:41:15.525>improved <00:41:15.700>models <00:41:15.875>is <00:41:16.050>a <00:41:16.225>little <00:41:16.400>bit, <00:41:16.575>um, <00:41:16.750>failing <00:41:16.925>on,

784
00:41:18.465 --> 00:41:19.765
<00:41:18.465>on <00:41:18.855>the <00:41:19.245>difficult <00:41:19.634>cases.

785
00:41:22.165 --> 00:41:25.135
<00:41:22.165>Alright, <00:41:22.320>so, <00:41:22.476>um, <00:41:22.631>as <00:41:22.787>a <00:41:22.942>conclusion, <00:41:23.098>we <00:41:23.253>can <00:41:23.409>say, <00:41:23.565>okay,

786
00:41:25.135 --> 00:41:28.055
<00:41:25.135>uncertainty <00:41:25.335>based <00:41:25.535>recording <00:41:25.735>already <00:41:25.935>with <00:41:26.135>this <00:41:26.335>naive, <00:41:26.535>um,

787
00:41:28.725 --> 00:41:33.095
<00:41:28.725>soft <00:41:28.924>max <00:41:29.125>based, <00:41:29.325>um, <00:41:29.525>uh, <00:41:29.725>uncertainty, <00:41:29.924>uh, <00:41:30.125>calculation,

788
00:41:33.865 --> 00:41:37.125
<00:41:33.865>um, <00:41:34.064>yeah, <00:41:34.265>provides <00:41:34.464>the <00:41:34.665>right <00:41:34.865>data <00:41:35.064>for <00:41:35.265>normal

789
00:41:37.625 --> 00:41:40.645
<00:41:37.625>and <00:41:37.825>also <00:41:38.025>for <00:41:38.225>the <00:41:38.425>d <00:41:38.625>difficult, <00:41:38.825>uh, <00:41:39.025>scenarios.

790
00:41:41.785 --> 00:41:46.445
<00:41:41.785>And <00:41:41.901>now, <00:41:42.018>um, <00:41:42.134>um, <00:41:42.251>I <00:41:42.368>want <00:41:42.484>to <00:41:42.601>you <00:41:42.718>to <00:41:42.835>give <00:41:42.951>it <00:41:43.068>some <00:41:43.185>thoughts.

791
00:41:46.585 --> 00:41:51.525
<00:41:46.585>Um, <00:41:46.712>maybe <00:41:46.839>we <00:41:46.966>can <00:41:47.094>go <00:41:47.221>back <00:41:47.348>to <00:41:47.475>two <00:41:47.603>months, <00:41:47.730>uh, <00:41:47.857>slides, <00:41:47.985>share

792
00:41:51.685 --> 00:41:54.485
<00:41:51.685>where, <00:41:51.918>where <00:41:52.151>we <00:41:52.384>can <00:41:52.618>present <00:41:52.851>the, <00:41:53.085>um,

793
00:41:54.675 --> 00:41:56.165
<00:41:54.675>outcome <00:41:55.010>of, <00:41:55.345>of <00:41:55.680>the <00:41:56.016>questions.

794
00:41:57.875 --> 00:42:01.535
<00:41:57.875>Um, <00:41:58.155>could <00:41:58.435>someone <00:41:58.715>switch? <00:41:58.995>Yeah. <00:41:59.275>Great.

795
00:42:01.635 --> 00:42:05.575
<00:42:01.635>So <00:42:01.790>basically <00:42:01.946>the <00:42:02.101>next <00:42:02.257>question <00:42:02.412>that <00:42:02.568>Tim, <00:42:02.723>maybe <00:42:02.879>you <00:42:03.035>can,

796
00:42:07.135 --> 00:42:10.115
<00:42:07.135>so <00:42:07.310>what <00:42:07.485>are <00:42:07.660>potential <00:42:07.835>weaknesses <00:42:08.010>in <00:42:08.185>this <00:42:08.360>presented <00:42:08.535>approach?

797
00:42:11.185 --> 00:42:15.405
<00:42:11.185>Are <00:42:11.340>there <00:42:11.496>maybe <00:42:11.651>things <00:42:11.807>you <00:42:11.962>where <00:42:12.118>we <00:42:12.273>would <00:42:12.429>say, <00:42:12.585>okay,

798
00:42:15.405 --> 00:42:18.725
<00:42:15.405>this <00:42:15.580>is <00:42:15.755>not <00:42:15.930>the <00:42:16.105>ideal <00:42:16.280>way <00:42:16.455>of <00:42:16.630>selecting <00:42:16.805>data,

799
00:42:18.775 --> 00:42:23.295
<00:42:18.775>there <00:42:18.930>might <00:42:19.086>be <00:42:19.241>better <00:42:19.397>ways <00:42:19.552>of <00:42:19.708>doing <00:42:19.863>this, <00:42:20.019>unknown <00:42:20.175>scenarios?

800
00:42:23.295 --> 00:42:27.725
<00:42:23.295>Very <00:42:23.645>good, <00:42:23.995>yes. <00:42:24.345>More <00:42:24.695>ideas.

801
00:42:31.555 --> 00:42:33.925
<00:42:31.555>Yeah, <00:42:31.694>I <00:42:31.835>mean, <00:42:31.975>the <00:42:32.115>data <00:42:32.254>set <00:42:32.395>we <00:42:32.535>used <00:42:32.674>here <00:42:32.815>is, <00:42:32.955>um,

802
00:42:34.715 --> 00:42:36.805
<00:42:34.715>it's <00:42:34.995>a <00:42:35.275>publicly <00:42:35.555>available <00:42:35.835>dataset, <00:42:36.115>right?

803
00:42:36.805 --> 00:42:37.885
<00:42:36.805>The <00:42:37.048>A <00:42:37.291>2D <00:42:37.534>two <00:42:37.777>dataset.

804
00:42:37.945 --> 00:42:41.285
<00:42:37.945>It <00:42:38.178>is <00:42:38.411>already, <00:42:38.645>uh, <00:42:38.878>kind <00:42:39.111>of <00:42:39.345>curated

805
00:42:41.545 --> 00:42:44.685
<00:42:41.545>and <00:42:41.778>well <00:42:42.011>selected <00:42:42.245>scenes <00:42:42.478>in <00:42:42.711>this <00:42:42.945>dataset.

806
00:42:44.705 --> 00:42:47.045
<00:42:44.705>So <00:42:44.985>there's <00:42:45.265>not <00:42:45.545>so <00:42:45.825>much, <00:42:46.105>uh,

807
00:42:47.045 --> 00:42:49.645
<00:42:47.045>maybe <00:42:47.185>there <00:42:47.325>are <00:42:47.465>not <00:42:47.605>too <00:42:47.745>much <00:42:47.885>corner <00:42:48.025>cases <00:42:48.165>in <00:42:48.305>that <00:42:48.445>actually,

808
00:42:50.025 --> 00:42:53.925
<00:42:50.025>so <00:42:50.165>that <00:42:50.305>the <00:42:50.445>dataset, <00:42:50.585>uh, <00:42:50.725>or <00:42:50.865>scenario <00:42:51.005>side <00:42:51.145>could <00:42:51.285>be <00:42:51.425>one.

809
00:43:03.575 --> 00:43:05.665
<00:43:03.575>Yeah. <00:43:03.714>Things <00:43:03.855>that <00:43:03.995>we <00:43:04.134>don't <00:43:04.274>have <00:43:04.415>in <00:43:04.555>the <00:43:04.694>data <00:43:04.835>at <00:43:04.975>all,

810
00:43:16.855 --> 00:43:20.675
<00:43:16.855>Aliens <00:43:16.982>is <00:43:17.109>also <00:43:17.236>very, <00:43:17.364>um, <00:43:17.491>I <00:43:17.618>think <00:43:17.745>we, <00:43:17.873>we <00:43:18.000>have <00:43:18.127>to <00:43:18.255>consider,

811
00:43:20.855 --> 00:43:22.955
<00:43:20.855>uh, <00:43:21.088>having, <00:43:21.321>we'll <00:43:21.555>talk <00:43:21.788>about <00:43:22.021>safety <00:43:22.255>later.

812
00:43:23.095 --> 00:43:27.475
<00:43:23.095>So, <00:43:23.250>uh, <00:43:23.406>maybe <00:43:23.561>this <00:43:23.717>comes <00:43:23.872>in <00:43:24.028>there, <00:43:24.183>yes, <00:43:24.339>different <00:43:24.495>kind

813
00:43:27.555 --> 00:43:31.915
<00:43:27.555>of <00:43:27.788>road <00:43:28.021>conditions, <00:43:28.254>um, <00:43:28.488>different, <00:43:28.721>um, <00:43:28.955>uh,

814
00:43:32.355 --> 00:43:34.665
<00:43:32.355>countries <00:43:32.705>and <00:43:33.055>so <00:43:33.405>on, <00:43:33.755>right?

815
00:43:34.685 --> 00:43:38.025
<00:43:34.685>The, <00:43:34.801>the <00:43:34.918>A <00:43:35.035>2D <00:43:35.151>two <00:43:35.268>data <00:43:35.384>set <00:43:35.501>is <00:43:35.618>from, <00:43:35.735>uh, <00:43:35.851>three <00:43:35.968>German <00:43:36.085>cities,

816
00:43:38.245 --> 00:43:40.625
<00:43:38.245>so <00:43:38.444>it's <00:43:38.645>not <00:43:38.845>really <00:43:39.045>representing <00:43:39.245>the <00:43:39.444>whole <00:43:39.645>world.

817
00:43:42.375 --> 00:43:46.225
<00:43:42.375>Okay. <00:43:42.550>Um, <00:43:42.725>yeah, <00:43:42.900>maybe <00:43:43.075>go <00:43:43.250>back <00:43:43.425>to <00:43:43.600>my <00:43:43.775>slides.

818
00:43:46.305 --> 00:43:48.345
<00:43:46.305>I <00:43:46.585>have <00:43:46.865>some <00:43:47.145>solution <00:43:47.424>to <00:43:47.705>that.

819
00:43:49.765 --> 00:43:54.245
<00:43:49.765>Um, <00:43:49.998>could <00:43:50.231>you <00:43:50.464>switch <00:43:50.698>to <00:43:50.931>my <00:43:51.165>print?

820
00:43:54.245 --> 00:43:57.045
<00:43:54.245>Yeah, <00:43:54.525>thanks. <00:43:54.805>Um, <00:43:55.085>so <00:43:55.365>basically, <00:43:55.645>uh,

821
00:43:57.225 --> 00:43:59.165
<00:43:57.225>we <00:43:57.458>have <00:43:57.691>several <00:43:57.924>challenges <00:43:58.158>in <00:43:58.391>this <00:43:58.625>approach.

822
00:43:59.345 --> 00:44:02.085
<00:43:59.345>Um, <00:43:59.500>of <00:43:59.656>course, <00:43:59.811>as <00:43:59.967>a <00:44:00.122>data <00:44:00.278>scientist, <00:44:00.433>we <00:44:00.589>always <00:44:00.745>want

823
00:44:02.085 --> 00:44:03.525
<00:44:02.085>to <00:44:02.344>have <00:44:02.603>more <00:44:02.862>and <00:44:03.121>better <00:44:03.381>data.

824
00:44:04.305 --> 00:44:07.645
<00:44:04.305>Um, <00:44:04.412>so <00:44:04.520>we <00:44:04.628>go <00:44:04.735>to <00:44:04.843>our <00:44:04.951>management, <00:44:05.058>say, <00:44:05.166>okay, <00:44:05.274>A <00:44:05.381>2D <00:44:05.489>two <00:44:05.597>is <00:44:05.705>nice,

825
00:44:07.665 --> 00:44:09.765
<00:44:07.665>but <00:44:07.865>give <00:44:08.065>us <00:44:08.265>the, <00:44:08.465>the, <00:44:08.665>the <00:44:08.865>full <00:44:09.065>thing.

826
00:44:10.305 --> 00:44:13.965
<00:44:10.305>Um, <00:44:10.444>but <00:44:10.585>there <00:44:10.725>are <00:44:10.865>other <00:44:11.004>things <00:44:11.145>like <00:44:11.285>how <00:44:11.424>many <00:44:11.565>data <00:44:11.705>set,

827
00:44:14.105 --> 00:44:16.605
<00:44:14.105>how <00:44:14.280>many <00:44:14.455>samples <00:44:14.630>shall <00:44:14.805>we <00:44:14.980>select <00:44:15.155>in <00:44:15.330>the <00:44:15.505>beginning

828
00:44:17.105 --> 00:44:18.205
<00:44:17.105>of <00:44:17.435>our <00:44:17.765>initial <00:44:18.095>training?

829
00:44:18.505 --> 00:44:21.885
<00:44:18.505>Or <00:44:18.705>if <00:44:18.905>you <00:44:19.105>repeat <00:44:19.305>this <00:44:19.505>several <00:44:19.705>times, <00:44:19.905>um,

830
00:44:21.985 --> 00:44:23.485
<00:44:21.985>how <00:44:22.177>often <00:44:22.370>do <00:44:22.563>we <00:44:22.756>have <00:44:22.949>to <00:44:23.142>repeat <00:44:23.335>this?

831
00:44:23.485 --> 00:44:25.725
<00:44:23.485>And <00:44:23.765>with <00:44:24.045>which <00:44:24.325>increases <00:44:24.605>of <00:44:24.885>with,

832
00:44:25.725 --> 00:44:27.365
<00:44:25.725>with <00:44:26.005>which <00:44:26.285>increments <00:44:26.565>of <00:44:26.845>the <00:44:27.125>data?

833
00:44:27.465 --> 00:44:31.365
<00:44:27.465>So <00:44:27.605>how <00:44:27.745>many, <00:44:27.885>um, <00:44:28.025>uh, <00:44:28.165>samples <00:44:28.305>we <00:44:28.445>select <00:44:28.585>in <00:44:28.725>each <00:44:28.865>iteration.

834
00:44:32.225 --> 00:44:36.205
<00:44:32.225>Uh, <00:44:32.380>then <00:44:32.536>it's <00:44:32.691>still <00:44:32.847>a <00:44:33.002>challenge <00:44:33.158>of <00:44:33.313>evaluating <00:44:33.469>this <00:44:33.625>properly

835
00:44:36.205 --> 00:44:40.485
<00:44:36.205>because <00:44:36.345>as <00:44:36.485>you <00:44:36.625>saw, <00:44:36.765>if <00:44:36.904>we <00:44:37.045>randomly, <00:44:37.185>um, <00:44:37.325>select <00:44:37.465>data <00:44:37.605>to,

836
00:44:40.505 --> 00:44:42.965
<00:44:40.505>for <00:44:40.680>the <00:44:40.855>evaluation <00:44:41.030>set <00:44:41.205>for <00:44:41.380>the <00:44:41.555>test <00:44:41.730>set <00:44:41.905>here,

837
00:44:43.625 --> 00:44:45.885
<00:44:43.625>we <00:44:43.858>don't <00:44:44.091>see <00:44:44.325>a <00:44:44.558>big <00:44:44.791>improvement, <00:44:45.025>uh,

838
00:44:45.885 --> 00:44:47.365
<00:44:45.885>maybe <00:44:46.218>only <00:44:46.551>very <00:44:46.884>small <00:44:47.216>ones,

839
00:44:47.365 --> 00:44:51.645
<00:44:47.365>but <00:44:47.520>in <00:44:47.676>the, <00:44:47.831>in <00:44:47.987>the <00:44:48.142>very <00:44:48.298>corner, <00:44:48.453>Casey <00:44:48.609>difficult <00:44:48.765>scenes,

840
00:44:51.705 --> 00:44:53.085
<00:44:51.705>we <00:44:52.015>see <00:44:52.326>a <00:44:52.636>large <00:44:52.947>improvements.

841
00:44:53.455 --> 00:44:57.045
<00:44:53.455>Maybe <00:44:53.654>we <00:44:53.855>need <00:44:54.055>to <00:44:54.255>have, <00:44:54.455>um, <00:44:54.654>better, <00:44:54.855>um,

842
00:44:57.195 --> 00:44:59.125
<00:44:57.195>data <00:44:57.661>selecting <00:44:58.128>strategy <00:44:58.595>here.

843
00:44:59.625 --> 00:45:02.005
<00:44:59.625>And <00:44:59.858>also, <00:45:00.091>um, <00:45:00.325>the <00:45:00.558>model, <00:45:00.791>uh, <00:45:01.025>the,

844
00:45:02.005 --> 00:45:03.565
<00:45:02.005>the <00:45:02.285>whole <00:45:02.565>thing <00:45:02.845>is <00:45:03.125>biased <00:45:03.405>towards

845
00:45:03.565 --> 00:45:04.765
<00:45:03.565>this <00:45:03.925>specific <00:45:04.285>model, <00:45:04.645>right?

846
00:45:04.765 --> 00:45:08.805
<00:45:04.765>We <00:45:04.904>don't <00:45:05.045>collect <00:45:05.185>data <00:45:05.325>for <00:45:05.464>the <00:45:05.605>next, <00:45:05.745>um, <00:45:05.884>I <00:45:06.025>don't <00:45:06.165>know,

847
00:45:08.805 --> 00:45:12.405
<00:45:08.805>pedestrian <00:45:08.944>detector <00:45:09.085>because <00:45:09.225>this <00:45:09.365>one <00:45:09.504>is <00:45:09.645>a, <00:45:09.785>um, <00:45:09.924>a <00:45:10.065>new <00:45:10.205>network

848
00:45:12.545 --> 00:45:14.365
<00:45:12.545>for <00:45:12.895>only, <00:45:13.245>uh, <00:45:13.595>traffic <00:45:13.945>lights.

849
00:45:14.465 --> 00:45:17.925
<00:45:14.465>And <00:45:14.640>so <00:45:14.815>it's <00:45:14.990>a <00:45:15.165>bit <00:45:15.340>difficult, <00:45:15.515>um, <00:45:15.690>for, <00:45:15.865>for,

850
00:45:17.955 --> 00:45:19.605
<00:45:17.955>it's <00:45:18.235>not <00:45:18.515>a <00:45:18.795>general <00:45:19.075>data <00:45:19.355>selector.

851
00:45:19.745 --> 00:45:22.085
<00:45:19.745>So, <00:45:19.944>um, <00:45:20.145>also <00:45:20.345>what <00:45:20.545>I <00:45:20.745>didn't <00:45:20.944>tell <00:45:21.145>you,

852
00:45:22.105 --> 00:45:24.325
<00:45:22.105>we <00:45:22.338>didn't <00:45:22.571>look <00:45:22.805>here <00:45:23.038>at <00:45:23.271>false <00:45:23.505>negatives

853
00:45:24.325 --> 00:45:26.925
<00:45:24.325>because <00:45:24.500>we <00:45:24.674>only <00:45:24.850>look <00:45:25.024>at <00:45:25.200>the, <00:45:25.375>the <00:45:25.549>positive <00:45:25.725>detections.

854
00:45:27.185 --> 00:45:28.845
<00:45:27.185>So <00:45:27.535>that's <00:45:27.884>a <00:45:28.235>problem. <00:45:28.585>Yeah.

855
00:45:28.865 --> 00:45:32.005
<00:45:28.865>So <00:45:29.004>there <00:45:29.145>are <00:45:29.285>a <00:45:29.424>few <00:45:29.564>more, <00:45:29.705>um, <00:45:29.845>actually <00:45:29.984>that's, <00:45:30.125>um, <00:45:30.265>the,

856
00:45:32.025 --> 00:45:33.925
<00:45:32.025>the <00:45:32.258>next <00:45:32.491>question, <00:45:32.725>but <00:45:32.958>I <00:45:33.191>think, <00:45:33.425>uh,

857
00:45:33.925 --> 00:45:35.245
<00:45:33.925>Fabian, <00:45:34.162>we <00:45:34.400>are <00:45:34.637>running <00:45:34.875>out <00:45:35.113>time.

858
00:45:35.245 --> 00:45:37.445
<00:45:35.245>Maybe <00:45:35.444>we <00:45:35.645>should <00:45:35.845>switch <00:45:36.045>to, <00:45:36.245>to <00:45:36.444>your <00:45:36.645>Yes.

859
00:45:37.905 --> 00:45:41.925
<00:45:37.905>Um, <00:45:38.138>next, <00:45:38.371>uh, <00:45:38.605>section, <00:45:38.838>so <00:45:39.071>thank <00:45:39.305>you.

860
00:45:42.545 --> 00:45:45.085
<00:45:42.545>Yes, <00:45:42.745>yes, <00:45:42.945>yes. <00:45:43.145>So <00:45:43.345>let <00:45:43.545>me <00:45:43.745>briefly, <00:45:43.945>uh,

861
00:45:45.435 --> 00:45:48.165
<00:45:45.435>give <00:45:45.634>you <00:45:45.835>an <00:45:46.035>introduction <00:45:46.235>to, <00:45:46.435>uh, <00:45:46.634>AI <00:45:46.835>safety.

862
00:45:48.225 --> 00:45:51.965
<00:45:48.225>That's <00:45:48.400>basically, <00:45:48.575>uh, <00:45:48.750>my <00:45:48.924>major <00:45:49.100>topic <00:45:49.275>as <00:45:49.450>introduced <00:45:49.625>earlier.

863
00:45:52.545 --> 00:45:55.325
<00:45:52.545>So <00:45:52.778>there, <00:45:53.011>the <00:45:53.245>main <00:45:53.478>question <00:45:53.711>is <00:45:53.945>like,

864
00:45:55.345 --> 00:45:59.365
<00:45:55.345>if <00:45:55.484>we <00:45:55.625>put <00:45:55.765>the <00:45:55.904>deep <00:45:56.044>neural <00:45:56.185>networks <00:45:56.325>into <00:45:56.464>the <00:45:56.605>car, <00:45:56.745>like

865
00:45:59.865 --> 00:46:02.685
<00:45:59.865>how <00:46:00.040>can <00:46:00.214>we <00:46:00.390>actually <00:46:00.564>make <00:46:00.740>that <00:46:00.915>ver <00:46:01.089>verifiable, <00:46:01.265>right?

866
00:46:02.685 --> 00:46:05.725
<00:46:02.685>Because <00:46:02.860>we <00:46:03.035>know <00:46:03.210>that, <00:46:03.384>uh, <00:46:03.560>deep <00:46:03.735>neural <00:46:03.910>networks, <00:46:04.085>uh,

867
00:46:05.985 --> 00:46:08.965
<00:46:05.985>is <00:46:06.185>usually <00:46:06.385>they're <00:46:06.585>produced <00:46:06.785>just <00:46:06.985>throwing <00:46:07.185>some <00:46:07.385>data

868
00:46:09.825 --> 00:46:13.165
<00:46:09.825>and <00:46:10.024>having <00:46:10.225>some, <00:46:10.424>uh, <00:46:10.625>maybe <00:46:10.825>intelligent <00:46:11.024>loss <00:46:11.225>functions

869
00:46:13.305 --> 00:46:14.365
<00:46:13.305>to <00:46:13.543>fit <00:46:13.781>on <00:46:14.020>that <00:46:14.259>data.

870
00:46:14.985 --> 00:46:19.365
<00:46:14.985>But, <00:46:15.125>uh, <00:46:15.265>we <00:46:15.405>all <00:46:15.545>seen <00:46:15.685>that, <00:46:15.825>like <00:46:15.965>when <00:46:16.105>you <00:46:16.245>encounter <00:46:16.385>situation

871
00:46:19.365 --> 00:46:20.885
<00:46:19.365>that <00:46:19.707>has <00:46:20.049>not <00:46:20.391>been <00:46:20.733>seen

872
00:46:20.885 --> 00:46:25.365
<00:46:20.885>before, <00:46:21.040>like <00:46:21.196>in <00:46:21.351>this, <00:46:21.507>uh, <00:46:21.662>image <00:46:21.818>here, <00:46:21.973>the <00:46:22.129>car <00:46:22.285>with,

873
00:46:25.385 --> 00:46:29.925
<00:46:25.385>or <00:46:25.501>the <00:46:25.618>sun, <00:46:25.735>like <00:46:25.851>with <00:46:25.968>a <00:46:26.085>big, <00:46:26.201>uh, <00:46:26.318>big, <00:46:26.435>uh, <00:46:26.551>shadows <00:46:26.668>here, <00:46:26.785>uh,

874
00:46:29.945 --> 00:46:31.165
<00:46:29.945>it <00:46:30.219>can <00:46:30.494>be <00:46:30.768>difficult, <00:46:31.043>right?

875
00:46:31.425 --> 00:46:34.525
<00:46:31.425>And, <00:46:31.775>um, <00:46:32.125>therefore <00:46:32.475>we, <00:46:32.825>uh,

876
00:46:34.785 --> 00:46:38.805
<00:46:34.785>we <00:46:34.960>are <00:46:35.134>currently <00:46:35.310>working <00:46:35.484>on <00:46:35.660>like, <00:46:35.835>uh, <00:46:36.009>approaches <00:46:36.185>to,

877
00:46:39.345 --> 00:46:42.925
<00:46:39.345>to <00:46:39.500>actually <00:46:39.656>verify <00:46:39.811>deep <00:46:39.967>neural <00:46:40.122>networks <00:46:40.278>when <00:46:40.433>we <00:46:40.589>put <00:46:40.745>them

878
00:46:43.235 --> 00:46:47.765
<00:46:43.235>into, <00:46:43.435>actually, <00:46:43.635>into <00:46:43.835>the, <00:46:44.035>the <00:46:44.235>reliable <00:46:44.435>part, <00:46:44.635>um,

879
00:46:48.425 --> 00:46:49.525
<00:46:48.425>for <00:46:48.920>autonomous <00:46:49.415>driving.

880
00:46:49.585 --> 00:46:52.965
<00:46:49.585>So <00:46:49.740>when <00:46:49.896>there's <00:46:50.051>actually <00:46:50.207>no <00:46:50.362>driver, <00:46:50.518>uh, <00:46:50.673>in <00:46:50.829>the <00:46:50.985>loop.

881
00:46:53.145 --> 00:46:56.965
<00:46:53.145>Um, <00:46:53.320>and <00:46:53.495>then <00:46:53.670>the, <00:46:53.845>the <00:46:54.020>autonomous, <00:46:54.195>uh, <00:46:54.370>car <00:46:54.545>has

882
00:46:56.965 --> 00:46:58.685
<00:46:56.965>to <00:46:57.245>basically <00:46:57.525>take <00:46:57.805>care <00:46:58.085>of <00:46:58.365>everything.

883
00:46:58.745 --> 00:47:01.045
<00:46:58.745>So <00:46:58.872>it <00:46:58.999>has <00:46:59.126>to <00:46:59.254>find <00:46:59.381>all <00:46:59.508>the <00:46:59.635>pedestrians <00:46:59.763>if <00:46:59.890>we <00:47:00.017>look <00:47:00.145>here

884
00:47:01.045 --> 00:47:02.165
<00:47:01.045>on <00:47:01.297>the <00:47:01.549>right <00:47:01.801>hand <00:47:02.053>side.

885
00:47:02.945 --> 00:47:07.525
<00:47:02.945>Um, <00:47:03.225>so <00:47:03.505>I'm, <00:47:03.785>I'm <00:47:04.065>sure <00:47:04.345>you,

886
00:47:07.525 --> 00:47:10.085
<00:47:07.525>you <00:47:07.758>probably <00:47:07.991>see, <00:47:08.225>uh, <00:47:08.458>those <00:47:08.691>two <00:47:08.925>pedestrians,

887
00:47:10.185 --> 00:47:14.925
<00:47:10.185>but <00:47:10.384>if <00:47:10.585>that's <00:47:10.785>like <00:47:10.985>something, <00:47:11.185>um, <00:47:11.384>uh, <00:47:11.585>that's,

888
00:47:15.305 --> 00:47:17.365
<00:47:15.305>uh, <00:47:15.480>not <00:47:15.654>obvious <00:47:15.830>is <00:47:16.004>there <00:47:16.180>is <00:47:16.355>another <00:47:16.529>one <00:47:16.705>there.

889
00:47:18.045 --> 00:47:21.325
<00:47:18.045>I <00:47:18.220>think <00:47:18.395>one <00:47:18.570>can <00:47:18.745>imagine <00:47:18.920>many, <00:47:19.095>uh, <00:47:19.270>many <00:47:19.445>situations

890
00:47:21.325 --> 00:47:22.365
<00:47:21.325>where <00:47:21.636>something <00:47:21.948>like <00:47:22.261>that.

891
00:47:22.545 --> 00:47:26.165
<00:47:22.545>And, <00:47:22.745>and <00:47:22.945>if <00:47:23.145>we <00:47:23.345>talk <00:47:23.545>about, <00:47:23.745>um, <00:47:23.945>pedestrians

892
00:47:26.425 --> 00:47:30.765
<00:47:26.425>and <00:47:26.600>vulnerable, <00:47:26.775>uh, <00:47:26.950>objects, <00:47:27.125>uh, <00:47:27.300>uh, <00:47:27.475>on <00:47:27.650>the <00:47:27.825>road,

893
00:47:31.025 --> 00:47:35.005
<00:47:31.025>that's <00:47:31.200>for <00:47:31.375>sure, <00:47:31.550>um, <00:47:31.725>part <00:47:31.900>of <00:47:32.075>the <00:47:32.250>safety <00:47:32.425>side.

894
00:47:35.705 --> 00:47:39.685
<00:47:35.705>And, <00:47:35.845>um, <00:47:35.985>what <00:47:36.125>we've <00:47:36.265>seen <00:47:36.404>earlier <00:47:36.545>is, <00:47:36.685>is, <00:47:36.825>uh, <00:47:36.965>is <00:47:37.105>uncertainty,

895
00:47:39.985 --> 00:47:41.205
<00:47:39.985>uh, <00:47:40.534>uh, <00:47:41.083>problem.

896
00:47:41.425 --> 00:47:46.085
<00:47:41.425>So <00:47:41.541>what, <00:47:41.658>what <00:47:41.775>how <00:47:41.891>we <00:47:42.008>call <00:47:42.125>that, <00:47:42.241>um, <00:47:42.358>from, <00:47:42.475>from <00:47:42.591>the <00:47:42.708>DNN <00:47:42.825>and

897
00:47:46.085 --> 00:47:47.765
<00:47:46.085>and <00:47:46.365>safety <00:47:46.645>side, <00:47:46.925>we <00:47:47.205>say, <00:47:47.485>well,

898
00:47:48.225 --> 00:47:50.925
<00:47:48.225>we <00:47:48.505>have <00:47:48.785>some <00:47:49.065>unreliable <00:47:49.345>confidence <00:47:49.625>information.

899
00:47:50.985 --> 00:47:53.125
<00:47:50.985>So <00:47:51.140>the <00:47:51.296>DNN <00:47:51.451>doesn't <00:47:51.607>really <00:47:51.762>tell <00:47:51.918>us <00:47:52.073>when <00:47:52.229>it's <00:47:52.385>wrong.

900
00:47:53.265 --> 00:47:56.205
<00:47:53.265>So <00:47:53.420>it <00:47:53.576>says, <00:47:53.731>well, <00:47:53.887>I <00:47:54.042>have <00:47:54.198>those <00:47:54.353>two <00:47:54.509>detections <00:47:54.665>here,

901
00:47:56.205 --> 00:47:57.365
<00:47:56.205>this <00:47:56.466>one <00:47:56.727>and <00:47:56.988>this <00:47:57.249>one.

902
00:47:58.185 --> 00:48:00.765
<00:47:58.185>And, <00:47:58.340>uh, <00:47:58.496>it <00:47:58.651>says, <00:47:58.807>well, <00:47:58.962>I'm <00:47:59.118>sure <00:47:59.273>there's <00:47:59.429>a <00:47:59.585>pedestrian,

903
00:48:00.765 --> 00:48:03.045
<00:48:00.765>it's <00:48:00.964>0.7 <00:48:01.165>confidence, <00:48:01.365>but <00:48:01.565>there's <00:48:01.765>actually <00:48:01.964>no <00:48:02.165>pedestrian.

904
00:48:03.345 --> 00:48:05.605
<00:48:03.345>Um, <00:48:03.694>that <00:48:04.044>would, <00:48:04.395>would <00:48:04.745>lead

905
00:48:05.605 --> 00:48:08.045
<00:48:05.605>to <00:48:05.760>the <00:48:05.916>car <00:48:06.071>stopping <00:48:06.227>all <00:48:06.382>the <00:48:06.538>time <00:48:06.693>and <00:48:06.849>so <00:48:07.005>on.

906
00:48:08.065 --> 00:48:10.365
<00:48:08.065>And, <00:48:08.220>but <00:48:08.376>also <00:48:08.531>it, <00:48:08.687>it <00:48:08.842>won't <00:48:08.998>find <00:48:09.153>this <00:48:09.309>one <00:48:09.465>here.

907
00:48:10.505 --> 00:48:12.325
<00:48:10.505>So, <00:48:10.855>and, <00:48:11.205>and <00:48:11.555>it's, <00:48:11.905>uh,

908
00:48:12.585 --> 00:48:14.485
<00:48:12.585>and <00:48:12.760>it <00:48:12.935>doesn't <00:48:13.110>really <00:48:13.285>tell <00:48:13.460>us <00:48:13.635>that, <00:48:13.810>that <00:48:13.985>it's,

909
00:48:14.795 --> 00:48:16.605
<00:48:14.795>it's <00:48:15.261>maybe <00:48:15.728>wrong, <00:48:16.195>right?

910
00:48:16.785 --> 00:48:19.765
<00:48:16.785>So <00:48:16.912>that's <00:48:17.039>why <00:48:17.166>if <00:48:17.294>we <00:48:17.421>put <00:48:17.548>up <00:48:17.675>the <00:48:17.803>scheme, <00:48:17.930>we <00:48:18.057>say, <00:48:18.185>well,

911
00:48:19.995 --> 00:48:22.045
<00:48:19.995>with <00:48:20.170>DNN <00:48:20.345>when <00:48:20.520>we <00:48:20.694>use <00:48:20.870>them, <00:48:21.045>we <00:48:21.220>have <00:48:21.395>to,

912
00:48:22.345 --> 00:48:23.485
<00:48:22.345>uh, <00:48:22.687>be <00:48:23.029>careful, <00:48:23.371>right?

913
00:48:23.485 --> 00:48:27.565
<00:48:23.485>Because <00:48:23.660>we <00:48:23.835>have <00:48:24.010>some <00:48:24.185>specific <00:48:24.360>concerns, <00:48:24.535>uh, <00:48:24.710>that <00:48:24.885>occur.

914
00:48:27.865 --> 00:48:30.805
<00:48:27.865>Uh, <00:48:28.040>and <00:48:28.214>they, <00:48:28.390>they're <00:48:28.564>all <00:48:28.740>kind <00:48:28.915>of <00:48:29.089>related <00:48:29.265>to

915
00:48:30.995 --> 00:48:34.165
<00:48:30.995>what <00:48:31.275>we <00:48:31.555>call <00:48:31.835>insufficient <00:48:32.115>generalization <00:48:32.395>capabilities.

916
00:48:34.545 --> 00:48:37.965
<00:48:34.545>So, <00:48:34.700>uh, <00:48:34.856>uh, <00:48:35.011>we <00:48:35.167>know <00:48:35.322>sometimes <00:48:35.478>that <00:48:35.633>the <00:48:35.789>DNS <00:48:35.945>are,

917
00:48:37.965 --> 00:48:41.725
<00:48:37.965>are <00:48:38.140>super <00:48:38.315>human, <00:48:38.490>but <00:48:38.665>um, <00:48:38.840>maybe <00:48:39.015>they, <00:48:39.190>they, <00:48:39.365>they,

918
00:48:42.315 --> 00:48:44.565
<00:48:42.315>they <00:48:42.515>don't <00:48:42.715>generalize <00:48:42.915>as <00:48:43.115>well <00:48:43.315>as <00:48:43.515>humans, <00:48:43.715>right?

919
00:48:44.565 --> 00:48:48.365
<00:48:44.565>They, <00:48:44.720>they <00:48:44.876>haven't <00:48:45.031>seen <00:48:45.187>from <00:48:45.342>a <00:48:45.498>kid's <00:48:45.653>perspective <00:48:45.809>like <00:48:45.965>all

920
00:48:48.365 --> 00:48:50.565
<00:48:48.365>different <00:48:48.564>kinds <00:48:48.765>of <00:48:48.964>how, <00:48:49.165>how <00:48:49.365>humans <00:48:49.564>look <00:48:49.765>like

921
00:48:50.665 --> 00:48:52.165
<00:48:50.665>and <00:48:51.002>carnival <00:48:51.340>and <00:48:51.677>so <00:48:52.015>on.

922
00:48:52.305 --> 00:48:56.005
<00:48:52.305>And <00:48:52.460>they, <00:48:52.616>they <00:48:52.771>won't, <00:48:52.927>won't <00:48:53.082>find, <00:48:53.238>uh, <00:48:53.393>all <00:48:53.549>the <00:48:53.705>pedestrians.

923
00:48:56.005 --> 00:48:59.005
<00:48:56.005>So <00:48:56.160>we <00:48:56.316>might <00:48:56.471>have <00:48:56.627>earners <00:48:56.782>outputs <00:48:56.938>in <00:48:57.093>there, <00:48:57.249>for <00:48:57.405>example,

924
00:48:59.145 --> 00:49:02.725
<00:48:59.145>if <00:48:59.320>we <00:48:59.495>know <00:48:59.670>that, <00:48:59.845>uh, <00:49:00.020>we, <00:49:00.195>we <00:49:00.370>make <00:49:00.545>errors

925
00:49:02.745 --> 00:49:05.485
<00:49:02.745>and <00:49:02.920>we, <00:49:03.095>we, <00:49:03.270>we <00:49:03.444>don't <00:49:03.620>actually <00:49:03.795>detect <00:49:03.970>it, <00:49:04.145>uh,

926
00:49:05.485 --> 00:49:07.925
<00:49:05.485>because <00:49:05.660>we <00:49:05.835>don't <00:49:06.010>have <00:49:06.185>a, <00:49:06.360>a <00:49:06.535>good <00:49:06.710>confidence <00:49:06.885>information,

927
00:49:08.505 --> 00:49:10.045
<00:49:08.505>we <00:49:08.782>have <00:49:09.059>to, <00:49:09.336>uh, <00:49:09.613>tackle <00:49:09.891>this.

928
00:49:10.045 --> 00:49:11.965
<00:49:10.045>So <00:49:10.278>that's <00:49:10.511>what <00:49:10.745>we <00:49:10.978>call <00:49:11.211>the <00:49:11.445>measures.

929
00:49:13.065 --> 00:49:16.045
<00:49:13.065>And <00:49:13.298>uh, <00:49:13.531>what <00:49:13.765>we <00:49:13.998>can, <00:49:14.231>for <00:49:14.465>example,

930
00:49:16.105 --> 00:49:19.645
<00:49:16.105>do <00:49:16.221>is <00:49:16.338>we <00:49:16.455>don't <00:49:16.571>only <00:49:16.688>have <00:49:16.805>the <00:49:16.921>DNN, <00:49:17.038>uh, <00:49:17.155>working <00:49:17.271>on <00:49:17.388>the <00:49:17.505>image

931
00:49:19.945 --> 00:49:22.405
<00:49:19.945>and <00:49:20.178>giving <00:49:20.411>it <00:49:20.645>us <00:49:20.878>then <00:49:21.111>later <00:49:21.345>on

932
00:49:22.405 --> 00:49:24.485
<00:49:22.405>to <00:49:22.638>the <00:49:22.871>autonomous <00:49:23.105>driving <00:49:23.338>stack, <00:49:23.571>uh, <00:49:23.805>the,

933
00:49:24.665 --> 00:49:27.725
<00:49:24.665>the <00:49:24.898>attention <00:49:25.131>prediction, <00:49:25.365>uh, <00:49:25.598>planning <00:49:25.831>or <00:49:26.065>driving.

934
00:49:28.385 --> 00:49:30.365
<00:49:28.385>Uh, <00:49:28.665>we <00:49:28.945>also <00:49:29.225>put <00:49:29.505>this <00:49:29.785>uncertainty

935
00:49:30.365 --> 00:49:31.565
<00:49:30.365>monitor <00:49:30.634>as <00:49:30.904>we've <00:49:31.175>seen <00:49:31.445>earlier.

936
00:49:32.345 --> 00:49:36.365
<00:49:32.345>And <00:49:32.472>when <00:49:32.599>we, <00:49:32.726>when <00:49:32.854>we <00:49:32.981>see <00:49:33.108>a <00:49:33.235>high <00:49:33.363>uncertainty <00:49:33.490>for <00:49:33.617>the <00:49:33.745>given

937
00:49:36.885 --> 00:49:40.445
<00:49:36.885>situation, <00:49:37.118>we <00:49:37.351>basically <00:49:37.585>put <00:49:37.818>actions <00:49:38.051>in <00:49:38.285>the,

938
00:49:40.445 --> 00:49:43.485
<00:49:40.445>in <00:49:40.585>the <00:49:40.725>later <00:49:40.865>part <00:49:41.005>of <00:49:41.145>the, <00:49:41.285>of <00:49:41.425>the <00:49:41.565>actual <00:49:41.705>system, <00:49:41.845>right?

939
00:49:43.625 --> 00:49:46.365
<00:49:43.625>So <00:49:43.765>we <00:49:43.905>can, <00:49:44.045>uh, <00:49:44.185>go <00:49:44.325>to <00:49:44.465>a <00:49:44.605>safe <00:49:44.745>stop, <00:49:44.885>for <00:49:45.025>example,

940
00:49:46.425 --> 00:49:49.725
<00:49:46.425>or <00:49:46.580>go <00:49:46.736>to <00:49:46.891>a <00:49:47.047>more <00:49:47.202>defensive <00:49:47.358>driving <00:49:47.513>strategy <00:49:47.669>like, <00:49:47.825>uh,

941
00:49:50.555 --> 00:49:51.725
<00:49:50.555>growing <00:49:51.081>the <00:49:51.607>speed.

942
00:49:52.345 --> 00:49:55.965
<00:49:52.345>Um, <00:49:52.472>yeah, <00:49:52.599>due <00:49:52.726>to <00:49:52.854>time <00:49:52.981>I <00:49:53.108>just <00:49:53.235>wanted <00:49:53.363>to <00:49:53.490>ask <00:49:53.617>like, <00:49:53.745>uh,

943
00:49:56.645 --> 00:49:59.725
<00:49:56.645>question <00:49:56.785>maybe <00:49:56.925>we, <00:49:57.065>we <00:49:57.205>don't <00:49:57.345>do <00:49:57.485>the <00:49:57.625>full <00:49:57.765>quiz <00:49:57.905>now <00:49:58.045>due

944
00:49:59.725 --> 00:50:02.085
<00:49:59.725>to <00:49:59.958>the <00:50:00.191>time, <00:50:00.424>but <00:50:00.658>what <00:50:00.891>other <00:50:01.125>DNN

945
00:50:02.085 --> 00:50:03.765
<00:50:02.085>insufficiencies <00:50:02.551>could <00:50:03.018>be <00:50:03.485>there?

946
00:50:04.465 --> 00:50:07.525
<00:50:04.465>Um, <00:50:04.605>but <00:50:04.745>due <00:50:04.885>to <00:50:05.025>time, <00:50:05.165>uh, <00:50:05.305>i, <00:50:05.445>I <00:50:05.585>come <00:50:05.725>with, <00:50:05.865>uh,

947
00:50:07.525 --> 00:50:09.405
<00:50:07.525>with <00:50:07.805>some <00:50:08.085>results <00:50:08.365>here, <00:50:08.645>uh, <00:50:08.925>quickly.

948
00:50:10.025 --> 00:50:11.405
<00:50:10.025>So <00:50:10.273>there, <00:50:10.521>there <00:50:10.770>is <00:50:11.018>many, <00:50:11.267>uh,

949
00:50:11.405 --> 00:50:14.525
<00:50:11.405>like <00:50:11.605>the <00:50:11.805>main <00:50:12.005>insufficiency <00:50:12.205>we <00:50:12.405>say <00:50:12.605>is <00:50:12.805>generalization

950
00:50:14.525 --> 00:50:17.325
<00:50:14.525>capability, <00:50:14.875>or <00:50:15.225>insufficient <00:50:15.575>generalization <00:50:15.925>capability.

951
00:50:18.105 --> 00:50:21.605
<00:50:18.105>And <00:50:18.280>that <00:50:18.455>can, <00:50:18.630>uh, <00:50:18.805>come <00:50:18.980>from <00:50:19.155>like <00:50:19.330>DNN <00:50:19.505>characteristics.

952
00:50:21.825 --> 00:50:23.805
<00:50:21.825>Uh, <00:50:22.000>as <00:50:22.174>I <00:50:22.350>said, <00:50:22.524>it's <00:50:22.700>all <00:50:22.875>just <00:50:23.049>this <00:50:23.225>fitting.

953
00:50:24.185 --> 00:50:27.680
<00:50:24.185>Uh, <00:50:24.384>therefore <00:50:24.585>we <00:50:24.785>get <00:50:24.985>this <00:50:25.185>unreliable <00:50:25.384>confidence <00:50:25.585>information.

954
00:50:27.795 --> 00:50:29.445
<00:50:27.795>Also, <00:50:28.145>we <00:50:28.495>have <00:50:28.845>sub <00:50:29.195>brittleness.

955
00:50:29.585 --> 00:50:32.925
<00:50:29.585>So <00:50:29.725>if <00:50:29.865>you <00:50:30.005>slightly <00:50:30.145>change <00:50:30.285>something <00:50:30.425>due <00:50:30.565>to <00:50:30.705>rain <00:50:30.845>or <00:50:30.985>fork

956
00:50:32.945 --> 00:50:35.525
<00:50:32.945>or <00:50:33.178>something <00:50:33.411>like <00:50:33.645>that, <00:50:33.878>your, <00:50:34.111>your, <00:50:34.345>your,

957
00:50:35.595 --> 00:50:40.045
<00:50:35.595>your <00:50:35.750>whole <00:50:35.906>performance <00:50:36.061>can <00:50:36.217>deteriorate, <00:50:36.372>um, <00:50:36.528>for <00:50:36.683>the <00:50:36.839>DNN <00:50:36.995>or,

958
00:50:40.065 --> 00:50:41.845
<00:50:40.065>or <00:50:40.265>they, <00:50:40.465>they <00:50:40.665>usually <00:50:40.865>look <00:50:41.065>at <00:50:41.265>single <00:50:41.465>frames

959
00:50:41.845 --> 00:50:44.965
<00:50:41.845>or <00:50:42.000>they, <00:50:42.156>they <00:50:42.311>don't, <00:50:42.467>don't <00:50:42.622>really <00:50:42.778>find, <00:50:42.933>uh, <00:50:43.089>uh, <00:50:43.245>stuff

960
00:50:45.205 --> 00:50:47.925
<00:50:45.205>that <00:50:45.404>moves, <00:50:45.605>uh, <00:50:45.805>quickly <00:50:46.005>or <00:50:46.205>don't <00:50:46.404>follow <00:50:46.605>objects.

961
00:50:48.625 --> 00:50:49.965
<00:50:48.625>Uh, <00:50:49.228>they're <00:50:49.831>incomprehensible.

962
00:50:50.085 --> 00:50:51.925
<00:50:50.085>I <00:50:50.260>mean, <00:50:50.435>that, <00:50:50.610>that's <00:50:50.785>not, <00:50:50.960>not <00:50:51.135>a <00:50:51.310>blocker, <00:50:51.485>right?

963
00:50:52.865 --> 00:50:55.125
<00:50:52.865>We <00:50:53.214>know <00:50:53.564>they're <00:50:53.915>superhuman <00:50:54.265>sometimes,

964
00:50:55.265 --> 00:50:58.165
<00:50:55.265>but, <00:50:55.498>uh, <00:50:55.731>we, <00:50:55.964>we <00:50:56.198>can't <00:50:56.431>explain <00:50:56.665>like

965
00:50:58.185 --> 00:50:59.845
<00:50:58.185>how <00:50:58.418>they <00:50:58.651>actually, <00:50:58.884>uh, <00:50:59.118>do <00:50:59.351>the <00:50:59.585>decision.

966
00:51:00.585 --> 00:51:03.285
<00:51:00.585>And <00:51:00.785>we <00:51:00.985>don't <00:51:01.185>have <00:51:01.385>any <00:51:01.585>like <00:51:01.785>plausibility <00:51:01.985>check,

967
00:51:03.435 --> 00:51:05.005
<00:51:03.435>like <00:51:04.835>currently.

968
00:51:05.285 --> 00:51:06.605
<00:51:05.285>I <00:51:05.522>know <00:51:05.760>there <00:51:05.997>is <00:51:06.235>approaches, <00:51:06.473>but,

969
00:51:06.625 --> 00:51:09.205
<00:51:06.625>but <00:51:06.780>by <00:51:06.936>by <00:51:07.091>nature, <00:51:07.247>they, <00:51:07.402>they <00:51:07.558>don't <00:51:07.713>say, <00:51:07.869>well, <00:51:08.025>ah,

970
00:51:09.205 --> 00:51:11.605
<00:51:09.205>there's <00:51:09.380>a <00:51:09.555>hat <00:51:09.730>and <00:51:09.904>the <00:51:10.080>hand, <00:51:10.255>it's <00:51:10.430>a <00:51:10.605>pedestrian,

971
00:51:11.605 --> 00:51:13.085
<00:51:11.605>therefore <00:51:11.871>it <00:51:12.137>should <00:51:12.404>be <00:51:12.670>a <00:51:12.937>pedestrian.

972
00:51:13.705 --> 00:51:15.565
<00:51:13.705>And, <00:51:13.985>but <00:51:14.265>that's <00:51:14.545>just <00:51:14.825>one <00:51:15.105>part.

973
00:51:15.705 --> 00:51:18.045
<00:51:15.705>And <00:51:15.860>the <00:51:16.016>other <00:51:16.171>part <00:51:16.327>we <00:51:16.482>say <00:51:16.638>is <00:51:16.793>data <00:51:16.949>related <00:51:17.105>concern.

974
00:51:18.105 --> 00:51:20.685
<00:51:18.105>So <00:51:18.245>if <00:51:18.385>we <00:51:18.525>really, <00:51:18.665>uh, <00:51:18.805>don't <00:51:18.945>do <00:51:19.085>everything <00:51:19.225>right <00:51:19.365>with <00:51:19.505>data,

975
00:51:21.625 --> 00:51:23.645
<00:51:21.625>um, <00:51:21.905>we, <00:51:22.185>we <00:51:22.465>get <00:51:22.745>messed <00:51:23.025>up.

976
00:51:23.865 --> 00:51:28.685
<00:51:23.865>So <00:51:23.964>we, <00:51:24.064>what <00:51:24.165>we <00:51:24.265>have <00:51:24.365>to <00:51:24.464>do <00:51:24.564>is, <00:51:24.665>uh, <00:51:24.765>kind <00:51:24.865>of <00:51:24.964>have <00:51:25.064>to <00:51:25.165>show <00:51:25.265>that,

977
00:51:28.825 --> 00:51:32.045
<00:51:28.825>uh, <00:51:29.058>uh, <00:51:29.291>data <00:51:29.524>distribution <00:51:29.758>from <00:51:29.991>the <00:51:30.225>training

978
00:51:32.105 --> 00:51:35.045
<00:51:32.105>and <00:51:32.260>from <00:51:32.416>the <00:51:32.571>test <00:51:32.727>is, <00:51:32.882>is <00:51:33.038>a <00:51:33.193>good <00:51:33.349>approximation <00:51:33.505>of

979
00:51:35.045 --> 00:51:36.805
<00:51:35.045>where <00:51:35.325>the <00:51:35.605>system <00:51:35.885>is <00:51:36.165>deployed <00:51:36.445>later.

980
00:51:36.905 --> 00:51:38.125
<00:51:36.905>So <00:51:37.271>that's <00:51:37.637>2.1 <00:51:38.003>here.

981
00:51:39.025 --> 00:51:43.685
<00:51:39.025>Uh, <00:51:39.165>or <00:51:39.305>we <00:51:39.445>don't <00:51:39.585>really <00:51:39.725>carefully, <00:51:39.865>uh, <00:51:40.005>put, <00:51:40.145>uh, <00:51:40.285>um, <00:51:40.425>um,

982
00:51:44.595 --> 00:51:47.485
<00:51:44.595>test <00:51:44.750>data <00:51:44.906>that <00:51:45.061>that <00:51:45.217>has <00:51:45.372>not <00:51:45.528>been <00:51:45.683>seen <00:51:45.839>before, <00:51:45.995>right?

983
00:51:47.545 --> 00:51:50.165
<00:51:47.545>And, <00:51:47.720>and, <00:51:47.895>uh, <00:51:48.070>say, <00:51:48.245>oh, <00:51:48.420>well, <00:51:48.595>we're <00:51:48.770>good, <00:51:48.945>but,

984
00:51:50.185 --> 00:51:52.605
<00:51:50.185>but <00:51:50.325>then <00:51:50.465>we, <00:51:50.605>we, <00:51:50.745>we <00:51:50.884>don't <00:51:51.025>have <00:51:51.165>the <00:51:51.305>real <00:51:51.445>world <00:51:51.585>situation

985
00:51:52.605 --> 00:51:55.645
<00:51:52.605>there, <00:51:52.838>or <00:51:53.071>labeling <00:51:53.305>quality, <00:51:53.538>missing <00:51:53.771>labeling <00:51:54.005>details,

986
00:51:55.785 --> 00:51:57.965
<00:51:55.785>or <00:51:56.018>we <00:51:56.251>don't <00:51:56.484>specify <00:51:56.718>well <00:51:56.951>with <00:51:57.185>the

987
00:51:57.965 --> 00:51:59.445
<00:51:57.965>where <00:51:58.231>the <00:51:58.497>system <00:51:58.764>should <00:51:59.030>should <00:51:59.297>work.

988
00:52:00.225 --> 00:52:03.165
<00:52:00.225>And <00:52:00.380>we, <00:52:00.536>we <00:52:00.691>shift <00:52:00.847>over <00:52:01.002>time, <00:52:01.158>people <00:52:01.313>start <00:52:01.469>going, <00:52:01.625>uh,

989
00:52:03.535 --> 00:52:06.325
<00:52:03.535>going <00:52:03.690>out <00:52:03.846>with <00:52:04.001>this, <00:52:04.157>uh, <00:52:04.312>uh, <00:52:04.468>what <00:52:04.623>was <00:52:04.779>it <00:52:04.935>called?

990
00:52:06.535 --> 00:52:08.285
<00:52:06.535>Swiss <00:52:06.815>roller, <00:52:07.095>uh, <00:52:07.375>over <00:52:07.654>the <00:52:07.935>streets.

991
00:52:08.385 --> 00:52:09.845
<00:52:08.385>And <00:52:08.604>we, <00:52:08.823>we <00:52:09.042>have <00:52:09.261>never <00:52:09.480>seen <00:52:09.698>them.

992
00:52:10.625 --> 00:52:12.045
<00:52:10.625>And <00:52:10.944>we <00:52:11.264>don't <00:52:11.583>know, <00:52:11.903>like

993
00:52:12.045 --> 00:52:13.805
<00:52:12.045>what <00:52:12.325>happens <00:52:12.605>in <00:52:12.885>rare <00:52:13.165>critical <00:52:13.445>situations,

994
00:52:13.805 --> 00:52:15.525
<00:52:13.805>like <00:52:14.085>the <00:52:14.365>situations <00:52:14.645>that <00:52:14.924>never <00:52:15.205>occur

995
00:52:15.525 --> 00:52:16.565
<00:52:15.525>that <00:52:15.712>you've <00:52:15.899>seen <00:52:16.086>in <00:52:16.273>the <00:52:16.461>beginning.

996
00:52:18.105 --> 00:52:20.325
<00:52:18.105>Uh, <00:52:18.280>yeah. <00:52:18.455>And <00:52:18.630>we <00:52:18.805>have <00:52:18.980>to <00:52:19.155>put <00:52:19.330>right <00:52:19.505>metrics.

997
00:52:20.505 --> 00:52:24.045
<00:52:20.505>So <00:52:20.621>like <00:52:20.738>we, <00:52:20.855>we <00:52:20.971>shouldn't <00:52:21.088>just <00:52:21.205>say, <00:52:21.321>I <00:52:21.438>mean, <00:52:21.555>it's <00:52:21.671>in, <00:52:21.788>in, <00:52:21.905>uh,

998
00:52:24.195 --> 00:52:26.285
<00:52:24.195>like <00:52:24.428>a <00:52:24.661>mean, <00:52:24.895>it's <00:52:25.128>good, <00:52:25.361>uh, <00:52:25.595>enough,

999
00:52:26.785 --> 00:52:28.645
<00:52:26.785>but, <00:52:26.940>uh, <00:52:27.096>we <00:52:27.251>have <00:52:27.407>to <00:52:27.562>say <00:52:27.718>at <00:52:27.873>least <00:52:28.029>it <00:52:28.185>has

1000
00:52:28.645 --> 00:52:31.045
<00:52:28.645>to <00:52:28.995>have <00:52:29.345>some <00:52:29.695>certain <00:52:30.045>performance.

1001
00:52:31.265 --> 00:52:34.525
<00:52:31.265>Um, <00:52:31.392>so <00:52:31.519>we <00:52:31.646>have <00:52:31.774>to <00:52:31.901>come <00:52:32.028>up <00:52:32.155>with <00:52:32.283>metrics <00:52:32.410>that <00:52:32.537>actually <00:52:32.665>cover

1002
00:52:34.525 --> 00:52:37.045
<00:52:34.525>that <00:52:34.700>to, <00:52:34.875>to <00:52:35.050>argue <00:52:35.225>the <00:52:35.400>safety <00:52:35.575>of <00:52:35.750>our <00:52:35.925>system.

1003
00:52:37.985 --> 00:52:41.725
<00:52:37.985>Um, <00:52:38.185>and, <00:52:38.385>uh, <00:52:38.585>now <00:52:38.785>I <00:52:38.985>I <00:52:39.185>would <00:52:39.385>like

1004
00:52:41.725 --> 00:52:43.085
<00:52:41.725>to <00:52:41.969>briefly <00:52:42.214>jump <00:52:42.459>into <00:52:42.704>the <00:52:42.949>topic.

1005
00:52:43.285 --> 00:52:46.245
<00:52:43.285>I <00:52:43.401>know <00:52:43.518>it's, <00:52:43.634>it's <00:52:43.751>maybe <00:52:43.868>too <00:52:43.984>much <00:52:44.101>for <00:52:44.218>the <00:52:44.335>short <00:52:44.451>time <00:52:44.568>I <00:52:44.685>have,

1006
00:52:46.385 --> 00:52:49.245
<00:52:46.385>but, <00:52:46.585>um, <00:52:46.785>like <00:52:46.985>how <00:52:47.185>do <00:52:47.385>we <00:52:47.585>actually <00:52:47.785>develop

1007
00:52:49.465 --> 00:52:52.965
<00:52:49.465>and, <00:52:49.815>uh, <00:52:50.165>argument, <00:52:50.515>uh, <00:52:50.865>systems

1008
00:52:53.105 --> 00:52:55.725
<00:52:53.105>or <00:52:53.305>argument <00:52:53.505>the <00:52:53.705>safety <00:52:53.905>of <00:52:54.105>our <00:52:54.305>systems <00:52:54.505>systems?

1009
00:52:56.585 --> 00:53:00.245
<00:52:56.585>So <00:52:56.740>therefore, <00:52:56.896>like, <00:52:57.051>like <00:52:57.207>just <00:52:57.362>quick, <00:52:57.518>uh, <00:52:57.673>jump <00:52:57.829>into <00:52:57.985>like

1010
00:53:00.465 --> 00:53:03.805
<00:53:00.465>how <00:53:00.745>development, <00:53:01.025>uh, <00:53:01.305>uh, <00:53:01.585>is <00:53:01.865>working.

1011
00:53:04.065 --> 00:53:06.285
<00:53:04.065>So <00:53:04.220>first <00:53:04.376>of <00:53:04.531>all, <00:53:04.687>you <00:53:04.842>have <00:53:04.998>to <00:53:05.153>specify <00:53:05.309>the <00:53:05.465>functionality.

1012
00:53:06.395 --> 00:53:09.045
<00:53:06.395>Well, <00:53:06.570>system <00:53:06.745>has <00:53:06.920>to <00:53:07.095>find <00:53:07.270>pedestrian, <00:53:07.445>for <00:53:07.620>example, <00:53:07.795>like,

1013
00:53:09.185 --> 00:53:12.885
<00:53:09.185>but <00:53:09.340>uh, <00:53:09.496>you <00:53:09.651>won't, <00:53:09.807>uh, <00:53:09.962>in <00:53:10.118>the <00:53:10.273>open <00:53:10.429>world <00:53:10.585>scenario,

1014
00:53:12.885 --> 00:53:14.885
<00:53:12.885>like <00:53:13.085>autonomous <00:53:13.285>driving, <00:53:13.485>you <00:53:13.685>won't <00:53:13.885>be <00:53:14.085>able <00:53:14.285>to,

1015
00:53:14.945 --> 00:53:16.205
<00:53:14.945>to <00:53:15.323>fully <00:53:15.701>specify <00:53:16.078>that.

1016
00:53:16.205 --> 00:53:19.805
<00:53:16.205>It <00:53:16.321>has <00:53:16.438>to <00:53:16.555>find, <00:53:16.671>uh, <00:53:16.788>uh, <00:53:16.904>what <00:53:17.021>kind <00:53:17.138>of, <00:53:17.255>uh, <00:53:17.371>pedestrians <00:53:17.488>it <00:53:17.605>has

1017
00:53:19.805 --> 00:53:22.845
<00:53:19.805>to <00:53:19.932>find <00:53:20.059>or <00:53:20.186>so, <00:53:20.314>and <00:53:20.441>can <00:53:20.568>you <00:53:20.695>specify <00:53:20.823>where <00:53:20.950>it <00:53:21.077>should <00:53:21.205>work,

1018
00:53:22.845 --> 00:53:24.965
<00:53:22.845>should <00:53:23.000>it <00:53:23.156>work <00:53:23.311>at <00:53:23.467>night <00:53:23.622>or <00:53:23.778>rain <00:53:23.933>or <00:53:24.089>so <00:53:24.245>on?

1019
00:53:25.385 --> 00:53:27.085
<00:53:25.385>You <00:53:25.735>specify <00:53:26.085>some <00:53:26.435>system <00:53:26.785>architecture.

1020
00:53:27.085 --> 00:53:31.005
<00:53:27.085>Then <00:53:27.240>you <00:53:27.396>built, <00:53:27.551>uh, <00:53:27.707>your <00:53:27.862>AI <00:53:28.018>function <00:53:28.173>where, <00:53:28.329>uh, <00:53:28.485>data

1021
00:53:31.775 --> 00:53:35.165
<00:53:31.775>model <00:53:31.950>design, <00:53:32.125>training <00:53:32.300>and <00:53:32.475>so <00:53:32.650>on, <00:53:32.825>testing <00:53:33.000>and <00:53:33.175>maintenance.

1022
00:53:35.185 --> 00:53:37.125
<00:53:35.185>And <00:53:35.360>then <00:53:35.535>on <00:53:35.710>the <00:53:35.884>other <00:53:36.060>side, <00:53:36.235>you <00:53:36.410>basically <00:53:36.585>have

1023
00:53:37.125 --> 00:53:40.325
<00:53:37.125>to <00:53:37.265>do <00:53:37.405>a <00:53:37.545>safety <00:53:37.685>augmentation <00:53:37.825>once, <00:53:37.965>uh, <00:53:38.105>the <00:53:38.245>system <00:53:38.385>you <00:53:38.525>built,

1024
00:53:40.625 --> 00:53:42.365
<00:53:40.625>uh, <00:53:40.905>with <00:53:41.185>d <00:53:41.465>analysis, <00:53:41.745>safety <00:53:42.025>critical.

1025
00:53:43.065 --> 00:53:46.685
<00:53:43.065>So <00:53:43.205>you <00:53:43.345>have <00:53:43.485>to <00:53:43.625>identify, <00:53:43.765>uh, <00:53:43.905>how <00:53:44.045>that <00:53:44.185>risks, <00:53:44.325>um, <00:53:44.465>um,

1026
00:53:46.835 --> 00:53:49.045
<00:53:46.835>what <00:53:47.035>could <00:53:47.235>possibly <00:53:47.435>happen <00:53:47.635>from <00:53:47.835>the <00:53:48.035>system <00:53:48.235>perspective.

1027
00:53:49.145 --> 00:53:52.325
<00:53:49.145>And <00:53:49.300>when <00:53:49.456>you <00:53:49.611>decompose <00:53:49.767>it <00:53:49.922>into <00:53:50.078>the, <00:53:50.233>to <00:53:50.389>the, <00:53:50.545>um,

1028
00:53:52.375 --> 00:53:55.125
<00:53:52.375>perception <00:53:52.608>component, <00:53:52.841>for <00:53:53.075>example, <00:53:53.308>like <00:53:53.541>the <00:53:53.775>detection,

1029
00:53:55.465 --> 00:53:57.685
<00:53:55.465>you <00:53:55.620>kind <00:53:55.776>of, <00:53:55.931>uh, <00:53:56.087>kind <00:53:56.242>of <00:53:56.398>come <00:53:56.553>up <00:53:56.709>with <00:53:56.865>requirements,

1030
00:53:57.685 --> 00:54:00.525
<00:53:57.685>which <00:53:57.884>is <00:53:58.085>actually, <00:53:58.285>actually <00:53:58.485>component <00:53:58.685>has <00:53:58.884>to, <00:53:59.085>uh,

1031
00:54:01.305 --> 00:54:02.365
<00:54:01.305>uh, <00:54:01.622>be <00:54:01.941>capable <00:54:02.259>of.

1032
00:54:02.425 --> 00:54:05.085
<00:54:02.425>And <00:54:02.775>there, <00:54:03.125>um, <00:54:03.475>argumenting <00:54:03.825>this,

1033
00:54:05.085 --> 00:54:08.445
<00:54:05.085>when <00:54:05.240>we <00:54:05.396>know <00:54:05.551>there's <00:54:05.707>DNS <00:54:05.862>involved, <00:54:06.018>uh, <00:54:06.173>in <00:54:06.329>the <00:54:06.485>perception,

1034
00:54:08.445 --> 00:54:11.325
<00:54:08.445>for <00:54:08.620>example, <00:54:08.795>we <00:54:08.970>say, <00:54:09.145>well, <00:54:09.320>we <00:54:09.495>know <00:54:09.670>there <00:54:09.845>can,

1035
00:54:11.455 --> 00:54:13.485
<00:54:11.455>stuff <00:54:11.688>can <00:54:11.921>go <00:54:12.154>wrong <00:54:12.388>with <00:54:12.621>the <00:54:12.855>dns.

1036
00:54:13.625 --> 00:54:16.165
<00:54:13.625>And <00:54:13.800>there <00:54:13.975>we, <00:54:14.150>we <00:54:14.325>take <00:54:14.500>this <00:54:14.675>approach <00:54:14.850>that, <00:54:15.025>that

1037
00:54:16.165 --> 00:54:19.845
<00:54:16.165>by <00:54:16.340>the <00:54:16.515>way <00:54:16.690>are, <00:54:16.865>uh, <00:54:17.040>um, <00:54:17.215>inspired <00:54:17.390>from <00:54:17.565>the

1038
00:54:20.565 --> 00:54:22.125
<00:54:20.565>project <00:54:21.031>publicly <00:54:21.498>funded <00:54:21.965>project.

1039
00:54:22.865 --> 00:54:26.925
<00:54:22.865>Um, <00:54:23.004>we, <00:54:23.145>we <00:54:23.285>know <00:54:23.424>those <00:54:23.564>concerns <00:54:23.705>and <00:54:23.845>then <00:54:23.984>we <00:54:24.125>take <00:54:24.265>measures.

1040
00:54:27.025 --> 00:54:29.285
<00:54:27.025>So <00:54:27.225>we <00:54:27.425>say, <00:54:27.625>well, <00:54:27.825>we <00:54:28.025>know <00:54:28.225>it's <00:54:28.425>not,

1041
00:54:29.665 --> 00:54:31.405
<00:54:29.665>has <00:54:30.015>no <00:54:30.365>reliable <00:54:30.715>confidence <00:54:31.065>information,

1042
00:54:31.405 --> 00:54:33.445
<00:54:31.405>therefore <00:54:31.638>we, <00:54:31.871>we <00:54:32.105>apply <00:54:32.338>some <00:54:32.571>uncertainty <00:54:32.805>mechanism

1043
00:54:34.185 --> 00:54:36.845
<00:54:34.185>and, <00:54:34.418>um, <00:54:34.651>yeah, <00:54:34.884>then <00:54:35.118>measure <00:54:35.351>it <00:54:35.585>and,

1044
00:54:36.905 --> 00:54:38.005
<00:54:36.905>and <00:54:37.103>put <00:54:37.301>it <00:54:37.499>into <00:54:37.697>the <00:54:37.895>system.

1045
00:54:38.745 --> 00:54:41.965
<00:54:38.745>And, <00:54:38.920>uh, <00:54:39.095>therefore <00:54:39.270>in <00:54:39.444>the <00:54:39.620>end <00:54:39.795>come <00:54:39.970>with, <00:54:40.145>uh,

1046
00:54:41.995 --> 00:54:45.245
<00:54:41.995>with <00:54:42.150>evidences <00:54:42.306>from, <00:54:42.461>from <00:54:42.617>all <00:54:42.772>sides, <00:54:42.928>from <00:54:43.083>the <00:54:43.239>DNN <00:54:43.395>side,

1047
00:54:45.305 --> 00:54:48.205
<00:54:45.305>or <00:54:45.432>we, <00:54:45.559>we <00:54:45.686>get, <00:54:45.814>generate <00:54:45.941>more <00:54:46.068>data <00:54:46.195>as <00:54:46.323>we've <00:54:46.450>seen <00:54:46.577>in <00:54:46.705>the,

1048
00:54:48.225 --> 00:54:50.285
<00:54:48.225>in <00:54:48.458>the <00:54:48.691>other <00:54:48.924>part <00:54:49.158>of <00:54:49.391>the <00:54:49.625>talk.

1049
00:54:50.985 --> 00:54:53.325
<00:54:50.985>Um, <00:54:51.218>or <00:54:51.451>we <00:54:51.685>do <00:54:51.918>a <00:54:52.151>different <00:54:52.385>architecture,

1050
00:54:53.325 --> 00:54:56.645
<00:54:53.325>like <00:54:53.500>different <00:54:53.674>several <00:54:53.850>sensor <00:54:54.024>or <00:54:54.200>redundancy <00:54:54.375>and <00:54:54.549>so <00:54:54.725>on.

1051
00:54:57.105 --> 00:55:01.605
<00:54:57.105>So <00:54:57.245>that <00:54:57.385>in <00:54:57.525>the <00:54:57.665>end, <00:54:57.805>like <00:54:57.945>in <00:54:58.085>an <00:54:58.225>iterative <00:54:58.365>process, <00:54:58.505>uh,

1052
00:55:01.785 --> 00:55:04.125
<00:55:01.785>we <00:55:02.065>kind <00:55:02.345>of, <00:55:02.625>uh, <00:55:02.904>can <00:55:03.185>argument

1053
00:55:04.125 --> 00:55:07.645
<00:55:04.125>that <00:55:04.300>the <00:55:04.475>system <00:55:04.650>which <00:55:04.825>uses <00:55:05.000>the <00:55:05.175>DNS <00:55:05.350>say, <00:55:05.525>um,

1054
00:55:08.025 --> 00:55:10.485
<00:55:08.025>and <00:55:08.305>that's <00:55:08.585>really <00:55:08.865>raising <00:55:09.145>new <00:55:09.425>challenges

1055
00:55:10.625 --> 00:55:11.725
<00:55:10.625>in, <00:55:10.955>in <00:55:11.285>development <00:55:11.615>process.

1056
00:55:12.585 --> 00:55:17.205
<00:55:12.585>And, <00:55:12.760>um, <00:55:12.935>we <00:55:13.110>actually, <00:55:13.285>uh, <00:55:13.460>at <00:55:13.635>carrier, <00:55:13.810>I'd <00:55:13.985>have

1057
00:55:17.205 --> 00:55:19.165
<00:55:17.205>to <00:55:17.555>build <00:55:17.904>this <00:55:18.255>iteratively, <00:55:18.605>right?

1058
00:55:19.265 --> 00:55:21.685
<00:55:19.265>You <00:55:19.440>don't <00:55:19.615>know <00:55:19.790>how <00:55:19.964>the, <00:55:20.140>the <00:55:20.315>full <00:55:20.490>specification <00:55:20.665>is,

1059
00:55:21.905 --> 00:55:24.525
<00:55:21.905>and <00:55:22.080>you <00:55:22.255>have <00:55:22.430>to <00:55:22.605>test, <00:55:22.780>uh, <00:55:22.955>against <00:55:23.130>like, <00:55:23.305>uh,

1060
00:55:24.945 --> 00:55:27.645
<00:55:24.945>the <00:55:25.145>specification <00:55:25.345>of <00:55:25.545>that <00:55:25.745>open <00:55:25.945>world, <00:55:26.145>uh, <00:55:26.345>that,

1061
00:55:27.675 --> 00:55:29.325
<00:55:27.675>that <00:55:28.025>also <00:55:28.375>evolves <00:55:28.725>over <00:55:29.075>time.

1062
00:55:29.385 --> 00:55:32.485
<00:55:29.385>So <00:55:29.585>it's <00:55:29.785>like <00:55:29.985>a <00:55:30.185>very <00:55:30.385>iterative, <00:55:30.585>um, <00:55:30.785>thing.

1063
00:55:32.745 --> 00:55:36.525
<00:55:32.745>And, <00:55:32.920>um, <00:55:33.095>that's <00:55:33.270>just <00:55:33.444>like <00:55:33.620>maybe <00:55:33.795>as, <00:55:33.970>as <00:55:34.145>some,

1064
00:55:36.595 --> 00:55:39.205
<00:55:36.595>some <00:55:36.770>last <00:55:36.944>message <00:55:37.120>as <00:55:37.294>we <00:55:37.470>are <00:55:37.645>already <00:55:37.819>over <00:55:37.995>time,

1065
00:55:39.385 --> 00:55:41.245
<00:55:39.385>um, <00:55:39.851>comes <00:55:40.318>over <00:55:40.785>here.

1066
00:55:41.345 --> 00:55:42.525
<00:55:41.345>So, <00:55:41.557>so <00:55:41.769>you <00:55:41.982>really <00:55:42.194>need <00:55:42.407>to,

1067
00:55:42.585 --> 00:55:44.365
<00:55:42.585>you <00:55:42.818>can <00:55:43.051>put <00:55:43.285>everything <00:55:43.518>in <00:55:43.751>Word <00:55:43.985>documents

1068
00:55:44.365 --> 00:55:48.085
<00:55:44.365>and <00:55:44.540>Excel <00:55:44.714>sheets, <00:55:44.890>so <00:55:45.064>really <00:55:45.240>need <00:55:45.415>a, <00:55:45.589>um, <00:55:45.765>uh,

1069
00:55:48.085 --> 00:55:50.965
<00:55:48.085>like <00:55:48.285>a <00:55:48.485>model <00:55:48.685>driven, <00:55:48.885>uh, <00:55:49.085>it <00:55:49.285>software <00:55:49.485>development.

1070
00:55:51.225 --> 00:55:53.405
<00:55:51.225>And, <00:55:51.924>uh, <00:55:52.625>um,

1071
00:55:54.585 --> 00:55:57.125
<00:55:54.585>that's <00:55:54.760>may <00:55:54.935>maybe <00:55:55.110>brings <00:55:55.285>us <00:55:55.460>to, <00:55:55.635>to <00:55:55.810>the <00:55:55.985>summary.

1072
00:55:57.625 --> 00:56:02.365
<00:55:57.625>So <00:55:57.752>like, <00:55:57.879>if <00:55:58.006>you <00:55:58.134>use <00:55:58.261>dns, <00:55:58.388>well <00:55:58.515>be <00:55:58.643>careful, <00:55:58.770>um, <00:55:58.897>in <00:55:59.025>general,

1073
00:56:03.025 --> 00:56:05.125
<00:56:03.025>uh, <00:56:03.258>so <00:56:03.491>for <00:56:03.725>example, <00:56:03.958>for <00:56:04.191>unreliable, <00:56:04.425>uh,

1074
00:56:05.125 --> 00:56:07.685
<00:56:05.125>confidence <00:56:05.405>information, <00:56:05.685>like, <00:56:05.965>uh, <00:56:06.245>there, <00:56:06.525>there's,

1075
00:56:07.685 --> 00:56:09.365
<00:56:07.685>there's <00:56:07.965>stuff <00:56:08.245>that <00:56:08.525>helps <00:56:08.805>with <00:56:09.085>this.

1076
00:56:09.465 --> 00:56:14.125
<00:56:09.465>So <00:56:09.640>for <00:56:09.815>example, <00:56:09.990>we've <00:56:10.165>seen <00:56:10.340>this, <00:56:10.515>uh, <00:56:10.690>data, <00:56:10.865>uh,

1077
00:56:14.825 --> 00:56:16.925
<00:56:14.825>um, <00:56:15.524>recording <00:56:16.225>part

1078
00:56:17.065 --> 00:56:18.805
<00:56:17.065>or <00:56:17.415>this <00:56:17.765>online <00:56:18.115>monitor <00:56:18.465>part

1079
00:56:18.815 --> 00:56:22.005
<00:56:18.815>where <00:56:19.015>we <00:56:19.215>actually <00:56:19.415>have <00:56:19.615>measures <00:56:19.815>available <00:56:20.015>that, <00:56:20.215>uh,

1080
00:56:22.005 --> 00:56:26.165
<00:56:22.005>help <00:56:22.160>me <00:56:22.316>help <00:56:22.471>with, <00:56:22.627>uh, <00:56:22.782>with, <00:56:22.938>with <00:56:23.093>this, <00:56:23.249>um, <00:56:23.405>insufficiencies.

1081
00:56:26.825 --> 00:56:29.245
<00:56:26.825>And <00:56:27.000>what <00:56:27.174>I <00:56:27.350>also <00:56:27.524>said, <00:56:27.700>uh, <00:56:27.875>in <00:56:28.049>the <00:56:28.225>end,

1082
00:56:29.245 --> 00:56:31.965
<00:56:29.245>basically <00:56:29.525>when <00:56:29.805>we <00:56:30.085>put <00:56:30.365>Datadriven <00:56:30.645>functions,

1083
00:56:32.425 --> 00:56:33.765
<00:56:32.425>we <00:56:32.827>really <00:56:33.229>have <00:56:33.631>big,

1084
00:56:33.825 --> 00:56:36.165
<00:56:33.825>big <00:56:34.058>challenges <00:56:34.291>in <00:56:34.524>the <00:56:34.758>development <00:56:34.991>processes, <00:56:35.225>right?

1085
00:56:36.225 --> 00:56:37.605
<00:56:36.225>Is <00:56:36.432>we <00:56:36.639>need <00:56:36.846>this <00:56:37.053>big <00:56:37.260>loop <00:56:37.467>thing

1086
00:56:37.605 --> 00:56:40.205
<00:56:37.605>with <00:56:37.780>the <00:56:37.955>new <00:56:38.130>situations <00:56:38.305>we <00:56:38.480>haven't <00:56:38.655>seen, <00:56:38.830>we <00:56:39.005>need

1087
00:56:40.205 --> 00:56:42.525
<00:56:40.205>to <00:56:40.485>apply <00:56:40.765>measures <00:56:41.045>during <00:56:41.325>the <00:56:41.605>development,

1088
00:56:42.545 --> 00:56:44.645
<00:56:42.545>we <00:56:42.778>can <00:56:43.011>fully <00:56:43.245>specify <00:56:43.478>the <00:56:43.711>whole <00:56:43.945>system

1089
00:56:44.865 --> 00:56:46.885
<00:56:44.865>and <00:56:45.098>then <00:56:45.331>build <00:56:45.564>it <00:56:45.798>like <00:56:46.031>a <00:56:46.265>waterfall.

1090
00:56:47.105 --> 00:56:50.085
<00:56:47.105>Uh, <00:56:47.305>so <00:56:47.505>that's <00:56:47.705>something <00:56:47.905>really <00:56:48.105>has <00:56:48.305>new <00:56:48.505>challenges,

1091
00:56:50.185 --> 00:56:54.085
<00:56:50.185>but, <00:56:50.340>uh, <00:56:50.496>at <00:56:50.651>carrier, <00:56:50.807>we <00:56:50.962>basically <00:56:51.118>as <00:56:51.273>a <00:56:51.429>software <00:56:51.585>company,

1092
00:56:54.675 --> 00:56:57.605
<00:56:54.675>have <00:56:54.830>the <00:56:54.986>right <00:56:55.141>minds <00:56:55.297>and <00:56:55.452>really, <00:56:55.608>uh, <00:56:55.763>nice <00:56:55.919>interactive, <00:56:56.075>uh,

1093
00:56:57.605 --> 00:57:01.565
<00:56:57.605>community <00:56:57.760>to, <00:56:57.916>to <00:56:58.071>bring <00:56:58.227>up, <00:56:58.382>uh, <00:56:58.538>the <00:56:58.693>right, <00:56:58.849>right <00:56:59.005>tools

1094
00:57:01.665 --> 00:57:03.365
<00:57:01.665>and <00:57:02.015>processes, <00:57:02.365>uh, <00:57:02.715>for <00:57:03.065>that.

1095
00:57:03.625 --> 00:57:07.165
<00:57:03.625>Um, <00:57:03.800>yeah, <00:57:03.975>I <00:57:04.150>hope, <00:57:04.325>uh, <00:57:04.500>that <00:57:04.675>gave <00:57:04.850>you <00:57:05.025>some,

1096
00:57:07.275 --> 00:57:10.285
<00:57:07.275>some <00:57:07.475>good <00:57:07.675>insights <00:57:07.875>into <00:57:08.075>what <00:57:08.275>actually <00:57:08.475>uncertainty, <00:57:08.675>uh,

1097
00:57:10.825 --> 00:57:12.525
<00:57:10.825>is <00:57:11.105>about <00:57:11.384>in, <00:57:11.665>in <00:57:11.944>machine <00:57:12.225>learning

1098
00:57:12.825 --> 00:57:14.525
<00:57:12.825>or <00:57:13.105>specifically <00:57:13.384>in <00:57:13.665>deep <00:57:13.944>neur <00:57:14.225>networks

1099
00:57:15.145 --> 00:57:17.845
<00:57:15.145>and <00:57:15.345>how <00:57:15.545>we, <00:57:15.745>uh, <00:57:15.945>work <00:57:16.145>with <00:57:16.345>this <00:57:16.545>at,

1100
00:57:19.065 --> 00:57:20.245
<00:57:19.065>uh, <00:57:19.330>thank <00:57:19.596>you <00:57:19.861>very <00:57:20.127>much.

1101
00:57:26.255 --> 00:57:27.865
<00:57:26.255>Okay. <00:57:26.605>Thank <00:57:26.955>you <00:57:27.305>very <00:57:27.655>much.

1102
00:57:28.095 --> 00:57:30.425
<00:57:28.095>That <00:57:28.444>was <00:57:28.794>really, <00:57:29.145>really <00:57:29.495>interesting.

1103
00:57:30.925 --> 00:57:35.545
<00:57:30.925>Um, <00:57:31.065>we've <00:57:31.205>got <00:57:31.345>quite <00:57:31.485>some <00:57:31.625>questions <00:57:31.765>on <00:57:31.905>the, <00:57:32.045>on <00:57:32.185>the <00:57:32.325>chat

1104
00:57:36.045 --> 00:57:39.145
<00:57:36.045>and, <00:57:36.220>uh, <00:57:36.395>thanks <00:57:36.570>for <00:57:36.745>answering <00:57:36.920>them <00:57:37.095>on <00:57:37.270>the <00:57:37.445>way.

1105
00:57:39.605 --> 00:57:43.185
<00:57:39.605>Uh, <00:57:39.745>I <00:57:39.885>would <00:57:40.025>say <00:57:40.165>that, <00:57:40.305>um, <00:57:40.445>I <00:57:40.585>have <00:57:40.725>also <00:57:40.865>a <00:57:41.005>question,

1106
00:57:43.185 --> 00:57:46.585
<00:57:43.185>like <00:57:43.325>if <00:57:43.465>someone <00:57:43.605>would <00:57:43.745>like <00:57:43.884>to <00:57:44.025>enter <00:57:44.165>into <00:57:44.305>this <00:57:44.445>field <00:57:44.585>and,

1107
00:57:46.685 --> 00:57:48.225
<00:57:46.685>and <00:57:47.031>start <00:57:47.377>working <00:57:47.724>there <00:57:48.071>or,

1108
00:57:48.365 --> 00:57:51.945
<00:57:48.365>or <00:57:48.492>switch <00:57:48.619>to <00:57:48.746>this <00:57:48.874>field, <00:57:49.001>like <00:57:49.128>what <00:57:49.255>would <00:57:49.383>be <00:57:49.510>the <00:57:49.637>next <00:57:49.765>steps

1109
00:57:52.125 --> 00:57:53.785
<00:57:52.125>and <00:57:52.475>are <00:57:52.825>there <00:57:53.175>any <00:57:53.525>preconditions

1110
00:57:54.005 --> 00:57:56.825
<00:57:54.005>or <00:57:54.205>what <00:57:54.405>knowledge <00:57:54.605>should <00:57:54.805>you <00:57:55.005>bring <00:57:55.205>with <00:57:55.405>you?

1111
00:58:00.585 --> 00:58:01.995
<00:58:00.585>Yeah, <00:58:00.796>maybe <00:58:01.008>I <00:58:01.219>can <00:58:01.431>answer <00:58:01.642>to <00:58:01.854>this.

1112
00:58:02.015 --> 00:58:05.875
<00:58:02.015>So <00:58:02.248>basically, <00:58:02.481>uh, <00:58:02.714>in <00:58:02.948>carat, <00:58:03.181>uh, <00:58:03.415>we,

1113
00:58:05.935 --> 00:58:07.915
<00:58:05.935>we <00:58:06.285>need <00:58:06.634>data <00:58:06.985>scientists, <00:58:07.335>right?

1114
00:58:08.055 --> 00:58:10.035
<00:58:08.055>As <00:58:08.288>we've <00:58:08.521>heard, <00:58:08.754>to <00:58:08.988>really <00:58:09.221>understand <00:58:09.455>what,

1115
00:58:10.035 --> 00:58:11.195
<00:58:10.035>what <00:58:10.296>the <00:58:10.557>AI <00:58:10.818>is <00:58:11.079>about.

1116
00:58:11.505 --> 00:58:16.155
<00:58:11.505>What <00:58:11.645>we <00:58:11.785>also, <00:58:11.925>what <00:58:12.065>what's <00:58:12.205>also <00:58:12.345>needed <00:58:12.485>is <00:58:12.625>like <00:58:12.765>the <00:58:12.905>overall,

1117
00:58:16.415 --> 00:58:19.675
<00:58:16.415>uh, <00:58:16.695>openness <00:58:16.975>for, <00:58:17.255>for <00:58:17.535>development, <00:58:17.815>right?

1118
00:58:19.775 --> 00:58:22.155
<00:58:19.775>So <00:58:20.008>what, <00:58:20.241>what's <00:58:20.475>actually, <00:58:20.708>uh, <00:58:20.941>the <00:58:21.175>interplay

1119
00:58:22.225 --> 00:58:26.715
<00:58:22.225>between <00:58:22.505>different, <00:58:22.785>uh, <00:58:23.065>uh, <00:58:23.345>um, <00:58:23.625>groups,

1120
00:58:27.095 --> 00:58:29.675
<00:58:27.095>uh, <00:58:27.294>like <00:58:27.495>the <00:58:27.694>safety <00:58:27.895>people <00:58:28.095>and, <00:58:28.294>and <00:58:28.495>uh,

1121
00:58:29.695 --> 00:58:31.235
<00:58:29.695>and <00:58:30.041>uh, <00:58:30.388>the <00:58:30.734>function <00:58:31.081>people

1122
00:58:31.895 --> 00:58:34.075
<00:58:31.895>and <00:58:32.095>the <00:58:32.295>product <00:58:32.495>people <00:58:32.695>and, <00:58:32.895>and <00:58:33.095>so <00:58:33.295>on.

1123
00:58:34.295 --> 00:58:36.075
<00:58:34.295>So <00:58:34.575>maybe <00:58:34.855>somebody <00:58:35.135>can <00:58:35.415>add, <00:58:35.695>but,

1124
00:58:36.135 --> 00:58:38.275
<00:58:36.135>but <00:58:36.368>it's <00:58:36.601>like, <00:58:36.835>like <00:58:37.068>it's <00:58:37.301>really, <00:58:37.535>uh,

1125
00:58:38.415 --> 00:58:39.555
<00:58:38.415>an <00:58:38.928>interesting <00:58:39.441>environment.

1126
00:58:43.045 --> 00:58:45.135
<00:58:43.045>Yeah. <00:58:43.200>And <00:58:43.356>may <00:58:43.511>maybe, <00:58:43.667>maybe <00:58:43.822>I <00:58:43.978>can <00:58:44.133>add <00:58:44.289>one <00:58:44.445>point.

1127
00:58:45.235 --> 00:58:46.735
<00:58:45.235>So <00:58:45.403>I, <00:58:45.572>I <00:58:45.741>know <00:58:45.910>there <00:58:46.078>are <00:58:46.247>a <00:58:46.416>lot <00:58:46.585>of

1128
00:58:46.795 --> 00:58:48.815
<00:58:46.795>of <00:58:47.075>developers <00:58:47.355>here <00:58:47.635>probably <00:58:47.915>as <00:58:48.195>well.

1129
00:58:49.355 --> 00:58:53.695
<00:58:49.355>Um, <00:58:49.495>and, <00:58:49.635>and <00:58:49.775>so, <00:58:49.915>um, <00:58:50.055>uh, <00:58:50.195>there's <00:58:50.335>a <00:58:50.475>lot <00:58:50.615>more <00:58:50.755>to,

1130
00:58:53.995 --> 00:58:56.735
<00:58:53.995>to <00:58:54.194>it <00:58:54.395>than, <00:58:54.595>than <00:58:54.795>just <00:58:54.995>doing, <00:58:55.194>I <00:58:55.395>mean,

1131
00:58:56.735 --> 00:58:58.215
<00:58:56.735>just <00:58:56.957>doing <00:58:57.179>the <00:58:57.401>data <00:58:57.623>science, <00:58:57.845>I <00:58:58.067>mean,

1132
00:58:58.215 --> 00:59:00.935
<00:58:58.215>this <00:58:58.355>is <00:58:58.495>very <00:58:58.635>interesting <00:58:58.775>parts, <00:58:58.915>but <00:58:59.055>bringing <00:58:59.195>it <00:58:59.335>all <00:58:59.475>to <00:58:59.615>life

1133
00:59:01.035 --> 00:59:03.935
<00:59:01.035>and, <00:59:01.162>and <00:59:01.289>bringing <00:59:01.416>it <00:59:01.544>to <00:59:01.671>a, <00:59:01.798>to <00:59:01.925>a <00:59:02.053>product <00:59:02.180>requires <00:59:02.307>a <00:59:02.435>lot,

1134
00:59:04.355 --> 00:59:05.375
<00:59:04.355>lot <00:59:04.814>of <00:59:05.273>engineering.

1135
00:59:05.835 --> 00:59:09.335
<00:59:05.835>Um, <00:59:05.975>and, <00:59:06.115>um, <00:59:06.255>so <00:59:06.395>people <00:59:06.535>that <00:59:06.675>have <00:59:06.815>a <00:59:06.955>good <00:59:07.095>understanding <00:59:07.235>of,

1136
00:59:09.555 --> 00:59:11.135
<00:59:09.555>of, <00:59:10.021>uh, <00:59:10.488>machine <00:59:10.955>learning

1137
00:59:11.435 --> 00:59:14.815
<00:59:11.435>but <00:59:11.668>are <00:59:11.901>excellent <00:59:12.134>engineers, <00:59:12.368>software <00:59:12.601>engineers, <00:59:12.835>um,

1138
00:59:15.115 --> 00:59:16.935
<00:59:15.115>we <00:59:15.290>are <00:59:15.464>always <00:59:15.640>looking <00:59:15.814>for <00:59:15.990>those <00:59:16.165>kind <00:59:16.339>of <00:59:16.515>people.

1139
00:59:39.755 --> 00:59:41.735
<00:59:39.755>So <00:59:39.910>maybe <00:59:40.066>you <00:59:40.221>could <00:59:40.377>put <00:59:40.532>it <00:59:40.688>also <00:59:40.843>in <00:59:40.999>the <00:59:41.155>chat,

1140
00:59:42.035 --> 00:59:43.055
<00:59:42.035>so <00:59:42.341>it's, <00:59:42.647>yeah, <00:59:42.953>okay.

1141
00:59:43.055 --> 00:59:46.335
<00:59:43.055>Just <00:59:43.194>put <00:59:43.335>it <00:59:43.475>in <00:59:43.615>the <00:59:43.754>chat. <00:59:43.895>Yes. <00:59:44.035>Yeah, <00:59:44.174>yeah, <00:59:44.315>good <00:59:44.455>idea.

1142
00:59:47.155 --> 00:59:51.135
<00:59:47.155>Um, <00:59:47.295>and <00:59:47.435>with <00:59:47.575>that <00:59:47.715>said, <00:59:47.855>I, <00:59:47.995>I <00:59:48.135>would, <00:59:48.275>uh, <00:59:48.415>really, <00:59:48.555>uh,

1143
00:59:51.145 --> 00:59:52.255
<00:59:51.145>close <00:59:51.394>the <00:59:51.644>session <00:59:51.894>right <00:59:52.144>now

1144
00:59:52.255 --> 00:59:55.055
<00:59:52.255>because <00:59:52.455>we <00:59:52.655>have <00:59:52.855>another <00:59:53.055>super <00:59:53.255>interesting <00:59:53.455>workshop, <00:59:53.655>uh,

1145
00:59:55.055 --> 00:59:56.135
<00:59:55.055>that <00:59:55.379>has <00:59:55.703>just <00:59:56.027>started.

1146
00:59:56.325 --> 01:00:00.015
<00:59:56.325>It's <00:59:56.524>about <00:59:56.725>developing <00:59:56.924>an <00:59:57.125>AI <00:59:57.325>SDK, <00:59:57.524>uh, <00:59:57.725>presented

1147
01:00:00.035 --> 01:00:02.735
<01:00:00.035>by <01:00:00.190>some <01:00:00.346>colleagues <01:00:00.501>of, <01:00:00.657>of <01:00:00.812>three <01:00:00.968>and <01:00:01.123>Fabian <01:00:01.279>and <01:00:01.435>Nico

1148
01:00:03.155 --> 01:00:04.775
<01:00:03.155>and <01:00:03.505>I <01:00:03.855>would <01:00:04.205>basically <01:00:04.555>everyone

1149
01:00:04.795 --> 01:00:08.095
<01:00:04.795>who <01:00:04.935>joined <01:00:05.075>this <01:00:05.215>workshop <01:00:05.355>advise <01:00:05.495>to, <01:00:05.635>to <01:00:05.775>switch <01:00:05.915>over <01:00:06.055>to <01:00:06.195>this,

1150
01:00:08.195 --> 01:00:09.215
<01:00:08.195>uh, <01:00:08.654>next <01:00:09.113>workshop.

1151
01:00:09.715 --> 01:00:12.215
<01:00:09.715>And <01:00:09.870>therefore, <01:00:10.026>I <01:00:10.181>would <01:00:10.337>say, <01:00:10.492>uh, <01:00:10.648>thank <01:00:10.803>you <01:00:10.959>very <01:00:11.115>much,

1152
01:00:12.325 --> 01:00:13.775
<01:00:12.325>Tillman, <01:00:12.759>Fabian, <01:00:13.195>and <01:00:13.630>Nico,

1153
01:00:13.995 --> 01:00:16.175
<01:00:13.995>and <01:00:14.275>thanks <01:00:14.555>everyone <01:00:14.835>also <01:00:15.115>for <01:00:15.395>watching

1154
01:00:16.195 --> 01:00:17.495
<01:00:16.195>and <01:00:16.429>seeing <01:00:16.663>in <01:00:16.897>the <01:00:17.131>other <01:00:17.365>session.

1155
01:00:19.655 --> 01:00:22.385
<01:00:19.655>Alright, <01:00:19.830>thank <01:00:20.005>you <01:00:20.180>everyone. <01:00:20.355>Thank <01:00:20.530>you <01:00:20.705>everyone <01:00:20.880>for <01:00:21.055>bye.

1156
01:00:22.405 --> 01:00:27.145
<01:00:22.405>Bye <01:00:23.105>bye. <01:00:23.805>Thanks.