Scripted Demos vs. Real Autonomy: The Truth About Dancing Robots

00:00:00: Welcome back to another episode of Rob Talk, the RobCoast podcast where we talk about everything that is related to robots and the future of robotics.

00:00:08: And today we have a really interesting topic in which talking better than any human could do them.

00:00:32: And this is something that I think has always gone viral around the internet since already two thousand sixteen.

00:00:38: Rob talk, The Autonomous Robotics Podcast.

00:00:42: physical AI no theory just reality.

00:00:48: Yeah my question to you Clemens and Clemens obviously working at Robco as one of the principal engineers in really deep into this topic have we solved human robotics now when i see all these videos?

00:01:01: They're extremely impressive.

00:01:03: The technology has improved so much over the recent years, but we are far away from having solved humanoids.

00:01:12: But I see these humanoids like they do on stage with thirty other performers and there's thirty robots all doing in synchricity doing same moves in perfect execution?

00:01:26: Yes!

00:01:27: It is just a very fancy way to... A scripted sequence.

00:01:33: Okay, so

00:01:34: basically

00:01:34: a demo right?

00:01:36: It is a demo but an extremely good one

00:01:41: okay.

00:01:42: So we have this because like with also the videos Right.

00:01:45: But The question for me Is When they can do This Couldn't They Do The Same Choreography In A Factory?

00:01:54: They Could.

00:01:57: So I mean, we can talk about all sorts of different things.

00:01:59: Obviously there's not only the dancing robots is also the marathon that you should probably talk about okay?

00:02:04: But what's?

00:02:05: What's the challenge in a factory?

00:02:08: maybe You could have like scripted movements but I Mean they're.

00:02:17: what was the point right?

00:02:18: either you Have a script at motion then Probably wheels are good enough and much more stable.

00:02:26: Makes sense, yeah?

00:02:28: Or you want to solve a much larger problem which is okay.

00:02:33: what do I do as human when i move through the factory and have goal in mind And then movement comes just by reaching that goal right it's means to an end.

00:02:49: so That part completely not covered by dancing robots.

00:02:55: talk a little bit about how they came about.

00:02:59: Yes, and the whole development there because it's been a long time coming right?

00:03:06: People have been building humanoid robots for thirty years but then something happened which was the Fukushima accident.

00:03:16: so suddenly you had very nasty environment And that was in two thousand eleven, right?

00:03:25: Yes.

00:03:26: So the DARPA, Defense Advanced Research Project Agency... The arm of US Army which hands out research grants.

00:03:39: they are sometimes doing these challenges on autonomous driving But I also did challenges around locomotion with humanoids.

00:03:50: And so then that was two thousand thirteen, fourteen fifteen.

00:03:57: So they said okay what if we had to move about in an environment like Fukushima?

00:04:03: could these robots do things completely autonomously?

00:04:08: Like where do we stand?

00:04:10: and the results were not really good at point.

00:04:15: Yeah, you sent me a video in advance.

00:04:17: It was really funny and I'll just describe a few scenes.

00:04:20: so there's robots trying to get out of cars like simple cars which are open it's not?

00:04:24: they don't have the door that just has to step out off an Open Jeep.

00:04:28: They start jittering falling over.

00:04:30: then we have other robots trying To open doors.

00:04:32: where?

00:04:32: They missed the handle or they grab their handle but Then Just fall backwards And cannot and Cannot move anymore.

00:04:38: and then There's also something That is Really interesting.

00:04:40: So they try to manipulate Their environment Like this valve that they want to turn, turned something off like a gas leak or steam coming out.

00:04:50: And the robot misses the handle and grabs the air just falls over because he doesn't have the counterweight.

00:04:56: so this is not one example.

00:04:58: I would say at least twenty five in his video.

00:05:00: it's really interesting how fragile they are right now.

00:05:07: we humans never had an issue with that as we count our balance In two thousand fifteen at least they couldn't.

00:05:15: Yeah, so I mean especially like when you think about the time.

00:05:19: where did we stand in terms of technology there?

00:05:22: So all of the controls for the robots were still very hand tuned and people using MATLAB to understand their model robot.

00:05:33: And then they built a program For particular movement.

00:05:37: They optimized tons of parameters to make it work.

00:05:41: The compute was very limited, not even the drives were very good.

00:05:51: they didn't have we call this back drivability or this kind of sensing fashion back then.

00:05:57: so a lot of engineering and trying was extremely challenging and it took years to really get that really under control.

00:06:16: People got very far with without like AI over the years, so much later we had robots said were able to climb on boxes even do backflips and stuff like that.

00:06:35: But but, but even that was just very constrained to this one situation.

00:06:41: so then what changed?

00:06:44: And I think the big change is first of all we had neural nets coming in and reinforcement learning and simulation.

00:06:56: because simulation environments What they do is at the core of a simulation environment, it's physics engine.

00:07:06: And we understand physics in the normal three D environment pretty well basically Newton mechanics right?

00:07:17: So just understanding how gravity reacts to like forces that gravity pulls on particular limb over robot can be simulated extremely well already.

00:07:32: And so what people did then was to take a digital twin off of their robot and put it in simulation, and then parallelize that massively gave the task to solve and randomized certain things through balls at the robots.

00:07:52: And so, the task for the controller was then okay.

00:07:58: remain stable while I'm getting all these forces being sent to me in a randomized fashion.

00:08:07: So that's one and other ones.

00:08:10: we suddenly started data sets of people doing things dancing or moving around, doing Kung Fu movements and so on.

00:08:23: So once you have a dataset then research community can start to deal with these things.

00:08:28: And suddenly it becomes an optimization problem because You put up kind of success metric.

00:08:35: Now everybody just do different techniques for optimizing that success metric.

00:08:43: And in that case, it was okay?

00:08:44: Can I can I execute that movement that was previously recorded from a human while also taking the constraint like why not falling over basically?

00:08:59: and so this was around two thousand eighteen to two thousand nineteen first papers around.

00:09:05: that came about and saw.

00:09:06: for years it got better and also to a certain extent more generalized.

00:09:13: So now we have, you can first of all record a human either through motion captures suite like... We've been using that for decades in movies.

00:09:30: so the suit on has round elements on it and then you have forty eight cameras around, which each costs six thousand euros.

00:09:44: And so you film the person from all sides while they're doing their stuff?

00:09:49: So now we can reverse project points in three D space with a very high accuracy.

00:10:00: movies have replaced real actors with animated characters for, I think probably since the nineties.

00:10:09: And so this technique was used.

00:10:11: meanwhile also purely computer vision based models become better and better.

00:10:16: So you can take a YouTube video of couple dancing And even if that has occlusions and they're in close contact, you can actually extract the movements from people.

00:10:32: So this is basically where we stand today.

00:10:35: It doesn't mean to see hand movements or tight interactions with certain materials but at least find out what the limbs are.

00:10:47: We can also generalize to a certain extent so that you have one movement and followed by another movement.

00:10:55: And meanwhile, I want to carry a box.

00:11:01: So probably the movement itself... ...I'm only applying it on top of my body while legs maintain balance Right, so you split it up in a way.

00:11:17: So there's lot of different pieces are already there and this is why when we see these robots move they become very human like but the case for dancing robot a very famous presentation at the Chinese New Year's Festival.

00:11:37: Very

00:11:38: impressive!

00:11:39: I mean, these are just clones and they're all executing the same movements.

00:11:43: that was pre-recorded.

00:11:45: it also runs only for couple of minutes.

00:11:48: so... The robot task is really to fulfill one thing you can train in simulation.

00:11:56: You have hundreds or thousands like overall billions of examples that flow into these models.

00:12:06: And all it does is just, you know becoming a very robust system because it has seen everything that was thrown on him or maybe even variations on the floor right?

00:12:19: That needs to counterbalance and so then given a machine learning function.

00:12:28: So specifically talking about like this demo or scripted setup.

00:12:33: They basically have been trained either by somebody in a motion suit, Or the virtual world so running millions of iterations off The thing they are going to do In a virtual environment and then This is being programmed into the robot And then it's in a controlled environment

00:12:53: in a certain controlled environment.

00:12:55: So for example, there you were on stage or you had only the

00:12:58: same line surface.

00:12:59: yeah and

00:13:01: does that so?

00:13:03: That is also then the execution's very impressive but obviously it's very controlled.

00:13:07: And what?

00:13:07: how does that help us like looking at looking at robots that are not used Or they're more used into an uncontrolled environment.

00:13:15: because The thing I think is very impressive Is the counterbalance

00:13:21: being

00:13:22: not as clumsy as the ones used in DARPA, right?

00:13:24: Because him doing a backflip and then standing is I think quite impressive.

00:13:31: But how does that help us going forward using them in our daily lives outside of scripted

00:13:36: environments?".

00:13:37: Yeah!

00:13:38: That's where the gap comes in... So first off all these robots do NOT interact a lot with the environment they're just standing on the floor.

00:13:49: so we are now getting the first demos where you know, You can lift a box and you can walk around with the box.

00:13:57: So it's much harder problem because now we have to leave the whole scripted thing.

00:14:03: The other is if you are thinking of doing very or maybe I should say this first so why?

00:14:11: Why Is the development still extremely impressive?

00:14:15: Because We Have Figured Out How To Build a humanoid with very similar motion characteristics to the human.

00:14:27: So like how fast leg or an arm is moving, of how fast it reacts?

00:14:33: Is already very... Very similar to humans and we have made work with compute units that can deal with all this in fast enough as part of the torso.

00:14:49: usually, that's a part of robot itself just driven by batteries.

00:14:54: So like ten years ago many robots then were still tethered and so now we can do it with batteries only.

00:15:04: Also huge advancement right?

00:15:07: It is.

00:15:07: but obviously the question how efficient or often does a motor break down which means The robot falls over and then everything breaks.

00:15:20: You know, there's a ton of problems where you think okay in the factory environment.

00:15:24: do I need to really solve this problem now?

00:15:27: In order to produce value?

00:15:29: probably not but obviously it is big development And i think we're gonna see We are going to be becoming more useful Over the years.

00:15:40: Current state Of Things Is That Now Also In China Since Last Year We Have Half Marathons Being run by humans and then there's a separate track for humanoid.

00:15:53: And so last year it was still also probably bit similar to the DARPA challenge, which is better!

00:16:01: I think they were teams that made it but still a lot slower than the humans.

00:16:07: This year with first team... The best team broke half marathon record.

00:16:15: Right, so we are getting into space where the humanoids can solve a problem of getting to goal without being controlled externally and also having endurance.

00:16:35: To do it not only for three minutes but maybe four an hour or even longer like you had one case where robot walked for hundred kilometers And so some of the teams there were still remote controlling it and they got a penalty.

00:16:53: But I think we're getting to this state where these things can actually do that autonomously, So It's a process but still We are not in the realm of really tight interaction with the environment.

00:17:06: Yeah That would be my next question as well.

00:17:08: like what happens for one of these script shows if somebody runs on stage?

00:17:13: Like, you know those people like on famous artists they climb onto the stage and run up.

00:17:19: What would happen with the robots?

00:17:20: continue to show or what?

00:17:21: They have I don't know obviously different depending on sensors but how will they interact with their changes happening?

00:17:29: so first of all i think that the robots do these shows are actually blind So eyes open or ears in order

00:17:44: to

00:17:44: do a movement, right?

00:17:45: What they've memorized is somebody kicking me in the simulator.

00:17:50: So if you kick such a role there will very likely balance it out up to certain point.

00:17:58: so I think that pretty robust against that.

00:18:02: and then for the marathon i mean there's much more interesting because obviously its like twenty one points something kilometers And there's going to be way more obstacles because other robots are running, so they must be super adaptive.

00:18:14: Like how not run into the man in front of them or something like that?

00:18:18: Yeah!

00:18:18: So I don't really know the exact details for different pieces... Kind imagine if you have navigation systems and some kind of obstacle avoidance.

00:18:28: but obviously for a marathon like this it would also do only minimal parts right?

00:18:33: There was still separate track.

00:18:36: Some robots ran into each other.

00:18:38: One robot just collapsed on the starting line, so yeah I mean it's still funny and very interesting to watch.

00:18:48: So there is one thing we really have to differentiate then?

00:18:51: We must not mistake movement or perfect execution of movements with understanding of real world which probably next step way more important for valuable help from robots, a valuable usage of robots.

00:19:12: Right?

00:19:13: So I think if you think about it many of these robots don't even have hands like they have stumps.

00:19:20: true You don't have to solve the problem off interacting with physical object apart.

00:19:26: form the floor right.

00:19:28: The arms are there for balance.

00:19:31: So I think it can be seen that if they swing their arms are faster and stuff like that.

00:19:35: Yeah, so in that sense is becomes more human-like.

00:19:40: but we talked about this before.

00:19:44: you have an object has a material object has certain distribution of weight.

00:19:53: as a human You would always look at an object predict its properties Before even touch

00:20:00: Right?

00:20:01: And then, I mean even these memes on the internet where you know... You can show your friend okay this is heavy and that the friend lifts it.

00:20:12: Yeah he'll put much more force inside.

00:20:14: Exactly

00:20:16: so we do these kind of predictions all the time um and-and We deal with different materials All The Time and we have learned It over the course Of y'know the first i don't Know maybe a few years right, to deal with the different materials.

00:20:31: And so that's a completely different game.

00:20:34: and The other part is we're not just randomly repeating stuff We are always goal-oriented Yeah...we have certain end state in mind when we try to manipulate something.

00:20:51: That also interesting.

00:20:52: I mean.

00:20:52: you talked about how robot learns and we already talked to this also in another episode.

00:20:58: So by the way, if you watch that episode were really detailed talking about how robots learn because also compared with how babies learned And it's very impressive.

00:21:06: How is done?

00:21:07: Because actually like a human needs about nine months To walk or maybe even ten and two grab objects and stuff needs Also some time.

00:21:13: so

00:21:14: too

00:21:14: to understand.

00:21:15: This Is A process but We now have it internalized But also get.

00:21:18: The machine can't but the computing power is Actually quite intense to do all That on All the prediction.

00:21:24: So I find that very fascinating.

00:21:26: And i would have thought, or really thought the dancing... Because it's so impressive to see a robot in humanized form-dancing That this will be like huge milestone and solve everything already.

00:21:40: But my question is what kind of demonstration would impress you more than the dancing?

00:21:44: Looking at from usage perspective how these robots can be more valuable in human society?

00:21:51: Well It all boils down to how can you manipulate things on a very intricate form, like let's say.

00:22:04: You have a circuit board and you want to put something in.

00:22:07: it need to be extremely detailed.

00:22:10: various little forces are sensitive versus you wanna just carry crate of beer and stack them everywhere.

00:22:22: there is forces, but it also depends on the materials.

00:22:26: It depends a lot on prediction especially now when you have moving objects as well And doing that in a safe manner.

00:22:35: so I think this is more like what we're interested.

00:22:40: Just a question for me because I don't know how it's done.

00:22:42: So looking at the circuit board, i always thought they were being like set by machine... Like a machine dozer and that's done but I have no idea how its done.

00:22:50: to be honest?

00:22:51: It is done in many cases But there are still things where you need to plug-in something or Where the circuitboard already exists.

00:22:59: There are certain plugs that im missing

00:23:02: Thinking about a German or EU plug which is very interesting.

00:23:08: thinking about, you know the big plugs.

00:23:11: I think that's also an interesting challenge because putting it into a wall can be... You need exact amount of pressure but not too little.

00:23:19: to push through the walls could have been interesting for robots to solve without breaking the plug as well.

00:23:29: yeah and i think one way to do this again in simulation.

00:23:35: This is a much harder problem than just gravity, right?

00:23:40: So suddenly you have to deal with contact rich movements of like forces or maybe different materials.

00:23:52: There's really only very recently simulators that are getting close to really simulating in any realistic manner.

00:24:02: So we're really on the forefront there.

00:24:05: If you think about

00:24:07: a normal

00:24:07: simulation, it consists of triangles?

00:24:11: Yes!

00:24:11: Right.

00:24:11: so now you have two objects with triangles.

00:24:13: obviously that doesn't work out right and so I need to do... You know having different mathematical representation of these objects.

00:24:22: And so now if were seeing this is starting too.

00:24:28: I found it fascinating because for me, really simple things like that.

00:24:32: Seeing how difficult they are actually for robots to do but then seeing how impressive the dancing is.

00:24:36: so The Dancing Robots Are A Milestone But Not The Solution.

00:24:41: Yet They're... Probably data helps you a lot i guess right?

00:24:46: It's not where we think human robotics has solved and machines walking on earth already Because its' not Like That.

00:24:55: So I Think We're Just Climbing the ladder of difficulty and.

00:25:00: Difficulty being constrained by the technology that is available, uh, And by the sensors are being available?

00:25:07: Obviously there are tactile sensors so if you put the plug in they're obviously systems.

00:25:14: it would allow you to sense the forces their.

00:25:17: now.

00:25:17: then The next question as okay how do you teach them without losing those forces?

00:25:21: or How Do You Simulate Them?

00:25:23: But it is in principle possible, then the question is how long does this sensor last?

00:25:28: So I think that's a development.

00:25:30: That will probably take years to solve but we're climbing the ladder also from a compute perspective of the difficulty and things you can deal with.

00:25:43: And actually currently one thing Keep me busy and keep a bunch of people busy in the team because we get all these customer requests.

00:25:56: Yeah, really understanding okay?

00:25:57: Oh when can he do this?

00:25:59: Can you do this now?

00:26:00: would it take us twelve or twenty four months?

00:26:02: for thirty six months Because we also need to pick our battles.

00:26:07: Yes Of course.

00:26:08: I think something We talked about as well When were talking About selfless driving Autonomic Driving is that there's like the level You can Get To Eighty percent really fast, but obviously nobody wants to drive in a car that is eighty percent safe.

00:26:23: Ninety percent also fast than ninety.

00:26:25: nine percent Is quite steep?

00:26:28: But then it's the ninety-nine point nine The nineteen nine point nine nine nine and they continuous.

00:26:32: And this is basically It gets deeper and steeper because there are so millions of variations.

00:26:38: So this seems to be something.

00:26:40: when I looked at from A normal standpoint without having deep engineering look into it seemed like Why haven't they solved it yet?

00:26:49: You explained to me that there's infinite possibilities and you don't want a car to crash at any point.

00:26:54: So this is like, problems still need to be solved.

00:26:57: for robots I guess This going to very similar but further manufacturing will be fine At the point before because we can create safe environment And We Don't wanna manipulate The whole world But Very specific things.

00:27:11: so That means the humanoid robot Is not solved yet But the progress is very real and Robco's progress, it was very real.

00:27:18: And the usability of robots is increasing massively through you to things that are developing?

00:27:24: Yeah exactly I think this is what makes this area so interesting from like a technologist point-of-view because especially when we think about autonomous driving It took ten or fifteen years To get into the state where cars driving around autonomously in the cities.

00:27:47: Still getting feedback, still being improved and also still in very controlled environments.

00:27:56: so if you drive one of those autonomous vehicles from San Francisco to Alaska they wouldn't work right because it is still constrained on where.

00:28:07: And so I think just going from where we are right now, where things are pre-programmed on the level of up to what we call Level Two autonomy.

00:28:20: So computer vision... To the next step is when you say hey!

00:28:25: We can do a small task here?

00:28:27: Start dealing with that and build feedback loops.

00:28:32: Thank you so much, Clemens.

00:28:33: It was really insightful and thank you for showing us that the dancing robots have not solved The human robot problem yet but they're a milestone definitely And I think it's still very impressive at all their companies who are working in this space.

00:28:45: creating these demos is definitely impressive Especially comparing how from two thousand fifteen falling over robots can now dance.

00:28:54: But yeah That doesn't mean we'll have a robot Butler arriving for us tomorrow about four factories.

00:28:59: This might be huge step.

00:29:01: And yeah, the next step is not about whether robots can impress us but more about how they can do something and help us for hours.

00:29:07: An hour's an end.

00:29:08: I think it's also really important that a three-hour... Three minute demonstration Is Something That It Not As Valuable as Like Running A Marathon For Maybe Two Hours Or Working In A Factory For Eight Hours Which Is Way More Valuable And Much More Well Important i would say.

00:29:27: So Thank you for pointing that out.

00:29:29: And if you've enjoyed the episode, obviously then please give us a like subscribe and leave a comment there.

00:29:36: we'd love to answer them and definitely contact as well If You're interested and share this episode with anybody who wants to learn more about robots?

00:29:43: And has been asking you how they dancing robots have not why We don't Have A Housemade or Butler house already.

00:29:52: Yeah, Clemens just answered That Why?

00:29:55: Yes!

00:29:57: have our next episode with Rob Talk from RobCo.

00:29:59: And obviously, if you want to know more about RobCo or if you wanna get in touch with Clemens please look him up on LinkedIn and also If You Want To Get In Touch With The Company visit our website at rob.co.

00:30:11: We're working on the future of robotics all the time!

00:30:15: Yeah...you are really into details every day.

00:30:19: so thank-you for being here giving us your insights.

00:30:22: Thank-you!

00:30:23: Hope to see ya soon.

00:30:25: This is RobTalk A podcast by Robco.

00:30:29: Subscribe on Spotify,

00:30:30: Apple Podcasts

00:30:32: and YouTube.

00:30:33: See you on the factory floor.