Empowering the Blind with Pedestrian Autonomous Driving | dotLumen CEO
.Lumen (pronounced dot Lumen) is a Romanian unicorn start-up that specialises in pedestrian autonomous navigation. They opened the Nvidia GTC keynote this year showcasing their glasses for the blind that mimic the benefits of a guide dog, allowing those who cannot see to navigate and interact with the world. Join us in this interview with their CEO Cornel Amariei as we delve into the fascinating story of dotLumen and their incredible technology. Learn about the glasses, the future of humanoid robotics, the excitement and challenges of innovation at this scale and beyond…
Apologies for no audio or video for this interview. Due to issues recording we deemed this the best format to share the conversation.
Credit: .Lumen
Introduction
Cornel: So I’m Cornell, I’m CEO and founder of dotLumen. At dotLumen, we built pedestrian autonomous driving AI, first showcased in the glasses for the blind.
A bit of context about me, I was born in a family of people with disabilities. I’m the only person without a disability in my family. And that really showed me a lot about how much technology can help, but also how little technology exists. When we were looking at problems and everything, after working at one of the largest autonomous driving companies in the world, I realized that, hey, I think we can use autonomous driving for something else except driving cars on the road.
There are a bunch of problems which on the pedestrian side are still unresolved. For example, we spent 62 billion euros on delivering last mile packages by hand because there’s no, there’s very few examples of autonomous robots doing anything on the pedestrian side. And also another interesting problem is that we spent half a billion euros per year into training only a bit over 2,000 guide dogs.
So half a billion, 2,000 guide dogs, that’s not good unit economics. And if you look in the world, we only have 28,000 guide dogs worldwide to basically more than 300 million visually impaired out of which 43 million are fully blind.
And we said, hey, I think we can do something better here. So we took autonomous driving from the road and we scaled it down to the pedestrian side. And scaled it down is not the correct term because definitely in some cases more complex than autonomous driving on the road. The infrastructure for the pedestrian is much more complex than the infrastructure for the road.
In the end the first product we decided to apply this pedestrian autonomous driving was glasses for the blind. Now, the glasses for the blind, they replicate what the guide dog does. So if a guide dog works by pulling your hand away from obstacles, keeping you on the side, avoiding you from obstacles, keeping you on crossings, helping you navigate, the glasses do exactly the same. They even do more.
But if a guide dog works by pulling your hand, the glasses actually work by “pulling your head”. And it’s pulling you in a direction you have to go, over one hundred times a second, it allows you to avoid obstacles, it keeps you on the sidewalk and everything. It’s a self-driving car on your head. It talks to you, you can talk to it, so it does more than a guide dog can do at this point.
There’s been a lot of excitement. There are videos of people testing our glasses and they’ve gone viral. It’s been a lot of work, but everything works very, very well. And yeah, that’s pretty much myself and what we’re doing here at Lumen. With time, we’re going to apply the pedestrian autonomous driving to other fields as well, ranging from humanoid robots or stuff like this.
We already know how to make things navigate to the pedestrian side. So one day you order a pizza and something will knock on your door and you’ll open and there’s going to be a humanoid robot giving you the pizza. And that humanoid robot got there because our work.
Harry: So your vision for dotLumen actually goes broader than visual impairment?
Cornel: Pedestrian autonomous driving, this is what we do. The first application is helping the visual impairment. That’s the first product line, but we see multiple product lines.
Inspiration for dotLumen
Harry: I was particularly inspired by your story. I was wondering if you could tell me a little bit more about that and what the process was from your first thoughts of wanting to help people. How did you actually turn that into this amazing product you’ve got now and what was your path along that journey?
Cornel: Sure. So all my family members have disabilities. It means my parents and my sister. My sister, she’s 15 years older than me, she has severe mental disabilities. My parents have severe automotive disabilities. They cannot walk without wheelchairs or similar. And growing up in such an environment in Eastern Europe where I’m from, in Romania, it was first of all very, very tough.
I remember from a very young age, my mother said it. I was saying to her that when I grow up, I’m going to be a doctor to fix their disabilities. I did build a very, very big project. I lead a very big project as a start-up at some point. I exited and it got absorbed into a company for locomotive disability assistive technology. It’s not just on the market, I’m not sure when it’s going to be on the market, but it’s something pretty cool. I have a few patents in that.
So I think that was like the proper first effort, which I did like months upon months, working on something to aid people with disabilities, specifically for my family. But then, what happened with dotLumen is that while I was working in the automotive field, I said, “Hey, I think we can use autonomous driving for something else.”
Autonomous Driving Background and Founding dotLumen
Harry: So tell me a bit more about your work in the automotive industry.
Cornel: I was at Continental, so one of the largest automotive companies in the world in Romania. My studies in Germany briefly went back to Romania, went back to Germany, back and forth. And that’s where I think the idea actually materialized a bit better. In terms of thoughts, not of the idea. But that’s where my observation was like, “Hey, I think we can do something more.” But that was the idea. My idea is worthless. It’s the implementation and everything and all the effort was spent.
I founded the company in the pandemic. It was the first day out of curfew when I founded the company. And then curfew came back one week after I founded the company. So it’s a full pandemic startup doing hardware and testing and medical devices during the pandemic after being horrible. But it really, really got very nice.
And it really grew very, very fast. And it grew at a rate not really pursued before by startups in my country. We were the first to take startup in my country and now they’re like three in total. And last month we opened [Nvidia] GTC. So we came to be incredibly proud and we were still running a startup. But then the first company on stage at GTC…
Harry: How did that feel for you?
Cornel: I’m going to publish a video.
Harry: Oh, are you?
Cornel: I have a second video. Somebody was filming me while I was looking at it. I’ll publish that. Maybe next week I’m going to publish that video.
Harry: I’ll definitely look out for that.
Cornel: It was really, really nice. There were 11,000 people in that room and I was watching it and now 13 million people have seen the keynote. It’s really cool. It just cements my belief. It’s a very hard work towards the right direction. It brings things and another thing which is immense. I really believe in planting a lot of seeds and see which ones grow.
For example, we have an internal example. We won a tremendous amount of awards and we were part of so many programs and we network in so many places. And the way we got there is because we continuously check what opportunities are there. So we actually have a rigorous monthly process where we check what opportunities are in the world and then we decide yes we like, we don’t.
But it brings so many, so many great things. And out of this process, some of the things which were amazing for us were, for example, when we go and become the first startup in my country to win funding from the European Innovation Council. So they gave us 9.7 million euros and a blended finance of 9.7 million. And that was the result of this process. The company was not one year old when we wrote that application.
Then now we have gotten to the EIC Scaling Club which is basically EU’s effort or EU’s bet. The European Union has made a bet of 120 startups, deep tech startups, that they will become unicorns in 2 years. We are one of them. And only 48 are announced at this point. More will be announced by the end of the year, but we are part of the first 48. And it’s, you know, again it’s an opportunity which happened by the network we have and everything and by being well known. So we do plant a lot of seeds.
At the same time I love to have the opportunity to say no. So when you only have two opportunities a year you have to say yes to both, but when you have a hundred opportunities a month you have enough options to say no.
We actually prefer saying we have that sort of opportunity, so we do invest in getting opportunities and everything and network, we network a lot. We could do even more, but now we network a lot, a lot of people know us and that’s a good thing. Because it’s so hard what we’re doing that we need all the support in the world and we need to find these people to offer us support.
I am aware of people who wanted to go in the field in which we are, so as competitors. And we discussed a couple of times that now they are our partners, they are working with us. It’s really, really amazing where we can get.
I will end with this. Jensen (CEO and founder of Nvidia) was asked last month, “with everything you know today, would you go back to the day you founded Nvidia?”. He said absolutely not, because it’s so much work and it’s so much pressure, work, fame, humility, all of it.
It’s so tough what we’re doing, so tough. I was discussing yesterday another interview for Romanian Press. In Romania we have this issue, but not only Romania, it’s everywhere I think. We see a lot of students starting startups, I completely disagree, students should not be starting startups.
It’s not a game of young people, it’s a game of experienced people. Just because we have some exceptions (Apple, Meta, Google), that’s not the norm. The most successful companies were not founded by people which were young. Because you need so much experience and you need to be through so much and you just can’t learn it on the work.
When I started the company I was 26, 27, I’m 30 now. I’m by far the youngest person in this game. My country has produced maybe three startups which are internationally known, we are one of them. I’m by far the youngest, 10-15 years younger than everyone else. Even they don’t feel they don’t feel ready for this.
This is one thing I see, so many people just starting and quitting in six months. The only thing which I found, the only recipe for success which I ever found is not only my research, it’s resilience. You can be anything you can have any better, it doesn’t matter. Resilience is the only trade which I found for successful entrepreneurs.
That moment when you’re alone at 2am in the office crying and still excited to get back to something, you need to have that. Otherwise you will never do something, you’ll never be successful in this field. And that’s perfectly ok. Any rational person, this is not a game for rational people. Rational people are taking a defeat and nobody can accuse them of anything bad. This is only a game of irrational people which want to sacrifice their lives and towards building great stuff. This company has a few of them.
How the Glasses work
Harry: Can you tell me about the glasses and the processing pipeline powering them?
Cornel: It’s the same technology regardless of whether you have a headset or another thing, but I will give you the example of the headset. The headset has six cameras, the dotLumen glasses as we call them, it’s a headset we call them glasses, it sounds better.
You have six cameras, we use those cameras to understand the world, so we have three of the cameras we watch near field. By your feet up to a few meters in front, and then we have the far field cameras which see up to 10 meters away in front. We use them to perceive depth, so other people will be using LIDARs, we don’t use LIDARs, there are no small LIDARS which will work out at this point.
So cameras are still better, stereoscopic cameras, it’s like we have two eyes, the same for cameras works better. In this use case, we can integrate anything, but we selected this for this use case. We take information from all of those six cameras 90 times a second, and we take the information and we do a bunch of processing.
Geometric processing is one of the first things, so we understand the world geometrically, where is the ground, what obstacles are above and what obstacles are below it. This is something which was done in the robotics world previously, we optimized some of these because in the use case of the head, very comparable to the use case of the robot, in a robot you have a constant sight from the ground, the sensor, you know where your sensor is.
The problem with something you wear on your head, you know, the head moves with so many degrees of freedom, the height changes depending on what you’re wearing, depending on the period of the day, everything changes, much more difficult to do that. But still, the robots, that’s what typically they do, and a pitfall of the robotics world is that they assume if the surface is flat, they can drive on it. Now, if you think from just a geometrical standpoint, a lake is perfectly flat. But you cannot drive on it.
That’s where we integrate AI. We use AI tremendously to understand the world semantically, so we understand, okay, this glass is a sidewalk, this glass is a road, this is a lake, this is a water, this is a mud, this is terrain, etc. And we take all that information and we combine it, and we also understand that, you know, a robot might not be able to go on stairs and stuff like that, but at least we understand where there are stairs, where there are curbs, which can go up, which can go down.
A lot of the things which we do as visually capable individuals, they are so hard to replicate, but we’ve done a lot. And then, with all of this technology, all of this understanding of the world, then we path plan, we understand where we can take you to. And we then do that for a couple of seconds, we compute where we can take you through, and with all the particularities, so if you are doing perception, like a processor, it will tell you which way to go, which options you have, you can try left, you can go forward, or you can pass certain objects or details, it will tell you about it, so you can hear them, the glasses will talk to you, and you can talk to them back. Right now, it can talk to you in 70 languages, but you can only talk with it in English. It will take a while for other languages, but we’ll get there.
Illustration of the processing pipeline. Credit: .Lumen
At this point, we have three ways of communicating with the device. One is voice, so you can talk to it. That’s most probably the least used from all the testing we know. How often we actually talk to your earphones. We have a button interface with audio menu, so we have some buttons on the device, you can hear the menu, you can go to the basic commands, or if you want to go to a complex destination, you go on Google Maps, which usually people do, they go on their smartphone on Google Maps, or whichever app they prefer, they find a destination, they press share, and they can share it with the glasses, and then the glasses will take them there.
Yeah, so getting back to the processing, that’s like the basic building blocks, on top of it we do a bunch more stuff, so in the way we create a more beautiful experience, a more useful experience, that is something which is on top of it, and there are a bunch of interesting interesting things which we have created, but not all of them are patented.
So yeah, that’s basically how the glasses work.
Empowering the Blind
Harry: I saw a video from CES of that, there was this guy using it for the first time, and I can understand why it’s gone viral. It’s amazing to see people using it.
Cornel: You know what’s interesting about that video, it wasn’t one, it was like 1% of the features of the glasses, like the building blocks were running, I mean, you can do so much more, but just those simple videos went incredibly viral.
That video was shared by some influencers, it got in total around more than 30 million views. It’s absolutely amazing, it’s such a simple video, it was filmed, like, it was nothing planned, the person was there, okay, take the device, and suddenly one of our members was filming it.
It’s a nice video, I think it’s very good. A lot of videos which are professionally done and we haven’t published them, we will publish them, but they don’t have the same, like, amazingness as the user-generated content, like, here, it just works.
Harry: That’s what I couldn’t believe is, you know, someone putting it on for the first time, because, sure, people can learn to use stuff, but wearing something for the first time like that and being able to use it so effectively means it must be phenomenally intuitive…
Cornel: I mean, we take people, we blindfold them, and we just give them a device and they do it, I mean, we have people, obviously for people which are visually capable, to be blindfolded, it’s a bit difficult…
Harry: Yeah, because they’re not used to it
Cornel: Yeah, yeah. They go very slowly until they get empowered, but for people who are visually impaired, people are used to it, and then we give them a device and they’re just running with it.
So we’re gonna, we’re gonna publish it in the videos. And it’s, it’s pretty amazing what we do here.
Building dotLumen and Looking into the Future
Harry: Jensen (Nvidia’s CEO) spoke at the Nvidia GTC 2024 keynote about the exponential growth of accelerated computing power, what is your vision of the future of AI? And more specifically, I guess, in your field?
Cornel: The thing which I recently saw, and I know also from the inside, what’s happening with autonomous driving, whether it’s pedestrian or not, it’s very interesting. We build ourselves tools, we build ourselves some tools, we build ourselves some very large models, which can do a lot of things. We have our own architecture, our own models, our own data set of millions of images, hand-labeled. It takes 5 to 30 minutes to do one photo, we have a million plus.
Harry: How did you, how did you, how did you accomplish that?
Cornel: We hired a lot of people to hand-label, here in Romania, and we did. We have data from 20+ countries, from 3 continents, and the models are, the models have our custom architectures trained on our own data, and, you know, it’s data worth millions. And, I still remember, like, a couple of years ago, we were discussing that, hey, one day, we were using public data, but it wasn’t really working, because there’s no real data on the pedestrian side, so a lot of the road, you move the point of view from the road to the side, it doesn’t work anymore properly. And, like, okay, we’ll develop as good as we can, and one day we’ll need to do data, we’ll need to do data-labeling.
Harry: What is the most significant technical challenge that you had to overcome in building the dotLumen glasses?
Cornel: Um, it’s not just one, but I can give you a sort of technical challenge. When we started to build them, it wasn’t technically possible. It was compute, it was the models, and we didn’t have data. When we began building it, we began building it on a bet that the technology would catch up, and we would be able to do what we do.
The first prototype, which didn’t have enough computing power to handle it, were three and a half kilograms, and the battery was not an hour. So three years ago, we couldn’t make it.
We didn’t have enough computing power and it was a three and a half kilos. And in just less than three years, we got to 800 grams, roughly the same weight as the [Apple] Vision Pro, much more computing power. And that’s the hardware challenge.
The software was even more intense. When we started, the model architecture couldn’t handle the MOU scores, which we needed in regards to the data. We didn’t have enough to handle real-time with six cameras, and multi-modal and everything else, which were nothing. And we tracked everything, and nobody was doing what we were doing. From a technical perspective, not from a use case, nobody was trying to do what we were doing. So we had to build it all. And now it works, it works amazingly, and now we have a full day battery life.
Harry: How did you get an investment? It was such a big bet. I mean, like you said, people like to plant seeds and stuff like that, but how did you really communicate to investors what you were trying to do?
Cornel: It would have been so much easier to be in the United States than in Eastern Europe. Having a great network was critical.
Harry: Who funded you at the early stages? Was it an angel?
Cornel: Angels and myself. We grew the valuation of the company 10x in the first six months, and then 4x in the following year. So, the first angels, they got 40x or something like that. I’m approximating, but it was a huge number. And I think that we’re still only beginning. What we can do with the pedestrian autonomous driving, where we can get to, it’s really limitless. I see a future in which all the tasks which require pedestrian mobility are automated, are done by robots, and those robots are guided by our employees.
Harry: When you were talking about robots earlier, you mentioned humanoid robots. Do you envision humanoid robots as opposed to wheeled robots?
Cornel: No one knows how to guide them. You know what, the funny thing, we are going to know how to guide humanoid robots. Because we’re going to know how to guide humans outdoors. So while others build robots, we will guide them. The entire infrastructure is built for legged things, like, for example, robot dogs or humanoid robots. Those are some of the directions which are incredibly, incredibly sexy, because it solves some problems. Right now, you delivery robotics companies. One of them, I know, personally, is very, very nice, nice, incredible engineering. But they are based on wheels. They can deliver the parts from the house. But when I order a pizza, the person who brings me the pizza comes to my door. I want that. And we know how to do that. Not helping the robot put a foot on the stairs, on the steps, that’s not our job. Others are doing it and they’re doing it well. We know how to tell them where the stairs are.
Harry: Interesting. So do you see yourself collaborating then with someone like Boston Dynamics or Tesla? Or do you see you building your own in-house robotics hardware?
Cornel: No, I don’t think we’re going to build other robotics. I think there are people much smarter than us who do that. We know how to do pedestrian robotics. I think we want to offer developers.
Harry: And what do you think is the most exciting, you know, humanoid robotics? Because Boston Dynamics for a long time has just blown my mind, but there’s lots of them now though, isn’t there?
Cornel: There’s lots of them. I mean check Nvidia project Groot and what they are down there and they integrate with multiple robots. I like their approach and we have a good partnership with Nvidia.
Want to learn more?
Check out:
Nvidia GTC presentation by Cornel about the technical aspects of the glasses
dotLumen’s website and LinkedIn
Cornel’s LinkedIn (he’s on tour at the moment so theres plenty of content to watch out for)
