At DVN, we want to keep the ADAS/AV community informed about sensor technology, architecture, and applications. Our second DVN conference on this topic will take place in Stuttgart on November 16-18. For the first time, there will be special sessions on dual use and “Road to Type Approval: Mastering E2E AI Systems.” In preparation for this event, we are presenting important players and their results/opinions via exclusive interviews in this field in our newsletter.
The first contribution to this series is a report on a preview drive with the latest Wayve vehicle. Further interviews will follow every 14 days until the start of the conference.
In this episode, we would like to introduce one of the big players in this game, NVIDIA.
DVN-Interview with Prof. Dr. Ralf Guido Herrtwich, Senior Director Automotive Software at NVIDIA in Germany, about NVIDIA plans and strategy on ADAS/AV.
DVN-Dickmann: If you were to describe NVIDIA’s offering in the automotive sector in one sentence, what exactly do you supply to OEMs and mobility providers?
Prof. Herrtwich: We provide AI solutions to the automotive industry from car to cloud, particularly for autonomous driving. This includes development tools, high-performance systems for training, platforms for simulation and testing, and, of course, the in-vehicle compute platform.
DVN-Dickmann: And in terms of product portfolio or markets, you address fleet vehicles as well as passenger cars, so the private sector, equally?
Prof. Herrtwich: Technologies such as L2, L2+, or L2++ are becoming increasingly universal and effective. Elevating these vehicles to L4—so they can also operate in fleet environments—appears to be a logical next step. The transition from L2++ to L4 primarily requires elevating safety concepts and system redundancy to a fully fail-operational level. The perception and reasoning AI are not fundamentally different; the key difference lies in the underlying safety architecture and redundancy. Fleet applications may additionally require teleoperation capabilities on top.
DVN-Dickmann: Okay, so for you, it is basically a “one-fits-all” solution that only requires minor adaptation? Have I understood that correctly?
Prof. Herrtwich: It is better described as a layered architecture. The AI core remains consistent, while different safety structures are added depending on the application. We refer to these safety layers as ‘NVIDIA Halos’—they act as safety nets beneath the AI.
DVN-Dickmann: A question that always comes up is: to what extent are you hardware/sensor agnostic? You always have Hyperion, or whatever the nomenclature will be in the future. But you would presumably also be free to serve other OEMs who have a completely different sensor set?
Prof. Herrtwich: Regarding the SoC, please, forgive us that we rely on NVIDIA hardware. It has evolved across generations while remaining compatible. For sensors, we are more flexible. However, introducing a new sensor requires new data collection and retraining, which significantly increases cost. Our Hyperion approach therefore defines the required sensor types, numbers, and placement for specific applications and ODDs and recommends configurations of proven sensors we and our OEM partners already worked with. The cost advantage of reusing validated sensors is substantial unless there is a compelling reason to change.
DVN-Dickmann: You have Hyperion 10 as your current sensor set. With that, you manage all L2 to L4 ODDs that are currently pending. If something in sensor technology were to evolve on the market (e.g., from 16×16 to real imaging radars with 32×32 antennas). One could rethink the sensor setup with that. Would you as NVIDIA use the optimisation opportunity and finance the delta learning yourselves? Or would you say, I would try to push this Hyperion 10 in re-use until we hit hard scenarios or safety limits and then rethink?
Prof. Herrtwich: We generally embrace innovation. If a technological step genuinely improves performance or cost, we will adopt it. However, focusing solely on the unit price of a sensor can be misleading. A small hardware saving may trigger significant expenses for new data collection and validation. Decisions must be evaluated at the holistic system level.
DVN-Dickmann: Perhaps that’s the link to the next question: Once you as NVIDIA have established L4 and can master the required ODD, is that it for L4, or will the ODDs continue to evolve? In a few years, you would be as good as Waymo and all those who have been doing this for decades longer; how do you see yourselves there, and is there then a necessity to rely on new sensors, for example, to differentiate yourselves? Or is differentiation only achievable via AI?
Prof. Herrtwich: In recent years, advances in AI—particularly through Large Language Models and Vision-Language-Action models—have significantly increased capabilities. Beyond object perception where we label world objects (a lane, a tree, a person, etc.), we now see reasoning models trained on causal chains, such as understanding that a ball rolling onto the road may imply a child following. Labelling and training such scenarios was once extremely difficult, now we have tools that facilitate it. All in all, the improvements in reasoning have had greater impact than incremental sensor upgrades over the past years.
DVN-Dickmann: Allow me to follow up once more. Regarding sensor technology, we have now reached a state that already enables you to be competitive in the L4 market today. Improvements in driving functions come more through software. This raises the question: “Why should companies still work on improvements or new ones?” Is that your message?
Prof. Herrtwich: One major challenge remains the integration of sensors into vehicles. Instead of focusing only on higher resolution or longer range, the industry might want to emphasize better packaging, cost-effective integration, coordinated field-of-view optimization, and potentially reducing the overall number of sensors while maintaining performance.
DVN-Dickmann: How do you see sensor development and sensor setup in the context of redundancy? Keyword: Vision only?
Prof. Herrtwich: Redundancy remains essential, including across different sensor modalities. While there has been a stronger shift toward cameras in certain phases, a pure vision-only approach for L4 is unlikely.
DVN-Dickmann: Regarding the topic of redundancy and plausibility checking of sensor data, does the central architecture help there?
If we send raw data to the CPU instead of a point cloud, then all sensor inputs can be combined in any way. And also be treated differently algorithmically. So, they can be viewed as statistically independent inputs. Doesn’t that also allow for the sensor set to be optimised? Thus, cameras and radar could do the job and the expensive infrastructure costs of Lidars could be saved. How do you see that?
Prof. Herrtwich: We strongly favor processing raw sensor data because filtering always reduces information content. Advances in compute performance now make centralized raw data processing feasible. This allows sensors to remain simpler. Whether LiDAR can be entirely eliminated depends on the application; many vehicles can operate effectively without it, but we are not declaring it obsolete.
DVN-Dickmann: In terms of redundancy, would it then be a perfectly plausible thought for safety reasons to move away from the former logic that at least two different sensor technologies out of three must provide plausibility?
Prof. Herrtwich: Sensor-level redundancy is important but not sufficient for L4. Our architecture combines a reasoning stack that generates human-like trajectories with a classical planning stack that ensures safety. The reasoning stack is, if you want, the creative component of the system.
Now, this creativity is—as we all know from AI systems—sometimes a bit excessive and leads to situations that might not be safe. We had to think about how we handle such cases. We then came up with something very simple: wait a minute, from our previous work we already have a software stack for autonomous driving that is implemented classically, for example in its planning components. And which delivers a safe trajectory. The problem with this classic stack is that it always drives correctly, but sometimes not entirely comfortably.
What we do now is take the trajectories coming from our reasoning model and filter them again against what our classic software stack would do. This means the classic stack is fed with trajectories that it would never have come up with on its own. But about which it can, of course, say in every case whether they are safe to drive.
DVN-Dickmann: That means the final decision-making instance is then ultimately traced back to classic software?
Prof. Herrtwich: It functions as a safety filter. Today, it may intervene in a good percentage of cases, but this rate decreases as the reasoning model improves. Nevertheless, architecturally, this safety layer remains fundamental—like a safety net beneath a trapeze artist.
DVN-Dickmann: But that is exactly the exciting point. Other firms that are strong proponents of E2E architectures argue: I no longer need classic signal processing. If I have understood correctly, you use the classic algorithmic approach as the final instance. Does that mean the innovation bottleneck is the AI approach with a 40% error rate, comparable to LLMs?
Prof Herrtwich: It would be if we were really talking about 40 per cent. But we aren’t. We are assuming 98 per cent and we filter perhaps 2 per cent. And in a few iterations of the system, we will only filter 0.2 per cent in the future.
DVN-Dickmann: As a consequence of the announcements at CES ’26, there were questions regarding the hurdle to a rapid market introduction of L4 systems: registration/type approval. One point in this is the view of E2E software as an AI black box. And the checking, scene-generating instance is partly an AI black box, too. How can one argue against these concerns of “black-box ping pong” to public authorities such as the Federal Motor Transport Authority (KBA)?
Prof. Herrtwich: It is not truly a black box. Training data is curated and traceable. At CES, we open-sourced not only software but also data and simulation tools. Transparency strengthens trust and understanding in the development community and among authorities.
And one more thing: your notion of a “black box” implies that one would not know what went inside it. That’s not the case. Training data undergoes heavy curation. But its generation has become so much easier. I recall that Mercedes at one point initiated a major campaign to collect data of deer crossings. As they could not find much in the wild, they eventually resorted to taking a huge stuffed toy dear to simulate these scenarios. Now, it is so much easier to generate such data in the machine.
DVN-Dickmann: Ah, okay. Does that also mean, in terms of competitive differentiation, that while the AI can do a lot, it only generates from what it has been trained on. It can’t do new things, or “know the unknown”—completely unforeseen things. Therefore, is it the personal experience of the NVIDIA developer team that will make the difference?
Prof. Herrtwich: Experience remains decisive. Autonomous driving systems cannot be created by simply prompting an AI. Developers must understand the problem space and use the appropriate tools. NVIDIA provides not only training, simulation, and in-vehicle compute platforms but a comprehensive ecosystem that allows developers to guide AI training toward critical edge cases. The input space now includes real-world data, synthetic data, and curated internet data, particularly for causal reasoning scenarios.
DVN-Dickmann: Final question: beyond 2030, where do you or NVIDIA see the biggest market? L2+, L4 fleet, or L4 private sector?
Prof Herrtwich: I always think there are reasons to have your own car and there are reasons not to have your own car. For private vehicles, autonomous capability will increasingly be expected as costs decline with scale. While not every vehicle may include L4, broad adoption is realistic. How OEMs package the functionality—standard or optional—will remain their strategic choice.
DVN-Dickmann: And in the fleet sector? Do you believe that will increase strongly?
Prof Herrtwich: The fleet sector will increase because the barrier to entry will become lower. As these functions become available in more vehicles, I can more easily—as someone who wants to operate such a fleet—simply buy myself a couple of automated vehicles to compose the fleet. I no longer need to invest in developing all the automation technology. But I can decide whether I then operate this fleet in isolation or make it part of a larger system such as Uber. In the ten years you mentioned, the technical hurdle for this will be gone.
DVN-Dickmann: Wow, I have learned a lot during this interview.
Ralf, thank you for the deep insights into NVIDIA’s philosophy and the outlook for the future. I hope we will be able to welcome NVIDIA representatives at the DVN conference.
