Behind the Largest Human Brain Reconstruction to Date | Prof. Jeff Lichtman
Hello and welcome to this interview with Jeff Lichtman, the Harvard University neuroscientist behind several highly ambitious projects to digitally reconstruct the brain’s neural wiring at an unfathomable scale. His work is pushing the boundaries of data-intensive neuroscience with AI. In close collaboration with Google Research, Jeff and his team recently reconstructed a cubic millimeter of human brain tissue. In this interview we delve into this fascinating work, as well as the importance and nature of Google’s involvement, the challenges of working with such vast volumes of data, and even a sneak peek into into some mind-boggling future projects. This includes attempting to reconstruct an entire mouse brain, approximately 1000x larger than their latest achievement.
Unfortunately, a short part of our introduction wasn’t captured due to recording issues. During this missing section, Jeff describes his work toward building a complete map of the neural connections in the brain. Called the connectome, this concept is similar to that of the human genome project.
Throughout his career, Jeff has focused on developing innovative imaging techniques to study the structure and function of the nervous system at the synaptic level. He is particularly known for his work developing the Brainbow technique, which enables the visualization of individual neurons in the brain with distinct fluorescent colors and has revolutionised the study of neural circuits. More recently his work has focused on using serial electron microscopy to reconstruct neural circuits with nanometer resolution, providing unprecedented insights into the complex wiring of the brain. Our conversation begins with Jeff discussing the path that led to his latest project in brain reconstruction. Join us as we dive deep into the fascinating world at the intersection of cutting-edge neuroscience and AI.
Introduction to Jeff’s Research
Jeff: I’m basically a one-trick pony. I’m doing that same experiment on ever larger and more complicated data sets, but doing it largely to understand how in development the wiring diagram gets established, because as you may or may not know, that early on the wiring diagram is quite different when mammals are born, including human beings, than the way it is later on. And a lot of the change in the wiring diagram is due to the elimination of a subset of connections and the selective enhancement of certain connections as other connections get pruned away. And so that’s basically what I’ve been studying my whole life.
At the beginning, I was doing it with physiology, then with what we call XFP mice, mice where a fluorescent protein was expressed in some subset of cells. And then I began crossing those cells to get multiple colors and then finally made these mice called brainbow mice, which were really amazing, but for very fine structure, they didn’t have enough resolution. So then I switched over to serial section electron microscopy, which I had dabbled in earlier and realized that this was a way to get enough resolution.
Then I partnered with Google just by accident almost.They were looking for a big project and we had just finished a big volume and they said they’d be interested in trying their hand at whether they could segment out learning, do all the stitching, alignment, segmentation, and ultimately synapse annotation on the big data set. So we did the data set that you’ve just read about. And it’s about 10 years of effort.
Harry: It’s absolutely astonishing. I mean, you know, this is exactly the kind of thing that inspired, I mean, I remember one of my first lectures, I saw the brainbow and I became mesmerized by that. And, you know, it’s these things like optogenetics and all of these really exciting things make neuroscience, I guess it’s the sexier part of neuroscience, but it makes it.
Jeff: I mean, it’s kind of it’s interesting because I see such a fundamental difference between tools like optogenetics and tools like brainbow. Maybe it’s worth just saying a little bit about what I see as the fundamental difference. And that is that this is a tool that generates data, but doesn’t test hypotheses. It generates hypotheses, but it doesn’t test them. Whereas optogenetics is basically to test an idea. You know, you stimulate an area you think is important or you block it, depending on which optogenetic probe you’re using. And then you see what happens.
Whereas brainbow and the seroleum is really descriptive where we’re not manipulating anything. I mean, if we are manipulating it, we’re not doing it on purpose. And then from this data, you look for things that are surprising, things that are not the way it is in the textbooks. And that is how you move forward is that those odd oddities you see force you to generate new hypotheses to change the way we think about the brain.
Differences in scientific approaches
Whereas optogenetics and tools like that basically say we know how it works. We have a model of how it works. And now we’re going to do a test. And you only really publish those tests when they’re compatible with your model. If they don’t work, it’s usually considered negative data. And, you know, there’s an infinite number of reasons why an experiment might not work.
So you only see the successful cases, but the successful cases only perpetuate the theories you already have. So it’s not a very for me personally, not a very effective way to move forward. So I’m very much in favor of descriptive neurobiology, just as, you know, Darwin or the Hubble telescope or the James Webb telescope. I like that. Looking at things, then you start seeing things and then you have to come up with an explanation.Usually it’s based on what you’ve observed, not on what you hoped to observe.
Research findings
Harry: I think that’s a great way of putting it. And you did discover some quite interesting findings from your research. You discovered some new neuron types.
Jeff: Yes, neuron types and different strength, these super powerful, very rare connections, which we didn’t expect. And all these other things. I think what we found was mainly proof of concept. The main point of this study is to generate a data set that is so large that no human being or no laboratory can sort of say we’ve picked all the high level results and now we can pick the bones. If you’re another lab. That’s so large. There’s an unlimited amount of things you could do with it, which we did a minuscule number. And in several cases, we’ve already found things that were not consistent with the world view that’s in the textbooks. And that to us is suggests that there’s probably a gazillion other things in this data set and subsequent data sets that will be forthcoming, you know, that will be, you know, transformative in the sense that they’ll force us to have to reconsider what we think how the brain works.
So, you know, for me, personally, the most interesting result was the last result described in the text of the main results, which is that among the 99% of connections between axons and dendrites that are very weak. There is a very small percentage somewhere between one and 500 to one in 1000 axons make disproportionately large numbers of synapses on one of the many target cells that acts on innervates.
So an axon, and it’s not just a special subset of axons, as far as we can tell that any targets that we look at will receive a huge number of weak connections from axons and yet there will always be a one or two outliers. And the, if you trace those outliers those axons don’t always make strong connections. In fact, they also only make weak connections except on one or two target cells so it’s as if the nervous system has in the cerebral cortex, two modes of action. One is related to things like temporal and spatial summation, where it’s the synchronous activity of multiple axons that may be distributed in multiple places on the neuron that are necessary to bring the cell’s membrane potential at the initial segment of the axon to threshold.
That’s the standard view of how information flows through the nervous system. And that raises an interesting paradox that you don’t see written about much but it’s pretty obvious that if only weak connections are there, and the way you get a cell to reach threshold is synchronously firing, let’s say for the sake of argument, 10 axons that have to all release neurotransmitter at the same time. And that target set that cell when it fires an action potential only makes one synapse on each of its target cells to get one of those target cells to fire, you would need nine other axons that are synchronous with this one, each of which are driven by 10 axons themselves you would need 100 synchronous axons to get one cell downstream to fire. And if you’re talking about a cell downstream another synapse you would need now 10 times more of those in each of them would require 10 synapses. But ultimately if you had a relay that had 10 synapses in it, you would need 10 to the 10th axons synchronously to get that to happen and that’s more neurons than there are even in the human brain so I don’t think that is a very reasonable way to go.
It is quite interesting that in fact there’s another way, you can have a super powerful connection where one axon is enough to drive a cell to threshold. And then that cell once it’s driven to threshold has one target cell, where it has enough power to drive that cell to threshold. And then that target cell has one output, where it can drive its target cell to threshold, then you have a fast path to get information let’s say from your eyeball if you’re driving a car and you have to take your foot off the gas pedal and put it on the brake when you see a red light. You just, you don’t have to think about it. There’s no summation there’s nothing going on you’re going to be listening talking to somebody listening to the radio doesn’t matter. It’s post cognitive you’re not even thinking anymore. And maybe that’s what we’re seeing in these brains is the things that haven’t yet been learned, which is the substrate for all future learning, and then this, you know, amazing subset of very fast powerful connections which are the memories that are now instantiated in the brain. So, that to me is really interesting, because I don’t know the better theory for how the brain can do things quickly. Once they’re learned. And that would have maybe even some input and insight for the way one thinks about artificial networks, you know how you might do that, as opposed to just changing the weights of connections. You could trim away all but a small subset of them. So, it’s interesting.
Collaboration with Google
Harry: So tell me a bit about the collaboration Google because I find that very interesting and how Google’s technology must have because I mean, the amount of data you’re talking about here 1.4 petabytes of data. And just tell me how this collaboration Google works and how this is enabled you to how that technology enabled you to do this work?
Jeff: I mean it’s been really a fantastically useful and wonderful collaboration in all respects. Google I know as a company and it makes money, but I’m dealing with a group there that is just very dedicated to solving hard problems, not for some commercial. But I think it’s part of the DNA of that company that they believe they make their success is partly because they don’t shy away from the hardest problems. And this is how they learn to be a better company. And so for one reason or another, there, they were eager to do this project I didn’t offer them this project they came to me.
They knew we had big data, I guess, and they were saying that they thought they could handle this, but it was not trivial for them to do machine learning on 1.4 petabytes of data. It did require a lot of engineering on their end, and they did it. Then they did all the synapse annotation hundreds of millions of synapses, and we’re now doing many other projects with them because of this, that are even more ambitious than this one.
Future endeavours: the mouse brain and is it possible to reconstruct the human brain
Harry: Could you tell me about any of these projects?
Jeff: Yeah. We have done a connectome of a larval zebra fish. A larval zebra fish sounds like it’s very immature but this is a free swimming animal that catches paramecia it’s a, you know, it’s visual system is good enough that it can chase around the small protozoa and eat them, and they can do many other behaviors. So although it’s a immature fish, it’s not immature in the sense that it can’t fend for itself. It’s a free swimming independent animal. We did a connectome of an animal that had, at Janelia the Howard Hughes site in Virginia, already been studied by doing activity dependent labeling with calcium imagers imaging agents, all the neurons in the brain of this same animal, not the same species but literally the same animal had been studied as the animal went through 10 different behaviors. So there is this activity map of every cells activity for all these behaviors. And then we took that same animal, and we did the serial section reconstruction of the animal. And the goal is to superimpose now activity on top of the wiring diagram. So that project is in progress. The heavy lifting for us, the imaging, is complete. Now Google is doing the segmentation of the fish based on what we imaged and that’s largely complete. The synapse annotation is underway. And now there is the beginnings of trying to superimpose the activity map on top of the wiring diagram.
An event more new project is that the federal government has funded here. The first proof of principle projects to try to do a whole mouse brains connectome so instead of doing something that is petabyte or two to do something that is about 1000 times bigger, an exabyte.
Harry: And that’s something, do you think it’s possible?
Jeff: I sure hope so. We have 30 million dollars to do the proof of principle with Google and a number of other groups. I’m the principal investigator, but there’s a lot of groups and a lot of different universities plus Google that are working on doing the proof of principle only 10 cubic millimeters something 10 times bigger than this data set. But with techniques that we think can scale we have to prove that the techniques could scale. So it’s just a matter of buying more machines to get this to work. So we have five years to do the proof of principle if we’re successful. Then presumably the government would go out and find the money which would be a lot of money to do an exabyte. That’s the next big project.
Harry: And what’s the main limitation of that? Is it the compute or is it the imaging?
Jeff: Yes, yes. Both. I mean everything about this, you know, doing a petabyte, 1.4 petabytes was hard. I mean everything about it was hard. But when you start doing a whole brain then you even have to have the challenge of getting the whole brain well stained. That’s not easy either. You have to have a well stained brain and then you have to not lose any of it when you start doing the imaging and now you’re imaging so much data. Even things that happen very rarely, let’s say something that happens one every 10,000 times, that can be devastating if you’re trying to do a whole brain and you lose a piece so you can’t connect the wires. Because you’re going to have to do about 80,000 sections. This was 5,000 sections. You have to do 80,000 sections of something much bigger.
Harry: I mean, before this I wasn’t aware of being able to do image annotation on this scale with electron microscopy I’ve been following. I don’t know if you’ve heard of Gubra? They’re a company in Copenhagen and they do some amazing light sheet microscopy, whole mouse brain imaging and I find that mesmerizing but this is just on a whole other level.
Jeff: If this could be done with light sheet it would be much easier. It’s just that it doesn’t have enough resolution. Usually the light sheet is at best a micron thick and here we’re dealing with things that are 100 times thinner.
Harry: Looking to the future, what’s your vision of it being done on a human brain?
Jeff: You know, a lot of people are curious about, well, if you’re going to do a whole mouse, why not just do a whole human brain? And there’s a whole bunch of reasons why that’s impossible to do right now. One reason is the obvious one that the digital data set would be closer to a zettabyte than a hexabyte, so another thousand fold bigger. So you would be talking about, instead of a million terabytes, a billion terabytes or something like that, you know, it’s just, it’s basically equivalent to the digital content of the world in a year. So it’s too much data. The other thing is how to get a really well-preserved human brain. You know, the data that we’re using here is from an operating table from a living patient. Once a patient dies, usually death is determined by brain death, you know, when there’s no more brain activity, the brain is already changing. And so you would like to get a fresh brain, but there’s no way to get a fresh brain, and certainly a fresh, healthy human brain is harder to imagine. So I think for the time being, we are more than happy because it’s a big enough challenge to do a mouse brain.
I think maybe a more scientifically interesting question to ask is, at some point we will presumably not be surprised by what we see in these brains. That is, we will have done enough in mouse brains that no longer are we shocked because everything we’ve seen, we’ve seen before. That may be the best humans can do, by the way, when it comes to understanding, is to not be surprised, even though we can’t explain many things. And because we’ll continue to do human samples, the same will occur. At some point, we’re not going to be surprised. And then the question is, well, what’s the upside of doing a whole human brain as opposed to doing little pieces if there’s no longer any surprises? And that’s a really important question because maybe to get the pathways that connect different parts of the brain, you don’t need this kind of resolution. So you can get that in another way, and now you’ve looked at enough regions and you’ve looked at a whole mouse, so you’re not, there’s nothing surprising anymore. So maybe it would be just redundant. It’s also, you know, you’re not going to learn anything new. It’s also true that humans probably store in the wiring diagram a lot of their memories. And so everyone’s brain is going to be different.
It’s not that everything you’re going to see in that brain is fundamental. It’s the unique experiences of the individual. So for all those reasons, I’m willing to put off until I come back in my next karmic reincarnation because I don’t think it’ll happen in my lifetime to do a whole human brain. Not that I haven’t thought about it, and I think a lot of people have thought about it. And some of the people, like Ken Hayworth at Janelia, who used to be a postdoc with me, now works on these newer techniques to scale up to do a whole mouse brain. I think he is interested in doing whole human brains, largely with the hope that one might be able to, you know, upload someone’s consciousness in silico or even more way out is broadcasted out at the speed of light into the universe and hope some advanced people somewhere…
Harry: I’ve not heard that one, actually. Broadcasting… Yeah, I’ve not heard that. …reincarnate your brain. Wow. Do you think that if you could do a digital twin, do you think that would replicate consciousness?
Jeff: I don’t. I think, you know, part of the digital twin problem here is that a brain’s activity is stimulated by its sense organs. And so you would really have to recreate everything, including the retinas and the hair cells in the ear, and they’d have to work because the patterns of activity going through those wires is not implicit in the wiring diagram. You know, it depends on the ion channels in each of these axons and the dendritic computations that are going on related to non-linearities related to the density of ion channels and dendrites and how many neurotransmitter molecules are released at the various synapses. None of that is sort of easy to understand just by looking at a wiring diagram at one moment in time. So I think it’s going to be very hard to, you know, in the movies you sort of upload the brain and then the computer screen says hello. But how does the brain actually type? It’s never really clever. It’s letters come out of the brain and onto the computer screen because there is no interface there. So, yeah, I’m a little skeptical that we’re going to have, that artificial intelligence is going to be replaced by human intelligence that’s now in silico. I think that’s unlikely, but maybe I’m missing something, and I am old, so there’s lots of new ideas.
Harry: Have you had much interest in organoids?
Jeff: Yeah, I’ve worked on with Paola Arlotta, we’ve worked on organoids here. She had an organoid paper that we participated in that where the organoid, if you shine blight on it, the neurons would be activated because there were retinal cells in there connected to neurons. And, yeah, we did a serial reconstruction of the organoid to see if there were synapses. There were synapses in it, but it was so disorganized compared to a normal brain. It was very hard, you know, you couldn’t find anything recognizably that reminded you of a brain. But they were not, the density was very low, and the cell’s shapes were very weird. It was, after all, an organoid.
But I think organoids, you know, they’re not going away, they’re becoming very interesting, especially for looking at human cognitive diseases. Thinking organoids from humans who have various forms of either adult psychiatric diseases like schizophrenia or developmental disorders like autism is a cottage industry that’s being done a lot now, with the hope that something will be discovered that gives us some insight into what actually is physically wrong.
Harry: I think it’s fantastic that you’ve open sourced all of your data and all the information. Like you said in the paper, there’s so much that it’s a drop in the ocean what you’ve done, but what are you excited about for other people to do with your work, or what are you hoping that people will do with the data?
Jeff: Yeah, so I’ve already told you my view of inductive science, that is hypothesis generating science as opposed to hypothesis testing science, and I would like people to feel that there’s nothing wrong with doing description. So having data sets where you can describe, you can be an explorer, it’s like this is a Galapagos island that is largely unexplored, and here is the boat ramp, just get off the boat and just go in the woods, maybe you’ll find something fantastic. And I think that’s what I would like to see is people, including amateur scientists, not necessarily professional neuroscientists, just explore. For example, lots of amateurs who are not astronomers are fascinated by the James Webb Telescope. So imagine if there was a website somewhere where every picture they took of the universe is available and you just look around, and here you can look around and click on something and render it in 3D, and you can click on its neighbor or an axon that’s innervating that cell. It’s amazing what you can do. The amazing thing for me is that here is 1.4 petabytes, and it feels like it’s a miniscule small data set.
Harry: Have they actually rendered the whole three dimensional image in one view? Have you seen that?
Jeff: We have made a render. But one of the reasons their approach works is that when you want to render the whole volume in something the size of your screen, it’s very low resolution. It looks great. And then if you zoom up, I want to only render half of it, it will use twice the resolution, but I’ll show you half of it. And then if you want to zoom up even more… So the amount of data it’s showing you, whether you look at the whole volume or just one dendritic spine, it’s the same amount of data. It’s just a clever… So you always think you’ve got all this data there, but when you’re looking at it so you can see the whole thing, it is 1.4 petabytes, but you’re actually rendering it…
Harry: You mentioned your interest in space a couple of times and I resonate with that. And I don’t know if it’s just a neuroscience thing, but are you quite inspired by it?
Jeff: I love descriptive studies of anything. I find paleontology, again, completely… There’s no manipulation. All you’re doing is digging up bones, reconstructing ancient animals. I find that very interesting. I think you would call a person like me a naturalist, but it’s basically the same feeling that people who are explorers have, which is my exploration is of the natural world. I’m more interested in that than looking at buildings, for example, or even being in an art museum. I’m very interested in the natural world, but whether it’s bones or brains or… I don’t know a bee for something out in the cosmos, but big galaxies, they’re all equally fascinating to me. It’s just I happen to know a lot more about brains, so my questions are a little more in depth there. Fundamentally, I think there’s a kind of person, and I’m just one of a gazillion people like this, who are just curious about what the world actually is. In my view, the best humans can do is describe the world until finally we’ve described everything. We’re far from describing everything, but that’s very different from the modern scientific view that we don’t understand something today, but if we work a little harder, we’ll finally have the mechanism and we’ll do the experiments and we’ll understand. I think understanding is overrated. I think it’s almost impossible to understand a brain. It’s just too complicated. It’s beyond comprehension. I think we’re not going to get to a place where some day you say, “I get it. I know why every synapse is where it is.” I don’t think that’s going to happen. It’s like explaining London or New York City. You can’t explain it. What do you mean, “understand London”? There’s so many things happening at the same time, and my view is if you can’t understand London, forget about trying to understand the brain because the brain is far more complicated.
