Depth and Breadth for an Efficient Brain: No Short Cuts

Maturation of a neural network. Over time, new nodes are formed 

with their respective connections, and existing connections are 

strengthened. The overall system, still maintains a small world 

structure, and its original base structure.

What are the sorts of activity over the lifespan that shape the efficiency of our brains? Short of taking a pill or undergoing microneurosurgery, how do we engage in cognitive processes that encourage favorable levels of neurotransmitter activity and optimal configurations of neural connectivity? For that matter, is it possible to bypass all the "hard work" of thinking and doing and just pop a pill to make our minds more intelligent? To preempt the latter question, perhaps there is no short cut. But that does not mean we give up. Rather, it means it is all the more critical to lay the right foundations, and it also means, it is never too late to start.

A recent development in our understanding of neural structure might be mapped onto this set of physical properties. Based on graph theory, we now know that the way in which the human brain is wired resembles a small-world network. That is, neurons are connected to each other in the brain such that there is an optimal balance between short-distance, local, connections with close neighboring neurons, as well as long-distance connections via hub neurons. This balance of having both types of connections results in the most efficient structure with which information can be transmitted from one neuron to another, on average. Too many local connections, and information must shuttle through an adverse number of short-range synapses before reaching a distant neuron, increasing time of transfer. Too many long-distance connections, and also information must ridiculously pass through distant neurons before arriving at the neuron which is just beside. Other properties emerge that also are used to characterize the degree to which a network is a small-world network - level of clustering and randomness of connections. Using such indices, we now know that the evident connectivity of the brain seems to represent a high-level of efficiency with regards to the processing of information pertaining to stimuli, memory, thought, and action. Because of such neural organization, we are able to read or hear, comprehend, remember, reason, and respond, all literally within the blink of an eye.

With this background, we come back to the opening questions. If our brains are generally already efficient, how does this efficiency change with age, and if it goes down (as we are apt to assume), how do we keep it at optimum efficiency for as long as possible apart from the use of chemical and physical interventions? How do we optimize our small-world networks via mental interventions?

Origami and the Brain

Origami. The art of folding paper into shapes using a single sheet of paper without tearing or cutting. Perhaps, at an abstract level, this may be likened to what our brains do. We have one brain. We can't make big changes to it, like take one part of the brain and manually "connect" it to another part of the brain. Rather, we have to work within the limits of certain neural connection rules to establish a certain way to get to an end state.

For example, some rules may be related to the fact that our neurons have many short range local connections with neighboring neurons, as well as, some long range connections to more distant groups of neurons. Establishing and pruning these connections is dependent on time and stimulation from external as well as internal events. These events can be cognitive or biological or physical (e.g. the intention to retrieve a memory, or some neurotransmitter regulation, or some visual energy input, respectively). Within this system, our brains try to represent external information, and to generate certain actions or responses.

In a similar manner, in origami, each fold is like an imprint of an event that happens. The effect of folding, however, is limited by the thickness, elasticity, and size of the paper, as well as the force of the folding. Folding could be a sharp strong crease, a light depression, or a curve. Folding also occurs along specific lines or regions on the paper at a time. Finally, folding has temporal order. Through a combination of these factors, the paper encodes what forces have been exerted on it, and represents all of that in a particular physical form. The end state.

The end state maybe be a meaningful shape, or it may have a meaningful function. We can transform a simple piece of paper into a form of a crane, or a box, or a really complex shape (origami experts have been able to do wonders!). We can even use the tension inherent in the folded paper as a spring with tremendous kinetic energy when released. We can also use folding to allow a large piece of material that ordinarily would not fit in specific area to conform to the shape and therefore fit in the area.

Likewise, the brain performs an interesting function in incorporating sensory information from the physical world and representing all the rich material within a single piece of organic tissue. This "folding" of information from one state to another may be a framework to understand neural function.

Consider that we can quantify the physical forces and characteristics of a piece of paper and its folds. Based on low level parameters, we can then determine what the origami will look like, what it can do, what properties its resulting form maintains. Applying a similar method to parameterize neural function may allow us to better describe how the properties of the brain relate to cognition and behavior. For example, the ease with which a paper folds may be dependent on the thickness of the paper (for a given material elasticity/rigidity/brittleness). This will in turn determine how much force must be applied to the paper to achieve a fold of a certain angle. In the same way, one property of the brain may be how strong the connections in a certain neuronal region may be. The stronger the connections, the easier it may be for a signal in one region to affect the activity in another. Another case in point, the brain maintains a certain level to generate new neurons in key parts of the cortex. Neurogenesis is known to occur even in late adulthood in the hippocampus and the peri-ventricular walls. Importantly, recent studies have shown that neurogenesis may be helpful in overcoming drug addiction. A possible mechanism might be that the new neurons enable the brain to represent existing addiction behaviors (information "folding"), in a new way that discourages addiction [link to relevant post]. Moreover, it is possible that different individuals have different rates, or ability, of neurogenesis, and external events or neurochemical interventions may also encourage neurogenesis. It is this rate of neurogenesis that might be a candidate parameter that determines how much a particular brain can fold.

Of course, this is all analogical. There is no necessary association between paper and brain. But, this presents an interesting way to approach the problem of quantifying brain function. Paper folding has been applied to several interesting real life problems. For example, the folding of solar-energy panels into a satellite so that large plates fit into a small structure for launching, and unfold in space to achieve maximum surface area for efficient energy collection. In addition, protein folding occurs according to the electro-chemical forces at the molecular level. Paper folding has been applied to understanding and even manipulating these forces to make protein molecules that achieve specific helpful biomolecular functions. Here's an example of applying origami to practical problem from an MIT group [link].

After all, the reason why origami is meaningful, is because we perceive cranes in a few simple folds.

The Automation of Science

Article from Science:

[REPORTS] The Automation of Science
"A robot scientist discovers orphan enzymes that take part in yeast metabolism."

This was published a while ago. But it may be worth mentioning because it could be the pivotal moment in AI.

Increasing neurogenesis might prevent drug addiction and relapse

Article from ScienceDaily:

Increasing neurogenesis might prevent drug addiction and relapse
"Researchers hope they have begun paving a new pathway in the fight against drug dependence."

This makes computational sense. Adding new neurons creates the possibility of forming new inhibitory connections, as well as de-potentiating the strength, or contribution, of existing ones. Such predifferentiated neurons serve as fresh unwritten computational space for which new behaviors and cognitions can be learned. In addition, old pathways which have been entrained and which are hard to change (because of prolonged experience or intensity) can have their effects counterbalanced.

Separation vs. Association

A key function of the brain is to first, process the fact that we are encountering different types of stimuli at every moment, and second, process the simultaneous fact that while there are these different types of stimuli, there are also many consistencies that reflect modifications of the same stimulus at a higher level of abstraction.

One way to evaluate what a cortical region may be doing with respect to this separation/association dichotomy may be to determine the number of neurons at the first level relative to the second level.

If the ratio of neurons at the first relative to second level is large, then the function of the second level is probably to associate. This is a many-to-few limitation. So various permutations and combinations at the first level are funneled into the reduced dimensionality of the second level. Therefore, some combinations are subsumed.

If the ratio of neurons from first to second levels is small, then there is the potential for expansion. The problem becomes a few-to-many scenario. The same combination at the first level may elicit several possible outcomes at the second level. There is information expansion.

...

Agnostic Brain, Biased Mind - what does the FFA do?

Many neuroimaging studies have repeatedly found an area in the human brain that seems to be involved in processing visual faces. This area located in the fusiform gyri in humans, has been affectionately named the fusiform face area or FFA. The FFA is most active when we are looking at pictures of faces, and almost non-responsive to other types of visual items such as objects, houses, scenes, random textures, or a blank screen. Prosopagnosics, who are not able to recognize faces, but are still able to detect the presence of a face and also show no difficulty in processing other types of visual stimuli, have been shown to involve less FFA activity. Even more compelling, patients with lateral occipital lobes lesioned lose some form of object-processing, but show intact face processing. And yet other patients with lesions that have affected the FFA, have problems with face processing (acquired prosopagnosia) but intact processing for other stimuli. The evidence strongly suggests that there is something special about faces, and something about the FFA that deals with this specialization.

The debate regarding the FFA pertain to whether it is the only region or even a critical region that does face processing. Some labs have shown that face processing information can be found in other regions of the brain that are not the FFA. Yet some labs have shown that the FFA is recruited to process fine levels of category distinctions. For example, bird and car experts have been shown to engage some level of FFA activity when processing these stimuli compared to novices. These findings suggest that the FFA is not processing faces per se, but visual representations that have come to require high-levels of fine discrimination through experience, of which faces are the best example of this currently.

I suggest that a more flexible definition is called for when thinking about the FFA and its role in processing visual information. Certainly, it does seem that faces occupy a special place in human experiences. On the other hand, it is difficult to explain why there would be a brain region that codes for faces and faces along based simply based on genetic or biologically determined causes.

In terms of a neural network, if indeed the brain consists of many different sub-types of neural networks that conglomerate to form one large complex network, the FFA is a sub-network specialized to perform a specific operation that is maximized and specialized (trained) for a specific information domain - faces. This or these specific operation(s) could involve identification, discrimination, recognition, or all of these, or even a yet unknown operation. Certainly neural network non-linearities can surprise us! Moreover, these operations have been tuned for a specialized class of stimuli that consists of eyes, nose, mouths, and other visual characteristics of faces when occurring together as a whole (whether from external input, or through internal imagination or retrieval).

What this means is that if you were able to "remove" the FFA, and plug it into a computer so that you can feed this FFA network with inputs and measure its outputs, you could theoretically feed it anything, but the information would be most meaningful or organized when the inputs correspond to information about a face. Of course, this would require us to know what is the language of the input to perform such an experiment.

Other types of inputs may elicit some level of meaningful output of the FFA. Neural network do that. Yet other types may elicit nothing at all. This does not necessarily mean that the FFA outputs from such inputs is useless, nor does necessarily mean that it is used! It is just output. What higher-level brain mechanisms do with the output depends on the task, and how the brain is wired to treat outputs from its sub-networks. It may be ignored, or it may actually incorporate relevant information. That is, the FFA is agnostic to the incoming information. It does not care. It will process it anyway. But other regions decide whether what is it saying needs to be incorporated or not, or if it should be further modified even.

Such a view would reconcile why the FFA is special for faces, yet seems to be carrying some information about other stimuli. It would also be consistent with the idea that information about faces is certainly also available to a certain extent in non-FFA regions, the same principles being applied to these other sub-networks. It would also be consistent with how self-organizing behavior in neural network (see von der Malsburg article [link]) can lead to a consistent topology across every person that processes a particular stimulus in a particular way in a particular spatial location.

This is probably not a new idea, but needs to be clarified in the literature I think.

VSS Conference Day 4: My Poster

This poster was presented at VSS Conference 2009 [link to VSS website], morning session [download poster pdf]. This study investigated the effect of task instructions on repetition suppression in the brain. Repetition suppression refers to the phenomenon that the brain response to repeated stimuli is usually reduced or attenuated. It is thought that such reduction in brain response reflects less neuronal recruitment, and hence, a more "efficient" way of processing the same information.
In this study, however, I postulated that under certain circumstances, the brain requires more neuronal recruitment in order to effectively process information for task demands. That is, repetition suppression becomes inefficient because it reduces the degrees of freedom that the brain can use to manipulate existing representations.

The study evaluated brain response in the fusiform region to face-pairs morphed at different levels of similarity. The idea is that the more similar face-pairs are, the more repetition suppression should be observed in the fusiform face area. Participants viewed the face-pairs under two different task instructions. The first task made face-pair similarity irrelevant. In this task, repetition suppression was observed to repeated faces. In the second task, face-pairs were made critical as participants had to make same-different judgments about the pairs. In this task, repetition suppression was eliminated.

The idea here is that in the same-different judgment task, the brain has to represent faces as distinctinctively as possible so that subtle morph differences can be detected. Thus, repetition suppression is prevented, possibly from executive function areas that process task instruction and exert a top-down modulatory control in the fusiform area.

The study also shows that there are individual differences in participants ability to exert this top-down modulation to regulate repetition suppression in the fusiform regions. This study was also performed in older adults, which will be reported in a subsequent research article. Briefly though, it is thought that older adults show declines in behavioral performance because of less distinctiveness in cognitive representations. This design is thus useful as a means to measure and related distinctinveness of representations in the brain and how that affects behavior.

A structural model of aging, brain and behavior

Possible working structural model that can be tested with measures of stimulus, behavior, neuro-functional, neuro-anatomical variables. The dynamic influences of age and "culture" can also be tested. Culture here refers to long-term experiences of any kind. More complex models can be postulated from this current framework by adding more factors, or measures, and by also constraining the specific weights and covariances. In the broadest sense, the weights and covariances are modeled linearly. However, certainly, non-linear functions can be imposed. The result of such impositions would be a neural network with non-linear activation functions.

I Think That I Shall Never See, A Brain as Pretty as a Tree

What if, sunlight to a tree, is as information to the brain? A tree needs sunlight to survive, to produce food. In response to this basic need, the tree spends a lot of its effort to maximize its ability to obtain sunlight. It does this by forming more leaves, and by spreading those leaves out at widely as possible to cover as much area as possible. Pushing this idea even further, to the extent that the tree covers a portion of area, that is the amount of sunlight it can absorb. Sunlight falling on other areas will be lost to the tree (albeit there might be secondary or tertiary transfer of energy via light reflection, diffusion, and other means). One important parameter that would determine the success of sunlight absorption for a tree would then be leaf surface area. Specifically, greater surface area would increase sunlight absorption rate.

Now, we project this idea onto the brain, of course being well aware that the brain is much different from a tree, although, probably not very very different. The brain is in the business of representing information. Its very function is the processing and retaining of all the information fed into it from the moment of its development. It would be interesting to pursue when this onsets, but that is a digression for later. Nevertheless, the brain develops in tandem with its experience with information. Some of that information is hardwired, or genetic. Some of that information is nurtured, or environmentally experienced. The role of each neuron then, in cooperation with all the other neurons in the brain, is to keep EVERY SINGLE EXPERIENCE, whether internal or external.

Why does the brain want to do that? Well, that's the same as answering why does a tree want so much sunlight for? We can only provide partial understanding here, because this borders on the domain of philosophical and religious pursuits. Biologically, a tree seeks sunlight as part of its nutritional source for the purpose of ultimately creating more trees. That is about as far as we can describe based on observation. This is, in a way, a tautology. Because what we impose as the purpose of the tree, is in fact, what we see the tree already doing. Therefore, such an answer may not satisfy some, but it is a partial answer at the least. Turning back to the brain, again, only a partial answer is given. The brain seeks to contain as much information as possible, because that's what is already observed that it is doing, and perhaps, this information helps the organism to survive, and to produce more organisms of this kind.

More importantly here, we shall consider how does the brain perform this function of representing as much information as possible. The tree does it by increasing surface area exposed to the sunlight. The brain's equivalent would be to increase the number of neurons it has, the connections between these neurons, the variability in the way these neurons can activate. Some smart person might be able to come up with an equation that tells us how much information a given brain with a given number of neurons and connections, and variability in activity, can hold. This could somehow be mathematically related to the concept of surface area...

However, there is a problem in terms of space. While the brain is fantastic and has way superior computational capacity, it is still finite. That is, there may come a time when a given person's brain can no longer process anymore new information. Maybe it has come already, just that we don't know it or that its not as big of a problem as we might think, given we have external aids for our memory now, through things such as computers, books, paper, language, and symbols. This finite capacity is indeed a problem, but our brains have a rather interesting way of solving it, at least to a great extent. Lets turn back to the tree for a moment, because its a greener thought. Lets say that to get more sunlight, the tree has two ways of doing it given a fixed amount of material. It can send more branches out with many leaves, or it can make fewer but bigger leaves. If it sends out more branches, it would have to content with using some of that material to make tree-parts that don't absorb sunlight (branches). If it makes bigger leaves, it may have to content with those leaves blocking each other out, since they will be close together as there are no branches to help spread them out. In the same way, the brain might have two ways of holding information within a limited amount of material. It can create more and more connections with more neurons, or it can use the existing neurons and connections in different ways. Here is where the tree analogy might break down. Unlike information, sunlight to a tree is a one-dimensional problem in the sense that it only needs to worry about expose area. Information, however, is obviously multi-dimensional, with auditory, visual, tactile, odor, taste modalities in the sensory domains, and countless of types of dimensions when you think about concepts and their associations, temporal information etc. Another dissimilarity between a tree and the brain is that most trees only make one kind of leaf, or grow with a certain fixed physical structure. The brain is able to flexibly use neuronal connections to group neurons in very dynamic ways. So, while a tree either has small or big leaves, the brain may use both small groups of neurons encoding some type of information, as well as bigger groups encoding other types of information.

The maximum surface area of the brain (meaning the physical ability of the brain to differentiate between its billions of states of activity using its neuronal connections), limits the total amount of different information the that brain can keep. This will be developed in later blogs. Here are some teasers. One brilliant way of reducing the space needed would be to encode information in terms of similarities and differences. And also, unlike a tree, the brain in the organism makes decisions about what the organism should do, affecting the environment and modifying subsequent experiences, as opposed to being completely at the mercy of the experiences.

Next time...."Similarities and Differences", and "The Brain, The Tree, Intentions, and Decisions".

Models that Account for the Same Data

Perhaps what our minds are doing is accounting for the data. The data is everything around us that we experience with our senses. This information is fed into our neurons, which, by virtue of their network organization, perform some sort of operation on this information. This operation can be likened to a form of model fitting (for those of you who are familiar with the modeling world). Our neurons constantly flux in an effort to represent the information we encounter in the most stable possible way, that allows us to incorporate new information as well as to maintain old information, and even to allow old information to modify new ones.

Consider the method called principal components analysis. This is nothing but redefining the data in terms of another set of dimensions. It thus appears that the same data can be understood in different ways, without changing the data one bit. Furthermore, using one set of dimensions over another set is simply dependent on one's goals or assumptions when trying to arrive at an explanation or investigation.

So then, the question is, which approach is scientific?

Reverse Engineering Brain Networks: Testing the Brain like One would Test a Neural Network

Typical steps in a neural network modeling study are the definition of a particular cognitive phenomenon, the creation and definition of the network model by specifying the neural architecture, activation functions and the learning rules. One then sets out to test
how well the network model matches up with the phenomenon, and to the extent that it does and is parsimonious, and has neural plausibility, it is a good model.

It is then not difficult to imagine how we can do the same thing with the brain by treating it as a neural model that's already built, and we're just trying to discover its architecture, its activation functions and its learning rules. Thus, we run the brain through simulations, observe the input and resulting output, and hypothesize the parameters that led to the observation.

We can then reverse engineer these parameters into the model (which is what we do anyway), and again, test how good the model is.

Unsupervised Learning

Here's an R code that implements unsupervised learning.

vdmulearning.R

Here's the code to display the hexagonal outputs you see in this page.

vdmhexplot.R

This is a specific instance of an unsupervised learning network used by Von Der Malsburg, hence VDM. He was interested in getting the network to exhibit similar behavior to what is observed about the human primary visual cortex. In humans, the primary visual neurons are organized in a columnar fashion according to their sensitivity and selectivity to visual line orientations. That is, each neuron in the primary visual cortex is maximally active for a specific orientation of lines that it receives visual signals from in the environmental space. Furthermore, these neurons are grouped together such that adjacent neurons are each sensitive to close orientations.



This R code implements the VDM network specifically using the following line orientation stimuli. The stimuli consist of 19 input units selectively made active (1 or 0) to give rise to "orientation". In fact, the input stimuli is realized in R as a matrix of 1s and 0s in the right positions.





At first, the network outputs a roughly clustered pattern of activity to a particular orientation (bottom left). But after several training iterations (about 100 cycles, which is quite fast!), it displays columnar organization (bottom right).













Interesting directions to pursue from this code are: object-level representation, color, moving stimuli, 3D representation, binding, repetition suppression.

Here's my paper which describes the model in greater detail [VDM.pdf].

Perceptron Neural Network: Backpropagation

Here's an R [http://www.r-project.org/] implementation of a backpropagation network.

trainnet_perceptron.R
testnet_perceptron.R

The network learns by propagating the input activity to the output layer, then comparing the resulting output with desired outputs. The difference is computed as an error which is backpropagated to the lower layers to effect a weight change that will reduce this error magnitude.

The network is then tested with original or distorted inputs. In general, this network can compute input-output mappings effectively (within network limits which are a function of the number of bits of information required to distinguish inputs, and the number of hidden layers and units). However, it is poor at generalization and distorted inputs compared to the Hopfield network.

Check out my paper that explains in greater detail [Backprop paper].
Also check out this website http://www.gregalo.com/neuralnets.html

Hopfield Neural Network

Here's an R [http://www.r-project.org/] implementation of the Hopfield, auto-associative network.

trainnet_hopfield.R
testnet_hopfield.R

Here's an brief on how it works. Every unit in the network is connected to every other unit (see weight matrix configuration in figure). Input patterns are used to trained the network using Hebbian learning. The network learns by additively changing its weights to reflect instances of unit co-activation. Unit dissimilarities and inactivations are ignored.

The network is then tested on original or distorted inputs, and it will robustly return one of the original trained inputs (within limits).

Check out my paper that explains in greater detail [Hopfield paper].
Also check out this website http://www.gregalo.com/neuralnets.html