0:02 in this video I'm going to provide a
0:04 brief overview of the application of
0:06 representational similarity analysis or
0:10 RSA to ERP data RSA is widely used in
0:13 fMRI but it can also be used with ERP is
0:16 to generate some amazing results it's a
0:17 general-purpose method for assessing
0:19 links among different kinds of neural
0:21 measures computational models and
0:23 behavior each of these sources of data
0:25 has a different format which makes them
0:27 difficult to compare directly for
0:30 example we have bold activation across
0:32 the set of voxels in fMRI voltage over
0:35 time at a bunch of electrodes in EEG and
0:37 a pattern of activation across the units
0:39 of a neural network model how can we
0:42 compare these different data formats RSA
0:44 solves this problem by converting each
0:46 source of data into a representational
0:49 similarity matrix or RSM we can then
0:51 look at how well the are SMS from the
0:53 different data sources are correlated
0:54 with each other
0:56 okay let's unpack all of this to get a
0:58 representational similarity matrix you
1:00 need to look at the pattern of activity
1:03 produced by several different inputs for
1:04 example imagine that we showed
1:06 participants a set of twenty scenes and
1:08 for each scene we obtain the pattern of
1:11 bold activation across visual cortex we
1:13 could then ask how similar to the
1:15 pattern of activation is for scene one
1:17 versus scene two there are many ways of
1:19 quantifying similarity but we could
1:21 simply calculate the Pearson R
1:23 correlation coefficient between the two
1:25 patterns of bold activation we might get
1:28 a correlation of 0.84 between the fMRI
1:31 patterns for scene 1 and scene 2 and we
1:33 might find a correlation of 0.09 between
1:35 the fMRI pattern for scene 1 and the
1:38 fMRI pattern for scene 3 we have a total
1:40 of 20 scenes so we'd end up with a 20 by
1:43 20 matrix of correlations the result is
1:45 called a representational similarity
1:48 matrix or RSM because it expresses the
1:50 pattern of similarity in the neural
1:52 representations of the 20 scenes we'd
1:55 get a separate RSM for each subject the
1:57 correlation between a given scene and
1:59 itself is always 1 so we ignore the
2:01 diagonal and the upper and lower
2:03 triangles are mirror images so we ignore
2:04 the upper triangle
2:07 some researchers prefer to use a
2:09 representational dissimilarity matrix
2:11 which is just one minus the correlation
2:13 in the end you get exactly the same
2:16 results either way we could also get a
2:18 representational similarity matrix for a
2:20 neural network model that's trained on
2:22 scene recognition we could feed each of
2:25 the 20 scenes into the network and note
2:26 the pattern of activation across the
2:28 units for each scene we could then
2:30 examine the correlation between the
2:32 active pattern for each pair of scenes
2:34 this would give us a 20 by 20
2:36 representational similarity matrix for
2:38 the neural network we could then ask
2:41 whether the RSM for the model is similar
2:44 to the RSM for the fMRI data we would do
2:46 this by just looking at the correlation
2:48 between the lower triangles of the two
2:51 representational similarity matrices we
2:53 use a rank order correlation because we
2:54 don't want to assume anything about the
2:58 scaling of these two matrices if the
2:59 matrices are correlated with each other
3:01 this indicates that the representational
3:04 geometry of the model is predictive of
3:06 the representational geometry of the
3:09 fMRI data and vice-versa we could also
3:11 have subjects view these twenty scenes
3:13 while we record the EEG we could then
3:15 make an average ERP for each scene and
3:18 calculate the similarity between each
3:20 pair of scenes in terms of the ERP data
3:22 to take advantage of the millisecond
3:24 level temporal resolution of the ERP
3:26 data we could do this separately for
3:28 each time point that is for each time
3:30 point in an average we can compute a
3:33 scalp distribution we can then get a 20
3:35 by 20 representational similarity matrix
3:38 for that time point by computing the
3:39 correlation between the scalp
3:42 distributions for each pair of scenes we
3:43 actually have a separate scalp
3:45 distribution for each time point in the
3:48 ERP waveform so we have a separate RSM
3:51 for each time point for example if we
3:53 have a sampling rate of 250 Hertz one
3:56 sample every 4 milliseconds we'd get
3:58 something like this you have an RSM at
4:00 time 0 which is the onset of the
4:02 stimulus then we have 1 at 4
4:04 milliseconds after stimulus onset 8
4:06 milliseconds after stimulus onset etc
4:09 this would go on for several hundred
4:10 milliseconds depending on the length of
4:12 the epoch that was used during averaging
4:15 of course the RSM stirring the period
4:17 immediately following stimulus onset
4:18 will be noise because information about
4:20 the stimulus hasn't reached the cortex
4:23 yet for a visual stimulus the RSM will
4:25 start being structured around
4:27 50 to 70 milliseconds after stimulus
4:29 onset remember each cell in these our
4:32 SMS is just the correlation between the
4:34 ERP scalp distributions for a pair of
4:36 scenes at that time point we can then
4:38 correlate the RSM at each time point
4:41 with the RSM for the fMRI data or the
4:43 neural network model this is just the
4:45 rank order correlation between the our
4:48 SMS we'd expect pretty low correlations
4:51 at time zero because the ERP RSM should
4:53 just be noise at that time we then
4:55 compute the correlation between the ERP
4:58 RSM at four milliseconds with the fMRI
5:00 and model our SMS and then we'd repeat
5:02 this for every time point in the ERP
5:05 waveform each Spearman Rho correlation
5:07 value indicates the representational
5:09 similarity between the ERP data at a
5:12 given time and either the fMRI data or
5:13 the pattern of activation in the model
5:16 we can plot these values as waveforms
5:18 much the way we plot ERP waveforms
5:21 except now the y-axis is the correlation
5:23 between the ERP RSM at a given time
5:26 point and either the fMRI RSM or the
5:30 model RSM so by abstracting away from
5:31 the original units of measure and
5:33 creating representational similarity
5:36 matrices we can quantify the extent to
5:38 which the representational geometry of
5:39 one source of data matches the
5:41 representational geometry of the other
5:44 sources up to this point I've been
5:46 showing artificial data but here's a
5:48 real study they used event related
5:50 magnetic fields rather than ERPs but the
5:53 principle is the same you tend to get
5:55 stronger effects with M eg than with EEG
5:57 but given how much cheaper EEG is I'm
6:00 not tempted to move to M eg subjects in
6:02 this study viewed a sequence of natural
6:03 images and performed an orthogonal
6:06 vigilant's task to keep them alert and
6:08 attentive each subject was tested in
6:11 separate M eg and fMRI sessions with the
6:13 same images the goal was to link the
6:15 temporal resolution of the event related
6:17 magnetic fields with the neuroanatomical
6:21 specificity of the ERP data to
6:22 accomplish this they calculated a
6:24 representational dissimilarity matrix
6:27 for the M eg data at each time point and
6:29 a representational dissimilarity matrix
6:31 for the fMRI data for each brain area
6:33 remember a representational
6:36 dissimilarity matrix is just one - the
6:38 representational similarity matrix
6:40 the next step was to calculate the
6:42 spearmen row rank order correlation
6:44 between the m EG matrix at each time
6:47 point with the fmri matrix for each
6:49 region this resulted in one Spearman row
6:51 correlation value for each combination
6:54 of M eg time point and fMRI brain region
6:58 here's what they found the intensity at
7:00 each voxel is the representational
7:02 similarity between that brain region in
7:04 the fMRI data and the magnetic field
7:06 distribution at a given time point in
7:08 the m EG data this is completely
7:10 different from most source localization
7:12 methods there making no assumptions
7:14 about the physics of EEG e or m EG
7:17 instead these results are obtained by
7:19 using a variety of visual stimuli to
7:21 probe the brain and comparing the
7:23 representational geometries of the fMRI
Chrome Extension