Add ivscc_apfrequency operation

[MichaelHuth]

close #2581

• Set the opacity of individual traces to 0.2. Leave average trace opacity at 1.
• adapt xvalues operation to extract also from xvalues meta data
• add support to plotter for opacity meta data
• Support fitting the data (use the error bars of the average trace to weight the fit). Consider implementing a new operation that's passed as an optional argument into ivscc_APfrequency.
• find out how to get automatically determined xOffset constant from e.g. exp_XOffset integrated fit func after fit -> W_fitConstants
• ??add argument to ivscc_apfrequency that enables that all experiments/cells are fitted.
• add argument to preparefit to limit the fitting range. For flexibility and simplicity I suggest to add this through a mask wave in fit2 then.
• Make the fit parameters accessible with a related operation
• ??the fit result may be based on the acquisition/sweep sequence rather than an ordered list of current intensity (x-scale values) -> 3rd avg mode?
• add a way to expose the number and type of required coefficients to the user. Perhaps a link to the relevant Igor documentation, and/or a pop-up? -> add to help of preparefit
• Another piece of data I'd like the operation to return is the individual and average DA scales at which a negative slope was first observed in the FI data.
• add argument to the operation to limit the current range of the avg trace points that are fitted. The current is in pA and refers to an x-axis range. The operation crops the point range that goes into the fit automatically. The argument is like [start_pA, end_pA].
• ?? for the single experiment f-I plots should the f-I pairs be sorted by I before displaying (prevents zig-zag connecting lines)

TBs new todos:

• Data on the FTP for regression tests: pr-2599/inputRegressionTest
• C&P into Excel does not work as multiple entries are added into one cell
• Fix else // SF_OP_IVSCCAPFREQUENCY_NONE branches and add fatal error
• Allow to add tags to experiments in the analysis browser
  - Tags are simple strings (igor object names) separated by e.g. comma: Add input functionality to tag experiments in AB. Tags are not saved to disk (except experiment is saved), but only there while the window is open.
  - Show them in an analysis browser column
  - We expect 10 or more experiments per tag, so convenient GUI is important
  - Add select variant to select data by one of multiple tags: This new select filter should work with select itself and as additional optional argument for ivscc_apfrequency. seltag([tag1, tag2, ...]) For ivscc_apfrequency we support that the seltag is given as array [seltag([tag1, tag2, ...]), seltag([tag1, tag2, ...]), ...]. ivscc_apfrequency should create one group for every seltag in the array.
  - If no seltag is given then ivscc_apfrequency should default to create one group for every list of tags present in the experiments, create one group for experiments without tags.
  - Mention list of tags in the legend.
  - When used with ivscc_apfrequency the following special analysis is applied:
  - Support seltag in ivscc_apfrequency` so that each group of experiments from the same seltag generates:
  - single experiment f-I plot (same as now)
  - one additional f-I plot where all f-I pairs from the single experiments are concatenated (and sorted?)
  - average as before (just including data for this seltag)
  - fit as before
• Add argument option for ivscc_apfrequency to turn off the traces for single experiment plots
  (Original text :one f-I plot in addition to the single experiment plots. Generate one average for each group and no global (over all data across groups) average anymore. Allow to turn off the single experiment plots.)
• IVSCC apfrequency: Prep 3 of x #2706
• getmeta
  • Tests
  • Docu
• fit2
  • Tests
  • Docu
• preparefit
  • Tests
  • Docu
• ivscc_apfrequency
  • Tests
  • Docu
• ihf docu

close #2628

[MichaelHuth]

@timjarsky This is a first version to play around. There are still a few things I have to add, that I discuss farther below.

ivscc_apfrequency([xaxisOffset, yaxisOffset, xAxisPercentage, yAxisPercentage]) where [] denotes that the parameters are optional.

xaxisOffset and yaxisOffset are strings that can be min, max and none.

General Plotting Behavior

The operation itself returns internally a full plotting specification that is inserted by the formula plotter at the location where the operation appears in the notebook code.

The operation creates only traces that are separated by with. This means preceding and succeeding formulas that are also separated with with go to the same subwindow.

xAxisPercentage and yAxisPercentage are treated as properties for the plot by the plotter. Specifically plot means in this context the subwindow where the traces go. Plot properties from the last formula in the with chain are applied. (i.e. there is currently no gathering of plot properties).

Thus, the plotter applies the 10% for x and y-axis when used like this, where the formula setting the plot properties is last in the chain:

1
with
ivscc_apfrequency(min, min, 10, 10)

but not for this:

ivscc_apfrequency(min, min, 10, 10)
with
1

because 1 does not include any plot properties.

Operation Arguments

Currently:

ivscc_apfrequency([xaxisOffset, yaxisOffset, xAxisPercentage, yAxisPercentage])

with xaxisOffset and yaxisOffset as min, max, none and xAxisPercentage, yAxisPercentage a number between 0 and 100.

The (later) final arguments should also expose arguments from apfrequency, so it will change to ivscc_apfrequency([xaxisOffset, yaxisOffset, xAxisPercentage, yAxisPercentage, method, level, resultType, normalize, xAxisType).

The default for xaxisOffset and yaxisOffset is min.

On the basis of the experiment avgMethodTesting2.pxp the generated code is:

sel = select(selsweeps(), selstimset("*rheo*", "*supra*"), selvis(all))
selexpAD0 = select(selexp("C57BL6J-734969.15.10A.01.nwb"), $sel, selchannels(AD0), selrange(E1))
selexpDA0 = select(selexp("C57BL6J-734969.15.10A.01.nwb"), $sel, selchannels(DA0), selrange(E1))
freq0 = apfrequency(data($selexpAD0))
current0 = max(data($selexpDA0))
currentNorm0 = $current0 - extract($current0, 0) #*1
selexpAD1 = select(selexp("C57BL6J-734969.15.10B.01.nwb"), $sel, selchannels(AD0), selrange(E1))
selexpDA1 = select(selexp("C57BL6J-734969.15.10B.01.nwb"), $sel, selchannels(DA0), selrange(E1))
freq1 = apfrequency(data($selexpAD1))
current1 = max(data($selexpDA1))
currentNorm1 = $current1 - extract($current1, 0) #*2
ivsccavg = avg([$freq0,$freq1], group)
ivscccurrentavg = avg([$currentNorm0,$currentNorm1], group)

$freq0 - extract($freq0, 0) #*3
vs
$currentNorm0
with
$freq1 - extract($freq1, 0) #*4
vs
$currentNorm1
with
$ivsccavg - extract($ivsccavg, 0) #*5
vs
$ivscccurrentavg - extract($ivscccurrentavg, 0) #*6

I added a #* to the formulas that change depending on the min, max, none setting.

#*1, #*2 and #*6 depend on the xaxisOffset argument with the following logic:

min: currentNormX = $currentX - extract($currentX, 0)
max: currentNormX = $currentX - max(flatten($currentX))
none: currentNormX = $currentX

for #*6 it is:
min: $ivscccurrentavg - extract($ivscccurrentavg, 0)
max: $ivscccurrentavg - max(flatten($ivscccurrentavg))
none: $ivscccurrentavg

#*3, #*4 and #*5 depend on the yaxisOffset argument with the following logic:

min: $freqX - extract($freqX, 0)
max: $freqX - max(flatten($freqX))
none: $freqX

for #*5 it is:
min: $ivsccavg - extract($ivsccavg, 0)
max: $ivsccavg - max(flatten($ivsccavg))
none: $ivsccavg

I need to add a flatten operation because the result in e.g. $freqX are single values in 7 datasets (for avgMethodTesting2.pxp). Thus, our max() operation would determine the max in each dataset individually, but that is not what is wanted. The flatten operation should change n datasets with a single data point to an array with n elements.

Therefore, the max argument is currently not implemented yet until I have the flatten operation implemented.

An additional task from the issue is to add a variable that contains the names of the experiments. I can create this variable in the operation and add it to the variable storage of the formula notebook. It would be available then after the operation ran.
For now I thought of $ivscc_apfrequency_explist as name and it would be a string array.

[timjarsky]

@MichaelHuth

Thanks for handling the metadata management for mismatched sweep numbers across experiments.

A few points for discussion:

1. The marker coloring makes it hard for me to evaluate the output (see plot image below). Can we use a single color for each experiment? Can the data be a single 1D wave instead of multiple x-y waves (this will enable fitting)?

2. There are many zero-frequency measurements spread across the x-axis.

3. The average seems more variable than the input data, perhaps because there are too many data points?

4. I'm not sure about the utility of the negative current values with "min" and "max" (second image). The two axes options that I'm sure are needed are none (where each FI curve starts at zero current and zero frequency) and the absolute current values.

ivscc_apfrequency(none, none)

ivscc_apfrequency()

[Copilot]

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[MichaelHuth]

@timjarsky

1. I changed that. There are now experiment + 1 traces with markers. The +1 trace is the average in the average color. The experiment traces use the common table for trace colors, with one color per experiment. I also added code to show the experiment name in the legend for these.
2. I still have to look into this, why a lot of the results return a zero. Currently the standard setting of apfrequency for the internal apfrequency call are used.
3. Yes, it looks a bit like that. I did not see that in my local testing with the three experiments though. I guess in the graphs you pasted there is for some of the average points only a single sweep for an index in the groups.
   e.g. first exp has 10 and second exp has 11 sweeps selected, then sweep data for index 10 is only present from the second experiment -> only one sweep goes into the average and then the trace point from the experiment and the trace point from the average are equal. The average is shown in front and covers the point from the experiment. You may check for the average points on the very top if there is a data point from an experiment underneath.
4. I think it is the other way around, with none no offset is applied. min moves the trace to zero and max moves the trace such that the former maximum point is at zero.
   What I think might be unexpected with the initial formula construct is that there is an offset used per trace. Thus, when changing none -> min then each trace start is moved to zero separately and the visual relation between the traces shifts. (same applies for max)
   This per-trace behavior is the same for the xaxisOffset and yaxisOffset.

[MichaelHuth]

@timjarsky
Regarding 2.: The default level for apfrequency is 0. This results in zero peaks found for sweeps where E1 does not cross the zero line and for this apfrequency returns 0 as result with the default arguments.

I have added support for the apfrequency argument block after the first four argument for ivscc_apfrequency. The arguments are now:
ivscc_apfrequency([xaxisOffset, yaxisOffset, xAxisPercentage, yAxisPercentage, method, level, resultType, normalize, xAxisType])

The last four arguments are "forwarded" to apfrequency.

[timjarsky]

@MichaelHuth, Are failing sweeps included or filtered out? If included, please update to use only passing sweeps.

[MichaelHuth]

rebased

@timjarsky The axisOffset specification is applied at two locations in the calculation. In one of the first steps the apfrequency vs current is determined for all experiments and their selection. This basically results in apfrequency vs current data for each experiment. These offsets are applied to this apfrequency vs current data for each experiment separately.

This offsetted data is then used as input for the averaging.
The result from averaging is offsetted then again with the same axisOffset settings (for averaged freq and averaged current).

Another thing I noticed: The initial apfrequency vs current result from each experiment is not sorted by current. This results in the plots that the line between the markers can go back and forth. The original order follows the order from the selection. Is it important to keep the result order or would it be better for the visualization if the data is sorted by current before being displayed?

The current version of ivscc_apfrequency has the "average DA scales at which a negative slope was first observed in the FI data." supported. I added the following:
The apfrequency vs current result from each experiment is sorted by current. Then apfrequency for pairs with the same current is averaged. Then I determine the slope and find the first occurrence starting from the lowest current where the slope of the frequency changes to negative. For each experiment the resulting apfrequency / current pair is determined. If there was no location where the slope changed then the pair is NaN.
This result is plotted as trace with reed diagonal cross markers.

I kept the y-value at the frequency such that the cross is on top of the marker from the apfrequency vs current graph. Alternatively visualization could be to keep the y-value zero and plot it just as vertical lines.

A second trace is added that is the averaged pair of apfrequency vs. current from the previous traces. This is only a single point that is the average where the slope changed (inflection) for all experiments.
The marker is a red filled caro.

The current version of this PR also has documentation for the new operations in the .rst file.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

[t-b]

@MichaelHuth The seltag spec we came up with looks good. I've adapted one thing as [seltag(...), ...] input to ivscc_apfrequency should be optional.

Diff: (also available in the ... menu on github)

ivscc_apfrequency should default to create one group for every list of tags present in the experiments, create one group for experiments without tags. Mention list of tags in the legend.

[t-b]

Note to self: Chat with @MichaelHuth about finding neg slope algorithm. We should only consider neg slope value from a DAScale value which is higher than the maximum firing frequency.

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 3 comments.

[MichaelHuth]

@timjarsky I have added a first version that allows adding tags to experiments in the analysis browser. The tags are stored in a column in the experiment list.
The tags are currently rather volatile, thus if you close and reopen the AB the column is cleared, also if you refresh the AB.

On the left there is a new button that open/hides the tag control side panel.

The top input field is for adding single tags to selected experiments. Press enter or the button right of it to execute.

The tag list below shows the unique tags present for the selected experiments. If a tag there has a green background then all selected experiments have that tag.

You can select one or more tags in that list.
The Remove tag(s) removes the selected tags from the selected experiments.

For selecting experiments based on tags there are three ways:

1. Select tags from the list, then press "Contains at least one tag" -> select experiments that contain at least one of the tags selected in the tags list
2. Select tags from the list, then press "Contains all tags" -> select experiments that have exactly all tags selected in the tags list
3. Enter some comma separated list of tags in the input field, then press enter or the Select button -> select experiments that have exactly all tags from the entered list

Tags are case-insensitive. Tags must be clean names.

I also added a small convenience tweak to the experiment list, where you can press CTRL-A now to select all rows.

[timjarsky]

@MichaelHuth the analysis browser implementation of tags looks nice. Can you help me understand what Contains at least one tag , Contains all tags, and tag list: do?

Are the tags implemented in IVSCC_ApFrequency yet?

[MichaelHuth]

@MichaelHuth the analysis browser implementation of tags looks nice. Can you help me understand what Contains at least one tag , Contains all tags, and tag list: do?

All three options allow to select experiments in the experiment list by tag.
You select tags in the tags list and then button Contains at least one tag selects all experiments in the experiment list that have at least one tag from your selected tags.

Button Contains all tags select all experiments that have exactly all your selected tags.

The tag list input allows you to enter a tag list as comma separated list. Select then selects all experiments that contain exactly these tags.

Are the tags implemented in IVSCC_ApFrequency yet?

Not yet, the next step is to take them over when loading into the sweepmap, such that the tags are also available in the sweep browser. Then I can access the tags from sweepformula to add seltag as operation.

[timjarsky]

@MichaelHuth thanks for the helpful response. I'm finding that Contains at least one tag, Contains all tags, and tag list don't select experiments except in once case: a single tag to' tag list'.

[MichaelHuth]

I select tags in the tags list on the right:

Then after pressing "Contains at least one tag":

With the same selection as in the first picture and then pressing "Contains all tags":

If you have no tags selected in the tags list (blue background) then no experiment is selected.

Your entry in the lower input field is "control, test" (notice the space before test). Thus, the tag parsed is " test". It should work with "control,test" (without space). I add some trimming that removes spaces from the list elements, such that "control, test" is parsed as "control" and "test".

[Copilot]

Pull request overview

Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 21 out of 21 changed files in this pull request and generated 12 comments.

[Copilot]

Pull request overview

Copilot reviewed 20 out of 20 changed files in this pull request and generated 4 comments.