Two probes have non-unique contact ids

[alejoe91]

These two probes fail to load because they have non-unique contact ids:

['ASSY-325D-P-1', 'ASSY-325D-P-2']

@WeissShahaf

[WeissShahaf]

Thanks
Will fix and update

[cambridge-neurotech]

should be fixed now.

[alejoe91]

He @cambridge-neurotech

The last upload changed the format, so that in the contact xlsx you have several sections with contour and metadata as well.

Can you refert to the format of single table per probe? This will make it much easier to parse.

[WeissShahaf]

Oops. Will do

[alejoe91]

Also I noticed some column name changed. For us it's better to hav x/y/width/height (instead of x_um, y_um, etc), so we don't have to change the names.

Since you are at it, can you also rename the side column to contact_sides? This would map to our naming convention 1-to-1

[shahafweissMPI]

must have been a debug step i forgot to remove.
check now.
column names also fixed

[alejoe91]

I think something got messed up. Now all these probes have non-unique contact ids:

['ASSY-325D-E-1',
 'ASSY-325D-E-2',
 'ASSY-325D-F',
 'ASSY-325D-H10',
 'ASSY-325D-H7',
 'ASSY-325D-P-1',
 'ASSY-325D-P-2']

[cambridge-neurotech]

can you check again?

[alejoe91]

Seems the same as before. Here is the full error:

Problem loading ASSY-325D-E-1: Contact positions must be unique within a probe. Found 8 duplicate(s): Position (0.0, 0.0) appears at indices [0, 1, 2, 3, 4, 5, 6, 7]; Position (5.0, 0.0) appears at indices [8, 9, 10, 11, 12, 13, 14, 15]; Position (250.0, 0.0) appears at indices [16, 17, 18, 19, 20, 21, 22, 23]; Position (255.0, 0.0) appears at indices [24, 25, 26, 27, 28, 29, 30, 31]; Position (500.0, 0.0) appears at indices [32, 33, 34, 35, 36, 37, 38, 39]; Position (505.0, 0.0) appears at indices [40, 41, 42, 43, 44, 45, 46, 47]; Position (750.0, 0.0) appears at indices [48, 49, 50, 51, 52, 53, 54, 55]; Position (755.0, 0.0) appears at indices [56, 57, 58, 59, 60, 61, 62, 63]
Problem loading ASSY-325D-E-2: Contact positions must be unique within a probe. Found 8 duplicate(s): Position (0.0, 0.0) appears at indices [0, 1, 2, 3, 4, 5, 6, 7]; Position (5.0, 0.0) appears at indices [8, 9, 10, 11, 12, 13, 14, 15]; Position (250.0, 0.0) appears at indices [16, 17, 18, 19, 20, 21, 22, 23]; Position (255.0, 0.0) appears at indices [24, 25, 26, 27, 28, 29, 30, 31]; Position (500.0, 0.0) appears at indices [32, 33, 34, 35, 36, 37, 38, 39]; Position (505.0, 0.0) appears at indices [40, 41, 42, 43, 44, 45, 46, 47]; Position (750.0, 0.0) appears at indices [48, 49, 50, 51, 52, 53, 54, 55]; Position (755.0, 0.0) appears at indices [56, 57, 58, 59, 60, 61, 62, 63]
Problem loading ASSY-325D-F: Contact positions must be unique within a probe. Found 12 duplicate(s): Position (0.0, 0.0) appears at indices [0, 1, 2, 3, 4]; Position (17.0, 0.0) appears at indices [5, 6, 7, 8, 9]; Position (200.0, 0.0) appears at indices [10, 11, 12, 13, 14, 15]; Position (217.0, 0.0) appears at indices [16, 17, 18, 19, 20]; Position (400.0, 0.0) appears at indices [21, 22, 23, 24, 25, 26]; Position (417.0, 0.0) appears at indices [27, 28, 29, 30, 31]; Position (600.0, 0.0) appears at indices [32, 33, 34, 35, 36, 37]; Position (617.0, 0.0) appears at indices [38, 39, 40, 41, 42]; Position (800.0, 0.0) appears at indices [43, 44, 45, 46, 47, 48]; Position (817.0, 0.0) appears at indices [49, 50, 51, 52, 53]; Position (1000.0, 0.0) appears at indices [54, 55, 56, 57, 58]; Position (1017.0, 0.0) appears at indices [59, 60, 61, 62, 63]
Problem loading ASSY-325D-H10: Contact positions must be unique within a probe. Found 6 duplicate(s): Position (0.0, 0.0) appears at indices [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]; Position (19.0, 0.0) appears at indices [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21]; Position (38.0, 0.0) appears at indices [22, 23, 24, 25, 26, 27, 28, 29, 30, 31]; Position (150.0, 0.0) appears at indices [32, 33, 34, 35, 36, 37, 38, 39, 40, 41]; Position (169.0, 0.0) appears at indices [42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53]; Position (188.0, 0.0) appears at indices [54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
Problem loading ASSY-325D-H7: Contact positions must be unique within a probe. Found 4 duplicate(s): Position (0.0, 0.0) appears at indices [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]; Position (52.0, 0.0) appears at indices [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]; Position (250.0, 0.0) appears at indices [32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47]; Position (302.0, 0.0) appears at indices [48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
Problem loading ASSY-325D-P-1: Contact positions must be unique within a probe. Found 8 duplicate(s): Position (0.0, 0.0) appears at indices [0, 1, 2, 3, 4, 5, 6, 7]; Position (23.0, 0.0) appears at indices [8, 9, 10, 11, 12, 13, 14, 15]; Position (250.0, 0.0) appears at indices [16, 17, 18, 19, 20, 21, 22, 23]; Position (273.0, 0.0) appears at indices [24, 25, 26, 27, 28, 29, 30, 31]; Position (500.0, 0.0) appears at indices [32, 33, 34, 35, 36, 37, 38, 39]; Position (523.0, 0.0) appears at indices [40, 41, 42, 43, 44, 45, 46, 47]; Position (750.0, 0.0) appears at indices [48, 49, 50, 51, 52, 53, 54, 55]; Position (773.0, 0.0) appears at indices [56, 57, 58, 59, 60, 61, 62, 63]
Problem loading ASSY-325D-P-2: Contact positions must be unique within a probe. Found 8 duplicate(s): Position (0.0, 0.0) appears at indices [0, 1, 2, 3, 4, 5, 6, 7]; Position (23.0, 0.0) appears at indices [8, 9, 10, 11, 12, 13, 14, 15]; Position (250.0, 0.0) appears at indices [16, 17, 18, 19, 20, 21, 22, 23]; Position (273.0, 0.0) appears at indices [24, 25, 26, 27, 28, 29, 30, 31]; Position (500.0, 0.0) appears at indices [32, 33, 34, 35, 36, 37, 38, 39]; Position (523.0, 0.0) appears at indices [40, 41, 42, 43, 44, 45, 46, 47]; Position (750.0, 0.0) appears at indices [48, 49, 50, 51, 52, 53, 54, 55]; Position (773.0, 0.0) appears at indices [56, 57, 58, 59, 60, 61, 62, 63]

[alejoe91]

I think that for those probes you inverted y and z!

[alejoe91]

yes I confirm, swapping y/z for these probes fixes it.

[WeissShahaf]

Not my day. I will review and get back to you

[cambridge-neurotech]

ok. i was running the wrong script.
i believe it's all fixed now.

[alejoe91]

Still same issues: here is an example

This is a simple script I'm using to test. Can you run it on your side and let me know when you encounter no issues?

import pandas as pd
import numpy as np
import probeinterface as pi
from probeinterface.plotting import plot_probe
from tqdm.auto import tqdm


sheet_names = list(pd.read_excel("probe_contacts.xlsx", sheet_name=None).keys())

wrong_contours = []
sheets_with_issues = []
double_sided_probes = []

for sheet_name in tqdm(sheet_names, "Exporting CN probes"):
    wrong = False
    plot = True
    contacts = pd.read_excel("probe_contacts.xlsx", sheet_name=sheet_name)
    contour = pd.read_excel("probe_contours.xlsx", sheet_name=sheet_name)

    if np.all(pd.isna(contacts["contact_sides"])):
        contacts.drop(columns="contact_sides", inplace=True)
    else:
        double_sided_probes.append(sheet_name)
        plot = False

    if "z" in contacts.columns:
        contacts.drop(columns=["z"], inplace=True)
    try:
        probe = pi.Probe.from_dataframe(contacts)
        probe.manufacturer = "cambridgeneurotech"
        probe.model_name = sheet_name
        probe.set_planar_contour(contour)
        # plot_probe(probe)
        min_x = np.min(probe.contact_positions[:, 0] - contacts["width"][0] / 2)
        max_x = np.max(probe.contact_positions[:, 0] +  contacts["width"][0] / 2)
    
        if min_x  < np.min(contour["x"]):
            print(f"Problem with {sheet_name} on left side: {plot}")
            wrong_contours.append(sheet_name)
            wrong = True
        if max_x  > np.max(contour["x"]):
            print(f"Problem with {sheet_name} on right side: {plot}")
            wrong_contours.append(sheet_name)
            wrong = True
    
        if wrong and plot:
            plot_probe(probe)

    except Exception as e:
        print(f"Problem loading {sheet_name}: {e}")
        sheets_with_issues.append(sheet_name)

[alejoe91]

Actually, instead of checking only right and left boundaries I added a proper contour check, so now the list has grown:

Wrong contours:
['ASSY-1-E-1', 'ASSY-1-E-2', 'ASSY-116-E-1', 'ASSY-116-E-2', 'ASSY-116-H10b', 'ASSY-116-H7b', 'ASSY-156-E-1', 'ASSY-156-E-2', 'ASSY-156-H10', 'ASSY-156-H5', 'ASSY-156-H7', 'ASSY-156-M1v1', 'ASSY-156-M1v2', 'ASSY-156-M2v1', 'ASSY-156-M2v2', 'ASSY-158-E-1', 'ASSY-158-E-2', 'ASSY-158-H10', 'ASSY-158-H5', 'ASSY-158-H7', 'ASSY-158-M1v1', 'ASSY-158-M1v2', 'ASSY-158-M2v1', 'ASSY-158-M2v2', 'ASSY-196-E-1', 'ASSY-196-E-2', 'ASSY-196-H10b', 'ASSY-236-E-1', 'ASSY-236-E-2', 'ASSY-236-H10', 'ASSY-236-H5', 'ASSY-236-H7', 'ASSY-236-M1v1', 'ASSY-236-M1v2', 'ASSY-236-M2v1', 'ASSY-236-M2v2', 'ASSY-276-E-1', 'ASSY-276-E-2', 'ASSY-276-H10', 'ASSY-276-H5', 'ASSY-276-H7', 'ASSY-276-M1v1', 'ASSY-276-M1v2', 'ASSY-276-M2v1', 'ASSY-276-M2v2', 'ASSY-325-H5', 'ASSY-325-M1v2', 'ASSY-325-M2v2', 'ASSY-350-H12', 'ASSY-350-H13', 'ASSY-350-H14-1', 'ASSY-350-H14-2', 'ASSY-37-E-1', 'ASSY-37-E-2', 'ASSY-37-H10b', 'ASSY-37-H7b', 'ASSY-77-E-1', 'ASSY-77-E-2', 'ASSY-77-H10', 'ASSY-77-H5', 'ASSY-77-H7', 'ASSY-77-M1v1', 'ASSY-77-M1v2', 'ASSY-77-M2v1', 'ASSY-77-M2v2', 'ASSY-79-E-1', 'ASSY-79-E-2']

Sheets with issues:
['ASSY-325D-E-1', 'ASSY-325D-E-2', 'ASSY-325D-F', 'ASSY-325D-H10', 'ASSY-325D-H7', 'ASSY-325D-P-1', 'ASSY-325D-P-2']

Here is a zip with the generated figures and JSON files:

cn_test.zip

I'm using this script from this PR: SpikeInterface/probeinterface#382