Unexpected mismatch between camera and harp trigger timestamps

[NgD-Huy]

Session: 715869\20240507T080801

fps: 120 - time between frames: 8.33ms

For this session, two harp timestamps were spaced ~6 ms from each other instead of 8.33 ms. This does not seem to be a dropped frame issue as (1) camera timestamps and frame diff show no anomaly and (2) no compensations were observed, meaning the diff goes back to the expected 8.33 ms. This is consistent across FaceCamera/Metadata.csv, SideBodyCamera/Metadata.csv, and Camera0Frame.bin as timestamps matched exactly.

Check that the timestamps match exactly. Delta = 0 means no difference between the timestamp of these three files.

The time between frames dipped twice in Harp Timestamp (blue) while Camera timestamp (orange) and frame diff are normal

Harp Timestamp only view

Dip location

Index: 187420
Timestamp: [2992896.995136, 2992897.001824, 2992897.010176]
Diff: [0.006688 0.008352]

Index: 341023
Timestamp: [2994176.998432, 2994177.00464, 2994177.01296 ]
Diff: [0.006208 0.00832 ]

[bruno-f-cruz]

Tracking here harp-tech/protocol#48

Code to reproduce:

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import harp
from pathlib import Path
from typing import Literal

ROOT = Path(r"...\20240507T080801")
pd.options.display.float_format = '{:.6f}'.format

def filter_message_type(df: pd.DataFrame,
                        message_type: Literal["READ", "WRITE", "EVENT"],
                        mode: Literal["include", "exclude"] = "include"):
    if mode == "include":
        return df[df["MessageType"] == message_type]
    elif mode == "exclude":
        return df[df["MessageType"] != message_type]


# Harp Trigger
behavior_root = ROOT / "behavior.harp"
harp_behavior = harp.create_reader(behavior_root)

camera_fps = harp_behavior.Camera0Frequency.read(behavior_root / "Register__Camera0Frequency.bin", keep_type=True)



frame_trigger = harp_behavior.Camera0Frame.read(behavior_root / "Register__Camera0Frame.bin", keep_type=True)
frame_trigger = filter_message_type(frame_trigger, message_type="EVENT")
frame_trigger["Diff"] = frame_trigger.index.diff()

start_trigger = harp_behavior.StartCameras.read(behavior_root / "Register__StartCameras.bin", keep_type=True)
start_trigger = filter_message_type(start_trigger, message_type="WRITE")
assert len(start_trigger) == 1, "More than one start trigger found"
assert start_trigger.CameraOutput0.iloc[0] == 1, "Camera 0 was not started"

stop_trigger = harp_behavior.StopCameras.read(behavior_root / "Register__StopCameras.bin", keep_type=True)
stop_trigger = filter_message_type(stop_trigger, message_type="EVENT")
assert len(stop_trigger) == 1, "More than one start trigger found"
assert stop_trigger.CameraOutput0.iloc[0] == 1, "Camera 0 was not stopped"


# Check if any dropped frames are present

print(f"""
      Harp Trigger diagnostics:
        - Frame FPS: \n{camera_fps}\n
        - Start Trigger: \n{start_trigger}\n
        - Stop Trigger: \n{stop_trigger}\n
      """)


# Camera metadata
camera_root = ROOT / "FaceCamera"
camera_metadata = pd.read_csv(camera_root / "metadata.csv", header=None, names=["Time", "FrameNumber", "CameraTime"])
camera_metadata.set_index("Time", inplace=True)

## Check if dropped frames are present
assert all(np.diff(camera_metadata["FrameNumber"].values) == 1), "Dropped frames detected"

## Check if first frame arrives after the start trigger
assert camera_metadata.index[0] > start_trigger.index[0], "First frame arrives before the start trigger"

## Check if last frame arrives before the stop trigger
assert camera_metadata.index[-1] < stop_trigger.index[0], "Last frame arrives after the stop trigger"

## check if number of frames match
bounded_triggers = frame_trigger[(frame_trigger.index >= start_trigger.index[0]) & (frame_trigger.index <= stop_trigger.index[0])]
assert len(bounded_triggers) == len(camera_metadata), "Number of frames do not match"

## Plot timestamp differences
plt.figure()
#plt.plot(np.diff(camera_metadata.index), label="Camera")
plt.plot(np.diff(bounded_triggers.index), label="Harp")
plt.plot(np.diff(camera_metadata["CameraTime"] * 1e-9), label="CameraTime")
plt.xlabel("Frame")
plt.ylabel("Timestamp difference (s)")
plt.legend()
plt.show()


target_period = 1.0 / camera_fps["Camera0Frequency"].values[0]

faulty_frames = np.where(np.abs(np.diff(camera_metadata.index) - target_period) >  0.001)[0]
for fault in faulty_frames:
    print(f"\n\nFrame {fault} has a timestamp difference of {np.diff(camera_metadata.index)[fault]:.5f} s")
    print(f"{frame_trigger.iloc[fault-1 : fault+3]}")