[GSoC] Persistent & Scheduled Firmware Upgrades

[nemesifier]

Official thread for this GSoC idea, ask questions and announce your interest here.

[SAMurai-16]

Hi @nemesifier I am interested in this one, I was going through the issue raised in which you mentioned using existing UpgradeOperation model with retry_count and scheduled_time fields.

Would you be open to enhancing the current status field into a more comprehensive state machine? For example:

Current: status = 'failed' (ambiguous - permanently failed vs waiting for retry?)
Proposed: status with states like 'waiting_for_device', 'scheduled_retry', 'permanently_failed'

This would provide clearer semantics while still using the existing model structure. What's your preference?

[nemesifier]

Current: status = 'failed' (ambiguous - permanently failed vs waiting for retry?)

I argue this is not needed. The definition of failed is exactly that: could not reconnect after the upgrade.

Proposed: status with states like 'waiting_for_device', 'scheduled_retry', 'permanently_failed'

This would provide clearer semantics while still using the existing model structure. What's your preference?

We could add a status for the scheduled upgrades. Not sure about waiting_for_device, it could be useful, but I'll seek an opinion from our UX experts before committing. This detail doesn't impact the rest of the code in a way that it needs to be decided now or does it?

[SAMurai-16]

I believe it will not impact the rest of the code much.

[SAMurai-16]

Hi @nemesifier ,
I was going through the latest GSoC recommendation document shared on the discussion forum, and I had a small doubt regarding the persistence behavior.
The document mentions that persistence should be available per device, but in issue #379 it states that persistence is only intended for mass upgrade operations for now, while single-device upgrades should retain their current behavior.
These two seem slightly conflicting to me—am I misunderstanding something here?

[nemesifier]

Thank you for bringing this up @SAMurai-16!

Yes, the focus is on batch upgrades, but we can add persistence to single upgrades later. I didn't want to add too much work in one iteration.

However, I think it's better to update that part of the text so we avoid ambiguity, this is the new version of the text:

This change is mainly aimed at mass upgrade operations for now; But it should be possible to trigger persistent single device upgrades too, for example, we could support doing this only via REST API and Python scripting for now, and we'll implement the UI for persistent single upgrades in a future iteration.

Is it clear?

[SAMurai-16]

@nemesifier it is clear now. Thanks for the clarification!

[UltraBot05]

Hi, @nemesifier

I have taken my sweet time delving into this issue, and now am in the later stages (mostly the end ones) of drafting my first version of the complete proposal...

The recent clarification regarding exposing single-device persistent upgrades via the REST API and Python scripts (while deferring the UI) makes perfect sense. I also spent time reviewing PR #377 - Improve UX and Flexibility of the Firmware Upgrader Module. It is clear this GSoC project will directly expand upon the excellent foundation that pandafy and the team built there!

In my v1 of the draft:
I am heavily focused on the backend reliability of the operations. Specifically, I am mapping out how we handle the Celery Beat periodic sweeps, manage the retry_count state, and utilize database-level locking (select_for_update) to prevent race conditions - for instance, if a device unexpectedly reconnects at the exact millisecond a retry task is triggered.

I am drawing some diagrams to to add to the proposal - about the complete upgrade lifecycle, and will shortly add a few UI Mockup Images aswell - for the mass-upgrade scheduling interface

Since I am nearing completion, If you do prefer me sending the first iteration of the proposal here I shall send here today evening or tomorrow afternoon. I would love to get your feedback on my approach before the final GSoC submission deadline.

Looking forward to it!

[nemesifier]

Submit the proposal in the gsoc system, feel free to send a commentable google doc link to me privately, I'll share it with the other mentors.

[UltraBot05]

@nemesifier Greetings!

I have submitted my proposal to the GSoC portal. Regarding your request for a Google Doc: I sincerely apologize, but I don’t have a Google Doc version as the proposal was authored in Markdown + LaTeX and built via Pandoc. I chose this workflow to ensure the code blocks, FSM diagrams, and hyperlinked technical thumbnails were rendered with 100% precision, which can sometimes break in Google Docs.

However, to make it as easy as possible for you and the other mentors to review, I have shared the PDF with you via Element/Matrix DM. Most modern PDF viewers (and even browsers like Edge) allow for direct commenting and annotations. I am ready to jump on any feedback or modifications you have immediately!

[SAMurai-16]

Hi @nemesifier , I had a question regarding notification tracking for persistent upgrades.

Since we want to send a single reminder per batch upgrade (to avoid excessive notifications), I currently store last_reminder_sent at the BatchUpgradeOperation level and send aggregated notifications with details about pending devices.

However, since we’re keeping the persistent logic at the UpgradeOperation level, I was wondering how best to model reminder tracking in a way that remains consistent.

My current approach is:

• Use batch-level tracking (BatchUpgradeOperation.last_reminder_sent) for mass upgrades to ensure a single aggregated notification
• Potentially introduce operation-level tracking later if/when single-device persistent upgrades need independent reminders

Does this approach sound reasonable, or would you prefer keeping reminder tracking at the operation level even for batch upgrades?

[nemesifier]

Yes, sounds reasonable.

[Eeshu-Yadav]

Hi @nemesifier

How to pass persistent for single-device upgrades via REST API?

Ref: [feature] Persistent mass upgrade operations #379

it should be possible to trigger persistent single device upgrades too, for example, we could support doing this only via REST API and Python scripting for now

For batch upgrades, persistent propagates from BatchUpgradeOperation to child ops in create_upgrade_operation() — that part is clear.But for single-device upgrades, the PATCH on DeviceFirmwareDetailView goes through DeviceFirmware.save() → create_upgrade_operation(batch=None), and right now there's no way to pass persistent through that chain.
I'm thinking of adding it as a param to create_upgrade_operation() and threading it from DeviceFirmwareSerializer. Would that work or do you have a different preference?

[nemesifier]

It should be treated similarly as upgrade_options.