Commit 30f5a9e
[FEATURE] Drive Identification and Refactoring (automatic-ripping-machine#1259)
* Add support for more device info
USB devices frequently change their mount points on system reboot,
causing faulty job associations to drives. To prevent this, drive
recognition based on pyudev serial information (maker, model, serial
number) of optical drives is introduced.
This allows us to always associate a physical drive to the drives stored
in the database without relying on fixed /dev/sr* numbers. On system
startup, drives are now always scanned to recognize hardware changes.
For backward compatibility, the stored serial is used primarily to
identify the drive, and the mount point is only used as a fallback if
the identification fails. This change also makes the drive name
non-editable by the user. We are now also able to mark stale drives that
were previously attached but are not found by the current pyudev run.
This commit also addresses the unreliable eject status that was stored
in the database. - If the drive was ejected (e.g. before system start)
without the database being updated, this drive status became outdated.
We have also added an image to illustrate that a drive was not found and
was marked as stale in the settings overview. We reworked the Drive
Settings Overview to use collapsible accordion items of bootstrap card
items. The icons are now (to some extend) scalable instead of fixed
size. The layout of the drive tab is now 2 column based to give the icon
some space to extend to the bottom.
This commit also addresses some of the pylint errors, introduces
dataclasses for pyudev object handling and aims for better code
readability.
* Enhance MakeMKV Calls with Parser and Error Handling
With this update, we've improved our MakeMKV integration by introducing
a better parser that handles errors more robustly and provides greater
clarity. The module is now also Pylint compatible, making it easier to
maintain. To reduce noise in the logs, we've removed links to stdout
logfiles, instead directing error reports through Python's built-in
logger. This change reduces the file size of logs for users on info log
level or higher. Additionally, when an internal error occurs that is
likely to be expected due to a known failure, it will be marked as
critical.
These Changes will allow further improvements on working with the
MakeMKV output. We introduce modern python concepts like dataclass and
enums.
* Raise Version to 2.16
* Handle MakeMKV info calls
MakeMKV info calls create zombie processes and syslog errors at MakeMKV
level. This changes help to reduce the amount of disturbing MakeMKV info
calls.
- cache MakeMKV disc number for disc:9999 calls
- wait for MakeMKV processes to finish before calling info.
- Give other MakeMKV processes time to call info before advancing in
processing. This helps if multiple discs are inserted at the same time
but waits for makemkv ripping to finish before calling info.
- Update job status to reflect better, at which stage we are.
* Maintainablility: Rename class
A field should not duplicate the name of its containing class
python:S1700
* Maintainability: Remove unnecessary pass
Nested blocks of code should not be left empty python:S108
* Fix flake8 compatibility
* Add python 3.6 compatibility
* Reduce Function Complexity
* Upgrade Process for Multiple Database Upgrades
The current implementation assumed a commit-by-commit upgrade. As we
reasonable cannot assume this, the system drives cannot get updated when
the database updates as the table may have other pending changes that
conflict with the function call.
* Alter logging output.
A lot of the sleep check messages are spammed to the log file. Reduce
Messages by half the amount while preserving the information.
After the sleep check was exited. A message is also added on info level
that the job now continues.
* Add more job status messages
The job status messages are extended to show in the UI which process is
stuck at waiting.
* Consistency: remove button role
Use <button> instead of the button role to ensure accessibility across
all devices.
* Refactor main makemkv function.
Refactor this function to reduce its Cognitive Complexity from 17 to the
15 allowed.
* Cleanup: Too many return statements
Reduce the amount of return statements for better readability.
* Messages to stdout as default
Instead of duplicating this literal "--messages=-stdout" 5 times, we set
it as default for makemkv in the run() function since we also direct
stdout to a piped buffer in this function. It should be safe to assume
that makemkv always accepts this cli parameter for all calls.
* Remove label css class for non-inputs
A form label must be associated with a control. Label elements should
have a text label and an associated control Web:S6853 This applies for
all fields that are not changeable by the user.
* Add job status messages as variables
Use only a defined set of status messages for the ripping process
instead of the current various different (hard to track) status
messages.
* Add config option for concurrent MakeMKV info
This setting turns on/off the limit for concurrent makemkv info calls to
avoid crashes.
* Fix: location of sleep
* Keep Job Status Updates Where They Happen
Move the job status updates to the makemkv info call. This ensures
readability of status updates so it becomes clear who changes them.
* Readability: Fix equality check.
Previous type checks suggest that operands are int but the dataclass
input is not type safe. To not confuse the reader we do a type
conversion to int and then to bool to convert "1" to True.
* Remove ToDo
Make the ToDo an info message. It won't fix that easy and is not of much
importance to store makemkv info persistent.
* Merge this if statement with the enclosing one.
Mergeable "if" statements should be combined python:S1066
* Adjust wait time
Some processes continue early and leave makemkv info processes behind.
Setting the wait time to higher values for finished processes should
allow waiting processes to better pick up the slot.
* docstrings
* Add more Error Messages
Add some more potentially fatal error messages.
* Add traceback to logged exceptions
For better error reporting, include the trace back in the log output.
* BugFix: inverse logic
* Add known error to tray status queries
If a drive was recently detached (without calling scan drives), the
error is known and the drive status should return None.
* Order Drives by description
This will allow the user to define the ordering of the drives in the
settings page. E.g. adding 1st usb drive, 2nd drive, internal drive will
order the drives accordingly.
* Handle Not-Existing Drives in eject page
The eject page did show poor error messages if the drive was not found
in the database. This should help the user to get notified that we
could not eject the requested drive.
* BugFix: Don't use str.upper()
If the connection information could not get retrieved, we do not expect
string output here. Let jinja filter handle upper conversion and do not
put a string if we have None here.
* Use Enum for CDS (IOctl Drive Status)
Use an Enum to aggregate the possible outputs of the Kernel drive status
into readable output. Use tooltips to show this to the user. Make the
drive status a callable function instead of a (magic) property to
indicate that this function call might take some time and is not pure
state conversion (which is what properties should do).
* Fix pylint Error
* DB Drive relationship in job is read only
To avoid conflicts and tackle the SAWarning, we explicitly mark
job.drive as read only. The drive is associated by setting
job_id_current.
* Tackle encoding for None values
Since we are using .get to fill up the pyudev content, there might be a
None value here and there. To make the string conversions fail safe, we
check for this.
* BugFix: TackID messed up
* BugFix: Cancel Wait
* Add eject with tray toggle to drive utils
Handing over the logic to "eject -T" for toggling the tray open/close.
Prevent opening the tray if a job is in progress and associated to a
drive.
* BugFix: Preserve Previous Job ID
The Previous Job ID gets deleted when a new job is added and the
job_finished method had been called prior.
* Get drive debug info back
Add the drive debug information (previously drive_status_debug) back to
the output.
* Preserve Job Association on Database deletion
If a job in the database is deleted, we cannot know for sure if the
association between job.devpath and job_id_previous is actually the
right choice here. It may be that the server restarted in-between
deletions and the mounts have changed. It may also be that the current
job is still running during deletions. In general, this call adds
unnecessary obfuscation to where the job_id_previous is actually changed.
Adding a better (Exception) message to the job info page if a job
is requested but missing and hiding unassociated job ids completley from
the drive settings page.
* Remove Redundant Clutter
We want to check for an associated job id and can simply check if the
Database association exists by using the None object. This should handle
these cases similarly.
* Handle pyudev calls during running MakeMKV
If MakeMKV is running, pyudev does not report a proper drive. We do not
want to mark these drives as stale and handle deletion of properties as
the last step.
* Add indicator for closed drive that is processing
Indicate a running job on the drive by adding a green cycle wheel.
* Add ripping finished ready criterion for drives
The drive status is only updated for "success" and "fail" status.
However, the drive is free to use after the Ripping process finished.
This is currently whenever we go to the transcoding step. Some rework of
the state logic is needed in v3 but this is good for now.
Issue automatic-ripping-machine#1236
* Readability
* Remove obfuscating log message
* Allow int input to sleep_check_process
For better readability, allow integer intput to sleep_check_process.
This converts the time range passed to random to a single value output.
* Fix flake8 and pylint Errors
flake8:
- fix spaces around variable
pylint:
- disable exception for no-member
- fix bad-exception-cause
* Add serial_id field to Database
Make the drive name user editable and add a serial_id field to the
database that contains the static information. Use name for user-space
messages (e.g. flash) and serial_id for the logger.
* Mask the serial Number to the User Frontend
The last digits of the serial number is masked here. It is a good
practice from the MakeMKV forum to not leave your serial number out in
the open. The logs, however, still contain the unmasked numbers for
unique identifiers with serial_id.
* Update Documentation for new Drive Information
Include the newly added fields into the drive information wiki page.
* Mask Serial Number in Dataclass repr
The Serial Number is not something we want to distribute with the logs.
* BugFix: sqlalchemy backref
sqlalchemy backrefs needs only be defined in one place. The way it was
done, created 3 independent backrefs which was a bit confusing and not
working properly when updating one of them. We do not need a
backref for the last job association since the job does not know if it
is the current or previous job for the drive. Only the job needs to
know, what drive it was assigned to. This simplifies the backrefs to just
one (active drive <--> active job) instead of three.
* Only delete mdisc association at program start
With the current implementation, mdisc could get deleted by the UI
settings "scan for drives" operation. If this is done in-between a rip
that requires the makemkvcon disc association, the job will fail due to
missing mdisc id. Only deleting the mdisc once on UI startup ensures
that it will get filled by the first call to makemkvcon info and not get
emptied afterwards.
* Remove explicit session.add of changed entities
A session needs to get informed only on newly created entities. Having
the session.add() in places where we changed an entity is not necessary.
* Be less strict about when ripping
Do update all mount points in the database with mdisc ids. This should
ensure that the mdisc id is stored across all devices.
* Update Progressbar instead of redrawing
* Check for associated drive when ejecting
Sometimes the drive might have been disassociated from the job by other
means and thus cannot get ejected here. Check this and log the case.
* Add more known Errors from MakeMKV
Add the error code for a failure in disc opening. This probably means
that the disc is beeing read by another process. We differentiate
between fatal errors which raise (MakeMKV exit status is unreliably
sometimes) and errors which are only directed to a higher log level to
get user awareness.
* Organize Possible Job Status and changes.
Created an enum for JobState to keep better track of the different
possible job states throughout the project. It also prevents new jobs to
get created when any ripping (audio/video) is active. A job is not
released when its status changes. The associated drive keeps track if
the drive can be safely re-used by another job.
Issue automatic-ripping-machine#1411
* Prevent too frequent updates of mdisc
If multiple jobs call this piece of code too frequently after each
other, the database will be locked. Using a deferred write here to limit
the number of database writes during mdisc update.
* Define a constant instead of duplicating literals
The literal "route_settings.settings" was used multiple times.
String literals should not be duplicated python:S1192
* Reduce Cognitive Complexity of makemkv function.
The cognitive complexity for check_output and track_info was too high.
Using a class based approach here to distinguish between possible error
states.
---------
Co-authored-by: Mtech <62650032+microtechno9000@users.noreply.github.com>1 parent cdc7c1b commit 30f5a9e
28 files changed
Lines changed: 2974 additions & 708 deletions
File tree
- arm_wiki
- images
- arm
- migrations/versions
- models
- ripper
- ui
- jobs
- settings
- templates/settings
- static
- img
- js
- setup
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
Lines changed: 49 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
Lines changed: 25 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
Lines changed: 26 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1 | | - | |
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
1 | 2 | | |
2 | 3 | | |
3 | 4 | | |
4 | 5 | | |
5 | 6 | | |
6 | 7 | | |
7 | 8 | | |
| 9 | + | |
8 | 10 | | |
| 11 | + | |
| 12 | + | |
9 | 13 | | |
10 | 14 | | |
11 | 15 | | |
| |||
15 | 19 | | |
16 | 20 | | |
17 | 21 | | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
18 | 83 | | |
19 | 84 | | |
20 | 85 | | |
| |||
28 | 93 | | |
29 | 94 | | |
30 | 95 | | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
31 | 102 | | |
32 | 103 | | |
33 | 104 | | |
| |||
211 | 282 | | |
212 | 283 | | |
213 | 284 | | |
214 | | - | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
215 | 293 | | |
216 | 294 | | |
| 295 | + | |
217 | 296 | | |
218 | | - | |
219 | | - | |
220 | | - | |
221 | | - | |
222 | | - | |
223 | | - | |
224 | | - | |
225 | | - | |
226 | | - | |
227 | | - | |
228 | | - | |
229 | | - | |
230 | | - | |
231 | | - | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
0 commit comments