-
Notifications
You must be signed in to change notification settings - Fork 3
Adding support for .jpk-qi-data and .bin files and moving to dataclasses for the returned loaded data #190
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
ahobbs7
wants to merge
54
commits into
main
Choose a base branch
from
ahobbs7/jpk-qi-data
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
54 commits
Select commit
Hold shift + click to select a range
d25abaf
Added jpk-qi-data loading functionality. Two possible channels: by tr…
ahobbs7 9d28dec
Adding get available channels function to each file format as well as…
ahobbs7 4bd6bb3
Making load jpk qi data function use zipfile instead of afmformats
ahobbs7 28f7633
Fixing scaling
ahobbs7 a67b11f
Updating general_loader and dependencies
ahobbs7 37e1ae2
Adjusting h5_jpk so it works for different shaped images
ahobbs7 16a827c
Making the jpk-qi-data processing save the h5 jpk file in the correct…
ahobbs7 c780de5
Add the ability to save the metadata to h5 from jpk qi. Additionally …
ahobbs7 a764ca6
Adding ability to load force curves from h5 file
ahobbs7 405a95d
Improving speed of loading qi curve data by loading all the curves at…
ahobbs7 3198567
Making loading jpk-qi-data return all the curves
ahobbs7 c25fb75
Adding .bin files support
ahobbs7 085b252
Adding returning of metadata
ahobbs7 06ec0de
Refactoring jpk-qi-data reader to use a loader class for greater modu…
ahobbs7 78c54a7
Starting to implement lazy loading
ahobbs7 3c327e9
Made force curves lazy loaded for h5-jpk and jpk-qi-data
ahobbs7 e776c66
Implementing caching of heavy data objects/ references to large open …
ahobbs7 3e0b8b1
Adjusting curve data access method to work more like a 2D array for m…
ahobbs7 b8ec43a
Separating saving functionality for jpk-qi-data and adding function t…
ahobbs7 10c1050
Fixing duplicated converting to nm bug
ahobbs7 b510141
Minor changes to fix double scaling on current channel as well
ahobbs7 afc65cc
Fixing minor error
ahobbs7 b1ec527
Added timing for testing and started converting to more memory and ti…
ahobbs7 1fd1eed
Making jpk-qi-data loading stream data into h5 file rather than savin…
ahobbs7 4e90188
Changing metdata data saving so 'changing keys' are assumed based on …
ahobbs7 66f388a
Pre-sizing the curve data to make loading faster (using a best guess)
ahobbs7 f286e57
Improving performance by removing javaproperties reliance in loop
ahobbs7 5391179
Making the saving to h5 save in sections using a buffer
ahobbs7 f0235ad
Removing redundant functions and fixing minor index bugs
ahobbs7 62481da
removing possibility of size 1 image stack
ahobbs7 588c725
adding comments and formatting
ahobbs7 4c7ea88
Ensuring that the h5-jpk copy of the jpk-qi-data doesn't get overwritten
ahobbs7 62637c2
Improving __iter__ function so loading of all curves for analysis is …
ahobbs7 b9104cc
Fixing errors with tests caused by logging problems
ahobbs7 0a1f459
Updating tests for jpk-qi-data and h5-jpk with curve data
ahobbs7 58a3a0f
Updating documentation
ahobbs7 c0a2924
Reformatting to match pre-commit conditions and make more robust
ahobbs7 e646205
Minor formatting changes on tests
ahobbs7 a6388fd
Indices spelling correction
ahobbs7 b88ee26
Skipping tests requiring large test files which cannot be added to repo
ahobbs7 be2840e
[pre-commit.ci] Fixing issues with pre-commit
pre-commit-ci[bot] a1764fe
Fixing pre-commit problems
ahobbs7 603d7bf
chore: removing print statements
ahobbs7 43640ac
tests: adding tests for get channel functions
ahobbs7 0146542
fix: stopped unnecessary redefining of nested row proxy classes
ahobbs7 0f6dabf
docs: comments and improving variable naming
ahobbs7 612d4e9
refactor: simplifying dataclass structure by making them work closer …
ahobbs7 b104b3b
tests: updating curve loading tests (currently being skipped) for ref…
ahobbs7 2394556
chore: renaming get_pixel_metadata to get_point_metadata
ahobbs7 699c55b
feat: moving channel_units to be stored per volume rather than in met…
ahobbs7 773daca
refactor: using a dataclass to represent all return data consistently
ahobbs7 0b29bbb
refactor: moving jpk-qi-data logic in general_loader into jpk_qi
ahobbs7 23523fb
chore: moving channels logic out of general_loader and into topostats
ahobbs7 724460f
fix: resizing of datasets while saving not including current buffered…
ahobbs7 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,6 +3,9 @@ __pycache__/ | |
| *.py[cod] | ||
| *$py.class | ||
|
|
||
| AFMReader/data/* | ||
| AFMReader/notebooks/* | ||
|
|
||
| # C extensions | ||
| *.so | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't have time to check this manually - does it work? There isn't a test but frankly we don't have the dev time to move slowly. If you say it works, this is fine with me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The channel fetching seems to work though I'm happy to quickly make some tests for them as that should be pretty quick.