nvidia-cdi-refresh service causing errors at boot #1873
Replies: 14 comments 1 reply
-
|
@pallaswept I have included these changes in the v1.19.0-rc.3 release. If you could install and test this using the following instructions, that would be very helpful: |
Beta Was this translation helpful? Give feedback.
-
Hi mate, I'd love to help out but I'm on opensuse tumbleweed, I'm not sure if we have an equivalent repo? I've just applied the changes in 5fe6b42 to my service unit file and it seems to do the trick, no errors at startup, and the CDI file was generated as normal. |
Beta Was this translation helpful? Give feedback.
-
|
The current changes made here prevent me from creating any systemd service with an Would it be possible to narrow down a more specific dependency and set When I have a little more time, I will be happy to look into it myself and make suggestions on an alternative if I can find one. |
Beta Was this translation helpful? Give feedback.
-
The initial problem here is the service starting before filesystems are mounted, so it's a pretty safe bet it won't get to multi-user before then, but I agree, it is quite broad. Perhaps you can find a better spot, the Perhaps a better approach rather than |
Beta Was this translation helpful? Give feedback.
-
|
The If a missing mount is the cause then yes, I think On my own systems, I can't duplicate the original issue using the older version of the service unless I completely skip loading the How and when are you loading the |
Beta Was this translation helpful? Give feedback.
-
No idea to be honest, I'm using whatever openSUSE use by default. I can tell you it loads right after the bootloader (grub) because that's when my monitors wake up, but I haven't dug into this myself.
I didn't do any troubleshooting at all beyond reading the logs to find out what service had failed, which is when I noticed this:
(That last part is noteworthy because I know that service which loads the crashkernel is intentionally designed to load before anything else, so that it can reserve memory for its use... but the cdi-refresh was loading before even that) That is to say, this service which generates a file to write to disk, starts before said disk is mounted (and before most drivers have been loaded, it looks like it's before parts of the nvidia driver have loaded, although, at least some of nvidia is loaded because I do see the error on all my displays, which aren't active until after the driver is). I don't really know what's happening other than it's obviously starting way too soon. I saw this in the logs, so I just naively picked an ordering dependency that was right in front of my face, just enough to avoid the error, and reported the issue here. I didn't try to solve it, I just got a workaround in place, and reported it. Sorry I don't have a lot of useful info here, and I'm sorry that the workaround I used has caused you problems. |
Beta Was this translation helpful? Give feedback.
-
|
I did some testing today with both the latest stable and RC release as of Jan 24th to try and duplicate your issue and I just could not get it to break on my Leap 15.6 system. It seems pretty clear that the issue you have is that the service is started before the driver is loaded. Since I couldn't duplicate that, I can only offer ideas, not tested alternatives. Feel free to test on your own system though if you would like. The best idea I can come up with is to add something like @elezar what do you think? Is something like this a better approach? |
Beta Was this translation helpful? Give feedback.
-
|
I'm very happy to test changes to this package to try and resolve this, but I won't be making changes to my nvidia driver installation - it's very strictly by-the-book as it needs to be 100% vanilla as I'm a tester of the package so it has to be just like everyone else's. That being said, I'm absolutely certain that the nvidia module is loaded when that error occurs, because I don't get multi-head displays, only the primary display, until after that module is loaded, and I see the error on both displays. I am also certain that the mountpoint where the cdi file would be written is not mounted at that point in time, so even if we got past that error about the driver, the service still wouldn't work. Now, while the I'm not "users with special requirements", it's all stock out there, with one exception - the only thing that's been altered at that point in the boot process is to use a different font. But honestly it really wouldn't matter. The way this service was configured, it'll start as soon as it can, and that's too soon. It does require some kind of ordering. Obviously, the one I picked out of a hat is no good, but it does need something. This isn't a special case. |
Beta Was this translation helpful? Give feedback.
-
|
So I removed the The nvidia driver is 100% definitely loaded when the error occurs. You may notice from the log in the first post here, that it does succeed on a retry, just one second later (the retry attemps have since been removed, but it used to do that). So, I looked to see where the string Obviously, the latter is not a dependency here, so, I set an Before: After: Before: After @brycehalling would I have no theory about why this works, but it seemingly does.... Maybe it really is a dependency, or maybe I'm just artificially avoiding the race condition that existed before. I'd have to ask an nvidia genius about that. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, that would still work on my system, but even that is not explicitly needed for me.
This is exactly what I am having a hard time getting behind. It seems that the code change to add In my own opinion (feel free to disagree), more analysis needs to be done on a system where this problem is present to determine exactly what condition exists that causes the failure so we can add a dependency that explicitly addresses that condition and nothing more. For example, we know that the unit can't start unless needed libraries are accessible. That is a condition we can create a dependency for without it causing any unnecessary restriction for users because it is a real and known requirement for the software to function. I would love to do more analysis myself to find whatever real requirement is not met to cause your problem, but my system simply does not have the problem. If it helps at all with your own analysis, here is the critical path for this unit in the latest stable 1.18 release on my Leap 15.6 system. Each thing in this chain must be done in order, and this is enough for me to avoid the problem. In my opinion, it should also be noted that dependencies can be added in drop-in files by users who need them, but the ones shipped in the main unit file cannot be removed in a similar way. This should cause us to be quite conservative when adding dependencies to the packaged unit. It is much easier for a user to add a drop-in one time if needed than it is for another user to remember to constantly edit the shipped unit file to remove a poorly-considered dependency every time there is a package update. |
Beta Was this translation helpful? Give feedback.
-
I see where you're coming from but you're missing an important facet of this - it's a race condition. You're not seeing it fail because you're lucky. This isn't a case of "people who need the fix should apply it", it's a case of "everyone needs the fix but if they're lucky they might not notice". It should not be possible for the service to start too early, for anyone, lucky or not. There's no doubt that the problem is that the service is starting too early. But I agree, nobody seems to know exactly what it is too early for.
It doesn't just seem that way, I explicitly said that it definitely is that way. I made it very clear from my very first post that I just picked the first naive shot at it and suggested that there might be a better dependency:
Agree with this let me be very clear. The analysis does not need to happen on a system where the problem is present. The dependencies are the dependencies whether they are met or not, and they should be set, for everyone, even if they don't see the problem that I did. But the analysis of the dependencies is the way forward, no question. I do agree with what you're saying, that a better dependency is needed, one that will fix it, and importantly, will not break it for others (like this one has for you, which is very unfortunate!), I agree with all of that... except for this whole "people should fix it with a drop in if they need it". No, it should just not break in the first place. The troubleshooting should happen here, and a fix applied, here, once, not individually on a case-by-case basis. Let's not confuse "this fix is no good" with "lave it broken and let everyone fix it themselves". I agree this fix is no good, but leaving it deliberately broken with a "fix it yourself if your PC breaks like theses other ones we already knew about", is also no good. The correct path forward here is not to leave it as-is (because it's broken!), and it is not to leave it as it was (because it's broken!) and tell users to fix it themselves with a drop-in. The correct path forward here is exactly what you said - figure out exactly what the dependencies are, and configure the unit accordingly. So let's just leave this "use a drop-in" thing behind, and actually fix it, yeh? I understand that you're not able to reproduce this fault, that's why I went and removed this 'fix' that we don't like, so I could see what changed inbetween "too early" and "not too early" and that's what I described to you above. I didn't list everything that changed between the failure and the success, because it is a LOT because the service starts EXTREMELY early. I'd have to share logs of my entire network config and filesystem and security config and more, with github. I just can't do that, for obvious reasons. So, I scoped the changes to things which touched the relevant filesystems (which as I mentioned was not a solution), and (working on your theory that the driver may not be loaded yet) searched for anything with the word Looking at the critical chain is a good call. Here's mine now, with Here it is with no Anything jump out at you there? |
Beta Was this translation helpful? Give feedback.
-
|
Yeh this is just race-y as can be. I rolled through each of those units, setting them as the Just to help visualise it: This fails: This succeeds: I really think someone from nvidia is going to need to explain to us what the actual dependency is here. It sure ain't jumping out at me! |
Beta Was this translation helpful? Give feedback.
-
|
Something that may be relevant here - I have three separate bugs open with systemd right now, related to their own services starting too soon. They've all come with fairly recent versions of systemd. You're on a pretty old OS there, I wonder if maybe you've got ye olde systemd and it's actually a newer systemd bug that I have and you don't? What version of systemd do you have there on 15.6? I'm troubled by the scenario shown in that first screenshot above - that's set to For reference: I don't really know what else I could add at this point. I've stepped through every possible dependency and while several do work, none of them make any sense. |
Beta Was this translation helpful? Give feedback.
-
|
The NVIDIA Container Toolkit 1.19.1 release removes the dependency on the I understand the original issue was raised due to the the |
Beta Was this translation helpful? Give feedback.
-
|
Hi @tariq1890 Thanks for removing the bad ordering dependency. I mentioned in the first post, that it was chosen thoughtlessly and might be a problem. It will be good if we can find a better ordering dependency.
I may be mistaken but I'm pretty sure that would not prevent the service from starting, if those conditions were not met, it would start, and fail. I'm not a huge fan of using Speaking of me testing stuff, I'm no longer seeing the race condition. Even if I remove all the new changes, the cdi-refresh service now starts just fine without errors the first time. Maybe this was fixed on the systemd side (as discussed above)?
The service succeeding eventually with restarts is the problem that was reported here. To be specific, the initial failures are the problem - the cdi file was always created, but the failing and restarting service caused the log monitor to alert the administrator. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Since updating CTK today, I'm seeing red in the logs at boot:
This service start and error is occurring very early in the boot sequence, for example, before network is up, disks are checked, or crash kernel is loaded.
This seems to fix it:
I just picked that as an ordering dependency based on the
WantedByso perhaps there might be something better.Beta Was this translation helpful? Give feedback.
All reactions