CPU fallback (Update and partial workaround)

2

Comments

  • reserv888reserv888 Posts: 1,143

    barbult said:

    If your CPU is going up to 100 degrees C, you don't have adequate cooling in your computer.

    That is correct.
    Question is why the temperature goes from 70 degrees C to 100 in a matter of seconds. I don't have this problem with any other application than DS, and only when the CPU fallback kicks in. 
    The CPU fallback option shouldn't trigger since the GPU's memory limit isn't reached.

  • PerttiAPerttiA Posts: 10,024

    reserv888 said:


    The CPU fallback option shouldn't trigger since the GPU's memory limit isn't reached.

    While running out of VRAM is the most common cause for falling back to CPU, in your case the cause is the "unknown error (while launching CUDA renderer in <internal>:937)" after having successfully rendered for quite a while already...

  • reserv888reserv888 Posts: 1,143

     

    Have you installed the (free) Octane plugin? - Remove
    Have you installed the Nvidia vMaterials 1.7/2.0? - Remove
     
    I don't have any of the two plug-ins installed.

    What happens if you load the Genesis 8 Basic Female and try to render her as she is, without anything else in the scene?
    It crashed (black screen) so fast I didn't even had put the GPU-Z monitor on. If you look in the log file you'll see there is a ton of warnings (again), 

     

    How long have you had the 3090, can it be dirty and overheating in areas that are not monitored?

    The graphics card is NVIDIA Geforce RTX 2080 Ti (not 3090). I've had for almost three years. The warranty lasts until november this year. I've had problems since early january 2021. DAZ says it is a hard ware problem. The retailer have tested it and can't find anything. The say it doesn't black screen in their tests. So I'm making this last push to find out if it is a hardware or software problem.

    Have you tried any stress tests? Check this out;
    https://www.daz3d.com/forums/discussion/comment/6215226/#Comment_6215226
    (The heureka moment)
    Havent tested yet. I'll be back.

    How big is your PSU?
    1000W.

    Have you tried underclocking the GPU?
    No.

    I am truly grateful for your effort. You have helped me more (regardless of the outcome) and analyzed the problem deeper in a couple of days than Daz Support has done in fifteen months.

    txt
    txt
    DAZ Error log 2022-05-10 B.txt
    210K
  • PerttiAPerttiA Posts: 10,024

    At 09:57:06, DS successfully finished rendering and at 09:57:27 started rendering again, this time failing with "unknown error" after 633 iterations - Did you start it again, or what's going on?

  • PerttiAPerttiA Posts: 10,024

    If you have some excess hobby-money, the 12GB RTX 3060 costs 390-499 Eur (VAT 24% included) on this side of Gulf of Bothnia, shouldn't be much different over there either.
    One way to eliminate a faulty GPU, and if the 2080Ti starts working, you could leave both of them in.

  • reserv888reserv888 Posts: 1,143
    edited May 2022

    PerttiA said:

    At 09:57:06, DS successfully finished rendering and at 09:57:27 started rendering again, this time failing with "unknown error" after 633 iterations - Did you start it again, or what's going on?

    I have mistakenly believed, until today, that "Cancel" a render would pause the rendering until you confirmed or denied. I see now that this assumption was wrong. I "paused" to start the TechPower monitoring program. The render obviously continued in the background anyway (but not updating the image in the window). I believe that is what looks like "stop rendering","start rendering". It was just a matter of seconds but I got a black screen before i started the monitoring application.

    Post edited by reserv888 on
  • Richard HaseltineRichard Haseltine Posts: 100,990

    It has to wait for the process to release its current task before it will stop.

  • reserv888reserv888 Posts: 1,143

    PerttiA

    Have you tried any stress tests? Check this out;
    https://www.daz3d.com/forums/discussion/comment/6215226/#Comment_6215226
    (The heureka moment)

    I have now run:
    - a one hour CPU test. Temperatures between 95 and 97 degrees C. Went well. No crash or black screen.
    - a one hour VRAM test. Went well. No crash or black screen.

  • PerttiAPerttiA Posts: 10,024

    You said, you have been having problems since January 2021... DS 4.15 was released in Jan 19th 2021 and Genesis 8.1 on Jan 26th 2021... Coincidence?

    Are you sure you have everything installed for your current DS 4.20.0.2, the Default Resources comes to mind.
    https://www.daz3d.com/downloader/customer/files#prod_13176

  • stefan.humsstefan.hums Posts: 132

    reserv888 said:

    I have now run:
    - a one hour CPU test. Temperatures between 95 and 97 degrees C. Went well. No crash or black screen.
    - a one hour VRAM test. Went well. No crash or black screen.

    A CPU temperature of 95 °C is much too high! Intel CPUs shouldn't run higher than 70°C, AMD CPUs run save until 80 to 85°C Anything above WILL damage the CPU sooner or later. So take care for a good cooling. Well, and those cheap OEM coolers from the CPU manufacturers are NOT made for heavy load...

    A simple VRAM test for the graphics card is also no real stress. Use FurMark for that - or the MSI Kombustor that is based on FurMark but with even more stress for the GPU. Both tools you will find at https://www.geeks3d.com/

  • reserv888reserv888 Posts: 1,143

    Thank you stefan.hums. I will check that.

     

  • reserv888reserv888 Posts: 1,143

     

    PerttiA said:

    You said, you have been having problems since January 2021... DS 4.15 was released in Jan 19th 2021 and Genesis 8.1 on Jan 26th 2021... Coincidence?

    I know, and I think it is a bit suspicious.
     

    Are you sure you have everything installed for your current DS 4.20.0.2, the Default Resources comes to mind.
    https://www.daz3d.com/downloader/customer/files#prod_13176

    Not entirely sure what you mean but I have now reinstalled all starter essentials

    There is a line that often occurs before the black screens and CPU overheating:
    2022-05-10 22:53:13.125 [WARNING] :: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(367): Iray [WARNING] - IRAY:RENDER ::   1.7   IRAY   rend warn : All available GPUs failed.

    I am curious if it indicates a software or hardware problem. Or neither.

  • Richard HaseltineRichard Haseltine Posts: 100,990

    reserv888 said:

     

    PerttiA said:

    You said, you have been having problems since January 2021... DS 4.15 was released in Jan 19th 2021 and Genesis 8.1 on Jan 26th 2021... Coincidence?

    I know, and I think it is a bit suspicious.
     

    Are you sure you have everything installed for your current DS 4.20.0.2, the Default Resources comes to mind.
    https://www.daz3d.com/downloader/customer/files#prod_13176

    Not entirely sure what you mean but I have now reinstalled all starter essentials

    There is a line that often occurs before the black screens and CPU overheating:
    2022-05-10 22:53:13.125 [WARNING] :: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(367): Iray [WARNING] - IRAY:RENDER ::   1.7   IRAY   rend warn : All available GPUs failed.

    I am curious if it indicates a software or hardware problem. Or neither.

    Does it say why they are failing? It means that the render will either stop or drop to CPU, depending on settings, but whether it indicates an issue  with the GPU would depend on why it was failing.

  • robertswwwrobertswww Posts: 790

    @reserv888 Your log says: (NVIDIA GeForce RTX 2080 Ti): Device failed while rendering

    This error sounds like a hardware problem with your GPU which is most likely caused by one of these 3 things:
    1. Your GPU ran out-of-memory
    2. Your GPU is over-heating
    3. Your power supply is too weak for your GPU (the PSU needs some overhead from the total watts drawn by all of the various PC components).

    Try a few things...

    1.  Purge Daz Memory:
    My DAZ 3D Library -> Scripts -> Utilities -> Purge Memory script

    2.  Close Daz Studio

    3.  Restart your PC

    4.  Launch Daz Studio

    5.  In DS, go to: Help-->About Installed Plugins and at the top of the list, make sure 3Delight Renderer is checked on... even though you are using NVIDIA Iray Renderer, unchecking 3Delight Renderer has been known to cause iRay renders to fail.

    6. Load a single Genesis 8 base character and try to render

    7. If it worked, try loading a more complex scene, but make sure you close the old/completed render window first, and then purge Daz memory before loading the more complex scene to test render

  • PerttiAPerttiA Posts: 10,024

    reserv888 said:

     

    PerttiA said:

    Are you sure you have everything installed for your current DS 4.20.0.2, the Default Resources comes to mind.
    https://www.daz3d.com/downloader/customer/files#prod_13176

    Not entirely sure what you mean but I have now reinstalled all starter essentials

    The starter essentials are for the figures, the Default Resources are for DS.
    The current version of default resources is called "IM00013176-42_DefaultResourcesforDAZStudio420.zip"

     

    reserv888 said:

    There is a line that often occurs before the black screens and CPU overheating:
    2022-05-10 22:53:13.125 [WARNING] :: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(367): Iray [WARNING] - IRAY:RENDER ::   1.7   IRAY   rend warn : All available GPUs failed.

    Before that line, the log is telling why the GPU failed 

  • reserv888reserv888 Posts: 1,143
    edited May 2022

    @robertswww
     

    This error sounds like a hardware problem with your GPU which is most likely caused by one of these 3 things:
    1. Your GPU ran out-of-memory
    2. Your GPU is over-heating
    3. Your power supply is too weak for your GPU (the PSU needs some overhead from the total watts drawn by all of the various PC components).

    1. I get black screens even if GPU memory is used below 50%
    2. According to TeckPower GPU-Z monitoring program the GPU is att steady level all the time. I ran a test with OCCT. It had the same level for an hour without errors.
    3. The PSU is 1000W witch according to "everyone" should be more than enough for my setup.

    Purge Daz memory.

    This was interesting. I have made to few tests to say anything conclusive but so far the purging seems to affect the black screens in a good way. I still get them but it seems the are less frequent.
     

     

    Post edited by reserv888 on
  • PerttiAPerttiA Posts: 10,024

    One thing did come to mind... Having 11GB's of VRAM for any other purpose is way more than enough and hasn't even been available for that long.

    Is the amount of VRAM too much for the test programs to test the GPU thoroughly, in a similar way that DS does when rendering Iray scenes.

  • reserv888reserv888 Posts: 1,143

     

     @PerttiA

    The current version of default resources is called "IM00013176-42_DefaultResourcesforDAZStudio420.zip"
    If that is the same thing as "Deafult resources for DS Stuidio 4.20+", then I have it installded. (To be sure I have uninstalled and reinstalled it).

     

    There is a line that often occurs before the black screens and CPU overheating:
    2022-05-10 22:53:13.125 [WARNING] :: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(367): Iray [WARNING] - IRAY:RENDER ::   1.7   IRAY   rend warn : All available GPUs failed.

    Before that line, the log is telling why the GPU failed 
    And that's where the infamous "unknown error" line shows its ugly face.

     

  • reserv888reserv888 Posts: 1,143

    Richard Haseltine said:

    reserv888 said:

     

    PerttiA said:

    You said, you have been having problems since January 2021... DS 4.15 was released in Jan 19th 2021 and Genesis 8.1 on Jan 26th 2021... Coincidence?

    I know, and I think it is a bit suspicious.
     

    Are you sure you have everything installed for your current DS 4.20.0.2, the Default Resources comes to mind.
    https://www.daz3d.com/downloader/customer/files#prod_13176

    Not entirely sure what you mean but I have now reinstalled all starter essentials

    There is a line that often occurs before the black screens and CPU overheating:
    2022-05-10 22:53:13.125 [WARNING] :: ..\..\..\..\..\src\pluginsource\DzIrayRender\dzneuraymgr.cpp(367): Iray [WARNING] - IRAY:RENDER ::   1.7   IRAY   rend warn : All available GPUs failed.

    I am curious if it indicates a software or hardware problem. Or neither.

    Does it say why they are failing? It means that the render will either stop or drop to CPU, depending on settings, but whether it indicates an issue  with the GPU would depend on why it was failing.

    No, it only says "Unknown error".

    I have ticked the box "CPU  fallback". There are two (at least) issues here.
    1. Why does the GPU leave the job to the CPU in the first place? It can happen while the VRAM is used to less than 50%. I have received black screens with just a plain Genesis 8 Figure and nothing but lighting in the scene.
    2. Why do i get black screens when the CPU fallback kicks in? (I can render with CPU only. It takes forever but it doesn't black screen.) I am aware that the CPU black screens when I use post denoiser but I get black screens  after CPU fallback even when i don't use post denoiser.

  • PerttiAPerttiA Posts: 10,024
    edited May 2022

    I have seen the same error code in context with "memory allocation" problem, for some reason yours says "unknown error", but points to the same direction, which strongly suggests there is a hardware problem with your GPU.

    Post edited by PerttiA on
  • reserv888reserv888 Posts: 1,143
    edited May 2022

    PerttiA said:

    I have seen the same error code in context with "memory allocation" problem, for some reason yours says "unknown error", but points to the same direction, which strongly suggests there is a hardware problem with your GPU.

    I have ran tests with OCCT and Furmark. No problems what so ever. Furmark wasn't very demanding regarding the amount of VRAM though (below 1GB was used).
    Since the retailer already have tested my GPU and said they found no problem I am not sure what to say to them. 

    Post edited by reserv888 on
  • PerttiAPerttiA Posts: 10,024

    reserv888 said:

    PerttiA said:

    I have seen the same error code in context with "memory allocation" problem, for some reason yours says "unknown error", but points to the same direction, which strongly suggests there is a hardware problem with your GPU.

    I have ran tests with OCCT and Furmark. No problems what so ever. Furmark wasn't very demanding regarding the amount of VRAM though (below 1GB was used).
    Since the retailer already hade tested my GPU and said they found no problem I am not sure what to say to them. 

    You need to find better tests, ones that test the whole VRAM, not just the usual 2-4GB's 

  • robertswwwrobertswww Posts: 790

    reserv888 said:

    @robertswww
     

    This error sounds like a hardware problem with your GPU which is most likely caused by one of these 3 things:
    1. Your GPU ran out-of-memory
    2. Your GPU is over-heating
    3. Your power supply is too weak for your GPU (the PSU needs some overhead from the total watts drawn by all of the various PC components).

    1. I get black screens even if GPU memory is used below 50%
    2. According to TeckPower GPU-Z monitoring program the GPU is att steady level all the time. I ran a test with OCCT. It had the same level for an hour without errors.
    3. The PSU is 1000W witch according to "everyone" should be more than enough for my setup.

    Purge Daz memory.

    This was interesting. I have made to few tests to say anything conclusive but so far the purging seems to affect the black screens in a good way. I still get them but it seems the are less frequent.

    @reserv888 Regarding the nVidia driver you are using... I would stick with the Studio Driver as it is generally more stable than the Game Driver.
    https://www.nvidia.com/Download/index.aspx

    Have you run into problems using your GPU with apps other than Daz Studio?  If you can try using your GPU to play a demanding game that uses RTX-on (ray-tracing) or try some other demanding iRay app and see if you encounter issues.
    https://www.nvidia.com/en-us/geforce/rtx/ (RTX-on game list at bottom of page)
    https://www.nvidia.com/en-us/geforce/news/nvidia-rtx-games-engines-apps/

    I'm glad to hear that the Purge Memory script is helping, as your issue definitely seems to be memory related and not power releated.

    Purge Memory will clear out the Daz Studio memory, but your GPU memory can still have data from pervious renders.  To fix that, try this...

    1. Run the purge memory script (Scripts -> Utilities -> Purge Memory script)
    2. Restart your graphics drivers: Press Win+Ctrl+Shift+B (note: for just a second, your screen will go black and there will be a beep)
    Full details here: https://www.howtogeek.com/351164/secret-windows-hotkey-restarts-your-graphics-card-drivers/

  • PerttiAPerttiA Posts: 10,024

    You don't happen to have any friends with 8+GB Nvidia RTX GPU's that you could borrow, or take your computer to the retailer if they have a card that you could test with your current setup in Iray rendering.

    There has been a number of posts where Iray rendering on DS has been proven to tax the system and the GPU in ways that a few (if any) other programs and/or games can.

  • reserv888reserv888 Posts: 1,143

    PerttiA said:

    You need to find better tests, ones that test the whole VRAM, not just the usual 2-4GB's 

    I ran a new session with OCCT, maximum memory load. It used all of the VRAM. I ran it for fifteen minutes but didn't get a black screen. It might not be enough but when rendering I get the black screen much sooner. 

  • reserv888reserv888 Posts: 1,143

    Thanks for all valuable input so far, especially from PerttiA and robertswww.

    Here is a summary of the situation regarding renders (and dforce simulations) with my NVIDIA Geforce RTX 2080 Ti graphics card.

    I get a lot of black screens when rendering (or running dForce Simulations). I ONLY get black screens with Daz Studio. There are games that can crash (Diablo II Resurrected, Schenck death animation scene, for those familiar with the game), but when it happens it only shuts down the game, not the entire computer.

    For some reason something triggers the CPU fallback while rendering and then the CPU temp sky rockets from 70 degrees C to 100 degrees C in a matter of seconds. I dont have a CPU temp problem with any other program than DS.

    The CPU fallback occurs without the GPU is out of memory, the GPU is overheated or the PSU (1000W) is to weak.
    Please look at the screenshot from TechPowerUp GPU-Z, where everything is normal...until it's not.

    The Daz log doesn't give much of clue either.
    Attached is the DAZ log from another black screen, The rendered scene was a single figure, lightning and nothing more. No makeup, hair or anything.The GPU memory was used way below 50%.

    I have run stress tests with OCCT and Furmark. Maximum load on the GPU doesn't cause the temperature to rise. Maximum load (11GB) on the GPU memory doesn't cause black screens.

    Since I still have warranty on the graphics card I would very much like to know if my computer problems are hardware or software related.
    (If it is a hardware problem I would like to know to what to say to the retailer. They have already tested the card and claims there is no problem.)

    One more thing: the problems began in january 2021. I have since then upgraded the motherboard, CPU, RAM and PSU. Still got these issues).

    GPU-Z Sensor log 2022-05-18.jpg
    1920 x 1038 - 792K
    txt
    txt
    DAZ Error log 2022-05-10 B.txt
    210K
  • PerttiAPerttiA Posts: 10,024

    What bothers me is this;

    2022-05-10 09:57 [INFO]... Received update to 00633 iterations after 18.333s.
    2022-05-10 09:57 [ERROR]... unknown error (while launching CUDA renderer in <internal>:937)

    Your GPU was rendering fine, did 633 iterations already and then it says it has problems launching the CUDA renderer... Why would it be launching the renderer when it's already rendering...

    Maybe you should present your case in the Nvidia Forums as well; https://www.nvidia.com/en-us/geforce/forums/discover/

     

  • AgitatedRiotAgitatedRiot Posts: 4,437

    PerttiA said:

    What bothers me is this;

    2022-05-10 09:57 [INFO]... Received update to 00633 iterations after 18.333s.
    2022-05-10 09:57 [ERROR]... unknown error (while launching CUDA renderer in <internal>:937)

    Your GPU was rendering fine, did 633 iterations already and then it says it has problems launching the CUDA renderer... Why would it be launching the renderer when it's already rendering...

    Maybe you should present your case in the Nvidia Forums as well; https://www.nvidia.com/en-us/geforce/forums/discover/ ;

    What about this?

    2022-05-10 09:57:53.582 Iray [ERROR] - IRAY:RENDER ::   1.6   IRAY   rend error: CUDA device 0 (NVIDIA GeForce RTX 2080 Ti): unknown error (while de-allocating memory)
    
  • PerttiAPerttiA Posts: 10,024

    AgitatedRiot said:

    PerttiA said:

    What bothers me is this;

    2022-05-10 09:57 [INFO]... Received update to 00633 iterations after 18.333s.
    2022-05-10 09:57 [ERROR]... unknown error (while launching CUDA renderer in <internal>:937)

    Your GPU was rendering fine, did 633 iterations already and then it says it has problems launching the CUDA renderer... Why would it be launching the renderer when it's already rendering...

    Maybe you should present your case in the Nvidia Forums as well; https://www.nvidia.com/en-us/geforce/forums/discover/ ;

    What about this?

    2022-05-10 09:57:53... unknown error (while de-allocating memory)
    

    That's caused by the first error 

  • AgitatedRiotAgitatedRiot Posts: 4,437

    Thanks

     

Sign In or Register to comment.