Performance improvement suggestion when loading scenes
Lantia Vertel
Posts: 19
In my workfllow I usually move between scene files which share a lot of content: models, assets, textures and images, morphs, etc... I guess this is the same situation for most people.
However, when I open scene A, do some work, then open scene B, scene B takes basically the same time to be ready than scene A took in first place. This is a bit frustrating as the scene load times are not small to say the least, and this ruins the momentum in your work.
My suggestion for Daz team: Is it possible to make the scene load a bit "smarter" so it doesn't unload/load the same resources when loading a new scene? I think this could be a huge boost when opening scenes sharing a lot of resources (which again, I think it's a very common case). It doesn't need to be "perfectly smart", I understand that reset some resources to their initial state is not easy, but other resources which should be static, like images, geometry (?), etc... could be reused in the opened scene.
Daz 3D is part of 
Connect
DAZ Productions, Inc.
7533 S Center View Ct #4664
West Jordan, UT 84084
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
I'd also like to have such an improvement from Daz but I just wonder if that'll be a mission impossible or not.... Daz has already given a caching function.. if without cache, it'll be even worse...
Not sure if this is a great feature, I would like to have a blank starting point with a new scene. The OS already provides disk caching, etc.. You can try to create a scene for common stuff and render that using Iray canvases.
You wouldn't notice any difference as the state of the program after loading a scene would be exactly the same, what yor are descriving is like restarting your pc everytime you what to navigate to a new new tab in your browser.
The os can cache the files, but it won't cache the data structures generated from those files, which is what ultimatelly the scene uses and takes time to generate.
Another analogy is think about it like compressed files, the os may caches the disk access for a zip file, but everytime you want to use it you still need to uncompress it, which takes some time, so... once you uncompress a file and you have it ready to be used, why to uncompress it again if you needed again?
For super complex stuff I'm not sure DAZ keep up with performance. Models and environments are usually not super optimized, even if you speedup loading time, you will have problems with a lot of photorealistic objects. Unreal is probably more performance-oriented, while DAZ is photorealistic and easier-to use. In your case I will evaluate if is worth to use Unreal.
I don´t need more performance other than the scene load times, as I said, it't about not breaking your focus and momentum when working on the edition stage. Seems you are suggesting a solution for a problem I don't have...
Everything runs smoothly (at least for me) except the scene load!
So normally how long does it take to load the typical scene you mentioned above? Is there any error in log file that might delay the loading time, e.g. duplicate formulas or sth. like that?
I don't think you'd see a big improvement in performance from this change, and probably a degradation in performance when the scene you're loading isn't a version of the one you already have. Think about what you have to do.
If you're actually loading a new and different scene, you'll end up with both of them in memory at the same time as laoding nears completion - which might be more than your machine can handle. And the whole process is made more complex, which means potential bugs can be introduced. You don't want bugs in something as fundamental as loading a new scene.
If you have scenes that have a large number of never-changing objects, and a small number of changing ones, it might be possible to rig up an approximation of the set-up you propose using scene subsets. Produce the big background scene and call it stage.duf. Pose your foreground figures, put them in a group and save it as a scene subset called actors01.duf. Move them around and save them as actors02.duf. Rinse and repeat ad nauseam. You can get to any scene by deleting the actors group and loading a new one from the appropriate subset. You might even be able to use a hierarchical pose preset instead of a subset. It's probably not worth the added complication for the amount of time it would save.
Daz is a funny program in general, I've noticed it sometimes sort of stall on loading assets on to figure (like hair or piece of clothing), and it suddenly wakes up just from clicking a mouse on the asset loading window
Just now tested leading G8.1 basic female into a blank scene repeatedly, load, delete, load, delete etc, if I keep Daz window in focus, the loading time is consistently around 15 seconds, 14.8~15.6 (looking into the log file messages), the really funny thing is, if I immediately alt-tab out to something like a web browser after clicking the load figure icon in Daz, then this load time drops to 12.5~13 seconds! and this quirk is repeatable
same as above but with G8.1 female dev load - keeping Daz in focus - 14.5sec aprox, tabbing out - drops to ~10sec and as low as 9.3sec
It's not exactly like that, this is just the old and well known cache invalidation problem. From what you said:
No need to retain anything, the old scene is already loaded in memory when you are using it (obviously) and that doesn't need to change. It remains as it is today.
The steps in Daz are different. At the moment Daz does:
Basically it deletes data generated from the current scene, just... (in most cases) to load and generate it again. After reading the asset in the first step, it should be trivial to check what you should unload/destroy, and what you shoud load/create, actually Daz do this for some assets (I believe), as in the logs you can see a "Determining missing assets..." line when loading a new scene.
Unfurtunatelly without access to the source code or more detailed logs I can just guess what is really happening behind the scenes, but all I see makes me think it shouldn't be complicated reuse more stuff and not only the assets Daz reuses now.
As example, If I load a scene and I load the same scene again without changing anything, it takes 23 seconds to load, according to the logs this is what happens:
If a scene loads in 23 seconds, that's lightning fast...
For a comparison, if loading a G8 character takes 3minutes, that's just normal for someone with many installed morph packs and characters. Loading a scene with several characters, environment and architechture, can easily take 10-20 minutes.
I think it really depends on the assets / resources that need to be loaded... My single V9 scene is even slower: ( latest Public Build )
2023-04-20 08:16:57.108 [INFO] :: Prepare asset load (open): /H:/G9Dev.duf(7 secs with PBRSkin + those big maps. Each time it must load the map files on disk. No cache for materials)2023-04-20 08:16:57.108 [INFO] :: Locking viewport redraw...
2023-04-20 08:16:57.108 [INFO] :: Viewport redraw locked.
2023-04-20 08:16:57.211 [INFO] :: Native format content directories: 3
2023-04-20 08:16:57.211 [INFO] :: Poser format content directories: 2
2023-04-20 08:16:57.211 [INFO] :: Other import format content directories: 0
2023-04-20 08:16:57.211 [INFO] :: Begin asset load (open): /H:/G9Dev.duf
2023-04-20 08:16:57.246 [INFO] :: Clearing the scene...
2023-04-20 08:16:57.258 [INFO] :: *** Scene Cleared ***
2023-04-20 08:16:57.258 [INFO] :: Determining missing assets...
2023-04-20 08:16:57.731 [INFO] :: Setting textures...
2023-04-20 08:16:57.897 [INFO] :: Scanning for addons...
2023-04-20 08:17:03.677 [INFO] :: Creating node geometry...
2023-04-20 08:17:03.687 [INFO] :: Creating UV sets...
2023-04-20 08:17:03.688 [INFO] :: Creating materials...
2023-04-20 08:17:03.707 Iray [INFO] - MDLC:COMPILER :: 1.0 MDLC comp info : Loading module "......\pbr_skin.mdl".(16 secs, fr projection queue of anatomy to redraw unlock, as no cache for projection.)2023-04-20 08:17:10.566 [INFO] :: Resolving legacy figures...
2023-04-20 08:17:10.566 [INFO] :: Preparing modifiers...
2023-04-20 08:17:10.617 [INFO] :: Creating modifiers...
2023-04-20 08:17:11.563 [INFO] :: Creating property aliases...
2023-04-20 08:17:11.590 [INFO] :: Resolving 186055 formulas...
2023-04-20 08:17:11.590 [INFO] :: Finding formula properties...
2023-04-20 08:17:12.680 [INFO] :: Sorting formula properties...
2023-04-20 08:17:12.724 [INFO] :: Creating ERC Links...
2023-04-20 08:17:14.204 [INFO] :: Finished resolving formulas.
2023-04-20 08:17:14.263 [INFO] :: Processing scene data...
2023-04-20 08:17:14.264 [INFO] :: Finalizing modifiers...
2023-04-20 08:17:14.265 [INFO] :: Finalizing channels...
2023-04-20 08:17:14.265 [INFO] :: Finalizing materials...
2023-04-20 08:17:14.265 [INFO] :: Sorting property groups...
2023-04-20 08:17:14.271 [INFO] :: Setting up follow targets...
2023-04-20 08:17:14.271 [INFO] :: Start following: Victoria 9 HD << Genesis 9 Eyelashes
2023-04-20 08:17:14.271 [INFO] :: Following started: Victoria 9 HD << Genesis 9 Eyelashes
2023-04-20 08:17:14.272 [INFO] :: Connect base morphs: Victoria 9 HD << Genesis 9 Eyelashes
2023-04-20 08:17:14.288 [INFO] :: Applying projection morphs...
2023-04-20 08:17:14.291 [INFO] :: Loaded morph deltas: 0m 0.3s - /data/Daz 3D/Genesis 9/Genesis 9 Eyelashes/Morphs/Daz 3D/Base/head_bs_EyelashesProjectionShape.dsf
2023-04-20 08:17:14.335 [INFO] :: Creating morph projection map: Victoria 9 HD << Genesis 9 Eyelashes
2023-04-20 08:17:14.364 [INFO] :: Base morphs connected: Victoria 9 HD << Genesis 9 Eyelashes
2023-04-20 08:17:14.364 [INFO] :: Applying projection morphs...
2023-04-20 08:17:14.364 [INFO] :: Creating morph projection map: Victoria 9 HD << Genesis 9 Eyelashes
................................................................
2023-04-20 08:17:14.801 [INFO] :: Setting current camera...
2023-04-20 08:17:14.801 [INFO] :: Finalizing scene data...
2023-04-20 08:17:14.802 [INFO] :: Finished asset load (open): 0m 17.590s - /H:/G9Dev.duf
2023-04-20 08:17:14.803 [INFO] :: Processing morph projection queue: Victoria 9 HD << Genesis 9 Eyelashes
................................................................2023-04-20 08:17:30.004 [INFO] :: Unlocking viewport redraw...
2023-04-20 08:17:30.004 [INFO] :: Viewport redraw unlocked.
Asset loading time 17.6s, total loading time 34.6s before viewport redraw, even if my computer is pretty robust... But the loading time in latest Public build was shortened by 10% at least comparing to General release, I was still a bit satisfied...
That's absolutely non-trivial and from your assumption it seems that you do not understand how Daz Studio, and 3D assets in general, work.
In order to compare two scenes, you have to compare the full scene tree hierarchy (recursion). That's already computationaly more expensive than just deleting and loading everything from scratch.
For the sake of an argument, let's say you have a simple scene with just one G8F figure.
In your current scene she is wearing a bikini, in the one you want to load she is wearing a dress. When you start loading new scene Daz Studio would according to you need to somehow determine that the figures are the same. Let's say that's an easy task (even if it isn't). Your ideal outcome would be Daz Studio unloading bikini and loading dress. How do you propose they determine what should be kept? What if the figure is also wearing sunglasses with the bikini? Should they remain and be paired with a dress or be removed? What if the new figure differs in just one morph state or it has different face texture (makeup)? Is it still the same figure?
That is already a complex and hard to define task, and we haven't even considered world XYZ coordinates of the figure (what should happen if they aren't the same? Should the figure move to a new position or not?), morphs which are dialed in (should old morphs remain and be combined with new morphs or be fully replaced with new morphs?), geografts, geoshells, deformers, cameras, lights, multiple objects, instancing, etc, etc.
Trust me when I say this, doing that would just make things slower.
Right now, the only things that make scene loading slow are:
1. Having too much installed 3D assets (the less you have the faster your scenes load, and this is especially true for figure morphs)
2. The efficiency of find/add/remove operations on data structures used to represent scene objects as GUI elements in memory (this is what could probably be further optimized by Daz 3D)
So if you want your scene load time to be faster my advice is to have two separate Daz Studio installs (release and beta), and two separate content folders. In release content you keep only the minimum amount of 3D assets required for working on your current project, and everything else keep in beta content.
Most people complaining about slow scene loading have hundreds of gigabytes of accumulated 3D assets installed -- it's like having a cluttered desk and then complaining you can't find that one particular paper you need quickly.
You can easily test if doing what I suggest would help in your particular workflow -- just move everything out of your content directory to a safe place, then load the scene, scroll through the missing files dialog, and copy only those missing files from the safe place to your content directory. Once you have all the files used by the scene try loading it, I'd be surprised if you reported back that it isn't considerably faster.
Hope this helps.
I think the biggest, even bigger than no.1, are the conflicts between installed assets, errors within installed content and references to content not found in the system - All these are making DS 'think' what to do with them, and some of them cause DS to 'think' pretty long in computer time before continuing with the loading process.
Recently there have been improvements in handling duplicate formulas in Daz Studio (a major point of slowdown) so that's pretty much the best it can be right now.
As for conflicts and errors -- that's why I suggested having a working content dir with assets only for the current project. It helps avoid those problems.
Daz only gave a 'new layout / format' in log file to show the messages of duplicate formulas / formula input / output property not found ... etc etc., however , fixing them still either counts on PA or yourself.
I fixed lots of these stuff in the age of G3/G8, now another wave is coming from G9....
What I was saying is that they also made speed improvement in handling duplicate formulas. It was in the changelog.
Yea, I've seen that. Better than before~
Listen my friend, don't want to have a pointless discussion and I really think you are being defensive here. I think you are asserting a lot which is not true and I don't really have time to explain point by point each thing. Just looking the start of the comment:
First, that's not recursion (you can implement a diff recursively, but you don't need to) and in any case, with a recursive implementation or not, that's nothing for even a low spec pc, it will take a few ms. Second: Every asset/resource should be indexed in a pool for direct access (that's one of the reasons for having a unique id for each thing) which leaves diff problem in nothing. It's very basic resource manamgent. Third: In the worst case scenario (the indexed pool is not part of the scene file (at the moment it's not, and I don't think that needs to change) you only need to read the scene file in a single pass, which is what it happens now. That would be dramatically faster than regenerating everything from scratch, and definitely no way to be slower as you say.
Don't want to continue this discussion as it doesn't seem constructive or productive. I'm an experienced dev (20+ years of commercial experience (7 in graphics)) (so I think I know what I was tallking about), found a basic thing that could (should) improve the performance when loading scenes, I left my suggestion here hoping it helps. But now, seeing how this comunity receips feedback and proposals I understand why Daz works as it does. This explains everything.