Tutorial on the idiosyncrasies of the Instancing laboratory. Part 5A of 6
pbudarick_4a3d2ac478
Posts: 0
For Part 1 of 6 of my tutorial, please go to [ http://www.daz3d.com/forums/discussion/3637/ ]
For Part 2 of 6 of my tutorial, please go to [ http://www.daz3d.com/forums/discussion/3924/ ]
For Part 3 of 6 of my tutorial, please go to [ http://www.daz3d.com/forums/discussion/5270/ ]
For Part 4 of 6 of my tutorial, please go to [ http://www.daz3d.com/forums/discussion/6506/ ]
For Rashad Carters Comprehensive Tutorial, please go to [ http://www.daz3d.com/forums/discussion/3381/ ]
Well this Part of my tutorial is going to be rather long so i will divide it into several sections [A, B, C, D....] because i can only use 5 images per section.
The following is an interpolation within the tutorial as a whole before we get on with things. It was prompted by Horo's latest response concerning instancing, memory size and compression [ http://www.daz3d.com/forums/discussion/6506/ ].
I created 5 files which are all the same except that different types of spheres are used. A 2D array of 81 spheres [9x9] was used and every file had the same material applied to all the spheres. That is important because the materials have a great influence on the file size, loading and rendering speed. I will go into that i bit more later.
The files are as follows:
(1) 81 Bryce Native Spheres.
(2) 81 MetaBalls of the same size which just begin to merge together.
(3) 81 MetaBalls converted into a single mesh object inside of Bryce. It took 3 hours and 25 minutes to convert the hull !!!
(4) 81 Mesh Spheres created form the Native Bryce Spheres at the Highest Wireframe Mesh Resolution [128] and of the same size.
(5) 1 Mesh Sphere created form the Native Bryce Sphere at the Highest Wireframe Mesh Resolution [128] and in addition 80 Instances of this object added.
Picture_811 shows the file names and the size of the files. I also observed saving and loading speed. I did not get any crashes. I used the Multi-Replicate tool as appropriate. For file (5) above i ticked the "instance" button. I was able to not only create instances from the master but also instances from the other instances. That was necessary because the master was in the middle of the array and i did not want to duplicate it.
Conclusion: The advantages of the Bryce primitives and MetaBalls are enormous. They not only result in a small file size but also very fast loading, saving and rendering. The Mesh Objects were the worst - very slow to save and load and producing huge file sizes [2,633,472 polygons]. None of this was surprising as i have been researching this for many months but i have not made such highly controlled experiments before.
Picture_812 shows the Array of Spheres. Picture_813 shows the MetaBall Array. Picture_814 shows the render times.
Observations: It appears that when Bryce Primitives and/or MetaBalls are used the objects are NOT saved but only each Matrix is saved as well the Materials. The Matrix for each Instance takes up very little memory, compared to a very smooth Polygonal Sphere which requires an enourmous amount of memory. Compression helps to make the file even smaller. Since i used only one material, and a very simple one at that, the files are very small and they load, save and render very fast. The Bryce Native Spheres render the fastest. Metaballs can be rather slow but that is understandable because Bryce must solve a complex surface. When these files are loaded a pointer is re-created to the Mathematical Master Description for each Instance. In addition, the outline is perfectly smooth when rendered because the actual form is solved at this time [For CSG primitives, BSOps and MetaBalls]. It is therefore a perfect system and partly why Bryce has been the most supperior program in many respects for over a decade. It is also clear why an additional instancing system for Mesh Objects was introduced in Bryce 7. These files also save and load quite quickly and the file size is much less than files constituted of copies. The instancing systems [Old and New] both work very well and are quite reliable. It is only a metter of knowing how and when to use each option available.
In regard to Bryce primitives and MetaBalls, it is also interesting to observe file size as a function of the materials used. Because the material must be saved for each Native Bryce Instance, [in case we have a different material on each object] and some complex multi-channel/multi-component materials have a huge database, the materials have a very great influence not only on render speed but also on saving and loading speed. This means that you can create MegaScenes with very simple materials on your objects and then save the scenes. Later you can load the scenes and then apply your special materials and then render. The resulting file cannot be saved anymore [this is when i get a crash] but it does not matter because you have accomplished your goal and you still have the master file.
Picture_815 shows the file sizes with various multiples of different materials applied. We notice that with a single material on all objects, the result is a large file size. As the number of different materials increases, the file size goes down [very strange]. Then as the number of different materials is increased even more, the file size goes up again [as we would expect]. Also complex materials result in large file sizes compared to very simple materials [as we would expect].
Note the difference between simple materials [no DTE channels] and those by David Brinnen [3 x DTE channels]:
81_Spheres_Tex_David_01_3CH and 81_Spheres_Tex_Simple.
729_Spheres_Tex_David_01_3CH and 729_Spheres_Tex_Simple.
The number of Texture components in the DTE [one, two or three] also would have a significant influence.
Well we can now move on to Part 5B
Kind regards
Peter
Daz 3D is part of 
Connect
DAZ Productions, Inc.
7533 S Center View Ct #4664
West Jordan, UT 84084
Licensing Agreement | Terms of Service | Privacy Policy | EULA
© 2024 Daz Productions Inc. All Rights Reserved.
Comments
Tutorial on the idiosyncrasies of the Instancing laboratory. Part 5B of 6
It has come to my notice that there remains some confusion about Instancing in Bryce, therefore i will dwell on this a little bit longer than i originally intended.
There is a difference between Instancing and Replication.
Because all objects take up memory, ways had to be found to reduce the amount of memory required for MegaScenes [this is an issue for ALL 3D/4D programs and not just for Bryce]. Replication makes a copy of the entire object database which includes the object description, the Linear transform matrix, the material and much else. If there are100 replica of an object then nearly 100 times as much memory is required. If those objects are Mesh Objects in Bryce then that is not good [see Part 5A above in this thread]. If the objects all look the same then that is a great waste of valuable resources.
Suppose you are building the interior of a great cathedral which has many elaborate ornaments and structural details which are repeated many times. It would make sense to have just one master of each type and then somehow create the duplicates from each master at rendertime. However it is not so simple because the non existent duplicates also have to be represented on the screen before rendertime in a simplified and fast rendering form [the most extreme of which is the rectilinear bounding box]. Otherwise we could not set up the scene and adjust it before rendering.
To understand what an instance is you need to understand how object data is stored in Bryce. When Bryce was first developed and before its primary function became a render cow for imported meshes, it had 5 master objects called primitives and also the 7th star of the combo called the Terrain which was [and still is] very special [ i think the Torus Primitive was added a little later - but i can't remember when]. There were also a few other things like infinite planes and so on. The 6 master objects [if we include the Torus] were mathematical descriptions and did not actually exist [there were also a few other Easter-egg Bryce Primitives which mysterious appeared from time to time]. In fact everything was procedural. What you had to do is create instances of the primitives by clicking on representations of them on the Create Palette. A stand-in representation was then created [looks like a mesh object but is nothing of the kind] and a matrix datebase was created for each instance. The description of the object was not included in the instance database.
Those days computers did not have much RAM and this Bryce technology meant that you could do things with Bryce you could not do with any other program. Also because Bryce objects were not mesh objects you could render perfectly smooth surfaces/edges which were resolution independed. In the early days you could have many more objects in Bryce than you could have in other 3D programs. That was only possible because Bryce was engineered on Instancing and on procedural "Master Objects". Gradually that changed over a decade. I won't spend too much time on the history but there was market pressure to either extend the procedural engines [as has happened in Cinema4D for example] or to make Bryce Mesh-Import-Export-Capable [to interface with DAZ-Studio]. So it came to pass that the procedural engines were never extended except to add Boolean Set Operations.
The stones were an interesting addition which came about from an idea to use the wireframe representation code. Since the wireframe representation can be easily converted to a mesh object [the data is already there] why not make a mesh generation engine? The Bryce Stones were the first Bryce Mesh Objects. Today we have several Bryce-generated Mesh Objects some are good and some are problematic [inverted normals] and the methods of making them are varied. But the Bryce Stones were not instances. Once a stone was created it was a Mesh Object. The more of them you had, the more memory they would eat up.
We will now have a look at how the Stones are created. There is a tree structure for the process and at each node a random decision is made [it would be nice to have a control panel so that we could make some of those decisions ourselves]. The first decision is to use either the Spherical or Cubic procedural generation engine [they are the sphere and cube Master Objects]. Next comes the perturbation of the mathematical "surface" with noise. Bryce already had a good fractal noise generator. The type and amount of noise are randomly selected. Next comes the conversion into the wireframe representation during which all the n-gons are triangulated [to avoid non-planar polygons being produced later]. At this stage also a random decision is made to select one of 3 degrees of resolution [the stones come in 3 different resolutions, low, medium and high - it would be nice to have a dialog to select that too]. Then the display representation is converted into a Mesh Object Database, describing vertices, edges and faces [which in Bryce cannot be selected and edited]. Finally the matrix is distorted randomly [most often squashed down the Y-axis] and a random material is added. Then it appears in the scene. Interestingly we can keep the applied smoothing across the edges or we can remove it by using the "E" button at the bottom of the attributes stack.
I know many Brycers have spent some time stone collecting. I have collected stones for over a decade and sometimes found interesting ones. In the early days the stones had a very high resolution but later the resolution was cut down to half.
In other 3D applications Mesh Objects can have a very large and complicated data base. This is necessary for character animation where individual vertices have to be moved smoothly. When a Mesh finally comes into Bryce it is as a cut down version of what it once was [ as the Bryce Stones are ] and you can't edit either vertices, edges or faces. The only option is to import a Mesh Object which has been deliberately cut up into many different pieces. You can then put a different material on each piece or even explode the object in an animation.
As more and larger Meshes were imported into Bryce a solution had to found to get around the limited memory problem. The instancing system native to Bryce could not be used for Mesh Objects. So the software engineers created a new instancing system specifically for Mesh Objects. There would be a master Mesh Object and then a very small database for each instance. Each database had a pointer to the Master. Furthermore if we change the material of the master all the instances would be updated. The Instance Database contains the matrix of the object - its position, size, orientation and so on. It also must be able to show a primitive wireframe representation without slowing down the computer too much. It is best to convert this to a bounding box which allows faster navigation. Grouped objects could not be instanced directly. To get around this there is a procedure already mentioned by Rashad [ he calls it “Trickery”] which i will again describe in detail below. Many thanks to Rashad for pointing it out!
There are many issues with instances which include saving, loading and rendering. Different 3D programs handle these things differently. Some have a pre-render pass with a message box stating "preparing objects". When you are loading Bryce MegaFiles full of instances Bryce often does informs you at the bottom left: "preparing objects". It is obvious that if you have 1000 instances of a polygonal mirrror sphere all over your scene reflecting each other there are are going to be a lot of jump vectors back and forth to the master not to mention the amount of stack space required which could result in memory overrun. So there is always a price to be paid. I don't know exactly how Bryce handles all this, but my impression now is that it is quite good at it, so long as one takes precautions. MegaScenes with a lot of instances take less time to save and load and seem to do so reliably, also memory size is very much reduced.
Now to some more interesting stuff.
Bryce has 4 tools for creating instances. Three of these tools are normally used for "replicating" objects. Remember when you "replicate" Bryce primitives, you are actually creating native instances [ so they don't need the instancing option selected and with some of those tools if you select it, nothing is replicated! ]. However when you replicate mesh objects, memory will be quickly eaten up. For mesh objects [and also for Bryce Native Trees] use the instancing option. It is as simple as that!
Now on to pictures and many details.
[075] The first tool is found under "Edit, Instance". There is no dialog for this one. If the option is greyed out it means the object cannot be instanced by the new Mesh Instancing System. When the tool tells you that you can instance an object, then select this option to make one instance which is created on top of the master. If you need many it is best to use another tool [see below] or select many objects without grouping them. It is best to keep all the masters in some location "underground" and identify them with a special mesh colour. This is where you go to change the materials of all your instances in a MegaScene. Very convenient because you don't have to find and select the instances in your MegaScene. If you do change the material of instances the master may update as well [depending on what they are and how they were created].
[076] Have a look at Picture_81 which shows the Advanced Replication and Distribution Lab [called "Instancing Lab"] which is what this tutorial was originally all about. Here you see a green button ["instance"] which you can turn off by clicking on it. Normally it should be left on because Bryce will not crash if you are working with objects which can't be instanced [unlike another tool i will describe below]. If they are Native Bryce and not grouped then they are automatically instances and if they are grouped objects of any kind [except for Trees and instance groups made by 'Trickery"] then they are simply replicated [and random rotation then does not function]. So you can leave this button on all the time. Only if you want to replicate meshes [deliberately make copies of them] should you turn this button off.
[077] When Native Bryce Objects are selected they are already instances. Picture_82 shows how when the instance of a Bryce Sphere is selected, "Instance" is greyed out. Rule No 1: Instances cannot be nested but replicates can [more about that in Parts 5C, 5D etc.].
Terrains are special objects. They are neither Mesh Objects nor instances of a mathematical equation. Picture_83 shows that Terrains also cannot be instanced. If you need many of them replicated, it is best to convert the master into a mesh object and export it and then import it into Bryce again. Bryce has a special Lab for doing that [with many optimization options]. After you have imported the Terrain Mesh Object back into Bryce you can instance it, which saves much memory. Native Bryce Terrains can have very high resolutions and relatively small memory requirements. They are very much superior to the terrains that can be created in other 3D software. So normally you don't need instances of Mesh Terrains. But if you use terrains and SymLattices to model non Terrain Objects then that is one way to go.
[078] Bryce comes with a library of many ready-made Mesh Objects which are quite useful. Picture_84 shows one of the Helical Spring Objects from the object library. It is a Mesh Object but it is a group of two pieces. I have ungrouped them and we can see that the parts [both selected] can now be instanced. The Group Object [ "container" ] cannot be instanced at the moment but the ARDL can handle them as you will see below.
[079] Trees also can be instanced even though they contain MetaBalls which normally cannot be instanced on their own. Please have a look at Picture_85. The Bryce software engineers did a great job in making this possible.
[080] Stones are Mesh Objects so they can obviously be instanced as shown by Picture_87.
[081] When the MetaBall Hull [constituted of two or more grouped MBs] is converted into a Mesh Object it can be instanced. Please have a look at Picture_88. If you look at Picture_811 of Part 5A of this tutorial you will notice the file size is is much larger [ 54,553 KB compared to 457 KB in Native Bryce Form]. Also the object does not render smoothly and can't be edited nor animated. So there is often not much advantage to doing this conversion which can take many hours. However if you create 30 instances of that thing [ 81_MetaBalls_Converted_Instance_Form.br7 ], the memory only goes from 54,553 KB to 54,728 KB and you have the equivalent of 2430 MetaBalls [but they no longer interact].
[082] Next we will consider what to do about Mesh Objects which are grouped [made of two or more parts]. Please have a look at Picture_89. On the left are two objects from the Object Library which are grouped to make one object. That is the Root Master. They were ungrouped, remained selected and one instance was created of each in the same relative position. The pair was then moved to the right [Please see green Mesh which is dotted]. The pair was then grouped and lo and behold, they remained instances. This is the Second Master to be used for instancing on a Surface Object.
Picture_90 shows how i used the ARDL to make a pattern on a Squashed Cone. The only drawback is that groups don't rotate randomly. If you want that, then you must create a family of Second Masters each rotated differently and then use a mixed brush in the ARDL. Please refer to Picture_91 to find out typical memory requirements. You can compare the 4 combinations: In the ARDL: Instances_Grouped and Non_Instances_Grouped and when using the Multi-Replicate Tool: Instances and No_Instances [copies]. Picture_92 shows the dotted wireframes in green. This is another bit of evidence that these are true instances, even though they are grouped. Unfortunately you can't remotely update the Materials of the Instances because they are grouped. To overcome this you can plan ahead and give each of the two parts a different wireframe colour and name. Then later you can ungroup the Instances and globally adjust the Materials on each subgroup.
[083] We come now to compare the Multi-Replicate and Random-Replicate tools. Picture_93 shows an Instance of the Bryce Sphere being replicated. Since the Bryce Sphere is already a native instance of the "old system" since Bryce 2, we do NOT check the "Instance" button. If you do click on the "Instance" button, Bryce takes some time to figure it out, finds recursive instancing and produces nothing. Fortunately i have not been able to induce a crash but we don't want to waste a lot of time waiting around for Bryce to produce nothing. Perhaps they can redesign this tool to conform to the ARDL because it must be very confusing and frustrating for Newbees.
The Random-Replicate tool behaves in the same way. Please compare Picture_95 with Picture_96. The Random-Replicate Tool has many useful options but in its current state of development will be very frustrating for beginners to learn and use. You need to know the Object Class you are using [Native Instances or Mesh Objects]. It is best to refer to Table_01 in Part 4 of this tutorial [ http://www.daz3d.com/index.php?&ACT=50&fid=38&aid=12961_DFu52tfnCRoMD7Jvppvk&board_id=1 ]. If in doubt you can find out!
There is an unfortunate bug with the Random-Replicate Tool. If you use a Tree as the Master and tick "Instance", Bryce will crash! It will always crash very reliably. Therefore use Multi-Replicate and/or the ARDL - Bryce does not crash when you use those.
[084] Picture_97 and Picture_98 show the Multi_Replicate Tool being used to either replicate or instance stones. It behaves predictably and please note the dotted wireframeswhich result when the "Instance" button is ticked. Also file sizes support that everything is working as we imagine and should work.
[085] Since Stones are Mesh Objects - but are created inside Bryce [using a process over which we have no control], when Stones are Random Replicated [not instanced] using the Random-Replicate Tool, every stone will be different. Please study Picture_99. This is useful when you want to find the right kinds of stones for your purposes. Simply create a large number of them and delete the ones you don't want.
However when the "Instance" button is checked, the Random-Replicate Tool behaves differently. All the stones are the same as the Master but they are rotated. These are true Instances as shown by the dotted wireframes you see in Picture_100.
There is still more we can investigate [ such as how different kinds of groupings behave and multiple replications done simultaniously ] but i think it is sufficient for us to move forward with the more creative parts of this tutorial.
What is to come will include building forms to be coated with MetaBalls, distributing Composit Plates on Terrrains and using Nested Masters.
kind regards,
Peter
Once again, thank you Peter for sharing your insights. You have answered questions that were only really beginning to form in my mind and provided some interesting information about the workings of the primitives.
You have overcome what to me seems an unwieldy interface in the IL and uncovered the hidden gems. I look forwards to further revelations!
But I'm going to have to read it all over again though, probably many times, before it sticks. I get about half way through any page and it all becomes a bit too much to take in.
Hello David.
Yes i understand were you are coming from. Unlike the wonderful German Cinema4D software, Bryce has many different systems of objects which are not integrated. I can't do anything about that, because i have not been invited to help the software engineers - not even as a beta tester.
It is, i suppose a question of money! Still we have to be thankful for having such a wonderful tool which is offered [at the moment free of charge] thanks to DAZ3D. I absolutely support DAZ3D in this project and i support your vision for Bryce too.
There is nothing in Bryce 7.1 Pro i need to complain about .
The displacement option in the Materials Lab works fine - if not as good as it might be [as in Cineme4D were we have a superb implementation of that concept ]. I know it is an experimental feature in Bryce. It works fine for me [ it can cause crashes if one is not careful ]. Interestingly the displacement channel is driven by the colour channel, leaving the alpha and bump free for other purposes. As a final judgement i dare say that displacement is not a particularly urgently needed feature [ not as bad as the stupid Easter Egg Particle Emitter which is totally useless waste of code ].
Well i will just battle on.
Kind regards
Peter.
I suppose i may show a flowchart of all the Bryce Object Systems and how the interact.
Thank you for the series! That's really wide knowledge, detailed and extremely useful guide.
@Peter - well, I'm collecting your insights and will finally have to print and read it once in one go to get a complete overview and then going to experiment with the details. You are doing an incredible job with this series.
I am confident that there will come a time when Bryce is going to get a bit of attention from DAZ 3D. Once this happens, new beta testers will be invited. I was on the team starting with the 6.0 development cycle and with each cycle, a few beta testers were exchanged. After the development of 7.1 was concluded, the team was dissolved (is this the correct word?). With the 6.3 cycle, a small steering commettee was assembled, which was also terminated at the end of 7.1. I wonder whether DAZ 3D will assemble such a commettee again, but in my view, it proved to be very productive.
"the stupid Easter Egg Particle Emitter" was there already from an earlier version, probably Bryce 4. One of the programmers stumbled over the code more or less by accident and we decided to have it activated but still as easter egg. It does work but the use is limited. There was no time to develop it further because we had more pressing matters on our minds in the SC.
"I suppose i may show a flowchart of all the Bryce Object Systems and how the interact." If you're a fan of such flow charts (I am), there is one I compiled from all the DTE videos David made. DAZ 3D said it would be added as an appendix to the doc but since that doc has been left dormant for 1-1/2 years now, I got the permission to host it on my website. If you're interested, use this link (but only when your series is concluded ;)): http://www.horo.ch/raytracing/tuts/pdf/DTEdoc.zip To be precise, it is not actually a flow chart but a block diagram of the "synthesizer".
Thanks dwsel,
As an old person who will soon pass away [soon in my time-scale of course] i have a duty to pass on to the younger generation all i know.
My knowledge of Bryce is not really "wide" [compared to Horo, Rashad and David] but i am learning all the time. When i encounter a problem i try to get to the bottom of it.
Kind regards,
Peter
Thanks Horo,
I must get on with the remaining Parts of this tutorial because i have some others in the pipeline. I am sorry this one is so fragmented. I should have completed all of it before posting anything [and let you read it firsts and then perhaps you give me advice]. I won't make that mistake again. Perhaps later we can consolidate this tutorial and clean it up and emit it through other channels.
BYW i have been doing a lot of other tests [including many different Render Perspective Transformers [RPTs] - many suitable for making light probes in Bryce [in 1:1 and 1:2] but also for many kinds of curved perspective]. Because i have two computers and each can handle two tasks at the same time [up to 4 tasks per computer when Bryce uses only one thread] i can afford to take some risks and tie up a computer for day or two. I have done a Multi-Replicate of 10,000 MetaBalls to make them 20,000. Yes it worked OK! Bryce used only one thread to do this [25%]. It took 19 hours !!! [i would not even try this with 2000 Mesh Objects!] I used the simplest material possible. Various bits of evidence i have collected tend to reinforce my suspicion that the Native Bryce MB Instances each contain an Independent Material Database [ with the size depending on the material]. When i tried to change the material to something complex on all the objects at once, Bryce crashed on me. "Run out of memory" dialog wanted me to klick OK 20,000 times! It would be nice that when Bryce stores 20,000 instances, all with the same material, it could save only one copy of the Material Database and not 20,000 copies! In practice of course you would not have two objects composed of 10,000 MetaBalls each. More likely 20 objects of 1000 MetaBalls each [on average]. If i have time i might test that idea. Store the 20 objects in the User Object Library each with a different material(s) and then load them into one of my scenes.
Ah the "synthesizer". The people that founded Bryce were into Electronic Music and so were you and i. In fact i was working with "synthesizers" for music at the time i discovered Bryce 1 entirely by chance. Yes i know basically how the Bryce "synthesizer" works. I am sure many Newbees are perplexed why some things in the DTE are called "noise", "phase" and there are things called "filters". I am sure Horo that with your professional background, you feel at home in there.
Well i must stop talking and battle on.
Kind regards
Peter
I do feel very comfortable with all those terms and what they mean. Where I struggle is the art part. A techie seldom makes a good artist. Oh, we do have a success now an then, mostly by chance. But I don't mind, it's fun. We can't be professonals in every domain. Push the limits as far as you can but don't get frustrated when you hit a wall you cannot climb, and be happy with what you've managed to accomplish.