Reduce texture sizes easily with this script.

24567

Comments

  • hphoenixhphoenix Posts: 1,335
    edited December 2016

    I found an issue with the Image.scale() and Image.smoothScale() methods in Qt......they trash any alpha channel in the image.  I tried this method, and the alpha ends up pure black.

    edit:  Just tried opening and resaving the image in Photoshop.  Fixed the issue.  So something in how DS is loading the image that Image.scale() and Image.save() encodes messes up the alpha channel.  Photoshop shows the alpha in the saved image normally, just resaved it as a PNG (no interlacing) and DS loads it properly again.  Not sure just what is happening, but it looks like it gets saved incorrectly and Qt can't correctly load the alpha back into DS.

     

    Post edited by hphoenix on
  • EsemwyEsemwy Posts: 578
    hphoenix said:

    I found an issue with the Image.scale() and Image.smoothScale() methods in Qt......they trash any alpha channel in the image.  I tried this method, and the alpha ends up pure black.

     

    Interesting. Shouldn't make any difference with JPEGs, but could completely screw PNGs. I haven't used scale on anything where alpha mattered. 

  • barbultbarbult Posts: 24,244
    Esemwy said:
    barbult said:

    I'm getting a bunch of warnings in the log file, even when the script seems to be working. Do I need to be concerned about them?

    2016-12-29 22:55:03.743 WARNING: general\dzcontentmgr.cpp(4341): Content directory index out of range in DzContentMgr::getContentDirectory()

    Change g_CONTENT_PATH_NUM at the top of the script from 1 to 0. 

    That eliminated the logged warning, thanks.

    As I continue to use this script, I found a scenario that I had trouble with. I had a model that used LIE on the face and lips. The CTRL-script created a reduced texture file, in the same folder as the original texture file. But in the case of LIE, that is the tmp folder. When I saved the scene with reduced textures and reloaded it, DS couldn't find the tmp file. I wasn't surprised by this. In fact, I would have been surprised if it worked, actually. It is just something to be aware of if you plan to save scenes with reduced textures and you have used LIE. You can work around it by saving the reduced texture tmp file in your content library and substituting that saved file for the tmp file in the surfaces pane before saving the scene.

     

  • mjc1016mjc1016 Posts: 15,001
    hphoenix said:

    I found an issue with the Image.scale() and Image.smoothScale() methods in Qt......they trash any alpha channel in the image.  I tried this method, and the alpha ends up pure black.

     

    If Daz's implementation of it is the full version (it should be) then it shouldn't trash the alpha...

    http://doc.qt.io/qt-4.8/qimage.html

  • hphoenixhphoenix Posts: 1,335
    edited December 2016
    mjc1016 said:
    hphoenix said:

    I found an issue with the Image.scale() and Image.smoothScale() methods in Qt......they trash any alpha channel in the image.  I tried this method, and the alpha ends up pure black.

     

    If Daz's implementation of it is the full version (it should be) then it shouldn't trash the alpha...

    http://doc.qt.io/qt-4.8/qimage.html

    Looks like it's a bug in Qt.  Just ran it through the script debugger, watching carefully.  The original Image pixel at 0,0 is <255,255,255,255> ARGB (100% transparent).  After the scale, the new image has a pixel at 0,0 of <255,0,0,0> ARGB.  The original has a value of 255, the scaled has a value of 0 (hue and saturation remained the same).

    That's not correct.  Color values should be retained. (the adjacent pixels are also full 100% transparent white)  But they aren't.

    So technically, it isn't trashing the alpha, it's trashing the colors due to the alpha....

     

    Post edited by hphoenix on
  • barbultbarbult Posts: 24,244

    Here are some log statistics:

    Full resolution textures:
    2016-12-30 00:08:10.362 Iray VERBOSE - module:category(IRAY:RENDER):   1.6   IRAY   rend stat : Texture memory consumption: 1.49292 GiB (device 0)

    Reduced resoluton textures:
    2016-12-29 23:57:12.543 Iray VERBOSE - module:category(IRAY:RENDER):   1.6   IRAY   rend stat : Texture memory consumption: 718.188 MiB (device 0)

    This was a simple test with Darius 7 wearing the Santa Suit, no fancy lighting or posing. I can see tiny differences in the rendered images, but I really have to toggle between them and pixel peep to detect the difference. In the case of this simple test, both full resolution and reduced resolution fit easily on my graphics card, so render time was about the same for both. I look forward to trying a scene where the reduced images make the difference between GPU vs CPU rendering.

     

     

    Full Resolution texture files.jpg
    2000 x 1500 - 809K
    Reduced texture files.jpg
    2000 x 1500 - 818K
  • barbultbarbult Posts: 24,244

    I'm trying to run the script on the free Christmas Dinner scene, because the textures are PNG files, so I can see what happens. I can't get the script to do anything with this product. The log says the script ran successfully, but none of the textures were changed. In this case, I have the product installed with the DIM, not Daz Connect. The props in the scene are all parts of DS groups. Is that an issue? Even if I unparent them from the group, the script still doesn't create the reduced resolution files.

     

  • EsemwyEsemwy Posts: 578
    barbult said:

    I'm trying to run the script on the free Christmas Dinner scene, because the textures are PNG files, so I can see what happens. I can't get the script to do anything with this product. The log says the script ran successfully, but none of the textures were changed. In this case, I have the product installed with the DIM, not Daz Connect. The props in the scene are all parts of DS groups. Is that an issue? Even if I unparent them from the group, the script still doesn't create the reduced resolution files.

    Try this version. I was trying to force a conversion to JPEG, but that might not work.

     

  • thank you so much!

  • mjc1016mjc1016 Posts: 15,001
    hphoenix said:
    mjc1016 said:
    hphoenix said:

    I found an issue with the Image.scale() and Image.smoothScale() methods in Qt......they trash any alpha channel in the image.  I tried this method, and the alpha ends up pure black.

     

    If Daz's implementation of it is the full version (it should be) then it shouldn't trash the alpha...

    http://doc.qt.io/qt-4.8/qimage.html

    Looks like it's a bug in Qt.  Just ran it through the script debugger, watching carefully.  The original Image pixel at 0,0 is <255,255,255,255> ARGB (100% transparent).  After the scale, the new image has a pixel at 0,0 of <255,0,0,0> ARGB.  The original has a value of 255, the scaled has a value of 0 (hue and saturation remained the same).

    That's not correct.  Color values should be retained. (the adjacent pixels are also full 100% transparent white)  But they aren't.

    So technically, it isn't trashing the alpha, it's trashing the colors due to the alpha....

     

    It looks like it may be premultiplying the alpha..which it shouldn't do unless it's told to (at least that's my understanding of Qt docs).

  • barbultbarbult Posts: 24,244
    edited December 2016
    Esemwy said:
    barbult said:

    I'm trying to run the script on the free Christmas Dinner scene, because the textures are PNG files, so I can see what happens. I can't get the script to do anything with this product. The log says the script ran successfully, but none of the textures were changed. In this case, I have the product installed with the DIM, not Daz Connect. The props in the scene are all parts of DS groups. Is that an issue? Even if I unparent them from the group, the script still doesn't create the reduced resolution files.

    Try this version. I was trying to force a conversion to JPEG, but that might not work.

     

    No, it still seems to do nothing with PNG texture files. I also noticed that the beginning still say var g_CONTENT_PATH_NUM = 1, where before you asked me to change it from 1 to 0. Is that change from 1 to 0 still recommended?

    Edited: sometimes png works.

    Post edited by barbult on
  • barbultbarbult Posts: 24,244

    No, my previous conclusion is not correct. I can get it to work in some scenes with PNG files. Sometimes, in some scenes, it works and sometimes, in other scenes it does not. I will continue to investigate and see if I can conclude why that is.

  • barbultbarbult Posts: 24,244
    edited December 2016

    I think it is working on "figures" but not "props"  for both JPEG and PNG. Everything in that Christmas Dinner scene that didn't work for me was a "prop". (I hope I have that terminology correct. If not, please correct me.)

     

    reduce images problem.jpg
    580 x 321 - 76K
    Post edited by barbult on
  • EsemwyEsemwy Posts: 578

    @barbult : the change to g_CONTENT_PATH_NUM was just for people that get an error. Normally 1 will be the correct value.

  • barbultbarbult Posts: 24,244

    I confirm that the reduced size PNG files with transparency are not working as expected. However, a reduced TIFF with transparency seems to work OK for me so far.

  • hphoenixhphoenix Posts: 1,335
    edited December 2016
    mjc1016 said:
    hphoenix said:
    mjc1016 said:
    hphoenix said:

    I found an issue with the Image.scale() and Image.smoothScale() methods in Qt......they trash any alpha channel in the image.  I tried this method, and the alpha ends up pure black.

     

    If Daz's implementation of it is the full version (it should be) then it shouldn't trash the alpha...

    http://doc.qt.io/qt-4.8/qimage.html

    Looks like it's a bug in Qt.  Just ran it through the script debugger, watching carefully.  The original Image pixel at 0,0 is <255,255,255,255> ARGB (100% transparent).  After the scale, the new image has a pixel at 0,0 of <255,0,0,0> ARGB.  The original has a value of 255, the scaled has a value of 0 (hue and saturation remained the same).

    That's not correct.  Color values should be retained. (the adjacent pixels are also full 100% transparent white)  But they aren't.

    So technically, it isn't trashing the alpha, it's trashing the colors due to the alpha....

     

    It looks like it may be premultiplying the alpha..which it shouldn't do unless it's told to (at least that's my understanding of Qt docs).

    That was my conclusion as well.....which means it's a pixel format issue.  There are some ways to try to tell Qt to use a specific one, which I'll try here in a bit to see if I can work around the issue.....

    edit:  Grr....not enough exposed by DS in the Image wrapper to access QImage Format data.  That basically puts the nail in the coffin for me using the internal scaling.  Too many materials use alpha channels and PNG format.

    So I guess the question is, is this a bug in DS (and how it handles the Format parameter during image scaling for PNG fomats) or in Qt itself (and can Daz do a workaround fix in DS)?

    Post edited by hphoenix on
  • hphoenix said:
    mjc1016 said:
    hphoenix said:
    mjc1016 said:
    hphoenix said:

    I found an issue with the Image.scale() and Image.smoothScale() methods in Qt......they trash any alpha channel in the image.  I tried this method, and the alpha ends up pure black.

     

    If Daz's implementation of it is the full version (it should be) then it shouldn't trash the alpha...

    http://doc.qt.io/qt-4.8/qimage.html

    Looks like it's a bug in Qt.  Just ran it through the script debugger, watching carefully.  The original Image pixel at 0,0 is <255,255,255,255> ARGB (100% transparent).  After the scale, the new image has a pixel at 0,0 of <255,0,0,0> ARGB.  The original has a value of 255, the scaled has a value of 0 (hue and saturation remained the same).

    That's not correct.  Color values should be retained. (the adjacent pixels are also full 100% transparent white)  But they aren't.

    So technically, it isn't trashing the alpha, it's trashing the colors due to the alpha....

     

    It looks like it may be premultiplying the alpha..which it shouldn't do unless it's told to (at least that's my understanding of Qt docs).

    That was my conclusion as well.....which means it's a pixel format issue.  There are some ways to try to tell Qt to use a specific one, which I'll try here in a bit to see if I can work around the issue.....

    edit:  Grr....not enough exposed by DS in the Image wrapper to access QImage Format data.  That basically puts the nail in the coffin for me using the internal scaling.  Too many materials use alpha channels and PNG format.

    So I guess the question is, is this a bug in DS (and how it handles the Format parameter during image scaling for PNG fomats) or in Qt itself (and can Daz do a workaround fix in DS)?

    You could make a bug report/feature request and see what the response is.

  • hphoenixhphoenix Posts: 1,335
    hphoenix said:
    mjc1016 said:
    hphoenix said:
    mjc1016 said:
    hphoenix said:

    I found an issue with the Image.scale() and Image.smoothScale() methods in Qt......they trash any alpha channel in the image.  I tried this method, and the alpha ends up pure black.

     

    If Daz's implementation of it is the full version (it should be) then it shouldn't trash the alpha...

    http://doc.qt.io/qt-4.8/qimage.html

    Looks like it's a bug in Qt.  Just ran it through the script debugger, watching carefully.  The original Image pixel at 0,0 is <255,255,255,255> ARGB (100% transparent).  After the scale, the new image has a pixel at 0,0 of <255,0,0,0> ARGB.  The original has a value of 255, the scaled has a value of 0 (hue and saturation remained the same).

    That's not correct.  Color values should be retained. (the adjacent pixels are also full 100% transparent white)  But they aren't.

    So technically, it isn't trashing the alpha, it's trashing the colors due to the alpha....

     

    It looks like it may be premultiplying the alpha..which it shouldn't do unless it's told to (at least that's my understanding of Qt docs).

    That was my conclusion as well.....which means it's a pixel format issue.  There are some ways to try to tell Qt to use a specific one, which I'll try here in a bit to see if I can work around the issue.....

    edit:  Grr....not enough exposed by DS in the Image wrapper to access QImage Format data.  That basically puts the nail in the coffin for me using the internal scaling.  Too many materials use alpha channels and PNG format.

    So I guess the question is, is this a bug in DS (and how it handles the Format parameter during image scaling for PNG fomats) or in Qt itself (and can Daz do a workaround fix in DS)?

    You could make a bug report/feature request and see what the response is.

    I'm now thinking it's a problem with the actual libpng library.  ImageMagick is doing the same thing.  I can dissociate the alpha from the color channels, but that basically removes it.  Not what is correct.  The PNG standard states it does NOT pre-multiply the alpha with the RGB space, but it certainly seems like it is doing it.

    Wherever the PNG is 100% transparent, the color gets changed to black.  This is a bad thing for overlay textures, as the base color won't come through (as it will if the color is white.)

    But it only does it on output.....it seems to read the original just fine, but when it goes to encode the data as PNG, it munges the color data where there is alpha.

     

  • EsemwyEsemwy Posts: 578

    @barbult : I think I have a solution for you but it will take me a bit to re-jigger things. 

  • EsemwyEsemwy Posts: 578

    OK, easier than I thought. This should keep it from skipping props/primitives. I've also replaced the script in the original post.

    dsa
    dsa
    Reduce Texture.dsa
    3K
  • barbultbarbult Posts: 24,244
    Esemwy said:

    OK, easier than I thought. This should keep it from skipping props/primitives. I've also replaced the script in the original post.

    Nope, this version behaves the same as the previous one. It works on the primitive sphere that I converted to a figure, but not on the original primitive sphere prop. It doesn't work on props that I load from the Content Library, either. I tried that Christmas Dinner product again, and I tried a very old product, the Baking Set. Those both load the items as props rather than figures, and the script won't make reduced textures.

    Did it work for you?

  • EsemwyEsemwy Posts: 578
    barbult said:
    Esemwy said:

    OK, easier than I thought. This should keep it from skipping props/primitives. I've also replaced the script in the original post.

    Nope, this version behaves the same as the previous one. It works on the primitive sphere that I converted to a figure, but not on the original primitive sphere prop. It doesn't work on props that I load from the Content Library, either. I tried that Christmas Dinner product again, and I tried a very old product, the Baking Set. Those both load the items as props rather than figures, and the script won't make reduced textures.

    Did it work for you?

    It worked just fine for me. I wonder if the forum software cached it wrongly when you downloaded it. It should now iterate over all selected nodes. I just verified that what's posted is the latest version. I may have to send you a version with some logging to figure it out.

  • barbultbarbult Posts: 24,244
    edited December 2016

    It is working for me today! I wonder if rebooting my computer and restarting Daz Studio made a difference? Do you suppose an old version could have been cached and still been used, even though I loaded the new version yesterday? I've never noticed that happening before. Maybe user error?

    I got it to work on the primitive sphere today, and I can get it to work on the items in Christmas Dinner as long as I unparent them from the nested Groups that the artist put them in. Thanks for all your hard work to perfect this. Is there anyway for it to look inside Groups to find things? (I feel guilty even asking you to consider doing more work. blushIt is already so great!)

    Edit: Maybe the script doesn't need to look inside groups. The scene right click context menu will find those things for me. See my post below.

    Post edited by barbult on
  • barbultbarbult Posts: 24,244
    edited December 2016

    This Christmas Dinner product is a bear to reduce. It is full of nested Groups and parented items and instances. But I vow to get it done.

    Edit: Darn, I thought I was onto something. I selected the top level group and expanded everything and then unparented it. Then I selected all the props and ran the script, which worked fine. Unfortunately, unparenting things messed up the scene and things are in the wrong place now. Back to the drawing board.

    Edit 2: OMG! This was easy! I didn't have to unparent anything or sort through any nested things. All I had to do was right click in the Scene pane and choose Select All Objects. Then I held down the CTRL key and run the script. It took a few seconds for the script window to open and show progress, but then it took off and completed the job!

    Post edited by barbult on
  • EsemwyEsemwy Posts: 578
    barbult said:

    This Christmas Dinner product is a bear to reduce. It is full of nested Groups and parented items and instances. But I vow to get it done.

    Edit: Darn, I thought I was onto something. I selected the top level group and expanded everything and then unparented it. Then I selected all the props and ran the script, which worked fine. Unfortunately, unparenting things messed up the scene and things are in the wrong place now. Back to the drawing board.

    Edit 2: OMG! This was easy! I didn't have to unparent anything or sort through any nested things. All I had to do was right click in the Scene pane and choose Select All Objects. Then I held down the CTRL key and run the script. It took a few seconds for the script window to open and show progress, but then it took off and completed the job!

    Excellent! Select children should also work if you want to limit the effect to only some scene elements. 

  • barbultbarbult Posts: 24,244
    edited December 2016

    Here is an image I did last night that never would have fit in my graphics card memory without reducing textures. I tried. It crashed Daz - twice. It didn't even roll over to CPU. It just crashed for some reason. Even after reducing textures (some more than once) it barely fit. So thank you again, Esemwy, for this great new tool. It kept me from having to render separate images and layer them.


    Where did I put the Keys? Oh, Daddy, not again!

    Lighting and background are from No Emotion HDRs http://noemotionhdrs.net/index.html. It is released under the Creative Commons Attribution-NoDerivatives 4.0  https://creativecommons.org/licenses/by-nd/4.0/

    Where did I put the keys.jpg
    2000 x 1500 - 2M
    Post edited by barbult on
  • hphoenixhphoenix Posts: 1,335

    Okay, a little more research done through the debugger......It looks like Qt (or the DAZ Wrapper class) is ignoring the alpha channel values.  When I load a PNG file with an alpha channel, it recognizes it as having an alpha channel (image.alphaBuffer = true) or not, but EVERY pixel has alpha set to 255.....regardless of the ACTUAL alpha value (tested with known PNG images, and with pixels validated as having 0% or 100% opacity.)  If the original image pixel is 100% transparent, the image.pixel(x,y).alpha = 255.  If the original image pixel is 0% transparent, the image.pixel(x,y).alpha = 255!

    Even when the alpha channel is not present in the PNG (image.alphaBuffer = false), any pixel value retrieved will have a 100% transparent alpha (image.pixel(x,y).alpha = 255) which is DEFINITELY NOT CORRECT.

    This is the problem.

     

  • mjc1016mjc1016 Posts: 15,001
    hphoenix said:

    Okay, a little more research done through the debugger......It looks like Qt (or the DAZ Wrapper class) is ignoring the alpha channel values.  When I load a PNG file with an alpha channel, it recognizes it as having an alpha channel (image.alphaBuffer = true) or not, but EVERY pixel has alpha set to 255.....regardless of the ACTUAL alpha value (tested with known PNG images, and with pixels validated as having 0% or 100% opacity.)  If the original image pixel is 100% transparent, the image.pixel(x,y).alpha = 255.  If the original image pixel is 0% transparent, the image.pixel(x,y).alpha = 255!

    Even when the alpha channel is not present in the PNG (image.alphaBuffer = false), any pixel value retrieved will have a 100% transparent alpha (image.pixel(x,y).alpha = 255) which is DEFINITELY NOT CORRECT.

    This is the problem.

     

    Yee-ouch!

    Bug report time?

  • hphoenixhphoenix Posts: 1,335
    mjc1016 said:
    hphoenix said:

    Okay, a little more research done through the debugger......It looks like Qt (or the DAZ Wrapper class) is ignoring the alpha channel values.  When I load a PNG file with an alpha channel, it recognizes it as having an alpha channel (image.alphaBuffer = true) or not, but EVERY pixel has alpha set to 255.....regardless of the ACTUAL alpha value (tested with known PNG images, and with pixels validated as having 0% or 100% opacity.)  If the original image pixel is 100% transparent, the image.pixel(x,y).alpha = 255.  If the original image pixel is 0% transparent, the image.pixel(x,y).alpha = 255!

    Even when the alpha channel is not present in the PNG (image.alphaBuffer = false), any pixel value retrieved will have a 100% transparent alpha (image.pixel(x,y).alpha = 255) which is DEFINITELY NOT CORRECT.

    This is the problem.

     

    Yee-ouch!

    Bug report time?

    Yep.  And what's more telling is the original (unresized) image loads and is used fine by DS.  But when the script tries to examine the same image (loaded via the Image() wrapper) it shows the incorrect behavior.  This leads me to believe it's a bug in the DS Script Wrapper class, NOT Qt.

    Now, I've been testing in DS 4.8, as I just recently re-loaded the OS on my big-honking graphics monster (the one with dual 1080s) and while I have DS 4.9 (both beta and release) loaded on it, I haven't transferred the needed files over to test with.  But I can do that test pretty quickly.  Will do it after dinner, and post results.  If it still shows the incorrect behavior, I'll be filing a bug report on it.

     

  • 3Diva3Diva Posts: 11,527

    Oh nice! This works great! :D 

    No need for high res textures for things way in the background of the scene, so this is brilliant! :) Thank you!

Sign In or Register to comment.