GLBasic forum

Codesnippets => Code Snippets => Topic started by: Wampus on 2012-Feb-05

Title: Compressing a file/array by saving as PNG
Post by: Wampus on 2012-Feb-05
Its possible to use the compression method used when saving PNG files to compress pretty much anything. Attached to this post is some code which does just that. Works nicely for large files. For smaller files the advantage diminishes since saving array data as a PNG file will add ~800 bytes of data.

The functions:-

LoadDataSprite( filename$ )
Loads a file into the array datasprite%[]

SaveDataSprite( filename$ )
Saves a file from the data contained in the array datasprite%[]

DeflateDataSprite (filename$ )
Compresses and saves the data in array datasprite%[]

InflateDataSprite ( filename$ )
Loads and uncompresses a file and puts the data into datasprite%[]

ClearDataSprite()
Clears the array datasprite%[] and size register

[attachment deleted by admin]
Title: Re: Compressing a file/array by saving as PNG
Post by: Wampus on 2012-Feb-06
n/p if I could find a way to get GLBasic to turn off saving superfluous palette info when saving PNG files then that reduces every file by 780 bytes. I know there is an option in pnglib to turn off palette saving for RGB files so maybe there is a way to override default GLBasic using that.
Title: Re: Compressing a file/array by saving as PNG
Post by: Moru on 2012-Feb-06
Wow, great idea, thanks!
Title: Re: Compressing a file/array by saving as PNG
Post by: Kitty Hello on 2012-Feb-06
Blimey, is this an awesome idea.
I'll try to cut the palette out. Do you know the parameter off your mind?
Title: Re: Compressing a file/array by saving as PNG
Post by: Wampus on 2012-Feb-06
Cool! Without the palette files would be real small. :)

I'm not sure how to cut the palette out completely but I have a couple of guesses. The official documentation for libpng says the palette (PLTE chunk) is optional for truecolor images with alpha but doesn't say what parameter to change when writing a file! In the source code I found a PNG_NO_READ_OPT_PLTE defined to prevent reading unnecessary optional PLTE data. I couldn't find the equivalent for turning off writing the optional palette.

In the source code it may be that color_type parameter being set to PNG_COLOR_TYPE_RGB_ALPHA is all that's needed. It is probably more complicated than that.

If the PLTE chunk can't be switched off in some way, maybe it can be reduced. Currently GLBASIC saves the full 256 optional palette. If the number of palette entries could be reduced to 0, 1 or 2 the PLTE chunk might still appear in file but only take up a few bytes instead of 780. There is a num_palette parameter I believe.
Title: Re: Compressing a file/array by saving as PNG
Post by: Kitty Hello on 2012-Feb-07
Code (glbasic) Select

   png_set_IHDR(png_ptr, info_ptr, width, height, 8, PNG_COLOR_TYPE_RGB_ALPHA,
      PNG_INTERLACE_NONE, PNG_COMPRESSION_TYPE_DEFAULT, PNG_FILTER_TYPE_DEFAULT);



   /* set the palette if there is one.  REQUIRED for indexed-color images */
   palette = (png_colorp)png_malloc(png_ptr, PNG_MAX_PALETTE_LENGTH
             * sizeof (png_color));
   /* ... set palette colors ... */
   png_set_PLTE(png_ptr, info_ptr, palette, PNG_MAX_PALETTE_LENGTH);
   /* You must not free palette here, because png_set_PLTE only makes a link to
      the palette that you malloced.  Wait until you are about to destroy
      the png structure. */


just remove the last 2 calls?
Title: Re: Compressing a file/array by saving as PNG
Post by: mentalthink on 2012-Feb-07
HI I don´t understand very well this functions... but I think it´s a png... very compresed in a xml file?¿ sorry for my ignorance...

And a question some time ago... I comment about streaming reading great images... whit this functions can be read a bit package of info, and finaly have the complete info and draw the huge Image?¿... I read the info goes into an Array... can be possible...

Title: Re: Compressing a file/array by saving as PNG
Post by: Kitty Hello on 2012-Feb-07
@metalthink - He uses PNG "data" to save anything. Sound, Music, Level data, ...
Title: Re: Compressing a file/array by saving as PNG
Post by: mentalthink on 2012-Feb-07
ok, don´t understand too much, but I suppose someone in the forum put an example...   :-* :-*
Title: Re: Compressing a file/array by saving as PNG
Post by: Ruidesco on 2012-Feb-07
He is using the compression algorithm that PNG files use for any kind of file.
Title: Re: Compressing a file/array by saving as PNG
Post by: mentalthink on 2012-Feb-07
ok  :good:, curious!!!, it´s the first time I look use a image compresion for any kind of file...  :blink: really interesting... I have to read a lot of more about programming    :-[ :-[ :-[
Title: Re: Compressing a file/array by saving as PNG
Post by: Slydog on 2012-Feb-07
@mentalthink, don't worry, I wouldn't consider this traditional programming in any way.
Wampus is exploiting the png compression algorithm to store ANY type of data, not just image related.

Ha, I've been programming for 30 years, and have never done this, or have even thought about doing this.  Very ingenious idea!

You no longer need to program your own zip libraries, just use the device's built in png libraries for the same purpose.  (I assume it use's the device's native png library?).  And PNG files are lossless, so this works great (for example, JPGs wouldn't work as they loose some data in order to make the compression better, so when restoring, the data would have changed).

:good:
Title: Re: Compressing a file/array by saving as PNG
Post by: spacefractal on 2012-Feb-07
Actuelly a very cool abuse of the png compression.

The only I concern os about the Loading:

- If graphics is set to 16 or 15bit, often the texture would loaded as 16bit, and hence content would been changed. Howover since I seen its using memory and not graphics memory, I think its dont happens here?
Title: Re: Compressing a file/array by saving as PNG
Post by: mentalthink on 2012-Feb-07
@Slydog,  HI thanks for your consideration, but sometimes... well , a lot  :(, I look very impressive things in your codes... I have envious healthy  :P :-[
Title: Re: Compressing a file/array by saving as PNG
Post by: Wampus on 2012-Feb-08
Slydog my code folder is full of failed experiments. This is a rare one that seems to have paid off...so far. I haven't tested it on all my devices yet. Tomorrow I will.


Quote from: Kitty Hello on 2012-Feb-07
just remove the last 2 calls?

I'm fairly sure removing those two calls will work as intended. Ultimately the test will be to see if the generated PNG files can be opened in browsers/image editors, etc. It should work though, in theory, since the palette is only strictly necessary for images with indexed color. I've seen source code that leaves out those two lines. Example from a cross-platform PNG library - file writepng.c (http://enchantia.com/graphapp/package/src/imgfmt/writepng.c) :-

Code (glbasic) Select
/* Set the palette if there is one.
* REQUIRED for indexed-color images. */
if (img->depth <= 8)
{
int i;
palette = (png_colorp) png_malloc(png_ptr,
256 * sizeof (png_color));
/* Set palette colors. */
for (i=0; i < img->cmap_size; ++i)
{
palette[i].red   = img->cmap[i].red;
palette[i].green = img->cmap[i].green;
palette[i].blue  = img->cmap[i].blue;
/* palette[i].alpha = 256 - img->cmap[i].alpha; */
}
png_set_PLTE(png_ptr, info_ptr, palette, img->cmap_size);
}


So, should work. :)


Quote from: mentalthink on 2012-Feb-07
ok, don´t understand too much, but I suppose someone in the forum put an example...   :-* :-*

I will do that soon. I need to improve how the files are saved. Currently the routine saves two files where only one should be needed. I'll update the code and post a better documented example...probably tomorrow.


Quote from: spacefractal on 2012-Feb-07
- If graphics is set to 16 or 15bit, often the texture would loaded as 16bit, and hence content would been changed. Howover since I seen its using memory and not graphics memory, I think its dont happens here?

I've just tested this: GLBasic will load PNGs with 256 indexed color, RGBA color with 8bit channels and RGBA with 16bit channels just fine. All work (on the PC at the least). However, when you save a PNG file in GLBasic it will always save as RGBA with 8bit channels. Because of this you'll never compress a file with the PNGZ code that will be treated as 16bit and look odd. Even if that were possible there would be a way to work with it.
Title: Re: Compressing a file/array by saving as PNG
Post by: hardyx on 2012-Feb-08
Quote from: Kitty Hello on 2012-Feb-07
just remove the last 2 calls?
I think so. If you are creating an RGB file you don't need a palette. Palette is only for indexed colours, that is, when the pixels are indexes.
Title: Re: Compressing a file/array by saving as PNG
Post by: spacefractal on 2012-Feb-08
I did not meant 16bit for each color, more when graphics set to rgb 565 format (64k colors). I have seen example in blitzmax, colors might been reduced under loading.

So it's must not load directly to hardware, but to memory? Hence no color loss.
Title: Re: Compressing a file/array by saving as PNG
Post by: Wampus on 2012-Feb-28
I modified the original code to describe what was happening on the screen as it compresses the example file. Should make more sense now. I also modified the way files are saved so that the extra data the routine needs to decompress a PNG to an array is included in the PNG file itself. If that doesn't make sense, er, just download the new attached zip file to the original post and use that version instead of the old.
Title: Re: Compressing a file/array by saving as PNG
Post by: spacefractal on 2012-Mar-24
I have still not tried this code, since I are not need to do. Today I have used samething to manipulate a png image to scramble it and reload it. Howover while rgb works fine, alpha got changed. So using alpha as data should not been used, but I can see its have been commented out as well.
Title: Re: Compressing a file/array by saving as PNG
Post by: Wampus on 2012-Mar-25
Ah, so I take it you are trying to use scrambling to protect your image data from content thieves? Nice idea.

I'm not sure why that technique wouldn't work. However, using the routine I posted I haven't come across any data corruption on any of my test devices. As it is, it works.
Title: Re: Compressing a file/array by saving as PNG
Post by: Qedo on 2021-Mar-13
I am interested in compressing a file using Png compression but unfortunately the source is no longer available on the forum.
It shouldn't be hard to rewrite it but I wouldn't want to reinvent the wheel.
Could someone please, Wampus ?, make it available on the forum?
Thank you
Ad maiora
Title: Re: Compressing a file/array by saving as PNG
Post by: Qedo on 2021-Mar-19
don't worry i solved it by myself. I finally have my personalized shoebox  which uses png compression.
  :good:
Title: Re: Compressing a file/array by saving as PNG
Post by: dreamerman on 2021-Mar-19
If You can don't forget to share it here ;) Every code, specially with some tricks can learn us something.
Can it store multiple files? I'm not sure if there is some Mem2Sound function so does it support different file types or only images, maybe with some encoding for protection.
Title: Re: Compressing a file/array by saving as PNG
Post by: Qedo on 2021-Mar-19
when it is finished I will definitely share it  :good:. Now I'm optimizing the speed because it's a bit slow at the moment. The customized shoebox is written in pure GLBasic and therefore, since the Mem2Sound command is not present, the sound files will not be managed normally. However, the logic of the program is this:
all files in the Media folder will be compressed with png logic. Once unpacked, only the bmp, jpg, png files can be managed thanks to Mem2Sprite in the program, while all the others must be saved unpacked in the same Media folder.
ad maiora