GLBasic forum

Main forum => GLBasic - en => Topic started by: Miroslav-Stimac on 2012-Oct-03

Title: Problem with Sprite Antialiasing
Post by: Miroslav-Stimac on 2012-Oct-03
Hello,

I have a problem with the antialiasing of some sprites.
To describe the problem, I added two images (the bottom one is just zoomed to show the differences).

I have a small sprite that is saved with Paint .NET (also with Photoshop, it does not matter) as a PNG file with 32 Bit and alpha channel.
It is the sprite on the left side of the images and it looks nice, the antialiasing is good. I simply pasted it in a new Paint .NET layer above the blue background that is the same one as in my GLBasic game.

The right sprite is the sprite that is drawn in GLBasic with Alphamode -0.999 (I cut it out of a screenshot of the game). In the game I use the same PNG file as the one I opened in Paint .NET.
As you can see the sprite drawn in GLBasic has no good antialiasing... the antialiasing is simply too "weak", the outline is not soft. The Paint .NET sprite has more transparency.

I tried all alphamode values from -1 to +1 by increasing step 0.001 and I was not able to find any value that makes the sprite look as good as in Paint .NET.
I also tried to change alphatesting (-1.000, -0.999, -0.998 ... 0.998, 0.999, +1.000), but it did not solve the problem, too.

How can I tell GLBasic to draw the Sprite in the same way with the same alpha mode settings as Paint .NET and Photoshop do?

Thanks for any hints!

Best wishes,
Miroslav

(http://www.haiku-survey.com/ProblemAntialiasing.jpg)
(http://www.haiku-survey.com/ProblemAntialiasingZoom.jpg)
Title: Problem with Sprite Antialiasing
Post by: Kitty Hello on 2012-Oct-03
Try alphatesting false.
Title: Re: Problem with Sprite Antialiasing
Post by: Miroslav-Stimac on 2012-Oct-03
Hello,

Thanks for the fast answer.
I tried it with Alphatesting False and Alphamode -1.
Now It looks a little bit better, but it is still not good. When I zoom it, I see that there are a few more transparent pixels when using Alphatesting False than there were before, but still there are not as many as in Paint .NET.
What else could I try?

Best wishes,
Miroslav
Title: Re: Problem with Sprite Antialiasing
Post by: erico on 2012-Oct-03
I wonder if our gfx board could also have anything to do with this.
I have noticed different GLB alpha behaviors according to different settings.
Title: Re: Problem with Sprite Antialiasing
Post by: Wampus on 2012-Oct-04
This has also been discussed here (http://www.glbasic.com/forum/index.php?topic=8038.0) as well as a couple other places.

Something to keep in mind is that whatever colour the transparent pixels are in RGB makes a huge difference to how they blend. Even if they're barely visible at all they have a strong impact on pixels next to them when zooming and rotating. So, a pixel with total transparency and an RGB value of 255,255,255 next to a solid pixel with an RGB value of 0, 0, 0 will create unexpected white blending when zooming or rotating. I've attached a little proggy I wrote to investigate this that demonstrates the principle.

Here is a non-GLBasic example of a related issue occurring in OpenGL (http://stackoverflow.com/questions/11711362/opengl-rendering-transparent-png-with-random-white-pixels). No easy solution I think. You just have to work with it as best you can, unless you can work out the OpenGL calls that might clear it up.

EDIT: Here is a much better discussion of this problem I found (http://answers.unity3d.com/questions/10302/messy-alpha-problem-white-around-edges.html). Miroslav-Stimac, I don't know if it makes sense why the white border problem mentioned in the discussion I linked to relates to what you're seeing with dark edges for your sprite. It is related though. They stem from the same issue when transparency is used. No easy solution.

[attachment deleted by admin]
Title: Re: Problem with Sprite Antialiasing
Post by: Wampus on 2012-Oct-04
Hmm. Having re-examined the first post something does puzzle me. If no zooming or rotation is being done, so the sprite mapping is 1 to 1, then transparency should be the same as you see in Pixel.net and Photoshop. I'm confused now.
Title: Re: Problem with Sprite Antialiasing
Post by: MrTAToad on 2012-Oct-04
What about using SMOOTHSHADING FALSE ?
Title: Re: Problem with Sprite Antialiasing
Post by: Miroslav-Stimac on 2012-Oct-04
Now I tried SMOOTHSHADING FALSE
Sadly it looks even worse than before.

Yeah, I know that there may be problems when zooming or rotating a sprite because OpenGL has it's own methods and algorithms that do this, but I neither rotate nor zoom the sprite. I just draw it on the screen by using the GLBasic method DRAWSPRITE.

Let me exactly describe what I do:
I create respectively import the sprite in Paint .NET (or Photoshop, the results look the same), test it on the background image and it looks nice. Then I save the sprite as a .PNG file with 32 Bit, close it, open it again and test it again in Paint .NET and it still looks nice.
After that, I load it in GLBasic with LOADSPRITE, use ALPHAMODE -1 respectively ALPHAMODE -0.999 and draw it with DRAWSPRITE on the background image. I also tried the ALPHATESTING settings and the SMOOTHSHADING FALSE setting, as described in the previous posts. Whatever I do, it does not look as nice as in Paint .NET.
I just want to draw the PNG sprite exactly the same way as I see it in Paint .NET respectively Photoshop. All the alpha blending information is saved in the alpha channel of the .PNG file, or?
Why is it not drawn 1 to 1 as it works in Paint .NET?
I am quite confused...

Best wishes,
Miroslav
Title: Re: Problem with Sprite Antialiasing
Post by: MrTAToad on 2012-Oct-04
Because OpenGL uses its own anti-aliasing system (I suspect Paint .Net uses DirectX).  It could also be depending on what your graphics card settings are too.

Title: Re: Problem with Sprite Antialiasing
Post by: Miroslav-Stimac on 2012-Oct-04
Thanks for your fast response. I also had a similar idea, but I think that Paint .NET uses only GDI (Win32 API) respectively GDI+ of the .NET Framework. I looked at the change log of Paint .NET. They only mention that the Text Tool (a tool in Paint .NET) may use DirectDraw if it is available, otherwise it uses GDI. They do not mention DirectX regarding any other tools and I do not use the Text Tool to create the sprite .PNG file. I have the same problem with Photoshop. The sprite looks the same (good) in both: Paint .NET and Photoshop.

May it be really a disadvantage of OpenGL?

Best wishes,
Miroslav
Title: Re: Problem with Sprite Antialiasing
Post by: erico on 2012-Oct-04
Agreed with Mr.T.

Maybe with smooth shading false, you may be able to keep the exact alpha without open gl adding to this alpha.
I do use the photoshop->glbasic pipeline with png 32bits using a single layer, the normal alpha channel does not export on my tests or cames pre-multiplied with some matte, which also changes the alpha behaviour.

Another thing is that I keep my gfx board on maximum everything overide all programs, and this was leading me to alpha problems on some glbbasic game.
By reducing a few things and setting to app controlled, I got things going ok.

Your example looks like part of the alpha is gone, probably by the alpha mode settings+gfx board settings.
Are you running your gfx board standard settings?

Care to post that png ball you used as example? Should make things easy to find out.
Title: Re: Problem with Sprite Antialiasing
Post by: erico on 2012-Oct-04
Quote from: Miroslav-Stimac on 2012-Oct-04
...May it be really a disadvantage of OpenGL?

Not at all, there has been lot´s of cases similar to this, can´t remember one that wasen´t solved.
Mine took a long while to figure out, and also noticed messing the gfx board usually alter things in all games, glb or not, on my machine.

Part of the solution involved smooth shade false, otherwise antialiased gfx that would perfectly fit on its own original background may come with alpha pre-multiplied with the colors around and won´t fit any other bkg without extra garbage.
Title: Re: Problem with Sprite Antialiasing
Post by: Miroslav-Stimac on 2012-Oct-04
Hello,

The hint regarding the graphic card settings was great!
Now I changed the graphic card setting to maximum antialiasing quality and it really looks better (not as good as in Paint .NET, but still better than before)!
But this is no solution for my problem. How shall I make sure that the antialiasing looks so good on all systems including mobile devices such as Android smart phones?
How can I set the quality setting of the graphic hardware to maximum in GLBasic for all platforms?

The more I experience OpenGL with PC and Android, the more I understand why many developers prefer standard console hardware such as Playstation 3, XBox 360 and Nintendo Wii. There they know what hardware and settings are default and can optimize the games for the platforms.

It is very difficult to make sure that all players on various systems experience the same graphics quality of a game, especially in case of Android where there are so many different devices.

Here is the original PNG:
(http://haiku-survey.com/target.png)

Is there any way to make it suitable and nice for all platforms?

Best wishes,
Miroslav
Title: Re: Problem with Sprite Antialiasing
Post by: MrTAToad on 2012-Oct-04
You can't - each system has its own limitations - some are 16 bit, some 24-bit and some are 32-bit.

Each has its own way of handling graphics (especially PNG's).  Actually it might be worth just using BMP files and use SETTRANSPARENCY to set the masking colour.
Title: Re: Problem with Sprite Antialiasing
Post by: Wampus on 2012-Oct-04
btw using the original PNG you supplied I get identical results, or visually identical results, to what I'm seeing in my browser when I compile for my set up in Win (AMD video card) or Android (Nexus 7).

Quote from: Miroslav-Stimac on 2012-Oct-04
The more I experience OpenGL with PC and Android, the more I understand why many developers prefer standard console hardware such as Playstation 3, XBox 360 and Nintendo Wii. There they know what hardware and settings are default and can optimize the games for the platforms.

This is most annoying for mobile development, especially with Android because there are so many hardware and software differences between devices. I was surprised to see Plants vs Zombies had some weird graphical glitches on the Nexus 7 compared to the Galaxy S2. Bad texture clamping and padding problems aren't what I expect from a game like that.
Title: Re: Problem with Sprite Antialiasing
Post by: Miroslav-Stimac on 2012-Oct-04
Strange, a friend of mine and I see differences between the .PNG file in our Opera browser, where it looks as nice as it looks in Paint .NET, and in the game running with GLBasic, where the antialiasing is not good.
But we have NVidia graphic cards, not AMD Radeon. Maybe that is the difference.
As I mentioned, when I change the graphic card driver settings to maximum quality, then it looks better. But the default settings are causing bad quality antialiasing in GLBasic.
Maybe your AMD Catalyst drivers have by default better antialiasing settings.
Is there any way by using GLBasic to force using the best graphic device settings respectively to use high quality software antialiasing as it is the case in Photoshop and Paint .NET?
Software antialiasing may be slow, but we have only 2D animations. Such games ran fast even on 15 years old computers.

Best wishes,
Miroslav
Title: Re: Problem with Sprite Antialiasing
Post by: spacefractal on 2012-Oct-04
I guess its could mightbeen the same issue I have in the early development of the game, seen in the attachment (the box behind the toy mouse was a alphabox test). Its could been a multiply issues, and they can been quite annoying. We use insane much antialasing and alpha in the game.

Here I use SETTRANSPARENCY -1, SetAlphaMode(1) and ALPHATESTING 1.0 in Greedy Mouse worked for me.

Code (glbasic) Select

FUNCTION SetAlphaMode: alpha#=-5
STATIC Lasta#
IF alpha=-5
ALPHAMODE Lasta#
RETURN
ENDIF
Lasta#=alpha#
IF alpha#>0
alpha#=0-alpha#
ELSE
alpha#=-alpha#
ENDIF
IF alpha#=0 THEN alpha#=0.1
ALPHAMODE alpha#
ENDFUNCTION


Yes I did not like that way ALPHAMODE worked and for me 0 should been off and 1 to on, not the other way, which cause some confuction for me. Its not a bug throught of course.

[attachment deleted by admin]
Title: Re: Problem with Sprite Antialiasing
Post by: erico on 2012-Oct-04
In my system I have to get the gfx settings to operate lower and allow program control, otherwise I get garbage.
But I mess with this settings a lot, can´t remember performance on default.

I think one way to keep it default on pcs/mac/linux is using the alpha mode false but if some system is set to override the application choice, then there isen´t much you can do.
Title: Re: Problem with Sprite Antialiasing
Post by: Wampus on 2012-Oct-04
Quote from: Miroslav-Stimac on 2012-Oct-04
But we have NVidia graphic cards, not AMD Radeon. Maybe that is the difference.
As I mentioned, when I change the graphic card driver settings to maximum quality, then it looks better. But the default settings are causing bad quality antialiasing in GLBasic.
Maybe your AMD Catalyst drivers have by default better antialiasing settings.

Catalyst is set to 'performance' for antialiasing, i.e. lowest quality, so it must a quirk of the nVidia OpenGL drivers. I've heard before that the nVidia 400/500 series have problems with OpenGL in some instances and aren't going to be kind on smaller multi platform games since they use OpenGL on the whole. Oddly, my nVidia 320m in my Mac Mini is very good with GLBasic games. Meh. What can you do?

If there is a way to force software rendering instead of hardware acceleration it would be a disaster for anything but Windows, Linux and MacOS. Yuck! If its driving you crazy one way to do that for development purposes only would be to compile on a virtual machine in VMware with hardware acceleration off.
Title: Re: Problem with Sprite Antialiasing (confirmed)
Post by: spacefractal on 2012-Oct-29
today I thinks this is a real glbasic bug. antialiasing and offbuffers simply dont work and you see that issue. I did never workaround that, and that why I avoid using offbuffers.

I used test code like this:
Code (glbasic) Select

IF OFFBUFFER=0
USESCREEN -1
SETTRANSPARENCY -1
ALPHATESTING 1.0
ELSE
USESCREEN -1
CLEARSCREEN RGB(0, 0, 0)
SETTRANSPARENCY -1
ALPHATESTING 1.0

USESCREEN 1
CLEARSCREEN RGB(0, 0, 0)
SETTRANSPARENCY -1
ALPHATESTING 1.0
ENDIF


this is the output:
(https://dl.dropbox.com/u/3236515/GreedyMouse/offbuffer_issue.jpg)

the upper is with USESCREEN 1 (and then paint the buffer directly to USESCREEN -1 with DRAWSPRITE), and second image is USESCREEN -1. The upper is wrong and the below is correct. they should show the same. Hence its a confirmed bug.

I guess GlBasic use wrong blendmode and I did never found what its happens and gave up offbuffers. So this is just a test for this bug.

So I thinks its should been moved to the bug forum? Its was not needed for me to create a new thread, since I think I experimentere extractly same issues.
Title: Re: Problem with Sprite Antialiasing
Post by: mentalthink on 2012-Oct-30
If I can help in something--- I always the .png I saved whit 24 Bits of depth and the black Background, and never I have extrange things, using alpha or not...

Title: Re: Problem with Sprite Antialiasing
Post by: Wampus on 2012-Oct-31
If you use CREATESCREEN then draw to it without using the CLEARSCREEN command (or a SHOWSCREEN command before drawing) then alpha transparency should be OK, i.e. set to 0. My Chaos remake relies on it to create new sprite sheets when in debug mode. The output is the equivalent of using TexturePacker or sprites straight out of Photoshop. However, using CLEARSCREEN will set the alpha to 255. Does that make sense?

EDIT: The thread How to clean a virtual screen (http://www.glbasic.com/forum/index.php?topic=8584.0) confirms this as MSX is finding the same thing. It isn't just my graphics card.