NETWEBGET$ not refreshing?

Previous topic - Next topic

SBlectric

When I try to use NETWEBGET$, I can get the contents of the webpage just fine. However, if I make the same call to a webpage that has been updated, it still returns the old information. It's almost like it's working off of a cache or something...
Cool stuff and whatever

Moru

Do you have a simple code example with homepage address that we can try?

SBlectric

I'm making a little program that will get your reddit karma for my friend, and it's pretty much this:
Code (glbasic) Select

// get your reddit info
dat$=NETWEBGET$("www.reddit.com","/user/"+user$+"/gilded/index.html",80,13000,1000)

where user$ is your reddit username. On reddit, when your karma clearly has changed, NETWEBGET$ returns the same value as before.
Also, it seems to take less time to access the information, as if it's accessing a local cache instead of the webpage.
Cool stuff and whatever

spicypixel

Reddit may cache the info and realise that the access request is from the same IP within too short a time period maybe?
http://www.spicypixel.net | http://www.facebook.com/SpicyPixel.NET

Comps Owned - ZX.81, ZX.48K, ZX.128K+2, Vic20, C64, Atari-ST, A500.600.1200, PC, Apple Mini-Mac.

SBlectric

I don't know if this matters, but changing the username or even the page of the user will generate new results... but then going back to the old URL returns the old ones again.  :noggin:
Cool stuff and whatever

Moru

I just picked a user (moru, not me :-) I get an almost empty page but it's still 37 kB. You have limited the download to 12.7 kB so mabe the changed data is at the end of the page and you cut it off?

SBlectric

As far as I know, the structure of the page stays the same. The karma is stored in between the 12kb mark and the 13kb mark.
Cool stuff and whatever

Moru

Maybe something happens when you don't download the whole file at once. Try downloading the whole file at once and see if that helps. In that case you found a bug that can be reproduced :-)

If that does not help, try this:

Code (glbasic) Select
text$ = NETWEBGET$("www.bjornrosenthal.eu","/assets/test/get_ip.php",80,10000,5000)
DEBUG "\n"+text$
text$ = NETWEBGET$("www.bjornrosenthal.eu","/assets/test/get_ip.php",80,10000,5000)
DEBUG "\n"+text$
text$ = NETWEBGET$("www.bjornrosenthal.eu","/assets/test/get_ip.php",80,10000,5000)
DEBUG "\n"+text$
text$ = NETWEBGET$("www.bjornrosenthal.eu","/assets/test/get_ip.php",80,10000,5000)
DEBUG "\n"+text$
text$ = NETWEBGET$("www.bjornrosenthal.eu","/assets/test/get_ip.php",80,10000,5000)
DEBUG "\n"+text$

Each call to this page is different, does it update for you? It works for me with version 12.001

spicypixel

The changes might be around the 12Kb mark, but if the page is updated with Javascript or using jQuery then we know that many times the full page has to be loaded in the browser before the JS is executed. Also if it is JS then the changes will only be local to the browser, and although the server-side of things will be updated it might be polled every few minutes to reduce server-strain and in that time the changes you see are only browser-based. This you might not see reflected with NETWEBGET$. There are many variables to consider other than GLB. The best way to test properly is to write your own PHP and MySQL, or say have the output from NETWEBGET$ displayed on-screen in a loop (with delay of SLEEP 500 say) and then update a html file yourself if you PHP/MySQL isn't too hot and then check the output window to see if the changes are reflected immediately when you update your own HTML file.
http://www.spicypixel.net | http://www.facebook.com/SpicyPixel.NET

Comps Owned - ZX.81, ZX.48K, ZX.128K+2, Vic20, C64, Atari-ST, A500.600.1200, PC, Apple Mini-Mac.

Moru

Quote from: spicypixel on 2013-Nov-26
...or say have the output from NETWEBGET$ displayed on-screen in a loop (with delay of SLEEP 500 say) and then update a html file yourself if you PHP/MySQL isn't too hot and then check the output window to see if the changes are reflected immediately when you update your own HTML file.
The code I posted does this all automatically (updates the data before sending to the client)
The data returned contains IP, Date and time, and a unique ID based on micro-time so every request will be unique. If you run the code with debug on, you will get all the responses in the debug window and can see directly that they all differ slightly. This means GLBasic is downloading data fine at least when it comes to small documents.

I don't have a reddit account so can't really say if anything updates with ajax or not. So far I don't see any ajax requests after loading the page but who knows how long that takes to refresh...

SBlectric

I see. Both NETWEBGET$ and NETWEBGET have the same issue for me.
Would there be any better way to perhaps access the karma and other info in a more direct way then NETWEBGET$ or NETWEBGET? Because I don't really like the fact that I'm either shoving 13KB into a string variable or downloading it onto the drive to then have to access later anyway.
Cool stuff and whatever

Moru

A quick google turns up that Reddit has some sort of API, have a look here for the documentation: http://www.reddit.com/dev/api

SBlectric

Thanks, Moru. I looked at the API info, and found that you can get a short piece of info about a user. Now it is working perfectly. It seems that Reddit has a 30-second cache for this info, whereas I don't know what was happening before.
Perhaps Reddit caches larger pages for longer or something.
Now, it's perfect. The file user/username/about.json file is less than 512bytes, and stores all the user info! Woo! :good:
Code (glbasic) Select
dat$=NETWEBGET$("www.reddit.com","/user/"+user$+"/about.json",80,512,1000)
Now I can continue! Thanks everyone.
Cool stuff and whatever