Buffer mystery!

[Guest_leke]

Regarding READ’s refinement:
/direct -- Opens the port without buffering.

I kind of know that a buffer is a place a script might store data, in either memory or a temporary file whilst being written. So when I got an ‘out of memory error’ in one of my scripts, I used this refinement to solve the problem.

So if I had buffering ‘turned off’, where was the data in my script being stored?
Whilst I am asking, would there be any way to count the bytes as they are being read?

Thanks.

[Gregg]

If you use /direct mode, the only place the data is buffered is in your variables (words you set, that refer to the data you've read), so you can buffer up as much data as you want, and dump it to recover memory once you're done with it.

[Anton]

It's not a simple example anymore, but it might just do what you want, too:

http://anton.wildit.net.au/rebol/util/batch-download.r
http://anton.wildit.net.au/rebol/util/demo-batch-download.r

This defines a BATCH-DOWNLOAD function which opens a window to download any number of urls in order, resuming when necessary. It opens the port to the url using direct mode and determines the total length with:

    total-size: to-integer port/locals/headers/content-length

Later it uses copy/part to specify a buffer size and if data was returned the data can be examined to find the length.