Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On 2/10/24 12:26 pm, avi.e.gross@gmail.com wrote:

The real problem is how the JSON is set up. If you take umpteen data
structures and wrap them all in something like a list, then it may be a tad
hard to stream as you may not necessarily be examining the contents till the
list finishes gigabytes later.

Yes, if you want to process the items as they come in, you might
be better off sending a series of separate JSON strings, rather than
one JSON string containing a list.
Or, use a specialised JSON parser that processes each item of the
list as soon as it's finished parsing it, instead of collecting the
whole list first.
--
Greg