Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

On 9/30/2024 11:30 AM, Barry via Python-list wrote:

 

On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list <python-list@python.org> wrote:
>
>
import polars as pl
pl.read_json("file.json")
>
>

 This is not going to work unless the computer has a lot more the 60GiB of RAM.
 As later suggested a streaming parser is required.

Streaming won't work because the file is gzipped.  You have to receive the whole thing before you can unzip it. Once unzipped it will be even larger, and all in memory.