Sujet : Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API
De : list1 (at) *nospam* tompassin.net (Thomas Passin)
Groupes : comp.lang.pythonDate : 30. Sep 2024, 20:05:36
Autres entêtes
Message-ID : <mailman.14.1727728109.3018.python-list@python.org>
References : 1 2 3
User-Agent : Mozilla Thunderbird
On 9/30/2024 11:30 AM, Barry via Python-list wrote:
On 30 Sep 2024, at 06:52, Abdur-Rahmaan Janhangeer via Python-list <python-list@python.org> wrote:
>
>
import polars as pl
pl.read_json("file.json")
>
>
This is not going to work unless the computer has a lot more the 60GiB of RAM.
As later suggested a streaming parser is required.
There is also the json-stream library, on PyPi at
https://pypi.org/project/json-stream/
Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API