Re: Help with Streaming and Chunk Processing for Large JSON Data (60 GB) from Kenna API

You can't validate an IP packet without having all of it. Your notion
of "streaming" is nonsensical.

Whoa, whoa, hold your horses! "nonsensical" needs a little bit of
justification :)

It seems you don't understand the difference between words and
languages! In my examples, IP _protocol_ is the language, sequences of
IP packets are the words in the language. A language is amenable to
streaming if the words of the language are repetition of sequences of
symbols of the alphabet of fixed length.  This is, essentially, like
saying that the words themselves are regular.

So, the follow-up question from you to me should be: how come strictly
context-free languages can still be parsed with streaming parsers? --
And the answer to that is it's possible to approximate context-free
languages with regular languages.  In fact, this is a very interesting
subject, which unfortunately is usually overlooked in automata
classes.  It's interesting in a sense that it's very accessible to the
students who already mastered the understanding of regular and
context-free formalisms.

So, streaming parsers (eg. SAX) are written for a regular language
that approximates XML.  This is because in practice we will almost
never encounter more than N nesting levels in an XML, more than N
characters in an element name etc. (for some large enough N).
Something which allows us to create a regular language from a
context-free one.

NB. "Nonsensical" has a very precise meaning, when it comes to
discussing the truth value of a proposition, which I think you also
somehow didn't know about.  You seem to use "nonsensical" as a synonym
to "wrong".  But, unbeknownst to you, you said something else.  You
actually implied that there's no way to tell if my notion of streaming
is correct or not.

But, for the future reference: my notion of streaming is correct, and
you would do better learning some materials about it before jumping to
conclusions.