@ChShersh or write the original file in chunks and have a custom file pruning implementation(i.e. if queries are time related), i.e. in a custom spark datasource?
@ChShersh loading from storage in 2s requires 400+ Gbps=scanning no good. What about streaming the file while being updated, to build an index? Inverted idx, ngram idx, time based idx(i guess updates/queries are time related). then query the idx.