Mark Klik

Lead Data Scientist at the Dutch Healthcare Authority. Working on Fraud detection on large data sets. Main author of R's #fstpackage.

[SpaceX @SpaceX]

Announcing the first commercial astronaut mission to orbit Earth aboard Dragon → https://t.co/MbESvnakAD

[Mark Klik @MarkKlik]

@giuliocurioni Yes, the current fst format is guaranteed to be supported indefinitely and can safely be used for long-time storage, no problem there! At the moment, fst is only available for R and Julia, but Python will be added in the near future.

[eval parse @evalparse]

{fst} still beats {arrow}'s read/write_parquet after {arrow}'s upgrade to v0.15.0. Heck, even writing CSVs with {data.table}'s fread/write is faster for datasets with lots of rows. H/T to the #rdatatable contributors! And ❤️ the new ETA messages! @ApacheArrow @ApacheParquet

[Mark Klik @MarkKlik]

@D_O_Arantes @MattDowle @arun_sriniv And if you would like to use the index to read the fst file as an ordered (keyed) data.table directly, the reverse indices would be required for fst to map the data to the correct in-memory elements (similar as with the a row selector, see also https://t.co/PhIwbIIVvk)

[Mark Klik @MarkKlik]

@D_O_Arantes @MattDowle @arun_sriniv Yes, interesting, you could store multiple indices separately for the same (unordered) data table and restore them as data.table attributes on read. The (integer) index attributes could also be stored in the fst file natively if that would be a useful feature.

[Mark Klik @MarkKlik]

@DanRosenheck @DanRosenheck, interesting question! Further parallelizing won't help you except with character columns. For those fst needs to use the single threaded R API to access R's global string cache and they are not using multiple threads yet (but they will in one of the next releases).

[Mark Klik @MarkKlik]

@yutannihilation the translator is playing games with your tweet :-), are you asking about compatibility between feather and fst?

[Hiroaki Yutani @yutannihilation]

fstはApache Arrowと互換性あるようになったりするのかな。Parquetについては対応されそうな感じがする。 https://t.co/ChnMiw32Vi

[Steven V. Miller @stevenvmiller]

Here's an R package of which I had known earlier. Have a data set you love that's more than 1 GB? Use the fst package in R. https://t.co/nJapWQygjJ

Here's a quick comparison of loading @ValuesStudies longitudinal data via haven::read_dta vs. fst::read_fst.

[Karl Broman @kwbroman]

Blog: sqlite, feather, & fst in #rstats. https://t.co/7GrVoHGFoD

[David Smith @revodavid]

The new fst package for R provides high-speed data import/export thanks to the magic of compression https://t.co/N9TXGxvlr9 #rstats

[Matt Dowle @MattDowle]

@dselivanov_ Sure - slides now uploaded. https://t.co/lJtAp8R7eK

[Karl Broman @kwbroman]

At @eddelbuettel’s suggestion, now trying https://t.co/KgbqyMFliF. 3x faster than feather, and file is 10x smaller https://t.co/E6Bt7stiJl

[Mark Klik @MarkKlik]

@bennetvoorhees If you're looking for random row and column access functionality, take a look at the fst package! https://t.co/AqSOAJ7P6S