Apache SystemDS

Open source system for end-to-end data science lifecycle over efficient, local and distributed ML.

Code:

[Matthias Boehm @matthiasboehm7]

The @SIGMODConf Programming Contest goes into another round. We (Bo Tang, Tilmann Rabl, and myself) just published the timeline and task overview:
https://t.co/0o5Ql8ed5i

Thanks to @carlo_curino and @Microsoft for the continued support.

[Arnab Phani @ArnabPhani]

Glad to share, our paper MEMPHIS is accepted at EDBT 2025! 🎉 MEMPHIS extends the LIMA framework and proposes a holistic approach for fine-grained reuse of intermediates and memory management across multiple backends.
Paper: https://t.co/KZGt2VJhBT

[Matthias Boehm @matthiasboehm7]

I'm grateful for being part of the great (and now accepted) collaborative research center FONDA II.

[Chip Huyen @chipro]

I went through the most popular AI repos on GitHub, categorized them, and studied their growth trajectories. Here are some of the learnings:

1. There are 845 generative AI repos with at least 500 stars on GitHub. They are built with contributions from over 20,000 developers, making almost a million commits.

2. I divided the AI stack into four layers: application, application development, model development, and infrastructure. The application and application development layers have seen the most growth in 2023. The infrastructure layer remains more or less the same. Some categories that have seen the most growth include AI interface, inference optimization, and prompt engineering.

3. The landscape exploded in late 2022 but seems to have calmed down since September 2023.

4. While big companies still dominate the landscape, there’s a rise in massively popular software hosted by individuals. Several have speculated that there will soon be billion-dollar one-person companies.

5. The Chinese’s open source ecosystem is rapidly growing. 6 out of 20 GitHub accounts with the most popular AI repos originate in China, with two from Tsinghua University and two from Shanghai AI Lab.

[Apache SystemDS @ApacheSystemDS]

@TheASF Major additions in this release include lineage-based reuse of Spark actions, parallel compressed feature transformation, bug fixes, and performance improvements.

https://t.co/wku2s9C06y

[The ASF - Apache Software Foundation @TheASF]

Apache SystemDS 3.1.0 is now available at:

https://t.co/eLj82EuR3T
https://t.co/QwLZB2ySLV

#SystemDS is an #opensource #machinelearning system for the end-to-end data science lifecycle.

#opensource #ML #datascience

[Apache SystemDS @ApacheSystemDS]

The SystemDS team is pleased to announce the release of version 3.0.0. This is the first release with Java 11 and Spark 3. Major additions include a feature-complete federated backend, Top-K cleaning, various new builtins, and performance improvements.

https://t.co/TI7kcdRLYF

[Matthias Boehm @matthiasboehm7]

Our preliminary experiments in @ApacheSystemDS with JDK 17's Float/DoubleVectors (for mapping to packed SIMD on x64 and aarch64) already look very promising:
https://t.co/qbYF6aRqAG

Thanks @KevinInnerebner for the initial exploration.

[Matthias Boehm @matthiasboehm7]

proud of our team (@ArnabPhani, Sebastian Baunsgaard) that we made reproducibility submissions for all of our SIGMOD'21 papers (I did it last minute, thanks to the CIDR camera-ready deadline extension).

https://t.co/1WGPx1kE9f

[Apache SystemDS @ApacheSystemDS]

The SystemDS team is pleased to announce the release of Apache SystemDS version 2.2.0. Major additions include full support for compression, multi-threaded feature transformations, improved Python API, and performance improvements.

Release Notes:
https://t.co/V9SOESuAAL

[Apache SystemDS @ApacheSystemDS]

The SystemDS team is pleased to announce the release of Apache SystemDS version 2.1.0. Major additions include a robust Federated backend w/ a Parameter Server for federated fraining, many new builtins, and performance improvements.

Release Notes: https://t.co/CdBRQHoLl9

[Matthias Boehm @matthiasboehm7]

4/4: All results are integrated and usable in @ApacheSystemDS. Btw, @ArnabPhani just today cut the next release candidate rc3 of Apache SystemDS 2.1, which after voting has passed will become available soon.

[Matthias Boehm @matthiasboehm7]

we're truly grateful for the recognition of our SliceLine work in the new data science & engineering track.
https://t.co/BAoQOxpRb0

Svetlana did awesome work and it's a nice example how a course project in OSS leads to a master thesis and back into @ApacheSystemDS.

[Matthias Boehm @matthiasboehm7]

Happy to announce Apache SystemDS 2.0.0, which we released mid October as the first release after merging SystemDS back into Apache SystemML:
https://t.co/nuejOoxvam