Philipp Brunenberg

Co-creator of @DayCaptainApp - developer productivity tool with vim-keybindings, clean UI, open API | Developer | Data Engineer

[Philipp Brunenberg @p_brunenberg]

Here's why you fail to accomplish what you want most.

Read this thread and learn something about yourself. 🚀

// A Thread //

#productivity #timemanagement #success

[Philipp Brunenberg @p_brunenberg]

Apache Spark with Scala Course - Video 10

🔨 By now, we have only talked about how to use Spark.
🕵️‍♀️ Today, we take the first glance at what it actually is and what we use it for ...

https://t.co/Vjz4EPVtaP

[Philipp Brunenberg @p_brunenberg]

Apache Spark with Scala Course - Video 9️⃣

Let's plug everything we learned together in our first assignment ...
Loading data, inferring schemas, accessing, renaming and transforming columns.

https://t.co/kj8uYAHIe5

[Philipp Brunenberg @p_brunenberg]

@DeltaLakeOSS is @databricks #LakeHouse implementation.

In today's video, we'll explore how to use Delta Lake from Apache Spark's SQL API.

[create table, append rows, add column, load versions]

https://t.co/YKWOBVw5SM

#ApacheSpark
#Scala

[Philipp Brunenberg @p_brunenberg]

Apache Spark with Scala Course - Video 8️⃣

🚀 There is also a SQL API in Spark ...
🧐 Should we use it or not?
💡 Which functions can we use in our SQL expressions?

https://t.co/UQXswM6hGO

[Philipp Brunenberg @p_brunenberg]

Apache Spark with Scala Course - Video 7️⃣

💡 There exist more functions to transform columns in Spark SQL.
🧚‍♂️ They offer a wide world of data transformations ...

We'll explore their implementation and documentation.

https://t.co/aWX02rKRzp

[Philipp Brunenberg @p_brunenberg]

Many companies are training their Data Engineers "in-production".

Why?

No time ...

Is that a good idea?

[Philipp Brunenberg @p_brunenberg]

Apache Spark with Scala Course - Video 6️⃣

❓ What can we do with Columns in the SparkSQL API?
💡 Have you ever looked at the Column Class in SparkSQL?
✅ We'll explore the source code and the documentation.

https://t.co/dfRO8IutHn

[Philipp Brunenberg @p_brunenberg]

Apache Spark with Scala Course - Video 5️⃣

How can we reference columns in the Spark DSL?
When working with the SparkSQL DSL, we mostly would like to do something with the columns of a Dataset.

https://t.co/1UR7LPJzj8

[Philipp Brunenberg @p_brunenberg]

High-quality software = ?

What do you think?

[Philipp Brunenberg @p_brunenberg]

Apache Spark with Scala Course - Video 4

💡 What's the Spark SQL API?
❓ How do Datasets and DataFrames relate?
🔎 What's the Spark DSL?

https://t.co/WV0f3JF6Nx

[Philipp Brunenberg @p_brunenberg]

Apache Spark with Scala Course - Video 3

✅ Inspect the schema of a DataFrame
📖 Explore Spark SQL data types in the official documentation
🔎 Inferring the schema while reading a csv file

https://t.co/TjtV53OYPL

#ApacheSpark #Scala #Course

[Philipp Brunenberg @p_brunenberg]

🚨 New video in Apache Spark with Scala Course - Video 10: What is Spark?

🔨 By now, we have only talked about how to use Spark.
🕵️‍♀️ Today, we take the first glance at what it actually is and what we use it for ...

https://t.co/Vjz4EPVtaP

[Philipp Brunenberg @p_brunenberg]

In part 2️⃣ of the #ApacheSpark with #Scala course, we will use our development setup to run our first Spark application. 💯

- Read csv file and explore documentation
- Run local standalone Spark (3.5.0 with Java 17)
- Explore @intellijidea run templates

https://t.co/mTk2E0Yx1Z

[Philipp Brunenberg @p_brunenberg]

I'm creating a free Apache Spark video course on YouTube. 🚀

✅ Now pre-knowledge required
💡 Step-by-step explanations
🏋️‍♀️ Hands-on exercises

Check out Video 1: Setting up Spark (IntelliJ, SBT, Scala, Spark)

https://t.co/WWANgNMLfy

#apachespark

[Philipp Brunenberg @p_brunenberg]

Consequently, Spark core’s optimization strategy is limited to pipeline narrow transformations into stages.
https://t.co/pMG1ZspYsg

[Philipp Brunenberg @p_brunenberg]

A sampling strategy enables us to find the structural contexts of two nodes in a graph.

We can use these contexts to find their similarity - for example using the Jaccard set similarity metric.

node2vec uses contexts to map nodes to vectors.
https://t.co/XNdkrB5vza

[Philipp Brunenberg @p_brunenberg]

What's the fundamental principle enabling Apache Spark to parallelize workload?

I'm creating a video series on the internals of Spark.

The first video is a deep dive into Spark's most profound principle: MapReduce.
https://t.co/X879w8XgKg

[Philipp Brunenberg @p_brunenberg]

How to build a recommendation engine in #neo4j.

⏫ Naive Bayes classification
🚀 Performs as good as a deep neural network
✅ Just a handful lines of code
https://t.co/KJSXSupQsE

[Philipp Brunenberg @p_brunenberg]

Naive Bayes classification is naive because it assumes independence of features.

The maths are very simple.

However, it works astonishingly well for our recommendation engine.
https://t.co/KJSXSupQsE

[Philipp Brunenberg @p_brunenberg]

To materialize a physical plan, #ApacheSpark schedules tasks on the workers.

In fact, the workers have one single functionality: to execute a task.

Let's explore how a physical plan is transformed into schedulable tasks.
https://t.co/uyTAkntQvG