Udemy – Data Engineering using Kafka and Spark Structured Streaming 2022-10

Published on: 2024-09-25 01:06:40

Categories: 28

Description

Data Engineering course using Kafka and Spark Structured Streaming. In this course, you will learn how to build a streaming pipeline by integrating Kafka and Spark Structured Streaming. Let’s go through the details of what is covered in the course. First, we need to have a suitable environment for building a streaming pipeline using Kafka and Spark Structured Streaming on Hadoop or any other distributed file system. As part of the course, you will start by setting up a self-supporting lab with all key components such as Hadoop, Hive, Spark and Kafka on a single-node Linux-based system. After setting up the environment, you will review the details of getting started with Kafka. As part of this process, you will create a Kafka topic, generate messages into the topic, and also consume messages from the topic.

What you will learn in the Data Engineering using Kafka and Spark Structured Streaming course

Setting up a self-supporting lab with Hadoop (HDFS and YARN), Hive, Spark and Kafka An overview of Kafka for building a streaming pipeline to receive data to Kafka topics using Kafka Connect using a file source Receiving data to HDFS using Kafka Connect with Using the HDFS 3 Connector Plugin Overview of Spark Structured Streaming to process data as part of a streaming pipeline Incremental data processing using Spark Structured Streaming using file source and file target Integrating Kafka and Spark Structured Streaming – Reading data from Kafka topics

This course is suitable for people who

Experienced ETL developers who want to learn Kafka and Spark to build streaming pipelines Experienced PL/SQL developers who want to learn Kafka and Spark to build streaming pipelines Beginner or experienced data engineers who want Kafka and Spark to build a streaming pipeline

Data Engineering using Kafka and Spark Structured Streaming course specifications

Publisher: Udemy
Teacher: Durga Viswanatha Raju Gadiraju
Training level: beginner to advanced
Training duration: 9 hours and 26 minutes

Course topics on 12/2023

Course prerequisites

Laptop with decent configuration
Decent internet speed to watch the lessons
Self Support lab (instructions will be provided as part of the course) or ITVersity labs
Knowledge about Functional Programming (preferably Python or Scala)
Knowledge or experience using Spark