独家优惠奖金 100% 高达 1 BTC + 180 免费旋转

Kurikulum Big Data Engineering

1. Basic Programming

a. SQL and Relational Algebra

b. Object Oriented using Java

c. Object Functional using Scala

d. Project Structure using Maven and SBT

2. Pengenalan Distributed System

a. Single Node Application

b. Application Architecture

c. Tiered Architecture

d. Microservice Architecture

3. Data Flow from Application to Data Warehouse or Reporting

a. RDBMS for transactional data in Application

b. Data Warehouse for Reporting Database and Query

c. OLTP vs OLAP

d. Query and Reporting problem with current approach

e. Apps development vs Data development

4. Entering Big Data in Theory

a. Roles in Big Data

i. Data Administrator

ii. Data Analyst

iii. Data Scientist

iv. Data Engineer

b. Difference between Data Engineer and Data Scientist

c. Data Engineering deep dive

5. Scaling

a. Horizontal

b. Vertical

6. Why we need ETL

a. Vendor Lock In

b. Difficult to modify existing applications

c. Straight to DB

d. Daily Batch

e. Not Realtime

7. Realtime Analytics in ETL / Daily Batch architecture

a. CDC from Database inside out

8. Lambda Architecture

a. Batch Layer

b. Speed Layer

c. Service Layer

9. Realtime Streaming pipeline

a. Microservice Architecture comes to rescue

b. Streaming changes from Services to Kafka

c. CDC at application level

d. Benefit : Spesific business event. Explicit

10. Big Data with Hadoop and Cloudera

a. Introduction

b. History of Big Data

11. Hadoop Ecosystem

a. Namenode

b. Datanode

c. YARN

d. Replication

e. Hadoop = Storage ( HDFS ) + Processing ( YARN MapReduce => Replaced by Spark )

f. Spark running on top of YARN Cluster

12. File System

a. Format

i. Avro

ii. Parquet

iii. Text File

b. Compression

i. Snappy

ii. Gzip

13. Cloudera VM QuickStart

14. Batch vs Streaming Pipeline

15. Batch Pipeline

a. Data Ingestion, Transform, Visualize

b. HDFS, Sqoop, Hive, Impala and Spark

c. Project

16. Streaming pipeline

a. Kafka, HBase, Spark Streaming

b. Project

17. Talend Big Data Realtime for Data Ingestion

18. Visualization with Tableau

19. Cloudera Installation

20. Confluent Kafka

Cheers

20. Confluent Kafka

Add a comment

What is your view?

Send

Related posts:

MobileRefunds at World Telemedia 2018

Our COO and CEO have returned to the UK, following a highly successful trip to World Telemedia 2018 — a conference for businesses involved in carrier billing and other alternative payment platforms…

15 Business Ideas for Kids

You might think that kids can’t start their own businesses, but with the proper knowledge and resources, children can develop innovative business ideas. Kids are often creative thinkers who can apply…