Skip to content
Home
Programming
Data Engineering

Data Engineering

Data engineering guides covering data pipelines, ETL/ELT, data warehousing, streaming, Spark, Airflow, and building reliable data infrastructure.

Articles in this section

Data Engineering: Building Reliable Data Pipelines at Scale Learn data engineering foundations — ETL vs ELT, batch and stream processing, data warehousing, orchestration, quality … 8 min ETL Pipelines: Building Reliable Data Workflows at Scale Master ETL pipeline design — extraction strategies, transformation logic, loading patterns, error handling, testing, … 8 min Streaming Data with Apache Kafka: Real-Time Pipelines Guide Master Apache Kafka for real-time streaming — topics, partitions, producers, consumers, consumer groups, schema … 8 min Data Warehousing: Architecture, Design, and Cloud Options Master data warehousing — star schema, Kimball vs Inmon, ETL vs ELT, dimensional modeling, columnar storage, Snowflake, … 8 min Data Orchestration with Apache Airflow: Complete Guide Master data orchestration with Apache Airflow — DAGs, operators, sensors, scheduling patterns, production best … 8 min Analytical SQL: Window Functions, CTEs, and Query Optimization Master analytical SQL — window functions, CTEs, recursive queries, GROUP BY extensions, query optimization, indexing … 8 min Data Quality Testing: Ensuring Reliable Data Pipelines Master data quality — validation rules, anomaly detection, data contracts, dbt tests, Great Expectations, dimensions of … 8 min Modern Data Stack and Lakehouse Architecture Explained Explore the modern data stack — lakehouse architecture, Apache Iceberg, Delta Lake, data mesh, reverse ETL, data … 8 min Apache Spark Guide: Distributed Data Processing at Scale Master Apache Spark for large-scale data processing — RDDs, DataFrames, Catalyst optimizer, Spark SQL, tuning, … 10 min dbt Guide: Analytics Engineering for Modern Data Teams Master dbt for analytics engineering — models, Jinja macros, incremental loads, testing, documentation, … 8 min Data Modeling Guide: Dimensional Design for Analytics Master dimensional modeling — fact tables, dimension tables, slowly changing dimensions, star schema, snowflake schema, … 9 min Data Lakes: Architecture, Governance, and Best Practices Master data lake architecture — medallion layers, partitioning, file formats, cataloging, open table formats, lakehouse … 8 min Data Pipeline Monitoring and Observability Learn data pipeline monitoring — observability pillars, metrics, monitoring tools, alerting, lineage tracking, and … 7 min Data Security and Governance Guide for Data Engineers Master data security — encryption at rest and in transit, RBAC, column-level security, data masking, audit logging, … 9 min Data Ingestion Guide Learn data ingestion — batch vs streaming ingestion, API ingest, database connectors, change data capture, file … 7 min Data Transformation Guide Learn data transformation — SQL vs Python transformations, dbt models, data cleansing, feature engineering, and … 8 min Data Lakehouse Architecture Guide Learn data lakehouse architecture — combining data lake and warehouse, Delta Lake, Apache Iceberg, ACID transactions, … 8 min Data Catalog and Discovery Guide Learn data catalog and discovery — metadata management, data lineage, data discovery tools, cataloging best practices, … 8 min