Who should attend
Developers, data engineers, database professionals and analysts moving into data pipeline development.
Prerequisites
SQL fundamentals and basic database concepts. Basic scripting is helpful.
Learning outcomes
- Design and implement maintainable ETL/ELT pipelines.
- Extract data from files, databases and APIs.
- Transform, cleanse, validate and load data into target structures.
- Build incremental and restartable workflows.
- Implement logging, error handling, scheduling and data-quality controls.
- Document and deploy a working pipeline.
Detailed curriculum
8 modules01Data engineering and pipeline design
- Data engineering lifecycle and common architectures
- Batch versus streaming and ETL versus ELT
- Source systems, landing, staging, curated and serving layers
- Data formats: CSV, JSON, XML, Parquet and relational tables
- Pipeline requirements, dependencies and design documentation
02SQL and data modelling for pipelines
- Advanced SELECT, joins, aggregations, CTEs and window functions
- DDL, DML, constraints, views and stored procedures
- Normalisation and dimensional modelling foundations
- Fact/dimension design and surrogate-key strategies
- Indexes, partitions and performance-aware data access
03Extraction patterns
- Extracting from relational databases and flat files
- Reading JSON/XML and API-based data sources
- Full extraction, incremental extraction and CDC concepts
- Pagination, API limits and retry handling
- Source metadata, audit timestamps and extraction checkpoints
04Transformation and data quality
- Cleansing, standardisation, type conversion and enrichment
- Business rules, lookups, derived columns and aggregations
- Handling missing, duplicate and invalid data
- Schema drift and data contract awareness
- Reusable transformation logic and validation checkpoints
05Loading and warehouse patterns
- Load strategies: append, overwrite, merge/upsert
- Staging and target loading patterns
- Dimension and fact loading
- SCD Type 1 and Type 2 implementation concepts
- Transaction boundaries and idempotent pipeline design
06Orchestration and automation
- Pipeline dependencies and DAG concepts
- Scheduling, parameters and environment configuration
- Control tables, watermarks and restartability
- Retries, backoff, alerts and failure routing
- Secrets and connection configuration
07Operations, monitoring and optimisation
- Structured logs, metrics and run identifiers
- Reject handling, dead-letter/quarantine patterns
- Data reconciliation and operational dashboards
- Query tuning, partitioning and incremental processing
- Access control, encryption and data governance basics
08Deployment and end-to-end project
- Development, test and production environment separation
- Version control, code review and deployment packaging
- Automated validation and pipeline smoke tests
- Runbook, lineage and support documentation
- End-to-end pipeline build and demonstration
How you will practise
Learn through instructor-led sessions using realistic, live applications. Then apply each concept through guided assignments on a separate hands-on practice application, with feedback to help you build practical skills and confidence.
Practical exercises and labs
- Build a file-to-database ingestion pipeline.
- Extract paginated API data and persist incremental checkpoints.
- Implement cleansing, deduplication and business-rule transformations.
- Load a dimensional model and create an operational run log.
Capstone project
Build a parameterised ETL pipeline that ingests sample sales/customer data, validates and transforms it, loads a target warehouse model, supports incremental re-runs, and produces logs/reconciliation output.
Tools and technologies
SQL plus one confirmed ETL/orchestration platform (e.g., Python, SSIS, Azure Data Factory, Airflow or another selected tool). Platform-specific depth depends on the batch plan.
Assessment approach
- Module-wise knowledge checks or short practical tasks.
- Hands-on exercise review using an agreed checklist or rubric.
- Final project demonstration and explanation of design decisions, test evidence or outputs, as applicable.
- Completion criteria, attendance requirements and any certificate conditions must be separately approved and published by TrainingRite.
Schedule and fees
Upcoming batch schedules are shared by our TrainingRite team. Please submit an enquiry and our team will contact you by email or phone with the latest schedule. TrainingRite does not publish course fees or discounts on the website. Our counselling team will share the applicable course fee and other details with you by email or phone after you submit an enquiry.
Frequently asked questions
Who should join this course?
Developers, data engineers, database professionals and analysts moving into data pipeline development.
Do I need prior experience?
SQL fundamentals and basic database concepts. Basic scripting is helpful.
Is the training online?
TrainingRite programmes are delivered online.
Will I get hands-on practice?
Sessions are instructor-led and use realistic applications. You then apply each concept through guided assignments on a separate practice application.
How can I enquire about fees and the next batch?
Fees and batch schedules are not published on the website. Submit the enquiry form on this page and our counselling team will share them with you by email or phone.