AWS Data Engineer

Job Description

Must Have Technical/Functional Skills

Technical Skills

Strong expertise in AWS services including:

o Lambda

o S3

o EventBridge

o Kinesis (Data Streams & Firehose)

o Glue (ETL + Crawlers)

o Step Functions

o Amazon Connect

o Athena

o Macie

Proficient in:

o Python

o PySpark

• Experience in:

o Glue Crawlers for schema discovery and cataloging

o Parquet-based data storage

o Building scalable data pipelines

• Strong understanding of event-driven architectures (Pub/Sub model)

• Hands-on experience with Terraform (Infrastructure as Code)

• Familiarity with CI/CD tools and automation pipelines

o Messaging platforms like Kafka and Amazon EventBridge

o Designing real-time data processing systems

o Using Glue Crawlers + Athena for data lake architectures

Roles & Responsibilities

• Develop a comprehensive plan for migrating near real-time fraud detection campaigns from on-premises systems to AWS.

• Design and implement event-driven architectures to process inbound dialer data (fraud events) using services such as Amazon EventBridge, Kafka, Kinesis Data Streams, and Kinesis Firehose.

• Build and manage scalable data pipelines using AWS Glue (ETL jobs, Crawlers), PySpark, and Python for data ingestion, transformation, and processing.

• Configure and manage Glue Crawlers to automatically discover schemas and update the Data Catalog.

• Store and optimize data using Parquet format and enable analytics through Amazon Athena for efficient querying.

• Develop integrations between Customer Profiles and messaging platforms to automatically trigger profile updates and downstream processes.

• Implement automation to trigger fraud-related outbound calls based on updates in customer profiles.

• Design and orchestrate workflows using AWS Step Functions to manage complex processing pipelines.

• Provision and manage cloud infrastructure using Terraform (Infrastructure as Code).

• Optimize system architecture for scalability, reliability, cost-efficiency, and ensure data integrity and security.

• Conduct end-to-end testing o f the entire framework to validate functionality, performance, and reliability.

• Deploy, automate, and manage resources using CI/CD pipelines.

• Continuously monitor system performance and implement optimizations post-deployment.

• Maintain detailed documentation of architecture, workflows, and operational processes.