Lead Data Engineer
Job Description
Job Description
We are looking for a highly skilled Lead Data Engineer to join our team. This role requires an individual with extensive experience in Databricks and a solid understanding of the insurance industry. You will play a pivotal role in designing and implementing scalable data architectures to support advanced analytics, reporting, and AI/BI/ML initiatives across various insurance functions.
Responsibilities:
• Develop and implement enterprise-level data architectures using Databricks Lakehouse, Delta Lake, and Azure cloud services.
• Create and maintain secure, scalable data pipelines to process structured and unstructured data from multiple sources.
• Define data integration, modeling, and governance frameworks tailored to insurance data domains such as policy, claims, underwriting, and billing.
• Collaborate with actuarial, underwriting, and BI teams to design semantic layers and analytics-ready datasets.
• Optimize Databricks cluster performance and workflows to ensure cost efficiency and reliability.
• Establish and enforce data security protocols aligned with industry regulations and compliance standards.
• Implement data quality, lineage, and metadata management frameworks using tools like Unity Catalog or Collibra.
• Mentor teams on best practices for data modeling, Databricks optimization, and cloud architecture.
• Review and validate solutions from divisional teams and vendors to ensure scalability, resilience, and cost-effectiveness.
• Evaluate emerging technologies and develop blueprints for adoption to enhance data capabilities.
• Bachelor’s or master’s degree in Computer Science, Data Engineering, Information Systems, or a related field.• At least 7 years of experience in data architecture or engineering, including 4 years working with Databricks.
• Expertise in insurance data models, including policy, claims, underwriting, and accounting domains.
• Proficiency in Python, PySpark, ETL processes, and Databricks workflows.
• Strong knowledge of Azure Databricks and modern data lakehouse architecture.
• Experience integrating data with BI tools such as Power BI, Tableau, or Looker.
• Familiarity with data governance tools like Unity Catalog or Collibra.
• Excellent communication and stakeholder management skills.
Recommended Jobs
RN - Telemetry
We are seeking a dedicated RN - Telemetry in Gastonia, NC, offering $1,912/week. Responsibilities Include Monitoring cardiac rhythms and vital signs of patients in telemetry units Assessing p…
Telesales Consultant
Join Our Dynamic Team and Propel Your Career to New Heights! Are you ready to seize an extraordinary opportunity with our rapidly expanding company? We're experiencing exponential growth, with a con…
Orthopedic Surgeon in Salisbury, NC - Affordable Living, and a Rewarding Mission! EDRP & Incentive Authorized !
Join Our Team: Fellowship-Trained Total Joint Orthopedic Surgeon in Salisbury, NC – Near Charlotte & Lake Norman! Enjoy Affordable Living, a Rewarding Mission, and a Dynamic Practice with Robotic, …
Associate Applications Engineer - Food and Packaging Science
Sealed Air designs and delivers packaging solutions that protect essential goods transported worldwide, preserve food, enable e-commerce and digital connectivity, and help create a global supply chai…
Building Envelope Sales Manager
Baker Roofing Company – Business Development Benefits: ~7 Paid Holidays ~ Medical Insurance ~ Dental Insurance ~ Vision Insurance ~401K Matching Program (100% up to 3% and 50% up to 5%) …
Software Engineer (C#)
Description Software Engineer The Company: Varonis (Nasdaq: VRNS) is a leader in data security, fighting a different battle than conventional cybersecurity companies. Our cloud-native D…
Angular Developer
Angular Developer Work location: Charlotte North Carolina Open for All visa and relocation but Candidate need to be from NJ/ NC/ TX/IL and GA. F2F Interview locations: Atlanta, GA/ Jersey Ci…
Project Engineer - Great Lakes
it's what's inside that counts _______________________________ There’s more to CMC than our products and the buildings, structures, and roads they go into. At CMC, it’s the people inside our re…
Accounts Payable/Title Clerk
Job Title: Automobile Administrative Assistant with Title Clerk Experience Location: Rocky Mount, NC Hours: Monday to Friday, 8:00 AM - 5:00 PM Benefits: 401(k), Medical and Dental Insurance, Compan…
Business Development Manager, National Builders
Job Description Job Description Job Description MUST HAVE: Experience with National Builders at the corporate level Location: Remotely – Atlanta, Charlotte, Chicago, Dallas, Orlando, Phoe…