AI Technical Lead

Job Description

Job Description

O

AI Technical LeadReports to: Chief Information Technology Officer

Our Organization

We are a medical specialty certifying board serving anesthesiologists. Since 1938, we have been administering certification exams and today we take an innovative approach to certification and continuous learning. We foster practice standards that instill confidence and trust that board-certified anesthesiologists have the knowledge and skills to provide high-quality patient care. We are dedicated to elevating expertise in an evolving profession. Our mission is to advance the highest standards of the practice of anesthesiology. We work together with physician anesthesiologists to ensure they provide the best care possible for every patient, every day.

Position Description

The ABA is seeking an experienced AI Technical Lead to build, maintain, and evolve production-grade AI/ML systems using state-of-the-art libraries and frameworks, grounded in a deep understanding of AI and machine learning concepts. The AI Technical Lead is primarily responsible for applying AI/ML theory to real-world problems, translating high-level objectives into concrete technical solutions, and connecting data, models, and systems into cohesive, end-to-end implementations. The role partners closely with AI, product, psychometrics, and engineering teams, and require the ability to communicate complex AI concepts clearly and precisely with highly technical AI and engineering audiences.

Education

• Bachelor’s or Master’s degree in computer science, Artificial Intelligence, Statistics, Mathematics, or a related field.

Skills

• Proactive self-starter with excellent interpersonal, communication, and customer service skills.
• Expert-level AI/ML and full-stack development skills, with strong hands-on experience building and integrating backend services and frontend applications using modern frameworks such as Node.js and React. Strong emphasis on clean, maintainable, reproducible, well-tested, and well-documented code.
• Ability to manage multiple tasks and projects simultaneously.
• Collaborative team player with a focus on achieving common goals.
• Meticulous attention to detail.
• Quick learner with a passion for staying current with emerging technologies and industry trends.

Experience

Required Experience

• Deep expertise in RAG systems, LLMs, embeddings, vector databases, and AI infrastructure
• Experience designing semantic retrieval and knowledge platforms, including curated corpora and grounding/citation patterns (e.g., “show your sources” for internal auditability)
• Experience evaluating AI models for different tasks
• Strong ability and experience to leverage cloud infrastructure
• Experience with data quality and metadata management (data lineage, dataset versioning, business glossary/taxonomy) and implementing automated quality checks and anomaly detection
• Experience implementing secure GenAI platform controls, including prompt logging, red-teaming, content filtering/leakage prevention, and model access controls/tenant isolation
• Excellent collaboration skills and ability to work with non-technical stakeholders
• Proven ability to work in existing codebases, improve reliability, performance, and readability over time.
• 5+ years of hands-on experience in machine learning engineering, with significant work developing and maintaining AI systems in production.
• Strong software engineering practices including modular design, refactoring, and technical debt management; unit, integration, and regression testing; as well as code reviews and shared coding standards
• Strong expertise in machine learning fundamentals and statistical modeling, including, but not limited to:
  • Supervised, unsupervised, and reinforcement learning
  • Model evaluation, bias, overfitting, and error analysis
  • Probabilistic and statistical reasoning
• Proficiency in Python and major ML libraries (e.g., TensorFlow, PyTorch, scikit-learn, HuggingeFace, LangChain, LlamaIndex).
• Hands-on experience with cloud-based ML platforms (AWS SageMaker, Azure ML, or Google AI Platform).

Valuable Additions

• Deep understanding of and experience with MLOps tooling (CI/CD for ML, experiment tracking, monitoring)
• Experience with model registry and release management, monitoring with drift detection, and operational governance (runbooks, incident response, and post-incident reviews) for AI/ML systems.
• Ability to translate complex business problems into machine learning solutions and communicate technical content to non-technical stakeholders.
• Experience working within agile, cross-functional teams.
• Experience with educational technology.
• Professional certifications such as AWS Certified Machine Learning or TensorFlow Developer Certificate are preferred.

Specific Responsibilities

• Build and deploy predictive and generative models (e.g., adaptive scoring, automated item classification, content gap analysis, chat-based candidate support).
• Own the end-to-end production ML pipeline: data ingestion and preprocessing, feature engineering, model training/validation, fine-tuning, model versioning and reproducibility, MLOps, monitoring.
  • Define and enforce data and knowledge governance for AI systems: curated sources for RAG, data lineage and dataset versioning, metadata standards, and business glossary/taxonomy management.
  • Build automated data quality checks and monitoring (freshness, schema/constraint checks, distribution shifts) with anomaly detection and alerting to protect downstream model performance.
• Partner with Psychometrics to validate model performance against gold-standard measurement theory. Work with Exam Development to pilot large-language model (LLM) workflows for draft item generation with subject-matter expert review.
• Build and improve LLM-based systems, including retrieval-augmented generation (RAG) pipelines.
• Implement grounding and citation requirements for LLM outputs where appropriate, enabling traceability to approved sources and supporting internal review/audit workflows.
• Own MLOps governance: model registry and promotion workflows, monitoring and drift detection, reproducible training/inference pipelines, and incident response for AI issues (triage, rollback, communication, and corrective actions)
• Optimize model performance, latency, and cost through profiling and experimentation.
• Own the full lifecycle of AI features: prototype → production → maintenance → iteration.
• Review and refactor existing AI/ML codebases to improve robustness and clarity.
• Collaborate with cross-functional teams to translate requirements into working, maintainable implementations.
• Establish and uphold coding standards, patterns, and best practices for AI development.
• Mentor engineers and analysts through code reviews and technical guidance (without formal people management)
• Write high-quality, well-documented, and testable code for AI-driven applications.
• Establish coding standards and reproducible research templates. Deliver periodic “AI Literacy” workshops for staff, board members, and external diplomates.
• Evaluate and negotiate with cloud/AI vendors, ensuring security, cost control, and integration with existing Microsoft/AWS stack.

Special Requirements

• Ability to lift 10 pounds maximum.
• Ability to sit for 6+ hours per day.
• Must be able to work onsite in Raleigh, NC, at least two days/week.