Tech Lead Data Scientist, AI Evaluation – Monitoring
Job Description:
- Principal technical expert for how Geisinger evaluates, monitors, and optimizes AI systems
- Set technical direction for AI evaluation across a growing portfolio
- Provide technical leadership to a team of data analysts
- Partner with AI program teams to improve validation and monitoring methods
- Design validation studies and monitoring plans for AI systems
- Instrument production monitoring and define measurable metrics
- Mentor analysts on methodology and technical rigor
Requirements:
- 6+ years in data science, statistics, ML engineering, or applied quantitative research
- Strong foundation in experimental design and causal inference
- Hands-on experience designing and running model evaluation studies in real production settings
- Experience evaluating LLM or generative AI systems
- Proven ability to translate ambiguous failure modes into concrete evaluation designs
- Strong fluency in Python and SQL
- Experience with fairness and equity evaluation for ML systems
- MS or PhD in a quantitative field preferred
Benefits:
- Healthcare benefits from day one
- Vision insurance
- Dental insurance
- Domestic partner benefits
- Collaboration and cooperation atmosphere
Apply tot his job Apply To this Job