📍 San Francisco Bay Area or New York City Area
🧠 Python | LLMs | NLP | Machine Learning | AWS
💰 $170K–$200K USD base salary + equity
🛂 Visa sponsorship available for candidates already in the US who require a visa transfer
🌱 Pre-seed AI startup | Early engineering team
Build the intelligence layer that makes AI customer conversations accountable
Are you a hands-on Machine Learning Engineer who enjoys turning modern AI technologies into reliable, measurable, production-grade systems?
We're looking for a Machine Learning Engineer to help build the intelligence and compliance layer of an early-stage AI startup tackling one of the most important challenges in enterprise AI: ensuring customer-facing AI agents are accurate, safe, compliant, and accountable.
You'll work closely with the CTO and engineering team, developing the models, evaluation frameworks, and production ML capabilities that power the company's AI monitoring and assurance platform.
This isn't a research-only position. You'll work with real-world data, evaluate model performance, build production solutions, and help solve challenging problems involving accuracy, reliability, and operational efficiency.
🧠 The opportunity
Enterprises are increasingly deploying AI agents to handle customer conversations across voice, chat, email, and SMS.
However, businesses need an independent way to measure their performance, identify failures, detect hallucinations, and demonstrate compliance.
The company is building an AI monitoring and assurance platform that evaluates customer interactions against applicable regulations and business requirements.
You'll contribute to the intelligence that makes this possible — from compliance detection and hallucination monitoring to evaluation frameworks, scoring models, and production ML observability.
🔧 What you'll work on
🛡️ Compliance, hallucination, and scoring
Develop AI-powered systems to detect compliance issues, hallucinations, ungrounded answers, and interaction-quality problems.
Work with rules, classifiers, LLM-based judges, and other evaluation approaches.
Help improve model accuracy, reliability, and confidence thresholds.
📊 Evaluation frameworks and data
Develop evaluation datasets and testing frameworks.
Work with human-labelled data and ground-truth datasets.
Build regression tests to identify model performance issues.
Explore synthetic data, weak supervision, and active learning where appropriate.
⚙️ Customer-specific AI evaluation
Help translate customer requirements and regulatory frameworks into measurable evaluation criteria.
Develop prompts, datasets, and workflows for customer-specific scoring.
Improve the consistency and reliability of AI evaluation across different customers.
🎙️ Speech and conversation intelligence
Work with noisy transcripts and unstructured conversational data.
Explore NLP and LLM techniques for conversation analysis.
Contribute to speech-processing capabilities, including ASR, speaker diarisation, and streaming transcription.
☁️ Production ML and performance
Develop and maintain production ML services using Python.
Work with model inference, routing, latency, and cost optimisation.
Monitor model performance and identify potential regressions.
Help build reliable ML workflows and monitoring capabilities.
🤝 Collaboration and engineering
Work closely with the CTO and engineering team.
Collaborate with customer-facing teams to understand real-world AI performance challenges.
Contribute to technical discussions and the evolution of the ML architecture.
Help establish effective ML engineering and evaluation practices.
🎯 What we're looking for
Applied ML experience: Practical experience developing or maintaining machine learning, NLP, or LLM-powered applications.
Strong Python skills: Comfortable writing production-quality Python code and working with data processing pipelines.
LLM experience: Hands-on experience with modern LLMs, prompting, evaluation, retrieval, and model integrations.
Model evaluation: Understanding of model accuracy, precision, recall, ground-truth datasets, and evaluation methodologies.
Machine learning fundamentals: Knowledge of classification, NLP, model selection, and practical ML techniques.
Production experience: Some experience deploying or maintaining ML-powered applications in production environments.
Data handling: Comfortable working with unstructured data, noisy datasets, and human-labelled information.
Cloud familiarity: Experience with AWS or similar cloud environments.
Practical engineering mindset: Able to recognise when a simple rule or traditional ML approach is more appropriate than an LLM.
Startup mindset: Comfortable working in a small engineering team where flexibility, initiative, and continuous learning are important.
AI-assisted development: Familiarity with coding agents and AI-assisted development tools is beneficial.
✨ Bonus points
Production speech and audio systems, including ASR, diarisation, and streaming transcription.
Custom NLP or scoring-model development.
LLM-as-judge systems and evaluation against human ground truth.
Weak supervision, programmatic labelling, or active learning.
Trust and safety, fraud detection, risk management, or regulatory compliance.
Contact-centre quality assurance or workforce optimisation.
🌟 Why consider this opportunity?
Join an early-stage AI company working on a meaningful enterprise problem.
Work directly with the CTO and engineering team.
Gain exposure to real-world applied AI and LLM challenges.
Build production ML solutions rather than focusing exclusively on research.
Contribute to the evolution of an AI platform from its early stages.
Work on problems involving model reliability, measurable accuracy, and AI accountability.
👉 Apply Now! 📩
SourceIn (sourcein.co) has been formally appointed by the client to provide recruitment and candidate pre-screening services for this role.