PunditSpace is seeking experienced AI Evaluation Engineers specializing in Engineering Simulation & Design. You will author and validate "model-breaking," simulation-based engineering design problems to train and evaluate state-of-the-art AI agents across major engineering disciplines. Your work will involve analyzing agent execution logs, exposing systemic reasoning gaps, and building automated, objective graders to elevate frontier model performance.

You will be an experienced AI Evaluation Engineer specializing in Engineering Simulation & Design. Your mission will be to author and validate "model-breaking," simulation-based engineering design problems to train and evaluate state-of-the-art AI agents. You'll operate across major engineering disciplines—including Electrical, Mechanical, Control Systems, Aerospace, Systems, and Robotics—creating complex, multi-constraint tasks where AI agents must interpret requirements, navigate trade-offs, configure open-source simulation tools, diagnose failures, and iterate toward valid solutions. You will analyze agent execution logs, expose systemic reasoning gaps, and build automated, objective graders to elevate frontier model performance. This is not a typical engineering role; it requires a deep understanding of engineering principles combined with expertise in AI evaluation to push the boundaries of AI agent capabilities.

What you'll do

Author original, self-contained engineering design tasks with competing constraints, explicit optimization targets, validated reference solutions, and objective autograders.
Build, run, and validate problem environments using open-source simulation tools and custom Python test benches.
Evaluate coding agent outputs and execution logs across repeated trials to identify systemic failure modes (e.g., misinterpreting simulator feedback, premature design convergence, physically impossible geometries).
Iteratively refine problem difficulty based on empirical model performance data without introducing ambiguity or missing information.
Partner with AI researchers, pod leads, and domain experts to integrate high-rigor benchmarks into the model evaluation pipeline.

Requirements

You have a Master’s degree or PhD in Electrical, Mechanical, or Aerospace, with 10+ years of hands-on engineering design experience.
You are proficient with at least one domain-relevant open-source simulation package (e.g., ngspice, PySpice, OpenFOAM, FEniCSx, CalculiX, python-control, CadQuery, build123d, OpenModelica, Cantera, Gmsh) combined with strong Python scripting skills.
You have hands-on experience with modern LLMs/coding agents and evaluation concepts (pass@k, failure-mode analysis, nondeterministic behavior), with the ability to audit trajectory logs and isolate core reasoning/tool-use failures.
You possess uncompromising attention to physical plausibility, unit consistency, boundary conditions, convergence criteria, and technical documentation.
You can commit 40 hours per week with 4 hours of overlap with PST and have weekend on-call availability.
You have a personal desktop/laptop equipped with a stable, high-speed internet connection in a remote setup.

Nice to have

Experience in AI evaluation, data annotation, content review, quality assurance, or a related analytical role is preferred but not required.

AI Evaluation Engineer (Engineering Simulation & Design)