AI Evaluation Engineer, Arizona State University, AZ, US

Job Profile

Applications Developer 2

Job Family

IT Applications

Time Type

Full time

Max Pay – Depends On Experience

$105,000.00 USD Annual

Apply before 11:59 PM Arizona time the day before the posted End Date.

Minimum Qualifications

Bachelor's degree and three (3) years of experience appropriate to the area of assignment/field; OR, Any equivalent combination of experience and/or training from which comparable knowledge, skills and abilities have been achieved.

Job Profile Summary

Assists in design, development, and troubleshooting software and applications in support of establishing a sustainable institutional technological infrastructure.

Job Description

Are you passionate about using AI to advance online higher education? Do you have strong analytical skills and solid, hands-on experience with AI-driven tools and model evaluation techniques? Then keep reading below for this AI Evaluation Engineer opportunity!

NOTE : This is not a fully remote position. Must be able to reliably commute to Scottsdale Arizona three days a week.

EdPlus at ASU is a dynamic unit of Arizona State University focused on the design and scalable delivery of digital teaching and learning models to increase student success and reduce barriers to achievement in higher education. We advance the economic, social, cultural, and overall health of the local, national, and international communities served by ASU.

Want to know more about working at EdPlus? Click here!

We are seeking an AI Evaluation Engineer to join our Solutions Engineering team. Every day you will make a difference in the lives of others by playing a key role in the learner’s experience across the EdPlus digital portfolio. You will be responsible for evaluating and optimizing AI models, particularly large language models (LLMs). You will be doing a combination of data preparation, performance assessment, bias detection, and model optimization. In addition, you will have close collaboration with cross-functional teams to ensure that AI models meet both technical and business objectives.

The Solutions Engineering team is responsible for creating solutions and enhancing our current technology stack for a variety of challenges as part of an innovative environment, refining how users engage with higher education. Working collaboratively in diverse teams creates forward-thinking and thoughtful experiences, ensuring web products support ASU’s and EdPlus’ overall strategy and goals.

Essential Duties

• Gather and preprocess structured and unstructured datasets, ensuring data quality and suitability for AI model evaluation.
• Utilize AI-driven tools to evaluate model outputs for factual accuracy, relevance, and completeness, especially for large language models (LLMs).
• Assess model performance using standard metrics (e.g., accuracy, precision, recall, F1 score) and advanced evaluation techniques.
• Conduct comparative analysis of multiple LLMs, algorithms and prompts to select the best-performing model based on specific KPIs and business goals.
• Ensure proper labeling and augmentation of the knowledge base, when necessary, to improve model accuracy.
• Detect and mitigate any biases in model predictions, ensuring fairness and reducing the risk of harmful outputs.
• Develop strategies to identify and eliminate hallucinations and other unintended behaviors in the model.
• Develop and implement continuous monitoring systems to track model performance in real time, detecting anomalies, degradation, or model drift.
• Proactively identify signs of data drift or model drift and suggest corrective actions.
• Monitor and optimize model latency, throughput, and infrastructure cost to ensure efficiency in production environments.
• Work closely with product owners and other engineers to analyze model outcomes and design experiments to validate results.
• Partner with other engineers to ensure smooth integration of AI models into production environments and address operational bottlenecks.
• Collaborate with business stakeholders to define AI performance metrics that align with overall business objectives and customer needs.
• Design and implement consistent evaluation frameworks and KPIs to assess model performance over time.
• Create and maintain automated pipelines for model evaluation, ensuring that performance checks are part of the model lifecycle from development to deployment.
• Stay updated on the latest advancements in AI research, methodologies, and industry trends to drive innovation and improve model performance.
• Conduct internal research and experiments to explore and validate new AI techniques, models, and tools that could improve evaluation and performance.
• Ensure that AI models meet ethical guidelines and comply with relevant industry standards and regulations (e.g., GDPR, privacy concerns, fairness audits).
• Provide insights into the ethical implications of AI models and advocate for responsible AI practices throughout the development and deployment process.
• Maintain clear and concise documentation of model performance assessments, evaluation processes, and changes to model configurations.
• Prepare detailed reports for stakeholders, summarizing findings from performance evaluations, including insights on strengths, weaknesses, and recommended
• Assumes or coordinates other duties or projects as assigned or directed.
  
  

NOTE : This is not a fully remote position. Must be able to reliably commute to Scottsdale Arizona.

Desired Qualifications

• Bachelor’s degree or higher in a related field.
• At least 4 years of hands-on software development and/or AI solutioning experience.
• At least 4 years of hands-on data modeling and predictive analytics experience.
• Ability to communicate with cross-functional teams about various AI topics such as LLMs, VectorDBs, RAGs, etc.
• Experience developing AI/ML solutions on platforms like AWS, GCP, Azure, OpenAI, Databricks, Snowflake, etc.
• Demonstrated understanding of software development lifecycle and standard methodologies.
• Demonstrated ability to communicate thoughtfully, using problem-solving skills and build positive working relationships with cross-functional teams.
• Experience developing applications that build AI APIs into smart agents.
• Experience developing applications using multimodal AI models.
• Experience designing, developing, and implementing generative AI models and algorithms, utilizing large language models (LLMs) for applications such as text generation, audio-to-text transcription, and qualitative data insights.
• Experience designing, developing, and deploying Retrieval-Augmented Generation (RAG) models using vector databases, including strategies for embedding, chunking, and indexing to build domain-specific chatbots.
• Experience developing resilient foundations for collecting, storing, modeling, and analyzing massive data sets from multiple sources.
• Experience with data manipulation and analysis using tools like Python and SQL
• Expertise with data cleaning and processing, model training, and evaluation techniques.
• Demonstrated ability to preprocess and curate text data, including cleaning, tokenization, and data augmentation, to prepare it for training and model evaluation.
• Demonstrated ability to build high-level and detailed design plans for AI and ML solutions, selecting suitable algorithms, data sources, infrastructure, and technologies that align with organizational goals.
• Experience conducting experiments, fine-tuning models, and optimizing their performance for accuracy and efficiency.
• Demonstrated passion for ethical AI and a drive to solve real-world problems using AI technology.
• Demonstrated ability to complete projects with minimal supervision and direction.
• Evidence of excellent verbal and written communication skills.
  
  

NOTE: Please answer the following questions in your cover letter:

• Are you able to commute three days a week to the office in Scottsdale, AZ?
• How are you using AI in your day-to-day work/life?
• Share a recent accomplishment you are proud of.
  
  

Salary & Benefits

$95,000 - $105,000 per year, DOE

ASU offers a total compensation package that includes valuable employee benefits. Click here to explore options.

• Healthcare
• Financial Security
• Retirement
• Family Resources
• Tuition Reduction (Eligible ASU Employee, their dependents and spouse.)
• Discounts
  
  

Working Environment

Your desk will be in the beautiful, cutting-edge, and collaborative workspace at SkySong, the ASU Scottsdale Innovation Center. The Center houses a diverse business community that links technology, research, education, and entrepreneurship to position ASU and Greater Phoenix as global leaders in the knowledge economy.

EdPlus at ASU supports flexible work options, ranging from alternate work schedules to hybrid remote work schedules, subject to approvals per ASU policy. (This is NOT a fully remote position.)

Applicant must be eligible to work in the United States. EdPlus at ASU will not be a sponsor for this position.

Department Statement

EdPlus defines itself through a culture of curiosity, risk-taking and refusing to accept the status quo. As a central enterprise unit for ASU focused on the design and scalable delivery of digital teaching and learning models to increase student success and reduce barriers to achievement in higher education, our commitments support the University charter. ASU is a national service university and aspires to accelerate positive social outcomes through the seamless integration of cutting-edge technological innovation and scalability.

EdPlus employees are valued, respected, and encouraged to be their unique selves. We know that our ability to deliver high-quality services and educational experiences is strengthened by our culture. Our innovative approach drives outcomes through serving learners, achieving milestones, striving for excellence, solving problems, embracing urgency, and being bold.

EdPlus at ASU

Driving Requirement

Driving is not required for this position.

Location:

Off-Campus: Scottsdale

Funding

No Federal Funding

Instructions To Apply

Current Employees and Students should apply directly within Workday using the Jobs Hub. Use this link and log in using SSO:

https://www.myworkday.com/asu/d/inst/1$9925/9925$12720.htmld

To be considered, your application must include all of the following attachments:

• Cover letter
• Resume/CV
  
  

Note: Multiple documents can be submitted into the attachment box. Alternatively, merge all documents into one PDF for submission.

Please include all employment information in month/year format (e.g., 6/88 to 8/94), job title, job duties and name of employer for each position. Resume should clearly illustrate how prior knowledge and experience meets the Minimum and Desired qualifications of this position. NOTE: GA and Intern positions are counted as .5 for job experience (ie. 1 year equals 6 months experience equivalency). Only electronic applications are accepted for this position.

Need help finding the right job?

We can recommend jobs specifically for you! Create a custom Job Alert by selecting criteria that suit your career interests.

ASU Statement

Arizona State University is a new model for American higher education, an unprecedented combination of academic excellence, entrepreneurial energy and broad access. This New American University is a single, unified institution comprising four differentiated campuses positively impacting the economic, social, cultural and environmental health of the communities it serves. Its research is inspired by real world application blurring the boundaries that traditionally separate academic disciplines. ASU serves more than 100,000 students in metropolitan Phoenix, Arizona, the nation's fifth largest city. ASU champions inclusive excellence, and welcomes students from all fifty states and more than one hundred nations across the globe.

ASU is a tobacco-free university. For details visit https://wellness.asu.edu/explore-wellness/body/alcohol-and-drugs/tobacco

All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, national origin, disability, protected veteran status, or any other basis protected by law.

Notice of Availability of the ASU Annual Security and Fire Safety Report:

In compliance with federal law, ASU prepares an annual report on campus security and fire safety programs and resources. ASU’s Annual Security and Fire Safety Report is available online at https://www.asu.edu/police/PDFs/ASU-Clery-Report.pdf. You may request a hard copy of the report by contacting the ASU Police Department at 480-965-3456.

Relocation Assistance – For information about schools, housing child resources, neighborhoods, hospitals, community events, and taxes, visit https://cfo.asu.edu/az-resources.

Employment Verification Statement

ASU conducts pre-employment screening which may include verification of work history, academic credentials, licenses, and certifications.

Background Check Statement

ASU conducts pre-employment screening for all positions which includes a criminal background check, verification of work history, academic credentials, licenses, and certifications. Employment is contingent upon successful passing of the background check.

Fingerprint Check Statement

This position is considered safety/security sensitive and will include a fingerprint check. Employment is contingent upon successful passing of the fingerprint check.