AI: Legitimate Tool for High-Stakes Test Scoring? A Privacy Debate

Speaker(s)

Gary Behrens

FifthTheory

Allison Mulford

Prometric

Alan Thiemann

Law Office of Alan J. Thiemann

Marc Weinstein

Caveon Test Security

In the past decade, use of AI for scoring high-stakes assessments and reporting results has grown steadily as data-analytic and machine-learning capabilities have improved. Yet, now there are increased concerns about the fairness, ethics and privacy of using automated AI in the scoring and reporting of high-stakes and even medium-stakes exams or tests. This debate will present the merits and challenges of utilizing AI within assessments to make hiring, certification or qualification decisions for attendees consideration in the context of the current environment.

Deja What's New?: Women's Careers in Assessment

Speaker(s)

Liz Burns

Appian

Tara Miller

Tableau

Gina Raney

Alpine Testing

Sierra Hampton-Simmons

PMI

Alina Von Davier

Duolingo

Twelve years ago at the 2008 ATP Innovations conference, a panel discussion on women's careers in assessment was convened. The topics ranged from how these women got their start, to career development, to balancing family, to leadership opportunities. As we look back over this time, much has changed. There are new faces at the table and new challenges to address. However, has the assessment landscape really improved for women in this field? Join us for a thought-provoking, interactive workshop as we learn from female assessment professionals at all levels of the organization. This workshop is an opportunity to discuss the progress that has been made by creating opportunities for women in assessment. It is an opportunity to contribute to the discussion, to learn from others, and to understand what needs to take place to make change happen.

How Do We Land This Thing? A User's Guide for Testing Programs Adjusting to Their Emergency Adjustments

Speaker(s)

Linda Adler

National Board of Medical Examiners

Kimberly Swygert

NBME

Over the past 12 months, testing programs were forced to rapidly adapt to the global pandemic. The need to adopt new practices, tools and delivery methods required innovating and building in real time. For some, it felt much like the expression of, building the plane while you're flying it. Eventually, of course, we will be expected to land. Experienced professionals will talk about the activities they would have taken, time permitting, and the activities they plan to take going forward. From risk assessments, change management, and psychometric studies to evidentiary thresholds and practices, we will explore the steps testing programs can take now to ensure a smooth landing. Whether you have made significant changes or are looking to adopt changes in the future, this session will provide you with meaningful information to build and land a better plane.

Incubator Live

Speaker(s)

Jenn McNamara

Breakaway Games

ATP's Incubator Live session will bring selected finalist innovators together with a panel of industry leader judges for a brief pitch on stage followed by a live, unscripted Q&A and discussion session. In this competitive event, following each presentation, the product is judged by the panel. At the conclusion of the event, votes are tallied, and the winning participant is announced.
Don't miss out on our four top-notch participants:

• Pymetrics, inc.
• Eduworks Corporation
• Qi Learning Research Group
• GoReact

Thank you to our sponsors!

Leading Incubator Sponsor
Ascend Learning

Incubator Contributing Sponsor
DRC

Innovation Demonstrations: A Natural Language Approach to Automated Item Generation; A Total View System for Remote Proctoring; Advances in digital simulations for practical assessments

Speaker(s)

Martin Yu

HumRRO

Kranthi Bathula

Proctor360

Kevin Leonard

Breakthrough Technologies

During this video, you will view the following three demonstrations.

A Natural Language Approach to Automated Item Generation

Writing quality test items takes time. Traditionally, such items are written by human subject matter experts. Automated item generation (AIG) technology has existed for decades. However, the application of natural language understanding and generation (NLU/NLG) to AIG remains in its infancy. For example, there has been a surge of new developments in the fields of NLU/NLG regarding applications of language models that have yet to be applied to AIG.

In this demo, we showcase an innovative prototype for on-demand automated generation of test items using new, pretrained NLU/NLG models. For the demo, we will demonstrate its application to the AIG of personality and vocational interest items. We also demonstrate the automated use of text quality metrics to inform item selection decisions for subsequent test development steps, such as expert item review and test form assembly.

Speaker: Martin Yu, HumRRO

A Total View System for Remote Proctoring

There has been a sudden transition for all of education to move online, and that includes testing. One of the biggest concerns for organizations is preventing academic dishonesty, preventing content theft, and, ultimately, maintaining fairness for all their test takers. Remote proctoring is a great solution for that though current remote technologies lend themselves to potential cheating due to their limited view. Many websites, articles and YouTube videos show how to hack the system. Proctor360's novel 360 camera technology captures the full environment and prevents academic dishonesty from occurring and records it if it does. Proctor360's unique approach to the remote proctoring market and its needs are the reason why it was named as one of the Top-10 EdTech companies of 2020 by Education Technology Insights.

Speaker: Kranthi Bathula, Proctor360

Advances in digital simulations for practical assessments

The year 2020 showed us that there is more need than ever to practice, demonstrate and assess skills remotely. In this session, you'll learn how organizations and corporations are utilizing technologies including Unity, VR, AR, AI in their formative and high-stakes assessments.

Speaker: Kevin Leonard, Breakthrough Technologies

Innovation Demonstrations: Innovative Technology Using Machine Learning and Natural Language Processing to Facilitate Content Development; More Effective and Powerful Surround AI-Assisted On-line Proctoring; Sero! for concept mapping-based learning assessment

Speaker(s)

Lance Blackstone

Certiverse

During this video you will view the following three demonstrations.

Innovative Technology Using Machine Learning and Natural Language Processing to Facilitate Content Development

This innovation showcase will demonstrate a groundbreaking, new, assessment- development platform, which leverages machine learning and natural language processing for item writing and review. With psychometric standards and best practices built into the underlying technology, high-quality content can be developed remotely, with subject matter experts working asynchronously. The ML & NLP automation, SME engagement tools and support enables item writers to produce content in a more efficient and cost-effective manner.. This revolutionary platform removes time-consuming and expensive elements from the traditional, exam-development model. Learn in this session how to leverage this new technology to solve the time and cost issues usually involved in content development.

Speaker: Lance Blackstone, Certiverse

More Effective and Powerful Surround AI-Assisted On-line Proctoring

The efficacy and validity of AI-assisted, online proctoring lies in its increasingly more efficient and accurate detections of actual test violations. The purpose of this presentation is to demonstrate our surround, AI-assisted, online proctoring system enhanced from our previous, one-camera, AI-assisted, online proctoring system. In addition to our oral summary, we intend to call upon 30 or more willing attendees to be fake cheaters to see how effectively and accurately our one human proctor and our surround, AI-assisted, online proctoring system can detect their simulated cheating behaviors. It can be concluded that our surround, AI-assisted, online proctoring technology not only increases accuracy but also efficiency. At the end of the demonstration, comprehensive explanations will be given regarding the principles, algorithms and technology as well as potential issues associated with AI-assisted, online proctoring in general.

Speaker: Xiang Wang, ATA

Sero! for concept mapping-based learning assessment

Concept mapping for learning assessment has a proven track record of tapping into higher-order thinking skills that are hard to get at with multiple choice tests but efficiency and scalability have hampered widespread use. Notably, concept mapping-based assessment is cited in the Science Framework for the National Assessment of Education Progress but it has not yet been piloted. Innovation is needed to bring the technique into use. A finalist for the 2018 ATP Innovation Lab, Sero! is the world's most advanced concept mapping-based learning assessment platform. Development of Sero! has been funded through three Department of Defense grants, the most recent aiming to introduce even more efficiency into the authoring process through the introduction of artificial-intelligence capabilities. This presentation will provide a brief overview of the goals, development history, and current and planned capabilities for Sero!

Speaker: Brian Moon, Sero! Learning Assessments

Lean-Agile Mindset: A Framework for Innovation

Speaker(s)

Susan Farago

Scaled Agile, Inc.

You have an interesting idea that has exciting potential to your customers and possible benefits to your program or organization. What do you do? It is tempting to jump straight in to finding or building the best solution but are you sure you are working on the right problem? What happens if you build it and no one buys it?

In this session, we will explore the principles and practices around a Lean-Agile mindset and how it will shift the way you work. These principles and practices include: identifying the problem to solve, creating a minimal viable product (MVP), working in small batches while leveraging customer feedback, and making pivot or persevere decisions. Case studies are used to provide examples of taking a Lean-Agile approach and include certification-related products developed in 2020 ? some highly successful and some that never made it to market.

You will be able to take your newfound Lean-Agile mindset back to your workplace and learn how to quickly sort through ideas, experiment to find out if an idea solves a customer problem, minimize wasted development time, and quickly bring your ideas to life (and to the market).

Models Are for More than Just Psychometrics: Innovations in Content Development that Every Testing Organization Should Still Be Considering Seriously

Speaker(s)

Drew Dallas

National Commission on Certification of Physician Assistants

Naomi Pitkin

American Board of Internal Medicine

The landscape of high-stakes certification assessment is changing. For many testing organizations, single-form administrations requiring small numbers of items attached to a bank scale are over. Many of today's assessments place greater demands on test developers and push traditional models of content development out of their comfort zones. Fortunately, theories of content development are also evolving. The emerging technology of model-based content development, exemplified by Automatic Item Generation, represents a proven method of developing content rapidly and effectively. This innovative approach to solving one of the chief concerns in test development represents an opportunity to modernize the testing industry. This presentation will address the lessons learned from two organizations that are actively investigating the operationalization of models as part of their content development processes. Participants are encouraged to develop models of their own as part of the workshop.

Remote Proctoring: A Practitioners' Guide to Implementation

Speaker(s)

Mike Olsen

Proctorio

Jim Larimore

RIID Labs

Carey Wright

State of Mississippi, National Assessment Governing Board

Chad Buckendahl

ACS Ventures

Ada Woo

Ascend Learning

Remote proctoring (RP) technology is attracting a lot of attention both in and outside of the assessment industry. The rapid and widespread adoption of RP has allowed many testing organizations to continuing serving their stakeholders during the pandemic offering a ready solution when traditional approaches may not have been available. While it is generally seen as a good tool by assessment professionals, the public is increasingly focusing on the privacy and surveillance concerns of RP. In addition, there are concerns about equity as it relates to technology access and the ability to offer appropriate accommodations.

This debate will provide a forum for attendees to hear both sides of the argument and engage in this important conversation about policies and practices of remote proctoring.

Should Interim Assessments replace Summative Assessments for K-12 Accountability?

Speaker(s)

John Oswald

The Oswald Consulting

Scott Marion

Center for Assessment

JT Lawrence

Cambium Assessments

Due to COVID19, and the fact that students were not in schools, all accountability testing was canceled in the spring of 2020. Some states have indicated their desire to also suspend accountability testing in the spring of 2021 and have asked the US Department of Education for a waiver under the requirements of ESSA . Many schools continue to administer -- often remotely at home -- interim assessments and other low-stakes assessments that have typically been used only for diagnosis and prescription. There have been opinions expressed for years that aggregating the results of low stakes assessments can be used for accountability in lieu of high-stakes summative assessments administered on demand in the spring of each year. Publishers and administers of high-stakes summative assessments are concerned that canceling their administration, or making their administration optional, for two years in a row would spell the end of a very large segment of the assessment business. A more important concern is that many experts in measurement question whether aggregating the results of interim and formative assessments can replace the quality of information that comes from on-demand summative assessments that are designed specifically for that purpose. This session will address the concerns on both sides of that argument, and its short-term and long-term effect on the educational assessment industry.

So You're Thinking About Implementing Automatic Item Generation at Your Organization? More Practical Issues You'll Wish You'd Considered Beforehand

Speaker(s)

Drew Dallas

National Commission on Certification of Physician Assistants

Naomi Pitkin

American Board of Internal Medicine

Model-based item development techniques are increasingly common. Many presentations have discussed the technical aspects of building models. These presentations raise the question: How do you go about implementing this initiative?

This session will answer this question by sharing lessons learned from two certification testing organizations in the process of operationalizing AIG. Topics covered will include the initial steps of recruiting and training of subject matter experts; transforming thinking among staff and other stakeholders; and, ultimately, bringing about an evolution in the operational process used to support content production for high-stakes examinations. Two new topics will receive attention this year: 1) results from pilot studies aimed at process improvement, and 2) strategies for incorporating model-based items into operational test forms and evaluating item functioning. Audience members will gain an understanding of how to make decisions critical to the success of AIG.

Test Accommodations: The Perils of the "Approve Everything" Model

Speaker(s)

Donald Balasa

American Association of Medical Assistants

Heather Case

International Council for Veterinary Assessment

The presenters drafted a journal article entitled, Testing Accommodations: The Perils of the Approve Everything Model, published in the Journal of the National Collegiate Testing Association in late 2019. The presenters will discuss the paper and its conclusions. We will discuss the range of business risks associated with the approve everything approach to accommodations, and we will articulate its potential pitfalls, including risks to test validity, test security, and the integrity of the testing process. Participants will have an opportunity to consider actual examples that will illustrate each point. The presenters will describe the advantages of taking an individualized, balanced and thoughtful approach to evaluating test accommodations requests one that ensures equal access for candidates with disabilities, while also taking into account test security, integrity, validity and operational considerations as well as the fundamental mission of the organization.

Test-taker centered design through prototyping and user testing

Speaker(s)

Ann Evers

GED Testing Service

Belinda Brunner

Inteleos

Do your test takers understand test interfaces the way you intend? With the tools you'll learn in this session, you can proactively make sure they do.

Item prototyping and user testing help you to learn, discover and pivot quickly. These tactics are important because any change to an item or test interface will impact the candidate's experience and has the potential to affect validity. To ensure that your innovations make sense to your candidates and support good measurement, you should try things out early and iterate often; this will lead to valuable improvements. For example, if you prototype a new item on paper, you can save resources. The prototype can then be used with real test takers in user testing to uncover pain points with the item and resolve them well in advance of testing.

The workshop emphasizes hands-on activities and how-to approaches. You will gain experience creating prototypes and learn user testing through small groups using reflective discussion.

Will Brick-and-Mortar Test Centers Even Be Relevant in the Future?

Speaker(s)

Kellie Early

NCBE

Bill West

Examity

John Fremer

Caveon

Jarret Dyer

College of DuPage

The pandemic has expanded acceptance of new education and test-delivery methods and proctoring tools. Much has been debated about the merits of remote testing and proctoring, but as an industry, we have yet to explore the converse: As more academic institutions and testing programs adapt to remote learning and testing, will brick-and-mortar test centers remain relevant? And if so, how should testing programs plan to address a world where individuals expect more immediate access to education and testing? Join testing professionals who are familiar with multiple modes of testing and the needs of different segments of the testing market to debate the longer-term viability of test centers and the impact of shifting learner and test-taker expectations on testing programs today.