The National Board of Fitness Examiners (NBFE) is a not-for-profit corporation founded in 2003 to fulfill two distinct needs. First, the NBFE addresses and defines a series of prescribed “scopes of practice” for fitness professionals. These definitions include personal fitness trainers and specialists in areas such as youth and senior fitness, and medical exercise. The second need is determining the “standards of practice” for each of these roles and assessing fitness professionals based on those standards. As they are developed, these standards will be articulated as formal statements of skills and knowledge that are associated with specific roles in the fitness industry. From these standards, the Board will generate examinations deployed in a secure and proctored testing environment.
Creating the first industry-wide examination for personal fitness trainers is a comprehensive process requiring input and feedback from many sources. The NBFE employs the very best practices in high-stakes testing and leading psychometric standards as defined by the American Psychological Association, American Educational Research Association, National Council of Measurement and Evaluation, and the American Evaluation Association. To assist us in accomplishing this task, we have contracted with LaserGrade, the world's foremost company for developing and deploying high-stakes examinations. Each year, LaserGrade deploys hundreds of thousands of tests in the medical, allied health, academic, and vocational arenas.
In the context of testing, validity means that an examination tests what it is intended to test. Reliability means that the examination consistently measures each test taker with predictable results. An unbiased test is one that provides no advantage to test takers other than that based on the knowledge associated with the examination. The partnership of NBFE and LaserGrade has resulted in a standardized examination that is valid, reliable and unbiased in the form of the NBFE Personal Fitness Trainer Examination, Part 1.
Generally, standards-based tests begin with a domain analysis, which determines the skills and knowledge of a defined practice or area of content. The NBFE has used several methods to establish and validate the practice domain for personal fitness trainers. Our domain analysis began with an open request for nominations for a focus group of leading subject matter experts that represent the best practices and broad perspectives of the industry. The NBFE received over 100 nominations for these positions. Several NBFE Executive Board members and staff reviewed the pool and selected 12 subject matter experts based on their professional merit and standards of selection provided by LaserGrade. These experts are all active and experienced personal trainers and most held multiple fitness certifications and/or academic training in related fields. We call this group of experts NBFE Fellows.
In August of 2004, the Fellows, several top psychometricians from LaserGrade and NBFE executive staff assembled in Provo, Utah to initiate the domain analysis. Work began with a pre-meeting survey to make some cursory determinations about the most relevant content areas in the industry to be included in the exam. From this survey, a preliminary Test Definition Document was composed. This document included proposed testing domains, a statement of presumed knowledge of testing candidates, identification of the primary industry stakeholders, and suggested testing format and modes. Using a process called Task Expansion, the Fellows refined and elaborated on areas of the Test Definition Document to identify seven major testing areas and sixteen measurable test objectives. This process finished with a more granular examination of the test objectives used to develop test items at a later point.
Following the Task Expansion, the Fellows worked on a series of exercises to weigh the test objectives based on their importance and criticality to practice. Each Fellow weighed the objectives individually. Each score was then collected, collated and tallied to examine the agreement in weight scores. The result showed a great deal of consistency between the Fellows when determining which test objectives were the most, and least, critical. The finalization of weights in this exercise required only slight adjustments. The workshop concluded with a revision of the Test Definition Document based on the work of the Fellows.
However, the focus groups of the NBFE Fellows represented only part of the domain analysis. Since the NBFE Fellows comprise a small, albeit authoritative perspective of the fitness industry, the NBFE conducted an industry-wide survey to validate our findings. The survey was composed in conjunction with the NBFE's testing vendor, LaserGrade. It invited responders to identify, rate and comment on major areas within personal training including each of the testing areas and their objectives. It also provided an area for qualitative comments and suggestions for additional testing areas.
The survey was deployed in two ways. First, it was implemented as a closed survey that was made available only to a sample of active personal trainers. This sample was created based on names and contact information received from several certification organizations and professional associations within the industry. Over 25,000 trainers received email inviting them to participate in the survey. Once the data from this survey were parsed and "scrubbed" (duplicate files removed, corrupt files corrected, formatting fixed, etc.), the NBFE had over 1,600 responses. The instrument was also implemented as an open survey available from the NBFE's web site, nbfe.org. The open survey gave access to a larger group of responders but did not provide for as much control of the sample as the closed survey. The process of data scrubbing and preparation was otherwise the same.
The survey was employed to accomplish two tasks. First, the NBFE wanted to be sure that the data received from both the closed and open survey was comparable and similar. There was a great deal of agreement in each item within the survey between the two samples. The primary reason for both surveys was to determine if the major testing areas and sixteen measurable test objectives and their weights identified by the NBFE Fellows were valid. Once again, the survey showed a great degree of agreement between the testing areas and objectives proposed by the Fellows and the data from the responses on the survey. When the final analysis was completed by LaserGrade, the Test Blueprint Document was validated with almost no revision. The NBFE's first Domain Analysis for the Personal Fitness Trainer Examination was complete.
Following the completion of the domain analysis, the Fellows assembled once again. This time, the group met just outside of Baltimore with NBFE staff and LaserGrade's top psychometric experts to write the test items (questions) for the first examination. The group began its work by learning how to write strong and psychometrically sound test items. The Fellows then split into small groups based on their specific content expertise to collectively write the 360 items required for the two forms of the first examination. One critical area of item authoring was the use of the NBFE Bibliography. Assembled by NBFE Board of Trustees and Fellows, this small library of authoritative texts was brought onsite for use by the Fellows in item writing. Each item required a specific citation from at least one of the materials.
Once the initial drafts of all items were completed, the Fellows divided into two groups for the first review. Each group evaluated half of the questions initially and then switched the item bank to evaluate the other half. In this manner, each question was evaluated twice by a total of 11 other subject-matter experts. The evaluations were based on three criteria: (1) Congruence: does the item test the knowledge, skill or ability as defined by the testing objective, (2) Difficulty: does the item reflect an examination of the proper degree of mastery of knowledge, skill or ability, and (3) Relevance: does the item represent a pertinent and applicable assessment of the testing objective. Upon completion of this exercise, the Fellows had written and evaluated all of the questions required for the beta-test and subsequent production test forms.
Following the completion of item database, LaserGrade created a test form with the entire item bank included for external review. The NBFE selected reviewers with extensive experience in auditing and editing test items in medicine and health science according to psychometric standards and structure. These individuals had no previous knowledge of the Personal Fitness Trainer Examination Part 1 and no relationship with the NBFE, its affiliates or LaserGrade. Each item was examined carefully based on the item development criteria. These edits were returned to the NBFE for a review.
Once the external audit was completed, the revised item bank was submitted to LaserGrade for preparation for the development of the beta-test form. LaserGrade generated multiple disks that were distributed to members of the NBFE Board of Examiners for content validation. Unlike the NBFE Fellows who are active personal trainers and have broad subject-matter expertise in all areas of the testing domain, the Board of Examiners have an advanced knowledge-base in specific areas of the domain, e.g., physiology, biomechanics, special populations, exercise programming, etc. Final edits and revisions based on content were made in the Spring of 2005.
In August, the NBFE deployed the Personal Fitness Trainer Examination, Part 1 beta-test. The entire test form was completed by 98 qualified personal fitness trainers, each of which held one or more certification from our affiliates: Aerobics and Fitness Association of America (AFAA), American Fitness Professionals and Associates (AFPA), International Sports Sciences Association (ISSA), Professional Fitness Instructor Training (PFIT), World Instructor Training Schools (WITS), and National Endurance and Sports Trainers Association (NESTA). All beta-tests were be completed at one of the hundreds of LaserGrade Testing Centers in the United States .
Following the beta-test, the psychometricians at LaserGrade conducted a series of statistical analyses to determine which test questions were the most valid, reliable, and fair for each of the testing objectives. In September, the Fellows met one final time to review the performance of the questions and complete exercises to determine the cut score, which is the minimum score a candidate must meet to pass the test. The final production test form is now complete and the NBFE Personal Fitness Trainer Examination Part 1 will be available to qualified testing candidates on October 29, 2005.