For general instructions, navigating, and actions related to Analysis Templates see Analysis Templates.

The following are details specific to Analysis Type: Item Analysis - Classical. 

When you are creating, editing, or duplicating an analysis template for Item Analysis - Classical, the following sub-menus and options are available:  

Processing Options
  • Combine Item Responses
    • single select option to configure if responses should be combined by accession # across forms/AUs or analyze the response within the specific form/AU
  • Edit Equivalent Values
    • single select option to edit equivalent values by type or value, if required
  • Max Popular Wrong Responses
    • single selection option to configure the maximum number of wrong responses for calculation and graphing purposes
  • Treatment of Omitted Selected Responses
    • toggle to exclude or include omitted Selected Responses.
  • Treatment of Omitted Constructed Responses
    • toggle to exclude or include omitted Constructed Responses.
  • Treatment of Not Reached Selected Responses
    • toggle to exclude or include not reached Selected Responses.
  • Treatment of Not Reached Constructed Responses
    • toggle to exclude or include not reached Constructed Responses
  • Automated Constructed Response Scoring Contributory Score
    • single select option for treatment of automated constructed response scoring contributory score by category weight or precise item score.

The image below shows all default selections for Processing Options:

Image 1


Correlations & Reliabilities 
  • Correlation(s) to Compute
    • multi-select option for correlations to compute
  • Correlation to Use for Flagging
    • single select option driven by configured correlations to compute
  • Correlation Calculation: Remove Item from Criterion Score
    • toggle to remove item from criterion score when calculating the correlation. Option is intended to be used when criterion score is a scaled score.
  • Produce Reliabilities
    • single select option 
    • If 'No' is selected, no further options are displayed
    • If 'Linear Form' is selected:
      • Reliability Calculation Options
        • single select option to configure the treatment of omits and not reached for reliability calculation purposes
      • Produce Reliabilities by
        • multi-select option 
    • If 'ISD Form' is selected:
      • Reliability Calculation Options
        • single select option to configure the treatment of omits and not reached for reliability calculation purposes.
      • Produce Reliabilities by
        • single select option 
      • ISD Testlets Identified by
        • manual entry text boxes to indicate the start position and length of the Testlet Type in AU Test Measure Type. 

Produce Reliabilities for a Linear Form:

Image 2


Produce Reliabilities for an ISD Form:

Image 3



EIRC & High Ability Group Options 
  • Produce Smoothed Empirical Item Response Curves
    • If toggle is set to 'No', no further options are displayed
    • If toggle is set to 'Yes':
      • Confidence Shading
        • option to configure confidence band around the correct responses in the EIRC curve.
      • Additional Partial Credit Item Curves
        • if toggle is set to 'No', no further options are displayed
        • if toggle is set to 'Yes':
          • multi-select option to include optional graphs other than Smoothed % Choosing for partial credit items
  • High Ability Group Selection Method
    • If 'None' is selected, no further options are displayed
    • If 'Constant N%' is selected:
      • manual entry text box to indicate the top % to define as the high ability group
    • if 'Volume Based N%' is selected:
      • table with manual entry text boxes where user can configure different percentages to define the high ability group within specific volume ranges
        • '+ Add Range' adds more rows to the table (max of 10) 
        • Note: N's must be continuous and non-overlapping.

Example with Constant N% for High Ability Group Selection Method:

Image 4


Volume Based N% for High Ability Group Selection Method:

Image 5


Validation message for Ns:

Image 5-2



Flag Items
If toggle is set to 'No', no further options display.
If toggle is set to 'Yes', all flags are selected by default and pre-populated with default values. 
 
List of flags available (see below list for details on Group By options):
  • ​​​v - Low N
    • manual entry text box  (value must be > 0)
    • apply flag if sample size does not meet threshold
  • r - Low Correlation With Criterion
    • manual entry text box (valid values 0.0 - 1.0)
    • apply flag if result is less than entered value
    • Group by Item Scoring or Response
  • A - Low Average Item Score
    • manual entry text box (valid values 0.0 - 1.0)
    • apply flag if result is less than entered value
    • Group by Item Scoring, Response or Number of Choices
  • J - Low PPlus
    • manual entry text box (valid values 0.0 - 1.0)
    • apply flag if result is less than entered value
    • Group by Item Scoring, Response or Number of Choices

Image 6
 
  • H - High Average Item Score
    • manual entry text box (valid values 0.0 - 1.0)
    • apply flag if result is greater than entered value
    • Group by Item Scoring or Response
  • Q - High PPlus
    • manual entry text box (valid values 0.0 - 1.0)
    • apply flag if result is greater than entered value
    • Group by Item Scoring or Response
  • L - Low Percentage At Score Level
    • manual entry text box (valid values 1 - 99)
    • apply flag if result value is less than % entered
    • Group by Item Scoring or Response

Image 7

 
  • N - High Percent Omits + Not Reached
    • manual entry text box (valid values 1 - 99)
    • apply flag if result value is greater than % entered
    • Group by Item Scoring or Response
  • O - High Percent Omits
    • manual entry text box (valid values 1 - 99)
    • apply flag if result value is greater than % entered
    • Group by Item Scoring or Response
  • S - High Ability Group Choosing Distractor
    • manual entry text box (valid values 1 - 99)
    • apply flag if result value is greater than % entered
    • Group by Item Scoring, Response or Number of Choices


Image 8
 
  • D - High Ability Group Choosing Distractor Over Key
    • Group by Item Scoring, Response or Evaluation
    • One or two options can be selected within each Group by category
    • Option to Include Differences of Distractor and Key proportion that are within a specified % value (valid values 1 - 99)
  • M - Mean Distractor Score Greater Than Key
    • Group by Item Scoring, Response or Evaluation
    • One or two options can be selected within each Group by category
    • Option to Include Differences of Distractor and Key Mean that are within a specified value (value must be > 0)
  • P - Distractor With Positive Correlation
    • manual entry text box (valid values 0.0 - 1.0)
    • apply flag if result is greater than entered value
    • Group by Item Scoring, Response or Evaluation


Image 9


Group By options include:
  • Item Scoring
    • Dichotomous
    • Polytomous
  • Response
    • Selected
    • Free
  • Evaluation
    • Subjective
    • Objective
  • Number of Choices 
    • 'Choices <='
    • 'Choices ='
    • 'Choices >='
    • 'Choices N/A'
      • Number of choices need to be continuous and non-overlapping

For a list of item types included in each classification see Flags and Automated Comments - Psychometric Analysis and Research Statistical Analysis Engine - Confluence (ets.org)



Delta Options
  • Compare Deltas
    • If toggle is set to 'No', no further options display.
    • If toggle is set to 'Yes':
      • Option to select 'Equate Item Statistics' in Test Settings is removed.
  • When using a template with Compare Deltas to generate a request, a warning will display in Request Details prompting the user to import a file with Reference Delta values to use in the comparison.  See Request Details for more information.

Image 10


Statistics Tags
Once all required selections in the Analysis Template tab have been entered, the Statistics Tags tab becomes available. 
  • Item Analysis - Classical templates have item-level tags and test-level tags available to be configured.
  • Tags selected in the analysis template are the default tags used in an analysis request using that template.  Tag overrides can be performed for the following:

Image 11



Viewing an Item Analysis - Classical analysis template displays the options as seen below:

Processing Options

Image 12

Correlations & Reliabilities 

Image 13
Image 14


EIRC & High Ability Group Options 

Image 15


Flag Options

Image 17
Image 18
Image 19


Delta Options  

Image 20

In the Statistics Tags tab:
  • Check marks show default item-level and test-level tag selections configured during template creation.  

Image 21



For Test Settings details see Test Settings for Item Analysis - Classical
For Request Details information see Request Details
For Statistics and Visualizations available for Item Analysis see Visualize Results
For Statistics Tags definitions and uses see Tagging Statistics
 

Copyright 2024 by Educational Testing Service. All rights reserved.
All trademarks are the property of their respective owners.