The Manage SRRs module provides the capability to create and/or update Statistical Retrieval Rules (SRRs) for your Program. PAR staff are responsible for working with AD staff to determine the optimal presentation of statistics for their program, and then creating and updating SRRs.
To navigate to the Manage SRRs module, select the SRRs tile on the Landing Page or from the Main Navigation select Data and SRRs, and then select the Manage tab. This tab is permission based and will only appear if you have a PAR Admin or Super Admin role.
Create an SRR
1. Select the Add SRR button
2. Enter in a Name (unique within the Program) and Description.
3. Select the appropriate options for the following fields:
- Assembly: Test specifications include both content and statistical specifications. The statistical specifications typically ask that the mean of some specified item statistical property (e.g. difficulty, discrimination, latency, etc.) be set as close as possible to some specified target value. For most tests, there will be just one SRR for test assembly. When getting test assembly statistics for equating items, the AD staff will supply an administration and/or form at the time they do the search. When getting test assembly statistics for non-equating items, they will not limit the SRR search. In rare cases where the list of statistics names to be populated varies by measure, then there may be a test assembly SRR for each measure. For example, if IRT work is not performed for the writing measure, but is performed for other measures, a separate writing test assembly SRR could be defined which does not return any IRT statistics.
- History: Display all instances of statistics (i.e. informational and test assembly and for item review). This SRR will typically be used only when looking at all instances of statistics for a single item. Since each instance will list the values of all tags, the history SRR should have no tags set allowing the user to see statistics instances regardless of tags. For most tests, there will be just one SRR for history requesting all statistics names (from all classes of analysis IA, DIF, IRT). However, if the PAR or AD staff want to view the history for one set of statistics names (e.g. IA and DIF) and at other times view the history for a different set of statistics names (e.g. IA and IRT), then PAR and AD should create several different SRRs tailored to the appropriate statistics names and selection rules.
- Internal: This category should be used if the SRR needs to be used internally in the IBIS® Application. For example, when pulling EIRC data.
- Inventory: On a periodic basis, an inventory of available items (and statistics) may be produced to guide AD of future item development. This might be used in a Test Analysis or other report. An inventory may require a more exhaustive set of statistical values (difficulty, discrimination, etc.) than are used for the Test Assembly purpose, for example, this may include response-level statistics like %choosing each option. An inventory may be exported as a spreadsheet containing items (rows) and item meta data including statistics (columns). When there are multiple inventory reports that contain different statistics names and/or select item statistics with different tags, you may opt to setup several different inventory SRRs.
- Post Admin Item Review: This is designed to detect problems with scoring keys and to flag items exhibiting questionable statistical characteristics. For PIA, it is done prior to equating and is generally limited to operational and external equating items. For FIA, this repeats the evaluation of individual items on the full population for an administration to determine whether there are individual items that should be excluded from use in future test assemblies.
- Print: This is used from Programs (e.g., AP, GRE Subject TOEIC, etc.) that use the print functionality in IBIS. Having SRRs configured as Print is an easy way to find the SRR for this functionality. Most Programs use other category SRRs (e.g., Assembly) for IBIS print functionality.
• SRR Use - Check off 'SRR designated for internal PARcore use only', if you do not want the SRR to be available externally to the IBIS Application. If left unchecked, it will be available to the IBIS Application.
• Context Resolution - When statistics from more than one type of analysis (IA, DIF, IRT, IA by Category) are retrieved by a single SRR, only one set of context information (program, administration, test, form, etc.) is displayed. Choose which of the following contexts should be displayed:
- Item Analysis
- DIF Analysis
• Sorting - When an SRR is used in a context like test assembly or item review, only one instance of IA, DIF, and IRT statistics is to be used. Under many circumstances, context tags may not be enough to uniquely identify a set of statistics for retrieval. For example, an item on a form that has been administered on several occasions will have several sets of statistics, each tagged as “Test Assembly” and “FIA”. When assembling a new form, the question is which of these sets of statistics should appear? A resolution rule is a tie-breaker, designed to select among sets of statistics that are indistinguishable from their context tags. A common rule will be to select the most recently created set of statistics from the most recent administration which fit the context criteria. In the IBIS Application, “most recent” can be based on any of the dates that appear in the Informational fields. These include administration date as well as the date the statistics were created.
- Admin Date: The date assigned to the administration that supplied the data used to create the statistics.
- Request Scheduled Date: The date the analysis request that produced the statistics was run.
• Tags/Category - Define the minimum set of tags which identify the statistics to be used in this SRR. This indicates which instances of statistics should be returned by the query. This can include both tags and informational field values that were applied to the statistics when they were imported/loaded. Only instances of statistics which match on tags and informational fields will be returned by the query. Most SRRs are expected to make use of the data quality and usage tags described in Tagging Statistics (even though additional descriptive tags may have also been assigned to statistics of interest). For example, some test assembly statistics may come from PIA analyses and others from FIA analyses, but as long as both sets have the test assembly tag set, then the test assembly selection criteria need only include the test assembly tag. Some specific examples will be provided below as to how tags and corresponding SRRs can be configured to support use of statistics with particular operational tasks. For example, an SRR designed to return statistics suitable for preliminary item review might specify that the Usage tag Item Review and the Data Quality tag PIA each be selected.
4. Once you have selected the statistics, select the Next button.
5. Select which order to display the selected statistics. This applies to both the SRR run output file in PARcore and the order the statistics will display in the IBIS Application.
- Statistics can be reordered by dragging and dropping the rows to the selected order. This can be done by either selecting the drag and drop handle or selecting anywhere within the row.
- Statistics can be deleted from the list by selecting the trash can button.
6. When you have finished ordering the statistics list, select the Save button to save the SRR. If you need to go back and make changes, select the Back button (you will receive a warning message that the order of the statistics will not be saved). Once the SRR has been saved it will appear in the Manage SRRs table as Draft.
- The SRR will need to be reviewed and approved before it will become available on the Run tab and in the IBIS Application.
SRRs View, Details, Actions
In the Manage SRRs table view you will see a list of all SRRs for your program (sorted alphabetically by Name), view the details of each SRR, and perform actions (Disable/Enable and Manage Versions) on the SRR.
Note:
- The initial view is pre-filtered to the program selected in the Global Navigation. Clearing the program filter will display the SRRs that are configured for All programs.
- All program SRRs are available so that if the SRR has all the details that your program will use, you can easily duplicate the SRR to your program. If you do not have access to the program, all actions will be disabled except for duplicate.
SRR Details
To view the details and ordered list of statistics for a saved SRR, or Approve an SRR in Draft Status, select the Name link.
1. To Approve an SRR, select the Approve button at the bottom of the page. Once the SRR is approved and set as the default, it will become available on the Run SRR tab, and in the IBIS Application (once the data refresh is done daily at 3am).
Enabled/Disabled
When an SRR is saved it is automatically Enabled. If you no longer want an SRR to be used and have it removed from the SRR Run tab and in the IBIS Application, you can change if from Enabled to Disabled. You can also Enable a Disabled SRR which will make it available on the SRR Run tab and in the IBIS Application. You can do this by selecting the ...More button in the Actions column, and either choose the Disable or Enable button.
Disable
Enable
Manage Versions
To view the SRR History, which includes Version details and actions (Edit, Save As New Version, Delete, Duplicate, Run, Set As Default), and Audit details, select the ...More button in the Actions column and choose the Manage Versions button. This will default to the Versions tab. (Note this button will be disabled for a Disabled SRR.)
1. Prior to approving an SRR, you may choose to Run the SRR version to ensure all the settings and output file are correct. You can do this within Manage Versions by selecting the ...More button and Run for the SRR version that you want to Run. For more details on running an SRR, see Run SRRs.
2. You can Edit an SRR Name, Description, or SRR Use by selecting the Edit button next to the SRR History - SRR Name. This can be done on both a Draft and Approved SRR.
Draft Version Actions
Approved Version Actions
3. The SRR version history can be exported to an Excel or CSV file by selecting Export List.
Edit an SRR
Edit is only available for SRR versions that are in a Draft Status.
1. Select the Edit button in the Actions column for the SRR version that you want to edit. The following fields are editable:
• Context Resolution
• Sorting
• Tags
• Statistics List
2. Select the Next button to edit the list and order of the statistics.
3. Select the Save button to save your changes. If you need to go back and make changes, select the Back button.
Save As New Version
Save as a new version is only available for SRR versions that are in an Approved Status.
1. To update an Approved SRR, select the Save As New Ver. button in the Actions column for the SRR version that you want to save as a new version. The following fields are editable:
• Context Resolution
• Sorting
• Tags
• Statistics List
2. Select the Next button to edit the list and order of the statistics.
3. Select the Save button to save your changes. If you need to go back and make changes, select the Back button.
Duplicate an SRR
1. To duplicate and create a new SRR, select the ...More button in the Actions column and Duplicate for the SRR version that you want to duplicate to the program that is selected in the Global Navigation. All fields are pre-populated with the duplicated SRR details and are editable. The Name fields will need to be edited to be a unique name within the program.
2. Select the Next button to edit the order of the statistics.
3. Select the Save button to save your changes. If you need to go back and make changes, select the Back button.
Delete an SRR
Delete is only available for SRR versions that are in a Draft Status.
1. Select the ...More button in the Actions column and Delete for the SRR version that you want to delete. A confirmation message will pop-up. Select Yes to delete or No to cancel this action. The SRR version will be removed from the SRR Manage table once deleted.
Set a Default SRR
Setting an SRR version as default will make the SRR available on the SRR Run tab and in the IBIS Application. Only an SRR version that is in an Approved Status can be set as the default. An SRR with only one version will automatically be set as the default once it is Approved.
1. Select the ...More button in the Actions column and Set As Default for the SRR version that you want to be made the default. The Default column will have a 'Yes' for the default version.
Audit Details
To view the history of all user actions performed on an SRR, select the Audit tab. The Audit details can be exported as an Excel or CSV file by selecting Export List.
Compare SRRs
Users can select two SRRs for comparison from the Manage tab or select two SRR versions for comparison in Manage Versions (on the Version tab).
Manage tab
Manage Versions
Upon selecting the Compare SRRs button the user is presented with a table of the details for each SRR with differences highlighted. The comparison can also be exported as an Excel or CSV file.
Related Articles
Run SRRs
Statistic Sets
Tagging Statistics