5.2 Assessment process stage 2
Full application assessment
The aim here is to assess quality of the research, ensure the research will deliver the outputs/outcomes planned and that the team is suitable with a demonstrated track record of publishing/producing outputs, and to determine whether the project is logistically feasible.
Reviewing
Full proposals will be sent to at least two reviewers for projects in Themes 1-3 and for projects proposing monitoring/underpinning science. Where warranted by the complexity of the project or the number of disciplines involved more reviews may be sought.
In the case of Frontier Science projects we will routinely seek at least three independent reviews.
NB: Where relevant expertise is not available on the ARAC to evaluate reviewer comments on the quality and relevance of Frontier Science proposals, additional expertise will be contracted to provide input to ARAC. Routinely an additional review will also be sought for all proposals in Frontier Science.
Criteria for full application assessment
Reviewers will be asked to recommend scores for Criteria 2-6 only.
The final score for all criteria will be assigned by the ARAC. Scores for Criteria 2-6 will be assigned based on the recommendations of the reviewers but moderated by the ARAC. Scores for Criterion 1 (Strategic fit) and Criterion 7 (Outreach) will be assigned by the ARAC.
Criterion 1 Strategic fit
Themes 1-3 [Maximum score = 40]
1a) Will the research proposed provide a significant advance in answering a Key Research Question in the Australian Antarctic Strategic Science Plan? [Max score = 15]
0 Not demonstrated [Score = 0]
1 Research is likely to provide limited relevant information toward answering the key research question(s) selected [Score = 1-5]
2 Research is likely to produce a useful contribution toward answering the key research question(s) selected [Score = 6-10]
3 Research is specifically designed to produce a significant contribution toward answering the key research question(s) selected [Score = 11-15]
1b) Has the applicant demonstrated there is a clearly identified end user for the research, and provided a credible path for adoption of the research within science and end-user communities? [Max score = 15]
0 Not demonstrated [Score = 0]
1 Research is unlikely to deliver useful outputs/outcomes in a form that is useful to the end user [Score = 1-5]
2 Research is likely to produce useable outputs/outcomes that should be useful for the end user [Score = 6-10]
3 Research is specifically designed to produce the required outputs/outcomes in a form appropriate for the end user. [A letter of end user support should be included with the application if the project is submitted as a “tactical’ or “strategic” project.] [Score = 11-15]
1c) Is this research identified as a priority in the Implementation Plan for this Stream? [Max score = 10]
0 Research planned is not considered relevant [Score = 0]
1 Research planned is identified in the Implementation Plan, but it is likely to only make a partial contribution [Score = 1-4]
2 Research planned is identified in the Implementation Plan, and it is likely to make a significant contribution [Score = 5-10]
3 The research approach is not identified in the Implementation Plan, but the approach proposed is considered better than that given in the implementation plan, and is likely to make a significant contribution [Score = 5-10]
Theme 4 (Frontier Science)
Has the applicant adequately demonstrated that the research will address a National Research Priority? Yes or No
[This is a threshold question – only those that address an NRP will proceed to be scored]
Monitoring and underpinning science [Maximum score = 40]
Threshold questions - a yes to at least one is required to proceed to be scored
a) Is the need for the monitoring well justified? [Y/N]
b) Is the data required for Australian Government responsibilities or an international agreement that requires access to an Antarctic/Southern Ocean monitoring location? [Y/N]
or
c) Is the need for this data included in the Implementation Plans for Theme 1, 2 or 3? [Y/N] If not does it replace an existing monitoring program and is it a better way to do it? [Y/N]
If yes to any of these threshold questions, will the monitoring proposed produce the required outputs/outcomes and is there an appropriate plan for archiving and analysing the data?
0 Monitoring/underpinning data that would be collected are unlikely to deliver useful outputs/outcomes [Score = 0-5]
1 Monitoring/underpinning data that would be collected are Likely to produce useable outputs/outcomes [Score = 6-20]
2 Monitoring/underpinning data designed specifically to produce outputs/outcomes at a high standard and in a readily available form [Score = 21-40]
Criteria 2-7 apply to Themes 1-3 and Monitoring/underpinning science. Frontier science will use the same criteria but will have different scores shown in italics - a summary of scoring is available.
Criterion 2 Scientific context
[Maximum score = 10 for Theme 1-3 and Monitoring; maximum score = 20 for Theme 4]
Has the applicant demonstrated sufficient knowledge of the context for this research?
0 Not demonstrated
1 Partial understanding of the scientific context for this research is demonstrated [Score = 1 / 2]
2 Good understanding of the scientific context for this research is demonstrated [Score = 2-5 / 4-10]
3 Very good understanding of the scientific context for this research is demonstrated [Score 6-10 / 11-20]
Criterion 3 Method and analysis
[Maximum score = 20 for Themes 1-3 and Monitoring; maximum score = 40 for Theme 4]
Are the methods proposed for data collection (including field work), analysis of samples and statistical analysis planned designed to appropriately meet the objectives of the project?
0 Not demonstrated [Score = 0]
1 Poorly designed collection of data, analysis of samples or statistical analysis [Score = 1-5 / 2-10]
2 Generally well designed collection of data, analysis of samples and statistical analysis [Score = 6-14 / 11-28]
3 All aspects of data collection, analysis of samples and statistical analysis are very well designed [Score = 15-20 / 29-40]
Criterion 4 Track record of team
[Maximum score = 10 for Themes 1-3 and Monitoring; maximum score = 15 for Theme 4]
Do the Project Leader and the Project Team have the skills and track record to deliver the outputs/outcomes of the project within the timeframe requested?
0 Not demonstrated [Score = 0]
1 Project leader and team have demonstrated limited capability to lead and deliver results of this project [Score = 1 / 1-4]
2 Project Leader and Team have a reasonable track record and have shown they can deliver projects relevant to this proposal on time [Score = 2-5 / 5-10]
3 Project Leader and team have demonstrated a very good track record and ability to deliver high quality results on time in projects relevant to this proposal [Score = 6-10 / 11-15]
Criterion 5 Budget
[Maximum score = 10 for Themes 1-3 and Monitoring; maximum score = 15 for Theme 4]
Is the budget appropriate to complete the project and does it represent good value for the investment required by the AAD?
0 Not demonstrated [Score = 0]
1 Budget partially appropriate [Score = 1 / 1-4]
2 Budget appropriate, and justified. Co-investment by CI and/or collaborators minimal. [Score = 2-5 / 5- 10]
3 Budget well planned, and justified. Significant co-investment demonstrated by either the CI or collaborators. [Score 6-10 / 11-15]
Criterion 6 Planning and timeline
[Maximum score 5 for all projects]
Is the project well planned, with roles and milestones clearly identified and the timeline appropriate?
0 Not demonstrated [Score = 0]
1 Planning and feasibility appropriate for some aspects but not all [Score = 1]
2 Reasonable planning and timeline outlined [Score = 2-3]
3 Project very well planned with clear roles and milestones and timeline well considered [Score = 4-5]
Criterion 7 Outreach and education
[Maximum score 5 for all projects]
*To be considered by the Assessment Committee only - not the reviewers*
Does the project have an appropriate outreach and education plan?
0 Not demonstrated [Score = 0]
1 Partially appropriate [Score = 1]
2 Good outreach/education planned [Score = 2-3]
3 High quality outreach and/or education planned [Score = 4-5]
