Learn how to be an AASHTO lab 24/7

"“…if you do everything that the AASHTO accreditation requires you to do, if you execute all of the requirements of [AASHTO re:source], you will have a well-run laboratory."

Florida Department of Transportation

Tim Ruelke, P.E., Director, Office of Materials

55+ years of experience
PSP participants
samples shipped per year
laboratory assessments per year
accredited labs

Proficiency Sample Ratings: Being Average Has Never Been So Good

By Brian Johnson, AASHTO Accreditation Program Manager

Posted: October 2010

So you opened up your email notification to see that the latest AASHTO re:source proficiency sample ratings were just posted, you log into the website to view your ratings (Figure 1), and you see ratings of **, -5, -3, 5,4. You think to yourself, "I know that 4 and 5 are good, but what about the negative numbers? Those are below 3, so they must be bad... and what are the stars for? I doubt they're like the stars that my elementary school teacher used to give me... and what is this repeatability rating?"

Figure 1: A Typical Line of Proficiency Sample Data (Color-Coded)

Calculating Averages and Standard Deviations
The first thing that you should understand is that laboratory ratings are based on the average of the results, although the reported averages are determined only after removing invalid and outlier results. It is important to eliminate them from the rating determination equations so that the ratings are not affected based on what some might consider to be "bad data." We determine a standard deviation for each data set (displayed as "1S" in Figure 1 above) and then begin the process of calculating ratings.

Calculating Z-Scores and Ratings
Each laboratory is rated with two values: a z-score and a lab rating. In statistics, the z-score, also known as the standard score, indicates how many standard deviations a result is from the average. The z-score is determined by the following calculation:

Z-Score   =     (Laboratory Test Result - Average Value)
                      (Standard Deviation)

The laboratory rating calculation is based on the absolute value of the z-score:

If Z-Score <= 1 Then Rating = 5

If Z-Score > 1 And <= 1.5 Then Rating = 4

If Z-Score > 1.5 And <= 2 Then Rating = 3

If Z-Score > 2 And <= 2.5 Then Rating = 2

If Z-Score > 2.5 And <= 3 Then Rating = 1

If Z-Score > 3 Then Rating = 0

Which Way Is Up? 
If you're confused by all of this, check out Figure 2 below for a graphical representation of z-scores and ratings. Here are a few quick points to remember:

  • Low z-scores are good.
  • High ratings are good.
  • A negative sign on a z-score or laboratory rating merely indicates that the laboratory's result was below the average, while a positive z-score or rating indicates that the laboratory's result was above the average.

Simply put, the closer your result is to the average, the better your rating. In the competitive world we live in, being average conjures up words like commonplace, mediocre, or ordinary; but in the world of proficiency testing, being average is the definition of excellence!


Figure 2: The Normal Distribution of AASHTO re:source Proficiency Sample Data

Low Ratings 
Any rating less than a 3 (z-score > 2) is considered a low rating according to the AASHTO Accreditation Program, but don't let that bother you unless you consistently receive low ratings. (See Figure 3 and the section below on Performance Charts.) Yes, low ratings are worth investigating, and you might even uncover an equipment problem or procedural mistake. Sometimes, however, your investigation of low ratings will lead you nowhere, and that's okay. The laws of statistics govern that some laboratories have to get low ratings - every lab will be on the low side of the ratings every once in a while. When an AASHTO-accredited laboratory receives low ratings for a given test, they are required to perform a root cause analysis and implement corrective action. If the laboratory receives low ratings again for that test, it might be a sign that either the corrective action was not effective or that the laboratory did not actually apply any corrective action. Now that you understand the concept of ratings, let's discuss a couple of other items that cause confusion.

The ** Rating 
The ** rating indicates that the test results have been suppressed. Ratings may be suppressed for several reasons, but usually this is an indication of one of three things: 1) The data collected was for informational purposes only and is not a measure of the laboratory's competency, 2) data received is unusual and does not fit a normal distribution, or 3) there were not enough data points to provide an accurate analysis.

Repeatability (Within-Lab)
Ratings Repeatability is an estimate of the variation in results that you might expect if you repeated the same test over and over in your laboratory. The within-lab rating is based on the difference between the two individual lab results, but also any actual differences between the two sample materials.

Performance Charts 
Performance charts provide an easy way to gauge your laboratory's proficiency testing performance over time (see Figure 3). As stated above, too much emphasis should not be placed on an occasional low rating. However, patterns in performance charts should be analyzed carefully, as they are usually good indicators of testing problems. The ideal scenario is to have all points over the center line - results right on the average time after time. Generally speaking, however, points scattered within the bands of +2 and -2 are indicative of good testing performance. Points drifting away from the centerline and points consistently on one side of the centerline are indicative of performance problems.


Figure 3: A Sample Performance Chart

Now What?
I'm glad you asked. You've just learned all that you need to know about the proficiency sample program and how the results are reported.  Now you have to take that knowledge and use it to get the most out of the program.  You'll be reviewing your results, repeatability ratings, performance charts, and taking meaningful corrective actions so that you can score 5's and -5's - and you'll be more excited than ever to be average!

Printer Friendly Version


Leave a comment
  1. Kimberly Swanson | Jan 07, 2021
    K. Chadburn - Not at this time. We are working on improving the report, so thank you for your feedback.
  2. K. Chadburn | Jan 06, 2021
    Any way of viewing color-coded lines (as shown above) for the actual report? It makes it so much easier to quickly view the separate samples.
  3. Kimberly Swanson | Dec 02, 2020

    Ahmad - The repeatability (within-lab) ratings do not factor into "low ratings" for accreditation purposes and corrective action is not required.  Tests ratings are where corrective action is required. 

    You may find the below helpful:

  4. Ahmad Q | Dec 02, 2020

    If the z-score of repeatability (within-lab) is >2 but the z-score for the tests is <2 ,is there a need for a corrective action? 

  5. Kimberly Swanson | Jul 08, 2020
    A. McGovern - no corrective action is not required for low lab-repeatability results.  
  6. A. McGovern | Jul 07, 2020
    Is a corrective action required for low lab-repeatability results?
  7. Kimberly Swanson | Jun 30, 2020
    Ibrahem - Most of the differences between standards are small enough that the results would not be statistically different, so the lab can use either method. If the difference is large enough to cause a change in results, the PSP directions will specify which procedure to follow. For instance, we require that labs reuse the soil in the soil compaction sample even though this isn't allowed by ASTM. 
  8. ibrahem taym | Jun 30, 2020
    I want to ask how you deal with the lab's send to you result in AASHTO Standard and other lab's send to you result in ASTM Standards in the same proficiency sample result Are you give ratings in ASTM without AASHTO? 
  9. harding barry | Aug 09, 2017
    Now you have to take that knowledge and use it to get the most out of the program.  
  10. hardik | Apr 05, 2017
    Nice Post. thanks to share.
  11. Katie gray | Mar 24, 2017
    Thanks for great post! 
  12. William Jack | Mar 17, 2017
    Great Post.
  13. Masum Villah | Jan 29, 2017
    I like this article.
  14. Kimberly Swanson | Nov 07, 2016
    The vertical axis on performance charts represents z-scores.
  15. | Nov 07, 2016

    Is the vertical axis of the performance chart standard deviation or z-score?

  16. Kimberly Swanson | Nov 02, 2016

    Vivian, a negative sign on a z-score or lab rating just tells you that your results were below the average, while a positive z-score or rating indicates that your results were above the average.  They don't factor into what is considered a "low rating."

    Accredited labs will be suspended if it is the second time in a row you received low ratings (±1 and 0) on the same test property. 

    Any rating less than a 3 (z-score > 2) is considered a low rating. When an AASHTO-accredited lab receives low ratings for a given test, they are required to perform a root cause analysis and implement corrective action.  More information on how to respond to low PSP ratings can be found here: http://aashtoresource.org/university/newsletters/newsletters/2016/08/11/i-received-a-low-psp-score-what-next

  17. Vivian Mendez | Nov 01, 2016
    I CAN'T find anywhere in the website where specific information regarding a suspension is explained.  Is a -2 or a 2 rating a "bad" score that can lead to suspension? What is the "threshold value" for a suspension?  Does a NEGATIVE rating have the same consequences as a POSITIVE one?

    Leave a comment