Learn how to be an AASHTO lab 24/7

"“…if you do everything that the AASHTO accreditation requires you to do, if you execute all of the requirements of [AASHTO re:source], you will have a well-run laboratory."

Florida Department of Transportation

Tim Ruelke, P.E., Director, Office of Materials

55+ years of experience
PSP participants
samples shipped per year
laboratory assessments per year
accredited labs

Debunking Myths about the AASHTO re:source Proficiency Sample Program

By Tracy Barnhart, Quality Manager

Posted: April 2015


First, a brief history lesson...
The AASHTO re:source Proficiency Sample Program (PSP) was created in March 1966 as a mechanism to evaluate testing competency by comparing the results of each participating laboratory to a large body of laboratories that are testing the same material. The program also provides laboratories with the means to check both the testing apparatus and the operator under actual testing conditions. Originally there were only four basic material types in our program. The program has grown extensively over the years and we now distribute fifteen different sample types each year. For some easy reading about other interesting PSP facts and figures, check out these articles:

Let the Debunking Begin!

The sample materials aren't real.

Fine_AggThe materials are as real as possible and are commercially available from uniform sources. Obviously, some materials like paint and asphalt binder aren't typically found in nature, but everything else we use is – rock, sand, and soil. You may occasionally find some odd particles in your aggregate or soil samples which are actually natural constituents of the material. We do not manufacture any materials that are used in our samples. However, we sometimes blend different materials together to create the samples before they are shipped to you. More on that later.

The sample materials are from Mars.

Many of the materials that we supply are actually from sources in our general geographical region (Maryland), mainly due to costs associated with shipping large quantities of materials to our facility. But, to keep things interesting, we have also used materials from California, Florida, Georgia, Iowa, New Jersey, Ohio, Pennsylvania, Rhode Island, Tennessee, Texas, etc. Oh, and Lowe’s, but not Mars (yet). Therefore, the materials may not be typical of your region, but we do try to provide materials covering a range of test values. Speaking of…

I received low ratings because the type of material you sent is not found in my geographical area, and I’m not familiar with testing it. That’s not fair.

coarse_aggOne of the objectives of the proficiency sample program is for laboratories to demonstrate their testing proficiency on a range of materials. In order to achieve that goal, it is necessary for us to switch up the materials year after year. For instance, some soils may have a higher plasticity than what a laboratory is normally used to testing, and it may require more time and effort to perform the testing. This allows a laboratory to demonstrate the range of their testing skills.

The samples in each pair are always identical (or they are supposed to be).

This is probably the #1 misconception about our samples, and it’s a great reason not to compare (or copy) the results from sample A to sample B, or to combine samples A and B before testing. There is no need to panic when you notice a visual difference between the samples, or a notable difference in the test results. The AASHTO re:source PSP is based on the testing of two samples of similar material having nearly identical, but not necessarily exact, test properties. We often adjust the properties of one sample in the pair by adding clay or sand to the soil, distilled water to the paint, etc. Also, the amounts of each component (i.e., aggregate size fractions, mineral filler, etc.) in the samples can intentionally be quite different between the samples in the pair. These adjustments are initially unknown to participating laboratories, but are usually apparent in the final data analysis. Each sample in the pair is evaluated separately, so the use of very different materials has no effect on the ratings issued to laboratories. That being said, we do everything we can to ensure that every sample A (or B) of the pair is as identical to every other sample A (or B) as possible, which leads us to…

There is no way that my samples are identical to everyone else’s.

I received low ratings and I did the testing right, so my samples must have been bad.

These myths go hand in hand. While it isn’t humanly possible to guarantee that everyone’s samples are 100% identical, rest assured that we take great care during the production process to ensure the uniformity of our samples. More information about how samples are processed can be reviewed in this article: Not Just a Box of Rocks: An Introduction to the AASHTO re:source Proficiency Sample Program.

Many quality control processes are in place during the production of our proficiency samples, from start to finish. Here are just a few:

  • The mass of each component in each sample, as well as the total mass of each sample, is determined before samples are packaged for delivery.
  • For proficiency samples that include coarse aggregate, the bulk aggregate we receive is carefully sieved into individual size fractions rather than repeatedly splitting the bulk material until the appropriate amount is obtained for each sample. Each size fraction is then placed in separate bins, and we carefully weigh each fraction for each individual sample during production.
  • We perform regular calibration, standardization and maintenance on all production equipment, and keep detailed records of these activities. Some of these checks are performed daily.
  • We carefully select our material suppliers and keep detailed sample preparation logs. Our records include specific information regarding the material source, preparation, blending, packaging, shipping, etc.
  • The data analysis is carefully reviewed before final proficiency sample reports are released to customers.
  • The three main programs of AASHTO re:source, including the Proficiency Sample Program, have been certified by DEKRA Certification (an external accrediting body) for ISO 9001 since 2006. We are audited by DEKRA yearly to ensure that our quality management system is in conformance with requirements.

We advise laboratories not to jump too quickly to the conclusion that we sent a “bad” sample when they can’t determine why low ratings were received. While it is certainly possible for a lab to receive a nonconforming sample (and that has happened on occasion), it is highly unlikely for a laboratory to receive two nonconforming samples together (A and B), let alone nonconforming samples year after year. The A and B samples in each pair are prepared and packaged separately, sometimes a week or more apart, and the samples are often shipped separately as well.

PSP-performance-chartWe know it can be very frustrating when you can’t determine why a low rating was received (see I Received a Low PSP Score, What Next?) Keep in mind that it is possible, and quite normal, for a good performing laboratory to occasionally receive low ratings. Things happen – human error, equipment problems, math errors, misinterpretation of the test method or instructions, sample variation, and random testing variation. Plus, the laws of statistics preclude laboratories from receiving 5’s on every test every time, even if the samples are tested “perfectly.” For these reasons, it is important to remember that significance need not be applied to a single low rating, or a pair of low ratings (i.e. z-scores beyond 2; or ratings of ± 2, ± 1, or 0). However, a continuing tendency to get results beyond two standard deviations from the grand average on several pairs of samples is of more importance. Laboratories can use tools such as the performance chart on the final reports to monitor their testing over time.

Mistakes happen, of course, and our system isn’t perfect. Things can slip through the cracks. If you suspect that you may have received a nonconforming sample, especially after considering that samples A and B in the pair aren’t necessarily identical (see Myth #4), please contact us immediately and we will be happy to assist you.

The sample materials are hazardous! 
Why else would
AASHTO re:source need to publish the SDS (Safety Data Sheet) information?

The materials that we use in our program are no more hazardous than any other normal materials that are tested by laboratories. However, some laboratories have safety measures in place that require SDS (formerly MSDS) information for every shipment they receive, in case there are some hazards. That is why we make the SDS available. The purpose of the SDS is to show that the materials are not hazardous. We do not provide soil from chemically or radiologically contaminated sites. Also, shipping companies have restrictions in place for certain materials, including hazardous materials, and we show that we are not shipping hazardous materials by way of the SDS information. We obtain the SDS from the suppliers to assure us that the materials are not hazardous, and then we share the information with participating laboratories.

Extra material isn’t provided in the samples because AASHTO re:source is cheap and lazy. 
Why not give us another chance to perform the tests again so that we can confirm our results before submitting our data?

SoilThe AASHTO re:source Proficiency Sample Program was developed based on requirements found in various ASTM and ISO standards applicable to proficiency testing schemes.  One of the requirements suggested in the standards is that no more material should be supplied than is required to perform the tests. Laboratories should only get one shot, rather than being allowed multiple attempts to get a result that they like, or in case they make mistakes. By providing two samples that are analyzed separately, our program actually does give participants two attempts to achieve satisfactory results. Failure to achieve satisfactory results on one or both of the samples could be a reflection of the laboratory’s ability to perform the testing properly.

AASHTO re:source’s analysis method assumes that all participating laboratories are perfect.

Actually, quite the contrary. Participating laboratories are generally driven to perform the tests to the best of their ability and in accordance with their best interpretation of the test methods. However, due to the large number of participants in the PSP, it is actually expected that the results submitted to AASHTO re:source for analysis will contain a few results from laboratories whose equipment and/or testing procedures may not have been in conformance with the test standards. That being said, the analysis technique that we use is an attempt to identify and eliminate those test results prior to determining the precision estimates and, ultimately, the ratings. If you are interested in learning more about the analysis technique that AASHTO re:source uses, here are some links to additional information:

The final ratings are (or should be) based on the results from 
one “expert” testing laboratory or on the results from
AASHTO re:source’s testing.

As mentioned at the beginning of this article, participating laboratories are evaluated by comparing their results to the results received from all other participating laboratories. Although there is some merit in using “expert” laboratories to determine the values of physical properties of materials for comparison, that method also has some drawbacks. Overall, we have found that using the “grand average” does a good job of determining the “true” value for comparison. And, there has been research to support our current method. Remember, the ratings you receive are in comparison to all the other laboratories in the world that performed the same tests on the same material. In the cases of the fine and coarse aggregate samples, we are talking about roughly 1,800 participants!

Although we typically perform initial testing on the sample materials to ensure that the materials are suitable for our program, AASHTO re:source’s test results are never incorporated into the final analysis. While it is difficult to determine “true” values for testing performed on a specific material, many excellent laboratories participate in the program, and nearly 50 years of experience has given AASHTO re:source confidence that the grand average provides an appropriate value for comparison.

Laboratories that are not accredited use equipment that isn’t calibrated regularly, which can really throw off the data. The data for accredited laboratories should be analyzed separately.

Almost all of the participants in our proficiency sample program are actually accredited by AASHTO. Many of the non-accredited laboratories are certainly competent but they simply do not need accreditation for their purposes. A previous study conducted by AASHTO re:source indicated that, overall, non-accredited laboratories do not throw off the data. Also, results that are out of the norm are identified as outliers and removed before the analysis is performed. The proficiency sample program data analysis has been designed to reflect the results of the best performing laboratories, and part of that process is the removal of outlying results before statistics are determined.

Many participating laboratories just enter results from past years, or they copy the results from sample A to sample B, instead of actually performing the tests.

Testing results are always the same and laboratories can just get the “answers” from previous sample round data that is published on the AASHTO re:source website.

We believe that laboratories actually do test the materials that are provided each time. Without giving away any of our secrets, I can tell you that we deliberately try to use different material suppliers for each round of testing so that laboratories do not know the results without actually performing the tests. The soil may be a clay, a silt, a sandy clay or a silty clay. The aggregate may be limestone, granite, pea gravel, concrete sand, or golf course sand. The asphalt we use may be a PG 64-22 or 58-28 or 82-22. The choices are endless! We often reuse leftover material from sample to sample as well. Go Green or Go Home!

YoudenThe similarity in results that you may notice over several rounds of testing is likely because the mineral makeup of the some of the materials did not vary to a great degree, rather than laboratories not performing the tests each time. Also, as mentioned earlier (see Myth #4), the A samples are often not identical to the B samples in the pair. If a laboratory were to enter results from past years, or only perform the tests on sample A and use the same results for sample B, they run the risk of receiving poor ratings on one or both of the samples. In the cases where samples A and B were intentionally produced to be dissimilar from each other, it would be evident in the data analysis and Youden diagram if laboratories had only tested one of the samples. When the data is analyzed, we actually look for and investigate anomalies such as a cluster of outlying results on the Youden diagram.

I know I submitted my data but my results were not included on the final report. 
Why am I being punished for
AASHTO re:source’s error?

We certainly make mistakes, and we take full responsibility for them when we do. However, some important things must be considered first. When data is submitted online through our website, the laboratory will receive a confirmation page that the data has actually been transmitted properly. Also, we send out emails one week before each sample’s closing date to laboratories that have not yet submitted data. We can confirm that these emails were sent to each applicable laboratory. It is always a good idea to print or save a copy of the confirmation page if you have concerns about us receiving your data. Also, be sure to check the AASHTO re:source website to confirm that the email contact information for your laboratory is accurate.

I hope this information has helped to clear up some of the more common misconceptions about our proficiency sample program. If you are seeking the truth about anything else that we do at AASHTO re:source, please let me know!

Printer Friendly Version