Raise your game and a glass

Scientific advisor to proficiency testing at Fapas, Mark Sykes, discusses the benefits of proficiency testing and how this applies to wine testing laboratories 

The global wine market is predicted to grow steadily and reach a global consumption of more than 30 billion litres by 2020, according to research organisation, Technavio, in its Global Wine Market 2016-2020 report. Both the growth in consumption and increasingly high value of wine is motivating producers to undertake testing to help ensure the quality and consistency of their products.

This in turn is increasing the role that laboratories play in this process and puts a dependency and focus on the results they provide. However, there are numerous examples of testing laboratories producing varying results after they have analysed the same sample. This can be caused by the implementation of different testing methods, the quality of the equipment used, the testing process or even the working practices of an individual laboratory technician. This problem can occur even when laboratories belonging to the same parent organisation are carrying out the testing.

In one instance, a sample of wine which was being transported was tested by both the exporting laboratory and the import laboratory – both of which produced different results. In these circumstances, whose results would you trust?

Proficiency testing provides a solution to this challenge and helps to generate confidence in the quality of testing procedures within laboratories. Over the last 18 years, Fapas, the proficiency testing arm of Fera Science, has published over 30 papers that have prompted improvements in proficiency testing schemes and laboratory analytical practices.

Why take part?

Proficiency testing provides wine testing laboratories with objective evidence of their ability to produce accurate and precise data by independently testing laboratory procedures. Participating in relevant testing schemes allows a laboratory to demonstrate its competence to managers and clients, and to gain and maintain ISO 17025 accreditation. The ISO 17025 standard covers testing and calibration performed using national standard, non-standard, and laboratory-developed methods, and it applies to all organisations that perform tests and/or calibrations. The standard can apply to all laboratories, regardless of their size or the scope of their testing or calibration activities.

Proficiency testing can also mitigate risks by identifying product failure and non-compliance, which, if they weren’t highlighted could lead to legal issues and damage brand reputation. In addition, the results provided by proficiency testing schemes draw attention to laboratories’ testing methods – which can then be improved to generate better performance.

The wine proficiency testing programme is relevant to everyone in the wine industry, from independent wineries to large, multi-site laboratories. Laboratories are invited to sign up online to participate and once they are registered each laboratory is sent an aliquot of the same test sample along with instructions describing what the particular proficiency test is looking for and how to submit results.

Generally, participants are given a period of four weeks to carry out their analysis, which could be anything from alcohol determination or heavy metals analysis to nutritional tests.

Once the participating laboratories submit their result, Fapas provides them with assessments based on how close their results were to the assigned value. Each laboratory receives a z-score, which is a common metric that indicates how close or far their result was from the correct result.

The further the result deviates from zero, the less accurate the analysis is; a z-score of zero means the result was absolutely correct, a positive z-score means the result was an overestimate, and a negative z-score means the result was an underestimate.

Generally, if the result is within ±2 z-scores, it is deemed satisfactory and anything outside of ±3 z-scores is unsatisfactory.

Between 2 and 3 z-scores is a grey area because the score may indicate there was a problem with the analysis, or it could simply be a statistical anomaly. The correct interpretation is something for the laboratory to rationalise.

It is recommended that laboratories take part in several proficiency tests throughout the year so they obtain a long-term assessment of their performance. This is because completing one test at a particular moment in time doesn’t necessarily reflect the ongoing performance of a laboratory.

With the wine market scheduled for increased growth, Fapas has worked in partnership with the Italian Wine Federation and Star Ecotronics in Italy, to develop a myriad of new wine proficiency tests. The tests span a range of wines, both red and white, and cover more than 60 parameters.

To the existing basic wine proficiency tests for alcohol content, sulphur dioxide and acidity, Fapas has added tests for alcohol-related congeners, inorganic ions, organic acids, quality indexes including colour, sugars, heavy metals, nutritional elements and oxygen-18 isotope ratio. The new proficiency tests are much wider in scope than anything Fapas has carried out before and aim to provide complete confidence in the products being produced. Therefore, the proficiency tests cover the entire spectrum of analyses for wine quality, authenticity and safety.

Ensuring quality

Forty-one laboratories submitted results in the first testing round, which took place in March-May 2018. The test materials were provided from a wine production line in the kind of container that would be associated with the purchase of wine. This was to ensure the homogeneity of the samples and ensure that all participants were effectively receiving ‘the same’ sample.

The outcome of the first round of proficiency testing has been published in an in-depth 300 page report and showed that key quality parameters were well reported and results demonstrated a high rate of proficiency, with laboratories across the board performing well. This is extremely promising for the critical parameters that are most frequently analysed by laboratories in this sector.

The key parameters of the quality of the wine were well reported and managed successfully by the participants, these were typically the alcoholic strength, some of the congeners such as ethanal, the acidity, the total sugars, a few of the organic acids, the total sulphur dioxide and some of the heavy metals such as Copper, Iron, Lead and Zinc.

What does success look like?

Take the example of pH in one of the wine samples. The distribution of z-scores was centered around zero, with a symmetrical distribution and the majority of participants have results between the +2 and -2 tramlines. These are the satisfactory performance limits, as described above.

We can contrast this with the distributions for volumic mass at the defined temperature of 20°C.

Most of the laboratories were successful in achieving a satisfactory z-score across the three wine samples. There is, however, a small population of laboratories that consistently over-estimated the result. This is evident in the z-score histograms below where there is a very tight distribution around the central zero z-score but a clear grouping to the right of each chart.

[caption id="attachment_68813" align="alignnone" width="620"] Figure 1[/caption]

 

 

[caption id="attachment_68809" align="alignnone" width="620"] Figure 2[/caption]

[caption id="attachment_68810" align="alignnone" width="620"] Figure 3[/caption]

This outcome is consistent to the same group of laboratories over the three different wine samples, highlighting that the problem is not related to issues with the test samples. Instead, there is a strong indication that the methods in practice at these laboratories are causing a systematic bias. At present there is insufficient method information from the proficiency test to troubleshoot the root cause of this observation but this is something for the wine industry to be aware of and advise on best practice.

Method dependency is a key parameter that can be investigated and observed in proficiency test data. To do this effectively, large volumes of data are required to ensure a robust analysis, as small amounts of data are limited in their statistical power. Repeat amounts of data are more beneficial as they allow trend analysis to be performed, rather than a one-off sample taken at a particular time, which could be subject to other outside factors. Also of importance is the method parameter information. Although method submission is not a mandatory part of the proficiency test results submission, it is an extremely useful tool for investigating unusual data distributions. Participants in proficiency tests are encouraged to submit method information for this purpose.

Making changes

Knowledge of the methods used in the tests will enable recommendations to be made on improving testing practices. For example, the measurement of alcohol volume can produce significantly different results, depending on whether this is carried out by direct or indirect methods. Knowledge of exactly which method was applied by each laboratory enables more reliable assessments of performance and demonstrates the effect of method dependency.

There is an opportunity here for all stakeholders to learn from the first round of proficiency testing as well as the second round. Fapas has reviewed the scope of the tests used in this second round following feedback from experts in the industry and from laboratories. The proficiency test itself is evolving and improving to enable laboratories to meet the changing and complex demands of their customers – who are looking for a consistent product which meets international standards.

Fapas is inviting laboratories – from those in small, independent wineries to large multi-site operations – to take part in its wine proficiency tests to help create an international standard for wine testing and ensure that high quality, consistent wine products reach consumers.

For further details visit www.fapas.com/wine

Author: Mark Sykes is Scientific advisor to proficiency testing at Fapas