As good as a human examiner - or even better

Artificial intelligence is becoming increasingly common in industrial settings and makes it possible to automate even more demanding tasks. Companies benefit from this through significant efficiency gains in various areas. Machine vision is particularly well suited to this field, as large quantities of images are quickly available as training data in production environments.

Pharmaceutical company Aspen, headquartered in Durban, South Africa, also saw an opportunity to benefit from the advantages of machine vision in combination with deep learning. The company, which operates in the healthcare and pharmaceutical industry, also has sites in Europe, including Notre-Dame-de-Bondeville in France. There, the company weighs and mixes the components of the drug formula and fills them into ampoules in a subsequent process.

"Our goal was to automate the inspection of ampoules for possible foreign particles. Quality assurance of pharmaceutical products is extremely important. Therefore, it was essential that the new solution matched the detection rates of the previous process, which involved inspection by human operators, or ideally even surpassed them," explains Mickael Denis, Manager Operationnel Vision Industrielle.

Vincent Trombetta, Automatic Visual Inspection Expert at Aspen, continues: "It was clear that such a task could only be automated using deep learning technologies. For the implementation, which required a great deal of expertise, we relied on the consulting services of MVTec Software. Since the machine vision solution on the inspection machine had already been implemented with MVTec HALCON, it made sense to use the services directly from the manufacturer."

Special challenges in the pharmaceutical industry

At the Notre-Dame-de-Bondeville site, where the corresponding plant is located, the plastic ampoules are first produced, in this case blown, then filled, and finally sealed. Since all these activities are carried out in one machine, this process is very hygienic. Foreign bodies can hardly get into the ampoules. Nevertheless, this process takes place under clean room conditions to further minimize the risk of contamination. Due to the activities involved, this process is also referred to as BFS. The acronym stands for "blow, fill, and seal".

Once the ampoules have been filled and sealed, they are transported to an inspection and packaging area. The ampoules and their contents are then checked for defects. Previously, this process involved employees picking up each ampoule individually and checking it from all sides to see if the ampoule and the fill level were OK and if there were any foreign objects in the liquid.

The big challenge here is that the contents of the ampoules may contain bubbles which are very difficult to distinguish from foreign objects. The particles floating in the vials are not always easy to detect, even for inspectors. They may be located on the side, sink to the bottom, or be unclear due to the viscosity of the liquid. It is therefore understandable that manual inspection is very time-consuming and costly. For this reason, the new process should be automated.

Mickael Denis explains: "Since the inspection has to be done visually, it was clear that we could only implement the process with machine vision and no other technology. We also had to adapt to the particularly demanding validation processes that apply in the pharmaceutical industry. This ensures that the new system tests the ampoules at the required speed, but at the same time with the highest accuracy and robustness."

Inspection of ampoules with 14 images

A total of 12 cameras that comply with the industry standard GigE Vision are used in the new system. Good lighting is also important to make the particles clearly visible. Machine Vision is performed on an industrial PC. On the software side, the machine vision solution is based on MVTec HALCON, the standard software for machine vision with over 2,100 operators for almost all image processing tasks including deep learning.

Due to the extremely difficult conditions in Aspen's application, deep learning had clear advantages over classic rule-based methods. With classic machine vision methods, it was not possible to find a set of rules that was robust and flexible enough to detect the defects.

In practice, the process now looks like this: the test vials are manually placed on the conveyor belt of the system and thus reach the inspection machine. The 12 cameras are positioned so that they capture a total of up to 14 images of each ampoule from different stations. The large number of images is helpful for deep learning-based inspection, as there are images in which the particles are not visible but can be seen when viewed from a different angle.

It should also be noted that a particle is only identified as such if it is found in a certain number of images. This successfully reduces the number of false positives.

Once the images have been captured, they are transmitted to MVTec HALCON. There, various machine vision methods are used to perform the checks. Aspen uses the deep learning-based semantic segmentation included in HALCON to detect foreign matters. In addition to inspecting the liquid for particles, other inspection tasks are also performed in parallel with MVTec HALCON. These include so-called cosmetic defects.

The system checks whether the fill level is correct, whether the colour is appropriate, and whether the closure complies with specifications. Classic machine vision methods such as matching and blob analysis are used for these tasks. Classic methods have the advantage that they deliver very robust results for suitable applications and enable very short processing times. At the end of the inspection, a clear decision is made as to whether the ampoule in question is OK or NOK.

Conceptual consulting and software support

"The task was very complex certainly one of the most challenging we have ever faced. This was particularly true when it came to preparing the images for training. We at MVTec were called in to assist with conceptual preparation, process implementation, and documentation," explains Patrick Ratzinger, Project Manager at MVTec.

Deep learning is only able to make robust decisions if the images are prepared appropriately. MVTec's support consisted of sorting, post-processing, and recompiling the data previously labeled by Aspen, and then training it multiple times. The same data set was trained multiple times in order to compare the different results.

MVTec thus created a neural network that Aspen can use to perform inspections on its own equipment. This network is trained to segment the particles from the background and thus reliably detect them.

For the training, test ampoules were manipulated to simulate possible defects that could occur. Images of these test ampoules were captured and labelled using the Deep Learning Tool. The Deep Learning Tool is an MVTec software product that allows images to be labelled for deep learning applications. The labelled data was combined with good images, meaning ampoules without any defects, to create a dataset used for training the deep learning networks. The training runs, which are carried out multiple times, are then evaluated to see if they work effectively in real-world use.

"Through our consulting and support work, we have built up extensive knowledge about the data and its use. For example, how best to label defects, what the ideal composition of datasets looks like, and how best to interpret the results. We have passed on this knowledge to Aspen as part of our consulting services," explains Patrick Ratzinger.

Robust detection rates

Both production lines are now running. "Our goal was to develop an application that reflects the current state of machine vision technology. It was clear that we wanted to use deep learning, also to expand our internal knowledge. With the support of our colleagues at MVTec, we have succeeded in significantly increasing the error detection rate and reducing false negative results," explains Vincent Trombetta.

The company plans to implement further automation based on machine vision in the future.