AI for virtual fitting: application scenarios and enable technologies
Online shopping has become an indispensable part of our everyday life. The demand has been noticeably accelerated due to the COVID-19 pandemic. It changed our shopping habits to embrace e-commerce more than before.
However, size solution is always an unavoidable topic. Therefore, virtual fitting rooms, also known as online fitting rooms, help us to recreate the dressing room experience digitally.
Customers are able to try on clothes, jewelry, and shoes, even makeup looks, more accurately.
Furthermore, virtual fitting rooms are now accessible to both online and brick-and-mortar retailers of all sizes and industries.
In this article, you will find out:
- What is virtual fitting technology?
- Examples of virtual fitting rooms in both online and regular retail.
- The essential technologies behind virtual fitting.
- The relevant open & commercial datasets.
What is virtual fitting technology?
Virtual fitting rooms, leverage the power of technology to allow shoppers to see the size, style and fit of items without physically touching or buying them.
Accordingly, the virtual fitting market was worth $2.97 billion in 2021 and is forecast to grow to more than $8.5 billion by 2028. 
Examples of virtual fitting rooms in both online and regular retail
Despite it being called Virtual Fitting Rooms, there is nothing about physical rooms, the only need here is screens. Therefore, the technology has wildly used in smartphone apps and brick-and-mortar retailers.
Regular fashion retailers use this technology by putting up large mirrors in-store. The mirror is commonly called Smart Mirror which in fact is a display screen.
It uses VR or AR to allow customers to virtually try on different styles and sizes of clothing, get personalized outfit suggestions, or play with different shades of makeup.
Here is a short video to show you a direct visual expression.
For instance, Ralph Lauren installed virtual mirrors in its in-store fitting rooms.
Same as Swedish fashion retailer H&M Group, which is rolling out a pilot in Cos stores in the US.
Normally, these smart mirrors are used to recognize products brought into the room (e.g. item, size, and color) with the possibility to offer personalized styling recommendations and virtual try-on.
Beauty brand Charlotte Tilbury installed a virtual mirror in its flagship London store. Customers are able to try on one of 10 famous makeup looks on their own faces and receive a list of makeup products from the makeup look they tried via email.
However, there is a more convenient and easier type of virtual fitting room — smartphone Apps.
Some brands have their own Apps.
Luxury retailer Gucci partnered with Snapchat to create its first AR shoes. It created a virtual lens that overlaid a digital version of the shoe onto a shopper’s foot.
A jeans brand called 1822 Denim, used 3DLOOK’s YourFit technology to create an online fitting room experience. With just two uploaded images of a shopper, 1822 Denim would provide personalized size and fit recommendations.
3DLOOK YouFit, this technology is one of those third-party platforms which connect customers and retailers together.
YouFit instantly generates a unique 3D avatar and over 86 points of measure from a quick scan on a shopper’s mobile device.
Then it matches these data to the brand or retailer’s product data and delivers a highly visual fit experience.
There are several similar Apps as well.
Style.me: shoppers upload a customer avatar using their basic body measurements and characteristics. Style.me then provides sizing and style recommendations along with 3D renderings of your apparel products.
YouCam Makeup: YouCam has a Shopify app where merchants can plug in to offer virtual try-ons for cosmetic and beauty products. You can do lipstick, eyeliner, eyeshadow, and blush.
More apps include AstraFit, Zeekit Fitting Room, Zugara, triMirror, Virtusize, WearFits and so on.
Here is an article that help you to know more about virtual fitting for makeup, please click the link below to read more.
The essential technologies behind virtual fitting.
Despite virtual Fitting Rooms looking like magic, the basic technologies are more complicated.
1. Image segmentation
For a fashion picture, the image segmentation technology can distinguish the clothes, pants, and various accessories of the figure in the picture at the pixel level. In the virtual fitting scene, it is often necessary to identify and segment the clothes worn by the experiencer, so as to replace the virtual clothes more accurately.
Mask R-CNN is often used as the Baseline network structure for image semantic segmentation task. Mask R-CNN can be decomposed into Resnet-FPN, RPN, Faster RCNN ,and Mask branch. Resnet-FPN performs convolution operations on the input images and extracts features, and then obtains feature images of different scales through the FPN layer. RPN selects the most suitable feature map for the candidate region. Finally, the network is divided into two branches, one is the traditional Faster RCNN detection and classification branch; In addition, the Mask prediction task is completed for the unique semantic segmentation branch of the Mask network.
2. Cross Modal Retrieval And Generation
Cross-modal technology in Fashion field can be divided into two aspects: generating text descriptions through images, and retrieving relevant clothing pictures from massive data through a descriptive text (cross-modal retrieval). Cross-modal techniques often require massive image text description and large-scale training of paired images. The commonly used framework of this kind of task is: by extracting image side features and text side features and optimizing the spatial distance of related modes in the same subspace, the spatial distance of unrelated modal features is gradually approaching, and the irrelevant modal features are gradually estranged.
3. Image Generation
Image generation can be divided into conditional generation and unconditional generation. Conditional generation provides examples of clothing to generate images of similar styles, represented by pix2PIx models. Random disturbance is used as input to generate more diversified images such as DCGAN and ProGAN.
In most generating tasks, antagonism loss is used as loss function, and reconstruction loss is added in some conditional generating tasks. The realization of anti-loss needs a discriminator and a generator as the medium. The generator generates more realistic pictures to mislead the discriminator, and the discriminator tries to distinguish the real picture and the generated picture. When the two reach Nash equilibrium, the realistic generation effect can be achieved. Since the conditional generation task usually has the reference image generated by the target, it is necessary to calculate the L1 loss between the generated image and the reference image pixels to complete the optimization.
4. 3D reconstruction
As the general example picture is a flat garment photo in 2d, 3d reconstruction technology is needed to re-model the garment in a 3D scene. 3d reconstruction virtual makeover 1 technology can be divided into two aspects: based on traditional graphics algorithm and based on deep learning algorithm.
Brouet et al.  proposed a set of automatic clothing transfer method, which can complete the fitting of clothing between different body types while retaining the clothing style. The method first formalizes the principles used in pattern-grading into a set of geometric constraints, including shape, style, proportion and fit. Then by adjusting the size of the virtual clothing and appropriate deformation to complete the clothing to the target human body transfer. This method ensures that the clothing style does not change after migration. Because the transfer process is carried out in three-dimensional space, the garment model needs to be transformed into two-dimensional pieces for production.
DRAPE proposed a garment deformation method that responds to changes in body shape and posture simultaneously. From a large number of human motion data, they learned the deformation mapping relationship between the garment parameters relative to the human body shape and the rotation of human mesh parts. At the same time, they learned the pose-related garment fold generation model and realized the garment migration between different poses and bodies.
4. The relevant open & commercial datasets
Visiting stores, queuing for fitting rooms and try-on, those days are long gone.
The Pandemic accelerated the development of virtual fitting technology.
Customers’ consumption behavior has completely changed. They are expecting faster, more personal and more convenient shopping experiences.
As the examples we mentioned above, not only big brands but also third-party platforms have infiltrated the market, making virtual fitting rooms accessible to retailers of all sizes.
The huge market of e-commerce, and the ever-changing trend of fashion, it is never too late to start AI projects to embrace the change.
As we all know, data is always the essential element for AI model training. So, we, Maadaa.ai, prepared some open and commercial datasets of the Fashion and eCommerce industries for you.
All these datasets are well-chosen, If you are interested in checking the datasets out, please click the link below for more Info.
- He, Kaiming, et al. “Mask r-cnn.” Proceedings of the IEEE international conference on computer vision. 2017.
- Brouet, R. , et al. “Design preserving garment transfer.” ACM Transactions on Graphics (TOG) — SIGGRAPH 2012 Conference Proceedings (2012).
- GUAN P, REISS L, HIRSHBERG D A, et al.Drape: Dressing any person[J]. ACM Transactions on Graphics (TOG), 2012,31(4):1–10.
- Li, Xirong, et al. “W2vv++ fully deep learning for ad-hoc video search.” Proceedings of the 27th ACM International Conference on Multimedia. 2019.