EyeWash

NOTE: Demo deployed on the cloud might not be available at evaluation time due to extra costs required with maintaining it on our side. Please write us an email to [email protected] if you'd like us to turn on our DigitalOcean machine so you can experiment with the dashboard.

For more detailed documentation and project description, see the README on our repository.

Repository URL: https://github.com/galpenergia/galp-hretail4-eyewash-car/

OVERALL SOLUTION DESCRIPTION

We addressed this challenge using a “divide and conquer” strategy, i.e., we tackled the problem by dividing it into smaller issues. The plan of resolution of the challenge consisted of developing:

• AI-based models - we approached this task from four different points of view:

  • Car Detection Model - assuming that Galp will use conventional cameras to detect the cars of their customers, it is of utmost importance that we have a model that is able to detect the cars we want to address. Therefore, we ended up using a pre-trained model (YOLOv4) to detect and crop the cars from an image.

  • Dirtyness Level Detection Model - after the detection of the car, we have to understand if it is dirty or not, so that we can propose a service of car washing. Hence, we trained a simple convolutional neural network (CNN) named MobileNet_v2 from Pytorch Library with our in-house dirtiness level detection data set.

  • 
    

    • Our model outputs a probability that is used as the dirtiness level of the car (i.e., the higher the probability, the higher the dirtiness level of the model). To follow the training progress of the model we used Weights & Biases tool.

    • One of the most prominent ways of producing a class-specific heatmap in a Convolutional Neural Network (CNN) model is using Gradient-Weighted Class Activation Mapping (Grad-CAM). It uses the gradients of any target concept (e.g., "dirty car" or "clean car") in a classification network to build a coarse localization map highlighting the essential regions in the image for predicting the concept. Grad-CAM requires three things: an input image, a trained Convolutional Neural Network (CNN), and a particular class of interest. Any CNN model with differentiable layers can be used to calculate Grad-CAM. This way,  we also added a GradCAM module to this model to highlight the locations of the image where the model detected dirtiness with higher confidence; this result is converted into an heatmap that can be placed on top of the original input image.
      
      

  • Car Model Detection Model - besides detecting the dirtiness level of the cars, we find it insightful to have a record of the car model and brand. To do this, we trained a ResNet50 deep neural network on the Stanford Cars Dataset. The output of this model is a string with the car brand and model.

  • Decision Process Pipeline - this module wraps up the complete pipeline. It receives an input image, detects all the cars in this image, crops these cars into smaller images, computes the car dirtiness level detection and predicts the car model and brand.

• Application Programming Interface (API) - to facilitate the deployment of our AI-based models, we decided to build an API.
  
  

• Dashboard - to facilitate the use of our AI-based models, we decided to build an interactive dashboard that allows the end-user to test and interact with our algorithms (via the Eyewash API). The dashboard has five different modes/tabs:

  • Main - this tab contains visual examples of a clean and dirty car on different photographs, and video that shows our dirtiness algorithm working in real-time.

  • Car Detection - this tab allows the user to interact with the “Car Detection Model” and visualize the outputs of this module.

  • Dirtiness Level Detection - this tab allows the user to interact with the “Dirtiness Level Detection Model” and visualize the outputs of this module.

  • Car Model Detection - this tab allows the user to interact with the “Car Model Detection Model” and visualize the outputs of this module.

  • Decision Process Pipeline - this tab allows the user to interact with the whole decision process pipeline algorithm.

• MongoDB Database - as previously mentioned, at the end of the decision process endpoint, the overall data resulting from the AI-based models is stored in a single MongoDB collection, thus establishing a Database to the process. The choice fell on this NoSQL database once the overall data is returned on dictionaries from the AI-based models, a data structure similar to a JSON file, which is the base object of this mapped based database. Another point is this database capability of being fully managed in a cloud service.

USING THE SOLUTION

Our solution is wrapped around in a set of Docker containers, orchestrated by a Docker-Compose file.

Follow the "deployment_instructions.md" to build the images locally and test the API and the streamlit dashboard.

If you want to deploy it, we recommend using the Terraform script to generate the DigitalOcean Droplet and use our instructions to copy the code.

REPOSITORY STRUCTURE

Code is available on https://github.com/galpenergia/galp-hretail4-eyewash-car.

The repository has the following structure:

• “app” - this directory contains all the files related to the development of the APIs for our AI-based models;

• “car_model_classifier” - this directory contains all the files related to the “Car Model Detection” AI model, including data processing files, model training files and model inference files;

• “clean_dirty_cars_classifier” - this directory contains all the files related to the “Dirtiness Level Detection” AI model, including data processing files, model training files, model inference files and heatmap (GradCAM) generation files;

• “dashboard” - this directory contains all the development files related to the Streamlit dashboard that we developed for the final demo;

• “detect_cars” - this directory contains all the files related to the “Car Detection” AI model, including data (i.e., images), data processing files, model training files and model inference files;

• “eyewash-terraform” - this directory contains all the Terraform files necessary to build a virtual machine (VM) at the DigitalOcean cloud service with the minimum requirements to deploy our demo application;

• “mongodb” - this directory contains a sub-directory “database” that acts has the storage path for the MongoDB Database we built for this demo application;

• “shared” - this directory works as a shared directory between all the Docker containers that compose our demo application, including the “tmp” sub-directory that allows us to write and read temporary files that may be created during the usage of the dashboard and the APIs.

PILOTS & IMPACT OF THE SOLUTION

This prototype assumes that Galp has an image acquisition infrastructure (e.g., cameras). To test the actionability of this minimum viable product, we would like to run a test-pilot during a year. During the two/three months of this test-pilot, the car drivers would enter a Galp station and a sequence of events would happen:

1. Detection of the car;

2. Detection of the level of dirtiness;

3. Car Washing Service Recommendation:

   1. 30% of probability of getting a random recommendation;

   2. 70% of probability of getting a recommendation based on the dirtiness level:

      1. Between 0 and 0.2 Dirtiness Score: Nothing

      2. Between 0.2 and 0.4 Dirtiness Score: Simple Program

      3. Between 0.4 and 0.6 Dirtiness: Super Program

      4. Between 0.6 and 0.8 Dirtiness: Special Program

      5. Between 0.8 and 1.0 Dirtiness: Extra Program

4. The user accepts, not accepts or selects a different car washing service;

5. The car brand and model is detected;

6. The level of dirtiness, the choice of the user and the car brand and model are saved into the database.

After this first phase, the main idea is to use the information about the choices of the users and their car brand and models to train a car washing recommendation system model based on this information. Afterwards, the sequence of events would be:

1. Detection of the car;

2. Detection of the level of dirtiness;

3. Car Washing Service Recommendation: 30% of probability of getting a random recommendation, 70% of probability of getting a recommendation based on the dirtiness level and previous user’s choices (e.g., “the users with your car brand preferred…”).

4. The user accepts, not accepts or selects a different car washing service;

5. The car brand and model is detected;

6. The level of dirtiness, the choice of the user and the car brand and model are saved into the database;

7. The recommendation system model is updated.

METRICS AND RESULTS

In a few weeks, the team was able to build an in-house database with different images of cars with variable dirtiness levels. We trained an AI-based model to automatically compute the level of dirtiness of a car that attained almost 100% accuracy (see confusion-matrix below). Moreover, we used state-of-the-art AI-based algorithms to perform car detection and car model and brand detection from high-resolution images. We built and deployed a final demo application based on a dashboard that allows the end-user to interact and test the different steps of the entire decision process pipeline we are proposing Galp to implement for a year in a test-pilot. We assured that the initial tests are completely free and that the amount of resources needed to perform the pilot-test is relatively low and cheap. Hence, we believe we are delivering a completely actionable solution that may impact the business model paradigm of Galp regarding the car-washing stations of the future.