- Services
- Solutions
- MVP-Calculator
- Portfolio
- FAQ
BI system for a global leader in cosmetic manufacturingBI system development for a global leader in cosmetic manufacturing
Business Result
Our team diligently tracked 21,000 SKUs across 18 countries on 9 different marketplaces, including Amazon. Through our comprehensive approach, we were able to gather the necessary data to develop a valuable business intelligence system.Project idea
Our objective was to develop a robust business intelligence system capable of extracting data on skincare products from various global marketplaces, while accurately parsing their components and pricing structures. Our system also aimed to provide weekly analytics and historical data to support informed decision-making.Tech Stack
AWSS3K8sApache AirflowPuppeteerSpaCy
Solution
Team Composition
2 software engineers, 2 data analytics, 1 PM
Project Duration
18 months
To accomplish this we’ve tried to use pretty standard approach but quickly faced two major challenges:
- Marketplaces actively protect their information. It enables requests throttling, cqrf, scraping detection, etc.
- Marketplaces actively change their layout for marketing purposes.This leads to a much more complex process of building reliable data extractors.
We’ve managed to successfully overcome these problems with intelligent requests, using a mesh of endpoints and AI to extract required data.
Key features
- Recognising ingredients on page for skincare products
- Preventing scrape discovery
- Not really sensitive to layout changes