Enterprise Data Platform

Our client’s backstory

Our client is one of the market leaders in smart bed manufacture, powering the sleeping experience with smart bed technology, including biometric data tracking over night, comfortable adjustable features like mattress firmness, temperature, etc., that allows for creating unique sleep experiences that improve health monitoring and the wellbeing of users.

Klika has been the preferred partner for the last 8 years working on almost all parts of their systems from hardware design, firmware development, mobile app development, backend development, data engineering, etc. We are in charge of supporting all client facing applications (external and internal).

The EDP was built to help the client unify all its operations and device information into a large system that tracks all the activities on bed users, as well as company activities in one place.

What need was identified?

Data nowadays are the most valuable asset a company can have with the big data concept, reporting and visualization, ant to get maximum value from data, enterprises need solutions that go beyond traditional BI tools. An Enterprise Data Platform (EDP) creates endless possibilities for data management and insight – breaking down organizational silos and allowing for informed decision-making across all company teams.

The EDP was built as a centralized point to access all internal and external data sources. Large number of IoT devices provide us with a vast amount of data that are used not just as data source for different reports on (sales and marketing), but also as a foundation for building other user features, like the integration with activity trackers (Fitbit, Under Armour, Nest).

Challenges

The immense data generated by IoT devices and millions of users, and company departments on the other hand, has brought organizations to rethink data collection and analysis. Handling different types of data, identifying sources where they are coming from and deciding on the best and most optimized methods to collect them, has become a challenge in a sea of data. Besides new, existing data need to be updated and integrated as well. To give data meaning and avoid the problem of getting unrelated data, companies prefer building a database that collects and analyzes data at one single point.

The second challenge is data automation and getting all the data to the right place at the right time. Assessing which data are to be moved where and how they will be stored is one of the determining factors before automating data processing pipelines.

The third is the lack of resources and tools within companies, which limits their capabilities to handle the distribution, speed, and variety of vast data. That’s why data analytics needs to be precis and accurate and deliver real-time detailed data ready to use and draw insights from in real time.

The solution?

The main part of the EDP was built around ingestion data from different

internal platforms:

• user profiles and sleep data
• Customer orders
• Net promoter scores

and external:

• (Fitbit, Under Armour, Nest).
• User profiles and sleep data
• Customer

Large portion of data was used for analytics and reporting to various departments: the Product team, Marketing team, Customer insights, Sales team. Dozens of reports for each team were implemented ranging from simple excel reports to Power BI real-time dashboards.

Data factories were used for ingestion purposes. Some of them were batch jobs triggered per time interval and some were using event hubs and streams for near real-time ingestion.

Regarding technology, we used HDInsights as managed Apache Hadoop service that lets us run Apache Spark, Apache Hive, Apache Kafka and HBase. Apache Hive the was main tool for querying huge amounts of data and building reports, but other tools and storages were also used (SQL database and ComsosDB).

Klika solutions

How we did it

External sources were ingested either using batch processes (API polling) like:

• Integration with Nest devices for tracking ambient temperature and associating it with sleep sessions (to provide smart sleep tips)

• Integrating with MixPanel to closely assess user behaviors, while using mobile applications, and to troubleshoot possible problems

• Integrating with Underarmour activity trackers

• Or using Events/Streams: Fitbit event streaming or Sleep sessions streaming

The largest portion of data comes in terms of sleep sessions - in total, the system had 1.2 billion sleep sessions stored. Around 1 million beds are pushing sleep sessions every day and night.