Accelerating data lake analytics at InCrowd with Dremio and Tableau

Data lakes store raw data without defining a schema before loading it, making it easier to collect raw data from multiple systems. Because the data is centralized, organizations are able to utilize best-in-class and cost-optimized approaches to analytics. This is why InCrowd Sports decided to leverage a cloud-based data lake architecture for their analytics needs, employing Dremio and Tableau.

According to Aberdeen research, organizations have ingested 25% more data than the previous year for the last five years. And while companies manage an average of 19 data sources for analysis, a massive 46% are inaccessible. This type of rapid growth and complexity can wreak havoc on internal efficiencies. At the same time, enterprises are always looking to reduce IT costs. This is why so many companies are modernizing their data architectures with data lakes as their new center of gravity.

Unlike with data warehouses, data lakes store raw data without defining a schema before loading it, making it easier to collect raw data from multiple systems. Because the data is centralized, organizations are able to utilize best-in-class and cost-optimized approaches to analytics. This is why InCrowd Sports decided to leverage a cloud-based data lake architecture for their analytics needs, employing Dremio and Tableau.

Rethinking data architecture to speed up results and lower costs

InCrowd Sports facilitates the sports marketing industry by enabling players, teams, and leagues to connect directly to their fans. The company analyzes a multitude of data sources from ticketing data to clickstream and social data in order to personalize the fan experience. However, InCrowd’s somewhat inflexible data warehouse prevented the fast access to fan insights they needed. It was also expensive, and InCrowd was looking for opportunities to lower costs.

To improve performance for Tableau users, InCrowd was running batch jobs with AWS Redshift to produce aggregate views of data. But this took hours to run and impacted other systems importing data into Redshift. It also required that Tableau users know which views to connect to in a complex, brittle environment. As it became more difficult to meet client expectations for immediate insights, InCrowd was missing opportunities for fan engagement.

Data security and lineage tracking processes were also time-consuming, labor-intensive, and therefore costly. InCrowd needed a way to raise productivity for their data teams while keeping budgets in check—implementing a cloud data lake provided a better way to consolidate and store their data at a much lower cost.

The first step was choosing inexpensive cloud storage for long-term data retention and consolidation. By itself, Amazon S3 was an incomplete solution—InCrowd also sought improved speed, interactive query performance, and a self-service semantic layer for the S3 data lake. So, InCrowd turned to Dremio.

With this modern cloud architecture in place, InCrowd avoided the myriad problems and costs associated with copying and moving significant volumes of data into a proprietary data warehouse. Now they can access, curate, and query the vast majority of their data versus the subsets that were previously. Moreover, the combination of Dremio and Tableau to power their customer analytics resulted in very happy data consumers.

Increasing revenue and loyalty through personalized fan experiences

Dremio enables InCrowd to be more flexible and agile in how they leverage data sources and bring them to life with Tableau. Previously, they would have to amend the structure of a database table in order to support new data or changes to existing data. Dremio’s support for Tableau’s native data source format (TDS) makes it easy to create and publish data sources. Their hosted instance of Tableau Server is connected to Dremio so they have a single integration to maintain and secure.

Each of InCrowd’s previously siloed data sources is represented inside Dremio as a physical data set. These are then joined together as a virtual dataset consisting entirely of metadata, enabling InCrowd’s hosted instance of the Tableau Server to connect directly to Dremio and avoid replicating or moving data. This also enables InCrowd’s data analysts to quickly understand their data structures and produce relevant, interactive dashboards.

This architecture enables InCrowd to create usage dashboards in Tableau, which are hosted in their client-facing insights tool called Bridge. They are able to join clickstream usage data from AWS Redshift with information such as football (soccer) match data from Opta to provide additional end-user insights with sub-second responses versus minutes or even hours.

With the combination of Dremio and Tableau, InCrowd Sports is able to provide its clients with faster, secure access to more data, improving insights and fan experiences. InCrowd’s clients can now get a complete view of how their fans are engaging and spending across all digital and transactional touchpoints, which powers personalized emails, push notifications, and other communication tools. Personalized analytics enable them to provide a better fan experience and make targeted offers to increase revenue spend.

Want to learn more?

Check out this video to learn how Dremio and Tableau enable cloud data lake analytics at InCrowd Sports.