Ticketmaster / Live Nation
I moved from warehouse engineering into leading data engineering teams whose work reached client reporting, finance, fraud detection, and live event operations. That meant rebuilding data models and pipelines, carrying business rules through warehouse migrations, and eventually delivering a replacement platform during the pandemic. The projects below had different constraints and required different combinations of direct engineering, product work, and team leadership.
Client analytics and the warehouse
The client analytics platform served about 10,000 venues, promoters, and artists with financial, ticket-sales, and attendance reporting. I rebuilt its data model and reporting pipelines to reduce latency, improve performance, and support much more detailed reporting. Changing the structure of the data and its transformations let the product offer detail that the previous model had struggled to support.
I worked in the product teams developing those pipelines, connecting what customers needed to see with physical database design, ETL mappings, data marts, and backfill plans. The work included real-time attendance and on-sale reporting for ticket sales, where the usefulness of a report depended on both its content and when it reached the client.
Alongside the analytics product, I wrote the core SQL Server exports for a company-wide financial clearing house. I also played an integral part in the teams moving the warehouse from SQL Server to Teradata and later from Teradata to Snowflake, carrying reporting and integration work through those changes.
Fraud data
Before the CDC program, I was the sole engineer supplying the fraud team's training and inference data in Snowflake. That combined Ticketmaster transactions, payment streams, and Accertify evaluation data for the models the fraud team built.
CDC sales reporting across the ticketing estate
In 2019 I led consumer-side delivery of change-data-capture reporting across roughly 500 Sybase ticketing databases. Sports and theater clients needed current sales figures when deciding whether to release more inventory. Busy sales approached 10,000 tickets a minute, and each transaction could change 9 to 15 related tables. Reporting had to keep up without adding load to the systems selling tickets. The contract carried substantial penalties, which I can discuss in person.
The source team built a Java producer that emitted Avro messages into Kafka. I worked with them on the architecture, including compaction, handling missing or stale changes, and monitoring producers during busy on-sales. A per-client view of lag helped the teams identify where source processing needed attention.
A principal engineer and his team handled Snowflake ingestion, historical retention, and current-state modeling. I reconstructed the reporting rules from the source stored procedures and wrote the consolidated SQL. Capturing the data correctly was necessary, but the reports also had to represent the sales, refunds, and other changes the client was asking about.
I also designed and wrote a Java Kafka Streams application applying the reporting logic into PostgreSQL on AWS, with infrastructure managed through Terraform. Keeping joined results current during bursts involved storage and server engineering as well as stream processing. Running the paths together let us compare their results with each other and the source systems, which also exposed defects in the original reporting procedures.
The Snowflake path reported about five minutes behind real time and was easier to scale and maintain. Changes to the reporting rules were more difficult to carry through the Kafka Streams application, and its joined updates struggled under the heaviest bursts. I retired that implementation and kept the Snowflake path.
I would apply the same reasoning to competing designs. The comparison has to include how other people will change and support each path after the original authors move on. My familiarity with my own implementation is useful during evaluation, but it is a poor substitute for the organization's ability to maintain it.
I would be equally direct about whose design survives. If the people responsible for operating a system can more readily scale and maintain a colleague's approach, the time I spent on mine does not create an obligation to keep it. A technical leader has to make it possible for that comparison to be candid, particularly when the person arguing for the other approach reports to them.
We kept Kafka topic, partition, timestamp, and client metadata through the warehouse layers so a reported number could be traced back to its source. That supported investigation of the data without implying global ordering across partitions or exactly-once delivery.
I worked directly with Ticketmaster Product, NFL stakeholders, and client account teams to pin down requirements and set shared expectations about what would actually ship.
Working with Snowflake on ingestion
A Snowflake proof of concept in May 2018 preceded the live reporting program. As an early customer collaborator, I worked with Snowflake's product teams on Kafka ingestion requirements, tested early releases, and supplied feedback about the data and metadata we needed. Later discussions included scheduled tasks and materialized views. This was customer-side specification and testing with the vendor's engineers.
A replacement platform during the pandemic
In a separate program in 2020, the collapse in live events made operating cost an urgent constraint. I was asked to deliver a replacement platform on Databricks and establish it through working sales and attendance products. The teams delivered in six months, with the platform and those products operating by December 2020.
The engineers already knew how to model the business in SQL. I designed the development framework around that expertise and tested its usability alongside the people building the data products. They could develop transformations in notebooks, iterate on individual stages, and publish models without first becoming specialists in Spark configuration and execution.
The platform engineers built the framework primarily in Python, with performance-critical components in Scala and transformations running through Spark SQL. Raw ingestion from configured Kafka topics was automated. The framework generated configuration around the engineers' models, and metadata described dependencies and scheduling, including triggers based on record counts or time.
Git promotion and pull requests governed deployment. Dependency handling, failure notifications, and Delta Lake-backed restore points supported recovery, so a failed stage did not always require a person to reconstruct the work around it. I led the migration and supplied business-logic expertise while the platform and data-delivery teams built and moved the products.
Sales reporting reached about 30-second freshness. Annual operating cost for that comparable workload fell from about $300,000 to $50,000, while the wider platform also supported attendance and further migrations.
These projects changed different things for the people relying on them. More reporting detail gave clients a better view of their business; fresher sales and attendance data mattered while an event was happening; and the development framework gave engineers a way to contribute using the SQL and business knowledge they already had. The useful measure of the platform was the work those people could now do, including the work of the engineers who would have to maintain it.
The pandemic platform let engineers contribute through SQL and business rules they already understood, while the framework carried recurring execution and recovery work. Under that deadline, the judgment was where to spend the team's learning effort and where shared engineering could absorb it. A framework takes time to build and support; its value depends on whether that effort reduces repeated work for the engineers delivering the data products.
In a crisis I make the cost to the team visible, and I leave behind a system people can sustain once the pressure passes. The pandemic delivery matters because of the platform and data products the teams delivered under those constraints, including a development framework that made their existing expertise useful quickly.
SAP Data Services, SQL Server, Teradata, Snowflake, Kafka, Kafka Streams, PostgreSQL, Databricks, Spark SQL, Python, Scala, Delta Lake, Accertify
Across my time at Ticketmaster I received two Live Nation IN awards and a Rock Star award.
Who did what
I combined direct work on data models, reporting logic, and the Kafka Streams implementation with leadership of the reporting and platform programs. The principal engineer and his team owned the Snowflake ingestion and modeling work; the source team implemented capture. For Databricks, I set the framework architecture and led migration, working with the engineers who built the platform components and the data products running on them.