Skip to content

Piwik-Bigquery-Project

by trên 21 Tháng Chín, 2017

BigQuery_500px.pngpiwik.png

  1. Introduction:
  • Nowadays, Piwik is the best open-source web analytics. It could be easy to use, simple to install and has a lot of customization. Many companies used it within their cooperation. With Piwik, people could easily to have in-house analytics tool with all their data.
  • Piwik could easily to handle 100.000 pageviews a day. The Piwik’s bottleneck is database resource for archive processing. I made Piwik to handle 40 millions requests a day. But with special customization – Oracle Cluster Database, Tracking Queue, Database sharding. These customization need a lot of development and system administration work. There is no effectiveness way to make Piwik has scalability.
  • BigQuery is a low cost enterprise data warehouse for analytics. That can easily to handle GB to PB data, without worry about scaling. BigQuery standard SQL is compliant with the SQL 2011.
  • As BigQuery cost is based on scan size of data. We could reduce the cost of running database cluster. 1TB for just $5, that is the amount need to processing 50GB a day (24 time running archive processing). With my own experience we need 4 server with 16 CPU core and 32GB Ram to handle such large data each day which might cost $2000 a month with AWS (and more if you use your own servers).
  • This project purpose is make a Piwik installation that could run easily on Google cloud environment with BigQuery as core Archive Processing. We might have our own analytics software running on Google Infrastructure, the same as current Google Analytics 360 offer without payment $150.000 a year to get our own raw data.

Next:

System Architecture

Advertisements
Gửi bình luận

Trả lời

Mời bạn điền thông tin vào ô dưới đây hoặc kích vào một biểu tượng để đăng nhập:

WordPress.com Logo

Bạn đang bình luận bằng tài khoản WordPress.com Đăng xuất / Thay đổi )

Twitter picture

Bạn đang bình luận bằng tài khoản Twitter Đăng xuất / Thay đổi )

Facebook photo

Bạn đang bình luận bằng tài khoản Facebook Đăng xuất / Thay đổi )

Google+ photo

Bạn đang bình luận bằng tài khoản Google+ Đăng xuất / Thay đổi )

Connecting to %s