Infrastructure - Revision history

Msemm: /* The brain */

2015-01-03T13:32:16Z

‎The brain

Msemm at 13:32, 3 January 2015

2015-01-03T13:32:02Z

Msemm at 13:31, 3 January 2015

2015-01-03T13:31:07Z

Msemm at 13:30, 3 January 2015

2015-01-03T13:30:34Z

Krzysztof at 10:04, 29 July 2014

2014-07-29T10:04:32Z

Nanako at 23:04, 30 January 2014

2014-01-30T23:04:38Z

Msemm at 09:56, 25 January 2014

2014-01-25T09:56:00Z

Msemm: /* The brain */

2014-01-23T18:08:08Z

‎The brain

Msemm at 18:07, 23 January 2014

2014-01-23T18:07:27Z

Msemm at 18:07, 23 January 2014

2014-01-23T18:07:03Z

@@ Line 88: / Line 88: @@
 Details about the implemented software and infrastructure can be found above.
-There are plans to publish the entire server software as open source for stimulating the contribution of software features of other community members of the OpenCellID project. This will most likely happen before the end of 2014.
+There are plans to publish the entire server software as open source for stimulating the contribution of software features of other community members of the OpenCellID project. This will most likely happen before the end of 2015.
 |}

@@ Line 88: / Line 88: @@
 Details about the implemented software and infrastructure can be found above.
-There are plans to publish the entire server software as open source for stimulating the contribution of software features of other community members of the OpenCellID project. This will most likely happen before the end of 2015.
+There are plans to publish the entire server software as open source for stimulating the contribution of software features of other community members of the OpenCellID project. This will most likely happen before the end of 2014.
 |}

@@ Line 88: / Line 88: @@
 Details about the implemented software and infrastructure can be found above.
-There are plans to publish the entire server software as open source for stimulating the contribution of software features of other community members of the OpenCellID project. This will most likely happen before the end of 2014.
+There are plans to publish the entire server software as open source for stimulating the contribution of software features of other community members of the OpenCellID project. This will most likely happen before the end of 2015.
 |}

@@ Line 66: / Line 66: @@
 The OpenCellID backend uses Kafka queuing system in order to be able to handle periodic peaks. Kafka producers embedded into the web application send all incoming data to Kafka brokers. Kafka consumers pull data from brokers, process measurements and store them in MongoDB.
-The database backend, with a current 4.4 million cell towers and about 565 million measurements (1.1.2014), is a MongoDB database cluster with six servers:
+The database backend, with a current 7 million cell towers and about 1.2 billion measurements (1.1.2015), is a MongoDB database cluster with six servers:
 *three servers are serving as MongoDB configuration servers and Zookeeper instances
 *the other three servers are serving as the database backend with one replication set spread across the three servers and Kafka brokers

@@ Line 64: / Line 64: @@
 ===Database Backend===
 The database backend, with a current 4.4 million cell towers and about 565 million measurements (1.1.2014), is a MongoDB database cluster with six servers:
-*three servers are serving as MongoDB configuration servers
+*three servers are serving as MongoDB configuration servers and Zookeeper instances
-*the other three servers are serving as the database backend with one replication set spread across the three servers
+*the other three servers are serving as the database backend with one replication set spread across the three servers and Kafka brokers
 ==Challenges and solutions==
@@ Line 75: / Line 77: @@
 *Data Processing<br>the analysis and process of the rapidly growing data must be constantly efficient
-The current solutions are based on MongoDB and its features:<br>
+The current solutions are based on Kafka queuing system and MongoDB with its features:<br>
 * Native Analytics<br>using the integrated aggregation framework and Map/Reduce to calculate aggregates and analyses in place without the need of prior exporting data to other systems
 * Advanced Geo Queries<br>using geospatial MongoDB support to execute complex queries
 * Horizontal Scaling<br>sharding makes it easy to scale applications horizontally on commodity hardware for accommodating constantly increased throughput
-* Reduced Total Cost of Ownership (TCO)<br>as open-source storage MongoDB is a very cost-effective solution
+* Reduced Total Cost of Ownership (TCO)<br>as open-source storage MongoDB and Kafka queuing system are a very cost-effective solution
 ==The brain==
-Krzysztof Ociepa (email: ociepa.krzysztof@gmail.com) has designed the big-data infrastructure as well as the new OpenCellID server software based on Java and MongoDB, and has also implemented most of the current features after two other developers failed to do so.
+Krzysztof Ociepa (email: ociepa.krzysztof@gmail.com) has designed the big-data infrastructure as well as the new OpenCellID server software based on Java, Kafka queuing system and MongoDB, and has also implemented most of the current features after two other developers failed to do so.
 Details about the implemented software and infrastructure can be found above.

@@ Line 58: / Line 58: @@
 ===Frontend===
-*The web frontend uses Apache web server as a proxy for serving web requests to Tomcat.
+*the web frontend uses Apache web server as a proxy for serving web requests to Tomcat
+*the OpenCellID web application is running on Tomcat and is reading and writing cell measurement data to/from the MongoDB database backend
-*The OpenCellID web application is running on Tomcat and is reading and writing cell measurements data to/from the MongoDB database backend.
+*jQuery Mobile is responsible for providing a cross-platform user interface
+*the map is displayed using OpenStreetMap combined with Leaflet library
-*jQuery Mobile is responsible for providing a cross-platform user interface.
-−
-*The map is displayed using OpenStreetMap combined with Leaflet library.
 ===Database Backend===
 The database backend, with a current 4.4 million cell towers and about 565 million measurements (1.1.2014), is a MongoDB database cluster with six servers:
-*Three servers are serving as MongoDB configuration servers
+*three servers are serving as MongoDB configuration servers
-*The other three servers are serving as database backend with one replication set spread across the three servers
+*the other three servers are serving as the database backend with one replication set spread across the three servers
 ==Challenges and solutions==
-The OpenCellID community is very strong and continously provides a high number of measurements.<br>
+The OpenCellID community is very strong and continuously provides a high number of measurements.<br>
 This immediately poses a few challenges:
-* High Volume<br>data arrives from many differnet sources and is rapidly growing
+*High Volume<br>data arrives from many different sources and is rapidly growing
-* Scale<br>growth of data should go along with predictable, incremental costs and no downtime should be needed when adding additional server resources
+*Scale<br>growth of data should go along with predictable, incremental costs and no downtime should be needed when adding additional server resources
-* Data Processing<br>analyzing and processing of rapidly growing data must be constantly efficient.
+*Data Processing<br>the analysis and process of the rapidly growing data must be constantly efficient
 The current solutions are based on MongoDB and its features:<br>
@@ Line 89: / Line 86: @@
 Details about the implemented software and infrastructure can be found above.
-There are plans to publish the entire server software as open source for stimulating the contribution of software features of other community members of the OpenCellID project. Most likely this is going to happen before the end of year 2014.
+There are plans to publish the entire server software as open source for stimulating the contribution of software features of other community members of the OpenCellID project. This will most likely happen before the end of 2014.
 |}

@@ Line 1: / Line 1: @@
-[[File:OpenCellID server strukture.PNG|right|600px]]
+[[File:OpenCellID server strukture.PNG|right|400px]]
 {| width=998px
+|