Retail Reference Architecture Part 3: Query Optimization and Scaling

Author: MongoDB.com ~ Published: May 20, 2015, 2 p.m.
In part one of this series on reference architectures for retail, we discussed how to use MongoDB as the persistence layer for a large product catalog. In part two, we covered the schema and data model for an inventory system. Today we’ll cover how to query and update inventory, plus how to scale the system. Inventory Updates and Aggregations At the end of the day, a good inventory system needs to be more than just a system of record for retrieving static data. We also need to be able to per ...

Retail Reference Architecture Part 3: Query Optimization and Scaling

Author: MongoDB.com ~ Published: May 20, 2015, 2 p.m.
In part one of this series on reference architectures for retail, we discussed how to use MongoDB as the persistence layer for a large product catalog. In part two, we covered the schema and data model for an inventory system. Today we’ll cover how to query and update inventory, plus how to scale the system. Inventory Updates and Aggregations At the end of the day, a good inventory system needs to be more than just a system of record for retrieving static data. We also need to be able to per ...

Server Discovery And Monitoring In PyMongo, Perl, And C

Author: A. Jesse Jiryu Davis ~ Published: May 20, 2015, 3:09 a.m.
(Cross-posted from the MongoDB Blog.) How does a MongoDB driver discover and monitor a single server, a set of mongos servers, or a replica set? How does it determine what types of servers they are? How does it keep this information up to date? How does it discover an entire replica set given an initial host list, and how does it respond to stepdowns, elections, reconfigurations, network error, or the loss of a server? In the past each MongoDB driver answered these questions a little differently ...

How MongoDB Powers The Social Platform Taking London By Storm

Author: MongoDB.com ~ Published: May 19, 2015, 10 a.m.

Many of the largest and most sophisticated companies in the world rely on MongoDB, including over a third of the Fortune 100. In addition to well established businesses using the modern database, innovative start ups from around the world put MongoDB at the heart of their data strategy.

This blog series highlights three UK-based start ups transforming their industries with MongoDB. This week, Urber. In part one of this series we looked at innovative ticketing site DICE.


Urber is a city blogging platform where users share what they love about their city; from food reviews to art events and everything in-between.

We developed the social platform when we realized that there was simply no great place to collectively share city stories, news and tips. The response has already been fantastic - over the last quarter we’ve seen 70% growth in our total number of users and 100% growth in the amount of content on the site.

However, building a platform with a high-level of functionality that could also scale like this, posed some interesting development challenges.

In order to create a data strategy on which to build our business, the development team turned to MongoDB for its high scalability and ease of development.

Why MongoDB

Urber is what’s known as a ‘document handling platform’. In development terms, it is classical CRUD (create, read, update, delete) without much of the D. The central data entity for us is an article, which aligns well with the MongoDB philosophy and how it models with data.

Although there are a number of related data entities to an article (for example, comments, loves, reposts), these are most frequently needed along with the rest of the article data, as that’s when social data becomes really powerful.

For instance, if you're reading about a new restaurant then you will want to know how many stars it received in a review. You may also be interested in its geographical nearness to you and if anyone in your network has eaten there. This ability to easily connect information that makes MongoDB’s document object model a fit for social media platforms, and our articles in particular. We no longer need a more traditional relational database.

Flexible Working

Flexibility is a key ingredient for success in the life of a startup or indeed any business that relies on rapidly evolving technology.

In my experience, flexibility in developing and evolving the application’s data model has been highly desirable but also rather difficult to manage. As document oriented databases have matured, capabilities are definitely changing. MongoDB has shown that much of that work can be effectively removed, so we can build applications faster and continuously innovate.

Our development team is able to keep pace with the changing data needs of the evolving platform thanks largely to MongoDB. The ability to augment data structures without the overhead of maintaining relational data structures has been key. It makes the development team’s life simple, which means we can focus on the important task of growing the business.

How MongoDB helps us scale

Our goal is to emulate the success of other social media platforms such as Twitter or Tumblr. A huge undertaking but we truly believe that we have the potential to achieve it and we need technology that can get us there. In the past quarter we’ve seen a year on year increase of 70% in users and more than 100% growth in content.

Twitter Integration

We have a close integration with Twitter. On Urber, users can tag Twitter handles in their articles and mention people or places they are writing about, as well as share t links automatically. MongoDB handles the unstructured social media data that Twitter produces seamlessly.

The goal of Urber is to provide readers with insider knowledge. We’re creating an experience and we hope to be the it place for city news, stories and inspiration. Establishing a start-up is always difficult, but one thing we’ve not lost sleep over is whether we have the right technology stack to support us. MongoDB is helping Urber expand in ways we couldn’t even imagine when we started.

To see how organizations around the world are building applications never before possible, read our Quantifying the Value of Database Selection white paper:

READ QUANTIFYING THE VALUE OF DATABASE SELECTION


How MongoDB Powers The Social Platform Taking London By Storm

Author: MongoDB.com ~ Published: May 19, 2015, 10 a.m.

Many of the largest and most sophisticated companies in the world rely on MongoDB, including over a third of the Fortune 100. In addition to well established businesses using the modern database, innovative start ups from around the world put MongoDB at the heart of their data strategy.

This blog series highlights three UK-based start ups transforming their industries with MongoDB. This week, Urber. In part one of this series we looked at innovative ticketing site DICE.


Urber is a city blogging platform where users share what they love about their city; from food reviews to art events and everything in-between.

We developed the social platform when we realized that there was simply no great place to collectively share city stories, news and tips. The response has already been fantastic - over the last quarter we’ve seen 70% growth in our total number of users and 100% growth in the amount of content on the site.

However, building a platform with a high-level of functionality that could also scale like this, posed some interesting development challenges.

In order to create a data strategy on which to build our business, the development team turned to MongoDB for its high scalability and ease of development.

Why MongoDB

Urber is what’s known as a ‘document handling platform’. In development terms, it is classical CRUD (create, read, update, delete) without much of the D. The central data entity for us is an article, which aligns well with the MongoDB philosophy and how it models with data.

Although there are a number of related data entities to an article (for example, comments, loves, reposts), these are most frequently needed along with the rest of the article data, as that’s when social data becomes really powerful.

For instance, if you're reading about a new restaurant then you will want to know how many stars it received in a review. You may also be interested in its geographical nearness to you and if anyone in your network has eaten there. This ability to easily connect information that makes MongoDB’s document object model a fit for social media platforms, and our articles in particular. We no longer need a more traditional relational database.

Flexible Working

Flexibility is a key ingredient for success in the life of a startup or indeed any business that relies on rapidly evolving technology.

In my experience, flexibility in developing and evolving the application’s data model has been highly desirable but also rather difficult to manage. As document oriented databases have matured, capabilities are definitely changing. MongoDB has shown that much of that work can be effectively removed, so we can build applications faster and continuously innovate.

Our development team is able to keep pace with the changing data needs of the evolving platform thanks largely to MongoDB. The ability to augment data structures without the overhead of maintaining relational data structures has been key. It makes the development team’s life simple, which means we can focus on the important task of growing the business.

How MongoDB helps us scale

Our goal is to emulate the success of other social media platforms such as Twitter or Tumblr. A huge undertaking but we truly believe that we have the potential to achieve it and we need technology that can get us there. In the past quarter we’ve seen a year on year increase of 70% in users and more than 100% growth in content.

Twitter Integration

We have a close integration with Twitter. On Urber, users can tag Twitter handles in their articles and mention people or places they are writing about, as well as share t links automatically. MongoDB handles the unstructured social media data that Twitter produces seamlessly.

The goal of Urber is to provide readers with insider knowledge. We’re creating an experience and we hope to be the it place for city news, stories and inspiration. Establishing a start-up is always difficult, but one thing we’ve not lost sleep over is whether we have the right technology stack to support us. MongoDB is helping Urber expand in ways we couldn’t even imagine when we started.

To see how organizations around the world are building applications never before possible, read our Quantifying the Value of Database Selection white paper:

READ QUANTIFYING THE VALUE OF DATABASE SELECTION


Announcing libmongoc 1.1.6

Author: A. Jesse Jiryu Davis ~ Published: May 18, 2015, 10:50 p.m.

Hans Egede sea serpent 1734

I released libmongoc 1.1.6 today with some bugfixes and a major performance enhancement.

  • mongoc_bulk_operation_execute now coalesces consecutive update operations into a single message to a MongoDB 2.6+ server, yielding huge performance gains. Same for remove operations. (Inserts were always coalesced.)
  • Large numbers of insert operations are now properly batched according to number of documents and total data size.
  • GSSAPI / Kerberos auth now works.
  • The driver no longer tries three times in vain to reconnect to a primary, so socketTimeoutMS and connectTimeoutMS now behave closer to what you expect for replica sets with down members. A full fix awaits 1.2.0.

I snuck in a feature:

I also released libbson 1.1.6 to maintain version parity; it's identical to libbson 1.1.5.

Release tarballs are available for download:

You'll notice this is just a week after the 1.1.5 release, since there were a users waiting on these particular fixes that I couldn't get in to last week's release.

It's my intention to do only the most critical work for the 1.1.x line of the driver libraries, and concentrate on shipping 1.2.0 as soon as possible: a reasonably tested beta in the middle of June and a stable version at the beginning of August. (Circumstances are likely to intervene, of course.) Shipping version 1.2.0 will offer you a C driver that conforms with the modern MongoDB specs: Server Discovery And Monitoring, and Server Selection. It will resolve a heap of replica set issues in the current driver.

For further information:

Thanks to those who contributed:

  • A. Jesse Jiryu Davis
  • Jason Carey
  • Kai Mast
  • Matt Cotter

Image: The "Great Sea Serpent" according to Hans Egede


Meteor: Build iOS and Android Apps that are a Delight to Use

Author: MongoDB.com ~ Published: May 13, 2015, 2 p.m.

Meteor provides a complete open source platform for building web and mobile apps in pure JavaScript. The Meteor team chose MongoDB as its datastore for its performance, scalability, and rich features for JSON. Meteor apps run using JavaScript via Node.JS on the server and JavaScript in the phone’s browser. You write both the client and the server sides of your application in JavaScript using the Meteor.JS framework.

What’s so great about Meteor? Here’s a few things that are particularly interesting to MongoDB developers:

  • Productivity Through Simplicity. Through reactivity and intelligent implication, Meteor requires much less code than other frameworks to get the same tasks done.
  • One Language. You write the client and server portions of your application in the same JavaScript language using the same framework. You can even place both in one file if you like.
  • Database Everywhere. The same methods access your MongoDB database on the server or on the phone.
  • Data on the Wire. Meteor does not send HTML over the network. The server sends data and the client renders it.
  • Latency Compensation. On the client, Meteor prefetches data from a local database to make it look like server method calls return instantly.
  • Full Stack Reactivity. All layers update themselves as data changes without any additional code.
  • Open Ecosystem. Open Source Meteor integrates with other open source tools and frameworks

The Database Everywhere and Latency Compensation concepts are especially interesting. Meteor maintains a local copy of data by implementing its own miniMongo database in phone memory. All of the phone-to-server data communication and synchronization is provided by Meteor. miniMongo is a JavaScript implementation of the MongoDB API.

With the automatic syncing provided by Meteor, you don’t need to have one set of developers working in one framework for the phone, and another set of developers working in another framework for the server. Now you can have full stack developers use one language and one framework. The phone code and the server code reside in the same JavaScript file. You build and deploy with a single command.

To show you the amazing simplicity of Meteor, let me show you the implementation of a barebones app that does this on a phone:

This HTML presents the page:

Meteor Blaze is a powerful library for creating live-updating user interfaces. The Blaze UI responds to messages from objects inside the <template> object. Blaze replaces {{counter}} with the value of the counter value in the JavaScript section. Intelligent “reactivity” will update the counter variable as it changes.

Now for the JavaScript. Note how the client and server code reside in a single file:

Blaze replaces frameworks such as Angular, Backbone, Ember, React or Knockout. Blaze updates HTML templates automatically without directives, model classes, data dependency declarations, or rendering functions. Blaze infers the data dependency of arbitrary JavaScript, sets up callbacks to detect changes in the template’s data sources, recompute values, and changes the DOM in place with the update. Meteor calls this feature “reactivity.” You just write simple HTML with variable names enclosed in double brackets as we did with {{counter}} in the HTML above.

Blaze lets you create rich phone user interfaces as in these screens from the Verso app built with Meteor and MongoDB:

The same codebase that powers the phone app can power a website for a PC or Mac in a browser. Employees can use their phones to run an app while managers use their desktops for reporting. For example, the same codebase that creates the phone interfaces above presents these web pages in a browser running on a PC or Mac desktop computer:

You can install and use Meteor for free. That link has excellent step by step tutorial showing you how to create a simple “todo” app that uses MongoDB for the datastore. At the end of the tutorial you can install a more elaborate sample todo app or a more complex customer engagement app that shows off native phone hardware functionality and social features. Meteor even provides a free server sandbox to deploy and test your apps.

If you’re interested in learning more about the architecture of MongoDB, download our guide:

DOWNLOAD ARCHITECTURE GUIDE


Meteor: Build iOS and Android Apps that are a Delight to Use

Author: MongoDB.com ~ Published: May 13, 2015, 2 p.m.

Meteor provides a complete open source platform for building web and mobile apps in pure JavaScript. The Meteor team chose MongoDB as its datastore for its performance, scalability, and rich features for JSON. Meteor apps run using JavaScript via Node.JS on the server and JavaScript in the phone’s browser. You write both the client and the server sides of your application in JavaScript using the Meteor.JS framework.

What’s so great about Meteor? Here’s a few things that are particularly interesting to MongoDB developers:

  • Productivity Through Simplicity. Through reactivity and intelligent implication, Meteor requires much less code than other frameworks to get the same tasks done.
  • One Language. You write the client and server portions of your application in the same JavaScript language using the same framework. You can even place both in one file if you like.
  • Database Everywhere. The same methods access your MongoDB database on the server or on the phone.
  • Data on the Wire. Meteor does not send HTML over the network. The server sends data and the client renders it.
  • Latency Compensation. On the client, Meteor prefetches data from a local database to make it look like server method calls return instantly.
  • Full Stack Reactivity. All layers update themselves as data changes without any additional code.
  • Open Ecosystem. Open Source Meteor integrates with other open source tools and frameworks

The Database Everywhere and Latency Compensation concepts are especially interesting. Meteor maintains a local copy of data by implementing its own miniMongo database in phone memory. All of the phone-to-server data communication and synchronization is provided by Meteor. miniMongo is a JavaScript implementation of the MongoDB API.

With the automatic syncing provided by Meteor, you don’t need to have one set of developers working in one framework for the phone, and another set of developers working in another framework for the server. Now you can have full stack developers use one language and one framework. The phone code and the server code reside in the same JavaScript file. You build and deploy with a single command.

To show you the amazing simplicity of Meteor, let me show you the implementation of a barebones app that does this on a phone:

This HTML presents the page:

Meteor Blaze is a powerful library for creating live-updating user interfaces. The Blaze UI responds to messages from objects inside the <template> object. Blaze replaces {{counter}} with the value of the counter value in the JavaScript section. Intelligent “reactivity” will update the counter variable as it changes.

Now for the JavaScript. Note how the client and server code reside in a single file:

Blaze replaces frameworks such as Angular, Backbone, Ember, React or Knockout. Blaze updates HTML templates automatically without directives, model classes, data dependency declarations, or rendering functions. Blaze infers the data dependency of arbitrary JavaScript, sets up callbacks to detect changes in the template’s data sources, recompute values, and changes the DOM in place with the update. Meteor calls this feature “reactivity.” You just write simple HTML with variable names enclosed in double brackets as we did with {{counter}} in the HTML above.

Blaze lets you create rich phone user interfaces as in these screens from the Verso app built with Meteor and MongoDB:

The same codebase that powers the phone app can power a website for a PC or Mac in a browser. Employees can use their phones to run an app while managers use their desktops for reporting. For example, the same codebase that creates the phone interfaces above presents these web pages in a browser running on a PC or Mac desktop computer:

You can install and use Meteor for free. That link has excellent step by step tutorial showing you how to create a simple “todo” app that uses MongoDB for the datastore. At the end of the tutorial you can install a more elaborate sample todo app or a more complex customer engagement app that shows off native phone hardware functionality and social features. Meteor even provides a free server sandbox to deploy and test your apps.

If you’re interested in learning more about the architecture of MongoDB, download our guide:

DOWNLOAD ARCHITECTURE GUIDE


Retail Reference Architecture Part 2: Approaches to Inventory Optimization

Author: MongoDB.com ~ Published: May 13, 2015, 10 a.m.
In part one of our series on retail reference architecture we looked at some best practices for how a high-volume retailer might use MongoDB as the persistence layer for a large product catalog. This involved index, schema, and query optimization to ensure our catalog could support features like search, per-store pricing and browsing with faceted search in a highly performant manner. Over the next two posts we will be looking at approaches to similar types of optimization, but applied to an enti ...

Retail Reference Architecture Part 2: Approaches to Inventory Optimization

Author: MongoDB.com ~ Published: May 13, 2015, 10 a.m.
In part one of our series on retail reference architecture we looked at some best practices for how a high-volume retailer might use MongoDB as the persistence layer for a large product catalog. This involved index, schema, and query optimization to ensure our catalog could support features like search, per-store pricing and browsing with faceted search in a highly performant manner. Over the next two posts we will be looking at approaches to similar types of optimization, but applied to an enti ...