What Storage Engine Should my Backups Use?

Author: MMS Release Notes ~ Published: April 24, 2015, 6:27 p.m.

What is WiredTiger? What is MMAPv1?

WiredTiger is MongoDB’s new storage engine. It is available as an option for all 64-bit MongoDB 3.0 and higher builds. Among other features, it supports document-level locking and compression on disk. Check out MongoDB’s docs on WiredTiger for more information.

MMAPv1 is MongoDB’s traditional memory-mapped files storage engine. In MongoDB 3.0 we added collection-level locking while remaining compatible with the on-disk storage of MongoDB 2.6 and older.

Why you should match

Matching your currently running storage engine to your backed-up engine is critical to easy restores. If your restore files are in a different format than what you are used to running, you will have to make certain that you set your command line options correctly, and differently from what you are running elsewhere.

Where to change your setting

First, you have to be running the new MMS. MMS Classic customers cannot change their backup format, and only have MMAPv1 backups available to them. You are running the new MMS if the upper-left of your MMS window looks like this:

new MMS

Once you are in the new MMS, you can change your backup format by going to the “Backup” tab, click the “…” for your replica set or cluster and choose “Edit Storage Engine”

... Edit Storage Engine

Once there, you can choose MMAPv1 (“MongoDB Memory Mapped Files”) or WiredTiger:

mmapv1 vs WT

Once you make this change, an initial sync will be triggered. Choose the server you want to sync from and confirm. An initial sync is required so we can build your new backup. This will not change your existing snapshot formats, so if you request an older restore, it will still be in MMAPv1. You can tell if a snapshot is in WiredTiger format by looking at the “Mongod Version” column on your snapshots listing page (just click on a replica set name on your Backup tab). If the version has “(wiredTiger)” after it, the snapshot is in WiredTiger format. You can see I converted this replica set to WiredTiger:

historical snapshots will not change


Samantha Ritter And Me At Open Source Bridge 2015

Author: A. Jesse Jiryu Davis ~ Published: April 24, 2015, 5:01 p.m.

Open source bridge 1

I'm so excited to tell you, my colleague Samantha Ritter and I were accepted to speak in Portland, Oregon this June.

Cat-herd's Crook: Enforcing Standards in 10 Programming Languages.

Samantha and I helped specify and test how MongoDB drivers behave in ten programming languages, and we persuaded dozens of open source programmers to implement these specifications the same. We created a test framework that verified compliance with YAML descriptions of how drivers should act, and we communicated updates to the specs by pushing changes to the YAML files.

We want to show you how we herded all the cats in the same direction.

How Do Python Coroutines Work?

I'll explain asyncio's new coroutine implementation in depth, including the mystical "yield from" statement. You'll know better than any of your peers how this amazing new programming model works. Plus, in a magical and entertaining feat of daring, I plan to live-code an asynchronous coroutine implementation before your very eyes!


MongoDB Storage Engine Summit - June 4th

Author: MongoDB.com ~ Published: April 24, 2015, 10:45 a.m.

When: June 4th, 9am to 4:30 pm
Where: MongoDB office, 229 W 43rd Street, 5th Floor, New York
Who: For engineers and architects implementing MongoDB storage engines
Questions: Contact henrik.ingo a t mongodb.com

To facilitate collaboration with the MongoDB team in charge of creating the MongoDB Storage Engine API, and with people implementing storage engines that consume that API, we are inviting you to a summit in our New York office. This is a full-day event taking place on June 4th.

Please RSVP to the event by May 20th, using this form, so that we can reserve enough seats and lunch. Note: MongoDB employees participating must RSVP too!

Below is a preliminary agenda for the day.

Agenda

Storage Engine API in MongoDB 3.0

  • Most participants will already be familiar with it, so this is just a short recap to allow for questions.
  • Feedback from community on the API in MongoDB 3.0.

Best practices for working with storage engine partners, by Mark Callaghan

  • From people with experience doing the same with MySQL, we'll be examining some best practices worth repeating and some mistakes to avoid in working with 3rd party storage engine implementers

MongoDB 3.1 Storage Engine API

  • Share current plans/design

MongoDB 3.1 Oplog and Replication enhancements

  • Change notification
  • Read committed

Community wishlist for MongoDB 3.1 Storage Engine API

  • Roundtable or just multiple speakers taking turns
  • From 3rd party engines to MongoDB

Unstructured time

  • Unconference and/or hackathon style

Lunch will be provided.

RSVP HERE


The Frills

Author: Hannes Magnusson ~ Published: April 23, 2015, 5:12 p.m.
I don't like ORMs. I really don't. It's not just because all of the ORM frameworks I've seen make mockery out of performance needs, but they also tend to be extremely over-engineered, which makes me want to cry. I'd wager that most people using ORMs are actually solving the wrong problem with the wrong solution, simply because there wasn't a better alternative for their data-modeling needs and the benefits of adopting an ORM outweighed its inherent problems.

It's like driving a gasoline-hungry car. You say that you care about its MPG rating, but you are still wasting a lot of gas without ever looking back. It is just a cost of living. Or insurance: You know you can save 15% or more... but you don't. You've accepted that bi-annual rip-off as cost of living.

Just like using an ORM, you have accepted the pain as cost of living. Well, I don't :)

ODS - Object<->Document Serializer

The new MongoDB PHP driver includes an experimental ODS interface (BSON\Persistable) that allows you to automatically store your object as documents in MongoDB the way you see fit. And the best part? When you retrieve the document from MongoDB, the driver will know which class it represented and can reconstruct the object (again, the way you see fit).

All you have to do is implement the two methods of the BSON\Persistable interface:
That's it! There is no special way of querying by type, parent object, or any of those weird things that ORMs have introduced to work around storing objects or documents in databases. Your document is your object. Your relations are your relations.

If you would like to build on top of this functionality, it's trivial to implement a trait that implements both methods and provides basic change tracking. At that point, we are entering ORM and ODM territory and are going beyond what we should ask from an extension. All we want from an extension is performance, and simplicity.


Much ado about nothing

Author: Mark Callaghan ~ Published: April 22, 2015, 6 p.m.
Kyle does amazing work with Jepsen and I am happy that he devotes some of his skill and time into making MongoDB better. This week a new problem was reported as stale reads despite the use of the majority write concern. Let me compress the bug report and blog post for you, but first this isn't a code bug this is expected behavior for async replication.

  1. MongoDB implements asynchronous master-slave replication
  2. Commits can be visible to others on the master before oplog entries are sent to the slave
The problem occurs when transaction 1 commits a change on the master, transaction 2 views that change on the master, the master disappears and a slave is promoted to be the new master where the oplog entries from transaction 1 never reached any slave. At this point transaction 1 didn't happen and won't happen on the remaining members of the replica set yet transaction 2 viewed that change. A visible commit has been lost.

When reading MongoDB source I noticed this in early 2014. See my post on when MongoDB makes a transaction visible. I even included a request to update the docs for write concerns and included this statement:
I assume the race exists in that case too, meaning the update is visible on the primary to others before a slave ack has been received.
This isn't a bug, this is async replication. You can fix it by adding support for sync replication. The majority write concern doesn't fix it because that only determines when to acknowledge the commit, it does not determine when to make the commit visible to others.  For now the problem might be in the documentation if it wasn't clear about this problem. The majority write concern is a lot like semisync replication in MySQL and then clever people added lossless semisync replication so that commits aren't visible on the master until they have been received by a slave. Finally, really clever people got lossless semisync replication running in production and we were much happier.


Announcing PyMongo 3.0.1

Author: A. Jesse Jiryu Davis ~ Published: April 21, 2015, 9:10 p.m.

Leaf

It's my pleasure to announce the release of PyMongo 3.0.1, a bugfix release that addresses issues discovered since PyMongo 3.0 was released a couple weeks ago. The main bugs were related to queries and cursors in complex sharding setups, but there was an unintentional change to the return value of save, GridFS file-deletion didn't work properly, passing a hint with a count didn't always work, and there were some obscure bugs and undocumented features.

For the full list of bugs fixed in PyMongo 3.0.1, please see the release in Jira.

If you are using PyMongo 3.0, please upgrade immediately.

If you are on PyMongo 2.8, read the changelog for major API changes in PyMongo 3, and test your application carefully with PyMongo 3 before deploying.


MongoDB : Playing With Arrays

Author: Tugdall Grall ~ Published: April 21, 2015, 1:45 p.m.
As you know, you have many differences between relational and document databases. The biggest, for the developer, is probably the data model: Row versus Document. This is particularly true when we talk about relations versus embedded documents (or values). Lets look at some examples, then see what are the various operations provided by MongoDB to help you to deal with this. I wont use this post to go in all the details about document design, but I just want to focus on the operations you can t ...

See Who is Speaking at MongoDB World

Author: MongoDB.com ~ Published: April 21, 2015, 10:30 a.m.

This year at MongoDB World, join us and 2000+ tech enthusiasts for two days of comprehensive technical sessions, networking opportunities, and one-on-one time with engineers. The most cutting edge thought leaders will take the stage at MongoDB World. They come from industry powerhouses and the most disruptive forces in the market. Come see what they’re excited about.

Featured Keynote Speakers including:

  • Mark Callaghan, Member of Technical Staff of Facebook
  • Ben Golub, CEO of Docker
  • Hilary Mason, Data Scientist and Founder of Fast Forward Labs
  • Scott Dietzen, CEO of Pure Storage
  • R “Ray” Wang, Principal Analyst, Founder and Chairman of Constellation Research, Inc.
  • Dev Ittycheria, President and CEO of MongoDB
  • Eliot Horowitz, Co-Founder and CTO of MongoDB

With over 100+ sessions to choose from, you will walk away with best practices, tips and tricks you need to know to build and manage modern applications.

Sign up today and save $400 on onsite ticket prices with Early Bird Pricing. Hurry, prices increase after Friday, May 1st.

REGISTER NOW


See Who is Speaking at MongoDB World

Author: MongoDB.com ~ Published: April 21, 2015, 10:30 a.m.

This year at MongoDB World, join us and 2000+ tech enthusiasts for two days of comprehensive technical sessions, networking opportunities, and one-on-one time with engineers. The most cutting edge thought leaders will take the stage at MongoDB World. They come from industry powerhouses and the most disruptive forces in the market. Come see what they’re excited about.

Featured Keynote Speakers including:

  • Mark Callaghan, Member of Technical Staff of Facebook
  • Ben Golub, CEO of Docker
  • Hilary Mason, Data Scientist and Founder of Fast Forward Labs
  • Scott Dietzen, CEO of Pure Storage
  • R “Ray” Wang, Principal Analyst, Founder and Chairman of Constellation Research, Inc.
  • Dev Ittycheria, President and CEO of MongoDB
  • Eliot Horowitz, Co-Founder and CTO of MongoDB

With over 100+ sessions to choose from, you will walk away with best practices, tips and tricks you need to know to build and manage modern applications.

Sign up today and save $400 on onsite ticket prices with Early Bird Pricing. Hurry, prices increase after Friday, May 1st.

REGISTER NOW


Vote For Me At Open Source Bridge!

Author: A. Jesse Jiryu Davis ~ Published: April 16, 2015, 10:28 p.m.

Portland old town

This is a shameless plug. I'm not ashamed of how much I want to speak at Open Source Bridge this year—it's my favorite conference with my favorite people and I desperately want to speak there again. So vote for me! Add stars and comments to the three talks I proposed.

Add star

Cat-herd's Crook: Enforcing Standards in 10 Programming Languages.

I'm proposing this talk with my colleague and first-time speaker Samantha Ritter. We helped MongoDB specify and test our specifications for driver APIs and behaviors, and we persuaded dozens of open source programmers to implement these specifications the same. We want to show you how we herded all the cats in the same direction.

Dodge Disasters and March to Triumph as a Mentor.

If you're ambitious to advance in your career, or you care about your junior colleagues' advancement, then it is time for you to learn how to be a great mentor. Especially if you’re committed to diversity: mentorship is critical to the careers of women and minorities in tech. I have failed at mentoring, then succeeded. Learn from me and march to mentorship triumph.

How Do Python Coroutines Work?

I'll explain asyncio's new coroutine implementation in depth, including the mystical "yield from" statement. You'll know better than any of your peers how this amazing new programming model works. Plus, in a magical and entertaining feat of daring, I plan to live-code an asynchronous coroutine implementation before your very eyes!