A Complete Server: Persistence
Our server is off to the races. We have a functioning API that can service a web front-end and other clients.
This is a great start, but as of right now our data is static. While it's true that we can dynamically return data to a client with paging and filtering, the source of the data can never change.
If we were building a company on top of this service, it wouldn't be very useful. What we need now is persistence. A client should be able to add, remove, and modify the data that our API serves.
Currently our data is stored on disk. Our app uses the filesystem to retrieve data. We could continue to use the filesystem when we add persistence. We would do this by using
fs.writeFile() and other
fs methods, but that's not a good idea for a production app.
When we run apps in production, we want to have multiple servers running at the same time. We need this for a variety of reasons: redundancy, scale, zero-downtime deploys, and handling server maintenance. If we continue to use the filesystem for persistence, we'll run into a problem when we try to run on multiple servers.
Without special setup, each app would only have access to its local filesystem. This means that if a new product is created on server A, any client connected to server B wouldn't be able to access it. Effectively, we'd have multiple copies of the app, but they would all have different data.
We can use a database to get around this issue. We can have as many instances of our apps as we'd like, and they can all connect to the same database, ensuring they all have access to the same data.
Databases allow us to separate the data from our running code, so that our app does not have to run at the same location where our data is stored. This allows us to run instances of our apps wherever we'd like. We could even run a local copy that is identical to the production version (assuming our local version has access to the production database).
Databases do add additional complexity to an app. This is why we started with the filesystem -- it's the simplest way to handle data. However, the small increase in complexity comes with a lot more power and functionality. In the previous chapter, our code was inefficient. When we wanted to find products matching specific criteria, we had to iterate over our full collection to find them. We'll use database indexes so that these lookups are much more efficient.
There are many different databases to choose from when creating a production app, and they each have their own tradeoffs and are popular for different reasons. When the LAMP stack was popular, MySQL was the go-to. Over time, developers gravitated to other databases that did not use SQL like MongoDB, Redis, DynamoDB, and Cassandra.
Additionally, SQL databases are easy for people to work with directly. By using SQL (structured query language), one can write mostly human readable commands to store and retrieve data with a direct interface to the database server.
This has the distinctive property of allowing us to be flexible with what data we decide to persist in the database. Unlike with SQL, we would not need to first update the database's schema before adding new documents to a collection. MongoDB does not require each document in a collection to be uniform, nor is it required that object properties need to be defined in a schema before being added.
Of course, this additional freedom is itself a tradeoff. It is often very nice to have this flexibility. When creating an app, it is useful to evolve the data as changes are made. Unfortunately, if we aren't careful, we can wind up with a collection of documents that are inconsistent, and our app will need to contend with collections of objects that vary in subtle (or not so subtle) ways. For this reason, we will use Mongoose to get the best of both worlds.
Over time, developers have come to rely on SQL databases to enforce data validation. If a row is inserted with the wrong types of data in its fields, the database can refuse with an error. The app doesn't need to worry about handling that type of validation logic. MongoDB has no opinions on what types of data belongs in a document, and therefore data validation needs to be done in the app.
Additionally when persisting data with a SQL database, its common to have store relationships between rows. When retrieving data, the database can automatically "join" related rows to conveniently return all necessary at once. MongoDB does not handle relationships, and this means that an app using MongoDB would need its own logic to fetch related documents.
Mongoose is an ODM (object data modeling) library that can easily provide both of these features (and more) to our app. Later in this chapter, we'll show how to add custom validation logic that goes well beyond what a SQL database can provide and how to store and retrieve related documents with ease.
We're now ready to convert our app to use MongoDB to store our data. Luckily, we've built our app so that switching the persistence layer will be very easy.
We've built our app using modules with specific responsibilities:
server.jscreates the server instance and connects endpoints to route handlers.
api.jscontains functions are responsible for converting HTTP requests into HTTP responses using our model module,
products.jsis responsible for loading and manipulating data.
We don't need to change anything about
api.js because the only difference is where data is stored and how it is loaded.
api.js can continue to call the same exact methods from
products.js. To get to the same functionality we have using the file system, the only thing we need to do is change the functionality of the
Products.get() methods so that they load data from MongoDB instead of the filesystem.
That said, if we start by making those modifications, it will be difficult to check if they're working correctly. For
Products.get() to pull data from MongoDB, we'll need to make sure there are documents in the database. It will be more helpful to first add the
Products.create() method to make that easier.
Once we add the
Products.create() method, we can easily import our existing products from the
products.json file. From there we can update
Products.get(), and we can verify that they are working as expected. Finally, after we have those three methods, we'll add