Seed MongoDB using external API in Node.js

Through teaching myself the MERN Stack, and building Full Stack applications whether, for personal projects, or freelance work, I was always curious about how I would be able to seed my MongoDB collection using an external dataset.

More precisely — How can I make a fetch request run and populate the database? Is there a command or a mongoose function that will do that? I know that I’m basically trying to emulate Ruby on Rails with a seed file. In this blog, I will demonstrate in a few simple steps how easy this actually is. You will want to already have some exposure using Node.js and Mongoose.

Step by step:

1. Provide an external API — I will be using a free dataset provided by NYC Open Data — Parks Closure Status Due to COVID-19: Skate Parks
2. Build a Mongoose model with all the attributes you'd like to save for each document.
3. Set up a seed file which we will configure accordingly
4. Run our seed file

Let’s create a new file and build our model. I’ll call this file SkateSpot.js. We’ll set up our Mongoose Model like this, with an attribute of name, an address, a description, and coordinates.

const mongoose = require("mongoose");
const Schema = mongoose.Schema;
const SkateSpotSchema = new Schema({
   name: {
      type: String,
      required: true
   },
   address: {
      type: String,
   },
   description: {
      type: String,
   },
   location: {
      type: {
         type: String,
         enum: ['Point'],
      },
      coordinates: {
         type: [Number],
      }
   },
});
module.exports = SkateSpot = mongoose.model("skatespots", SkateSpotSchema);

At the bottom, we’ll export our model and also pass two arguments, the first argument is the name of our MongoDB collection, and the second argument is the schema of the new model that we just created.

Now well create our seed file. Let's just call it seed.js. We’ll import the model that we just created above into our seed file, as well as our database connection and the node-fetch API.

const mongoose = require("mongoose");
const SkateSpot = require("./models/SkateSpot");
const db = require("./config/keys").mongoURI;
const fetch = require("node-fetch");
let resultData;
let saveCounter = 0;
mongoose.connect(db)
.then(() => console.log("mongodb connection success"))
.catch(error => console.log(error));
const url = ['https://data.cityofnewyork.us/resource/pvvr-75zk.json']
url.map(async url => {
try{
   const response = await fetch(url);
   const json = await response.json();
   resultData = [...json];
   for (let i = 0; i < resultData.length; i++) {
      let skateSpot = new SkateSpot({
         name: resultData[i].name,
         description: resultData[i].status,
         location: { coordinates:      [resultData[i].polygon.coordinates[0][0][1] , resultData[i].polygon.coordinates[0][0][0]]}
      })
   skateSpot.save(() => {
      console.log("saved" + skateSpot)
      
      saveCounter++;
  
      if (saveCounter === resultData.length) {
         mongoose.disconnect()
         .then(() => console.log("saved succesfully and mongodb   disconnected"))
         .catch(error => console.log(error));
         }
      });
   }
} catch (error) {
   console.log(error);
}
})

First and foremost, we’ll want to connect to our database in this file to make sure that we can create new documents. Once connected, we’ll define a variable for a counter, and also for our URL which we want to fetch from (it’s an array of URLs just in case later we wanna add more external APIs to fetch from). Then well map the array and fetch to the API which will return a response as an array of objects.

Next, a for loop will map the response array of objects, and for each object, create an instance of our SkateSpot model. We’ll save the park name as our name, park status as our description, and coordinates at position [0][0][1] and [0][0][0] as our locations’ coordinates. After each iteration, SAVE our mongoose object to our Database and increase the saveCounter variable. If our saveCounter is equal to the length of the array of results, DISCONNECT from the database.

Now we’re ready to run our seed file. It’s as simple as that! You may want to console.log() your objects first to make sure that the documents are being formatted correctly. When you're all set up, run the command below.

$node seed.js

That’s all! It’s actually quite simple once you have your database connection and model set up in your app. A new seed file takes the URL you want to fetch from and maps through the response. For each object, we create a new record and document in our database using the mongoose model. I hope this walkthrough helped you seed your database with data from an external API. The picture below also gives a pretty solid representation of what's going on behind the scenes. Best of luck!

Mongoose Model and Database relationship

Mongoose

Barak Saidoff | Full Stack Web Developer