Node.js is a JavaScript runtime environment that runs JavaScript code in web servers–the so-called web backend (server side)–instead of using a second language like Python or Ruby for server-side programs. The JavaScript language is already used in the modern frontend side of web applications, interacting with the HTML and CSS of the interface that the user interacts with a web browser. Using Node.js in tandem with JavaScript in the browser offers the possibility of just one programming language for the whole application.
The main reason for the existence of Node.js is the way it handles multiple concurrent connections in the backend. One of the most common ways that a web application server handles connections is through the execution of multiple processes. When you open a desktop application in your computer, a process starts and uses a lot of resources. Now think about when thousands of users are doing the same thing in a large web application.
Node.js avoids this problem using a design called the event loop, which is an internal loop that continously checks for incoming tasks to be computed. Thanks to the widespread use of JavaScript and the ubiquity of web technologies, Node.js has seen a huge adoption in both small and large applications. There are other characteristics that also helped Node.js to be widely adopted, such as asynchronous and non-blocking input/output (I/O) processing, which is explained later in this lesson.
The Node.js environment uses a JavaScript engine to interpret and execute JavaScript code on the server side or on the desktop. In these conditions, the JavaScript code that the programmer writes is parsed and compiled just-in-time to execute the machine instructions generated by the original JavaScript code.
Note
As you progress through these lessons about Node.js, you may notice that the Node.js JavaScript is not exactly the same as the one that runs on the browser (which follows the ECMAScript specification), but is quite similar.This section and the following examples assume that Node.js is already installed on your Linux operating system, and that the user already has basic skills such as how to execute commands in the terminal.
To run the following examples, create a working directory called node_examples. Open a terminal prompt and type node. If you have correctly installed Node.js, it will present a > prompt where you can test JavaScript commands interactively. This kind of environment is called REPL, for “read, evaluate, print, and loop”. Type the following input (or some other JavaScript statements) at the > prompts. Press the Enter key after each line, and the REPL environment will return the results of its actions:
> let array = ['a', 'b', 'c', 'd'];
undefined
> array.map( (element, index) => (`Element: ${element} at index: ${index}`));
[
'Element: a at index: 0',
'Element: b at index: 1',
'Element: c at index: 2',
'Element: d at index: 3'
]
>The snippet was written using ES6 syntax, which offers a map function to iterate over the array and print the results using string templates. You can write pretty much any command that is valid. To exit the Node.js terminal, type .exit, remembering to include the initial period.
For longer scripts and modules, it is more convenient to use a text editor such as VS Code, Emacs, or Vim. You can save the two lines of code just shown (with a little modification) in a file called start.js:
let array = ['a', 'b', 'c', 'd'];
array.map( (element, index) => ( console.log(`Element: ${element} at index: ${index}`)));Then you can run the script from the shell to produce the same results as before:
$ node ./start.js
Element: a at index: 0
Element: b at index: 1
Element: c at index: 2
Element: d at index: 3Before diving into some more code, we are going to get an overview of how Node.js works, using its single thread execution environment and the event loop.
It is hard to tell how much time your Node.js program will take to handle a request. Some requests may be short—perhaps just looping through variables in memory and returning them—whereas others may require time-consuming activities such as opening a file on the system or issuing a query to a database and waiting for the results. How does Node.js handle this uncertainty? The event loop is the answer.
Imagine a chef doing multiple tasks. Baking a cake is one task that requires a lot of time for the oven to cook it. The chef does not stay there waiting for the cake to be ready and then set out to make some coffee. Instead, while the oven bakes the cake, the chef makes coffee and other tasks in parallel. But the cook is always checking whether it is the right time to switch focus to a specific task (making coffee), or to get the cake out of oven.
The event loop is like the chef who is constantly aware of the surrounding activities. In Node.js, an “event-checker” is always checking for operations that have completed or are waiting to be processed by the JavaScript engine.
Using this approach, an asynchronous and long operation does not block other quick operations that come after. This is because the event loop mechanism is always checking whether that long task, such as an I/O operation, is already done. If not, Node.js can continue to process other tasks. Once the background task is complete, the results are returned and the application on top of Node.js can use a trigger function (callback) to further process the output.
Because Node.js avoids the use of multiple threads, as other environments do, it is called a single-threaded environment, and therefore a non-blocking approach is of the utmost importance. This is why Node.js uses an event loop. For compute-intensive tasks, Node.js is not among the best tools, however: there are other programming languages and environments that address these problems more efficiently.
In the following sections, we will take a closer look at callback functions. For now, understand that callback functions are triggers that are executed upon the completion of a predefined operation.
It is a best practice to break down complex functionality and extensive pieces of code into smaller parts. Doing this modularization helps to better organize the codebase, abstract away the implementations, and avoid complicated engineering problems. To meet those necessities, programmers package blocks of source code to be consumed by other internal or external parts of code.
Consider the example of a program that calculates the volume of a sphere. Open up your text editor and create a file named volumeCalculator.js containing the following JavaScript:
const sphereVol = (radius) => {
return 4 / 3 * Math.PI * radius
}
console.log(`A sphere with radius 3 has a ${sphereVol(3)} volume.`);
console.log(`A sphere with radius 6 has a ${sphereVol(6)} volume.`);Now, execute the file using Node:
$ node volumeCalculator.js
A sphere with radius 3 has a 113.09733552923254 volume.
A sphere with radius 6 has a 904.7786842338603 volume.Here, a simple function was used to compute the volume of a sphere, based on its radius. Imagine that we also need to calculate the volume of a cylinder, cone, and so on: we quickly notice that those specific functions must be added to the volumeCalculator.js file, which can become a huge collection of functions. To better organize the structure, we can use the idea behind modules as packages of separated code.
In order to do that, create a separated file called polyhedrons.js:
const coneVol = (radius, height) => {
return 1 / 3 * Math.PI * Math.pow(radius, 2) * height;
}
const cylinderVol = (radius, height) => {
return Math.PI * Math.pow(radius, 2) * height;
}
const sphereVol = (radius) => {
return 4 / 3 * Math.PI * Math.pow(radius, 3);
}
module.exports = {
coneVol,
cylinderVol,
sphereVol
}Now, in the volumeCalculator.js file, delete the old code and replace it with this snippet:
const polyhedrons = require('./polyhedrons.js');
console.log(`A sphere with radius 3 has a ${polyhedrons.sphereVol(3)} volume.`);
console.log(`A cylinder with radius 3 and height 5 has a ${polyhedrons.cylinderVol(3, 5)} volume.`);
console.log(`A cone with radius 3 and height 5 has a ${polyhedrons.coneVol(3, 5)} volume.`);And then run the filename against the Node.js environment:
$ node volumeCalculator.js
A sphere with radius 3 has a 113.09733552923254 volume.
A cylinder with radius 3 and height 5 has a 141.3716694115407 volume.
A cone with radius 3 and height 5 has a 47.12388980384689 volume.In the Node.js environment, every source code file is considered a module, but the word “module” in Node.js indicates a package of code wrapped up as in the previous example. By using modules, we abstracted the volume functions away from the main file, volumeCalculator.js, thus reducing its size and making it easier to apply unit tests, which are a good practice when developing real world applications.
Now that we know how modules are used in Node.js, we can use one of the most important tools: the Node Package Manager (NPM).
One of the main jobs of NPM is to manage, download, and install external modules into the project or in the operating system. You can initialize a node repository with the command npm init.
NPM will ask the default questions about the name of your repository, version, description, and so on. You can bypass these steps using npm init --yes, and the command will automatically generate a package.json file that describes the properties of your project/module.
Open the package.json file in your favorite text editor and you will see a JSON file containing properties such as keywords, script commands to use with NPM, a name, etc.
One of those properties is the dependencies that are installed in your local repository. NPM will add the name and version of these dependencies into package.json, along with package-lock.json, another file used as fallback by NPM in case package.json fails.
Type the following in your terminal:
$ npm i dayjsThe i flag is a shortcut for the argument install. If you are connected to the internet, NPM will search for a module named dayjs in the remote repository of Node.js, download the module, and install it locally. NPM will also add this dependency to your package.json and package-lock.json files. Now you can see that there is a folder called node_modules, which contains the installed module along with other modules if they are needed. The node_modules directory contains the actual code that is going to be used when the library is imported and called. However, this folder is not saved in versioning systems using Git, since the package.json file provides all the dependencies used. Another user can take the package.json file and simply run npm install in their own machine, where NPM will create a node_modules folder with all the dependencies in the package.json, thus avoiding version control for the thousands of files available on the NPM repository.
Now that the dayjs module is installed in the local directory, open the Node.js console and type the following lines:
const dayjs = require('dayjs');
dayjs().format('YYYY MM-DDTHH:mm:ss')The dayjs module is loaded with the require keyword. When a method from the module is called, the library takes the current system datetime and outputs it in the specified format:
2020 11-22T11:04:36This is the same mechanism used in the previous example, where the Node.js runtime loads the third party function into the code.
Because Node.js controls the back end of web applications, one of its core tasks is to handle HTTP requests.
Here is a summary of how web servers handled incoming HTTP requests. The functionality of the server is to listen for requests, determine as quickly as possible what response each needs, and return that response to the sender of the request. This application must receive an incoming HTTP request triggered by the user, parse the request, perform the calculation, generate the response, and send it back. An HTTP module such as Node.js is used because it simplifies those steps, alowing a web programmer to focus on the application itself.
Consider the following example that implements this very basic functionality:
const http = require('http');
const url = require('url');
const hostname = '127.0.0.1';
const port = 3000;
const server = http.createServer((req, res) => {
const queryObject = url.parse(req.url,true).query;
let result = parseInt(queryObject.a) + parseInt(queryObject.b);
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');
res.end(`Result: ${result}\n`);
});
server.listen(port, hostname, () => {
console.log(`Server running at http://${hostname}:${port}/`);
});Save these contents in a file called basic_server.js and run it through a node command. The terminal running Node.js will display the following message:
Server running at http://127.0.0.1:3000/Then visit the following URL in your web browser: http://127.0.0.1:3000/numbers?a=2&b=17
Node.js is running a web server in your computer, and using two modules: http and url. The http module sets up a basic HTTP server, processes the incoming web requests, and hands them to our simple application code. The URL module parses the arguments passed in the URL, converts them into an integer format, and performs the addition operation. The http module then sends the response as text to the web browser.
In a real web application, Node.js is commonly used to process and retrieve data, usually from a database, and return the processed information to the front end to display. But the basic application in this lesson concisely shows how Node.js makes use of modules to handle web requests as a web server.
Express.js, or simply Express, is a popular framework that runs on Node.js and is used to write HTTP servers that handle requests from web application clients. Express supports many ways to read parameters sent over HTTP.
Initial Server Script To demonstrate Express’s basic features for receiving and handling requests, let’s simulate an application that requests some information from the server. In particular, the example server:
- Provides an
echofunction, which simply returns the message sent by the client. - Tells the client its IP address upon request.
- Uses cookies to identify known clients.
The first step is to create the JavaScript file that will operate as the server. Using npm, create a directory called myserver with the JavaScript file:
$ mkdir myserver
$ cd myserver/
$ npm initFor the entry point, any filename can be used. Here we will use the default filename: index.js. The following listing shows a basic index.js file that will be used as the entry point for our server:
const express = require('express')
const app = express()
const host = "myserver"
const port = 8080
app.get('/', (req, res) => {
res.send('Request received')
})
app.listen(port, host, () => {
console.log(`Server ready at http://${host}:${port}`)
})Some important constants for the server configuration are defined in the first lines of the script. The first two, express and app, correspond to the inclused express module and an instance of this module that runs our application. We will add the actions to be performed by the server to the app object.
The other two constants, host and port, define the host and communication port associated to the server.
If you have a publicly accessible host, use its name instead of myserver as the value of host. If you don’t provide the host name, Express will default to localhost, the computer where the application runs. In that case, no outside clients will be able to reach the program, which may be fine for testing but offers little value in production.
The port needs to be provided, or the server will not start.
This script attaches only two procedures to the app object: the app.get() action that answers requests made by clients through HTTP GET, and the app.listen() call, which is required to activate the server and assigns it a host and port.
To start the server, just run the node command, providing the script name as an argument:
$ node index.jsAs soon as the message Server ready at http://myserver:8080 appears, the server is ready to receive requests from an HTTP client. Requests can be made from a browser on the same computer where the server is running, or from another machine that can access the server.
All transaction details we’ll see here are shown in the browser if you open a window for the developer console. Alternatively, the curl command can be used for HTTP communication and allows you to inspect connection details more easily. If you are not familiar with the shell command line, you can create an HTML form to submit requests to a server.
The following example shows how to use the curl command on the command line to make an HTTP request to the newly deployed server:
$ curl http://myserver:8080 -v
* Trying 192.168.1.225:8080...
* TCP_NODELAY set
* Connected to myserver (192.168.1.225) port 8080 (#0)
> GET / HTTP/1.1
> Host: myserver:8080
> User-Agent: curl/7.68.0
>Accept: /
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: text/html; charset=utf-8
< Content-Length: 16
< ETag: W/"10-1WVvDtVyAF0vX9evlsFlfiJTT5c"
< Date: Fri, 02 Jul 2021 14:35:11 GMT
< Connection: keep-alive
<
* Connection #0 to host myserver left intact
Request receivedThe -v option of the curl command displays all the request and response headers, as well as other debugging information. The lines starting with > indicate the request headers sent by the client and the lines starting with < indicate the response headers sent by the server. Lines starting with * are information generated by curl itself. The content of the response is displayed only at the end, which in this case is the line Request received.
The service’s URL, which in this case contains the server’s hostname and port (http://myserver:8080), were given as arguments to the curl command. Because no directory or filename is given, these default to the root directory /. The slash turns up as the request file in the > GET / HTTP/1.1 line, which is followed in the output by the hostname and port.
In addition to displaying HTTP connection headers, the curl command assists application development by allowing you to send data to the server using different HTTP methods and in different formats. This flexibility makes it easier to debug any problems and implement new features on the server.
The requests the client can make to the server depend on what routes have been defined in the index.js file. A route specifies an HTTP method and defines a path (more precisely, a URI) that can be requested by the client.
So far, the server has only one route configured:
app.get('/', (req, res) => {
res.send('Request received')
})Even though it is a very simple route, simply returning a plain text message to the client, it is enough to identify the most important components that are used to structure most routes:
- The HTTP method served by the route. In the example, the HTTP
GETmethod is indicated by thegetproperty of theappobject. - The path served by the route. When the client does not specify a path for the request, the server uses the root directory, which is the base directory set aside for use by the web server. A later example in this chapter uses the path
/echo, which corresponds to a request made tomyserver:8080/echo. - The function executed when the server receives a request on this route, usually written in abbreviated form as an arrow function because the syntax
=>points to the definition of the nameless function. Thereqparameter (short for “request”) andresparameter (short for “response”) give details about the connection, passed to the function by the app instance itself.
To extend the functionality of our test server, let’s see how to define a route for the HTTP POST method. It’s used by clients when they need to send extra data to the server beyond those included in the request header. The --data option of the curl command automatically invokes the POST method, and includes content that will be sent to the server via POST. The POST / HTTP/1.1 line in the following output shows that the POST method was used. However, our server defined only a GET method, so an error occurs when we use curl to send a request via POST:
$ curl http://myserver:8080/echo --data message="This is the POST request body"
* Trying 192.168.1.225:8080...
* TCP_NODELAY set
* Connected to myserver (192.168.1.225) port 8080 (#0)
> POST / HTTP/1.1
> Host: myserver:8080
> User-Agent: curl/7.68.0
>Accept: /
> Content-Length: 37
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 37 out of 37 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< X-Powered-By: Express
< Content-Security-Policy: default-src 'none'
< X-Content-Type-Options: nosniff
< Content-Type: text/html; charset=utf-8
< Content-Length: 140
< Date: Sat, 03 Jul 2021 02:22:45 GMT
< Connection: keep-alive
<
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Error</title>
</head>
<body>
<pre>Cannot POST /</pre>
</body>
</html>
* Connection #0 to host myserver left intactIn the previous example, running curl with the parameter --data message="This is the POST request body" is equivalent to submitting a form containing the text field named message, filled with This is the POST request body.
Because the server is configured with only one route for the / path, and that route only responds to the HTTP GET method, so the response header has the line HTTP/1.1 404 Not Found. In addition, Express automatically generated a short HTML response with the warning Cannot POST.
Having seen how to generate a POST request through curl, let’s write an Express program that can successfully handle the request.
First, note that the Content-Type field in the request header says that the data sent by the client is in the application/x-www-form-urlencoded format. Express does not recognize that format by default, so we need to use the express.urlencoded module. When we include this module, the req object—passed as a parameter to the handler function—has the req.body.message property set, which corresponds to the message field sent by the client. The module is loaded with app.use, which should be placed before the declaration of routes:
const express = require('express')
const app = express()
const host = "myserver"
const port = 8080
app.use(express.urlencoded({ extended: true }))Once this is done, it would be enough to change app.get to app.post in the existing route to fulfill requests made via POST and to recover the request body:
app.post('/', (req, res) => {
res.send(req.body.message)
})Instead of replacing the route, another possibility would be to simply add this new route, because Express identifies the HTTP method in the request header and uses the appropriate route. Because we are interested in adding more than one functionality to this server, it is convenient to separate each one with its own path, such as /echo and /ip.
Having defined which HTTP method will respond to the request, we now need to define a specific path for the resource and a function that processes and generates a response to the client.
To expand the echo functionality of the server, we can define a route using the POSTmethod with the path/echo`:
app.post('/echo', (req, res) => {
res.send(req.body.message)
})The req parameter of the handler function contains all the request details stored as properties. The content of the message field in the request body is available in the req.body.message property. The example simply sends this field is sent back to the client through the res.send(req.body.message) call.
Remember that the changes you make take effect only after the server is restarted. Because you are running the server from a terminal window during the examples in this chapter, you can shut down the server by pressing kbd:[Ctrl+C] on that terminal. Then rerun the server through the node index.js command. The response obtained by the client to the curl request we showed before is now successful:
$ curl http://myserver:8080/echo --data message="This is the POST request body"
This is the POST request bodyIt might be excessive to use the HTTP POST method if only short text messages like the one used in the example will be sent. In such cases, data can be sent in a query string that starts with a question mark. Thus, the string ?message=This+is+the+message could be included within the request path of the HTTP GET method. The fields used in the query string are available to the server in the req.query property. Therefore, a field named message is available in the req.query.message property.
Another way to send data via the HTTP GET method is to use Express’s route parameters:
app.get('/echo/:message', (req, res) => {
res.send(req.params.message)
})The route in this example matches requests made with the GET method using the path /echo/:message, where :message is a placeholder for any term sent with that label by the client. These parameters are accessible in the req.params property. With this new route, the server’s echo function can be requested more succinctly by the client:
$ curl http://myserver:8080/echo/hello
helloIn other situations, the information the server needs to process the request do not need to be explicitly provided by the client. For instance, the server has another way to retrieve the client’s public IP address. That information is present in the req object by default, in the req.ip property:
app.get('/ip', (req, res) => {
res.send(req.ip)
})Now the client can request the /ip path with the GET method to find its own public IP address:
$ curl http://myserver:8080/ip
187.34.178.12Other properties of the req object can be modified by the client, especially the request headers available in req.headers. The req.headers.user-agent property, for example, identifies which program is making the request. Although it is not common practice, the client can change the contents of this field, so the server should not use it to reliably identify a particular client. It is even more important to validate the data explicitly provided by the client, to avoid inconsistencies in boundaries and formats that could adversely affect the application.
As we’ve seen in previous examples, the res parameter is responsible for returning a response to the client. Furthermore, the res object can change other aspects of the response. You may have noticed that, although the responses we’ve implemented so far are just brief plain text messages, the Content-Type header of the responses is using text/html; charset=utf-8. Although this does not prevent the plain text response from being accepted, it will be more correct if we redefine this field in the response header to text/plain with the setting res.type('text/plain').
Other types of response adjustments involve using cookies, which allow the server to identify a client that has previously made a request. Cookies are important for advanced features, such as creating private sessions that associate requests to a specific user, but here we’ll just look at a simple example of how to use a cookie to identify a client that has previously accessed the server.
Given the modularized design of Express, cookie management must be installed with the npm command before being used in the script:
$ npm install cookie-parserAfter installation, cookie management must be included in the server script. The following definition should be included near the beginning of the file:
const cookieParser = require('cookie-parser')
app.use(cookieParser())To illustrate the use of cookies, let’s modify the route’s handler function with the``/root path that already exists in the script. Thereq` object has a `req.cookies` property, where cookies sent in the request header are kept. The res object, on the other hand, has a `res.cookie()` method that creates a new cookie to be sent to the client. The handler function in the following example checks whether a cookie with the name `known` exists in the request. If such a cookie does not exist, the server assumes that this is a first-time visitor and sends it a cookie with that name through the `res.cookie('known', '1')` call. We arbitrarily assign the value `1` to the cookie because it is supposed to have some content, but the server doesn’t consult that value. This application just assumes that the simple presence of the cookie indicates that the client has already requested this route before:
app.get('/', (req, res) => {
res.type('text/plain')
if ( req.cookies.known === undefined ){
res.cookie('known', '1')
res.send('Welcome, new visitor!')
}
else
res.send('Welcome back, visitor');
})By default, curl does not use cookies in transactions. But it has options to store (-c cookies.txt) and send stored cookies (-b cookies.txt):
$ curl http://myserver:8080/ -c cookies.txt -b cookies.txt -v
* Trying 192.168.1.225:8080...
* TCP_NODELAY set
* Connected to myserver (192.168.1.225) port 8080 (#0)
> GET / HTTP/1.1
> Host: myserver:8080
> User-Agent: curl/7.68.0
>Accept: /
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: text/plain; charset=utf-8
* Added cookie known="1" for domain myserver, path /, expire 0
< Set-Cookie: known=1; Path=/
< Content-Length: 21
< ETag: W/"15-l7qrxcqicl4xv6EfA5fZFWCFrgY"
< Date: Sat, 03 Jul 2021 23:45:03 GMT
< Connection: keep-alive
<
* Connection #0 to host myserver left intact
Welcome, new visitor!Because this command was the first access since cookies were implemented on the server, the client did not have any cookies to include in the request. As expected, the server did not identify the cookie in the request and therefore included the cookie in the response headers, as indicated in the Set-Cookie: known=1; Path=/ line of the output. Since we have enabled cookies in curl, a new request will include the cookie known=1 in the request headers, allowing the server to identify the cookie’s presence:
$ curl http://myserver:8080/ -c cookies.txt -b cookies.txt -v
* Trying 192.168.1.225:8080...
* TCP_NODELAY set
* Connected to myserver (192.168.1.225) port 8080 (#0)
> GET / HTTP/1.1
> Host: myserver:8080
> User-Agent: curl/7.68.0
>Accept: /
> Cookie: known=1
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Powered-By: Express
< Content-Type: text/plain; charset=utf-8
< Content-Length: 21
< ETag: W/"15-ATq2flQYtLMYIUpJwwpb5SjV9Ww"
< Date: Sat, 03 Jul 2021 23:45:47 GMT
< Connection: keep-alive
<
* Connection #0 to host myserver left intact
Welcome back, visitorThe developer should be aware of potential vulnerabilities when using cookies to identify clients making requests. Attackers can use techniques such as cross-site scripting (XSS) and cross-site request forgery (CSRF) to steal cookies from a client and thereby impersonate them when making a request to the server. Generally speaking, these types of attacks use non-validated comment fields or meticulously constructed URLs to insert malicious JavaScript code into the page. When executed by an authentic client, this code can copy valid cookies and store them or forward them to another destination.
Therefore, especially in professional applications, it is important to install and use more specialized Express features, known as middleware. The express-session or cookie-session module provide more complete and secure control over session and cookie management. These components enable extra controls to prevent cookies from being diverted from their original issuer.
Web servers have very versatile mechanisms to produce responses to client requests. For some requests, it’s enough for the web server to provide a static, unprocessed reponse, because the requested resource is the same for any client. For instance, when a client requests an image that is accessible to everyone, it is enough for the server to send the file containing the image.
But when responses are dynamically generated, they may need to be better structured than simple lines written in the server script. In such cases, it is convenient for the web server to be able to generate a complete document, which can be interpreted and rendered by the client. In the context of web application development, HTML documents are commonly created as templates and kept separate from the server script, which inserts dynamic data in predetermined places in the appropriate template and then sends the formatted response to the client.
Web applications often consume both static and dynamic resources. An HTML document, even if it was dynamically generated, may have references to static resources such as CSS files and images. To demonstrate how Express helps handle this kind of demand, we’ll first set up an example server that delivers static files and then implement routes that generate structured, template-based responses.
The first step is to create the JavaScript file that will run as a server. Let’s follow the same pattern covered in previous lessons to create a simple Express application: first create a directory called server and then install the base components with the npm command:
$ mkdir server
$ cd server/
$ npm init
$ npm install expressFor the entry point, any filename can be used, but here we will use the default filename: index.js. The following listing shows a basic index.js file that will be used as a starting point for our server:
const express = require('express')
const app = express()
const host = "myserver"
const port = 8080
app.listen(port, host, () => {
console.log(`Server ready at http://${host}:${port}`)
})You don’t have to write explicit code to send a static file. Express has middleware for this purpose, called express.static. If your server needs to send static files to the client, just load the express.static middleware at the beginning of the script:
app.use(express.static('public'))The public parameter indicates the directory that stores files the client can request. Paths requested by clients must not include the public directory, but only the filename, or the path to the file relative to the public directory. To request the public/layout.css file, for example, the client makes a request to /layout.css.
While sending static content is straightforward, dynamically generated content can vary widely. Creating dynamic responses with short messages makes it easy to test applications in their initial stages of development. For example, the following is a test route that just send back to the client a message that it sent by the HTTP POST method. The response can just replicate the message content in plain text, without any formatting:
app.post('/echo', (req, res) => {
res.send(req.body.message)
})A route like this is a good example to use when learning Express and for diagnostics purposes, where a raw response sent with res.send() is enough. But a useful server must be able to produce more complex responses. We’ll move on now to develop that more sophisticated type of route.
Our new application, instead of just sending back the contents of the current request, maintains a complete list of the messages sent in previous requests by each client and sends back each client’s list when requested. A response merging all messages is an option, but other formatted output modes are more appropriate, especially as responses become more elaborate.
To receive and store client messages sent during the current session, first we need to include extra modules for handling cookies and data sent via the HTTP POST method. The only purpose of the following example server is to log messages sent via POST and display previously sent messages when the client issues a GET request. So there are two routes for the / path. The first route fulfills requests made with the HTTP POST method and the second fulfills requests made with the HTTP GET method:
const express = require('express')
const app = express()
const host = "myserver"
const port = 8080
app.use(express.static('public'))
const cookieParser = require('cookie-parser')
app.use(cookieParser())
const { v4: uuidv4 } = require('uuid')
app.use(express.urlencoded({ extended: true }))
// Array to store messages
let messages = []
app.post('/', (req, res) => {
// Only JSON enabled requests
if ( req.headers.accept != "application/json" )
{
res.sendStatus(404)
return
}
// Locate cookie in the request
let uuid = req.cookies.uuid
// If there is no uuid cookie, create a new one
if ( uuid === undefined )
uuid = uuidv4()
// Add message first in the messages array
messages.unshift({uuid: uuid, message: req.body.message})
// Collect all previous messages for uuid
let user_entries = []
messages.forEach( (entry) => {
if ( entry.uuid == req.cookies.uuid )
user_entries.push(entry.message)
})
// Update cookie expiration date
let expires = new Date(Date.now());
expires.setDate(expires.getDate() + 30);
res.cookie('uuid', uuid, { expires: expires })
// Send back JSON response
res.json(user_entries)
})
app.get('/', (req, res) => {
// Only JSON enabled requests
if ( req.headers.accept != "application/json" )
{
res.sendStatus(404)
return
}
// Locate cookie in the request
let uuid = req.cookies.uuid
// Client's own messages
let user_entries = []
// If there is no uuid cookie, create a new one
if ( uuid === undefined ){
uuid = uuidv4()
}
else {
// Collect messages for uuid
messages.forEach( (entry) => {
if ( entry.uuid == req.cookies.uuid )
user_entries.push(entry.message)
})
}
// Update cookie expiration date
let expires = new Date(Date.now());
expires.setDate(expires.getDate() + 30);
res.cookie('uuid', uuid, { expires: expires })
// Send back JSON response
res.json(user_entries)
})
app.listen(port, host, () => {
console.log(`Server ready at http://${host}:${port}`)
})We kept the static files configuration at the top, because it will soon be useful to provide static files such as layout.css. In addition to the cookie-parser middleware introduced in the previous chapter, the example also includes the uuid middleware to generate a unique identification number passed as a cookie to each client that sends a message. If not already installed in the example server directory, these modules can be installed with the command npm install cookie-parser uuid.
The global array called messages stores the messages sent by all clients. Each item in this array consists of an object with the properties uuid and message.
What’s really new in this script is the res.json() method, used at the end of the two routes to generate a response in JSON format with the array containing the messages already sent by the client:
// Send back JSON response
res.json(user_entries)JSON is a plain text format that allows you to group a set of data into a single structure that is associative: that is, content is expressed as keys and values. JSON is particularly useful when responses are going to be processed by the client. Using this format, a JavaScript object or array can be easily reconstructed on the client side with all the properties and indexes of the original object on the server.
Because we are structuring each message in JSON, we refuse requests that do not contain application/json in their accept header:
// Only JSON enabled requests
if ( req.headers.accept != "application/json" )
{
res.sendStatus(404)
return
}A request made with a plain curl command to insert a new message will not be accepted, because curl by default does not specify application/json in the accept header:
$ curl http://myserver:8080/ --data message="My first message" -c cookies.txt -b cookies.txt
Not FoundThe -H "accept: application/json" option changes the request header to specify the response’s format, which this time will be accepted and answered in the specified format:
$ curl http://myserver:8080/ --data message="My first message" -c cookies.txt -b cookies.txt -H "accept: application/json"
["My first message"]Getting messages using the other route is done in a similar way, but this time using the HTTP GET method:
$ curl http://myserver:8080/ -c cookies.txt -b cookies.txt -H "accept: application/json"
["Another message","My first message"]Responses in formats such as JSON are convenient for communicating between programs, but the main purpose of most web application servers is to produce HTML content for human consumption. Embedding HTML code within JavaScript code is not a good idea, because mixing languages in the same file makes the program more susceptible to errors and harms the maintenance of the code.
Express can work with different template engines that separate out the HTML for dynamic content; the full list can be found at the Express template engines site. One of the most popular template engines is Embedded JavaScript (EJS), which allows you to create HTML files with specific tags for dynamic content insertion.
Like other Express components, EJS needs to be installed in the directory where the server is running:
$ npm install ejsNext, the EJS engine must be set as the default renderer in the server script (near the beginning of the index.js file, before the route definitions):
app.set('view engine', 'ejs')The response generated with the template is sent to the client with the res.render() function, which receives as parameters the template file name and an object containing values that will be accessible from within the template. The routes used in the previous example can be rewritten to generate HTML responses as well as JSON:
app.post('/', (req, res) => {
let uuid = req.cookies.uuid
if ( uuid === undefined )
uuid = uuidv4()
messages.unshift({uuid: uuid, message: req.body.message})
let user_entries = []
messages.forEach( (entry) => {
if ( entry.uuid == req.cookies.uuid )
user_entries.push(entry.message)
})
let expires = new Date(Date.now());
expires.setDate(expires.getDate() + 30);
res.cookie('uuid', uuid, { expires: expires })
if ( req.headers.accept == "application/json" )
res.json(user_entries)
else
res.render('index', {title: "My messages", messages: user_entries})
})
app.get('/', (req, res) => {
let uuid = req.cookies.uuid
let user_entries = []
if ( uuid === undefined ){
uuid = uuidv4()
}
else {
messages.forEach( (entry) => {
if ( entry.uuid == req.cookies.uuid )
user_entries.push(entry.message)
})
}
let expires = new Date(Date.now());
expires.setDate(expires.getDate() + 30);
res.cookie('uuid', uuid, { expires: expires })
if ( req.headers.accept == "application/json" )
res.json(user_entries)
else
res.render('index', {title: "My messages", messages: user_entries})
})Note that the format of the response depends on the accept header found in the request:
if ( req.headers.accept == "application/json" )
res.json(user_entries)
else
res.render('index', {title: "My messages", messages: user_entries})A response in JSON format is sent only if the client explicitly requests it. Otherwise, the response is generated from the index template. The same user_entries array feeds both the JSON output and the template, but the object used as a parameter for the latter also has the title: "My messages" property, which will be used as a title inside the template.
Like static files, the files containing HTML templates reside in their own directory. By default, EJS assumes the template files are in the views/ directory. In the example, a template named index was used, so EJS looks for the views/index.ejs file. The following listing is the content of a simple views/index.ejs template that can be used with the example code:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title><%= title %></title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="/layout.css">
</head>
<body>
<div id="interface">
<form action="/" method="post">
<p>
<input type="text" name="message">
<input type="submit" value="Submit">
</p>
</form>
<ul>
<% messages.forEach( (message) => { %>
<li><%= message %></li>
<% }) %>
</ul>
</div>
</body>
</html>The first special EJS tag is the <title> element in the <head> section:
<%= title %>
During the rendering process, this special tag will be replaced by the value of the title property of the object passed as a parameter to the res.render() function.
Most of the template is made up of conventional HTML code, so the template contains the HTML form for sending new messages. The test server responds to the HTTP GET and POST methods for the same path /, hence the action="/" and method="post" attributes in the form tag.
Other parts of the template are a mixture of HTML code and EJS tags. EJS has tags for specific purposes within the template:
<% … %>
- Inserts flow control. No content is directly inserted by this tag, but it can be used with JavaScript structures to choose, repeat, or suppress sections of HTML. Example starting a loop:
<% messages.forEach( (message) => { %>
<%# … %>
- Defines a comment, whose content is ignored by the parser. Unlike comments written in HTML, these comments are not visible to the client.
<%= … %>
- Inserts the escaped content of the variable. It is important to escape unknown content to avoid JavaScript code execution, which can open loopholes for cross-site Scripting (XSS) attacks. Example:
<%= title %>
<%- … %>
- Inserts the content of the variable without escaping.
The mix of HTML code and EJS tags is evident in the snippet where client messages are rendered as an HTML list:
<ul>
<% messages.forEach( (message) => { %>
<li><%= message %></li>
<% }) %>
</ul>In this snippet, the first <% … %> tag starts a forEach statement that loops through all the elements of the message array. The <% and %> delimiters let you control the snippets of HTML. A new HTML list item, <li><%= message %></li>, will be produced for each element of messages. With these changes, the server will send the response in HTML when a request like the following is received:
$ curl http://myserver:8080/ --data message="This time" -c cookies.txt -b cookies.txt
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>My messages</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="/layout.css">
</head>
<body>
<div id="interface">
<form action="/" method="post">
<p>
<input type="text" name="message">
<input type="submit" value="Submit">
</p>
</form>
<ul>
<li>This time</li>
<li>in HTML</li>
</ul>
</div>
</body>
</html>The separation between the code for processing the requests and the code for presenting the response makes the code cleaner and allows a team to divide application development between people with distinct specialties. A web designer, for example, can focus on the template files in views/ and related stylesheets, which are provided as static files stored in the public/ directory on the example server.
Although you can write your own functions to implement persistent storage, it may be more convenient to use a database management system to speed up development and ensure better security and stability for table-formatted data. The most popular strategy for storing data organized in interrelated tables, especially when those tables are heavily queried and updated, is to install a relational database that supports Structured Query Language (SQL), a language geared towards relational databases. Node.js supports various SQL database management systems. Following the principles of portability and user space execution adopted by Node.js Express, SQLite is an appropriate choice for persistent storage of data used by this type of HTTP server.
The Structured Query Language is specific to databases. Write and read operations are expressed in sentences called statements and queries. Both statements and queries are made up of clauses, which define the conditions for executing the operation.
Names and email addresses, for example, can be stored in a database table that contains name and email fields. A database can contain several tables, so each table must have a unique name. If we use the name contacts for the names and emails table, a new record can be inserted with the following statement:
INSERT INTO contacts (name, email) VALUES ("Carol", "carol@example.com");This insertion statement is composed of the INSERT INTO clause, which defines the table and fields where the data will be inserted. The second clause, VALUES, sets the values that will be inserted. It is not necessary to capitalize clauses, but it is common practice, so as to better recognize SQL keywords within a statement or query.
A query on the contacts table is done in a similar way, but using the SELECT clause:
SELECT email FROM contacts;
dave@example.com
carol@example.comIn this case, the SELECT email clause selects one field from entries in the contacts table. The WHERE clause restricts the query to specific rows:
SELECT email FROM contacts WHERE name = "Dave";
dave@example.comSQL has many other clauses, and we’ll look at some of them in later sections. But first it is necessary to see how to integrate the SQL database with Node.js.
SQLite is probably the simplest solution for incorporating SQL database features into an application. Unlike other popular database management systems, SQLite is not a database server to which a client connects. Instead, SQLite provides a set of functions that allow the developer to create a database like a conventional file. In the case of an HTTP server implemented with Node.js Express, this file is usually located in the same directory as the server script.
Before using SQLite in Node.js, you need to install the sqlite3 module. Run the following command in the server’s installation directory; i.e., the directory containing the Node.js script you will run.
$ npm install sqlite3Be aware that there are several modules that support SQLite, such as better-sqlite3, whose usage is subtly different from sqlite3. The examples in this lesson are for the sqlite3 module, so they might not work as expected if you choose another module.
To demonstrate how a Node.js Express server can work with an SQL database, let’s write a script that stores and displays messages sent by a client identified by a cookie. Messages are sent by the client via the HTTP POST method and the server response can be formatted as JSON or HTML (from a template), depending on the format requested by the client. This lesson won’t go into detail about using HTTP methods, cookies, and templates. The code snippets shown here assume that you already have a Node.js Express server where these features are configured and available.
The simplest way to store the messages sent by the client is to store them in a global array, where each message previously sent is associated with a unique identification key for each client. This key can be sent to the client as a cookie, which is presented to the server on future requests to retrieve its previous messages.
However, this approach has a weakness: because messages are stored only in a global array, all messages will be lost when the current server session is terminated. This is one of the advantages of working with databases, because the data is persistently stored and is not lost if the server is restarted.
Using the index.js file as the main server script, we can incorporate the sqlite3 module and indicate the file that serves as the database, as follows:
const sqlite3 = require('sqlite3')
const db = new sqlite3.Database('messages.sqlite3');If it doesn’t already exist, the messages.sqlite3 file will be created in the same directory as the index.js file. Inside this single file, all structures and respective data will be stored. All database operations performed in the script will be intermediated by the db constant, which is the name given to the new sqlite3 object that opens the messages.sqlite3 file.
No data can be inserted into the database until at least one table is created. Tables are created with the statement CREATE TABLE:
db.run('CREATE TABLE IF NOT EXISTS messages (id INTEGER PRIMARY KEY AUTOINCREMENT, uuid CHAR(36), message TEXT)')The db.run() method is used to execute SQL statements in the database. The statement itself is written as a parameter for the method. Although SQL statements must end with a semicolon when entered in a command-line processor, the semicolon is optional in statements passed as parameters in a program.
Because the run method will be performed every time the script is executed with node index.js, the SQL statement includes the conditional clause IF NOT EXISTS to avoid errors in future executions when the messages table already exists.
The fields that make up the messages table are id, uuid, and message. The id field is a unique integer used to identify each entry in the table, so it is created as PRIMARY KEY. Primary keys cannot be null and there cannot be two identical primary keys in the same table. Therefore, almost every SQL table has a primary key in order to track the table’s contents. Although it is possible to explicitly choose the value for the primary key of a new record (provided it does not yet exist in the table), it is convenient for the key to be generated automatically. The AUTOINCREMENT flag in the id field is used for this purpose.
Note
Explicit setting of primary keys in SQLite is optional, because SQLite itself creates a primary key automatically. As stated in the SQLite documentation: “In SQLite, table rows normally have a 64-bit signed integer ROWID which is unique among all rows in the same table. If a table contains a column of type INTEGER PRIMARY KEY, then that column becomes an alias for the ROWID. You can then access the ROWID using any of four different names, the original three names described above, or the name given to the INTEGER PRIMARY KEY column. All these names are aliases for one another and work equally well in any context.”The uuid and message fields store the client identification and message content, respectively. A field of type CHAR(36) stores a fixed amount of 36 characters, and a field of type TEXT stores texts of arbitrary length.
The main function of our example server is to store messages that are linked to the client that sent them. The client sends the message in the message field in the body of the request sent with the HTTP POST method. The client’s identification is in a cookie called uuid. With this information, we can write the Express route to insert new messages in the database:
app.post('/', (req, res) => {
let uuid = req.cookies.uuid
if ( uuid === undefined )
uuid = uuidv4()
// Insert new message into the database
db.run('INSERT INTO messages (uuid, message) VALUES (?, ?)', uuid, req.body.message)
// If an error occurrs, err object contains the error message.
db.all('SELECT id, message FROM messages WHERE uuid = ?', uuid, (err, rows) => {
let expires = new Date(Date.now());
expires.setDate(expires.getDate() + 30);
res.cookie('uuid', uuid, { expires: expires })
if ( req.headers.accept == "application/json" )
res.json(rows)
else
res.render('index', {title: "My messages", rows: rows})
})
})This time, the db.run() method executes an insert statement, but note that the uuid and req.body.message are not written directly into the statement line. Instead, question marks were substituted for the values. Each question mark corresponds to a parameter that follows the SQL statement in the db.run() method.
Using question marks as placeholders in the statement that is executed in the database makes it easier for SQLite to distinguish between the static elements of the statement and its variable data. This strategy allows SQLite to escape or sanitize the variable contents that are part of the statement, preventing a common security breach called SQL injection. In that attack, malicious users insert SQL statements into the variable data in the hope that the statements will be executed inadvertently; sanitizing foils the attack by disabling dangerous characters in the data.
As shown in the sample code, our intent is to use the same route to insert new messages into the database and to generate the list of previously sent messages. The db.all() method returns the collection of all entries in the table that match the criteria defined in the query.
Unlike the statements performed by db.run(), db.all() generates a list of records that are handled by the arrow function designated in the last parameter:
(err, rows) => {}This function, in turn, takes two parameters: err and rows. The err parameter will be used if an error occurs that prevents the execution of the query. Upon success, all records are available in the rows array, where each element is an object corresponding to a single record from the table. The properties of this object correspond to the field names indicated in the query: uuid and message.
The rows array is a JavaScript data structure. As such, it can be used to generate responses with methods provided by Express, such as res.json() and res.render(). When rendered inside an EJS template, a conventional loop can list all records:
<ul>
<% rows.forEach( (row) => { %>
<li><strong><%= row.id %></strong>: <%= row.message %></li>
<% }) %>
</ul>Instead of filling the rows array with all the records returned by the query, in some cases it might be more convenient to treat each record individually with the db.each() method. The db.each() method syntax is similar to the db.all() method, but the row parameter in (err, row) ⇒ {} matches a single record at a time.
So far, our client can only add and query messages on the server. Since the client now knows the id of the previously sent messages, we can implement a function to modify a specific record. The modified message can also be sent to an HTTP POST method route, but this time with a route parameter to catch the id given by the client in the request path:
app.post('/:id', (req, res) => {
let uuid = req.cookies.uuid
if ( uuid === undefined ){
uuid = uuidv4()
// 401 Unauthorized
res.sendStatus(401)
}
else {
// Update the stored message
// using named parameters
let param = {
$message: req.body.message,
$id: req.params.id,
$uuid: uuid
}
db.run('UPDATE messages SET message = $message WHERE id = $id AND uuid = $uuid', param, function(err){
if ( this.changes > 0 )
{
// A 204 (No Content) status code means the action has
// been enacted and no further information is to be supplied.
res.sendStatus(204)
}
else
res.sendStatus(404)
})
}
})This route demonstrates how to use the UPDATE and WHERE clauses to modify an existing record. An important difference from the previous examples is the use of named parameters, where values are bundled into a single object (param) and passed to the db.run() method instead of specifying each value by itself. In this case, the field names (preceded by $) are the object’s properties. Named parameters allow the use of field names (preceded by $) as placeholders instead of question marks.
A statement like the one in the example will not cause any modification to the database if the condition imposed by the WHERE clause fails to match some record in the table. To evaluate whether any records were modified by the statement, a callback function can be used as the last parameter of the db.run() method. Inside the function, the number of changed records can be queried from this.changes. Note that arrow functions cannot be used in this case, because only regular functions of the form function(){} define the this object.
Removing a record is very similar to modifying it. We can, for example, continue using the :id route parameter to identify the message to be deleted, but this time in a route invoked by the client’s HTTP DELETE method:
app.delete('/:id', (req, res) => {
let uuid = req.cookies.uuid
if ( uuid === undefined ){
uuid = uuidv4()
res.sendStatus(401)
}
else {
// Named parameters
let param = {
$id: req.params.id,
$uuid: uuid
}
db.run('DELETE FROM messages WHERE id = $id AND uuid = $uuid', param, function(err){
if ( this.changes > 0 )
res.sendStatus(204)
else
res.sendStatus(404)
})
}
})Records are deleted from a table with the DELETE FROM clause. We again used the callback function to evaluate how many entries have been removed from the table.
Once defined, the db object can be referenced at any time during script execution, because the database file remains open throughout the current session. It is not common to close the database while the script is running.
A function to close the database is useful, however, to avoid abruptly closing the database when the server process finishes. Although unlikely, abruptly shutting down the database can result in inconsistencies if in-memory data is not yet committed to the file. For instance, an abrupt database shutdown with data loss can occur if the script is terminated by the user by pressing the Ctrl+C keyboard shortcut.
In the Ctrl+C scenario just described, the process.on() method can intercept signals sent by the operating system and execute an orderly shutdown of both the database and the server:
process.on('SIGINT', () => {
db.close()
server.close()
console.log('HTTP server closed')
})The Ctrl+C shortcut invokes the SIGINT operating system signal, which terminates a foreground program in the terminal. Before ending the process when receiving the SIGINT signal, the system invokes the callback function (the last parameter in the process.on() method). Inside the callback function, you can put any cleanup code, in particular the db.close() method to close the database and server.close(), which gracefully closes the Express instance itself.
This documentation is provided by the Linux Professional Institute
Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0)