Category Archives: Node JS

Node JS with mySQL using Sequelize and Express

November 26, 2021Node JSadmin

ref – https://www.esparkinfo.com/node-js-with-mysql-using-sequelize-express.html

setup MySQL

Make sure your mySQL is installed on a mac like this:
http://chineseruleof8.com/code/index.php/2016/05/17/install-mysql-and-use-terminal-to-use-it/’

When you set up your mySQL, create a database called testdb

Node JS

$ mkdir nodejs-express-sequelize-mysql
$ cd nodejs-express-sequelize-mysql

npm init

name: (nodejs-express-sequelize-mysql)
version: (1.0.0)
description: Node.js Rest Apis with Express, Sequelize & MySQL.
entry point: (index.js) server.js
test command:
git repository:
keywords: nodejs, express, sequelize, mysql, rest, api
author: esparkinfo
license: (ISC)

1	npm install express sequelize mysql2 body-parser cors --save

Setting Up Express JS Web Server

in the root directory, create server.js

const express = require("express");

const bodyParser = require("body-parser");

const cors = require("cors");

const app = express();

var corsOptions = {

origin: "http://localhost:8081"

};

app.use(cors(corsOptions));

// parse requests of content-type - application/json

app.use(bodyParser.json());

// parse requests of content-type - application/x-www-form-urlencoded

app.use(bodyParser.urlencoded({ extended: true }));

// simple route

app.get("/", (req, res) => {

res.json({ message: "Welcome to my application." });

});

// set port, listen for requests

const PORT = process.env.PORT || 8080;

app.listen(PORT, () => {

console.log(`Server is running on port ${PORT}.`);

});

Configuring MySQL Database & Sequelize

In the root directory, create a config folder.
In that folder, create db.config.js

module.exports = {

HOST: "localhost",

USER: "root",

PASSWORD: "YOUR DB PASSWORD THAT MATCHES USER",

DB: "testdb",

dialect: "mysql",

pool: {

max: 5,

min: 0,

acquire: 30000,

idle: 10000

}

};

Note – The first five parameters are intended for MySQL. The pool parameter is optional and is deployed in the Sequelize connection pool config. The parameters are explained below.

max -maximum number of connections permissible in a pool
min – minimum number of connections permissible in a pool
idle – maximum time, in terms of milliseconds, that a connection can be held idly before being released
acquire – maximum time, in terms of milliseconds, that the pool seeks to make the connection before an error message pops up on screen

Sequelize

In the root directory, create a models folder

In the models folder, create index.js by entering the following code :

models/index.js

const dbConfig = require("../config/db.config.js");

const Sequelize = require("sequelize");

const sequelize = new Sequelize(dbConfig.DB, dbConfig.USER, dbConfig.PASSWORD, {

host: dbConfig.HOST,

dialect: dbConfig.dialect,

pool: {

max: dbConfig.pool.max,

min: dbConfig.pool.min,

acquire: dbConfig.pool.acquire,

idle: dbConfig.pool.idle

}

});

const db = {};

db.Sequelize = Sequelize;

db.sequelize = sequelize;

db.tutorials = require("./tutorial.model.js")(sequelize, Sequelize);

module.exports = db;

In MySQL databases, this model represents tutorial tables. The columns are automatically generated, a few of which are id, description, createdAt, and published.

tutorial.model.js

module.exports = (sequelize, Sequelize) => {

const Tutorial = sequelize.define("tutorial", {

title: {

type: Sequelize.STRING

description: {

type: Sequelize.STRING

published: {

type: Sequelize.BOOLEAN

}

});

return Tutorial;

};

Then in server.js, we require this model, and use

const db = require("./models");

db.sequelize.sync({ force: true }).then(() => {

console.log("Drop and re-sync db.");

});

What happens is that in in model/index.js, we allocated a new Sequelize object and initialized it with a mysql database connection using our db configuration. We import this functionality into our server.js. We call the sync function with the force option set to true in order to DROP TABLE IF EXISTS before trying to create the table – if you force, existing tables will be overwritten.

You’ll be signed in as db user root with your password before this takes place, so make sure its all correct.

Controller

Routes import Controllers and Controller’s exported functions create, find, update, delete, etc.
Controller’s create will first error check all the incoming parameters, then uses model. Thus controllers are the middleman.

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

const Sequelize = require('sequelize');

const Op = Sequelize.Op;

const db = require("../models"); // models path depend on your structure

const Tutorial = db.tutorials;

exports.create = (req, res) => {

// Validate request

if (!req.body.title) {

res.status(400).send({

message: "Content can not be empty!"

});

return;

}

// Create a Tutorial

const tutorial = {

title: req.body.title,

description: req.body.description,

published: req.body.published ? req.body.published : false

};

// Save Tutorial in the database

Tutorial.create(tutorial)

.then(data => {

res.send(data);

})

.catch(err => {

res.status(500).send({

message:

err.message || "Some error occurred while creating the Tutorial."

});

};

exports.findAll = (req, res) => {

const title = req.query.title;

var condition = title ? { title: { [Op.like]: `%${title}%` } } : null;

Tutorial.findAll({ where: condition })

.then(data => {

res.send(data);

})

.catch(err => {

res.status(500).send({

message:

err.message || "Some error occurred while retrieving tutorials."

});

};

exports.findOne = (req, res) => {

const id = req.params.id;

Tutorial.findByPk(id)

.then(data => {

res.send(data);

})

.catch(err => {

res.status(500).send({

message: "Error retrieving Tutorial with id=" + id

});

};

exports.update = (req, res) => {

const id = req.params.id;

Tutorial.update(req.body, {

where: { id: id }

})

.then(num => {

if (num == 1) {

res.send({

message: "Tutorial was updated successfully."

});

} else {

res.send({

message: `Cannot update Tutorial with id=${id}. Maybe Tutorial was not found or req.body is empty!`

});

}

})

.catch(err => {

res.status(500).send({

message: "Error updating Tutorial with id=" + id

});

};

exports.delete = (req, res) => {

const id = req.params.id;

Tutorial.destroy({

where: { id: id }

})

.then(num => {

if (num == 1) {

res.send({

message: "Tutorial was deleted successfully!"

});

} else {

res.send({

message: `Cannot delete Tutorial with id=${id}. Maybe Tutorial was not found!`

});

}

})

.catch(err => {

res.status(500).send({

message: "Could not delete Tutorial with id=" + id

});

};

exports.deleteAll = (req, res) => {

Tutorial.destroy({

where: {},

truncate: false

})

.then(nums => {

res.send({ message: `${nums} Tutorials were deleted successfully!` });

})

.catch(err => {

res.status(500).send({

message:

err.message || "Some error occurred while removing all tutorials."

});

};

exports.findAllPublished = (req, res) => {

Tutorial.findAll({ where: { published: true } })

.then(data => {res.send(data);})

.catch(err => {

res.status(500).send({message: err.message || "Some error occurred while retrieving tutorials."});

});

};

Routes

If the client sends a request for an endpoint via an HTTP request such as POST, DELETE, PUT, or GET, the user must determine how the server responds.

Such a response from the server is possible by setting up the below routes:

/api/tutorials: GET, POST, DELETE
/api/tutorials/:id: GET, PUT, DELETE
/api/tutorials/published: GET
To create a tutorial.routes.js inside the app/routes folder, the user performs the following steps:

module.exports = app => {

const tutorials = require("../controllers/tutorial.controller.js");

var router = require("express").Router();

// Create a new Tutorial

router.post("/", tutorials.create);

// Retrieve all Tutorials

router.get("/", tutorials.findAll);

// Retrieve all published Tutorials

router.get("/published", tutorials.findAllPublished);

// Retrieve a single Tutorial with id

router.get("/:id", tutorials.findOne);

// Update a Tutorial with id

router.put("/:id", tutorials.update);

// Delete a Tutorial with id

router.delete("/:id", tutorials.delete);

// Create a new Tutorial

router.delete("/", tutorials.deleteAll);

app.use('/api/tutorials', router);

};

In order to use routes, remember to put

1	require("./app/routes/tutorial.routes")(app);

in server.js, right above code for port.

Hence, our server.js has code for routes, which routes to middleman controller, which then uses models to make changes to the database.

In the end, your server should look like this:

server.js

const express = require("express");

const bodyParser = require("body-parser");

const cors = require("cors");

const app = express();

var corsOptions = {

origin: "http://localhost:8081"

};

app.use(cors(corsOptions));

// parse requests of content-type - application/json

app.use(bodyParser.json());

// parse requests of content-type - application/x-www-form-urlencoded

app.use(bodyParser.urlencoded({ extended: true }));

const db = require("./models");

db.sequelize.sync({ force: true }).then(() => {

console.log("Drop and re-sync db.");

});

// simple route

app.get("/", (req, res) => {

res.json({ message: "Welcome to esparkinfo application." });

});

require("./routes/tutorial.routes")(app);

// set port, listen for requests

const PORT = process.env.PORT || 8080;

app.listen(PORT, () => {

console.log(`Server is running on port ${PORT}.`);

});

Using Postman to test

In Postman, new tab:

POST on http://localhost:8080/api/tutorials

Body, raw, then select JSON.

In the body:

{

"title": "My tut 3",

"description": "Tut 3 Description"

}

In your database terminal, to see what you’ve inserted, get all results:

select * from tutorials;

Get all tutorials

Get specific tutorial

GET on http://localhost:8080/api/tutorials/2

Finding All Tutorials where Title=”node”

GET on http://localhost:8080/api/tutorials?title=tut

Find all published posts

GET on http://localhost:8080/api/tutorials/published

Delete depending on request param id

DELETE on http://localhost:8080/api/tutorials/2

Debug typescript in Node JS

March 22, 2021javascript, Node JSadmin

demo download

Create REST API with Typescript

mkdir Und-TSC-sec15
cd Und-TSC-sec15

Create Node Project
npm init
You’ll then have a package.json file.

1

tsc --init

You’ll then have a tsconfig.json file.

In tsconfig.json:

{

"compilerOptions": {

/* Visit https://aka.ms/tsconfig.json to read more about this file */

/* Basic Options */

// "incremental": true, /* Enable incremental compilation */

"target": "es2018", /* Specify ECMAScript target version: 'ES3' (default), 'ES5', 'ES2015', 'ES2016', 'ES2017', 'ES2018', 'ES2019', 'ES2020', or 'ESNEXT'. */

"module": "commonjs", /* Specify module code generation: 'none', 'commonjs', 'amd', 'system', 'umd', 'es2015', 'es2020', or 'ESNext'. */

"moduleResolution": "node",

"lib": [

"dom",

"es6",

"dom.iterable",

"scripthost"

], /* Specify library files to be included in the compilation. */

"sourceMap": true, /* Generates corresponding '.map' file. */

// "outFile": "./", /* Concatenate and emit output to single file. */

"outDir": "./dist", /* Redirect output structure to the directory. */

"rootDir": "./src", /* Specify the root directory of input files. Use to control the output directory structure with --outDir. */

// "composite": true, /* Enable project compilation */

// "tsBuildInfoFile": "./", /* Specify file to store incremental compilation information */

"removeComments": true, /* Do not emit comments to output. */

// "noEmit": true, /* Do not emit outputs. */

// "importHelpers": true, /* Import emit helpers from 'tslib'. */

// "downlevelIteration": true, /* Provide full support for iterables in 'for-of', spread, and destructuring when targeting 'ES5' or 'ES3'. */

// "isolatedModules": true, /* Transpile each file as a separate module (similar to 'ts.transpileModule'). */

/* Strict Type-Checking Options */

"strict": true, /* Enable all strict type-checking options. */

"noImplicitReturns": true, /* Report error when not all code paths in function return a value. */

// "noFallthroughCasesInSwitch": true, /* Report errors for fallthrough cases in switch statement. */

// "noUncheckedIndexedAccess": true, /* Include 'undefined' in index signature results

"typeRoots": [

"node_modules/@types"

], /* List of folders to include type definitions from. */

// "types": [], /* Type declaration files to be included in compilation. */

// "allowSyntheticDefaultImports": true, /* Allow default imports from modules with no default export. This does not affect code emit, just typechecking. */

"esModuleInterop": true, /* Enables emit interoperability between CommonJS and ES Modules via creation of namespace objects for all imports. Implies 'allowSyntheticDefaultImports'. */

}

Now let’s install the needed modules:

standard web server independencies:
npm install –save express body-parser

development independencies:
npm install –save-dev nodemon

create src folder:
mkdir src
cd src
touch app.ts

import express from 'express';

import { json } from 'body-parser';

const app = express();

app.use(json());

app.get('/', (req, res)=> {

console.log('entered root endpoint');

res.status(400).send('Welcome to Guang Dong!');

});

// error handling middleware function

app.use((err: Error, req: express.Request, res: express.Response, next: express.NextFunction) => {

res.status(500).json({message: err.message});

});

app.listen(3000);

When you are done, type tsc to compile the typescript into javascript.

Then you can debug it. Put a breakpoint there and make sure you’re on the index file. In our file, we’re on app.ts.

Once you’re on that file, go to Run > Start Debugging. You’ll see the bottom status bar turn orange. This means the server is running.

Click on View > Terminal, and you’ll see the terminal pop up. Then click on debug console. You’ll be able to see the console log from the node js app appear.

Creating Routes

cd src/routes
touch todos.ts

We extract Router from express. We can then register routes to hit functionality.
We set up the router to receive POST, GET, PATCH, and DELETE on ‘/’.
Each HTTP request will get mapped to certain functionalities that we declare in controllers/todos file.

routes/todos.ts

import { Router } from 'express';

import { createTodo, getTodos, updateTodo, deleteTodo} from '../controllers/todos';

const router = Router();

router.post('/', createTodo);

router.get('/', getTodos);

router.patch('/:id', updateTodo);

router.delete('/:id', deleteTodo);

export default router;

Finally, we need to connect our router to our server for endpoint /todos.

app.ts

import express from 'express';

import todoRoutes from './routes/todos';

const app = express();

app.use('/todos', todoRoutes); // localhost:3000/todos will hit todoRoutes

// depending on what HTTP verb it is, it will hit the functionalities mapped to that verb.

// error handling middleware function

app.use((err: Error, req: express.Request, res: express.Response, next: express.NextFunction) => {

res.status(500).json({message: err.message});

});

Add additional routes

First, we create a class Todos so we have a structure to work with. We export a class Todo so other modules can import it, and then new it to create an instance of Todo.

export class Todo {

constructor(public id: string, public text: string) {

console.log('Constructingn a Todo instance...');

}

Then import the class so that we get a simple data structure going.

We create functionality by implementing a function createTodo.
Note that we use RequestHandler interface, which contracts us to use
req, res, and next parameters.

controllers/todo.ts

import { RequestHandler } from 'express';

import { Todo } from '../models/todo';

// TODOS is the name of an array of type Todo

const TODOS: Todo [] = [];

export const createTodo: RequestHandler = (req, res, next) => {

console.log('entered createTodo');

const text = (req.body as {text:string}).text;

console.log('text', text);

const newTodo = new Todo(Math.random().toString(), text);

TODOS.push(newTodo);

res.status(201).json({

message: 'Created the todo',

createdTodo: newTodo

})

};

Since it takes care of a POST request, we use parse the body property in the request
object to get the body data. The body data’s key is text.

In order do that we need to import the body-parser package in app.ts, and then extract json.
This is so that we parse the body using json.

app.ts

import { json } from 'body-parser';

...

app.use(json());

Open up your postman and query it like so:

child processes in Node JS 2

March 15, 2021Node JSadmin

getCount.js

const slowFunction = (name) => {

let counter = 0;

console.log(`starting ${name}`)

while (counter < 1000000000 * 10) {

console.log(`counter ${counter}`);

counter++;

}

console.log(`finish ${name}`);

return counter;

}

process.on('message', (message) => {

const arr = message.split(' ');

if (arr[0] == 'START') {

slowFunction(arr[1]);

process.send(`FINISH ${arr[1]}`);

}

});

index.js

var express = require("express");

var app = express();

const { addLongTask, pauseQueue, resumeQueue } = require('./tasker.js');

app.get('/example',function(req,res){

console.log(req.query);

console.log(req.query.firstname);

res.send("Query String printed to console");

});

app.get('/task',function(req,res) {

addLongTask((pid) => {

res.setHeader('Content-Type', 'application/json');

res.writeHead(200);

res.end(`you've added process with id: ${pid}`);

}, (pid) => {

console.log(`running pid is ${pid}`);

});

app.get('/resume',function(req,res) {

console.log('Going to resume process');

resumeQueue();

});

app.get('/pause',function(req,res) {

console.log('Going to pause executing process');

pauseQueue();

// pause all process in the queue

// search out the process with the query ID.

// kill it

// resume queue

// if just kill...then we kill the current running process

// if given a pid, then we search it out.

// 1) is the current running process THAT id?

// 2) then look through the queue.

if (child) {

child.kill("SIGKILL");

console.log('kill the child process!!!!!!');

res.setHeader('Content-Type', 'application/json');

res.writeHead(200);

res.end(`${new Date().toISOString()} {"message":"process killed"}`);

}

});

app.get('/hello',function(req,res){

console.log('Returning /hello results');

res.setHeader('Content-Type', 'application/json');

res.writeHead(200);

res.end(`${new Date().toISOString()} {"message":"hello"}`);

});

app.listen(8080);

tasker.js

const { fork } = require('child_process');

const Queue = require('./queue.js');

let running = false;

let currentExecuting;

execute = async (obj) => {

const { pid, process } = obj;

process.send('START '+pid);

return new Promise((resolve, reject) => {

process.on('message', (message) => {

const [status, pid] = message.split(' ');

console.log(`${status} on process ${pid}`);

if (Queue.count() == 0 && running) {

running = false;

}

resolve({ status, pid });

});

}

exports.pauseQueue = async() => {

// All we have to do is PAUSE the executing process, and the whole queue will stand still.

const { pid, process } = currentExecuting;

console.log(`Pausing process ${pid}`);

process.kill('SIGSTOP');

}

exports.resumeQueue = async() => {

// first pause the executing process

const { pid, process } = currentExecuting;

console.log(`Resuming process ${pid}`);

process.kill('SIGCONT');

// then pause all the processes in the queue

}

exports.kill = async(pidToKill) => {

// first we pause the queue

this.pauseQueue();

// 1) is the executing process the pid to kill?

const { pid, process } = currentExecuting;

if (pidToKill == pid) {

// kill the current executing process!

process.kill("SIGKILL");

console.log(`killed the process ${$pid}`);

}

// 2) then search through the queue to kill the pid

Queue.find(item => {

const { pid, process} = item;

if (pid === pidToKill) {

process.kill("SIGKILL");

return true;

}

return false;

});

// then we resume

this.resumeQueue();

}

exports.addLongTask = async (addedCB, runningCB) => {

const child = fork(__dirname + '/getCount');

console.log(`enqueued child process ${child.pid}`);

Queue.enqueue({

pid: child.pid,

process: child

});

addedCB(child.pid);

// if its running, just add it to the queue to be auto processed

if (running) {

console.log(`Sorry process ${child.pid}! There are processes running, please wait your turn...`);

return;

}

// if its not running and the queue is empty, then we set it off

if (!running && !Queue.isEmpty()) {

console.log(`Nothing is running and there are ${Queue.count()} tasks on the queue`);

running = true;

do {

const obj = Queue.dequeue();

const { pid } = obj;

currentExecuting = obj;

runningCB(pid);

await execute(obj);

} while (Queue.count() > 0);

}

// always return the pid

return child.pid;

}

queue.js

let _queue = [];

exports.isEmpty = () => {

return (_queue.length == 0);

}

exports.enqueue = (item) => {

_queue.push(item);

}

exports.dequeue = () => {

return _queue.shift();

}

exports.count = () => {

return _queue.length;

}

exports.print = () => {

if (!isEmpty()) {

for (let i = 0; i < _queue.length; i++) {

_queue[i].toString();

}

child process in Node JS

March 12, 2021Node JSadmin

ref – https://www.digitalocean.com/community/tutorials/how-to-launch-child-processes-in-node-js

Creating a Child Process with fork()

index.js

const http = require('http');

const host = 'localhost';

const port = 8000;

// this function will take some time to run

const slowFunction = () => {

let counter = 0;

while (counter < 5000000000) {

counter++;

}

return counter;

}

const requestListener = function (req, res) {

if (req.url === '/total') {

let slowResult = slowFunction();

let message = `{"totalCount":${slowResult}}`;

console.log('Returning /total results');

res.setHeader('Content-Type', 'application/json');

res.writeHead(200);

res.end(message);

} else if (req.url === '/hello') {

console.log('Returning /hello results');

res.setHeader('Content-Type', 'application/json');

res.writeHead(200);

res.end(`{"message":"hello"}`);

}

};

const server = http.createServer(requestListener);

server.listen(port, host, () => {

console.log(`Server is running on http://${host}:${port}`);

});

Fork is a variation of spawn. To create a process that’s also a Node Process.

Main benefit of fork over (exec, spawn) is that fork enables communication between the parent and child process.

CPU intensive tasks (iterating over large loops, parsing large JSON) stop other JS code from running. If a web server is blocked, it cannot process any new incoming requests until the blocking code has completed its execution.

1) Now run the server
2) run /hello (1st time)
3) run /total
4) run /hello (2nd time)

So we’ll only see the result from the first /hello. The slow function will block other code, and that’s why we won’t see the 2nd /hello.

The blocking code is slowFunction.

So how do we solve this?

Move blocking code to own module:

getCount.js

const slowFunction = () => {

let counter = 0;

while (counter < 50000000000) {

console.log(`counter - ${counter}`);

counter++;

}

return counter;

}

Since it will be run with fork, we can add code to communicate with the parent process when slowFunction has completed processing. Let’s send a message to the parent process with JSON message.

process.on('message', (message) => {

if (message == 'START') { // server will do: child.send('START'), so we must receive

console.log('Child process received START message');

let slowResult = slowFunction(); // upon receiving START, we run our long task

let message = `${new Date().toISOString()} {"totalCount":${slowResult}}`;

process.send(message); // send data back to the parent process

}

});

– Why look for message ‘START’? Our server code will send the START event when someone access the ‘/total’ endpoint.
– Upon receiving that event, we run slowFunction().
– we use process.send() to send a message to the parent process.

index.js

const http = require('http');

const { fork } = require('child_process');

const host = 'localhost';

const port = 8000;

let child;

const requestListener = function (req, res) {

if (req.url === '/total') {

console.log(` ${new Date().toISOString()} - Start slow job`);

child = fork(__dirname + '/getCount');

console.log('child process id: ' + child.pid);

child.on('message', (message) => {

console.log('Returning /total results');

res.setHeader('Content-Type', 'application/json');

res.writeHead(200);

res.end(message);

});

child.send('START');

} else if (req.url === '/hello') {

console.log('Returning /hello results');

res.setHeader('Content-Type', 'application/json');

res.writeHead(200);

res.end(`${new Date().toISOString()} {"message":"hello"}`);

} else if (req.url === '/kill') {

if (child) {

child.kill("SIGKILL");

console.log('kill the child process!!!!!!');

res.setHeader('Content-Type', 'application/json');

res.writeHead(200);

res.end(`${new Date().toISOString()} {"message":"process killed"}`);

}

const server = http.createServer(requestListener);

server.listen(port, host, () => {

console.log(`Server is running on http://${host}:${port}`);

});

getCount.js

const slowFunction = () => {

let counter = 0;

while (counter < 50000000000) {

console.log(`counter - ${counter}`);

counter++;

}

return counter;

}

process.on('message', (message) => {

if (message == 'START') {

console.log('Child process received START message');

let slowResult = slowFunction();

let message = `${new Date().toISOString()} {"totalCount":${slowResult}}`;

process.send(message);

}

});

process.on('beforeExit', (code) => {

console.log('Process beforeExit event with code: ', code);

});

process.on('exit', (code) => {

console.log('Process exit event with code: ', code);

});

Run the app: node index.js

Open a 2nd terminal and type: curl http://localhost:8000/total
This will start up a long running task at say 7:41:18.

Open a 3rd terminal and type: curl http://localhost:8000/hello

Hi it a bunch of times. As you can see, it is running concurrently and not blocking the main JS code. This is because its a child process that’s running it concurrently. From the time stamp, you will also see that it’s happening after 7:41:18.

Then when the long task is done, it stops at 7:42:20. This proves that the /hello logs before 7:42:20 were all done concurrently with the task at /total.

Worker Thread with Node JS

March 4, 2021Node JSadmin

mkdir spawnEx
then ls into that directory

set it up all default
npm init

create an index.js file
touch index.js

Open it up with VCode
Code . index.js

copy and paste the code like so:
index.js

const http = require('http');

const hostname = '127.0.0.1';

const port = 3000;

const server = http.createServer((req, res) => {

res.statusCode = 200;

res.setHeader('Content-Type', 'text/plain');

res.end('Hello World');

});

server.listen(port, hostname, () => {

console.log(`Server running at http://${hostname}:${port}/`);

});

Then we create the run file. use worker_threads to extract Worker object. We create a new worker whenever we call runService().

run.js

const { Worker } = require('worker_threads')

function runService(name) {

return new Promise((resolve, reject) => {

const worker = new Worker('./worker.js', { workerData: name });

worker.on('message', data => {

console.log('------->', data);

resolve(data);

});

worker.on('error', reject);

worker.on('exit', (code) => {

if (code !== 0)

reject(new Error(`Worker stopped with exit code ${code}`));

})

}

runService("A").catch(err => console.error(err))

runService("B").catch(err => console.error(err))

runService("C").catch(err => console.error(err))

Create a worker class.
touch worker.js

worker.js

const { workerData, parentPort } = require('worker_threads');

const fs = require('fs');

for (let i = 0; i < 1000; i++) {

fs.appendFile('test.txt', `\n ${workerData} ${Date.now()}`, function (err) {

parentPort.postMessage({ workerData: `postMessage: ${workerData} ${Date.now()}` });

if (err) console.log('err', err);

});

}

test.txt

I will show the time at milliseconds of when the worker has written to the text.txt file. As you scroll down using the right hand scroll bar, you’ll notice this millisecond increase. This is the increase of time, and you’ll see exactly which worker was executing at that time.

When you should use Node JS

March 4, 2021Node JSadmin

ref – https://livecodestream.dev/post/when-you-should-and-should-not-use-nodejs-for-your-project/

A multi-threaded program has a pool of threads that run concurrently to accept requests. The number of threads used by the program is limited by the system’s RAM or other parameters. So, the number of requests it can process at a given time is limited to this number of threads.

Whereas the single-threaded Node.js program can process any number of requests at a given time that are in the event queue to be executed. Node’s event loop uses a single-threaded implementation when handling events. Each event is handled one after another in the order they arrive by a single processor thread. The requests received to a Node web server are served in the order they were received.

However, there is a downside to Node.js being single-threaded. The single-threaded implementation makes Node a bad choice for CPU-intensive programs. When a time-consuming task is running in the program it blocks the event loop from moving forward for a longer period. Unlike in a multi-threaded program, where one thread can be doing the CPU-intensive task and others can handle arriving requests, a Node.js program has to wait until the computation completes to handle incoming requests.

Node.js introduced a workaround for this problem in version 10.5.0: worker threads. You can read up more on this topic to understand how to use worker threads to solve this problem for CPU-intensive programs.

You may now have a question? How can a single thread handle more events at a given time than a thread pool if events are handled one after another and not concurrently? This is where Node’s asynchronous model steps in. Even though the Node’s event loop implementation is single-threaded, it achieves a concurrent-like behavior with the use of callbacks.

Assume you want to connect to a separate API from your Node server to retrieve some data.

You send a request to this particular API from the server. You also pass a callback function with the instruction on what to do when the response from the API is received.

After executing the task that makes the request to the API, the Node program doesn’t wait for the response from the that API. Instead, after sending the request, it continues to the next step of the program. When the response from the API arrives the callback function starts running, and therefore handles the received data. In this manner, the callback function runs concurrently to the main program thread.

In contrast, in a synchronous multi-threaded program, the thread sending making the API call waits for the response to arrive to continue to the next step. This does not stall the multi-threaded program because, even though this one thread is waiting, other threads in the thread pool are ready to accept receiving requests.

The asynchronous feature is what makes the single-thread of the Node.js program quite efficient. It makes the program fast and scalable without using as many resources as a multi-threaded application.

Naturally, this makes Node a good fit for data-intensive and I/O intensive programs. Data-intensive programs are focus on retrieving and storing data, while I/O-intensive programs focus on interacting with external resources.

When to Use Node.js?

Now back to our main question. When should you use Node.js for your projects? Considering the main features of Node.js and its strengths, we can say

– data-driven
– I/O-driven
– event-driven
– non-blocking

applications benefit the best from a Node.js implementation.

Especially, web backends that follow microservices architecture, which is now growing in popularity, are well suited for a Node implementation. With the microservices architecture, sending requests from one microservice to another becomes inevitable. With Node’s asynchronous programming, waiting for responses from outside resources can be done without blocking the program thread.

Unlike a multi-threaded program, which can run out of threads to serve incoming requests because many threads are waiting for responses from external resources, this I/O-intensive architecture does not affect the performance of the Node application. However, web servers that implement computational-heavy or processes-heavy tasks are not a good fit for Node.js. The programming community is still apprehensive of using Node with a relational database given Node tools for relational databases are still not as developed as compared to other languages.

Heavy Computational Applications

If your application is likely to run tasks that involve heavy computing and number crunching, like running the Fibonacci algorithm, Node.js is not the best language to turn to. As discussed above, the heavy computation blocks the single-thread running in the application and halts the progress of the event loop until the computation is finished. And it delays serving the requests still in the queue, which may not take as much time.

If your application has a few heavy computational tasks, but in general benefits from Node’s characteristics, the best solution is to implement those heavy computational tasks as background processes in another appropriate language. Using microservices architecture and separating heavy computational tasks from Node implementation is the best solution.

Dockerize Node app

January 21, 2021Node JSdockeradmin

ref – https://nodejs.org/en/docs/guides/nodejs-docker-webapp/

Creating the Node App

First, npm init, which crates a package.json file.

Then npm i express

Your package.json file should look something like this:

{

"name": "docker_web_app",

"version": "1.0.0",

"description": "Node.js on Docker",

"author": "First Last <first.last@example.com>",

"main": "server.js",

"scripts": {

"start": "node server.js"

"dependencies": {

"express": "^4.16.1"

}

Or you can just touch a package.json file, and run npm install. If you are using npm version 5 or later, this will generate a package-lock.json file which will be copied to your Docker image.

Then, create a server.js file that defines a web app using the Express.js framework:

'use strict';

const express = require('express');

// Constants

const PORT = 8080;

const HOST = '0.0.0.0';

// App

const app = express();

app.get('/', (req, res) => {

res.send('Hello World');

});

app.listen(PORT, HOST);

console.log(`Running on http://${HOST}:${PORT}`);

In the next steps, we’ll look at how you can run this app inside a Docker container using the official Docker image. First, you’ll need to build a Docker image of your app.

Dockerize it

A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image. Using docker build users can create an automated build that executes several command-line instructions in succession.

Create an empty file called Dockerfile in the main directory (where your package.json file is):

1	touch Dockerfile

Open the Dockerfile in your favorite text editor

The first thing we need to do is define from what image we want to build from. Here we will use the latest LTS (long term support) version 14 of node available from the Docker Hub:
FROM node:14

Next we create a directory to hold the application code inside the image, this will be the working directory for your application:

# Create app directory
WORKDIR /Users/ricky_tsao/Desktop/dockerizeNode

This image comes with Node.js and NPM already installed so the next thing we need to do is to install your app dependencies using the npm binary. Please note that if you are using npm version 4 or earlier a package-lock.json file will not be generated.

# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./

# Install app dependencies
RUN npm install
# If you are building your code for production
# RUN npm ci –only=production

Note that, rather than copying the entire working directory, we are only copying the package.json file.

To bundle your app’s source code inside the Docker image, use the COPY instruction:

# Bundle app source
COPY . .

Your app binds to port 8080 so you’ll use the EXPOSE instruction to have it mapped by the docker daemon:

EXPOSE 8080

Last but not least, define the command to run your app using CMD which defines your runtime. Here we will use node server.js to start your server:

CMD [ “node”, “server.js” ]

Dockerfile

# Here we will use the latest LTS (long term support) version 14

# of node available from the Docker Hub

FROM node:14

# Create app directory

WORKDIR /Users/ricky_tsao/Desktop/dockerizeNode

# This image comes with Node.js and NPM already installed so the next thing we need to do

# is to install your app dependencies using the npm binary.

# Install app dependencies

# A wildcard is used to ensure both package.json AND package-lock.json are copied

# where available (npm@5+)

COPY package*.json ./

RUN npm install

# If you are building your code for production

# RUN npm ci --only=production

# To bundle your app's source code inside the Docker image, use the COPY instruction:

COPY . .

# Your app binds to port 8080 so you'll use the EXPOSE instruction to have it mapped by the docker daemon:

EXPOSE 8080

# Last but not least, define the command to run your app using CMD which defines your runtime.

# Here we will use node server.js to start your server:

CMD [ "node", "server.js" ]

.dockerignore file

Create a .dockerignore file in the same directory as your Dockerfile with following content:

node_modules
npm-debug.log

Building your image

Go to the root directory that has your Dockerfile and run the following command to build the Docker image. The -t flag lets you tag your image so it’s easier to find later using the docker images command:

docker build -t rtsao6680/node-web-app .

Your image will now be listed by Docker:

Run the image

Running your image with -d runs the container in detached mode, leaving the container running in the background. The -p flag redirects a public port to a private port inside the container. Run the image you previously built:

docker run -p 40000:8080 -d rtsao6680/node-web-app

so now your web will run at localhost:

http://localhost:40000/

In our case, Docker mapped the 8080 port inside of the container to the port 40000 on your machine.

You can check the status of it:

docker ps

Server side rendering (isomorphic React apps) – Data

October 20, 2020Node JS, Reactadmin

Server

We start the server side.

At the GET verb of any url..we already have code where we store initial state in our redux. This time, let’s start it off by fetching data.

We create an initial state object, when store fetched data into an array called repositoryNames. Remember that in order to fetch in Node, we need to either use axios, or node-fetch package.

server.js

...

import thunk from 'redux-thunk';

import axios from 'axios';

...

app.get('*', async (req, res) => {

const scripts = ['vendor.js', 'client.js'];

let initialState;

if (req.url === '/') {

const repositories = await axios.get('https://jsonplaceholder.typicode.com/todos');

console.log('http get data');

initialState = {

repositoryNames: repositories.data.map((repo) => ({

id: repo.id,

})),

};

}

const store = createStore(reducers, initialState, applyMiddleware(thunk));

...

Notice thunk. Redux Thunk is a middleware that lets you call action creators that return a function instead of an action object. That function receives the store’s dispatch method, which is then used to dispatch regular synchronous actions inside the function’s body once the asynchronous operations have been completed.

Once that is done, we create our store, and place it in Provider component. Then place that into StaticRouter within our appMarkup.

const store = createStore(reducers, initialState, applyMiddleware(thunk));

const appMarkup = ReactDOMServer.renderToString(

<App />

</Provider>

</StaticRouter>

);

This is so that we can inject it into our HTML component.

const html = ReactDOMServer.renderToStaticMarkup(

<Html

children={appMarkup}

scripts={scripts}

initialState={initialState}

);

We then return this HTML component in our response object to the client.

const html = ReactDOMServer.renderToStaticMarkup(

<Html

children={appMarkup}

scripts={scripts}

initialState={initialState}

);

console.log('html', new Date());

res.send(`<!doctype html>${html}`);

Client Side

We do the same and apply thunk for the client. Since we already have the store and initial app state, it will match up with what the server has.

...

import thunk from 'redux-thunk';

import { applyMiddleware, createStore } from 'redux';

...

const store = createStore(reducers, { ...window.APP_STATE }, applyMiddleware(thunk));

ReactDOM.hydrate(

<App />

</Provider>

</BrowserRouter>,

document.getElementById('app')

);

Before in App.js, where we declare our mapping to props with initialText and changeText handler, we dispatch a very simple Action object with a literal object like so:

const mapDispatchToProps = (dispatch) => ({

changeText: () => dispatch({ type: 'CHANGE_TEXT' }),

});

which then get passed to reducers/index.js to be processed like so. The initialText is a simple string so we can do this:

function reducer(state, action) {

switch (action.type) {

case 'CHANGE_TEXT':

return { ...state, initialText: 'changed in the browser!' };

default:

return { ...state };

}

export default reducer;

But now, we use action objects defined in actions/index.js for more complicated procedures:

import axios from 'axios';

export const getRepositoryNames = () => (dispatch) => {

axios

.get('https://api.github.com/repositories')

.then(({ data }) => {

dispatch({

type: 'SET_REPOSITORY_NAMES',

payload: data.map((repo) => ({

id: repo.id,

})),

});

};

Thus, now in Home.js, we declare mapDispatchToProps with the action object getRepositoryNames:

import { getRepositoryNames } from '../actions';

...

const Home = ({ repositoryNames, getRepositoryNames }) => {

...

};

const mapStateToProps = ({ repositoryNames }) => ({

repositoryNames,

});

const mapDispatchToProps = (dispatch) => ({

getRepositoryNames: () => dispatch(getRepositoryNames()),

});

export default connect(mapStateToProps, mapDispatchToProps)(Home);

ref – https://reactjs.org/docs/hooks-effect.html

That way, we can use getRepositoryNames inside of Home component. Since Home is a functional component, we use useEffectuseEffect runs after every render!

By default, it runs both after the first render and after every update. Instead of thinking in terms of “mounting” and “updating”, you might find it easier to think that effects happen “after render”. React guarantees the DOM has been updated by the time it runs the effects.

This is the basic case in our example.

Hence we check for props repositoryNames. If its valid, we’ll display it. If it isn’t, we’ll fetch the data so it can be rendered.

Also, useEffect has a nifty feature where it can look at a state variable. If that state changes, it will run the effect. In our case, it would fetch. Hence, I put two use cases of this:

1) value changes after each click. useEffect will run every time the button is clicked because onClick updates value everytime.
2) value changes once after clicks. So useEffect will run the first time, value updates, runs second time, and that’s it.

const Home = ({ repositoryNames, getRepositoryNames }) => {

console.log('Home', new Date());

const [value, setValue] = useState(true);

const onClick = useCallback((a, b) => {

// always run useEffect because value is always changing

let changedValue = !value;

setValue(changedValue);

setValue(false);

// value changed once. useEffect will that one time.

// Then it will not run anymore because value will stay at 'false'.

console.log('value', value);

});

useEffect(() => {

console.log('---------> useEffect', new Date());

if (!repositoryNames) {

console.log('repositoryNames DNE exists, lets read it');

getRepositoryNames();

} else {

console.log('repositoryNames already exists');

}

}, [value]);

return (

<div>

<button onClick={onClick}>heheh</button>

<ul>

{repositoryNames && repositoryNames.map((repo) => (

{repo.name}

</li>

))}

</ul>

</div>

);

};

GET POST using fetch from client to web api

September 14, 2020javascript, Node JSAzureadmin

Using fetch, we get a GET request on a web API using Azure authentication and passport

node server

// API endpoint

app.get("/hello",

passport.authenticate('oauth-bearer', {session: false}),

(req, res) => {

console.log('User info: ', req.user);

console.log('Validated claims: ', req.authInfo);

if ('scp' in req.authInfo && req.authInfo['scp'].split(" ").indexOf("demo.read") >= 0) {

// Service relies on the name claim.

res.status(200).json({'name': req.authInfo['name']});

} else {

console.log("Invalid Scope, 403");

res.status(403).json({'error': 'insufficient_scope'});

}

);

client

// helper function to access the resource with the token

function callApiWithAccessToken(endpoint, token) {

const headers = new Headers();

const bearer = `Bearer ${token}`;

headers.append("Authorization", bearer);

const options = {

method: "GET",

headers: headers

};

fetch(endpoint, options)

.then(response => response.json())

.then(response => {

logMessage("Web API returned:\n" + JSON.stringify(response));

}).catch(error => {

logMessage("Error calling the Web api:\n" + error);

});

}

Using fetch, we get a POST request on a web API using Azure authentication and passport

node server

app.post("/hello",

passport.authenticate('oauth-bearer', {session: false}),

(req, res) => {

console.log(`POST on /hello`);

console.log('message: ', req.body);

res.status(200).json({'message':`received`});

}

);

client

function callApiWithAccessToken(endpoint, token) {

const headers = new Headers();

const bearer = `Bearer ${token}`;

headers.append("Authorization", bearer);

headers.append("Content-Type", "application/json");

headers.append("Accept", "application/json");

let data = {

message: 'please call me'

};

const options = {

method: "POST",

headers: headers,

body: JSON.stringify(data)

};

fetch(endpoint, options)

.then(response => response.json())

.then(response => {

logMessage("Web API returned:\n" + JSON.stringify(response));

}).catch(error => {

logMessage("Error calling the Web api:\n" + error);

});

}

Passing a form data to the Web API with no authentication

client

<label for="exampleInputEmail1">Email address</label>

<small id="emailHelp" class="form-text text-muted">We'll never share your email with anyone else.</small>

</div>

<label for="exampleInputPassword1">Password</label>

</div>

<label class="form-check-label" for="exampleCheck1">Check me out</label>

</div>

<button type="submit" class="btn btn-primary" onclick="passEmailToApi(event)">Submit</button>

</form>

function passEmailToApi(e) {

e.preventDefault(); // stop the submission

let form = document.getElementsByName('emailForm')[0];

storeEmailData(form[0].value, form[1].value);

}

function storeEmailData(email) {

const headers = new Headers();

headers.append("Content-Type", "application/json");

headers.append("Accept", "application/json");

let data = {message: 'please call me'};

const options = {

method: "POST",

headers: headers,

body: JSON.stringify(data)

};

fetch('http://localhost:5000/saveMessage', options)

.then(response => response.json())

.then(response => {

logMessage("Web API returned:\n" + JSON.stringify(response));

}).catch(error => {

logMessage("Error calling the Web api:\n" + error);

});

}

DNY client (12 – basic site)

February 10, 2020javascript, Node JS, Reactadmin

dnyNodeAPI-basic-site
npm install to install all node modules
Then, npm run dev to run the server

dny-client-basic-site
npm install to install all node modules
Then, npm run start to run the client

So far we have a basic site with these functionalities:

Create a user

POST on http://localhost:8080/signup
body: JSON.stringify(user)

src/auth/index.js

export const SignUpUser = user => {

return fetch('http://localhost:8080/signup', {

method: "POST",

mode: "cors",

headers: {

Accept : "application/json",

"Content-Type": "application/json",

'Access-Control-Allow-Origin':'*'

body: JSON.stringify(user),

})

.then(response => {

return response.json()

})

.catch(err => {

return err

})

}

src/user/Signup.js

clickSubmit = event => {

event.preventDefault()

const { name, email, password } = this.state

const user = { name, email, password }

SignUpUser(user)

.then(data => {

console.log('data', data)

if (data.error) {

this.setState({ error: data.error })

} else {

this.setState({

email: "",

password: "",

error: "",

open: true

})

}

})

}

When the user successfully creates an account, we will show a link to sign in.

Authenticate a user

POST on http://localhost:8080/signin
body: JSON.stringify(user)

We first use the email and password to try to get user user data and a token back, which are both packaged into a object called ‘data’. Once we get the data, we pass it into AuthenticateUser, which will store it via localStorage at property ‘jwt’. Thus, we can always access the value at property ‘jwt’. We get this value, which has properties token and user, which we can use.

src/user/Signup.js

clickSubmit = event => {

event.preventDefault()

this.setState({ loading: true })

const { email, password } = this.state

const user = { email, password }

SignInUser(user).then(data => {

this.setState({loading:false})

if (data.error) {

this.setState({ error: data.error })

} else {

console.log('Signin.js - let authenticate it')

console.log('data received is: ', data)

// 1) authenticate

AuthenticateUser(data, () => {

// 2) redirect to profile page

this.setState({ redirectToReferer: true})

})

}

})

}

Once we get the token and user data successfully into the localStorage, we redirect to the main page.
The main page’s component is Home

Also, we have a Menu for navigation and showing/hiding of links according to authentication.

User Profile

GET on http://localhost:8080/user/${userId}

The main idea is that we must give the userId so the backend knows which user to fetch. We also need to give a token so that the backend can authenticate it.

There is a situation where if you’re on someone else’s (say tom’s) profile, and you click on the user profile (john) link in the Menu, the Profile’s component render function will be called first, which will fetch and render tom’s data again. But soon after, your click on john’s profile link will trigger john’s data to be fetched also. Hence, there will two fetches.

Edit user profile

PUT on http://localhost:8080/user/${userId}
body: user

Editing is basically about updating the user on the backend.

UpdateUser(userId, token, this.userData).then(data => {

if (data.error) {

this.setState({ error: data.error, loading: false })

} else {

updateLocalStorageUser(data, () => {

this.setState({ redirectToProfile: true, loading: false })

})

}

})

Once we update the user, we must remember to update the localStorage with the latest retrieved user data.

Delete user profile

removeUser(userId, token)

.then(data => {

if (data.error) {

console.log(data.error)

} else {

// signout user

// we provide callback for next() in SignOutUser

SignOutUser( () => console.log('User is deleted')) ;

// redirect

this.setState({ redirect: true });

}

})

removeUser does a fetch for Delete on http://localhost:8080/user/${userId}
where SignOutUser does a fetch for GET on http://localhost:8080/signout

Here, we tell the backend to clear cookie for this particular client. Cookies are stored per-user on the user’s machine. A cookie is usually just a bit of information. Cookies are usually used for simple user settings colours preferences ect. No sensitive information should ever be stored in a cookie.

View all users

GET on http://localhost:8080/users