Nodejs - KidsIL

1. Home »
2. Nodejs
Packaging NodeJS scripts into a Binary
Posted on Mar 27, 2020
Hey Node devs, jealous of Go devs for being able to create single binaries without any dependencies?
Now you can do it too! With pkg - a binary compiler for NodeJS!
pkg - from NodeJS to binary
I recently developed a NodeJS script that automatically consumes SQS messages when an EC2 instance is initiated.
My most major issue has been the dependencies - NodeJS installation was often slow, and would sometimes fail or timeout.
On top of that, I had to also install the AWS SDK in order to get the SQS messages.
The entire thing was too much of a delay for my User Data script to handle.

Side Note: User Data is a script that executes when an EC2 instance is initialized. Very useful for pulling Docker images or running init scripts.

After an evening of debugging, I started thinking - this would be a whole lot easier with Go, as I would just have a single binary file and zero dependencies to install.
The issue was that converting my entire script to Go would've required much more time, and would likely create new issues (as AWS SDK varies between the languages, not to mention I'm much more experienced with NodeJS compared to Go).
I then thought that within the infinite universe of NPM there must be a tool that packages binaries out of NodeJS scripts, or at least somehow bundles the NodeJS runtime together with my script.
pkg to the rescue!
pkg allows you to bundle your NodeJS scripts into a single binary.
No more dependencies, no more node_modules/, package.json, npm install...
Simply download the single executable and... well... Execute!

I have successfully tested the following with pkg:
- NodeJS environment (obviously) - includes Node8, Node6, and Node4 (9 is not supported yet)
- Operating System - includes linux, mac, and windows
- NPM Libraries (aws-sdk in my case) - works without any configuration
- Environment Variables - works without any configuration
- Spawning Child Processes - works without any configuration
- Async/Await - works without any configuration
Install pkg
```
npm install -g pkg
```
Tip: it also works if you install it locally, you would simply have to run:
```
./node_modules/.bin/pkg
```
instead of just pkg
Run pkg
```
pkg --targets node8-linux server.js -o server-linux
```
The command above simply bundles server.js into a linux binary named server-linux

Until next time!
Read more ›
Tagged with nodejs, Binary
Posted in Technology
How to install global NPM packages locally
Posted on Sep 27, 2018
Too many NPM libraries are demanding to be globally installed, to name a few:
```
npm install -g serverless
npm install -g terraform
npm install -g mocha
```
While it's sometimes very useful to have some of these components globally available,
in many cases you only need them for 1-2 projects.
In some cases, you might not have permissions to install packages globally.

The reality is, you don't need to install them globally at all!
Installing a global NPM package locally
Let's take Serverless as an example. Let's say we want to deploy an existing project, but we don't have Serverless installed on our system.
First we install the package locally:
```
npm install serverless
```
Then, on our node_modules/ directory, there's a hidden .bin/ directory, where the serverless binary is located.
So from the main project directory, we simply run:
```
./node_modules/.bin/serverless deploy -v
```
Now if you want to make it a bit more snappy, you could add a shortcut (i.e. symbolic link) to the binary:
```
ln -s node_modules/.bin/serverless serverless
```
And from now on simply run any serverless command like so:
```
./serverless deploy -v
```
On rare occasions, some packages' binaries are not under node_modules/.bin/ but somewhere inside the package's directory, such as node_modules/serverless/bin/.

So far I've been using locally installed packages for Serverless, SailsJS, Mocha, Terraform, and a few others as well.

On the next post, we'll try playing around with Serverless and S3, stay tuned!
Read more ›
Tagged with npm, node nodejs
Posted in Technology

Adding Serverless to your Web Application

Posted on Oct 31, 2017

As you've probably heard, Serverless is the next big thing in the cloud industry.
Here's I'll go over some of the ways you can integrate Serverless technology in your existing web apps!

Serverless is essentially a system of Function-as-a-Service, where you have certain functions in the cloud, and you get billed by their usage & required computation.
This is a further abstraction from the actual Virtual Machines in common use today, and goes hand-in-hand with a Microservices approach to web development.
But enough about dictionary definitions, let's go to the good stuff:

Examples of integrating Serverless with your existing Web App

Reports

Database reports are a pretty common request, often by marketing/business departments to get a better understanding of your users.
This is more common in early-stage applications, before you connect them to proper analytics tools.
Using Serverless, it's quite simple to have a function that runs a report on the database.
We'll use AWS Lambda in this example:

Go to Lambda on your AWS Dashboard:
Choose "Author from scratch":

In your Lambda function, add the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
'use strict';
module.exports.runReport = (event, context, callback) => {
  const response = {
    statusCode: 200,
    body: {
      message: 'Result: ',
      input: event,
    },
  };
  const mysql = require('mysql');
  const connection = mysql.createConnection({
    host: 'localhost',
    user: 'user',
    password: 'password',
    database: 'database',
    port: 'port' //if you remove this line port defaults to 3306
  });
  connection.connect();
  connection.query('SELECT * FROM users', function(err, results) {
    if (err) throw err;
    response.body.message += JSON.stringify(results);
    response.body = JSON.stringify(response.body);
    callback(null, response);
  })

  connection.end();

};

As an easier alternative, get the entire Serverless Boilerplate here

Admin Panel

In most web applications there's some sort of an Admin Panel.
Ideally, the admin panel should be a separate application, to limit potential systemic weaknesses.
Using Serverless, we're able to take advantage of some of the most secure systems available - IAM permissions.
Examples of cool things you could do to further secure your Admin Panel:

Restrict Admin functions permissions - Since Lambda functions have their own IAM roles, you can have very fine-grained permission settings. A great example of this is with DynamoDB.
You can literally give your function access to specific tables with a custom IAM policy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1509470869566",
      "Action": [
        "dynamodb:BatchGetItem",
        "dynamodb:DescribeTable",
        "dynamodb:GetItem",
        "dynamodb:GetRecords",
        "dynamodb:ListTables",
        "dynamodb:Query",
        "dynamodb:Scan"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:dynamodb:YOUR-REGION:YOUR-AWS-ID:table/TABLE-NAME"
    }
  ]
}

Or another great example where a function can only write items to the table without reading any:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1509470869566",
      "Action": [
        "dynamodb:PutItem"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:dynamodb:YOUR-REGION:YOUR-AWS-ID:table/TABLE-NAME"
    }
  ]
}

Combine API Gateway with Cognito User Pools for authentication - This is quite complex, but this guide has a pretty good walkthrough. In the following days (probably weeks) I'll try to add this to the boilerplates repository.
Write the entire Admin Panel on Serverless - If your app doesn't have a fully fledged Admin Panel yet (and you understood the last 2 items on the list), This could offer both a high level of security as well as a great way to integrate Serverless into an important but not-yet-customer-ready parts of your system.

Cron Jobs

Most applications have some form of Cron Jobs, but more often than not,
there's a good chance that you don't have a lot of logging/analytics for your Cron Jobs.
With CloudWatch you can schedule functions to run every minute/hour/day/week, just like a normal Cron Job.
Combine that with CloudWatch Alarms, and you have both a logging system and an alarm system in case something goes wrong.

These were just a few examples of how you could integrate Serverless into your web app right away.
In the future, I'll try to work on the boilerplate repository, to add more cases.

View-specific JS Files in Sails.js

Posted on Jun 15, 2017

One thing I really loved about Meteor, is that you can fine-tune which files are included (and the order of inclusion).
This is especially useful when structuring everything in small Meteor packages.
So for example, if you simply want to add relevant client JS files (on package.js):

1
2
3
4
api.addFiles([
    'client/page-login.html',
    'client/page-login.js',
], 'client');

page-login.html:

1
2
3
4
5
6
7
<template name="PageLogin">
  <form id="login-form">
    <input name="username" type="text" />
    <input name="password" type="password" />
    <input id="submit-login" type="submit" />
  </form>
</template>

page-login.js:

1
2
3
4
5
6
Template.PageNewDomain.events({
  'submit #login-form': function (event, template) {
    event.preventDefault();
    //handle form submission..
  }
});

Together with specific Template namespaces, it allows you to have template-specific JS code that doesn't interfere with other templates. I looked for a way to have that feature in Sails.js.

Instead of trying to implement the entire complicated Template rendering engine, I figured it should be enough if every view gets a corresponding JavaScript file that is optionally loaded with said view.

How it should work

Each View in the views/ folder has a name, let's create a corresponding JS file with the same name
On Sails.js, JavaScript files are stored separately in the js/ folder, let's store all view-specific JS files under js/views/
JS filenames should be the same as view filenames. views/dashboard/login.ejs will have a JS file named js/views/dashboard/login.js

How to implement

add this to config/views.js:

1
2
3
4
locals: {
    existsSync: require('fs').existsSync,
    jsPath: __dirname + '/../assets/js/views/'
}

This is so that we're able to check if a JS file exists.

on the bottom of views/layout.ejs, under blocks.localScripts

1
2
3
<% if (existsSync(jsPath + req.options.controller + '/' + req.options.action + '.js')) { %>
  <script src="/js/views/<%= req.options.controller + '/' + req.options.action %>.js"></script>
<% } %>

In tasks/pipeline.js in the jsFilesToInject array, we need to remove
```
js/**/*.js
```
This is Critical. Otherwise, all JS files under js/views will be automatically included in ALL views.
Instead, to automatically inject other views we can have the following line:
```
js/dependencies/**/*.js
```

That's it! Now you'll be able to write view-specific JavaScript files!

Asaf Zamir – Chief Technology Officer and Cloud Consultant

Packaging NodeJS scripts into a Binary

pkg - from NodeJS to binary

pkg to the rescue!

Install pkg

Run pkg

How to install global NPM packages locally

Installing a global NPM package locally