IPFS is a peer-to-peer network for storing and sharing data in a distributed file system. There are various ways for uploading files. Among all using existing websites is the easiest option to try out, but if you want to upload programmatically you have to use APIs. Today I will be explaining two major services that are actively used for IPFS storage.

1. Web3 Storage
   Web3 storage is one of the best services to store data however it has limitations in the free plan as we can only upload up to 5 GiB of files. You can upload files vising the account tab. If you want to upload media using their API, then hover over to the account or visit this link to generate an API key which you can use later
2. NFT Storage
   NFT storage provides free storage for NFTs on IPFS and Filecoin. You can upload files in the Files table after creating an account. Also, we can generate an API key by visiting the API Keys tab which we will be using in the following section.

Using Web3 Storage

1. Create a project directory:

mkdir ipfsupload 

2. Initialize the project and link it to npm

cd ipfsupload
npm init -y

3. Install express and set up basic routes for file upload.

npm i express express-fileupload web3.storage dotenv nodemon

Add "type":"module" in package.json so that we can use module imports

Folder structure:

- server
    - controllers
    - routes
    - temp
    - index.js
- .env
- package.json

4. Roll up routes and express server

The final files look like this:

server/index.js

import express from 'express';

import routes from './routes/index.js';
const app = express();

const port = 3000;

app.get('/', (req, res) => {
    res.send('IPFS upload server is up and running!');
});

 // Mount endpoints
 app.use('/api/v1', routes)

app.listen(port, () => console.log('Server ready at: http://localhost:%s', port))

server/controllers/upload.controller.js

import dotenv from 'dotenv';
dotenv.config();

import fs from 'fs'
import { Web3Storage, File } from 'web3.storage';

const web3StorageAPIKey = process.env.WEB3_STORAGE_API_KEY;

function makeStorageClient() {
    return new Web3Storage({ token: web3StorageAPIKey });
}

async function makeFileObjects(name, description, image) {
    const obj = { name, description, image };
    const buffer = Buffer.from(JSON.stringify(obj))

    const files = [
        new File(['contents-of-file-1'], 'plain-utf8.txt'),
        new File([buffer], 'metadata.json')
    ]
    return files
}

async function storeFiles(files) {
    const client = makeStorageClient()
    const cid = await client.put(files)
    return cid
}

const fileFromPath = async (image, fileName) => {
    const filePath = image.tempFilePath;
    const mimeType = image.mimetype;

    const content = await fs.promises.readFile(filePath)
    const files = [
        new File(['contents-of-file-1'], mimeType),
        new File([content], fileName)
    ]
    return files
}


const web3StorageUpload = async (req, res) => {
    const { name, description } = req.body;
    const { image } = req?.files ?? {};
    console.log(`Uploading image: [${name}] to ipfs.`);
    try {
        if (!image && !name && !description || name === undefined) {
            return res.status(200).send({ message: 'invalid input' });
        }
        const imageName = `${new Date().getTime()}_${image.name.replaceAll(' ', '')}`;
        const file = await fileFromPath(image, imageName);
        const imageCid = await storeFiles(file);
        const files = await makeFileObjects(name, description, `https://${imageCid}.ipfs.w3s.link/${imageName}`);
        const metaDataCid = await storeFiles(files);
        await fs.promises.unlink(image.tempFilePath);
        const metadataUrl = `https://${metaDataCid}.ipfs.w3s.link/metadata.json`;

        const ipfsTierInfo = {
            name,
            description,
            ipfsUrl: metadataUrl,
        };
        // TODO: store metadata to db
        return res.send(ipfsTierInfo);
    } catch (error) {
        console.log(`Problem while uploading image to ipfs: ${error}`);
        return res.status(500).send({
            message: 'Problem while uploading image to ipfs'
        })
    }
}

export default {
    web3StorageUpload,
}

server/routes/updalod.route.js

import express from 'express';
import fileUpload from 'express-fileupload';
import controller from '../controllers/upload.controller.js';

const router = express.Router();
router.use(fileUpload({useTempFiles: true}));

router.route('/web3storage')
    .post(controller.web3StorageUpload);

export default router;

server/routes/index.js

import express from 'express';
import uploadRoute from './upload.route.js';

const router = express.Router();

// Mount upload endpoints
router.use('/upload', uploadRoute);

export default router;

.env

WEB3_STORAGE_API_KEY=<your_web3_storage_key>

5. Now you should be able to start the server and test the file upload from any rest clients.

npm run start

I am using postman and after successful upload API will respond following message.

GitHub Repository: IPFS Upload in NodeJs

In this article, I will discuss the working mechanism of HLS and how we can use HLS(HTTP Live Streaming) for audio/music streaming in NodeJs.

What is HLS?
According to Wikipedia: HLS is an HTTP-based adaptive bitrate streaming communications protocol developed by Apple Inc. It was released in 2009. Support for the protocol is widespread in media players, web browsers, mobile devices, and streaming media servers.

Why HLS?

1. Fast:
   As one large file is divided into segments a few segments in length and those files are fetched on respective player time with the help of the manifest/index file.
2. Adaptive Bitrate Streaming:
   It has the ability to adjust video quality in the middle of a stream as network conditions change. This ability allows videos to keep playing even if network conditions get worse; conversely, it also maximizes video quality to be as high as the network can support.
   If the network slows down, the user’s video player detects this, and adaptive bitrate streaming lowers the quality of the stream so that the video does not stop playing. If more network bandwidth becomes available, adaptive bitrate streaming improves the quality of the stream.
   Adaptive bitrate streaming is possible because HLS creates several duplicate segmented streams at different quality levels during the segmentation process. The user’s video player can switch from one of those streams to another one during video playback.

3. Unwanted i.e unwatched or unlistened portions of the file(audio/video) won’t be downloaded.

4. Less server and client configuration uses HTTP Protocol

How does HLS work?
Server: An HLS stream originates from a server where (in on-demand streaming) the media file is stored, or where (in live streaming) the stream is created. Because HLS is based on HTTP, any ordinary web server can originate the stream.

Two main processes take place on the server:

1. Encoding: The audio/video data is reformatted so that any device can recognize and interpret the data. HLS must use H.264 or H.265 encoding.
2. Segmenting: The audio/video is divided up into segments a few seconds in length.

• In addition to dividing the audio/video into segments, HLS creates an index file of the audio/video segments to record the order they belong in.
• HLS will also create several duplicate sets of segments at different quality levels: 480p, 720p, 1080p, and so on.

Distribution: The encoded video segments are pushed out to client devices over the Internet when client devices request the stream. Typically, a CDN or OFS like s3 will help distribute the stream to geographically diverse areas.

Client devices: The client device is the device that receives the stream and plays the video — for instance, a user's smartphone or laptop. The client device uses the index file(.m3u8) as a reference for assembling the audio/video in order, and it switches from higher quality to lower quality pictures (and vice versa) as needed.

Example: HLS Audio streaming(NodeJs)
1. MP3 to chunks:

First of all, initiate a project with npm init and add utils/mp3tochunks.js, the contents of mp2tochunks.js should be:

const util = require('util');
const exec = util.promisify(require('child_process').exec);
const fs = require('fs');
const path = require('path');

const dir = path.join(__dirname, '../songs');
const dest = path.join(__dirname, '../temp/chunks');

const startTime = new Date();
console.info('> Start reading files', startTime);

fs.readdir(dir, (readDirError, files) => {

if (readDirError) {
console.error(readDirError);
return;
}

const countFiles = files.length;

files.map(async (file, index) => {
const fileName = path.join(dir, file);
const { err, stdout, stderr } =
await exec(`ffmpeg -i ${fileName} -profile:v baseline -level 3.0 -s 640x360 -start_number 0 -hls_time 10 -hls_list_size 0 -f hls  ${dest}/${index}.m3u8`);

if (err) {
console.log(err);
}

if (countFiles - 1 === index) {
const endTime = new Date();
console.info('< End Preparing files', endTime);
}

});
});

In the above snippet, I have made a song folder that contains mp3 files that will later be converted into chunks and those chunks are stored on the temp/chunks directory.

The main task in this util is done by ffmpeg, which should be installed on your machine.

Command used:

ffmpeg -i ${fileName} -profile:v baseline -level 3.0 -s 640x360 -start_number 0 -hls_time 10 -hls_list_size 0 -f hls  ${dest}/${index}.m3u8

where,
-i : specifies the input file audio/video,
-profile: sets the audio/video encoding codec profiles, setting baseline, as it supports lower-cost applications with limited computing resources,
-level: level as mentioned in Annex A of the H.264 standard,
-s: sets frame size,
-start_number: Set the index of the file matched by the image file pattern to start to read from, the default value is 0.
-hls_time: segment duration in seconds,
-hls_list_size: Set the maximum number of playlist entries. If set to 0 the list file will contain all the segments. The default value is 5,
-f: Set the file format,
${dest}/${index}.m3u8: Sets Path of the chunks generated and manifest file m3u8

The content of the m3u8 file should be something like this:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:10
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:10.008778,
00.ts
#EXTINF:10.008778,
01.ts
#EXTINF:9.985556,
02.ts
#EXTINF:10.008778,
03.ts
#EXTINF:10.008778,
04.ts
#EXTINF:9.985556,
05.ts
#EXTINF:10.008778,
06.ts
#EXTINF:10.008778,
07.ts
#EXTINF:9.985556,
08.ts
#EXTINF:10.008778,
09.ts
#EXTINF:10.008778,
010.ts
#EXTINF:9.985556,
011.ts
#EXT-X-ENDLIST

where,
EXTM3U: this indicates that the file is an extended m3u file. Every HLS playlist must start with this tag.

EXT-X-VERSION: indicates the compatibility version of the Playlist file.

EXT-X-TARGETDURATION: this specifies the maximum duration of the media file in seconds.

EXT-X-MEDIA-SEQUENCE: indicates the sequence number of the first URL that appears in a playlist file. Each media file URL in a playlist has a unique integer sequence number. The sequence number of a URL is higher by 1 than the sequence number of the URL that preceded it. The media sequence numbers have no relation to the names of the files.

EXTINF: tag specifies the duration of a media segment. It should be followed by the URI of the associated media segment — this is mandatory. You should ensure that the EXTINF value is less than or equal to the actual duration of the media file that it is referring to

Once the manifest file is generated, we have to host it, lets add a node server and test it on hls player i.e. add main.js file that should have:

var http = require('http');
var fs = require('fs');

const port = 8000

http.createServer(function (request, response) {
console.log('request starting...');
var filePath = './temp/chunks' + request.url;

fs.readFile(filePath, function(error, content) {
response.writeHead(200, { 'Access-Control-Allow-Origin': '*' });

if (error) {
if(error.code == 'ENOENT'){
fs.readFile('./404.html', function(error, content) {
response.end(content, 'utf-8');
});
} else {
response.writeHead(500);
response.end('Sorry, check with the site admin for error:
'+error.code+' ..\n');
response.end();
}
} else {
  response.end(content, 'utf-8');
}
});
}).listen(port);

console.log(`Server running at http://127.0.0.1:${port}/`);

This basically takes the request URL and looks for the corresponding URL if that is generated in chunks, if that is generated it will respond with the contents of the file. for example: for request http://127.0.0.1:8000/0.m3u8 the manifest file will be downloaded, here to test the file you have to add one browser extension i.e. Native HLS Playback, that should play the media you have used. In the meantime, you can inspect the file sequence chunks requested by the client/browser from the server after every 10 seconds the chunk has been loaded.

That sums up the HLS audio/video streaming with nodejs. Here all the files are served locally, in an ideal case they are served through some online file storage service like AWS s3, for that, I have added uploadchunkstos3.js file in the repository of this article.

References

1. https://github.com/mondyfy/hls_demo
2. https://blog.boot.dev/javascript/hls-video-streaming-node/
3. https://hls-js.netlify.app/demo/
4. https://github.com/video-dev/hls.js/
5. https://scanskill.com/programming/how-to-use-hlsjs-for-video-streaming/
6. https://www.cloudflare.com/learning/video/what-is-http-live-streaming/
7. https://medium.com/sharma02gaurav/adaptive-bitrate-streaming-hls-vod-service-in-nodejs-8df0d91d2eb4
8. http://blog.mediacoderhq.com/h264-profiles-and-levels/

As per Wikipedia Basic Authentication is a method for an HTTP user agent (e.g. a web browser) to provide a user name and password when making a request. In basic HTTP authentication, a request contains a header field in the form of Authorization: Basic <credentials>, where credentials is the base64 encoding of id and password joined by a single colon :

The best way I have found is to do the job with request-promise (The simplified HTTP request client ‘request’ with Promise support.)

/**
* Basic authuthentication request in node
* @param {*} username
* @param {*} password
* @param {*} Method
* @param {*} requestURL
*/
async function basicAuthRequestInNode(username,password,Method,requestURL){
const jsonData = {
key:value,
........

}

const auth = 'Basic ' + new Buffer(username + ':' + password).toString('base64');

const options = {
headers: {
Authorization: auth,
'content-type': 'application/json',},
method: Method,
body: jsonData,
uri: requestURL,
json: true,
};

try {
const response = await rp(options); // rp is request-promise
return response;
} 
catch (error) {
return error;
}
}

Above function can handle the basic authentication in nodejs.

connect IPS ,said to be a single payments platform that allows the customers to link their bank account(s) to enable payment processor, fund transfer and bill payments. An extended product of Nepal Clearing House to support citizen-to-government (C2G), customer-to-business (C2B) and peer-to-peer (P2P) payment transactions directly from/to the bank accounts.

First of all any financial organizations get registered on connectIPS and connectIPS provides back with necessary credentials ,now it’s turn of developer to connect connectIPS and their app/product.From the starting of this week i was also trying to do so,finally i made it now, yay!!

** Probably connectIPS being new to the market doesn’t have good developer support. **

Talking about development

I have used request-promise, pem and node-rsa ,we need to use pem for extracting certificate information and private key values from .pfx file .

// Generate Private Key
async function generatePrivateKey(pfx) {
return new Promise(async (resolve, reject) => {
return pem.readPkcs12(pfx, pkOptions, (err, cert) => {
if (cert) {
resolve(cert.key);
} 
else {
reject(err);
}
});
});
}

Response of pem looks similar to

{ cert: "...", ca: ["subca", "rootca"], key: "..." }

Now you can see the information of certificate and private key, as pem returns only the encrypted private key (RSA PRIVATE KEY), but we need to use private key without encryption .Unfortunately pem doesn’t provide this, here we need to convert pkcs12 private key into pkcs8 private key.We can do this with:

const pfx = fs.readFileSync("test.pfx");
const RSAKey = await generatePrivateKey(pfx);
const key = new NodeRSA(RSAKey);
const privateKey = key.exportKey("pkcs8");

Now we can use this privateKey to generate hash,

// Generate Hash
/**
 * @param {[obj]} data [data that you have to send]
*/
async function generateHash(data) {
const pfx = fs.readFileSync("test.pfx");
const RSAKey = await generatePrivateKey(pfx);
const key = new NodeRSA(RSAKey);
const privateKey = key.exportKey("pkcs8");

const signer = crypto.createSign("sha256WithRSAEncryption");
signer.update(data);
const hash = signer.sign(privateKey, "base64");
return hash;
}

Now we have to send request to connectIPS with following options,

const options = await generateOptions(hash, gatewayURL, method);

We can get response at:

//rp(request-promise)
try {
const response = await rp(options);
return response;
} catch (error) {
console.log("error => ", error);
}

(…To Be Continued…)

Compiling the Kernel

This board uses a 4.1.15 Kernel on most of all images and is based on the NXP’s which are changed from upstream for their board support. We have to change them to support the various hardware that we use with this processor.

The “imx_4.1.15_1.0.0_ga” branch in embeddedarm/linux-3.10.17-imx6 includes support for TS-4900, TS-7970, TS-7990, and TS-4100.Also this kernel works with all Debian releases, Ubuntu, Yocto Jethro, and Yocto Krogoth.

The kernel branch can be selected once the git repository is cloned as outlined below. The kernel can be rebuilt by cross compiling from an X86/X86_64 Linux. The default kernels are rebuilt using the toolchains built by Yocto. You can download the appropriate cross toolchain for your Linux system here:

• x86_64
• i686

Now open your terminal (Assumed root directory here) and do,

chmod a+x poky-*toolchain-*.sh
sudo ./poky-*toolchain-*.sh

This will ask for the install directory for the toolchain. As this instructions assumes that the default path is used.

This also requires several tools from the host PC linux distribution. For Ubuntu/Debian:

sudo apt-get install git build-essential lzop u-boot-tools libncursesw5-dev

Once those are installed follow:

# This will pull our shared kernel, using the 4.1.15 branch, use a folder linux-imx6, and only
# download the latest changes.
git clone https://github.com/embeddedarm/linux-3.10.17-imx6.git -b imx_4.1.15_1.0.0_ga linux-imx6 --depth 1

Now change the arch/arm/boot/dts/imx6qdl-ts7970.dtsi file inside linux-imx6 folder like:

mxcfb1: fb@0 {
		compatible = "fsl,mxc_sdc_fb";
		disp_dev = "ldb";
		interface_pix_fmt = "RGB666";
		default_bpp = <16>;
		int_clk = <0>;
		late_init = <0>;
		status = "okay";
	};
in the mxcfb1 node (This change is for redirecting the default display from HDMI to LVDS)

Here define ldb( disp_dev = “ldb” ) as:

&ldb {
	status = "okay";
 
	lvds-channel@0 {
		crtc = "ipu1-di1";
		fsl,data-mapping = "spwg";
		fsl,data-width = <18>;
		status = "okay";
		primary;
 
		display-timings {
			native-mode = <&timing0>;
			timing0: hantronix-svga1 {
				clock-frequency = <40000000>;
				hactive = <800>;
				vactive = <600>;
				hback-porch = <46>;
				hfront-porch = <210>;
				vback-porch = <23>;
				vfront-porch = <12>;
				hsync-len = <20>;
				vsync-len = <10>;
				de-active = <1>;
				hsync-active = <1>;
				vsync-active = <1>;
				pixelclk-active = <0>;
			};
		};
	};
};

(Note : This is for the 10" Hantronix display used on the TS-TPC-8950:,please customize it according to your LCD )

Now,

cd linux-imx6
 
# These two must be run each time you open a terminal to build the kernel.
export CROSS_COMPILE="/opt/poky/2.2.2/sysroots/x86_64-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/arm-poky-linux-gnueabi-"
# Or if you are using the i686 toolchain:
# export CROSS_COMPILE="ccache /opt/poky/2.2.2/sysroots/i686-pokysdk-linux/usr/bin/arm-poky-linux-gnueabi/arm-poky-linux-gnueabi-"
export ARCH=arm
export LOADADDR=0x10008000
 
make ts4900_defconfig
 
## Make any changes in "make menuconfig" or driver modifications, then compile
make && make uImage

Installing the Kernel

To install this to an SD card, use a USB SD card reader connected to the host PC.

(Note: Before installing the kernel in SD card ,first backup the card.

On Linux, you can use the standard dd tool:

sudo dd if=/dev/sdx of=<your backup path>/backup.img

or for compression:

dd if=/dev/sdx | gzip > /<your backup path>/backup.img

Where /dev/sdx is your SD card.

To correctly identify your card, perform “df -h” before and after inserting the SD card and observe the new paths. These new paths represent your SD card.that new path might be /dev/sdb1 or /dev/sdc1. )

Assuming the SD card has the Linux rootfs set up on “sdc1” do:

export DEV=/dev/sdc1
sudo mount "$DEV" /mnt/sd

# By default your os might not provide the mnt directly and error with can't point to /mnt/sd might appear ,if that happen then make a directory with mkdir -p /mnt/sd .

sudo rm /mnt/sd/boot/uImage
sudo cp arch/arm/boot/uImage  /mnt/sd/boot/uImage
sudo cp arch/arm/boot/dts/imx6*ts*.dtb /mnt/sd/boot/
INSTALL_MOD_PATH="/mnt/sd" sudo -E make modules_install 
INSTALL_HDR_PATH="/mnt/sd" sudo -E make headers_install
sudo umount /mnt/sd/
sync

The modified kernel is installed in your SD card ,now insert the SD card in your device and start the raspberry pi ,wait for a minute and finally you will figure out your working LVDS display screen …Zomg !!! enjoy it.

References

1. https://wiki.embeddedarm.com/wiki/TS-7970#Compile_the_Kernel

Step 1:- First of all visit your github dashboard and then create a repo either private or public but don’t initiate with README and any License Templates,and then click create new repository button, after that you will be redirected to Quick Set up Page, now copy the project URL with HTTPS option.

Step 2:- Open Terminal and then change the directory to your working or project Directory.

Step 3:- Initialize your local project as a Git Repository with
$git init .

Step 4:- Add the file to a repository with $git add . , This adds the files of your local repository to staging for your initial commit.

Step 5:- Commit the files that you have added or staged in your local repository with $git commit -m “your commit message”.

Step 6:- Now add the URL of your remote repository that you have copied from Quick Setup Page with $git remote add origin < your remote repository URL> and then verify it with $git remote -v .

Step 7:- Now Push your staged files to remote repository with $git push origin master

After that, you will be asked for Github username and password please provide those credentials, as soon as you enter your password, your local project starts pushing to Github.

Last Saturday (January -5–2019 ) we had a talk on open source tools,softwares,utilities and philosophies. In the event Rameshwor Dhakal Dai from 8th semester(one of the executive member of Mozilla ASCOL) presented different open source tools alternative to paid products.

Mr. Dhakal also expained in brief about open source software,what it mean?,what it does ?,communities behind it ,activities happening these days and many more.

Some of the facsimile of the event are :-

About 24 attendees were present on the event from 1st,5th and 8th semester,they were interested in Open Street Mapping among all tools discussed on the session,also they were eager to listen about Toilet Mapping Initiation taken by Kathmandu Living Labs recently. I think we need to have one session soon. Here is the link for the rough slides we have presented in our talk.

Open Source Tools( Open meet week #3) was originally published in Mozilla ASCOL on Medium, where people are continuing the conversation by highlighting and responding to this story.

Program was officially started with the introduction of the Club,it’s members along with it’s gosls and objective by Prajwol Tiwari(an active member of Mozilla ASCOL) and then every participant introduced to other.

Thoroughly, Mr. Prajwol expained the programming language Python,its current status,Scope,resources to learn it and communities behind it. He justified why newbies should start with python with some satisfying examples. Also he have compared other languages like Java,PHP,R, .Net, React Native etc, their popular frameworks like Spring,Laravel,Flutter etc. with use and applications.

After a while Prajwol shown a demo of web scraping on the news portal using Beautiful Soup, One of the popular module of Python,The session was highly interactive though most of the participants were new comers i.e. from first semester.On the occasion Aayush Paudel(AI enthusiast,
City ambassador of nature.ai and one of the community lead of AID for Nepal) has added about different method of scraping on the varying DOM tree of different websites.

In the fullness of time our senior brother Mr. Roshan Gautam (Mozilla Reps) Introduced us about Common Voice project of Mozilla it’s need,importance and impacts. He also described how our contribution matters.

At last Mr. Nirmal Rijal, Mozilla ASCOL Club Captain summarized the day long work and started discussion for next session. Overall, the event was ended successfully.

Web Scraping With Python (Open Meet Week #2) was originally published in Mozilla ASCOL on Medium, where people are continuing the conversation by highlighting and responding to this story.