Flexible file chunking for Node.js applications with a mini-sdk

Introducing Triple Dot

If you’ve happened to see any of my recent tweets, im working on an OSS personal file system that can be controlled by the terminal

So, in order to be able to chunk my files into many, smaller files to upload them progressively and without causing a memory overflow.

We already know that node programs arent the most momory efficient ones and i dont want to make mine a nightmare. Soooooooo i had to build a file chunker.

This is where i built Trple Dot (or “…” or the chunking of a line)

Lets chunk a file together!

Installation

bun install @andriotis/triple-dot

Usage method 1: File System Chunks

import td from "@andriotis/triple-dot";

await td({
    // chunk size in KB
    chunkSize: 1024,
    filePath: "./bun.lockb",
    outputDir: ".triple-dot"
})

after a bit, .triple-dot will contain files with the format #.chunk where # is the number of the chunk

Before we move on to the second method, id like to mention the function getFileChunkStats that allows you to predict the behaviour of triple dot by providing you with the chunk amount it will generate for a given file as well as its stats from fs.

Usage

import { getFileChunkStats } from "@andriotis/triple-dot";

const stats = await getFileChunkStats("bun.lockb", 1024);
console.log(stats);

Output

{
  "chunks": 1 (or another number),
  "stats": ... (typeof fs.Stats)
}

Usage method 2: Callback Function Chunks

Triple Dot has the option to pass the chunk buffer to an onProgress function instead of writing to the file system. Lets see how this works

import td from "@andriotis/triple-dot";

const outputToProgress = true;

await td({
  chunkSize: 1024,
  filePath: "bun.lockb",
  outputDir: ".chunks",
  passChunkToCallback: outputToProgress,
  onProgress: (chunkNumber, totalChunks, buffer) => {
    // do stuff with the buffer....
    return;
  },
});

And that’s how you can chunk a file with triple dot! If you have any questions, feel free to email me at medium@andriotis.me

Oh, i almost forgot. Triple dot is OSS. You can check it out here: https://github.com/andriotisnikos1/file-chunker

thats all for today. adios!