If you have ever dealt with systems programming or data engineering or big data processing, it’s certain that you have shell scripting as one of the tools in your toolbox. We all use shell scripts - be it Bash or Korn or Zsh - to glue together workflows in the form of wrapper scripts.

In this post, let’s explore Nu Shell, which is written completely in Rust. Hear me out - I know you must be thinking, oh, great, one more shell, just what we need. However,

... think about bringing SQL to the command line, that is what Nu does and then some.

nu-logo

Now that I got your attention, let’s explore Nu. If you want to follow along and bring up Nu environment locally, please feel free to clone my github repo and compose up using,

git clone https://github.com/vangalamaheshh/rust-esh-cean
cd rust-esh-cean
docker network create rust-net
docker-compose up -d
docker exec -it rust-esh-cean bash
mkdir -p /mnt/test-ground/nu-shell && cd $_

nu command brings up the Nu Shell into scope.

Displaying as Table

[US/Eastern Asia/Kolkata Europe/Berlin US/Hawaii] | wrap Zone | upsert Time {|it| date now | date to-timezone $it.Zone | date format %c}

Output:
╭───┬───────────────┬──────────────────────────╮
│ # │     Zone      │           Time           │
├───┼───────────────┼──────────────────────────┤
│ 0 │ US/Eastern    │ Mon Mar 28 10:10:51 2022 │
│ 1 │ Asia/Kolkata  │ Mon Mar 28 19:40:51 2022 │
│ 2 │ Europe/Berlin │ Mon Mar 28 16:10:51 2022 │
│ 3 │ US/Hawaii     │ Mon Mar 28 04:10:51 2022 │
╰───┴───────────────┴──────────────────────────╯

Parallel execution using par-each

[1 2 3 4 5] | par-each -n { |it| echo $"sleeping in thread: ($it.index)"; sleep 2sec; }

Note: Depending on # of cores on your computer, that number of threads will execute in parallel.

Parallel execution using map-reduce

[1 2 3 4 5] | reduce -n { |it, acc| echo $"#($it.index) Item1: ($it.item); Item2: ($acc)"; $it.item + $acc } | str collect | lines | wrap total

Output:
#1 Item1: 2; Item2: 1
#2 Item1: 3; Item2: 3
#3 Item1: 4; Item2: 6
#4 Item1: 5; Item2: 10
╭───┬───────╮
│ # │ total │
├───┼───────┤
│ 0 │ 15    │
╰───┴───────╯

Nu strings

"hello how are you?"

Output:
hello how are you?

Nu lists

"hello how are you?" | split row " "

Output:
╭───┬───────╮
│ 0 │ hello │
│ 1 │ how   │
│ 2 │ are   │
│ 3 │ you?  │
╰───┴───────╯

Nu tables

"hello how are you?" | split row " " | wrap "Column1"

Output:
╭───┬─────────╮
│ # │ Column1 │
├───┼─────────┤
│ 0 │ hello   │
│ 1 │ how     │
│ 2 │ are     │
│ 3 │ you?    │
╰───┴─────────╯

Nu dataframes

Dataframes are similar to tables. However, Nu Dataframes are built on top of Apache Arrow and Polars, so they are blazing fast and memory sparing.

1..10 | each { |it| [[scores1 scores2]; [(random integer 60..100) (random integer 60..100)]] } | flatten | dfr to-df

Output:
╭───┬─────────┬─────────╮
│ # │ scores1 │ scores2 │
├───┼─────────┼─────────┤
│ 0 │      94 │      97 │
│ 1 │      71 │      63 │
│ 2 │     100 │      83 │
│ 3 │      90 │      60 │
│ 4 │      72 │      85 │
│ 5 │      66 │      86 │
│ 6 │      63 │      68 │
│ 7 │      89 │      90 │
│ 8 │      73 │      66 │
│ 9 │     100 │      99 │
╰───┴─────────┴─────────╯

We can perform operations on dataframes. For example,

1..10 | each { |it| [[scores1 scores2]; [(random integer 60..100) (random integer 60..100)]] } | flatten | dfr to-df | dfr aggregate sum

Output:
╭───┬─────────┬─────────╮
│ # │ scores1 │ scores2 │
├───┼─────────┼─────────┤
│ 0 │     795 │     807 │
╰───┴─────────┴─────────╯

In addition, Nu offers modules, custom functions, linux style piping with external commands and many more. I think of it as bringing Rust, Pandas and SQL combined power to the shell scripting.

Happy Nu shell coding!!