McFly upgrades Bash with artificial intelligence

The most popular Linux shell by far is the GNU Bourne-Again Shell (Bash for short). One of its many outstanding features is its integrated command history, where Bash saves previously run command lines so that you can recall them later on without a lot of typing.

Listing 1 shows the output you'll see if you enter the history command at the terminal prompt. Each line begins with a line number in ascending order. The last command line to have been entered is at the end of the history list. You can use the arrow keys to scroll backward or forward one step in the history in the terminal. Pressing the Enter key copies the displayed line and runs it again.

§§nonumer
01 $ history
02 [...]
03   446  nano .config/GIMP/2.10/scripts/image-subdivide.scm
04   447  nano .config/GIMP/2.10/scripts/slice-and-join.scm
05 [...]
06   901  yay -Syu
07   902  trizen -Syu
08   903  yay -Syu
09 [...]
10  1044  ++
11  1045  ++ /tmp/
12  1046  mv /tmp/*png . -v
13  1047  ++
14  1048  mv /tmp/*png . -v
15  1049  ++
16  1050  mv /tmp/linify.png .
17 $ history | grep slice
18   447  nano .config/GIMP/2.10/scripts/slice-and-join.scm
19  1051  history | grep slice

Bash supports two approaches to searching for specific strings directly in the history: Ctrl+R searches backwards from the current cursor position; Ctrl+S searches forwards from the current cursor position. Ctrl+S only makes sense if you have already advanced further back in the history and are no longer at the end. In addition, entering Ctrl+S only works in an appropriately configured terminal; normally, this keyboard shortcut blocks the terminal.

To indicate reverse searching in the history, Bash uses Ctrl+R with a special prompt:

(reverse-i-search)`':

i-search stands for an incremental search, which refines or enhances the search by entering a new character. The function finds the last line in the history that matches the search criteria; each time Ctrl+R is pressed, the search jumps one match further back into the past.

In practice, this form of search in the history proves to be as simple as it is effective. A few cleverly selected entries are usually sufficient to find the desired line. However, you need to know what you are looking for – and, above all, you need to know a good pattern that will quickly take you to the desired command. The pattern can be anywhere on the command line, so you don't have to type it from the beginning. (See also the "Search Patterns" box.)

For instance, to retrieve the command from Listing 1 (line 16)

mv /tmp/linify.png .

you would just need to type Ctrl+R I F Y. Command lines such as trizen -Syu or yay -Syu can be quickly found with Ctrl+R, Shift+S, or Ctrl+R - Shift+S.

As shown in Listing 1, the history usually contains command lines that are very similar to each other and can be described well with patterns, but cannot be easily distinguished. A remedy for this requires a new approach.

Modern Times

Bash history lacks the ability to search in a context-based way. For example, on many modern systems based on systemd, /var/log/journal/ contains large numbers of logfiles. Deleting outdated logfiles can often free up a large amount of storage space.

To see if, and how many, old logfiles are stored there, change to the directory. The du -sh command then determines the space used by the existing files. find -mtime 7 lets you search for all files that have not been changed in over one week; you can then delete them using rm [...].

You could now write a script or – preferably – a shell function for this task in a few moments, but the problem remains the same: You have to actively develop a solution for every situation. Wouldn't you prefer that the shell guessed what you were going to do based on the directory and offered you a corresponding action?

McFly [1] does this by reading the previous history, analyzing it in a synthetic neural network, and attempting to recognize correlations in the command lines by machine learning. These relationships then serve as the basis for the suggestions offered by the shell's search functions.

In contrast to the neural network's quite complicated structure, McFly's last step is very easy to understand: The previous-history function normally mapped to Ctrl+R is replaced by the new mcfly search function.

The following parameters serve as the basis for the neuronal evaluation of the history and the McFly suggestions for the search function, in the specified order:

• The current directory
• The previously entered command lines
• The frequency and order of the call
• Whether a command line was previously found by McFly
• Whether the command ran without errors

After setting up McFly (see the box "Installing McFly"), the shell behavior changes. Ctrl+R calls the tool directly; the titlebar changes to reflect this, and the new key bindings become active (Figure 1).

Figure 1: McFly extends the search for commands in the shell's command history.

yay -S mcfly

Unable to add cmd_tpl to commands: SqliteFailure([...]])',   src/libcore/result.rs:1009:5

^@^@^[...]^@^@sudo

§§nonumer
01 $ sudo apt install linuxbrew-wrapper
02 $ brew
03 $ echo 'PATH="/home/linuxbrew/.linuxbrew/bin:$PATH"' >> ~/.profile
04 $ echo 'MANPATH="/home/linuxbrew/.linuxbrew/share/man:$MANPATH"' >> ~/.profile
05 $ echo 'INFOPATH="/home/linuxbrew/.linuxbrew/share/info:$INFOPATH"' >> ~/.profile
06 $ PATH="/home/linuxbrew/.linuxbrew/bin:$PATH"
07 $ brew tap cantino/mcfly https://github.com/cantino/mcfly
08 $ brew install mcfly
09 $ source "$(brew --prefix)/opt/mcfly/mcfly.bash"

§§nonumer
if [ -f $(brew --prefix)/opt/mcfly/mcfly.bash ]; then
. $(brew --prefix)/opt/mcfly/mcfly.bash
fi

While initializing, McFly reads the existing history and evaluates it, which takes some time. You can press Esc to exit this mode and return directly to the prompt. Then use the up and down arrows to navigate the recommendations offered by McFly. When you enter a search term, McFly retrieves matching hits from the history.

Press Enter to accept the currently highlighted line from the history and execute the command immediately. The Tab key, on the other hand, transfers the highlighted history line to the input prompt without executing the command immediately. You can then edit the command and press Enter to execute it. Press F2 to delete individual lines from the list of suggestions after a prompt.

Artificial Intelligence

The interesting thing about McFly is its underlying artificial intelligence, the neural network. What used to require a huge amount of power and entire data centers, can now (with some limitations) even be performed by a Raspberry Pi. Today, such small neural networks are useful for all kinds of tasks, from image processing to character and pattern recognition.

These systems' missing logic often proves to be problematic. Neural networks learn coherencies without it being possible to clearly understand how they do it in each case. With McFly, however, you can still see the learning behavior, or at least the rudiments of it. If you call a command from the list of suggestions, McFly moves the command further up the list the next time it is called. However, it is less clear which commands appear (and in which order) initially in the proposal list. The rules used already seem so complex that some effort is needed to predict their impact.

You actually encounter a very similar system every day when searching on the Internet. Here, too, it is often impossible to identify how, where, and why certain results appear in the hit list while others do not. This problem is attributable to the principle of neural networks and is not completely harmless. For this reason, IT researchers repeatedly urge cautious use of the technology [4].

Conclusions

In addition to autocompletion, the history is an important shell feature for interactive use. McFly adds value to the history command. Fortunately, the developer implemented the tool as an independent program instead of adding it to the already complex Bash or developing a plugin. This minimizes the side effects and allows the tool to be tried out quickly and safely.

As far as McFly's AI performance is concerned, a typical problem of self-learning systems applies here. It is virtually impossible to make reliable statements about the software, since using the software changes the system itself at run time. In my test, McFly delivered very good results; most of the command lines I searched for were found in the first list of suggestions.

If you are just looking for an upgrade of history, there are simpler alternatives in the open source universe, such as hstr [5] and cdhist [6], which do without neural networks and take up far less disk space.