Working with files

Navigating through the Linux command line

Creating and navigating to directories

You can create directories with the mkdir command and navigate with the cd command.

# Make a directory "hpc_tutorial"
mkdir hpc_tutorial
# Change to the "hpc_tutorial" directory
cd hpc_tutorial
# Print the working directory path
pwd
# Move back to the parent directory
cd ..

Creating files and directories with names WITHOUT spaces is much easier to navigate on the command line.

. refers to the current directory and .. refers to the parent directory.

Creating and setting permissions on files

You can create and edit files either by using Vim or by using touch. The example below creates a file, uses echo to insert some text, makes the file executable, and then executes it.

# Make a file "hello.sh"
touch hello.sh
# Add the text "echo \"hello world\"" to hello.sh
echo "echo \"hello world\"" > hello.sh
# Print the contents of hello.sh
cat hello.sh
# Make hello.sh executable for the user
chmod u+x hello.sh
# Execute hello.sh
./hello.sh

chmod lets you control permissions on a file you own. For more info, see this tutorial.

To execute a file, include the absolute path In the preceding tutorial, ./ is a shortcut that means "this directory"

Listing files

You can list files in a directory using ls. The options -l and -a list the files with permissions and owner and list hidden files, respectively. You can use options together (for example, ls -la).

# List the files in your current directory
ls
# List the files in your current directory with permissions and owner
ls -l
# List the files in your current directory, including hidden files
ls -a

In the Linux file system, files named with a leading dot are hidden by default. Example: .bashrc

Moving, copying, and renaming files

Files can be moved or renamed with the mv command.

# Renames the file from hello.sh to hello-world.sh
mv hello.sh hello-world.sh
# Make a new directory "example"
mkdir example
# Move hello-world.sh into example
mv hello-world.sh example/
# Change directory into example
cd example
# List the files
ls
# Change directory back to the parent directory
cd ..

Similarly, you can copy files with the cp command. This leaves the original file in place.

# Copies the file hello.sh to example/
cp hello-world.sh example/

Transferring files

While logged in to CIRC systems, it isn’t possible to directly browse or use files on your local computer. In order to work with a file on a CIRC cluster or server, first transfer it to the system. There are several command-line and graphical user interface (GUI) options:

Command line options

The two main Linux command line options are scp and rsync. Both transfer files via SSH connection.

The below commands should be run from your desktop or laptop computer, not from the CIRC server.

`scp`

scp stands for "secure copy" and copies files from one computer to another through SSH. The command structure is

scp [OPTIONS] <[user@]host1:]file>  <[user@]host1:]file2>

For example, if user with NetID abc123123 wants to transfer a file example.txt from their desktop terminal to their Ganymede scratch directory, the command is

scp example.txt abc123123@ganymede.utdallas.edu:~/scratch/

Similarly, if user abc123123 wants to transfer the file from Ganymede to their computer, the command is

scp abc123123@ganymede.utdallas.edu:~/scratch/example.txt .

The . (period) symbol means "the current directory." You can also replace . with a path in the preceding command.

For more information on scp, see the scp documentation.

`rsync`

rsync is similar to scp in practice, but does not support moving files between two remote servers (you can only push or pull from a local computer). The command structure is

rsync [OPTIONS] <host> <destination>

For example, if user with NetID abc123123 wants to transfer a file example.txt from their desktop terminal to their Ganymede scratch directory, the command is

rsync -av example.txt abc123123@ganymede.utdallas.edu:~/scratch/

Similarly, if user abc123123 wants to transfer the file from Ganymede to their computer, the command is

rsync -av abc123123@ganymede.utdallas.edu:~/scratch/example.txt .

For more information on rsync, see the rsync documentation.

Graphical options

There are many options for GUI file transfers. One free and open source option is FileZilla. FileZilla is available for Linux, MacOS, and Windows computers.

To use, download and install the appropriate FileZilla version for your operating system. Then, add a connection to the CIRC system, (for example, Ganymede), by clicking on File > Site Manager and then selecting "new site."

You must go through site manager. The quick connect bar is not sufficient.

In the "protocol" dropdown, select SFTP - SSH File Transfer Protocol.

Type the CIRC host into the "host" field and enter your UT Dallas NetID credentials in the "user" and "password" fields, then click "connect."

For Europa, change Logon Type to 'Key file' and select your SSH key.

After connecting, you see your local files on the left side of the program and your remote server files on the right side. You can now drag and drop from your local computer to the remote server.