Remote and Network Computing

Users have been accessing sites remotely since the earliest days of what we now call the Internet.

telnet and ftp (and their secure successors)

These two commands and their secure successors have been for decades the two fundamental commands of the Internet / remote computing.

telnet (host) - Login to remote host
ftp (host) - Transfer files to/from remote host

With telnet you get a command-line, with ftp you get a read-write file system.

Origin:

telnet, 1971 (and here).
ftp, 1971.

These two commands have been replaced by secure versions:

telnet -> ssh
ftp -> sftp / scp (both use ssh) or ftps (uses ssl)

In the past: ftp of public data
- Read-only ftp (File Transfer Protocol) was what people used to publish files and archives online before the Web (http - Hypertext Transfer Protocol).
- This is rarely used any more. Though most browsers should be able to read public files through it.
Now the main use: ftp of private data
- (Secure) ftp is now far more often used in read-write mode for remote private data (needs password).
- e.g. Your web hosting company uses a Unix server. You periodically upload your web site (edited in Windows) onto it with a private ftp.
- There are many graphical drag-and-drop ftp clients, and even programs that make the site into a full Windows drive.

How to login to Linux at DCU

Remote Unix/Linux files

A common situation in IT is a Windows client, working with remote Unix/Linux files.
A nice way to do this is to make the remote files a drive (like the X: drive) on Windows.
Here is one way I like to work:

Windows client. Windows GUI. Windows editors.
Remote Linux files as a drive on Windows.
This requires software to turn an sftp site into a drive.
I use Windows editors to edit remote Linux files.
I have ssh always open to run scripts to process those files.

How to turn an ftp site into a drive

Remote Unix/Linux GUI

Run Unix GUI programs, visible on remote Unix machine:

X-Windows

Run Unix GUI programs, visible on remote Windows machine:

Xming
Or Cygwin/X or X-Win32

FTP scripting

FTP (sftp / scp / ftps) scripting is a good/essential tool for website maintenance.

Say you are working on an offline copy of a website of 10,000 files. You make changes to 137 files among the 10,000. You want to upload the 137 new files, but not the entire 10,000. Drag and drop the 137 to their correct destinations could be quite tedious.

For repetitive tasks, drag-and-drop is not a better interface than being able to write automated scripts (this will be a theme of this course). You can write ftp scripts ("macros") and call them from Shell scripts:

RFC 959
- low-level ftp commands
Unix client-level ftp commands (and here)
ftp macros (and here and example)
- Basically, you can put logins and command lists in $home/.netrc
- Any macro called init will be run automatically.
- You can include passwords - .netrc needs to be hidden from group and others.

Sample ftp scripting

This is how I work:
To make a new build of my (huge) genealogy site:
I just want to upload the small number of files that have changed since the last build.
I run a program to build an ftp script with all the changes, like the ftp script below.
I then run sftp with this ftp script.


cd public_html
lcd /users/local/humphrys/local/history-copy
mkdir ./Flanagan/NMI.bird
put ./Flanagan/NMI.bird/IMAG1024.jpg ./Flanagan/NMI.bird/IMAG1024.jpg
put ./Flanagan/NMI.bird/IMAG1025.jpg ./Flanagan/NMI.bird/IMAG1025.jpg
put ./Flanagan/the.bird.html ./Flanagan/the.bird.html
put blog.html blog.html

Changes directory on remote site.
Changes directory on local site.
Makes remote directories.
Puts various files into various destinations.

The above ftp script is automatically built by a program.

See also tools like rsync

HTTP scripting

You can also do HTTP GET or POST scripting from the command-line.
Some tools that do this:

wget
- gnu.org
- manual
- does HTTP GET:
```
  wget -q -O - URL
```
- See wget examples
Advanced wget:
- To crawl an entire site or check for broken links, something like:
```
  wget --spider --force-html -i file.html
```
- Sites that block programs:
  If a site won't let scripts/programs see its content (only browsers), you can set User agent to pretend to be a browser:
```
  UserAgent="Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
  wget -q -O - 	-U "$UserAgent"	URL
```

lynx

does HTTP GET:
```
  lynx -reload -source URL
```

does HTTP POST:

  cat DATA | lynx -reload -source -post_data URL

curl
- dumps to command line by default
- -s for silent mode
- does HTTP GET
- does HTTP POST

Working remotely

Idea: Your files are "on the network" ("in the cloud") somewhere. You can access them and make changes to them from anywhere. All copies stay in synch.

This is what you actually have within DCU (can move from terminal to terminal, accessing files at central server).

Various ways of working:

1 copy of files. Copy is on server. Read files from server when we need them, and copy any changed files back to server over the network, as you go along.
2 copies of files. Work on local machine which has synchronised mirror of server files. Have to keep copies fully in synch over the network.
Physically bring hardware to and from work (laptop or USB key or external hard disk).

Cloud computing is the logical extension of these ways of working.