The Linux System Administrator's Guide Version 0.8 Lars Wirzenius Joanna Oja Stephen Stafford Alex Weeks 2003-12-03 An introduction to system administration of a Linux system for novices. Copyright 1993--1998 Lars Wirzenius. Copyright 1998--2001 Joanna Oja. Copyright 2001--2003 Stephen Stafford. Copyright 2003--Present Stephen Stafford & Alex Weeks. Trademarks are owned by their owners. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.1; with no Invariant Sections, with no Front-Cover Texts, and with no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License". ----------------------------------------------------------------------------- Table of Contents Source and pre-formatted versions available 1. Introduction 2. About This Book 2.1. Acknowledgments 2.2. Typographical Conventions 3. Overview of a Linux System 3.1. Various parts of an operating system 3.2. Important parts of the kernel 3.3. Major services in a UNIX system 4. Overview of the Directory Tree 4.1. Background 4.2. The root filesystem 4.3. The /etc directory 4.4. The /dev directory 4.5. The /usr filesystem 4.6. The /var filesystem 4.7. The /proc filesystem 5. Device Files 5.1. The MAKEDEV Script 5.2. The mknod command 5.3. Device List 6. Using Disks and Other Storage Media 6.1. Two kinds of devices 6.2. Hard disks 6.3. Floppies 6.4. CD-ROMs 6.5. Tapes 6.6. Formatting 6.7. Partitions 6.8. Filesystems 6.9. Disks without filesystems 6.10. Allocating disk space 7. Memory Management 7.1. What is virtual memory? 7.2. Creating a swap space 7.3. Using a swap space 7.4. Sharing swap spaces with other operating systems 7.5. Allocating swap space 7.6. The buffer cache 8. Boots And Shutdowns 8.1. An overview of boots and shutdowns 8.2. The boot process in closer look 8.3. More about shutdowns 8.4. Rebooting 8.5. Single user mode 8.6. Emergency boot floppies 9. init 9.1. init comes first 9.2. Configuring init to start getty: the /etc/inittab file 9.3. Run levels 9.4. Special configuration in /etc/inittab 9.5. Booting in single user mode 10. Logging In And Out 10.1. Logins via terminals 10.2. Logins via the network 10.3. What login does 10.4. X and xdm 10.5. Access control 10.6. Shell startup 11. Managing user accounts 11.1. What's an account? 11.2. Creating a user 11.3. Changing user properties 11.4. Removing a user 11.5. Disabling a user temporarily 12. Backups 12.1. On the importance of being backed up 12.2. Selecting the backup medium 12.3. Selecting the backup tool 12.4. Simple backups 12.5. Multilevel backups 12.6. What to back up 12.7. Compressed backups 13. Keeping Time 13.1. The concept of localtime 13.2. The hardware and software clocks 13.3. Showing and setting time 13.4. When the clock is wrong 13.5. NTP - Network Time Protocol 13.6. Basic NTP configuration 13.7. NTP Toolkit 13.8. Some known NTP servers 13.9. NTP Links 14. Finding Help 14.1. Newsgroups and Mailing Lists 14.2. IRC A. GNU Free Documentation License 0. PREAMBLE 1. APPLICABILITY AND DEFINITIONS 2. VERBATIM COPYING 3. COPYING IN QUANTITY 4. MODIFICATIONS 5. COMBINING DOCUMENTS 6. COLLECTIONS OF DOCUMENTS 7. AGGREGATION WITH INDEPENDENT WORKS 8. TRANSLATION 9. TERMINATION 10. FUTURE REVISIONS OF THIS LICENSE How to use this License for your documents Glossary (DRAFT, but not for long hopefully) List of Tables 6-1. Partition types (from the Linux fdisk program). 9-1. Run level numbers 12-1. Efficient backup scheme using many backup levels List of Figures 3-1. Some of the more important parts of the Linux kernel 4-1. Parts of a Unix directory tree. Dashed lines indicate partition limits. 6-1. A schematic picture of a hard disk. 6-2. A sample hard disk partitioning. 6-3. Three separate filesystems. 6-4. /home and /usr have been mounted. 6-5. Sample output from dumpe2fs 10-1. Logins via terminals: the interaction of init, getty, login, and the shell. 12-1. A sample multilevel backup schedule. ----------------------------------------------------------------------------- Source and pre-formatted versions available The source code and other machine readable formats of this book can be found on the Internet via anonymous FTP at the Linux Documentation Project home page [http://www.tldp.org/] http://www.tldp.org/, or at the home page of this book at [http://www.taylexson.org/sag//] http://www.taylexson.org/sag/. Available are at least HTML and PDF formats. ----------------------------------------------------------------------------- Chapter 1. Introduction "In the beginning, the file was without form, and void; and emptiness was upon the face of the bits. And the Fingers of the Author moved upon the face of the keyboard. And the Author said, Let there be words, and there were words." The Linux System Administrator's Guide, describes the system administration aspects of using Linux. It is intended for people who know next to nothing about system administration (those saying "what is it?"), but who have already mastered at least the basics of normal usage. This manual doesn't tell you how to install Linux; that is described in the Installation and Getting Started document. See below for more information about Linux manuals. System administration covers all the things that you have to do to keep a computer system in usable order. It includes things like backing up files (and restoring them if necessary), installing new programs, creating accounts for users (and deleting them when no longer needed), making certain that the filesystem is not corrupted, and so on. If a computer were, say, a house, system administration would be called maintenance, and would include cleaning, fixing broken windows, and other such things. The structure of this manual is such that many of the chapters should be usable independently, so if you need information about backups, for example, you can read just that chapter. However, this manual is first and foremost a tutorial and can be read sequentially or as a whole. This manual is not intended to be used completely independently. Plenty of the rest of the Linux documentation is also important for system administrators. After all, a system administrator is just a user with special privileges and duties. Very useful resources are the manual pages, which should always be consulted when you are not familiar with a command. If you do not know which command you need, then the apropos command can be used. Consult its manual page for more details. While this manual is targeted at Linux, a general principle has been that it should be useful with other UNIX based operating systems as well. Unfortunately, since there is so much variance between different versions of UNIX in general, and in system administration in particular, there is little hope to cover all variants. Even covering all possibilities for Linux is difficult, due to the nature of its development. There is no one official Linux distribution, so different people have different setups and many people have a setup they have built up themselves. This book is not targeted at any one distribution. Distributions can and do vary considerably. When possible, differences have been noted and alternatives given. In trying to describe how things work, rather than just listing "five easy steps" for each task, there is much information here that is not necessary for everyone, but those parts are marked as such and can be skipped if you use a preconfigured system. Reading everything will, naturally, increase your understanding of the system and should make using and administering it more productive. [1] Like all other Linux related development, the work to write this manual was done on a volunteer basis: I did it because I thought it might be fun and because I felt it should be done. However, like all volunteer work, there is a limit to how much time, knowledge and experience people have. This means that the manual is not necessarily as good as it would be if a wizard had been paid handsomely to write it and had spent millennia to perfect it. Be warned. One particular point where corners have been cut is that many things that are already well documented in other freely available manuals are not always covered here. This applies especially to program specific documentation, such as all the details of using mkfs. Only the purpose of the program and as much of its usage as is necessary for the purposes of this manual is described. For further information, consult these other manuals. Usually, all of the referred to documentation is part of the full Linux documentation set. ----------------------------------------------------------------------------- Chapter 2. About This Book 2.1. Acknowledgments 2.1.1. Joanna's acknowledgments Lars has tried to make this manual as good as possible and I would like, as a current maintainer, to keep up the good work. I would really like to hear from you if you have any ideas on how to make it better. Bad language, factual errors, ideas for new areas to cover, rewritten sections, information about how various UNIX versions do things, I am interested in all of it. My contact information is available via the World Wide Web at [http://www.iki.fi /viu/] http://www.iki.fi/viu/. Many people have helped me with this book, directly or indirectly. I would like to especially thank Matt Welsh for inspiration and LDP leadership, Andy Oram for getting me to work again with much-valued feedback, Olaf Kirch for showing me that it can be done, and Adam Richter at Yggdrasil and others for showing me that other people can find it interesting as well. Stephen Tweedie, H. Peter Anvin, Remy Card, Theodore Ts'o, and Stephen Tweedie have let me borrow their work (and thus make the book look thicker and much more impressive): a comparison between the xia and ext2 filesystems, the device list and a description of the ext2 filesystem. These aren't part of the book any more. I am most grateful for this, and very apologetic for the earlier versions that sometimes lacked proper attribution. In addition, I would like to thank Mark Komarinski for sending his material in 1993 and the many system administration columns in Linux Journal. They are quite informative and inspirational. Many useful comments have been sent by a large number of people. My miniature black hole of an archive doesn't let me find all their names, but some of them are, in alphabetical order: Paul Caprioli, Ales Cepek, Marie-France Declerfayt, Dave Dobson, Olaf Flebbe, Helmut Geyer, Larry Greenfield and his father, Stephen Harris, Jyrki Havia, Jim Haynes, York Lam, Timothy Andrew Lister, Jim Lynch, Michael J. Micek, Jacob Navia, Dan Poirier, Daniel Quinlan, Jouni K Seppänen, Philippe Steindl, G.B. Stotte. My apologies to anyone I have forgotten. ----------------------------------------------------------------------------- 2.1.2. Stephen's acknowledgments I would like to thank Lars and Joanna for their hard work on the guide. In a guide like this one there are likely to be at least some minor inaccuracies. And there are almost certainly going to be sections that become out of date from time to time. If you notice any of this then please let me know by sending me an email to: . I will take virtually any form of input (diffs, just plain text, html, whatever), I am in no way above allowing others to help me maintain such a large text as this :) Many thanks to Helen Topping Shaw for getting the red pen out and making the text far better than it would otherwise have been. Also thanks are due just for being wonderful. The current web home of the guide is http://people.debian.org/~bagpuss ----------------------------------------------------------------------------- 2.1.3. Alex's Acknowledgments I would like to thank Lars, Joanna, and Stephen for all the great work that they have done on this document over the years. I only hope that my contribution will be worthy of continuing the work they started. There have been many people who have helped me on my journey through the "Windows-Free" world, the person I feel I need to thank the most is my first true UN*X mentor, Mike Velasco. Back in a time before SCO became a "dirty word", Mike helped me on the path of tar's, cpio's, and many, many man pages. Thanks Mike! You are the 'Sofa King'. ----------------------------------------------------------------------------- 2.2. Typographical Conventions Throughout this book, I have tried to use uniform typographical conventions. Hopefully they aid readability. If you can suggest any improvements please contact me. Filenames are expressed as: /usr/share/doc/foo. Command names are expressed as: fsck Email addresses are expressed as: URLs are expressed as: [http://www.tldp.org] http://www.tldp.org I will add to this section as things come up whilst editing. If you notice anything that should be added then please let me know. ----------------------------------------------------------------------------- Chapter 3. Overview of a Linux System "God saw everything that he had made, and saw that it was very good. " -- Bible King James Version. Genesis 1:31 This chapter gives an overview of a Linux system. First, the major services provided by the operating system are described. Then, the programs that implement these services are described with a considerable lack of detail. The purpose of this chapter is to give an understanding of the system as a whole, so that each part is described in detail elsewhere. ----------------------------------------------------------------------------- 3.1. Various parts of an operating system A UNIX operating system consists of a kernel and some system programs. There are also some application programs for doing work. The kernel is the heart of the operating system. [2] It keeps track of files on the disk, starts programs and runs them concurrently, assigns memory and other resources to various processes, receives packets from and sends packets to the network, and so on. The kernel does very little by itself, but it provides tools with which all services can be built. It also prevents anyone from accessing the hardware directly, forcing everyone to use the tools it provides. [3] This way the kernel provides some protection for users from each other. The tools provided by the kernel are used via system calls. See manual page section 2 for more information on these. The system programs use the tools provided by the kernel to implement the various services required from an operating system. System programs, and all other programs, run `on top of the kernel', in what is called the user mode. The difference between system and application programs is one of intent: applications are intended for getting useful things done (or for playing, if it happens to be a game), whereas system programs are needed to get the system working. A word processor is an application; mount is a system program. The difference is often somewhat blurry, however, and is important only to compulsive categorizers. An operating system can also contain compilers and their corresponding libraries (GCC and the C library in particular under Linux), although not all programming languages need be part of the operating system. Documentation, and sometimes even games, can also be part of it. Traditionally, the operating system has been defined by the contents of the installation tape or disks; with Linux it is not as clear since it is spread all over the FTP sites of the world. ----------------------------------------------------------------------------- 3.2. Important parts of the kernel The Linux kernel consists of several important parts: process management, memory management, hardware device drivers, filesystem drivers, network management, and various other bits and pieces. Figure 3-1 shows some of them. Figure 3-1. Some of the more important parts of the Linux kernel [overview-kernel] Probably the most important parts of the kernel (nothing else works without them) are memory management and process management. Memory management takes care of assigning memory areas and swap space areas to processes, parts of the kernel, and for the buffer cache. Process management creates processes, and implements multitasking by switching the active process on the processor. At the lowest level, the kernel contains a hardware device driver for each kind of hardware it supports. Since the world is full of different kinds of hardware, the number of hardware device drivers is large. There are often many otherwise similar pieces of hardware that differ in how they are controlled by software. The similarities make it possible to have general classes of drivers that support similar operations; each member of the class has the same interface to the rest of the kernel but differs in what it needs to do to implement them. For example, all disk drivers look alike to the rest of the kernel, i.e., they all have operations like `initialize the drive', `read sector N', and `write sector N'. Some software services provided by the kernel itself have similar properties, and can therefore be abstracted into classes. For example, the various network protocols have been abstracted into one programming interface, the BSD socket library. Another example is the virtual filesystem (VFS) layer that abstracts the filesystem operations away from their implementation. Each filesystem type provides an implementation of each filesystem operation. When some entity tries to use a filesystem, the request goes via the VFS, which routes the request to the proper filesystem driver. ----------------------------------------------------------------------------- 3.3. Major services in a UNIX system This section describes some of the more important UNIX services, but without much detail. They are described more thoroughly in later chapters. ----------------------------------------------------------------------------- 3.3.1. init The single most important service in a UNIX system is provided by init. init is started as the first process of every UNIX system, as the last thing the kernel does when it boots. When init starts, it continues the boot process by doing various startup chores (checking and mounting filesystems, starting daemons, etc). The exact list of things that init does depends on which flavor it is; there are several to choose from. init usually provides the concept of single user mode, in which no one can log in and root uses a shell at the console; the usual mode is called multiuser mode. Some flavors generalize this as run levels; single and multiuser modes are considered to be two run levels, and there can be additional ones as well, for example, to run X on the console. Linux allows for up to 10 runlevels, 0-9, but usually only some of these are defined by default. Runlevel 0 is defined as "system halt". Runlevel 1 is defined as "single user mode". Runlevel 6 is defined as "system reboot". Other runlevels are dependent on how your particular distribution has defined them, and they vary significantly between distributions. Looking at the contents of /etc/inittab usually will give some hint what the predefined runlevels are and what they have been defined as. In normal operation, init makes sure getty is working (to allow users to log in), and to adopt orphan processes (processes whose parent has died; in UNIX all processes must be in a single tree, so orphans must be adopted). When the system is shut down, it is init that is in charge of killing all other processes, unmounting all filesystems and stopping the processor, along with anything else it has been configured to do. ----------------------------------------------------------------------------- 3.3.2. Logins from terminals Logins from terminals (via serial lines) and the console (when not running X) are provided by the getty program. init starts a separate instance of getty for each terminal upon which logins are to be allowed. getty reads the username and runs the login program, which reads the password. If the username and password are correct, login runs the shell. When the shell terminates, i.e., the user logs out, or when login terminated because the username and password didn't match, init notices this and starts a new instance of getty. The kernel has no notion of logins, this is all handled by the system programs. ----------------------------------------------------------------------------- 3.3.3. Syslog The kernel and many system programs produce error, warning, and other messages. It is often important that these messages can be viewed later, even much later, so they should be written to a file. The program doing this is syslog. It can be configured to sort the messages to different files according to writer or degree of importance. For example, kernel messages are often directed to a separate file from the others, since kernel messages are often more important and need to be read regularly to spot problems. ----------------------------------------------------------------------------- 3.3.4. Periodic command execution: cron and at Both users and system administrators often need to run commands periodically. For example, the system administrator might want to run a command to clean the directories with temporary files (/tmp and /var/tmp) from old files, to keep the disks from filling up, since not all programs clean up after themselves correctly. The cron service is set up to do this. Each user can have a crontab file, where she lists the commands she wishes to execute and the times they should be executed. The cron daemon takes care of starting the commands when specified. The at service is similar to cron, but it is once only: the command is executed at the given time, but it is not repeated. See the manual pages cron(1), crontab(1), crontab(5), at(1) and atd(8) for more in depth information. ----------------------------------------------------------------------------- 3.3.5. Graphical user interface UNIX and Linux don't incorporate the user interface into the kernel; instead, they let it be implemented by user level programs. This applies for both text mode and graphical environments. This arrangement makes the system more flexible, but has the disadvantage that it is simple to implement a different user interface for each program, making the system harder to learn. The graphical environment primarily used with Linux is called the X Window System (X for short). X also does not implement a user interface; it only implements a window system, i.e., tools with which a graphical user interface can be implemented. Some popular window managers are: fvwm, icewm, blackbox and windowmaker. There are also two popular desktop managers, KDE and Gnome. ----------------------------------------------------------------------------- 3.3.6. Networking Networking is the act of connecting two or more computers so that they can communicate with each other. The actual methods of connecting and communicating are slightly complicated, but the end result is very useful. UNIX operating systems have many networking features. Most basic services (filesystems, printing, backups, etc) can be done over the network. This can make system administration easier, since it allows centralized administration, while still reaping in the benefits of microcomputing and distributed computing, such as lower costs and better fault tolerance. However, this book merely glances at networking; see the Linux Network Administrators' Guide [http://www.tldp.org/LDP/nag2/index.html] http:// www.tldp.org/LDP/nag2/index.html for more information, including a basic description of how networks operate. ----------------------------------------------------------------------------- 3.3.7. Network logins Network logins work a little differently than normal logins. There is a separate physical serial line for each terminal via which it is possible to log in. For each person logging in via the network, there is a separate virtual network connection, and there can be any number of these. [4] It is therefore not possible to run a separate getty for each possible virtual connection. There are also several different ways to log in via a network, telnet and rlogin being the major ones in TCP/IP networks. [5] Network logins have, instead of a herd of gettys, a single daemon per way of logging in (telnet and rlogin have separate daemons) that listens for all incoming login attempts. When it notices one, it starts a new instance of itself to handle that single attempt; the original instance continues to listen for other attempts. The new instance works similarly to getty. ----------------------------------------------------------------------------- 3.3.8. Network file systems One of the more useful things that can be done with networking services is sharing files via a network file system. The one usually used is called the Network File System, or NFS, developed by Sun. With a network file system any file operations done by a program on one machine are sent over the network to another computer. This fools the program to think that all the files on the other computer are actually on the computer the program is running on. This makes information sharing extremely simple, since it requires no modifications to programs. Another popular way of sharing files is Samba [http://www.samba.org] http:// www.samba.org. This protocol allows the sharing of files with MS Windows machines (via Network Neighbourhood). It also allows the sharing of printers across machines. ----------------------------------------------------------------------------- 3.3.9. Mail Electronic mail is the most popularly used method for communicating via computer. An electronic letter is stored in a file using a special format, and special mail programs are used to send and read the letters. Each user has an incoming mailbox (a file in the special format), where all new mail is stored. When someone sends mail, the mail program locates the receiver's mailbox and appends the letter to the mailbox file. If the receiver's mailbox is in another machine, the letter is sent to the other machine, which delivers it to the mailbox as it best sees fit. The mail system consists of many programs. The delivery of mail to local or remote mailboxes is done by one program (the mail transfer agent (MTA), e.g., sendmail or smail), while the programs users use are many and varied (mail user agent (MUA), e.g., pine, mutt or elm). The mailboxes are usually stored in /var/spool/mail. ----------------------------------------------------------------------------- 3.3.10. Printing Only one person can use a printer at one time, but it is uneconomical not to share printers between users. The printer is therefore managed by software that implements a print queue: all print jobs are put into a queue and whenever the printer is done with one job, the next one is sent to it automatically. This relieves the users from organizing the print queue and fighting over control of the printer. [6] The print queue software also spools the printouts on disk, i.e., the text is kept in a file while the job is in the queue. This allows an application program to spit out the print jobs quickly to the print queue software; the application does not have to wait until the job is actually printed to continue. This is really convenient, since it allows one to print out one version, and not have to wait for it to be printed before one can make a completely revised new version. ----------------------------------------------------------------------------- 3.3.11. The filesystem layout The filesystem is divided into many parts; usually along the lines of a root filesystem with /bin, /lib, /etc, /dev, and a few others; a /usr filesystem with programs and unchanging data; a /var filesystem with changing data (such as log files); and a /home filesystem for everyone's personal files. Depending on the hardware configuration and the decisions of the system administrator, the division can be different; it can even be all in one filesystem. Chapter 4 describes the filesystem layout in some little detail; the Filesystem Hierarchy Standard covers it in somewhat more detail. [7] ----------------------------------------------------------------------------- Chapter 4. Overview of the Directory Tree " Two days later, there was Pooh, sitting on his branch, dangling his legs, and there, beside him, were four pots of honey..." (A.A. Milne) This chapter describes the important parts of a standard Linux directory tree, based on the Filesystem Hierarchy Standard. It outlines the normal way of breaking the directory tree into separate filesystems with different purposes and gives the motivation behind this particular split. Not all Linux distributions follow this standard slavishly, but it is generic enough to give you an overview. ----------------------------------------------------------------------------- 4.1. Background This chapter is loosely based on the Filesystems Hierarchy Standard (FHS) [8] version 2.1, which attempts to set a standard for how the directory tree in a Linux [9] system is organized. Such a standard has the advantage that it will be easier to write or port software for Linux, and to administer Linux machines, since everything should be in standardized places. There is no authority behind the standard that forces anyone to comply with it, but it has gained the support of many Linux distributions. It is not a good idea to break with the FHS without very compelling reasons. The FHS attempts to follow Unix tradition and current trends, making Linux systems familiar to those with experience with other Unix systems, and vice versa. This chapter is not as detailed as the FHS. A system administrator should also read the full FHS for a complete understanding. This chapter does not explain all files in detail. The intention is not to describe every file, but to give an overview of the system from a filesystem point of view. Further information on each file is available elsewhere in this manual or in the Linux manual pages. The full directory tree is intended to be breakable into smaller parts, each capable of being on its own disk or partition, to accommodate to disk size limits and to ease backup and other system administration tasks. The major parts are the root (/), /usr, /var, and /home filesystems (see Figure 4-1). Each part has a different purpose. The directory tree has been designed so that it works well in a network of Linux machines which may share some parts of the filesystems over a read-only device (e.g., a CD-ROM), or over the network with NFS. Figure 4-1. Parts of a Unix directory tree. Dashed lines indicate partition limits. [fstree] The roles of the different parts of the directory tree are described below.   * The root filesystem is specific for each machine (it is generally stored on a local disk, although it could be a ramdisk or network drive as well) and contains the files that are necessary for booting the system up, and to bring it up to such a state that the other filesystems may be mounted. The contents of the root filesystem will therefore be sufficient for the single user state. It will also contain tools for fixing a broken system, and for recovering lost files from backups.   * The /usr filesystem contains all commands, libraries, manual pages, and other unchanging files needed during normal operation. No files in /usr should be specific for any given machine, nor should they be modified during normal use. This allows the files to be shared over the network, which can be cost-effective since it saves disk space (there can easily be hundreds of megabytes, increasingly multiple gigabytes in /usr). It can make administration easier (only the master /usr needs to be changed when updating an application, not each machine separately) to have /usr network mounted. Even if the filesystem is on a local disk, it could be mounted read-only, to lessen the chance of filesystem corruption during a crash.   * The /var filesystem contains files that change, such as spool directories (for mail, news, printers, etc), log files, formatted manual pages, and temporary files. Traditionally everything in /var has been somewhere below /usr, but that made it impossible to mount /usr read-only.   * The /home filesystem contains the users' home directories, i.e., all the real data on the system. Separating home directories to their own directory tree or filesystem makes backups easier; the other parts often do not have to be backed up, or at least not as often as they seldom change. A big /home might have to be broken across several filesystems, which requires adding an extra naming level below /home, for example / home/students and /home/staff. Although the different parts have been called filesystems above, there is no requirement that they actually be on separate filesystems. They could easily be kept in a single one if the system is a small single-user system and the user wants to keep things simple. The directory tree might also be divided into filesystems differently, depending on how large the disks are, and how space is allocated for various purposes. The important part, though, is that all the standard names work; even if, say, /var and /usr are actually on the same partition, the names /usr/lib/libc.a and /var/log/messages must work, for example by moving files below /var into /usr/var, and making /var a symlink to /usr/var. The Unix filesystem structure groups files according to purpose, i.e., all commands are in one place, all data files in another, documentation in a third, and so on. An alternative would be to group files files according to the program they belong to, i.e., all Emacs files would be in one directory, all TeX in another, and so on. The problem with the latter approach is that it makes it difficult to share files (the program directory often contains both static and sharable and changing and non-sharable files), and sometimes to even find the files (e.g., manual pages in a huge number of places, and making the manual page programs find all of them is a maintenance nightmare). ----------------------------------------------------------------------------- 4.2. The root filesystem The root filesystem should generally be small, since it contains very critical files and a small, infrequently modified filesystem has a better chance of not getting corrupted. A corrupted root filesystem will generally mean that the system becomes unbootable except with special measures (e.g., from a floppy), so you don't want to risk it. The root directory generally doesn't contain any files, except perhaps the standard boot image for the system, usually called /vmlinuz. All other files are in subdirectories in the root filesystems: /bin Commands needed during bootup that might be used by normal users (probably after bootup). /sbin Like /bin, but the commands are not intended for normal users, although they may use them if necessary and allowed. /sbin is not usually in the default path of normal users, but will be in root's default path. /etc Configuration files specific to the machine. /root The home directory for user root. This is usually not accessible to other users on the system /lib Shared libraries needed by the programs on the root filesystem. /lib/modules Loadable kernel modules, especially those that are needed to boot the system when recovering from disasters (e.g., network and filesystem drivers). /dev Device files. Some of the more commonly used device files are examined in Chapter 5 /tmp Temporary files. Programs running after bootup should use /var/tmp, not / tmp, since the former is probably on a disk with more space. Often /tmp will be a symbolic link to /var/tmp. /boot Files used by the bootstrap loader, e.g., LILO. Kernel images are often kept here instead of in the root directory. If there are many kernel images, the directory can easily grow rather big, and it might be better to keep it in a separate filesystem. Another reason would be to make sure the kernel images are within the first 1024 cylinders of an IDE disk. [10] /mnt Mount point for temporary mounts by the system administrator. Programs aren't supposed to mount on /mnt automatically. /mnt might be divided into subdirectories (e.g., /mnt/dosa might be the floppy drive using an MS-DOS filesystem, and /mnt/exta might be the same with an ext2 filesystem). /proc, /usr, /var, /home Mount points for the other filesystems. [11] ----------------------------------------------------------------------------- 4.3. The /etc directory The /etc directory contains a lot of files. Some of them are described below. For others, you should determine which program they belong to and read the manual page for that program. Many networking configuration files are in /etc as well, and are described in the Networking Administrators' Guide. /etc/rc or /etc/rc.d or /etc/rc?.d Scripts or directories of scripts to run at startup or when changing the run level. See Chapter 9 for further information. /etc/passwd The user database, with fields giving the username, real name, home directory, encrypted password, and other information about each user. The format is documented in the passwd manual page. The encrypted passwords are much more commonly found in the /etc/shadow these days. This means that almost everything about the user except the password is stored in the passwd file. History and convention make a name change undesirable. /etc/fdprm Floppy disk parameter table. Describes what different floppy disk formats look like. Used by setfdprm. See the setfdprm manual page for more information. /etc/fstab Lists the filesystems mounted automatically at startup by the mount -a command (in /etc/rc or equivalent startup file). Under Linux, also contains information about swap areas used automatically by swapon -a. See Section 6.8.5 and the mount manual page for more information. Also fstab usually has its own manual page in section 5. /etc/group Similar to /etc/passwd, but describes groups instead of users. See the group manual page in section 5 for more information. /etc/inittab Configuration file for init. /etc/issue Output by getty before the login prompt. Usually contains a short description or welcoming message to the system. The contents are up to the system administrator. /etc/magic The configuration file for file. Contains the descriptions of various file formats based on which file guesses the type of the file. See the magic and file manual pages for more information. /etc/motd The message of the day, automatically output after a successful login. Contents are up to the system administrator. Often used for getting information to every user, such as warnings about planned downtimes. /etc/mtab List of currently mounted filesystems. Initially set up by the bootup scripts, and updated automatically by the mount command. Used when a list of mounted filesystems is needed, e.g., by the df command. /etc/shadow Shadow password file on systems with shadow password software installed. Shadow passwords move the encrypted password from /etc/passwd into /etc/ shadow; the latter is not readable by anyone except root. This makes it harder to crack passwords. If your distribution gives you a choice (many do) of whether or not to use shadow passwords then you are highly recommended to do so. /etc/login.defs Configuration file for the login command. The login.defs file usually has a manual page in section 5. /etc/printcap Like /etc/termcap, but intended for printers. However it uses different syntax. The printcap has a manual page in section 5. /etc/profile, /etc/csh.login, /etc/csh.cshrc Files executed at login or startup time by the Bourne or C shells. These allow the system administrator to set global defaults for all users. See the manual pages for the respective shells. /etc/securetty Identifies secure terminals, i.e., the terminals from which root is allowed to log in. Typically only the virtual consoles are listed, so that it becomes impossible (or at least harder) to gain superuser privileges by breaking into a system over a modem or a network. Do not allow root logins over a network. Prefer to log in as an unprivileged user and use su or sudo to gain root privileges. /etc/shells Lists trusted shells. The chsh command allows users to change their login shell only to shells listed in this file. ftpd, the server process that provides FTP services for a machine, will check that the user's shell is listed in /etc/shells and will not let people log in unless the shell is listed there. /etc/termcap The terminal capability database. Describes by what "escape sequences" various terminals can be controlled. Programs are written so that instead of directly outputting an escape sequence that only works on a particular brand of terminal, they look up the correct sequence to do whatever it is they want to do in /etc/termcap. As a result most programs work with most kinds of terminals. See the termcap, curs_termcap, and terminfo manual pages for more information. ----------------------------------------------------------------------------- 4.4. The /dev directory The /dev directory contains the special device files for all the devices. The device files are named using special conventions; these are described in Chapter 5. The device files are created during installation, and later with the /dev/MAKEDEV script. The /dev/MAKEDEV.local is a script written by the system administrator that creates local-only device files or links (i.e. those that are not part of the standard MAKEDEV, such as device files for some non-standard device driver). ----------------------------------------------------------------------------- 4.5. The /usr filesystem The /usr filesystem is often large, since all programs are installed there. All files in /usr usually come from a Linux distribution; locally installed programs and other stuff goes below /usr/local. This makes it possible to update the system from a new version of the distribution, or even a completely new distribution, without having to install all programs again. Some of the subdirectories of /usr are listed below (some of the less important directories have been dropped; see the FSSTND for more information). /usr/X11R6 The X Window System, all files. To simplify the development and installation of X, the X files have not been integrated into the rest of the system. There is a directory tree below /usr/X11R6 similar to that below /usr itself. /usr/bin Almost all user commands. Some commands are in /bin or in /usr/local/bin. /usr/sbin System administration commands that are not needed on the root filesystem, e.g., most server programs. /usr/share/man, /usr/share/info, /usr/share/doc Manual pages, GNU Info documents, and miscellaneous other documentation files, respectively. /usr/include Header files for the C programming language. This should actually be below /usr/lib for consistency, but the tradition is overwhelmingly in support for this name. /usr/lib Unchanging data files for programs and subsystems, including some site-wide configuration files. The name lib comes from library; originally libraries of programming subroutines were stored in /usr/lib. /usr/local The place for locally installed software and other files. Distributions may not install anything in here. It is reserved solely for the use of the local administrator. This way he can be absolutely certain that no updates or upgrades to his distribution will overwrite any extra software he has installed locally. ----------------------------------------------------------------------------- 4.6. The /var filesystem The /var contains data that is changed when the system is running normally. It is specific for each system, i.e., not shared over the network with other computers. /var/cache/man A cache for man pages that are formatted on demand. The source for manual pages is usually stored in /usr/share/man/man?/ (where ? is the manual section. See the manual page for man in section 7); some manual pages might come with a pre-formatted version, which might be stored in /usr/ share/man/cat*. Other manual pages need to be formatted when they are first viewed; the formatted version is then stored in /var/cache/man so that the next person to view the same page won't have to wait for it to be formatted. /var/games Any variable data belonging to games in /usr should be placed here. This is in case /usr is mounted read only. /var/lib Files that change while the system is running normally. /var/local Variable data for programs that are installed in /usr/local (i.e., programs that have been installed by the system administrator). Note that even locally installed programs should use the other /var directories if they are appropriate, e.g., /var/lock. /var/lock Lock files. Many programs follow a convention to create a lock file in / var/lock to indicate that they are using a particular device or file. Other programs will notice the lock file and won't attempt to use the device or file. /var/log Log files from various programs, especially login (/var/log/wtmp, which logs all logins and logouts into the system) and syslog (/var/log/ messages, where all kernel and system program message are usually stored). Files in /var/log can often grow indefinitely, and may require cleaning at regular intervals. /var/mail This is the FHS approved location for user mailbox files. Depending on how far your distribution has gone towards FHS compliance, these files may still be held in /var/spool/mail. /var/run Files that contain information about the system that is valid until the system is next booted. For example, /var/run/utmp contains information about people currently logged in. /var/spool Directories for news, printer queues, and other queued work. Each different spool has its own subdirectory below /var/spool, e.g., the news spool is in /var/spool/news. Note that some installations which are not fully compliant with the latest version of the FHS may have user mailboxes under /var/spool/mail. /var/tmp Temporary files that are large or that need to exist for a longer time than what is allowed for /tmp. (Although the system administrator might not allow very old files in /var/tmp either.) ----------------------------------------------------------------------------- 4.7. The /proc filesystem The /proc filesystem contains a illusionary filesystem. It does not exist on a disk. Instead, the kernel creates it in memory. It is used to provide information about the system (originally about processes, hence the name). Some of the more important files and directories are explained below. The / proc filesystem is described in more detail in the proc manual page. /proc/1 A directory with information about process number 1. Each process has a directory below /proc with the name being its process identification number. /proc/cpuinfo Information about the processor, such as its type, make, model, and performance. /proc/devices List of device drivers configured into the currently running kernel. /proc/dma Shows which DMA channels are being used at the moment. /proc/filesystems Filesystems configured into the kernel. /proc/interrupts Shows which interrupts are in use, and how many of each there have been. /proc/ioports Which I/O ports are in use at the moment. /proc/kcore An image of the physical memory of the system. This is exactly the same size as your physical memory, but does not really take up that much memory; it is generated on the fly as programs access it. (Remember: unless you copy it elsewhere, nothing under /proc takes up any disk space at all.) /proc/kmsg Messages output by the kernel. These are also routed to syslog. /proc/ksyms Symbol table for the kernel. /proc/loadavg The `load average' of the system; three meaningless indicators of how much work the system has to do at the moment. /proc/meminfo Information about memory usage, both physical and swap. /proc/modules Which kernel modules are loaded at the moment. /proc/net Status information about network protocols. /proc/self A symbolic link to the process directory of the program that is looking at /proc. When two processes look at /proc, they get different links. This is mainly a convenience to make it easier for programs to get at their process directory. /proc/stat Various statistics about the system, such as the number of page faults since the system was booted. /proc/uptime The time the system has been up. /proc/version The kernel version. Note that while the above files tend to be easily readable text files, they can sometimes be formatted in a way that is not easily digestible. There are many commands that do little more than read the above files and format them for easier understanding. For example, the free program reads /proc/meminfo and converts the amounts given in bytes to kilobytes (and adds a little more information, as well). ----------------------------------------------------------------------------- Chapter 5. Device Files This chapter gives an overview of what a device file is, and how to create one. It also lists some of the more common device files. The canonical list of device files is /usr/src/linux/Documentation/devices.txt if you have the Linux kernel source code installed on your system. The devices listed here are correct as of kernel version 2.2.17. ----------------------------------------------------------------------------- 5.1. The MAKEDEV Script Most device files will already be created and will be there ready to use after you install your Linux system. If by some chance you need to create one which is not provided then you should first try to use the MAKEDEV script. This script is usually located in /dev/MAKEDEV but might also have a copy (or a symbolic link) in /sbin/MAKEDEV. If it turns out not to be in your path then you will need to specify the path to it explicitly. In general the command is used as: # /dev/MAKEDEV -v ttyS0 create ttyS0 c 4 64 root:dialout 0660 This will create the device file /dev/ttyS0 with major node 4 and minor node 64 as a character device with access permissions 0660 with owner root and group dialout. ttyS0 is a serial port. The major and minor node numbers are numbers understood by the kernel. The kernel refers to hardware devices as numbers, this would be very difficult for us to remember, so we use filenames. Access permissions of 0660 means read and write permission for the owner (root in this case) and read and write permission for members of the group (dialout in this case) with no access for anyone else. ----------------------------------------------------------------------------- 5.2. The mknod command MAKEDEV is the preferred way of creating device files which are not present. However sometimes the MAKEDEV script will not know about the device file you wish to create. This is where the mknod command comes in. In order to use mknod you need to know the major and minor node numbers for the device you wish to create. The devices.txt file in the kernel source documentation is the canonical source of this information. To take an example, let us suppose that our version of the MAKEDEV script does not know how to create the /dev/ttyS0 device file. We need to use mknod to create it. We know from looking at the devices.txt file that it should be a character device with major number 4 and minor number 64. So we now know all we need to create the file. # mknod /dev/ttyS0 c 4 64 # chown root.dialout /dev/ttyS0 # chmod 0644 /dev/ttyS0 # ls -l /dev/ttyS0 crw-rw---- 1 root dialout 4, 64 Oct 23 18:23 /dev/ttyS0 As you can see, many more steps are required to create the file. In this example you can see the process required however. It is unlikely in the extreme that the ttyS0 file would not be provided by the MAKEDEV script, but it suffices to illustrate the point. ----------------------------------------------------------------------------- 5.3. Device List This list which follows is by no means exhaustive or as detailed as it could be. Many of these device files will need support compiled into your kernel for the hardware. Read the kernel documentation to find details of any particular device. If you think there are other devices which should be included here but aren't then let me know. I will try to include them in the next revision. /dev/dsp Digital Signal Processor. Basically this forms the interface between software which produces sound and your soundcard. It is a character device on major node 14 and minor 3. /dev/fd0 The first floppy drive. If you are lucky enough to have several drives then they will be numbered sequentially. It is a character device on major node 2 and minor 0. /dev/fb0 The first framebuffer device. A framebuffer is an abstraction layer between software and graphics hardware. This means that applications do not need to know about what kind of hardware you have but merely how to communicate with the framebuffer driver's API (Application Programming Interface) which is well defined and standardised. The framebuffer is a character device and is on major node 29 and minor 0. /dev/hda /dev/hda is the master IDE drive on the primary IDE controller. /dev/hdb is the slave drive on the primary controller. /dev/hdc and /dev/hdd are the master and slave devices on the secondary controller respectively. Each disk is divided into partitions. Partitions 1-4 are primary partitions and partitions 5 and above are logical partitions inside extended partitions. Therefore the device file which references each partition is made up of several parts. For example /dev/hdc9 references partition 9 (a logical partition inside an extended partition type) on the master IDE drive on the secondary IDE controller. The major and minor node numbers are somewhat complex. For the first IDE controller all partitions are block devices on major node 3. The master drive hda is at minor 0 and the slave drive hdb is at minor 64. For each partition inside the drive add the partition number to the minor node number for the drive. For example /dev/hdb5 is major 3, minor 69 (64 + 5 = 69). Drives on the secondary interface are handled the same way, but with major node 22. /dev/ht0 The first IDE tape drive. Subsequent drives are numbered ht1 etc. They are character devices on major node 37 and start at minor node 0 for ht0 1 for ht1 etc. /dev/js0 The first analogue joystick. Subsequent joysticks are numbered js1, js2 etc. Digital joysticks are called djs0, djs1 and so on. They are character devices on major node 15. The analogue joysticks start at minor node 0 and go up to 127 (more than enough for even the most fanatic gamer). Digital joysticks start at minor node 128. /dev/lp0 The first parallel printer device. Subsequent printers are numbered lp1, lp2 etc. They are character devices on major mode 6 and minor nodes starting at 0 and numbered sequentially. /dev/loop0 The first loopback device. Loopback devices are used for mounting filesystems which are not located on other block devices such as disks. For example if you wish to mount an iso9660 CD ROM image without burning it to CD then you need to use a loopback device to do so. This is usually transparent to the user and is handled by the mount command. Refer to the manual pages for mount and losetup. The loopback devices are block devices on major node 7 and with minor nodes starting at 0 and numbered sequentially. /dev/md0 First metadisk group. Metadisks are related to RAID (Redundant Array of Independent Disks) devices. Please refer to the various RAID HOWTOs at the LDP for more details. Metadisk devices are block devices on major node 9 with minor nodes starting at 0 and numbered sequentially. /dev/mixer This is part of the OSS (Open Sound System) driver. Refer to the OSS documentation at [http://www.opensound.com] http://www.opensound.com for more details. It is a character device on major node 14, minor node 0. /dev/null The bit bucket. A black hole where you can send data for it never to be seen again. Anything sent to /dev/null will disappear. This can be useful if, for example, you wish to run a command but not have any feedback appear on the terminal. It is a character device on major node 1 and minor node 3. /dev/psaux The PS/2 mouse port. This is a character device on major node 10, minor node 1. /dev/pda Parallel port IDE disks. These are named similarly to disks on the internal IDE controllers (/dev/hd*). They are block devices on major node 45. Minor nodes need slightly more explanation here. The first device is /dev/pda and it is on minor node 0. Partitions on this device are found by adding the partition number to the minor number for the device. Each device is limited to 15 partitions each rather than 63 (the limit for internal IDE disks). /dev/pdb minor nodes start at 16, /dev/pdc at 32 and /dev/pdd at 48. So for example the minor node number for /dev/pdc6 would be 38 (32 + 6 = 38). This scheme limits you to 4 parallel disks of 15 partitions each. /dev/pcd0 Parallel port CD ROM drives. These are numbered from 0 onwards. All are block devices on major node 46. /dev/pcd0 is on minor node 0 with subsequent drives being on minor nodes 1, 2, 3 etc. /dev/pt0 Parallel port tape devices. Tapes do not have partitions so these are just numbered sequentially. They are character devices on major node 96. The minor node numbers start from 0 for /dev/pt0, 1 for /dev/pt1, and so on. /dev/parport0 The raw parallel ports. Most devices which are attached to parallel ports have their own drivers. This is a device to access the port directly. It is a character device on major node 99 with minor node 0. Subsequent devices after the first are numbered sequentially incrementing the minor node. /dev/random or /dev/urandom These are kernel random number generators. /dev/random is a non-deterministic generator which means that the value of the next number cannot be guessed from the preceding ones. It uses the entropy of the system hardware to generate numbers. When it has no more entropy to use then it must wait until it has collected more before it will allow any more numbers to be read from it. /dev/urandom works similarly. Initially it also uses the entropy of the system hardware, but when there is no more entropy to use it will continue to return numbers using a pseudo random number generating formula. This is considered to be less secure for vital purposes such as cryptographic key pair generation. If security is your overriding concern then use /dev/random, if speed is more important then /dev/urandom works fine. They are character devices on major node 1 with minor nodes 8 for /dev/random and 9 for /dev/urandom. /dev/zero This is a simple way of getting many 0s. Every time you read from this device it will return 0. This can be useful sometimes, for example when you want a file of fixed length but don't really care what it contains. It is a character device on major node 1 and minor node 5. ----------------------------------------------------------------------------- Chapter 6. Using Disks and Other Storage Media "On a clear disk you can seek forever. " When you install or upgrade your system, you need to do a fair amount of work on your disks. You have to make filesystems on your disks so that files can be stored on them and reserve space for the different parts of your system. This chapter explains all these initial activities. Usually, once you get your system set up, you won't have to go through the work again, except for using floppies. You'll need to come back to this chapter if you add a new disk or want to fine-tune your disk usage. The basic tasks in administering disks are:   *  Format your disk. This does various things to prepare it for use, such as checking for bad sectors. (Formatting is nowadays not necessary for most hard disks.)   *  Partition a hard disk, if you want to use it for several activities that aren't supposed to interfere with one another. One reason for partitioning is to store different operating systems on the same disk. Another reason is to keep user files separate from system files, which simplifies back-ups and helps protect the system files from corruption.   *  Make a filesystem (of a suitable type) on each disk or partition. The disk means nothing to Linux until you make a filesystem; then files can be created and accessed on it.   *  Mount different filesystems to form a single tree structure, either automatically, or manually as needed. (Manually mounted filesystems usually need to be unmounted manually as well.) Chapter 7 contains information about virtual memory and disk caching, of which you also need to be aware when using disks. ----------------------------------------------------------------------------- 6.1. Two kinds of devices UNIX, and therefore Linux, recognizes two different kinds of device: random-access block devices (such as disks), and character devices (such as tapes and serial lines), some of which may be serial, and some random-access. Each supported device is represented in the filesystem as a device file. When you read or write a device file, the data comes from or goes to the device it represents. This way no special programs (and no special application programming methodology, such as catching interrupts or polling a serial port) are necessary to access devices; for example, to send a file to the printer, one could just say $ cat filename > /dev/lp1 $ and the contents of the file are printed (the file must, of course, be in a form that the printer understands). However, since it is not a good idea to have several people cat their files to the printer at the same time, one usually uses a special program to send the files to be printed (usually lpr). This program makes sure that only one file is being printed at a time, and will automatically send files to the printer as soon as it finishes with the previous file. Something similar is needed for most devices. In fact, one seldom needs to worry about device files at all. Since devices show up as files in the filesystem (in the /dev directory), it is easy to see just what device files exist, using ls or another suitable command. In the output of ls -l, the first column contains the type of the file and its permissions. For example, inspecting a serial device might give $ ls -l /dev/ttyS0 crw-rw-r-- 1 root dialout 4, 64 Aug 19 18:56 /dev/ttyS0 $ The first character in the first column, i.e., `c' in crw-rw-rw- above, tells an informed user the type of the file, in this case a character device. For ordinary files, the first character is `-', for directories it is `d', and for block devices `b'; see the ls man page for further information. Note that usually all device files exist even though the device itself might be not be installed. So just because you have a file /dev/sda, it doesn't mean that you really do have an SCSI hard disk. Having all the device files makes the installation programs simpler, and makes it easier to add new hardware (there is no need to find out the correct parameters for and create the device files for the new device). ----------------------------------------------------------------------------- 6.2. Hard disks This subsection introduces terminology related to hard disks. If you already know the terms and concepts, you can skip this subsection. See Figure 6-1 for a schematic picture of the important parts in a hard disk. A hard disk consists of one or more circular platters, [12] of which either or both surfaces are coated with a magnetic substance used for recording the data. For each surface, there is a read-write head that examines or alters the recorded data. The platters rotate on a common axis; typical rotation speed is 5400 or 7200 rotations per minute, although high-performance hard disks have higher speeds and older disks may have lower speeds. The heads move along the radius of the platters; this movement combined with the rotation of the platters allows the head to access all parts of the surfaces. The processor (CPU) and the actual disk communicate through a disk controller. This relieves the rest of the computer from knowing how to use the drive, since the controllers for different types of disks can be made to use the same interface towards the rest of the computer. Therefore, the computer can say just "hey disk, give me what I want", instead of a long and complex series of electric signals to move the head to the proper location and waiting for the correct position to come under the head and doing all the other unpleasant stuff necessary. (In reality, the interface to the controller is still complex, but much less so than it would otherwise be.) The controller may also do other things, such as caching, or automatic bad sector replacement. The above is usually all one needs to understand about the hardware. There are also other things, such as the motor that rotates the platters and moves the heads, and the electronics that control the operation of the mechanical parts, but they are mostly not relevant for understanding the working principles of a hard disk. The surfaces are usually divided into concentric rings, called tracks, and these in turn are divided into sectors. This division is used to specify locations on the hard disk and to allocate disk space to files. To find a given place on the hard disk, one might say "surface 3, track 5, sector 7". Usually the number of sectors is the same for all tracks, but some hard disks put more sectors in outer tracks (all sectors are of the same physical size, so more of them fit in the longer outer tracks). Typically, a sector will hold 512 bytes of data. The disk itself can't handle smaller amounts of data than one sector. Figure 6-1. A schematic picture of a hard disk. [hd-schematic] Each surface is divided into tracks (and sectors) in the same way. This means that when the head for one surface is on a track, the heads for the other surfaces are also on the corresponding tracks. All the corresponding tracks taken together are called a cylinder. It takes time to move the heads from one track (cylinder) to another, so by placing the data that is often accessed together (say, a file) so that it is within one cylinder, it is not necessary to move the heads to read all of it. This improves performance. It is not always possible to place files like this; files that are stored in several places on the disk are called fragmented. The number of surfaces (or heads, which is the same thing), cylinders, and sectors vary a lot; the specification of the number of each is called the geometry of a hard disk. The geometry is usually stored in a special, battery-powered memory location called the CMOS RAM, from where the operating system can fetch it during bootup or driver initialization. Unfortunately, the BIOS [13] has a design limitation, which makes it impossible to specify a track number that is larger than 1024 in the CMOS RAM, which is too little for a large hard disk. To overcome this, the hard disk controller lies about the geometry, and translates the addresses given by the computer into something that fits reality. For example, a hard disk might have 8 heads, 2048 tracks, and 35 sectors per track. [14] Its controller could lie to the computer and claim that it has 16 heads, 1024 tracks, and 35 sectors per track, thus not exceeding the limit on tracks, and translates the address that the computer gives it by halving the head number, and doubling the track number. The mathematics can be more complicated in reality, because the numbers are not as nice as here (but again, the details are not relevant for understanding the principle). This translation distorts the operating system's view of how the disk is organized, thus making it impractical to use the all-data-on-one-cylinder trick to boost performance. The translation is only a problem for IDE disks. SCSI disks use a sequential sector number (i.e., the controller translates a sequential sector number to a head, cylinder, and sector triplet), and a completely different method for the CPU to talk with the controller, so they are insulated from the problem. Note, however, that the computer might not know the real geometry of an SCSI disk either. Since Linux often will not know the real geometry of a disk, its filesystems don't even try to keep files within a single cylinder. Instead, it tries to assign sequentially numbered sectors to files, which almost always gives similar performance. The issue is further complicated by on-controller caches, and automatic prefetches done by the controller. Each hard disk is represented by a separate device file. There can (usually) be only two or four IDE hard disks. These are known as /dev/hda, /dev/hdb, / dev/hdc, and /dev/hdd, respectively. SCSI hard disks are known as /dev/sda, / dev/sdb, and so on. Similar naming conventions exist for other hard disk types; see Chapter 5 for more information. Note that the device files for the hard disks give access to the entire disk, with no regard to partitions (which will be discussed below), and it's easy to mess up the partitions or the data in them if you aren't careful. The disks' device files are usually used only to get access to the master boot record (which will also be discussed below). ----------------------------------------------------------------------------- 6.3. Floppies A floppy disk consists of a flexible membrane covered on one or both sides with similar magnetic substance as a hard disk. The floppy disk itself doesn't have a read-write head, that is included in the drive. A floppy corresponds to one platter in a hard disk, but is removable and one drive can be used to access different floppies, and the same floppy can be read by many drives, whereas the hard disk is one indivisible unit. Like a hard disk, a floppy is divided into tracks and sectors (and the two corresponding tracks on either side of a floppy form a cylinder), but there are many fewer of them than on a hard disk. A floppy drive can usually use several different types of disks; for example, a 3.5 inch drive can use both 720 kB and 1.44 MB disks. Since the drive has to operate a bit differently and the operating system must know how big the disk is, there are many device files for floppy drives, one per combination of drive and disk type. Therefore, /dev/fd0H1440 is the first floppy drive (fd0), which must be a 3.5 inch drive, using a 3.5 inch, high density disk (H) of size 1440 kB (1440), i.e., a normal 3.5 inch HD floppy. The names for floppy drives are complex, however, and Linux therefore has a special floppy device type that automatically detects the type of the disk in the drive. It works by trying to read the first sector of a newly inserted floppy using different floppy types until it finds the correct one. This naturally requires that the floppy is formatted first. The automatic devices are called /dev/fd0, /dev/fd1, and so on. The parameters the automatic device uses to access a disk can also be set using the program setfdprm. This can be useful if you need to use disks that do not follow any usual floppy sizes, e.g., if they have an unusual number of sectors, or if the autodetecting for some reason fails and the proper device file is missing. Linux can handle many nonstandard floppy disk formats in addition to all the standard ones. Some of these require using special formatting programs. We'll skip these disk types for now, but in the mean time you can examine the /etc/ fdprm file. It specifies the settings that setfdprm recognizes. The operating system must know when a disk has been changed in a floppy drive, for example, in order to avoid using cached data from the previous disk. Unfortunately, the signal line that is used for this is sometimes broken, and worse, this won't always be noticeable when using the drive from within MS-DOS. If you are experiencing weird problems using floppies, this might be the reason. The only way to correct it is to repair the floppy drive. ----------------------------------------------------------------------------- 6.4. CD-ROMs A CD-ROM drive uses an optically read, plastic coated disk. The information is recorded on the surface of the disk [15] in small `holes' aligned along a spiral from the center to the edge. The drive directs a laser beam along the spiral to read the disk. When the laser hits a hole, the laser is reflected in one way; when it hits smooth surface, it is reflected in another way. This makes it easy to code bits, and therefore information. The rest is easy, mere mechanics. CD-ROM drives are slow compared to hard disks. Whereas a typical hard disk will have an average seek time less than 15 milliseconds, a fast CD-ROM drive can use tenths of a second for seeks. The actual data transfer rate is fairly high at hundreds of kilobytes per second. The slowness means that CD-ROM drives are not as pleasant to use as hard disks (some Linux distributions provide `live' filesystems on CD-ROMs, making it unnecessary to copy the files to the hard disk, making installation easier and saving a lot of hard disk space), although it is still possible. For installing new software, CD-ROMs are very good, since maximum speed is not essential during installation. There are several ways to arrange data on a CD-ROM. The most popular one is specified by the international standard ISO 9660. This standard specifies a very minimal filesystem, which is even more crude than the one MS-DOS uses. On the other hand, it is so minimal that every operating system should be able to map it to its native system. For normal UNIX use, the ISO 9660 filesystem is not usable, so an extension to the standard has been developed, called the Rock Ridge extension. Rock Ridge allows longer filenames, symbolic links, and a lot of other goodies, making a CD-ROM look more or less like any contemporary UNIX filesystem. Even better, a Rock Ridge filesystem is still a valid ISO 9660 filesystem, making it usable by non-UNIX systems as well. Linux supports both ISO 9660 and the Rock Ridge extensions; the extensions are recognized and used automatically. The filesystem is only half the battle, however. Most CD-ROMs contain data that requires a special program to access, and most of these programs do not run under Linux (except, possibly, under dosemu, the Linux MS-DOS emulator, or wine, the Windows emulator. [16] There is also VMWare, a commercial product which emulates an entire x86 machine in software [17]) . A CD-ROM drive is accessed via the corresponding device file. There are several ways to connect a CD-ROM drive to the computer: via SCSI, via a sound card, or via EIDE. The hardware hacking needed to do this is outside the scope of this book, but the type of connection decides the device file. ----------------------------------------------------------------------------- 6.5. Tapes A tape drive uses a tape, similar [18] to cassettes used for music. A tape is serial in nature, which means that in order to get to any given part of it, you first have to go through all the parts in between. A disk can be accessed randomly, i.e., you can jump directly to any place on the disk. The serial access of tapes makes them slow. On the other hand, tapes are relatively cheap to make, since they do not need to be fast. They can also easily be made quite long, and can therefore contain a large amount of data. This makes tapes very suitable for things like archiving and backups, which do not require large speeds, but benefit from low costs and large storage capacities. ----------------------------------------------------------------------------- 6.6. Formatting Formatting is the process of writing marks on the magnetic media that are used to mark tracks and sectors. Before a disk is formatted, its magnetic surface is a complete mess of magnetic signals. When it is formatted, some order is brought into the chaos by essentially drawing lines where the tracks go, and where they are divided into sectors. The actual details are not quite exactly like this, but that is irrelevant. What is important is that a disk cannot be used unless it has been formatted. The terminology is a bit confusing here: in MS-DOS and MS Windows, the word formatting is used to cover also the process of creating a filesystem (which will be discussed below). There, the two processes are often combined, especially for floppies. When the distinction needs to be made, the real formatting is called low-level formatting, while making the filesystem is called high-level formatting. In UNIX circles, the two are called formatting and making a filesystem, so that's what is used in this book as well. For IDE and some SCSI disks the formatting is actually done at the factory and doesn't need to be repeated; hence most people rarely need to worry about it. In fact, formatting a hard disk can cause it to work less well, for example because a disk might need to be formatted in some very special way to allow automatic bad sector replacement to work. Disks that need to be or can be formatted often require a special program anyway, because the interface to the formatting logic inside the drive is different from drive to drive. The formatting program is often either on the controller BIOS, or is supplied as an MS-DOS program; neither of these can easily be used from within Linux. During formatting one might encounter bad spots on the disk, called bad blocks or bad sectors. These are sometimes handled by the drive itself, but even then, if more of them develop, something needs to be done to avoid using those parts of the disk. The logic to do this is built into the filesystem; how to add the information into the filesystem is described below. Alternatively, one might create a small partition that covers just the bad part of the disk; this approach might be a good idea if the bad spot is very large, since filesystems can sometimes have trouble with very large bad areas. Floppies are formatted with fdformat. The floppy device file to use is given as the parameter. For example, the following command would format a high density, 3.5 inch floppy in the first floppy drive: $ fdformat /dev/fd0H1440 Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB. Formatting ... done Verifying ... done $ Note that if you want to use an autodetecting device (e.g., /dev/fd0), you must set the parameters of the device with setfdprm first. To achieve the same effect as above, one would have to do the following: $ setfdprm /dev/fd0 1440/1440 $ fdformat /dev/fd0 Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB. Formatting ... done Verifying ... done $ It is usually more convenient to choose the correct device file that matches the type of the floppy. Note that it is unwise to format floppies to contain more information than what they are designed for. fdformat will also validate the floppy, i.e., check it for bad blocks. It will try a bad block several times (you can usually hear this, the drive noise changes dramatically). If the floppy is only marginally bad (due to dirt on the read/write head, some errors are false signals), fdformat won't complain, but a real error will abort the validation process. The kernel will print log messages for each I/O error it finds; these will go to the console or, if syslog is being used, to the file /usr/log/messages. fdformat itself won't tell where the error is (one usually doesn't care, floppies are cheap enough that a bad one is automatically thrown away). $ fdformat /dev/fd0H1440 Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB. Formatting ... done Verifying ... read: Unknown error $ The badblocks command can be used to search any disk or partition for bad blocks (including a floppy). It does not format the disk, so it can be used to check even existing filesystems. The example below checks a 3.5 inch floppy with two bad blocks. $ badblocks /dev/fd0H1440 1440 718 719 $ badblocks outputs the block numbers of the bad blocks it finds. Most filesystems can avoid such bad blocks. They maintain a list of known bad blocks, which is initialised when the filesystem is made, and can be modified later. The initial search for bad blocks can be done by the mkfs command (which initializes the filesystem), but later checks should be done with badblocks and the new blocks should be added with fsck. We'll describe mkfs and fsck later. Many modern disks automatically notice bad blocks, and attempt to fix them by using a special, reserved good block instead. This is invisible to the operating system. This feature should be documented in the disk's manual, if you're curious if it is happening. Even such disks can fail, if the number of bad blocks grows too large, although chances are that by then the disk will be so rotten as to be unusable. ----------------------------------------------------------------------------- 6.7. Partitions A hard disk can be divided into several partitions. Each partition functions as if it were a separate hard disk. The idea is that if you have one hard disk, and want to have, say, two operating systems on it, you can divide the disk into two partitions. Each operating system uses its partition as it wishes and doesn't touch the other ones. This way the two operating systems can co-exist peacefully on the same hard disk. Without partitions one would have to buy a hard disk for each operating system. Floppies are not usually partitioned. There is no technical reason against this, but since they're so small, partitions would be useful only very rarely. CD-ROMs are usually also not partitioned, since it's easier to use them as one big disk, and there is seldom a need to have several operating systems on one. ----------------------------------------------------------------------------- 6.7.1. The MBR, boot sectors and partition table The information about how a hard disk has been partitioned is stored in its first sector (that is, the first sector of the first track on the first disk surface). The first sector is the master boot record (MBR) of the disk; this is the sector that the BIOS reads in and starts when the machine is first booted. The master boot record contains a small program that reads the partition table, checks which partition is active (that is, marked bootable), and reads the first sector of that partition, the partition's boot sector (the MBR is also a boot sector, but it has a special status and therefore a special name). This boot sector contains another small program that reads the first part of the operating system stored on that partition (assuming it is bootable), and then starts it. The partitioning scheme is not built into the hardware, or even into the BIOS. It is only a convention that many operating systems follow. Not all operating systems do follow it, but they are the exceptions. Some operating systems support partitions, but they occupy one partition on the hard disk, and use their internal partitioning method within that partition. The latter type exists peacefully with other operating systems (including Linux), and does not require any special measures, but an operating system that doesn't support partitions cannot co-exist on the same disk with any other operating system. As a safety precaution, it is a good idea to write down the partition table on a piece of paper, so that if it ever corrupts you don't have to lose all your files. (A bad partition table can be fixed with fdisk). The relevant information is given by the fdisk -l command: $ fdisk -l /dev/hda Disk /dev/hda: 15 heads, 57 sectors, 790 cylinders Units = cylinders of 855 * 512 bytes Device Boot Begin Start End Blocks Id System /dev/hda1 1 1 24 10231+ 82 Linux swap /dev/hda2 25 25 48 10260 83 Linux native /dev/hda3 49 49 408 153900 83 Linux native /dev/hda4 409 409 790 163305 5 Extended /dev/hda5 409 409 744 143611+ 83 Linux native /dev/hda6 745 745 790 19636+ 83 Linux native $ ----------------------------------------------------------------------------- 6.7.2. Extended and logical partitions The original partitioning scheme for PC hard disks allowed only four partitions. This quickly turned out to be too little in real life, partly because some people want more than four operating systems (Linux, MS-DOS, OS/ 2, Minix, FreeBSD, NetBSD, or Windows/NT, to name a few), but primarily because sometimes it is a good idea to have several partitions for one operating system. For example, swap space is usually best put in its own partition for Linux instead of in the main Linux partition for reasons of speed (see below). To overcome this design problem, extended partitions were invented. This trick allows partitioning a primary partition into sub-partitions. The primary partition thus subdivided is the extended partition; the sub-partitions are logical partitions. They behave like primary partitions, but are created differently. There is no speed difference between them. The partition structure of a hard disk might look like that in Figure 6-2. The disk is divided into three primary partitions, the second of which is divided into two logical partitions. Part of the disk is not partitioned at all. The disk as a whole and each primary partition has a boot sector. Figure 6-2. A sample hard disk partitioning. [hd-layout] ----------------------------------------------------------------------------- 6.7.3. Partition types The partition tables (the one in the MBR, and the ones for extended partitions) contain one byte per partition that identifies the type of that partition. This attempts to identify the operating system that uses the partition, or what it uses it for. The purpose is to make it possible to avoid having two operating systems accidentally using the same partition. However, in reality, operating systems do not really care about the partition type byte; e.g., Linux doesn't care at all what it is. Worse, some of them use it incorrectly; e.g., at least some versions of DR-DOS ignore the most significant bit of the byte, while others don't. There is no standardization agency to specify what each byte value means, but some commonly accepted ones are included in in Table 6-1. A more complete list is available in the Linux fdisk program. Table 6-1. Partition types (from the Linux fdisk program). +--+-------------------+---+----------------+---+------------------+ |0 |Empty |40 |Venix 80286 |94 |Amoeba BBT | +--+-------------------+---+----------------+---+------------------+ |1 |DOS 12-bit FAT |51 |Novell? |a5 |BSD/386 | +--+-------------------+---+----------------+---+------------------+ |2 |XENIX root |52 |Microport |b7 |BSDI fs | +--+-------------------+---+----------------+---+------------------+ |3 |XENIX usr |63 |GNU HURD |b8 |BSDI swap | +--+-------------------+---+----------------+---+------------------+ |4 |DOS 16-bit FAT <32M|64 |Novell |c7 |Syrinx | +--+-------------------+---+----------------+---+------------------+ |5 |Extended |75 |PC/IX |db |CP/M | +--+-------------------+---+----------------+---+------------------+ |6 |DOS 16-bit >=32M |80 |Old MINIX |e1 |DOS access | +--+-------------------+---+----------------+---+------------------+ |7 |OS/2 HPFS |81 |Linux/MINIX |e3 |DOS R/O | +--+-------------------+---+----------------+---+------------------+ |8 |AIX |82 |Linux swap |f2 |DOS secondary | +--+-------------------+---+----------------+---+------------------+ |9 |AIX bootable |83 |Linux native |ff |BBT | +--+-------------------+---+----------------+---+------------------+ |a |OS/2 Boot Manager |93 |Amoeba |  |  | +--+-------------------+---+----------------+---+------------------+ ----------------------------------------------------------------------------- 6.7.4. Partitioning a hard disk There are many programs for creating and removing partitions. Most operating systems have their own, and it can be a good idea to use each operating system's own, just in case it does something unusual that the others can't. Many of the programs are called fdisk, including the Linux one, or variations thereof. Details on using the Linux fdisk are given on its man page. The cfdisk command is similar to fdisk, but has a nicer (full screen) user interface. When using IDE disks, the boot partition (the partition with the bootable kernel image files) must be completely within the first 1024 cylinders. This is because the disk is used via the BIOS during boot (before the system goes into protected mode), and BIOS can't handle more than 1024 cylinders. It is sometimes possible to use a boot partition that is only partly within the first 1024 cylinders. This works as long as all the files that are read with the BIOS are within the first 1024 cylinders. Since this is difficult to arrange, it is a very bad idea to do it; you never know when a kernel update or disk defragmentation will result in an unbootable system. Therefore, make sure your boot partition is completely within the first 1024 cylinders [19] . Some newer versions of the BIOS and IDE disks can, in fact, handle disks with more than 1024 cylinders. If you have such a system, you can forget about the problem; if you aren't quite sure of it, put it within the first 1024 cylinders. Each partition should have an even number of sectors, since the Linux filesystems use a 1 kilobyte block size, i.e., two sectors. An odd number of sectors will result in the last sector being unused. This won't result in any problems, but it is ugly, and some versions of fdisk will warn about it. Changing a partition's size usually requires first backing up everything you want to save from that partition (preferably the whole disk, just in case), deleting the partition, creating new partition, then restoring everything to the new partition. If the partition is growing, you may need to adjust the sizes (and backup and restore) of the adjoining partitions as well. Since changing partition sizes is painful, it is preferable to get the partitions right the first time, or have an effective and easy to use backup system. If you're installing from a media that does not require much human intervention (say, from CD-ROM, as opposed to floppies), it is often easy to play with different configuration at first. Since you don't already have data to back up, it is not so painful to modify partition sizes several times. There is a program for MS-DOS, called fips [20] , which resizes an MS-DOS partition without requiring the backup and restore, but for other filesystems it is still necessary. ----------------------------------------------------------------------------- 6.7.5. Device files and partitions Each partition and extended partition has its own device file. The naming convention for these files is that a partition's number is appended after the name of the whole disk, with the convention that 1-4 are primary partitions (regardless of how many primary partitions there are) and number greater than 5 are logical partitions (regardless of within which primary partition they reside). For example, /dev/hda1 is the first primary partition on the first IDE hard disk, and /dev/sdb7 is the third extended partition on the second SCSI hard disk. ----------------------------------------------------------------------------- 6.8. Filesystems 6.8.1. What are filesystems? A filesystem is the methods and data structures that an operating system uses to keep track of files on a disk or partition; that is, the way the files are organized on the disk. The word is also used to refer to a partition or disk that is used to store the files or the type of the filesystem. Thus, one might say "I have two filesystems" meaning one has two partitions on which one stores files, or that one is using the "extended filesystem", meaning the type of the filesystem. The difference between a disk or partition and the filesystem it contains is important. A few programs (including, reasonably enough, programs that create filesystems) operate directly on the raw sectors of a disk or partition; if there is an existing file system there it will be destroyed or seriously corrupted. Most programs operate on a filesystem, and therefore won't work on a partition that doesn't contain one (or that contains one of the wrong type). Before a partition or disk can be used as a filesystem, it needs to be initialized, and the bookkeeping data structures need to be written to the disk. This process is called making a filesystem. Most UNIX filesystem types have a similar general structure, although the exact details vary quite a bit. The central concepts are superblock, inode, data block, directory block, and indirection block. The superblock contains information about the filesystem as a whole, such as its size (the exact information here depends on the filesystem). An inode contains all information about a file, except its name. The name is stored in the directory, together with the number of the inode. A directory entry consists of a filename and the number of the inode which represents the file. The inode contains the numbers of several data blocks, which are used to store the data in the file. There is space only for a few data block numbers in the inode, however, and if more are needed, more space for pointers to the data blocks is allocated dynamically. These dynamically allocated blocks are indirect blocks; the name indicates that in order to find the data block, one has to find its number in the indirect block first. UNIX filesystems usually allow one to create a hole in a file (this is done with the lseek() system call; check the manual page), which means that the filesystem just pretends that at a particular place in the file there is just zero bytes, but no actual disk sectors are reserved for that place in the file (this means that the file will use a bit less disk space). This happens especially often for small binaries, Linux shared libraries, some databases, and a few other special cases. (Holes are implemented by storing a special value as the address of the data block in the indirect block or inode. This special address means that no data block is allocated for that part of the file, ergo, there is a hole in the file.) ----------------------------------------------------------------------------- 6.8.2. Filesystems galore Linux supports several types of filesystems. As of this writing the most important ones are: minix The oldest, presumed to be the most reliable, but quite limited in features (some time stamps are missing, at most 30 character filenames) and restricted in capabilities (at most 64 MB per filesystem). xia A modified version of the minix filesystem that lifts the limits on the filenames and filesystem sizes, but does not otherwise introduce new features. It is not very popular, but is reported to work very well. ext3 The ext3 filesystem has all the features of the ext2 filesystem. The difference is, journaling has been added. This improves performance and recovery time in case of a system crash. This has become more popular than ext2. ext2 The most featureful of the native Linux filesystems. It is designed to be easily upwards compatible, so that new versions of the filesystem code do not require re-making the existing filesystems. ext An older version of ext2 that wasn't upwards compatible. It is hardly ever used in new installations any more, and most people have converted to ext2. reiserfs A more robust filesystem. Journalling is used which makes data loss less likely. Journalling is a mechanism whereby a record is kept of transaction which are to be performed, or which have been performed. This allows the filesystem to reconstruct itself fairly easily after damage caused by, for example, improper shutdowns. In addition, support for several foreign filesystems exists, to make it easier to exchange files with other operating systems. These foreign filesystems work just like native ones, except that they may be lacking in some usual UNIX features, or have curious limitations, or other oddities. msdos Compatibility with MS-DOS (and OS/2 and Windows NT) FAT filesystems. umsdos Extends the msdos filesystem driver under Linux to get long filenames, owners, permissions, links, and device files. This allows a normal msdos filesystem to be used as if it were a Linux one, thus removing the need for a separate partition for Linux. vfat This is an extension of the FAT filesystem known as FAT32. It supports larger disk sizes than FAT. Most MS Windows disks are vfat. iso9660 The standard CD-ROM filesystem; the popular Rock Ridge extension to the CD-ROM standard that allows longer file names is supported automatically. nfs A networked filesystem that allows sharing a filesystem between many computers to allow easy access to the files from all of them. smbfs A networks filesystem which allows sharing of a filesystem with an MS Windows computer. It is compatible with the Windows file sharing protocols. hpfs The OS/2 filesystem. sysv SystemV/386, Coherent, and Xenix filesystems. The choice of filesystem to use depends on the situation. If compatibility or other reasons make one of the non-native filesystems necessary, then that one must be used. If one can choose freely, then it is probably wisest to use ext3, since it has all the features of ext2, and is a journaled filesystem. There is also the proc filesystem, usually accessible as the /proc directory, which is not really a filesystem at all, even though it looks like one. The proc filesystem makes it easy to access certain kernel data structures, such as the process list (hence the name). It makes these data structures look like a filesystem, and that filesystem can be manipulated with all the usual file tools. For example, to get a listing of all processes one might use the command $ ls -l /proc total 0 dr-xr-xr-x 4 root root 0 Jan 31 20:37 1 dr-xr-xr-x 4 liw users 0 Jan 31 20:37 63 dr-xr-xr-x 4 liw users 0 Jan 31 20:37 94 dr-xr-xr-x 4 liw users 0 Jan 31 20:37 95 dr-xr-xr-x 4 root users 0 Jan 31 20:37 98 dr-xr-xr-x 4 liw users 0 Jan 31 20:37 99 -r--r--r-- 1 root root 0 Jan 31 20:37 devices -r--r--r-- 1 root root 0 Jan 31 20:37 dma -r--r--r-- 1 root root 0 Jan 31 20:37 filesystems -r--r--r-- 1 root root 0 Jan 31 20:37 interrupts -r-------- 1 root root 8654848 Jan 31 20:37 kcore -r--r--r-- 1 root root 0 Jan 31 11:50 kmsg -r--r--r-- 1 root root 0 Jan 31 20:37 ksyms -r--r--r-- 1 root root 0 Jan 31 11:51 loadavg -r--r--r-- 1 root root 0 Jan 31 20:37 meminfo -r--r--r-- 1 root root 0 Jan 31 20:37 modules dr-xr-xr-x 2 root root 0 Jan 31 20:37 net dr-xr-xr-x 4 root root 0 Jan 31 20:37 self -r--r--r-- 1 root root 0 Jan 31 20:37 stat -r--r--r-- 1 root root 0 Jan 31 20:37 uptime -r--r--r-- 1 root root 0 Jan 31 20:37 version $ (There will be a few extra files that don't correspond to processes, though. The above example has been shortened.) Note that even though it is called a filesystem, no part of the proc filesystem touches any disk. It exists only in the kernel's imagination. Whenever anyone tries to look at any part of the proc filesystem, the kernel makes it look as if the part existed somewhere, even though it doesn't. So, even though there is a multi-megabyte /proc/kcore file, it doesn't take any disk space. ----------------------------------------------------------------------------- 6.8.3. Which filesystem should be used? There is usually little point in using many different filesystems. Currently, ext3 is the most popular filesystem, because it is a journaled filesystem. Currently it is probably the wisest choice. Reiserfs is another popular choice because it to is journaled. Depending on the overhead for bookkeeping structures, speed, (perceived) reliability, compatibility, and various other reasons, it may be advisable to use another file system. This needs to be decided on a case-by-case basis. A filesystem that uses journaling is also called a journaled filesystem. A journaled filesystem maintains a log, or journal, of what has happened on a filesystem. In the event of a system crash, or if your 2 year old son hits the power button like mine loves to do, a journaled filesystem is designed to use the filesystem's logs to recreate unsaved and lost data. This makes data loss much less likely and will likely become a standard feature in Linux filesystems. However, do not get a false sense of security from this. Like everything else, errors can arise. Always make sure to back up your data in the event of an emergency. ----------------------------------------------------------------------------- 6.8.4. Creating a filesystem Filesystems are created, i.e., initialized, with the mkfs command. There is actually a separate program for each filesystem type. mkfs is just a front end that runs the appropriate program depending on the desired filesystem type. The type is selected with the -t fstype option. The programs called by mkfs have slightly different command line interfaces. The common and most important options are summarized below; see the manual pages for more. -t fstype Select the type of the filesystem. -c Search for bad blocks and initialize the bad block list accordingly. -l filename Read the initial bad block list from the name file. To create an ext2 filesystem on a floppy, one would give the following commands: $ fdformat -n /dev/fd0H1440 Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB. Formatting ... done $ badblocks /dev/fd0H1440 1440 $>$ bad-blocks $ mkfs -t ext2 -l bad-blocks /dev/fd0H1440 mke2fs 0.5a, 5-Apr-94 for EXT2 FS 0.5, 94/03/10 360 inodes, 1440 blocks 72 blocks (5.00%) reserved for the super user First data block=1 Block size=1024 (log=0) Fragment size=1024 (log=0) 1 block group 8192 blocks per group, 8192 fragments per group 360 inodes per group Writing inode tables: done Writing superblocks and filesystem accounting information: done $ First, the floppy was formatted (the -n option prevents validation, i.e., bad block checking). Then bad blocks were searched with badblocks, with the output redirected to a file, bad-blocks. Finally, the filesystem was created, with the bad block list initialized by whatever badblocks found. The -c option could have been used with mkfs instead of badblocks and a separate file. The example below does that. $ mkfs -t ext2 -c /dev/fd0H1440 mke2fs 0.5a, 5-Apr-94 for EXT2 FS 0.5, 94/03/10 360 inodes, 1440 blocks 72 blocks (5.00%) reserved for the super user First data block=1 Block size=1024 (log=0) Fragment size=1024 (log=0) 1 block group 8192 blocks per group, 8192 fragments per group 360 inodes per group Checking for bad blocks (read-only test): done Writing inode tables: done Writing superblocks and filesystem accounting information: done $ The -c option is more convenient than a separate use of badblocks, but badblocks is necessary for checking after the filesystem has been created. The process to prepare filesystems on hard disks or partitions is the same as for floppies, except that the formatting isn't needed. ----------------------------------------------------------------------------- 6.8.5. Mounting and unmounting Before one can use a filesystem, it has to be mounted. The operating system then does various bookkeeping things to make sure that everything works. Since all files in UNIX are in a single directory tree, the mount operation will make it look like the contents of the new filesystem are the contents of an existing subdirectory in some already mounted filesystem. For example, Figure 6-3 shows three separate filesystems, each with their own root directory. When the last two filesystems are mounted below /home and / usr, respectively, on the first filesystem, we can get a single directory tree, as in Figure 6-4. Figure 6-3. Three separate filesystems. [hd-mount-separate] Figure 6-4. /home and /usr have been mounted. [hd-mount-mounted] The mounts could be done as in the following example: $ mount /dev/hda2 /home $ mount /dev/hda3 /usr $ The mount command takes two arguments. The first one is the device file corresponding to the disk or partition containing the filesystem. The second one is the directory below which it will be mounted. After these commands the contents of the two filesystems look just like the contents of the /home and /usr directories, respectively. One would then say that "/dev/hda2 is mounted on /home", and similarly for /usr. To look at either filesystem, one would look at the contents of the directory on which it has been mounted, just as if it were any other directory. Note the difference between the device file, /dev/hda2, and the mounted-on directory, /home. The device file gives access to the raw contents of the disk, the mounted-on directory gives access to the files on the disk. The mounted-on directory is called the mount point. Linux supports many filesystem types. mount tries to guess the type of the filesystem. You can also use the -t fstype option to specify the type directly; this is sometimes necessary, since the heuristics mount uses do not always work. For example, to mount an MS-DOS floppy, you could use the following command: $ mount -t msdos /dev/fd0 /floppy $ The mounted-on directory need not be empty, although it must exist. Any files in it, however, will be inaccessible by name while the filesystem is mounted. (Any files that have already been opened will still be accessible. Files that have hard links from other directories can be accessed using those names.) There is no harm done with this, and it can even be useful. For instance, some people like to have /tmp and /var/tmp synonymous, and make /tmp be a symbolic link to /var/tmp. When the system is booted, before the /var filesystem is mounted, a /var/tmp directory residing on the root filesystem is used instead. When /var is mounted, it will make the /var/tmp directory on the root filesystem inaccessible. If /var/tmp didn't exist on the root filesystem, it would be impossible to use temporary files before mounting / var. If you don't intend to write anything to the filesystem, use the -r switch for mount to do a read-only mount. This will make the kernel stop any attempts at writing to the filesystem, and will also stop the kernel from updating file access times in the inodes. Read-only mounts are necessary for unwritable media, e.g., CD-ROMs. The alert reader has already noticed a slight logistical problem. How is the first filesystem (called the root filesystem, because it contains the root directory) mounted, since it obviously can't be mounted on another filesystem? Well, the answer is that it is done by magic. [21] The root filesystem is magically mounted at boot time, and one can rely on it to always be mounted. If the root filesystem can't be mounted, the system does not boot. The name of the filesystem that is magically mounted as root is either compiled into the kernel, or set using LILO or rdev. The root filesystem is usually first mounted read-only. The startup scripts will then run fsck to verify its validity, and if there are no problems, they will re-mount it so that writes will also be allowed. fsck must not be run on a mounted filesystem, since any changes to the filesystem while fsck is running will cause trouble. Since the root filesystem is mounted read-only while it is being checked, fsck can fix any problems without worry, since the remount operation will flush any metadata that the filesystem keeps in memory. On many systems there are other filesystems that should also be mounted automatically at boot time. These are specified in the /etc/fstab file; see the fstab man page for details on the format. The details of exactly when the extra filesystems are mounted depend on many factors, and can be configured by each administrator if need be; see Chapter 8. When a filesystem no longer needs to be mounted, it can be unmounted with umount. [22] umount takes one argument: either the device file or the mount point. For example, to unmount the directories of the previous example, one could use the commands $ umount /dev/hda2 $ umount /usr $ See the man page for further instructions on how to use the command. It is imperative that you always unmount a mounted floppy. Don't just pop the floppy out of the drive! Because of disk caching, the data is not necessarily written to the floppy until you unmount it, so removing the floppy from the drive too early might cause the contents to become garbled. If you only read from the floppy, this is not very likely, but if you write, even accidentally, the result may be catastrophic. Mounting and unmounting requires super user privileges, i.e., only root can do it. The reason for this is that if any user can mount a floppy on any directory, then it is rather easy to create a floppy with, say, a Trojan horse disguised as /bin/sh, or any other often used program. However, it is often necessary to allow users to use floppies, and there are several ways to do this:   * Give the users the root password. This is obviously bad security, but is the easiest solution. It works well if there is no need for security anyway, which is the case on many non-networked, personal systems.   * Use a program such as sudo to allow users to use mount. This is still bad security, but doesn't directly give super user privileges to everyone. [23]   * Make the users use mtools, a package for manipulating MS-DOS filesystems, without mounting them. This works well if MS-DOS floppies are all that is needed, but is rather awkward otherwise.   * List the floppy devices and their allowable mount points together with the suitable options in /etc/fstab. The last alternative can be implemented by adding a line like the following to the /etc/fstab file: /dev/fd0 /floppy msdos user,noauto 0 0 The columns are: device file to mount, directory to mount on, filesystem type, options, backup frequency (used by dump), and fsck pass number (to specify the order in which filesystems should be checked upon boot; 0 means no check). The noauto option stops this mount to be done automatically when the system is started (i.e., it stops mount -a from mounting it). The user option allows any user to mount the filesystem, and, because of security reasons, disallows execution of programs (normal or setuid) and interpretation of device files from the mounted filesystem. After this, any user can mount a floppy with an msdos filesystem with the following command: $ mount /floppy $ The floppy can (and needs to, of course) be unmounted with the corresponding umount command. If you want to provide access to several types of floppies, you need to give several mount points. The settings can be different for each mount point. For example, to give access to both MS-DOS and ext2 floppies, you could have the following to lines in /etc/fstab: /dev/fd0 /dosfloppy msdos user,noauto 0 0 /dev/fd0 /ext2floppy ext2 user,noauto 0 0 For MS-DOS filesystems (not just floppies), you probably want to restrict access to it by using the uid, gid, and umask filesystem options, described in detail on the mount manual page. If you aren't careful, mounting an MS-DOS filesystem gives everyone at least read access to the files in it, which is not a good idea. ----------------------------------------------------------------------------- 6.8.6. Checking filesystem integrity with fsck Filesystems are complex creatures, and as such, they tend to be somewhat error-prone. A filesystem's correctness and validity can be checked using the fsck command. It can be instructed to repair any minor problems it finds, and to alert the user if there any unrepairable problems. Fortunately, the code to implement filesystems is debugged quite effectively, so there are seldom any problems at all, and they are usually caused by power failures, failing hardware, or operator errors; for example, by not shutting down the system properly. Most systems are setup to run fsck automatically at boot time, so that any errors are detected (and hopefully corrected) before the system is used. Use of a corrupted filesystem tends to make things worse: if the data structures are messed up, using the filesystem will probably mess them up even more, resulting in more data loss. However, fsck can take a while to run on big filesystems, and since errors almost never occur if the system has been shut down properly, a couple of tricks are used to avoid doing the checks in such cases. The first is that if the file /etc/fastboot exists, no checks are made. The second is that the ext2 filesystem has a special marker in its superblock that tells whether the filesystem was unmounted properly after the previous mount. This allows e2fsck (the version of fsck for the ext2 filesystem) to avoid checking the filesystem if the flag indicates that the unmount was done (the assumption being that a proper unmount indicates no problems). Whether the /etc/fastboot trick works on your system depends on your startup scripts, but the ext2 trick works every time you use e2fsck. It has to be explicitly bypassed with an option to e2fsck to be avoided. (See the e2fsck man page for details on how.) The automatic checking only works for the filesystems that are mounted automatically at boot time. Use fsck manually to check other filesystems, e.g., floppies. If fsck finds unrepairable problems, you need either in-depth knowledge of how filesystems work in general, and the type of the corrupt filesystem in particular, or good backups. The latter is easy (although sometimes tedious) to arrange, the former can sometimes be arranged via a friend, the Linux newsgroups and mailing lists, or some other source of support, if you don't have the know-how yourself. I'd like to tell you more about it, but my lack of education and experience in this regard hinders me. The debugfs program by Theodore Ts'o should be useful. fsck must only be run on unmounted filesystems, never on mounted filesystems (with the exception of the read-only root during startup). This is because it accesses the raw disk, and can therefore modify the filesystem without the operating system realizing it. There will be trouble, if the operating system is confused. ----------------------------------------------------------------------------- 6.8.7. Checking for disk errors with badblocks It can be a good idea to periodically check for bad blocks. This is done with the badblocks command. It outputs a list of the numbers of all bad blocks it can find. This list can be fed to fsck to be recorded in the filesystem data structures so that the operating system won't try to use the bad blocks for storing data. The following example will show how this could be done. $ badblocks /dev/fd0H1440 1440 > bad-blocks $ fsck -t ext2 -l bad-blocks /dev/fd0H1440 Parallelizing fsck version 0.5a (5-Apr-94) e2fsck 0.5a, 5-Apr-94 for EXT2 FS 0.5, 94/03/10 Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Check reference counts. Pass 5: Checking group summary information. /dev/fd0H1440: ***** FILE SYSTEM WAS MODIFIED ***** /dev/fd0H1440: 11/360 files, 63/1440 blocks $ If badblocks reports a block that was already used, e2fsck will try to move the block to another place. If the block was really bad, not just marginal, the contents of the file may be corrupted. ----------------------------------------------------------------------------- 6.8.8. Fighting fragmentation When a file is written to disk, it can't always be written in consecutive blocks. A file that is not stored in consecutive blocks is fragmented. It takes longer to read a fragmented file, since the disk's read-write head will have to move more. It is desirable to avoid fragmentation, although it is less of a problem in a system with a good buffer cache with read-ahead. The ext2 filesystem attempts to keep fragmentation at a minimum, by keeping all blocks in a file close together, even if they can't be stored in consecutive sectors. Ext2 effectively always allocates the free block that is nearest to other blocks in a file. For ext2, it is therefore seldom necessary to worry about fragmentation. There is a program for defragmenting an ext2 filesystem called, strangely enough, defrag [24] . There are many MS-DOS defragmentation programs that move blocks around in the filesystem to remove fragmentation. For other filesystems, defragmentation must be done by backing up the filesystem, re-creating it, and restoring the files from backups. Backing up a filesystem before defragmenting is a good idea for all filesystems, since many things can go wrong during the defragmentation. ----------------------------------------------------------------------------- 6.8.9. Other tools for all filesystems Some other tools are also useful for managing filesystems. df shows the free disk space on one or more filesystems; du shows how much disk space a directory and all its files contain. These can be used to hunt down disk space wasters. Both have manual pages which detail the (many) options which can be used. sync forces all unwritten blocks in the buffer cache (see Section 7.6) to be written to disk. It is seldom necessary to do this by hand; the daemon process update does this automatically. It can be useful in catastrophes, for example if update or its helper process bdflush dies, or if you must turn off power now and can't wait for update to run. Again, there are manual pages. The man is your very best friend in linux. Its cousin apropos is also very useful when you don't know what the name of the command you want is. ----------------------------------------------------------------------------- 6.8.10. Other tools for the ext2/ext3 filesystem In addition to the filesystem creator (mke2fs) and checker (e2fsck) accessible directly or via the filesystem type independent front ends, the ext2 filesystem has some additional tools that can be useful. tune2fs adjusts filesystem parameters. Some of the more interesting parameters are:   *  A maximal mount count. e2fsck enforces a check when filesystem has been mounted too many times, even if the clean flag is set. For a system that is used for developing or testing the system, it might be a good idea to reduce this limit.   *  A maximal time between checks. e2fsck can also enforce a maximal time between two checks, even if the clean flag is set, and the filesystem hasn't been mounted very often. This can be disabled, however.   *  Number of blocks reserved for root. Ext2 reserves some blocks for root so that if the filesystem fills up, it is still possible to do system administration without having to delete anything. The reserved amount is by default 5 percent, which on most disks isn't enough to be wasteful. However, for floppies there is no point in reserving any blocks. See the tune2fs manual page for more information. dumpe2fs shows information about an ext2 filesystem, mostly from the superblock. Figure 6-5 shows a sample output. Some of the information in the output is technical and requires understanding of how the filesystem works (see appendix XXX ext2fspaper), but much of it is readily understandable even for layadmins. Figure 6-5. Sample output from dumpe2fs dumpe2fs 0.5b, 11-Mar-95 for EXT2 FS 0.5a, 94/10/23 Filesystem magic number:  0xEF53 Filesystem state:         clean Errors behavior:          Continue Inode count:              360 Block count:              1440 Reserved block count:     72 Free blocks:              1133 Free inodes:              326 First block:              1 Block size:               1024 Fragment size:            1024 Blocks per group:         8192 Fragment