URLS for UNIX

http://unixtips.hpage.co.in/
http://shtnawi.blogspot.in/2011/09/dns.html

DNS Configuration in Solaris 10

DNS Configuration in Solaris 10
I like to share configuration of DNS primary and secondary in solaris 10. This implementation was done in a solaris 10 X86.Solaris 10 comes with Bind 9.3 version.

Step 1:

Include the name server in a /etc/resolv.conf file



Here domain name is dev.test.com and there are two name servers defined(192.168.0.1 and 192.168.0.2)

step 2:

Change hosts in nsswitch.conf to point to dns.



Here while querying, it will first look in hosts file then it will go for dns server

I am going to include following host name and ip address in DNS. For this we need to create forward and reverse lookup zones.

Sun10 --> 192.168.0.1
sunclient --> 192.168.0.2
sunclient1 --> 192.168.0.3
sunclient2 --> 192.168.0.4
sunclient3 --> 192.168.0.7

Setting up Primary DNS server

In the below setups i am going to create a zone for domain dev.test.com

Step 1:

Configure /etc/named.conf file. If the file is not available areate new named.conf file




In the above file Secret keys are generated using command "rndc-confgen". We need to include this key in primary and secondary servers named.conf file
for secured zone transfer.

Directory is path where zone db files were kept. Here zone db files were kept in /var/named. We are using type as master since this is a primary server
and the db file name is db.dev and this is kept in path /var/named. Here i created both forward and reverse lookup zone


Step 2:

We need to create domain zone file in /var/named. I am going to create three files for the three zones.




Step 3:

Create named.boot file in /var/named



Thats it we are done

Start dns using either /usr/sbin/named or svcadm enable dns/server




Setting up secondary servers

A Secondary DNS Server is basically just a backup server. I does not hold the master versions of the zone information but rather it holds copies of them. Most sites use Secondary servers in remote locations or to cut down on the load on the Primary server. The Secondary server performs zone transfers at said times making sure it has the newest versions of the zone information.

In order to configure secondary server. we need to make few changes in primary servers zone db file.

Include secondary servers name in db file. I have included secondary server name sun10 in db.dev file, db.192.168.0



Here NS sun10 is secondary server. The same should be included in other zone file db.192.168.0.

create /etc/named.conf file in secodary server



Here i have included type as slave as it is secondary server. we are done now

Because this is a Secondary Server, there is no need to do anything with zone files. BONUS! As long as you have everything setup right on your Primary and your named.conf file is configured properly, everything will work properly.

Each time the Primary site's zone files are modified and the Serial # is incremented, BIND will send out a notify to all Secondaries (any server in the zone file with a IN NS statement) stating a change has been made. BIND on each of these servers will then check it's own zone files to see if it has the same version or not. If the version that the Primary has notified it about is newer then it will perform a zone transfer and obtain the newer version.

If everything is setup properly, you will never need to make any changes to your Secondary server except to upgrade BIND itself. All changes from here on out should be made on the Primary server. The exception is if a new zone is added on the primary, you need to add it to the secondaries also if you want them to be secondary to the new zone.

Thanks,
Rajeev

Solaris: Which process is associated with a socket connection?

Consider the following real world scenario

[...] one process is waiting for a very long time in a send system call. It is sending on a valid fd but the question we have is that, is there any way to find who is on the other end of that fd? We want to know to which process is that message being sent to. [...]

Here is how I proceed in finding the other end of the socket, and the state of the socket connection with Mozilla's Thunderbird mail client in one end of the socket connection:

1. Get the process id of the application
   

   % prstat
      PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP      
   22385 mandalik  180M   64M sleep   49    0   0:05:15 0.1% thunderbird-bin/5

   
   
2. Run pfiles on the pid - it prints a list of open files including open sockets (pfiles is the Solaris supported equivalent of lsof utility).
   

   % pfiles 22385
   22385:  /usr/lib/thunderbird/thunderbird-bin -UILocale C -contentLocale C
     Current rlimit: 512 file descriptors
     ...
     ...
   
     33: S_IFSOCK mode:0666 dev:280,0 ino:31544 uid:0 gid:0 size:0
         O_RDWR|O_NONBLOCK
           SOCK_STREAM
           SO_SNDBUF(49152),SO_RCVBUF(49640)
           sockname: AF_INET 192.168.1.2  port: 60364
           peername: AF_INET 192.18.39.10  port: 993
           ...
           ...

3. Locate the socket id and the corresponding sockname/port#, peername/port# in the output of pfiles pid (see step #2).
   
   Here my assumption is that I know the socket id I'm interested in. In the above output, 33 is the socket id. One end of the socket is bound to port 60364 on the local host 192.168.1.2; and the other end of the socket is bound to port 993 on the remote host 192.18.39.10.
   
   
4. Run netstat -a | egrep "| (get the port numbers from step 3); and check the state of the socket connection. If you see anything other than ESTABLISHED, it indicates trouble.
   

   % netstat -a | egrep "60364|993"
   solaris-devx-iprb0.60364 mail-sfbay.sun.com.993 48460      0 49640      0 ESTABLISHED

   
   If you want to see the host names in numbers (IP addresses), run netstat with option -n.
   

   %  netstat -an | egrep "60364|993"
   192.168.1.2.60364    192.18.39.10.993     49559      0 49640      0 ESTABLISHED

   
   Now since we know both ends of the socket, we can easily get the state of the socket connection at the other end by running netstat -an | egrep '|.
   
   If the state of the socket connection is CLOSE_WAIT, have a look at the following diagnosis: CPU hog with connections in CLOSE_WAIT.

Finally to answer ... which process is that message being sent to ... part of the original question:

Follow the above steps and find the remote host (or IP) and remote port number. To find the corresponding process id on the remote machine to which the other half of the socket belongs to, do the following:

1. Login as root user on the remote host.
   
2. cd /proc
3. Run pfiles * | egrep "^[0-9]|sockname" > /tmp/pfiles.txt.
   
4. vi /tmp/pfiles.txt and search for the port number. If you scroll few lines up, you can see the process ID, name of the process along with its argument(s).

http://www.feep.net/sendmail/tutorial/intro/MUA-MTA-MDA.html

MUAs, MTAs, and MDAs There are three main parts of the Unix E-mail system: Part What It Does Examples Mail User Agent (MUA) The MUA is the program which the user uses to read and send e-mail. It reads incoming messages that have been delivered to the user's mailbox, and passes outgoing messages to an MTA for sending. elm, pine, mutt Mail Transfer Agent (MTA) The MTA basicly acts as a "mail router". It accepts a message passed to it by either an MUA or another MTA, decides based upon the message header which delivery method it should use, and then passes the message to the appropriate MDA for that delivery method. sendmail, postfix, qmail Mail Delivery Agent (MDA) The MDA accepts a piece of mail from an MTA and performs the actual delivery. mail.local, procmail The MTA is the most important one of these. It is responsible for doing all the "intelligent" work of e-mail transfer. While it does not actually perform any of the delivery itself, it is the part which tells the other parts how to interact and what to do. In a sense, the MTA is the glue which holds the whole process together. To illustrate how the three parts of the email system work together, here's a very general example. This is what happens when the user jsmith@host1.uiuc.edu sends e-mail to johndoe@host2.uiuc.edu: jsmith's MUA (pine, elm, etc) on host1.uiuc.edu passes the message to the MTA (sendmail) on the local host. The MTA (sendmail) notices that the message is addressed to a user at host2.uiuc.edu. Since it is configured to know that it can reach host2.uiuc.edu via SMTP, it passes the message to the SMTP MDA (the SMTP MDA is actually builtin to sendmail, but all other MDAs are external programs). The SMTP MDA connects to the MTA on host2.uiuc.edu (sendmail) and sends it the message. The MTA on host2.uiuc.edu (sendmail) notices that the message is addressed to a user on the local host, so it passes the message to the local MDA. The local MDA saves the message in user johndoe's mailbox. The next time johndoe logs in to host2.uiuc.edu and runs his MUA, the message is there waiting for him to read. As you can see, none of this would be possible without the MTA!

Breaking RCS locks.

Breaking RCS locks. So you want to edit some awesome file, only to see this when you check it out: co -l /mnt/yp/netgroup /mnt/yp/RCS/netgroup,v --> /mnt/yp/netgroup co: /mnt/yp/RCS/netgroup,v: Revision 1.1008 is already locked by u0064824. Blood and ashes! Here's how to break the lock so the world can move forward. First move the file to a backup location: mv /mnt/yp/netgroup /mnt/yp/netgroup-lockedByu0064824-Sep272002 Then break the lock rcs -u /mnt/yp/netgroup RCS file: RCS/netgroup,v Revision 1.1008 is already locked by u0064824. Do you want to break the lock? [ny](n): y State the reason for breaking the lock: (terminate with single '.' or end of file) >> Testing lock breaking -SH. >> . 1.1008 unlocked done Now you can check out the file and using diff (or vimdiff) you can see what changes the previous user made so you can hopefully merge in any changes they might have made. co -l /mnt/yp/netgroup diff /mnt/yp/netgroup /mnt/yp/netgroup-lockedByu0064824-Sep272002 Then edit away! When you're done we check the file in such that it stays in place and isn't magicly whisked away in the RCS repository: ci -u /mnt/yp/netgroup

VMware Player vmnetcfg

VMware Player vmnetcfg


Recently I was using VMware Player 4 on my laptop but I was not able to get an IP address with Bridged Networking selected. Being a user of VMware products for many years, I knew there was a quick fix.

I navigated to the VMware Player directory to run vmnetcfg but it was not there. Unfortunately, the new version of VMware Player doesn’t include vmnetcfg in the installer anymore. Lucy for all of us, it is included in the installer.

To extract vmnetcfg.exe from the installer do the following:


1. Run the installer with /e option. For example:
VMware-player-4.0.0-197124.exe /e .\extract
Contents will be extracted to “extract” folder.
2. Open “network.cab” and copy vmnetcfg.exe to your installation folder,
typically “C:\Program Files\VMware\VMware Player\”.

Now you can run vmnetcfg to exclude network adapters from binding.

How Rsync Works

A Practical Overview
Foreword
The original Rsync technical report and Andrew Tridgell's Phd thesis (pdf) Are both excellent documents for understanding the theoretical mathematics and some of the mechanics of the rsync algorithm. Unfortunately they are more about the theory than the implementation of the rsync utility (hereafter referred to as Rsync).

In this document I hope to describe...

A non-mathematical overview of the rsync algorithm.
How that algorithm is implemented in the rsync utility.
The protocol, in general terms, used by the rsync utility.
The identifiable roles the rsync processes play.
This document be able to serve as a guide for programmers needing something of an entré into the source code but the primary purpose is to give the reader a foundation from which he may understand

Why rsync behaves as it does.
The limitations of rsync.
Why a requested feature is unsuited to the code-base.
This document describes in general terms the construction and behaviour of Rsync. In some cases details and exceptions that would contribute to specific accuracy have been sacrificed for the sake meeting the broader goals.

Processes and Roles
When we talk about Rsync we use specific terms to refer to various processes and their roles in the task performed by the utility. For effective communication it is important that we all be speaking the same language; likewise it is important that we mean the same things when we use certain terms in a given context. On the rsync mailing list there is often some confusion with regards to role and processes. For these reasons I will define a few terms used in the role and process contexts that will be used henceforth. client role The client initiates the synchronisation.
server role The remote rsync process or system to which the client connects either within a local transfer, via a remote shell or via a network socket.
This is a general term and should not be confused with the daemon.

Once the connection between the client and server is established the distinction between them is superseded by the sender and receiver roles.
daemon Role and process An Rsync process that awaits connections from clients. On a certain platform this would be called a service.
remote shell role and set of processes One or more processes that provide connectivity between an Rsync client and an Rsync server on a remote system.
sender role and process The Rsync process that has access to the source files being synchronised.
receiver role and process As a role the receiver is the destination system. As a process the receiver is the process that receives update data and writes it to disk.
generator process The generator process identifies changed files and manages the file level logic.



Process Startup
When an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

When Rsync communicates with a remote non-daemon server via a remote shell the startup method is to fork the remote shell which will start an Rsync server on the remote system. Both the Rsync client and server are communicating via pipes through the remote shell. As far as the rsync processes are concerned there is no network. In this mode the rsync options for the server process are passed on the command-line that is used to start the remote shell.

When Rsync is communicating with a daemon it is communicating directly with a network socket. This is the only sort of Rsync communication that could be called network aware. In this mode the rsync options must be sent over the socket, as described below.

At the very start of the communication between the client and the server, they each send the maximum protocol version they support to the other side. Each side then uses the minimum value as the the protocol level for the transfer. If this is a daemon-mode connection, rsync options are sent from the client to the server. Then, the exclude list is transmitted. From this point onward the client-server relationship is relevant only with regards to error and log message delivery.

Local Rsync jobs (when the source and destination are both on locally mounted filesystems) are done exactly like a push. The client, which becomes the sender, forks a server process to fulfill the receiver role. The client/sender and server/receiver communicate with each other over pipes.

The File List
The file list includes not only the pathnames but also ownership, mode, permissions, size and modtime. If the --checksum option has been specified it also includes the file checksums.
The first thing that happens once the startup has completed is that the sender will create the file list. While it is being built, each entry is transmitted to the receiving side in a network-optimised way.

When this is done, each side sorts the file list lexicographically by path relative to the base directory of the transfer. (The exact sorting algorithm varies depending on what protocol version is in effect for the transfer.) Once that has happened all references to files will be done by their index in the file list.

If necessary the sender follows the file list with id→name tables for users and groups which the receiver will use to do a id→name→id translation for every file in the file list.

After the file list has been received by the receiver, it will fork to become the generator and receiver pair completing the pipeline.

The Pipeline
Rsync is heavily pipelined. This means that it is a set of processes that communicate in a (largely) unidirectional way. Once the file list has been shared the pipeline behaves like this:
generator → sender → receiver
The output of the generator is input for the sender and the output of the sender is input for the receiver. Each process runs independently and is delayed only when the pipelines stall or when waiting for disk I/O or CPU resources.

The Generator
The generator process compares the file list with its local directory tree. Prior to beginning its primary function, if --delete has been specified, it will first identify local files not on the sender and delete them on the receiver.

The generator will then start walking the file list. Each file will be checked to see if it can be skipped. In the most common mode of operation files are not skipped if the modification time or size differs. If --checksum was specified a file-level checksum will be created and compared. Directories, device nodes and symlinks are not skipped. Missing directories will be created.

If a file is not to be skipped, any existing version on the receiving side becomes the "basis file" for the transfer, and is used as a data source that will help to eliminate matching data from having to be sent by the sender. To effect this remote matching of data, block checksums are created for the basis file and sent to the sender immediately following the file's index number. An empty block checksum set is sent for new files and if --whole-file was specified.

The block size and, in later versions, the size of the block checksum are calculated on a per file basis according to the size of that file.

The Sender
The sender process reads the file index numbers and associated block checksum sets one at a time from the generator.
For each file id the generator sends it will store the block checksums and build a hash index of them for rapid lookup.

Then the local file is read and a checksum is generated for the block beginning with the first byte of the local file. This block checksum is looked for in the set that was sent by the generator, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum”

If a block checksum match is found it is considered a matching block and any accumulated non-matching data will be sent to the receiver followed by the offset and length in the receiver's file of the matching block and the block checksum generator will be advanced to the next byte after the matching block.

Matching blocks can be identified in this way even if the blocks are reordered or at different offsets. This process is the very heart of the rsync algorithm.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfe), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file.

Generating the rolling checksums and searching for matches in the checksum set sent by the generator require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The Receiver
The receiver will read from the sender data for each file identified by the file index number. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance.

The Daemon
The daemon process, like many daemons, forks for every connection. On startup, it parses the rsyncd.conf file to determine what modules exist and to set the global options.
When a connection is received for a defined module the daemon forks a new child process to handle the connection. That child process then reads the rsyncd.conf file to set the options for the requested module, which may chroot to the module path and may drop setuid and setgid for the process. After that it will behave just like any other rsync server process adopting either a sender or receiver role.

The Rsync Protocol
A well-designed communications protocol has a number of characteristics.

Everything is sent in well defined packets with a header and an optional body or data payload.
In each packet's header a type and or command specified.
Each packet has a definite length.
In addition to these characteristics, protocols have varying degrees of statefulness, inter-packet independence, human readability, and the ability to reestablish a disconnected session.

Rsync's protocol has none of these good characteristics. The data is transferred as an unbroken stream of bytes. With the exception of the unmatched file-data, there are no length specifiers nor counts. Instead the meaning of each byte is dependent on its context as defined by the protocol level.

As an example, when the sender is sending the file list it simply sends each file list entry and terminates the list with a null byte. Within the file list entries, a bitfield indicates which fields of the structure to expect and those that are variable length strings are simply null terminated. The generator sending file numbers and block checksum sets works the same way.

This method of communication works quite well on reliable connections and it certainly has less data overhead than the formal protocols. It unfortunately makes the protocol extremely difficult to document, debug or extend. Each version of the protocol will have subtle differences on the wire that can only be anticipated by knowing the exact protocol version.

notes
This document is a work in progress. The author expects that it has some glaring oversights and some portions that may be more confusing than enlightening for some readers. It is hoped that this could evolve into a useful reference.
Specific suggestions for improvement are welcome, as would be a complete rewrite.