Configuring SnapVault on Netapp

Configuring SnapVault on Netapp

In this example, I will assume our Primary Filer as Filer1 and our secondary filer as filer2.

Step 1:

- Configure the Primary Filer for Snapvault, add the Snapvault license by using the following command on a terminal session:

license add sv_primary_license

Step 2:

- Enable snapvault on primary by using the following command

options snapvault.enable on

Step 3:

- Specify the name of the secondary storage system on the primary by using the command

options snapvault.access host=NameOfTheSecondaryFilerHere

Note that you can also use the following command to all all hosts as destination filers

options snapvault.access host=all

Step 4:

- Perform steps 1 and 2 on the secondary filer as well

Step 5:

- Enable the compliance clock on the secondary filer by using the command:

date -c initialize

Step 6:

- You can also optionally create a "log volume" for snapvault logs, but we will by pass it here

Step 7:

- Finally on the secondary filer for each qtree you need to backup on the primary, create a initial baseline transfer

For example, if you want the qtree called /vol/mail1/maillun1 on the primary to be backed up to a qtree on secondary called /vol/sv_mail1/maillun1 , you will give the following command:

snapvault start -S filer1:/vol/mail1/maillun1 /vol/sv_mail1/maillun1

Note that it is a good practiceto name your volumes with sv_ (indicating snapvault volume).

This initialization can take some time to complete., to check the progress, you can give the command:

snapvault status

That is it. The final step is to create a Snapvault schedule on Primary and secondary. Here are the steps

Step 8:

- Create a schedule on Primary by using the following command:

snapvault snap sched -x vol snap_name count [@day_list][@hour_list]

so for example, it will be:

snapvault snap sched -x filer1: /vol/mail1/maillun1 sv_nightly 150@mon-sat 0

meaning, snapvault will create backups with prefix sv_nightly for the primary qtree /vol/mail1/maillun1 at midnight monday to saturday and will retain 150 copies (one for every night)

Step 9:

Similar to step 8 configure the secondary, I will just demonstrate the command for the secondarysnapvault snap sched -x filer2: /vol/sv_mail1/maillun1sv_nightly 150@mon-sat 0

Creating (and Maintaining) Perl Modules

Creating (and Maintaining) Perl Modules
Goals

The goal of this web page is to help you write easily maintainable and re-usable code. In Perl, re-usability is implemented through modules, which are similar to libraries in other languages.

This page will guide you through creating your module and documenting it, as well as giving you some tips on how to make your code as maintainable and re-usable as possible.
Creating Perl Modules

Perl modules are those files that end in .pm. If you do things right, you can make the process of writing, testing, and installing your module really slick. You'll also be able to easily bundle up your module for testing and installation on other machines, or uploading to CPAN.

Here are the steps in creating a module:

* Create a place to develop your module
* Create skeleton files for your module.
* Document your module
* Write some Perl code
* Write some tests for your code
* Install the module
* Tips

Create a place to develop your module

The simplest way to do this is to create one directory per module. Give this directory any name that clearly identifies the module that it contains. See the Math::BaseCalc module for a simple example, or the Apache::Filter modules for a more involved one.
Create skeleton files for your module

Perl is distributed with a program called h2xs. This program, while initially intended to help programmers implement C extensions to Perl, can also be used to generate skeleton files for a new module.

Let's create a module called NewModule.pm that doesn't do very much. I'll run the h2xs program:

[~/modules],2:05pm% h2xs -AXc -n NewModule
Writing NewModule/NewModule.pm
Writing NewModule/Makefile.PL
Writing NewModule/test.pl
Writing NewModule/Changes
Writing NewModule/MANIFEST
[~/modules],2:05pm% cd NewModule/
[~/modules/NewModule],2:05pm% ls
Changes MANIFEST Makefile.PL NewModule.pm test.pl


The Changes file is where you might keep track keep track of changes you make to your module as you write new versions. If you're using RCS or CVS version control, you shouldn't use the Changes file, since all your history & logs will be in revision control and is much more reliable there (you are adding detailed revision notes in version control, aren't you?). I've found that the best scheme is to automatically build the Changes file from the revision control history, but your preferences might vary.

MANIFEST contains a list of files in this directory. If you add new files to the directory, you should also add them to the MANIFEST. The MANIFEST is used to create a tarball of your module for distribution, and it's also checked when people unpack the tarball and install the module.

Makefile.PL is a Perl program used to create a Unix Makefile. You'll use this Makefile to test and install your module.

NewModule.pm is your module. You'll write the code here in the next step.

test.pl is a Perl program that tests your module. You don't run it directly, you type "make test" at a Unix prompt and it runs it for you. We'll develop this test suite a little later.
Document your module

One of the great things about Perl modules is that they can have their documentation right in the same file. Once this module is installed, its documentation can be read by typing "perldoc NewModule" at a Unix prompt. Keeping the code and documentation together is a great thing, since it means you'll always have the most recent documentation if you've got the most recent code.

Here is some sample documentation.

=head1 NAME

NewModule - Perl module for hooting

=head1 SYNOPSIS

use NewModule;
my $hootie = new NewModule;
$hootie->verbose(1);
$hootie->hoot; # Hoots
$hootie->verbose(0);
$hootie->hoot; # Doesn't hoot


=head1 DESCRIPTION

This module hoots when it's verbose, and doesn't do anything
when it's not verbose.

=head2 Methods

=over 4

=item * $object->verbose(true or false)

=item * $object->verbose()

Returns the value of the 'verbose' property. When called with an
argument, it also sets the value of the property. Use a true or false
Perl value, such as 1 or 0.

=item * $object->hoot()

Returns a hoot if we're supposed to be verbose. Otherwise it returns
nothing.

=back

=head1 AUTHOR

Ken Williams (ken@mathforum.org)

=head1 COPYRIGHT

Copyright 1998 Swarthmore College. All rights reserved.

This library is free software; you can redistribute it and/or
modify it under the same terms as Perl itself.

=head1 SEE ALSO

perl(1).

=cut

When you create the module using h2xs it will create several sections for you automatically. They are:

* NAME - The name of the module and a very short description of what it does.
* SYNOPSIS - This should be a few lines of example code demonstrating how to use the major functions/methods of your module.
* DESCRIPTION - This should be some prose text describing what your module is for.
* AUTHOR - You.
* SEE ALSO - This should point the person reading your docs to other documentation that may be useful (docs for other Modules, C library docs, etc.)

One other critical section that you should create is FUNCTIONS or METHODS (depending on whether your code is function-based or object-oriented). This section should list every single function or method intended for public use. At the very minimum, these descriptions should list the parameters each function/method takes and the return values it can give back.

Feel free to expand your documentation beyond these sections. Make sure to note any areas where your module does something that might go against someone else's assumption. Also make sure to mention limitations of the module that might not be obvious without looking at the code.

Your documentation is complete when someone can use your module without ever having to look at its code. This is very important. This makes it possible for you to separate your module's documented interface from its internal implementation (guts). This is good because it means that you are free to change the module's internals as long as the interface remains the same.
POD (Plain Old Documentation)

The perldoc program expects your documentation to be in POD format. The pod format has a few (very few) tags that you use to markup plain text. As an aside, the Perl compiler ignores POD commands so they can be used for extended comments inside your code.

Here is a list of some of the tags, with some HTML tags that are similar in spirit:
POD tag HTML equivalent Description
=head1

Primary heading.
=head2

Secondary heading.
=over N

or
Indent N spaces until it finds a =back tag. The convention is generally to indent in multiples of 4, so you see =over 4 a lot.
=back

or

Indicates that you are done with indenting.
=item

Indicates a list item. The convention is to use =over to begin a list, and =back to end it. Generally you do =item *, which puts bullets in front of each list item.
=cut

Indicates the end of a POD section.

For more information on POD, type perldoc perlpod at a UNIX prompt. There's not much to POD, and it will behoove you to know it inside & out.
Write some Perl code

What you've got now is a documented, fully functional Perl module that doesn't do anything. We've got to write some code in NewModule.pm to make it do something. This code should implement the interface defined in the documentation we just wrote.

The NewModule.pm file will have this in it already:

package NewModule;

use strict;
use vars qw($VERSION @ISA @EXPORT @EXPORT_OK);

require Exporter;

@ISA = qw(Exporter AutoLoader);
# Items to export into callers namespace by default. Note: do not export
# names by default without a very good reason. Use EXPORT_OK instead.
# Do not simply export all your public functions/methods/constants.
@EXPORT = qw(

);
$VERSION = '0.01';


# Preloaded methods go here.

# Autoload methods go after =cut, and are processed by the autosplit program.

1;
__END__

Here is a line by line explanation of what this means:

*

use strict;
This turns on the strict pragmas. This means that you must declare all your variables (with "my") before using them, you are not allowed to use soft references (also known as symbolic or dynamic references), and that all subroutines used my be preceded by an ampersand (&) or followed by parentheses.

This is crucial to writing maintainable code for the following reasons:
1. It forces you either declare all your global variables in advance or to use lexically scoped variables (the preferred choice). This leads to cleaner code with cleaner function interfaces and fewer hidden dependencies.
2. It will help you track down typos in variable names because it will complain about undeclared variables when you make a typo.
3. It doesn't allow you to use soft references, which are almost never needed and are a maintenance nightmare.

It also makes your code run a bit faster, because the Perl interpreter can do some optimizations at compile-time rather than waiting until run-time to make all its decisions.
*

use vars qw($VERSION @ISA @EXPORT @EXPORT_OK);
The vars pragma is a way of declaring your global variables. Any variable listed here is available throughout the entire package scope.
*

require Exporter;
This loads the Exporter module, which allows you to export variables and subroutines into the calling packages namespace when the module is used with the use statement.
*

@ISA = qw(Exporter AutoLoader);
By placing a module name in the @ISA array, we are saying that the current package is a subclass (in the object-oriented sense) of that package. In practical terms, this means that if we try to use a method NewModule->foo and the &foo subroutine is not found in the NewModule package, then Perl will also check for &Exporter::foo and &AutoLoader::foo as well. If those aren't found, Perl will check Exporter's parents and AutoLoader's parents, and so on up the tree.

In general, you can remove the references to AutoLoader as you probably won't use this. If you are writing an object-oriented module (a class) then you should remove the Exporter-related code as well.
*

@EXPORT = qw(

);

Any variables or subroutines listed here will automatically be placed into the calling package's namespace. It is important to document these in order to prevent namespace conflicts.

*

$VERSION = '0.01';
All modules should have a version number. This helps with version control. In addition, it allows you to do:

use My::Module 1.21;

This will cause the compiler to die if it cannot find at least version 1.21 of the module. It checks the version by looking at the $VERSION variable.
If you're using RCS or CVS and you want to synchronize $VERSION with the revision numbers, you can do the following for the $VERSION scalar:
$VERSION = sprintf "%d.%03d", q$Revision: 1.6 $ =~ /: (\d+)\.(\d+)/;
This must be all on one line. The Revision tag will be kept up to date by the rcs programs. Perl in turn will use the regex to extract the major and minor numbers, which are then formatted by sprintf. In this case $VERSION will be set to '1.006'.

*

The references to preloaded and autoloaded methods are only relevant if you are using the AutoLoader module. Don't use the AutoLoader module unless you know what you're doing, it's generally not worth the trouble.
*

1;
When a module is loaded (via use) the compiler will complain unless the last statement executed when it is loaded is true. This line ensures that this is the case (as long as you don't place any code after this line). It's Perl's way of making sure that it successfully parsed all the way to the end of the file.
*

__END__
Anything after this token is ignored by the compiler. This is generally where you will place your documentation.

Let's create some sample code. Don't worry about what this code does or how it works. We're mostly concerned with having a few methods so we can demonstrate how to document a module. For reference, this code is using the Object Oriented Perl syntax and features that became available with Perl 5.

package NewModule;

use strict;
use vars qw($VERSION);
$VERSION = '0.01';

sub new {
my $package = shift;
return bless({}, $package);
}

sub verbose {
my $self = shift;
if (@_) {
$self->{'verbose'} = shift;
}
return $self->{'verbose'};
}

sub hoot {
my $self = shift;
return "Don't pollute!" if $self->{'verbose'};
return;
}

1;
__END__

Write some tests for your code

One of the benefits of developing modules this way is that you can maintain a list of tests for your code that make sure it's working properly. This is what the test.pl file is for. Let's put a couple of tests at the end of the file - here is the complete file now:

# Before `make install' is performed this script should be runnable with
# `make test'. After `make install' it should work as `perl test.pl'

######################### We start with some black magic to print on failure.

# Change 1..1 below to 1..last_test_to_print .
# (It may become useful if the test is moved to ./t subdirectory.)

BEGIN { $| = 1; print "1..1\n"; }
END {print "not ok 1\n" unless $loaded;}
use NewModule;
$loaded = 1;
print "ok 1\n";

######################### End of black magic.

# Insert your test code below (better if it prints "ok 13"
# (correspondingly "not ok 13") depending on the success of chunk 13
# of the test code):

# Test 2:
my $obj = new NewModule;
$obj->verbose(1);
my $result = $obj->hoot;
print ($result eq "Don't pollute!" ? "ok 2\n" : "not ok 2\n");

# Test 3:
$obj->verbose(0);
my $result = $obj->hoot;
print ($result eq "" ? "ok 3\n" : "not ok 3\n");

The first test has already been created by h2xs in step one. It just makes sure we can load NewModule.pm in the first place. The second and third tests check that the hoot method returns the right things. These tests were written by the programmer.

These tests should completely exercise every function/method of the entire module, as exhaustively as possible. This script should be the regression test for your module. Every time you make a change to the module's implementation, you can test it against this script to make sure that nothing is broken. It also lets you determine whether your code will work on different platforms.

While this is a signficant time commitment for a large module, it also has a big payoff. Whenever a change is made to this module, you can find out very quickly whether or not the existing functionality has been changed. And when a bug gets reported, the first thing you can do is add a test to test.pl that exhibits the bug - when we fix the bug, we'll never have to worry about it escaping our attention again.
Install the module

Now we've got everything written, we can try installing the module. The general procedure for installing any Perl module is:

perl Makefile.PL
make
make test
make install

Let's try it now.

[~/modules/NewModule],6:07pm% ls
Changes MANIFEST Makefile.PL NewModule.pm README test.pl
[~/modules/NewModule],6:07pm% perl Makefile.PL
Checking if your kit is complete...
Looks good
Writing Makefile for NewModule
[~/modules/NewModule],6:09pm% make
mkdir ./blib
mkdir ./blib/lib
mkdir ./blib/arch
mkdir ./blib/arch/auto
mkdir ./blib/arch/auto/NewModule
mkdir ./blib/lib/auto
mkdir ./blib/lib/auto/NewModule
mkdir ./blib/man3
cp NewModule.pm ./blib/lib/NewModule.pm
Manifying ./blib/man3/NewModule.3
[~/modules/NewModule],6:09pm% make test
PERL_DL_NONLAZY=1 /usr/local/bin/perl -I./blib/arch -I./blib/lib -I/usr/local/li
b/perl5/alpha-dec_osf/5.00404 -I/usr/local/lib/perl5 test.pl
1..1
ok 1
ok 2
ok 3
[~/modules/NewModule],6:10pm% su
s/key 1111 aa11111
Password:
[forum]:/home/ken/modules/NewModule# make install
Installing /usr/local/lib/perl5/site_perl/./NewModule.pm
Installing /usr/local/lib/perl5/man/man3/./NewModule.3
Writing /usr/local/lib/perl5/site_perl/alpha-dec_osf/auto/NewModule/.packlist
Appending installation info to /usr/local/lib/perl5/alpha-dec_osf/5.00404/perllocal.pod

Notice that I had to become root to install the module globally. Installation involves copying files into the Perl library directory, which most people don't have permission to copy into. Since this isn't a very useful module, I installed it and then immediately uninstalled it by deleting the first three files mentioned in the "make install" step.

If you want to install the module in some non-standard location (like /foo/bar/lib), you give a LIB directive in the Makefile.PL step, i.e. perl Makefile.PL LIB=/put/module/here. This can also be put inside Makefile.PL if you're writing a module that will only be used locally and has a specific installation location that you want to enforce.
Tips

* Write the documentation for a module first, before writing any code. Discuss the module with other people first, before writing any code. Plan the module first, before writing any code.
It's easy to come up with a solution to a problem. It takes planning to come up with a good solution. Remember: the documentation, not the code, defines what a module does.
* Every module should have a purpose. There's a proliferation of modules with names like "perlutils.pm", "rcs_utils.pm", and "utilUtils.pm" that have no obvious purpose, and it's difficult to know what each does. This leads to confusion and duplication of code.
* A note about naming conventions. The Perl standard for module names is that all modules start with a capital letter. Names starting with a lower case letter are reserved for pragmas. In addition, it is not a bad thing to have your name include two colons, as in the name Text::Wrap. When this module is installed, it will be placed in a directory named Text under the root library directory. The module code itself will be in a file called Wrap.pm. This helps keep the library directory more organized. In addition, the :: naming convention can also indicate class hierarchies, although it does not have to.
* If you're writing an object-oriented module, don't use the Exporter stuff. Exporter lets someone "use" your module and then use some of your module's functions without fully package-qualifying those functions. See the DataExchange module for an example, and read the Exporter documentation for the details.
* The AutoLoader stuff lets your module only compile certain subroutines as they're needed. It can make programs that use your module faster if they only use small parts of your module. It's usually not necessary to use this, but if you're writing a really huge module (like CGI.pm) it might be worth it. Read the AutoLoader docs for the details. In general, avoid it, it's not worth the headache.
* You can type "make realclean" to get rid of temporary files and directories that are created during the testing and installation process.
* You can type "make dist" to create a file suitable for uploading to CPAN or giving to your friends at the mall.

Avocent Cyclades ACS console server password reset or reset unit factory defaults

Avocent Cyclades ACS console server password reset or reset unit factory defaults

The Avocent Cycaldes ACS console servers are great little units that run Linux and even give you full root console access. The root password is “tslinux” by default, but if it has been changed then you can boot the unit into single user mode by supplying the argument “single” to the Linux kernel selection during the boot process (make sure you put a space between the existing Kernel parameters and “single”) which will drop you to a root prompt.

On my unit, this line comes up as right at the start of the boot process:

Linux/PPC load: root=/dev/ram ramdisk=0x0001F000

So you would type ” single” (remember the space!) to give you:

Linux/PPC load: root=/dev/ram ramdisk=0x0001F000 single

Then just hit enter and the unit will boot up into single user mode and give you the root prompt.

At this point, if you want to restore the entire unit to the factory default settings which will erase all of the configuration, then just run “defconf” and then reboot the unit.

If you want to keep the existing configuration intact but just reset the password then you can just use the traditional Linux passwd tool to edit /etc/passwd:

[root@(none) /]# passwd
New password:
Re-enter new password:
Password changed
[root@(none) /]# saveconf
Checking the configuration file list…
Compressing configuration files into /tmp/saving_config.tar.gz … done.
Saving configuration files to flash … done.
[root@(none) /]# reboot
[root@(none) /]# Restarting system.

DNS Configuration in Solaris 10

DNS Configuration in Solaris 10
I like to share configuration of DNS primary and secondary in solaris 10. This implementation was done in a solaris 10 X86.Solaris 10 comes with Bind 9.3 version.

Step 1:

Include the name server in a /etc/resolv.conf file



Here domain name is dev.test.com and there are two name servers defined(192.168.0.1 and 192.168.0.2)

step 2:

Change hosts in nsswitch.conf to point to dns.



Here while querying, it will first look in hosts file then it will go for dns server

I am going to include following host name and ip address in DNS. For this we need to create forward and reverse lookup zones.

Sun10 --> 192.168.0.1
sunclient --> 192.168.0.2
sunclient1 --> 192.168.0.3
sunclient2 --> 192.168.0.4
sunclient3 --> 192.168.0.7

Setting up Primary DNS server

In the below setups i am going to create a zone for domain dev.test.com

Step 1:

Configure /etc/named.conf file. If the file is not available areate new named.conf file




In the above file Secret keys are generated using command "rndc-confgen". We need to include this key in primary and secondary servers named.conf file
for secured zone transfer.

Directory is path where zone db files were kept. Here zone db files were kept in /var/named. We are using type as master since this is a primary server
and the db file name is db.dev and this is kept in path /var/named. Here i created both forward and reverse lookup zone


Step 2:

We need to create domain zone file in /var/named. I am going to create three files for the three zones.




Step 3:

Create named.boot file in /var/named



Thats it we are done

Start dns using either /usr/sbin/named or svcadm enable dns/server




Setting up secondary servers

A Secondary DNS Server is basically just a backup server. I does not hold the master versions of the zone information but rather it holds copies of them. Most sites use Secondary servers in remote locations or to cut down on the load on the Primary server. The Secondary server performs zone transfers at said times making sure it has the newest versions of the zone information.

In order to configure secondary server. we need to make few changes in primary servers zone db file.

Include secondary servers name in db file. I have included secondary server name sun10 in db.dev file, db.192.168.0



Here NS sun10 is secondary server. The same should be included in other zone file db.192.168.0.

create /etc/named.conf file in secodary server



Here i have included type as slave as it is secondary server. we are done now

Because this is a Secondary Server, there is no need to do anything with zone files. BONUS! As long as you have everything setup right on your Primary and your named.conf file is configured properly, everything will work properly.

Each time the Primary site's zone files are modified and the Serial # is incremented, BIND will send out a notify to all Secondaries (any server in the zone file with a IN NS statement) stating a change has been made. BIND on each of these servers will then check it's own zone files to see if it has the same version or not. If the version that the Primary has notified it about is newer then it will perform a zone transfer and obtain the newer version.

If everything is setup properly, you will never need to make any changes to your Secondary server except to upgrade BIND itself. All changes from here on out should be made on the Primary server. The exception is if a new zone is added on the primary, you need to add it to the secondaries also if you want them to be secondary to the new zone.

source :- http://unixtips.hpage.co.in/dns_8462084.html
 

URLS for UNIX

http://unixtips.hpage.co.in/
http://shtnawi.blogspot.in/2011/09/dns.html

DNS Configuration in Solaris 10

DNS Configuration in Solaris 10
I like to share configuration of DNS primary and secondary in solaris 10. This implementation was done in a solaris 10 X86.Solaris 10 comes with Bind 9.3 version.

Step 1:

Include the name server in a /etc/resolv.conf file



Here domain name is dev.test.com and there are two name servers defined(192.168.0.1 and 192.168.0.2)

step 2:

Change hosts in nsswitch.conf to point to dns.



Here while querying, it will first look in hosts file then it will go for dns server

I am going to include following host name and ip address in DNS. For this we need to create forward and reverse lookup zones.

Sun10 --> 192.168.0.1
sunclient --> 192.168.0.2
sunclient1 --> 192.168.0.3
sunclient2 --> 192.168.0.4
sunclient3 --> 192.168.0.7

Setting up Primary DNS server

In the below setups i am going to create a zone for domain dev.test.com

Step 1:

Configure /etc/named.conf file. If the file is not available areate new named.conf file




In the above file Secret keys are generated using command "rndc-confgen". We need to include this key in primary and secondary servers named.conf file
for secured zone transfer.

Directory is path where zone db files were kept. Here zone db files were kept in /var/named. We are using type as master since this is a primary server
and the db file name is db.dev and this is kept in path /var/named. Here i created both forward and reverse lookup zone


Step 2:

We need to create domain zone file in /var/named. I am going to create three files for the three zones.




Step 3:

Create named.boot file in /var/named



Thats it we are done

Start dns using either /usr/sbin/named or svcadm enable dns/server




Setting up secondary servers

A Secondary DNS Server is basically just a backup server. I does not hold the master versions of the zone information but rather it holds copies of them. Most sites use Secondary servers in remote locations or to cut down on the load on the Primary server. The Secondary server performs zone transfers at said times making sure it has the newest versions of the zone information.

In order to configure secondary server. we need to make few changes in primary servers zone db file.

Include secondary servers name in db file. I have included secondary server name sun10 in db.dev file, db.192.168.0



Here NS sun10 is secondary server. The same should be included in other zone file db.192.168.0.

create /etc/named.conf file in secodary server



Here i have included type as slave as it is secondary server. we are done now

Because this is a Secondary Server, there is no need to do anything with zone files. BONUS! As long as you have everything setup right on your Primary and your named.conf file is configured properly, everything will work properly.

Each time the Primary site's zone files are modified and the Serial # is incremented, BIND will send out a notify to all Secondaries (any server in the zone file with a IN NS statement) stating a change has been made. BIND on each of these servers will then check it's own zone files to see if it has the same version or not. If the version that the Primary has notified it about is newer then it will perform a zone transfer and obtain the newer version.

If everything is setup properly, you will never need to make any changes to your Secondary server except to upgrade BIND itself. All changes from here on out should be made on the Primary server. The exception is if a new zone is added on the primary, you need to add it to the secondaries also if you want them to be secondary to the new zone.

Thanks,
Rajeev

Solaris: Which process is associated with a socket connection?

Consider the following real world scenario

[...] one process is waiting for a very long time in a send system call. It is sending on a valid fd but the question we have is that, is there any way to find who is on the other end of that fd? We want to know to which process is that message being sent to. [...]

Here is how I proceed in finding the other end of the socket, and the state of the socket connection with Mozilla's Thunderbird mail client in one end of the socket connection:

1. Get the process id of the application
   

   % prstat
      PID USERNAME  SIZE   RSS STATE  PRI NICE      TIME  CPU PROCESS/NLWP      
   22385 mandalik  180M   64M sleep   49    0   0:05:15 0.1% thunderbird-bin/5

   
   
2. Run pfiles on the pid - it prints a list of open files including open sockets (pfiles is the Solaris supported equivalent of lsof utility).
   

   % pfiles 22385
   22385:  /usr/lib/thunderbird/thunderbird-bin -UILocale C -contentLocale C
     Current rlimit: 512 file descriptors
     ...
     ...
   
     33: S_IFSOCK mode:0666 dev:280,0 ino:31544 uid:0 gid:0 size:0
         O_RDWR|O_NONBLOCK
           SOCK_STREAM
           SO_SNDBUF(49152),SO_RCVBUF(49640)
           sockname: AF_INET 192.168.1.2  port: 60364
           peername: AF_INET 192.18.39.10  port: 993
           ...
           ...

3. Locate the socket id and the corresponding sockname/port#, peername/port# in the output of pfiles pid (see step #2).
   
   Here my assumption is that I know the socket id I'm interested in. In the above output, 33 is the socket id. One end of the socket is bound to port 60364 on the local host 192.168.1.2; and the other end of the socket is bound to port 993 on the remote host 192.18.39.10.
   
   
4. Run netstat -a | egrep "| (get the port numbers from step 3); and check the state of the socket connection. If you see anything other than ESTABLISHED, it indicates trouble.
   

   % netstat -a | egrep "60364|993"
   solaris-devx-iprb0.60364 mail-sfbay.sun.com.993 48460      0 49640      0 ESTABLISHED

   
   If you want to see the host names in numbers (IP addresses), run netstat with option -n.
   

   %  netstat -an | egrep "60364|993"
   192.168.1.2.60364    192.18.39.10.993     49559      0 49640      0 ESTABLISHED

   
   Now since we know both ends of the socket, we can easily get the state of the socket connection at the other end by running netstat -an | egrep '|.
   
   If the state of the socket connection is CLOSE_WAIT, have a look at the following diagnosis: CPU hog with connections in CLOSE_WAIT.

Finally to answer ... which process is that message being sent to ... part of the original question:

Follow the above steps and find the remote host (or IP) and remote port number. To find the corresponding process id on the remote machine to which the other half of the socket belongs to, do the following:

1. Login as root user on the remote host.
   
2. cd /proc
3. Run pfiles * | egrep "^[0-9]|sockname" > /tmp/pfiles.txt.
   
4. vi /tmp/pfiles.txt and search for the port number. If you scroll few lines up, you can see the process ID, name of the process along with its argument(s).

http://www.feep.net/sendmail/tutorial/intro/MUA-MTA-MDA.html

MUAs, MTAs, and MDAs There are three main parts of the Unix E-mail system: Part What It Does Examples Mail User Agent (MUA) The MUA is the program which the user uses to read and send e-mail. It reads incoming messages that have been delivered to the user's mailbox, and passes outgoing messages to an MTA for sending. elm, pine, mutt Mail Transfer Agent (MTA) The MTA basicly acts as a "mail router". It accepts a message passed to it by either an MUA or another MTA, decides based upon the message header which delivery method it should use, and then passes the message to the appropriate MDA for that delivery method. sendmail, postfix, qmail Mail Delivery Agent (MDA) The MDA accepts a piece of mail from an MTA and performs the actual delivery. mail.local, procmail The MTA is the most important one of these. It is responsible for doing all the "intelligent" work of e-mail transfer. While it does not actually perform any of the delivery itself, it is the part which tells the other parts how to interact and what to do. In a sense, the MTA is the glue which holds the whole process together. To illustrate how the three parts of the email system work together, here's a very general example. This is what happens when the user jsmith@host1.uiuc.edu sends e-mail to johndoe@host2.uiuc.edu: jsmith's MUA (pine, elm, etc) on host1.uiuc.edu passes the message to the MTA (sendmail) on the local host. The MTA (sendmail) notices that the message is addressed to a user at host2.uiuc.edu. Since it is configured to know that it can reach host2.uiuc.edu via SMTP, it passes the message to the SMTP MDA (the SMTP MDA is actually builtin to sendmail, but all other MDAs are external programs). The SMTP MDA connects to the MTA on host2.uiuc.edu (sendmail) and sends it the message. The MTA on host2.uiuc.edu (sendmail) notices that the message is addressed to a user on the local host, so it passes the message to the local MDA. The local MDA saves the message in user johndoe's mailbox. The next time johndoe logs in to host2.uiuc.edu and runs his MUA, the message is there waiting for him to read. As you can see, none of this would be possible without the MTA!

Breaking RCS locks.

Breaking RCS locks. So you want to edit some awesome file, only to see this when you check it out: co -l /mnt/yp/netgroup /mnt/yp/RCS/netgroup,v --> /mnt/yp/netgroup co: /mnt/yp/RCS/netgroup,v: Revision 1.1008 is already locked by u0064824. Blood and ashes! Here's how to break the lock so the world can move forward. First move the file to a backup location: mv /mnt/yp/netgroup /mnt/yp/netgroup-lockedByu0064824-Sep272002 Then break the lock rcs -u /mnt/yp/netgroup RCS file: RCS/netgroup,v Revision 1.1008 is already locked by u0064824. Do you want to break the lock? [ny](n): y State the reason for breaking the lock: (terminate with single '.' or end of file) >> Testing lock breaking -SH. >> . 1.1008 unlocked done Now you can check out the file and using diff (or vimdiff) you can see what changes the previous user made so you can hopefully merge in any changes they might have made. co -l /mnt/yp/netgroup diff /mnt/yp/netgroup /mnt/yp/netgroup-lockedByu0064824-Sep272002 Then edit away! When you're done we check the file in such that it stays in place and isn't magicly whisked away in the RCS repository: ci -u /mnt/yp/netgroup

VMware Player vmnetcfg

VMware Player vmnetcfg


Recently I was using VMware Player 4 on my laptop but I was not able to get an IP address with Bridged Networking selected. Being a user of VMware products for many years, I knew there was a quick fix.

I navigated to the VMware Player directory to run vmnetcfg but it was not there. Unfortunately, the new version of VMware Player doesn’t include vmnetcfg in the installer anymore. Lucy for all of us, it is included in the installer.

To extract vmnetcfg.exe from the installer do the following:


1. Run the installer with /e option. For example:
VMware-player-4.0.0-197124.exe /e .\extract
Contents will be extracted to “extract” folder.
2. Open “network.cab” and copy vmnetcfg.exe to your installation folder,
typically “C:\Program Files\VMware\VMware Player\”.

Now you can run vmnetcfg to exclude network adapters from binding.

How Rsync Works

A Practical Overview
Foreword
The original Rsync technical report and Andrew Tridgell's Phd thesis (pdf) Are both excellent documents for understanding the theoretical mathematics and some of the mechanics of the rsync algorithm. Unfortunately they are more about the theory than the implementation of the rsync utility (hereafter referred to as Rsync).

In this document I hope to describe...

A non-mathematical overview of the rsync algorithm.
How that algorithm is implemented in the rsync utility.
The protocol, in general terms, used by the rsync utility.
The identifiable roles the rsync processes play.
This document be able to serve as a guide for programmers needing something of an entré into the source code but the primary purpose is to give the reader a foundation from which he may understand

Why rsync behaves as it does.
The limitations of rsync.
Why a requested feature is unsuited to the code-base.
This document describes in general terms the construction and behaviour of Rsync. In some cases details and exceptions that would contribute to specific accuracy have been sacrificed for the sake meeting the broader goals.

Processes and Roles
When we talk about Rsync we use specific terms to refer to various processes and their roles in the task performed by the utility. For effective communication it is important that we all be speaking the same language; likewise it is important that we mean the same things when we use certain terms in a given context. On the rsync mailing list there is often some confusion with regards to role and processes. For these reasons I will define a few terms used in the role and process contexts that will be used henceforth. client role The client initiates the synchronisation.
server role The remote rsync process or system to which the client connects either within a local transfer, via a remote shell or via a network socket.
This is a general term and should not be confused with the daemon.

Once the connection between the client and server is established the distinction between them is superseded by the sender and receiver roles.
daemon Role and process An Rsync process that awaits connections from clients. On a certain platform this would be called a service.
remote shell role and set of processes One or more processes that provide connectivity between an Rsync client and an Rsync server on a remote system.
sender role and process The Rsync process that has access to the source files being synchronised.
receiver role and process As a role the receiver is the destination system. As a process the receiver is the process that receives update data and writes it to disk.
generator process The generator process identifies changed files and manages the file level logic.



Process Startup
When an Rsync client is started it will first establish a connection with a server process. This connection may be through pipes or over a network socket.

When Rsync communicates with a remote non-daemon server via a remote shell the startup method is to fork the remote shell which will start an Rsync server on the remote system. Both the Rsync client and server are communicating via pipes through the remote shell. As far as the rsync processes are concerned there is no network. In this mode the rsync options for the server process are passed on the command-line that is used to start the remote shell.

When Rsync is communicating with a daemon it is communicating directly with a network socket. This is the only sort of Rsync communication that could be called network aware. In this mode the rsync options must be sent over the socket, as described below.

At the very start of the communication between the client and the server, they each send the maximum protocol version they support to the other side. Each side then uses the minimum value as the the protocol level for the transfer. If this is a daemon-mode connection, rsync options are sent from the client to the server. Then, the exclude list is transmitted. From this point onward the client-server relationship is relevant only with regards to error and log message delivery.

Local Rsync jobs (when the source and destination are both on locally mounted filesystems) are done exactly like a push. The client, which becomes the sender, forks a server process to fulfill the receiver role. The client/sender and server/receiver communicate with each other over pipes.

The File List
The file list includes not only the pathnames but also ownership, mode, permissions, size and modtime. If the --checksum option has been specified it also includes the file checksums.
The first thing that happens once the startup has completed is that the sender will create the file list. While it is being built, each entry is transmitted to the receiving side in a network-optimised way.

When this is done, each side sorts the file list lexicographically by path relative to the base directory of the transfer. (The exact sorting algorithm varies depending on what protocol version is in effect for the transfer.) Once that has happened all references to files will be done by their index in the file list.

If necessary the sender follows the file list with id→name tables for users and groups which the receiver will use to do a id→name→id translation for every file in the file list.

After the file list has been received by the receiver, it will fork to become the generator and receiver pair completing the pipeline.

The Pipeline
Rsync is heavily pipelined. This means that it is a set of processes that communicate in a (largely) unidirectional way. Once the file list has been shared the pipeline behaves like this:
generator → sender → receiver
The output of the generator is input for the sender and the output of the sender is input for the receiver. Each process runs independently and is delayed only when the pipelines stall or when waiting for disk I/O or CPU resources.

The Generator
The generator process compares the file list with its local directory tree. Prior to beginning its primary function, if --delete has been specified, it will first identify local files not on the sender and delete them on the receiver.

The generator will then start walking the file list. Each file will be checked to see if it can be skipped. In the most common mode of operation files are not skipped if the modification time or size differs. If --checksum was specified a file-level checksum will be created and compared. Directories, device nodes and symlinks are not skipped. Missing directories will be created.

If a file is not to be skipped, any existing version on the receiving side becomes the "basis file" for the transfer, and is used as a data source that will help to eliminate matching data from having to be sent by the sender. To effect this remote matching of data, block checksums are created for the basis file and sent to the sender immediately following the file's index number. An empty block checksum set is sent for new files and if --whole-file was specified.

The block size and, in later versions, the size of the block checksum are calculated on a per file basis according to the size of that file.

The Sender
The sender process reads the file index numbers and associated block checksum sets one at a time from the generator.
For each file id the generator sends it will store the block checksums and build a hash index of them for rapid lookup.

Then the local file is read and a checksum is generated for the block beginning with the first byte of the local file. This block checksum is looked for in the set that was sent by the generator, and if no match is found, the non-matching byte will be appended to the non-matching data and the block starting at the next byte will be compared. This is what is referred to as the “rolling checksum”

If a block checksum match is found it is considered a matching block and any accumulated non-matching data will be sent to the receiver followed by the offset and length in the receiver's file of the matching block and the block checksum generator will be advanced to the next byte after the matching block.

Matching blocks can be identified in this way even if the blocks are reordered or at different offsets. This process is the very heart of the rsync algorithm.

In this way, the sender will give the receiver instructions for how to reconstruct the source file into a new destination file. These instructions detail all the matching data that can be copied from the basis file (if one exists for the transfe), and includes any raw data that was not available locally. At the end of each file's processing a whole-file checksum is sent and the sender proceeds with the next file.

Generating the rolling checksums and searching for matches in the checksum set sent by the generator require a good deal of CPU power. Of all the rsync processes it is the sender that is the most CPU intensive.

The Receiver
The receiver will read from the sender data for each file identified by the file index number. It will open the local file (called the basis) and will create a temporary file.

The receiver will expect to read non-matched data and/or to match records all in sequence for the final file contents. When non-matched data is read it will be written to the temp-file. When a block match record is received the receiver will seek to the block offset in the basis file and copy the block to the temp-file. In this way the temp-file is built from beginning to end.

The file's checksum is generated as the temp-file is built. At the end of the file, this checksum is compared with the file checksum from the sender. If the file checksums do not match the temp-file is deleted. If the file fails once it will be reprocessed in a second phase, and if it fails twice an error is reported.

After the temp-file has been completed, its ownership and permissions and modification time are set. It is then renamed to replace the basis file.

Copying data from the basis file to the temp-file make the receiver the most disk intensive of all the rsync processes. Small files may still be in disk cache mitigating this but for large files the cache may thrash as the generator has moved on to other files and there is further latency caused by the sender. As data is read possibly at random from one file and written to another, if the working set is larger than the disk cache, then what is called a seek storm can occur, further hurting performance.

The Daemon
The daemon process, like many daemons, forks for every connection. On startup, it parses the rsyncd.conf file to determine what modules exist and to set the global options.
When a connection is received for a defined module the daemon forks a new child process to handle the connection. That child process then reads the rsyncd.conf file to set the options for the requested module, which may chroot to the module path and may drop setuid and setgid for the process. After that it will behave just like any other rsync server process adopting either a sender or receiver role.

The Rsync Protocol
A well-designed communications protocol has a number of characteristics.

Everything is sent in well defined packets with a header and an optional body or data payload.
In each packet's header a type and or command specified.
Each packet has a definite length.
In addition to these characteristics, protocols have varying degrees of statefulness, inter-packet independence, human readability, and the ability to reestablish a disconnected session.

Rsync's protocol has none of these good characteristics. The data is transferred as an unbroken stream of bytes. With the exception of the unmatched file-data, there are no length specifiers nor counts. Instead the meaning of each byte is dependent on its context as defined by the protocol level.

As an example, when the sender is sending the file list it simply sends each file list entry and terminates the list with a null byte. Within the file list entries, a bitfield indicates which fields of the structure to expect and those that are variable length strings are simply null terminated. The generator sending file numbers and block checksum sets works the same way.

This method of communication works quite well on reliable connections and it certainly has less data overhead than the formal protocols. It unfortunately makes the protocol extremely difficult to document, debug or extend. Each version of the protocol will have subtle differences on the wire that can only be anticipated by knowing the exact protocol version.

notes
This document is a work in progress. The author expects that it has some glaring oversights and some portions that may be more confusing than enlightening for some readers. It is hoped that this could evolve into a useful reference.
Specific suggestions for improvement are welcome, as would be a complete rewrite.

CDE Login Problem, Remote X11, X-Server, XDMCP login problem

Sometimes I got an error after finishing on Solaris 10 box installation. After make some configuration then suddenly I can’t access my Solaris XDMCP remote session on my laptop.. Usually, I use XManager Enterprise to get Solaris GUI remote session XDMCP. here the step-by-step to troubleshoot if you got the same problem:

{Make sure that svc:/application/graphical-login/cde-login is enabled and online.

root@solaris10 # svcs cde-login
STATE STIME FMRI
online Mar_02 svc:/application/graphical-login/cde-login:default

root@solaris10 #netservices limited

restarting syslogd
restarting sendmail
dtlogin needs to be restarted. Restart now? [Y] y
restarting dtlogin
{Check dtlogin process:

root@solaris10 # ps -ef | grep dtlogin

root 29384 1 0 Mar 02 ? 0:00 /usr/dt/bin/dtlogin -daemon -udpPort 0 [should be TCP, not UDP]

{Modify the x11-server service:

—–>Show properties:
#svcprop svc:/application/x11/x11-server

——>Turn on tcp listen:
#svccfg -s svc:/application/x11/x11-server setprop options/tcp_listen=true

{Modify the dtlogin service:

—–>Show properties:
#svcprop svc:/application/graphical-login/cde-login:default
#svccfg -s svc:/application/graphical-login/cde-login setprop dtlogin/args=\”\”

—–>Then restart the X server:
#svcadm refresh svc:/application/graphical-login/cde-login:default;
#svcprop -p dtlogin svc:/application/graphical-login/cde-login:default

root@solaris10 #netservices open

restarting syslogd
restarting sendmail

root@solaris10# svcadm restart cde-login
root@solaris10# ps -ef |grep dtlogin
root 27722 1 0 15:08:37 ? 0:00 /usr/dt/bin/dtlogin -daemon
root 27724 26297 0 15:08:43 pts/3 0:00 grep dtlogin

tar file size limit

Generally, tar can't handle files larger than 2GB. I suggest using an alternative to tar, 'star'. A more comprehensive answer is available here:


--->
The quick answer is: 2^63-1 for the archive, 68'719'476'735 (8^12-1)
bytes for each file, if your environment permits that.

as I understand your question, you want to know if you can produce tar
files that are biger than 2 GBytes (and how big you can really make
them). The answer to this question depends on a few simple parameters:

1) Does your operating system support large files?
2) What version of tar are you using?
3) What is the underlying file system?

You can answer question 1) for yourself by verifying that your kernel
supports 64bit file descriptors. For Linux this is the case for
several years now. A quick look in /usr/include/sys/features.h will
tell you, if there is any line containing 'FILE_OFFSET_BITS'. If there
is, your OS very very probably has support for large files.

For Solaris, just check whether 'man largefile' works, or try 'getconf
-a|grep LARGEFILE'. If it works, then you have support for large files
in the operating system. Again, support for large files has been there
for several years.

For other operating systems, try "man -k large file', and see what you
get -- I'll be gladly providing help if you need to ask for
clarification to this answer. Something like "cd /usr/include; grep
'FILE_OFFSET_BITS' * */*" should tell you quickly if there is standard
large file support.

2) What version of tar are you using? This is important. Obviously,
older tar programs won't be able to handle files or archives that are
larger than 2^31-1 bytes (2.1 Gigabytes). Try running 'tar --version'.
If the first line indicates you are using gnu tar, then any version
newer than 1.12.64 will in principle be able to provide you with large
files. Try to run this command: "strings `which tar`|grep 64", and you
should see some lines saying lseek64, creat64, fopen64. If yes, your
tar contains support for large files.

If your tar program does not contain support for large files (most
really do, but maybe you are working on a machine older than 1998?),
you can download the newest gnu tar from ftp://ftp.gnu.org/pub/gnu/tar
and compile it for yourself.

The size of files you put into a tar archive (not the archive itself)
is limited to 11 octal digits, the max. size of a single file is thus
ca. 68 GBytes.

3) Given that both your operating system (and C library), and tar
application support large files, the only really limiting factor is
the file system that you try to create the file in. The theoretical
limit for the tar archive size is 2^63-1 (9'223'372 Terabytes), but
you will reach more practical limits (disk or tape size) much quicker.
Also take into consideration what the file system is. DOS FAT 12
filesystems don't allow files as big as the Linux EXT2, or Sun UFS
file systems.

If you need more precise data (for a specific file system type, or for
the OS, etc.) please do not hesitate to ask for clarification.

I hope my answer is helpful to you,
Rajeev

Amazon EC2 Instance Types

Amazon EC2 Instance Types

Amazon Elastic Compute Cloud (Amazon EC2) provides the flexibility to choose from a number of different instance types to meet your computing needs. Each instance provides a predictable amount of dedicated compute capacity and is charged per instance-hour consumed.
Available Instance Types
Standard Instances

Instances of this family are well suited for most applications.

Small Instance – default*

1.7 GB memory
1 EC2 Compute Unit (1 virtual core with 1 EC2 Compute Unit)
160 GB instance storage
32-bit platform
I/O Performance: Moderate
API name: m1.small

Large Instance

7.5 GB memory
4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each)
850 GB instance storage
64-bit platform
I/O Performance: High
API name: m1.large

Extra Large Instance

15 GB memory
8 EC2 Compute Units (4 virtual cores with 2 EC2 Compute Units each)
1,690 GB instance storage
64-bit platform
I/O Performance: High
API name: m1.xlarge
Micro Instances

Instances of this family provide a small amount of consistent CPU resources and allow you to burst CPU capacity when additional cycles are available. They are well suited for lower throughput applications and web sites that consume significant compute cycles periodically.

Micro Instance

613 MB memory
Up to 2 EC2 Compute Units (for short periodic bursts)
EBS storage only
32-bit or 64-bit platform
I/O Performance: Low
API name: t1.micro
High-Memory Instances

Instances of this family offer large memory sizes for high throughput applications, including database and memory caching applications.

High-Memory Extra Large Instance

17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform
I/O Performance: Moderate
API name: m2.xlarge

High-Memory Double Extra Large Instance

34.2 GB of memory
13 EC2 Compute Units (4 virtual cores with 3.25 EC2 Compute Units each)
850 GB of instance storage
64-bit platform
I/O Performance: High
API name: m2.2xlarge

High-Memory Quadruple Extra Large Instance

68.4 GB of memory
26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each)
1690 GB of instance storage
64-bit platform
I/O Performance: High
API name: m2.4xlarge
High-CPU Instances

Instances of this family have proportionally more CPU resources than memory (RAM) and are well suited for compute-intensive applications.

High-CPU Medium Instance

1.7 GB of memory
5 EC2 Compute Units (2 virtual cores with 2.5 EC2 Compute Units each)
350 GB of instance storage
32-bit platform
I/O Performance: Moderate
API name: c1.medium

High-CPU Extra Large Instance

7 GB of memory
20 EC2 Compute Units (8 virtual cores with 2.5 EC2 Compute Units each)
1690 GB of instance storage
64-bit platform
I/O Performance: High
API name: c1.xlarge
Cluster Compute Instances

Instances of this family provide proportionally high CPU resources with increased network performance and are well suited for High Performance Compute (HPC) applications and other demanding network-bound applications. Learn more about use of this instance type for HPC applications.

Cluster Compute Quadruple Extra Large Instance

23 GB of memory
33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core “Nehalem” architecture)
1690 GB of instance storage
64-bit platform
I/O Performance: Very High (10 Gigabit Ethernet)
API name: cc1.4xlarge
Cluster GPU Instances

Instances of this family provide general-purpose graphics processing units (GPUs) with proportionally high CPU and increased network performance for applications benefitting from highly parallelized processing, including HPC, rendering and media processing applications. While Cluster Compute Instances provide the ability to create clusters of instances connected by a low latency, high throughput network, Cluster GPU Instances provide an additional option for applications that can benefit from the efficiency gains of the parallel computing power of GPUs over what can be achieved with traditional processors. Learn more about use of this instance type for HPC applications.

Cluster GPU Quadruple Extra Large Instance

22 GB of memory
33.5 EC2 Compute Units (2 x Intel Xeon X5570, quad-core “Nehalem” architecture)
2 x NVIDIA Tesla “Fermi” M2050 GPUs
1690 GB of instance storage
64-bit platform
I/O Performance: Very High (10 Gigabit Ethernet)
API name: cg1.4xlarge
Selecting Instance Types

Amazon EC2 instances are grouped into six families: Standard, Micro, High-Memory, High-CPU, Cluster Compute, and Cluster GPU.

Standard Instances have memory to CPU ratios suitable for most general purpose applications; High-Memory instances offer larger memory sizes for high throughput applications, including database and memory caching applications; and High-CPU instances have proportionally more CPU resources than memory (RAM) and are well suited for compute-intensive applications.

Micro instances provide a small amount of consistent CPU resources and allow you to burst CPU capacity when additional cycles are available. They are well suited for lower throughput applications and web sites that consume significant compute cycles periodically. At steady state, Micro instances receive a fraction of the compute resources that Small instances do. Therefore, if your application has compute-intensive or steady state needs we recommend using a Small instance (or larger, depending on your needs). However, Micro instances can periodically burst up to 2 ECUs (for short periods of time). This is double the number of ECUs available from a Standard Small instance. Therefore, if you have a relatively low throughput application or web site with an occasional need to consume significant compute cycles, we recommend using Micro instances.

Cluster Compute instances provide a very large amount of CPU coupled with increased network performance making them well suited for High Performance Compute (HPC) applications and other demanding network-bound applications.

Cluster GPU instances provide general-purpose graphics processing units (GPUs) with proportionally high CPU and increased network performance making them well suited for applications benefitting from highly parallelized processing, including HPC, rendering and media processing applications.

When choosing instance types, you should consider the characteristics of your application with regards to resource utilization and select the optimal instance family and size. One of the advantages of EC2 is that you pay by the instance hour, which makes it convenient and inexpensive to test the performance of your application on different instance families and types. One good way to determine the most appropriate instance family and instance type is to launch test instances and benchmark your application.
Measuring Compute Resources

Transitioning to a utility computing model fundamentally changes how developers have been trained to think about CPU resources. Instead of purchasing or leasing a particular processor to use for several months or years, you are renting capacity by the hour. Because Amazon EC2 is built on commodity hardware, over time there may be several different types of physical hardware underlying EC2 instances. Our goal is to provide a consistent amount of CPU capacity no matter what the actual underlying hardware.

Amazon EC2 uses a variety of measures to provide each instance with a consistent and predictable amount of CPU capacity. In order to make it easy for developers to compare CPU capacity between different instance types, we have defined an Amazon EC2 Compute Unit. The amount of CPU that is allocated to a particular instance is expressed in terms of these EC2 Compute Units. We use several benchmarks and tests to manage the consistency and predictability of the performance of an EC2 Compute Unit. One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. This is also the equivalent to an early-2006 1.7 GHz Xeon processor referenced in our original documentation. Over time, we may add or substitute measures that go into the definition of an EC2 Compute Unit, if we find metrics that will give you a clearer picture of compute capacity.

To find out which instance will work best for your application, the best thing to do is to launch an instance and benchmark your own application. One of the advantages of EC2 is that you pay by the hour which makes it convenient and inexpensive to test the performance of your application on different instance types.
I/O Performance

Amazon EC2 provides virtualized server instances. While some resources like CPU, memory and instance storage are dedicated to a particular instance, other resources like the network and the disk subsystem are shared among instances. If each instance on a physical host tries to use as much of one of these shared resources as possible, each will receive an equal share of that resource. However, when a resource is under-utilized you will often be able to consume a higher share of that resource while it is available.

The different instance types will provide higher or lower minimum performance from the shared resources depending on their size. Each of the instance types has an I/O performance indicator (low, moderate, or high). Instance types with high I/O performance have a larger allocation of shared resources. Allocating larger share of shared resources also reduces the variance of I/O performance. For many applications, low or moderate I/O performance is more than enough. However, for those applications requiring greater or more consistent I/O performance, you may want to consider instances with high I/O performance.

Cluster Compute and Cluster GPU instances have very high I/O performance using 10 Gigabit Ethernet for high network throughput and reduced network latencies within clusters.

In addition, you can use Amazon EBS to get improved storage I/O performance for disk bound applications.



*The Small Instance type is equivalent to the original Amazon EC2 instance type that has been available since Amazon EC2 launched. This instance type is currently the default for all customers. To use other instance types, customers must specifically request them via the RunInstances API.

How to disable SSH host key checking

Remote login using the SSH protocol is a frequent activity in today's internet world. With the SSH protocol, the onus is on the SSH client to verify the identity of the host to which it is connecting. The host identify is established by its SSH host key. Typically, the host key is auto-created during initial SSH installation setup.

By default, the SSH client verifies the host key against a local file containing known, rustworthy machines. This provides protection against possible Man-In-The-Middle attacks. However, there are situations in which you want to bypass this verification step. This article explains how to disable host key checking using OpenSSH, a popular Free and Open-Source implementation of SSH.

When you login to a remote host for the first time, the remote host's host key is most likely unknown to the SSH client. The default behavior is to ask the user to confirm the fingerprint of the host key.

$ ssh Rajeev@192.168.0.100
The authenticity of host '192.168.0.100 (192.168.0.100)' can't be established.
RSA key fingerprint is 3f:1b:f4:bd:c5:aa:c1:1f:bf:4e:2e:cf:53:fa:d8:59.
Are you sure you want to continue connecting (yes/no)?


If your answer is yes, the SSH client continues login, and stores the host key locally in the file ~/.ssh/known_hosts. You only need to validate the host key the first time around: in subsequent logins, you will not be prompted to confirm it again.

Yet, from time to time, when you try to remote login to the same host from the same origin, you may be refused with the following warning message:

$ ssh Rajeev@192.168.0.100
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
3f:1b:f4:bd:c5:aa:c1:1f:bf:4e:2e:cf:53:fa:d8:59.
Please contact your system administrator.
Add correct host key in /home/peter/.ssh/known_hosts to get rid of this message.
Offending key in /home/rajeev/.ssh/known_hosts:3
RSA host key for 192.168.0.100 has changed and you have requested strict checking.
Host key verification failed.$


There are multiple possible reasons why the remote host key changed. A Man-in-the-Middle attack is only one possible reason. Other possible reasons include:

* OpenSSH was re-installed on the remote host but, for whatever reason, the original host key was not restored.
* The remote host was replaced legitimately by another machine.


If you are sure that this is harmless, you can use either 1 of 2 methods below to trick openSSH to let you login. But be warned that you have become vulnerable to man-in-the-middle attacks.

The first method is to remove the remote host from the ~/.ssh/known_hosts file. Note that the warning message already tells you the line number in the known_hosts file that corresponds to the target remote host. The offending line in the above example is line 3("Offending key in /home/rajeev/.ssh/known_hosts:3")

You can use the following one liner to remove that one line (line 3) from the file.

$ sed -i 3d ~/.ssh/known_hosts


Note that with the above method, you will be prompted to confirm the host key fingerprint when you run ssh to login.

The second method uses two openSSH parameters:

* StrictHostKeyCheckin, and

* UserKnownHostsFile.


This method tricks SSH by configuring it to use an empty known_hosts file, and NOT to ask you to confirm the remote host identity key.

$ ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no rajeev@192.168.0.100
Warning: Permanently added '192.168.0.100' (RSA) to the list of known hosts.
peter@192.168.0.100's password:


The UserKnownHostsFile parameter specifies the database file to use for storing the user host keys (default is ~/.ssh/known_hosts).

The /dev/null file is a special system device file that discards anything and everything written to it, and when used as the input file, returns End Of File immediately.

By configuring the null device file as the host key database, SSH is fooled into thinking that the SSH client has never connected to any SSH server before, and so will never run into a mismatched host key.

The parameter StrictHostKeyChecking specifies if SSH will automatically add new host keys to the host key database file. By setting it to no, the host key is automatically added, without user confirmation, for all first-time connection. Because of the null key database file, all connection is viewed as the first-time for any SSH server host. Therefore, the host key is automatically added to the host key database with no user confirmation. Writing the key to the /dev/null file discards the key and reports success.

Please refer to this excellent article about host keys and key checking.

By specifying the above 2 SSH options on the command line, you can bypass host key checking for that particular SSH login. If you want to bypass host key checking on a permanent basis, you need to specify those same options in the SSH configuration file.

You can edit the global SSH configuration file (/etc/ssh/ssh_config) if you want to make the changes permanent for all users.

If you want to target a particular user, modify the user-specific SSH configuration file (~/.ssh/config). The instructions below apply to both files.

Suppose you want to bypass key checking for a particular subnet (192.168.0.0/24).

Add the following lines to the beginning of the SSH configuration file.

Host 192.168.0.*
StrictHostKeyChecking no
UserKnownHostsFile=/dev/null


Note that the configuration file should have a line like Host * followed by one or more parameter-value pairs. Host *means that it will match any host. Essentially, the parameters following Host * are the general defaults. Because the first matched value for each SSH parameter is used, you want to add the host-specific or subnet-specific parameters to the beginning of the file.

As a final word of caution, unless you know what you are doing, it is probably best to bypass key checking on a case by case basis, rather than making blanket permanent changes to the SSH configuration files.

netapp command summary

sysconfig -a : shows hardware configuration with more verbose information
sysconfig -d : shows information of the disk attached to the filer
version : shows the netapp Ontap OS version.
uptime : shows the filer uptime
dns info : this shows the dns resolvers, the no of hits and misses and other info
nis info : this shows the nis domain name, yp servers etc.
rdfile : Like "cat" in Linux, used to read contents of text files/
wrfile : Creates/Overwrites a file. Similar to "cat > filename" in Linux
aggr status : Shows the aggregate status
aggr status -r : Shows the raid configuration, reconstruction information of the disks in filer
aggr show_space : Shows the disk usage of the aggreate, WAFL reserve, overheads etc.
vol status : Shows the volume information
vol status -s : Displays the spare disks on the filer
vol status -f : Displays the failed disks on the filer
vol status -r : Shows the raid configuration, reconstruction information of the disks
df -h : Displays volume disk usage
df -i : Shows the inode counts of all the volumes
df -Ah : Shows "df" information of the aggregate
license : Displays/add/removes license on a netapp filer
maxfiles : Displays and adds more inodes to a volume
aggr create : Creates aggregate
vol create : Creates volume in an aggregate
vol offline : Offlines a volume
vol online : Onlines a volume
vol destroy : Destroys and removes an volume
vol size [+|-] : Resize a volume in netapp filer
vol options : Displays/Changes volume options in a netapp filer
qtree create : Creates qtree
qtree status : Displays the status of qtrees
quota on : Enables quota on a netapp filer
quota off : Disables quota
quota resize : Resizes quota
quota report : Reports the quota and usage
snap list : Displays all snapshots on a volume
snap create : Create snapshot
snap sched : Schedule snapshot creation
snap reserve : Display/set snapshot reserve space in volume
/etc/exports : File that manages the NFS exports
rdfile /etc/exports : Read the NFS exports file
wrfile /etc/exports : Write to NFS exports file
exportfs -a : Exports all the filesystems listed in /etc/exports
cifs setup : Setup cifs
cifs shares : Create/displays cifs shares
cifs access : Changes access of cifs shares
lun create : Creates iscsi or fcp luns on a netapp filer
lun map : Maps lun to an igroup
lun show : Show all the luns on a filer
igroup create : Creates netapp igroup
lun stats : Show lun I/O statistics
disk show : Shows all the disk on the filer
disk zero spares : Zeros the spare disks
disk_fw_update : Upgrades the disk firmware on all disks
options : Display/Set options on netapp filer
options nfs : Display/Set NFS options
options timed : Display/Set NTP options on netapp.
options autosupport : Display/Set autosupport options
options cifs : Display/Set cifs options
options tcp : Display/Set TCP options
options net : Display/Set network options
ndmpcopy : Initiates ndmpcopy
ndmpd status : Displays status of ndmpd
ndmpd killall : Terminates all the ndmpd processes.
ifconfig : Displays/Sets IP address on a network/vif interface
vif create : Creates a VIF (bonding/trunking/teaming)
vif status : Displays status of a vif
netstat : Displays network statistics
sysstat -us 1 : begins a 1 second sample of the filer's current utilization (crtl - c to end)
nfsstat : Shows nfs statistics
nfsstat -l : Displays nfs stats per client
nfs_hist : Displays nfs historgram
statit : beings/ends a performance workload sampling [-b starts / -e ends]
stats : Displays stats for every counter on netapp. Read stats man page for more info
ifstat : Displays Network interface stats
qtree stats : displays I/O stats of qtree
environment : display environment status on shelves and chassis of the filer
storage show : Shows storage component details
snapmirror intialize : Initialize a snapmirror relation
snapmirror update : Manually Update snapmirror relation
snapmirror resync : Resyns a broken snapmirror
snapmirror quiesce : Quiesces a snapmirror bond
snapmirror break : Breakes a snapmirror relation
snapmirror abort : Abort a running snapmirror
snapmirror status : Shows snapmirror status
lock status -h : Displays locks held by filer
sm_mon : Manage the locks
storage download shelf : Installs the shelf firmware
software get : Download the Netapp OS software
software install : Installs OS
download : Updates the installed OS
cf status : Displays cluster status
cf takeover : Takes over the cluster partner
cf giveback : Gives back control to the cluster partner
reboot : Reboots a filer