How to install VMware tools on linux based vm

vSphere needs VMware tools to detect the IP of the vm. Following are the steps to install the VMware tools:

# To see the installed version 

vmware-toolbox-cmd -v

                             

# To see the currently installed packages

sudo apt --installed list | grep tools

# Steps to remove open-vm-tools

sudo apt-get remove --purge open-vm-tools sudo apt-get -f install sudo apt-get autoremove

# Steps to add vmware vm-tools 

sudo add-apt-repository "deb http://packages.vmware.com/tools/esx/6.0u1/ubuntu precise main" sudo apt-get update sudo apt-get install vmware-tools-core sudo apt-get install vmware-tools-services sudo apt-get install vmware-tools-esx-nox

# See the latest installed packages  

sudo apt --installed list | grep tools

           

# Verify latest version is installed

vmware-toolbox-cmd -v

Bash script to monitor uptime, cpu, memory, swap memory, load average of linux system

#!/bin/bash

# Print system resources
echo "HOSTNAME: "`hostname`
echo "SYSTEM: "`uname -a`
echo "IP ADDRESS: "; ifconfig eth0
echo "ROUTE TABLE:"; route -n
echo ""

# Start monitoring the system resources
while true
do
    echo ------ `date` ------
    top -b n1 | head -n 6
    sleep 60
done

Python code to read and write array to file in json format

First install and import 'json' module

import json

Reading JSON object from file:

def read_json_from_file(json_file):
    """
    Read array of objects from a file in json format.
    :param json_file: input json file name
    :return: array of objects from the json
    """
    with open(json_file, 'r') as f:
        exp_string = json.load(f)
        json_array = json.loads(exp_string)
    return json_array

Writing JSON to the file:

def write_json_to_file(array, out_file_name):
    """
    Dump an array to file in json format.
    :param array: Array to write to the file
    :param out_file_name: output file name
    """
    with open(out_file_name, 'w') as out_f:
        s = json.dumps([o.__dict__ for o in array], sort_keys=True)
        json.dump(s, out_f)

Get recursive file listing in python

Below is the python code:

def get_recursive_filelist(rootdir):
    outfiles = []
    if os.path.isfile(rootdir):
        outfiles.append(rootdir)
    for (dir, _, files) in os.walk(rootdir):
        for f in files:
            path = os.path.join(dir, f)
            if os.path.exists(path):
                outfiles.append(path)
    return outfiles

Easy way to check internet speed using command line speedtest tool

Here is the light weight command line speedtest tool to check the internet speed.

$ wget -O speedtest-cli https://raw.github.com/sivel/speedtest-cli/master/speedtest_cli.py
$ chmod +x speedtest-cli
$ ./speedtest-cli
Retrieving speedtest.net configuration...
Retrieving speedtest.net server list...
Testing from Comcast Cable (x.x.x.x)...
Selecting best server based on ping...
Hosted by FiberCloud, Inc (Seattle, WA) [12.03 km]: 44.028 ms
Testing download speed........................................
Download: 32.29 Mbit/s
Testing upload speed..................................................
Upload: 5.18 Mbit/s

Easy way to parse command line arguments in python using argparse

Here is the easiest way to parse command line arguments in python.

import argparse

args = argparse.ArgumentParser(description='Description of the script.')
args.add_argument('-i', '--input', help='Input file name', required=True)
args.add_argument('-o', '--output', help='Output file', required=True)
args = args.parse_args()

input = args.input
output = args.output

print "Input:" + input
print "Ouput:" + output

Output:

Rahuls-MacBook-Pro:exception rahuldiyewar$ python test.py -h
usage: test.py [-h] -i INPUT -o OUTPUT

Description of the script.

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT, --input INPUT
                        Input file name
  -o OUTPUT, --output OUTPUT
                        Output file

Rahuls-MacBook-Pro:exception rahuldiyewar$ python test.py -i myinput -o myoutput 
Input:myinput 
Ouput:myoutput

Python code to search and create bug in JIRA using rest api

Here is the Python code to search/open issues in JIRA:

1. Search for open issues
2. Create new issue

from jira import JIRA


class JIRARestAPI:
    url = 'https://<company-name>.atlassian.net'    
    username = '<username>'    
    password = '<password>'    
    jira = JIRA(url, basic_auth=(username, password))

    def search_issues(self, search_string):
        return JIRARestAPI.jira.search_issues(search_string)

    def print_bugs(self, bugs):
        for bug in bugs:
            self.print_bug(bug)

    def print_bug(self, bug):
        print ("%s, %s, \"%s\"" % (bug.key, bug.fields.status.name, bug.fields.summary))

    def create_cyg_bug(self, summary, description, label):
        issue_dict = {
            'project': {'id': 10000},
            'summary': summary,
            'description': description,
            'issuetype': {'name': 'Bug'},
            'labels': [label]
        }
        return JIRARestAPI.jira.create_issue(fields=issue_dict)

Per thread CPU usage on Linux

To investigate the per-thread CPU usage on Linux, use command ‘top’ with the -H option, which provides an additional per thread information, which is not provided by default ‘top’ usage.

The output of ‘top -H’ on Linux shows the breakdown of the CPU usage on the machine by individual threads. The top output has the following sections of interest:

 top - 16:15:45 up 21 days,  2:27,  3 users,  load   average: 17.94, 12.30, 5.52   
 Tasks: 150 total,  26 running, 124 sleeping,   0   stopped,   0 zombie     Cpu(s): 87.3% us,  1.2% sy,  0.0% ni, 27.6% id,  0.0%   wa,  0.0% hi,  0.0% si     Mem:   4039848k total,  3999776k used,   40072k free,    92824k buffers     Swap:  2097144k total,      224k used, 2096920k free,  1131652k cached   

The Cpus(s) row in this header section shows the CPU usage in terms of the following:

us - Percentage of CPU time spent in user space.

sy - Percentage of CPU time spent in kernel space.

ni - Percentage of CPU time spent on low priority processes.

id - Percentage of CPU time spent idle.

wa - Percentage of CPU time spent in wait (on disk).

hi - Percentage of CPU time spent handling hardware interrupts.

si - Percentage of CPU time spent handling software interrupts.

The "us", "sy" and "id" values are useful as the user, system (kernel) and idle CPU time respectively.

The next section shows the per-thread breakdown of the CPU usage.

PID USER    PR  NI  VIRT  RES  SHR S %CPU   %MEM    TIME+  COMMAND   
 31253 user1   16   0 2112m 2.1g 1764 R 37.0   53.2   0:39.89 java   
 31249 user1   16   0 2112m 2.1g 1764 R 15.5   53.2   0:38.29 java   
 31244 user1   16   0 2112m 2.1g 1764 R 13.6   53.2   0:40.05 java   
 31250 user1   16   0 2112m 2.1g 1764 R 13.6   53.2   0:41.23 java   
 31242 user1   16   0 2112m 2.1g 1764 R 12.9   53.2   0:40.56 java   
 31238 user1   16   0 2112m 2.1g 1764 S 12.6   53.2   1:22.21 java   
 31246 user1   16   0 2112m 2.1g 1764 R 12.6   53.2   0:39.62 java   
 31248 user1   16   0 2112m 2.1g 1764 R 12.6   53.2   0:39.40 java   
 31258 user1   16   0 2112m 2.1g 1764 R 12.6   53.2   0:39.98 java   
 31264 user1   17   0 2112m 2.1g 1764 R 12.6   53.2   0:39.54 java   
 31243 user1   16   0 2112m 2.1g 1764 R 12.2   53.2   0:37.43 java   
 31245 user1   16   0 2112m 2.1g 1764 R 12.2   53.2   0:37.53 java     ...

This provides the following information per thread basis:

PID - The thread ID. This can be converted into hexadecimal and used to correlate to the "native ID" in a java thread dump file.

USER - The user ID of the user that started the process.

PR - The priority of the thread.

NI - The "nice" value for the process.

VIRT - The virtual memory (allocated) usage of the process.

RES - The resident memory (committed) usage of the process.

SHR - The shared memory usage of the process.

S - The state of the thread. This can be one of the following:

R - Running

S - Sleeping

D - Uninterruptible sleep

T - Traced

Z - Zombie

%CPU - The percentage of a single CPU usage by the thread.

%MEM - The percentage of the memory used by the process.

TIME+ - The amount of CPU time used by the thread.

COMMAND - The name of the process executable.

Note that the "Cpu(s)" line in the header of the output shows the percentage usage across all of the available CPUs, whereas the %CPU column above represents the percentage usage of a single CPU. Thus for example, on a four-CPU machine the Cpu(s) row will total 100% and the %CPU column will total to 400%. To see the per cpu usage in header section, press 1.

What to look for from the top -H output?

In the per-thread breakdown of the CPU usage shown above, the Java process is taking approximately 75% of the CPU usage. This value is found by totaling the %CPU column for all the Java threads (not all threads are shown above) and dividing by the number of CPUs. The Java process is not limited by other processes, because the CPU there is still approximately 25% idle.

You can also see that the CPU usage of the Java process is spread reasonably evenly over all of the threads in the Java process. This spread implies that no one thread has a particular problem. Although the application is allowed to use most of the available CPU, approximately 25% of the total CPU is idle meaning that some points of contention or delay in the Java process can be identified.

A report indicating that active processes are using a small percentage of CPU, even though the machine appears idle, means that the performance of the application is probably limited by points of contention or process delay, preventing the application from scaling to use all of the available CPU.

If a deadlock is present, the reported CPU usage for the Java process is low or zero.

If threads are looping, the Java CPU usage approaches 100%, but a small number of the threads account for all of that CPU time.

Whenever you have threads of interest, note the PID values, convert them to a hexadecimal value and look up the threads in thread dump file to discover the name of application thread. Then look at the thread stack trace to understand the kind of work it is doing.

How to generate thread dump for Java process?

The thread dump for a java process can be generated using command:

/bin/kill -3 <pid>

“Too many open files” error in highly multithreaded application

Error:

Caused by: java.io.FileNotFoundException: errors.txt (Too many open files)

    at java.io.FileOutputStream.openAppend(Native Method)

    at java.io.FileOutputStream.<init>(FileOutputStream.java:192)

    at java.io.FileOutputStream.<init>(FileOutputStream.java:116)

    at hydra.FileUtil.appendToFile(FileUtil.java:528)

    ... 5 more

If we see “Too many open files” error in highly multi- threaded application, this means we have reached to maximum number of file handles.

To see where you are at with file handles on Linux:

Run the command

sysctl fs.file-nr

From this command you will get 3 numbers. First is the number of used file descriptor the second is the number of allocated but not used file descriptor and the last is the system max file descriptor.

To up this limit:
You will need to add something similar to the following in the file /etc/sysctl.conf:

fs.file-max = 204708

Also there is an individual users file handle limit, that is where the "ulimit -n" command. If you are running to this limit you will have to add entries to the /etc/security/limits.conf file for the user like the following:

@users hard nofile 81920
@users soft nofile 8192
@users hard nproc unlimited
@users soft nproc 501408
@users soft core unlimited
@users hard core unlimited

Apply the changes using :

sysctl -a

OutOfMemoryError because of default stack size

Error String:

       java.lang.OutOfMemoryError: unable to create new native thread
              at java.lang.Thread.start0(Native Method)
              at java.lang.Thread.start(Thread.java:597)

The application dies with “java.lang.OutOfMemoryError: unable to create new native thread.” The solution is to reduce the stack size. The JVM has an interesting implementation, the design of which I don’t completely understand, but the implication is that the more memory is allocated for the heap (not necessarily used by the heap), the less memory available in the stack, and since threads are made from the stack, in practice this means more “memory” in the heap sense (which is usually what people talk about) results in less threads being able to run concurrently.

Each thread by default gets a stack of 1MB. In a highly multithreaded system (~1500 threads), it is possible that we run out of contiguous blocks of memory on that machine. The limitation is normally stack space (which must be in contiguous blocks) and since every thread consumes this scattered about you rapidly run out of contiguous blocks. Reference:http://stackoverflow.com/questions/481900/whats-the-maximum-number-of-threads-in-windows-server-2003

To reduce the stack size, add “-Xss64kb” to the JVM options. Start with 64k, try the application, then if it doesn’t work (it will fail with a Java.lang.StackOverFlowError), increase the stack to 128k, then 256k, and so on. The default stack size is 8192k so there’s a wide range to test.
Also, on Linux, you’ll need to set the Linux thread stack size to the same value as the JVM stack size to get full benefits. To do that, use “ulimit -s <size in kb>”. Note that the stack size applies per user, so you have to modify the init script, or edit /etc/security/limits.conf (on Debian/Ubuntu at least).
 
GemFire is a highly multi-threaded system and at any given point in time there are multiple thread pools and threads that are in use. The default stack size setting for a thread in java is 1MB. Stack size has to be allocated in contiguous blocks and if the machine is being used actively and there are many threads running in the system (Task Manager shows the number of active threads), you may encounter OutOfMemoryError: unable to create new native thread , even though your process has enough available heap. If this happens, consider reducing the stack size requirement for threads on the cache server. The stack size setting for cache servers can be set using the -Xss vm arg. This can be set to 384k or 512k in such cases