Ravi Sharda's Weblog [BETA]

Bredemeyer's Software Architecture Links

2011-12-18T10:48:00.000+05:30

A comprehensive set of links on software architecture: http://www.bredemeyer.com/links.htm.

Design approaches

2011-09-04T18:03:00.000+05:30

Does you head spin when someone talks about design approaches such as responsibility driven design (RDD), behavior driven design (BDD), domain driven design (DDD) or model driven design (MDD)? Here is Rebecca J. Wirfs-Brock's article to the rescue.

Preserving architectural design in code

2011-06-23T09:06:00.000+05:30

Eoin Woods advices on writing code that preserves the initial architectural design using conventions, dependency analysis, module systems, augmenting the code & checking rules, and language extensions.: InfoQ: Where Did My Architecture Go?

Amazon Dynamo and its design

2011-05-29T15:57:00.004+05:30

Here is a seminal paper released by Amazon describing Dynamo, Amazon's proprietary highly available key-value based data store used for it's own data storage needs.

Many philosphies described in this paper coincide with those of other NoSql data stores. For example, consistent hashing for determining which partition to store data, hinted handoffs to ensure read/write operations do not get affected when nodes or parts of network go down temporarily, and so on. As such, this paper is a great reading material for anyone wanting to understand some of the key design considerations regarding NoSql DBs.

Understanding Weak references in Java

2011-01-07T14:20:00.000+05:30

Here's a nice article from Ethan Nicholas on Weak References - a topic that eludes many Java developers.

A list of refactorings

2010-11-26T22:48:00.000+05:30

Developers often encounter a need for refactoring code that has existed for a long time. This is because software tends to get complex and unmaintainable over time, unless time and effort is spent on preserving the modular structure of the software and its simplicity in the face of evolving needs.

Martin Fowler defines refactoring as "... a disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior". [Src]

A list of "refactorings" both from the original book by Fowler and some later sources, is available here.

jQuery 101 for folks on the run

2010-11-15T10:33:00.001+05:30

Here is Rebecca Murphy's article on jQuery fundamentals. It helped me learn enough jQuery in a couple of days to do/understand small to medium sized jQuery based UIs.

Sreekanth, my good friend, has published a comparision between jQuery and GWT here.

Rebecca Wirfs-Brock's columns and articles on software design

2010-10-15T18:01:00.003+05:30

Rebecca Wirfs-Brock's IEEE software "design" columns (published between Jan. 2006 and Dec. 2009) and several other articles on software design, can be found here. These are must reads for software designers/architects.

Floating Point Basics Made Easy

2010-09-29T14:09:00.003+05:30

1) Introduction

Almost every processor, computer, programming language and operating system supports floating-points. Despite its ubiquity, most programmers consider it an esotoric topic. James Gosling (the "father" of Java) once asserted "95% of folks out there are completely clueless about floating-point."

The vast majority of applications/programs can do just with integral/integer types. There are times when the need to use non-integral types and the vagaries of representing those, show up. High-level programming languages such as Java, C#, C++ all make it very easy to use floating-points. However, seemingly innocuous floating-point code can cause severe discrepancies and very hard to debug runtime errors. Therefore, understanding how it works under the hood helps!

The basic ideas behind floating point are not very difficult to understand. This article intends to introduce the basics to the uninitiated. Folks interested in a deeper treatment of floating-point are referred to David Goldberg's article "What Every Computer Scientist Should Know About Floating-Point Arithmetic".

"Section 2" provides the background on topics such as real numbers, decimal and binary numeral systems. Readers already well-versed with these topics may want to just skim through this section. "Section 3" gets into the basics of floating-point representation. "Section 4" describes some practical issues in representing real numbers in floating point representation.

2) Background

2.1 Real Numbers

Nearly any number you can think of is a real number. This includes:

Whole numbers: {0 and natural/counting numbers: 1, 2, 3…..upto infinity}
Negatives of counting numbers: {-infinity,….-3, -2, -1}
Other rational numbers/fractions: Any number that can be expressed as a fraction m / n where m and n are integers and n is not equal to 0. Put another way, rational numbers are numbers that can be written as a simple fraction. For example 0.5, 7 and 0.214 are all rational numbers, since they can be represented exactly as: 1/2, 7/1 and 214/100 respectively.
Irrational numbers: Any real number that is not a rational number, i.e., cannot be represented as a simple fraction. Some examples are:
- π ('Pi') = 3.14159....(and more). The popular approximation of 22/7 is close, but not accurate.
- √2 (Square root of 2) = 1.4142135623730950...(etc)

Whole numbers and negatives of whole numbers are also known as "integers" or "integral" types.

Note: A radix point is a symbol used to separate the the integer part of a number with from its fractional part. For e.g., in the number 1.23, the radix point (decimal point in this case) separates the integer 1 from the fraction 23.

2.2 Decimal Numeral System

The decimal number system that we use so frequently in our lives, has 10 digits (also known as decimal digits): 0, 1, 2, 3, 4, 5, 6, 7, 8 and 9. The "base" is the number of digits in a number system. So, the decimal number system is base-10 (has 10 as its base). The numbers 2525 or 25.25 use decimal digits, for example.

The decimal number system allows you to represent a real number by an ordered set of characters where the value of the character depends on its position. When you say 2525, you implicitly mean 2x10³ + 5x10² + 2x10¹ + 5 (which amounts to 2525). Therefore, it is a "positional" system of numeration.

A decimal fraction is a fraction where the denominator is a power of 10. For example 2.8 equates to 28/10, 0.28 equates to 28/(10x10).

2.3 Binary Numeral System

Unlike the decimal numeral system, which has 10 digits for representing characters in a number, most modern computers have only 0 or 1 to represent a number. Computers can be thought of as comprised of switches that are on or off (digital electronic circuitry using logic gates). These digits are called bits. Since there are only 2 digits available, the binary numeral system is a base-2 system.

Like the decimal numeral system, the binary numeral system is a "positional" system of numeration, that is it represents a number by an ordered set of characters where the value of the character depends on its position.

Representing Positive Integers:
Suppose you want to represent the positive integers 12 and 139 (both decimal numbers) in the binary system. You can represent them in the binary system as follows:

12 = 1x2³ + 1x2² + 0x2¹ + 0x2⁰= 1100
139 = 1x2⁷ + 0 + 0 + 0 + 1x2³ + 0 + 1x2¹ + 0 = 1000 1010

Notes:

Bytes are sequences of 8 bits and are now used widely as a fundamental unit. Most modern computers process information in 8-bit units, or some multiples of 8 at a time: 16, 32, 64 bits. A group of 8 bits is also commonly known as octet. Individual bits are not directly addressable and are manipulated as part of bigger units such as bytes.
As you can imagine, storing and processing the vast streams of bits can quickly become very inefficient. Moreover, it is tedious and error prone for system programmers to look at bits such as 1001001101110101 for debugging. Hexadecimal representation comes to the rescue. It is a compact representation of four bit groups and can be easily converted to the actual binary bits and back. A good description of hexadecimal is provided in [10].

Representing Negative Integers:
So far we have seen how positive integers can be represented using the binary numeral system. Negative numbers are typically represented using two common approaches: "signed magnitude" and "two's complement".

"Signed magnitude" is simple: simply use the leftmost bit (the "most significant" bit - remember binary system is positional) for the sign – 0 for positive and 1 for negative. For example, on 8-bit numbers:

-12 = 1000 1100
+12 = 0000 1100

However, using signed magnitude mechanism complicates matters for circuit builders when they need to take a sign bit into account. The "two's complement" representation solves this problem and is widely used. To form a two's complement of a number, flip all bits and add 1. For example, to represent the number -13:

Step 1: 13 = 0000 1101
Step 2: Flip all bits => 1111 0010
Step 3: Now add 1 (to represent negative sign) => 1111 0011

Therefore, -13 = 1111 0011, in two's complement representation.

Representing -13 as 1111 0011 might look odd at first, but try adding 13 with +13 and you'd get zero as expected. The rules of binary arithmetic addition operation apply: carry to the next position if the sum of the digits and the prior carry is 2 or 3. Refer to [12] for a description on binary arithmetic.

        11111 111
+13   0000 1101
-13    1111 0011
        ----------------
       10000 0000

But only the last 8 digits count, so +13 and -13 add up to 0, as you'd expect.

Java's Primitive Integral Data Types:
At this point, you might want to see how the concepts we have discussed so far map to data types of a programming language. Let's take Java's Integer (also referred to as "integral") data types, as examples.

Data Type	No. of Bits/Bytes	Signed/Unsigned	Covers what values?
boolean	1 bit	NA
byte	8 bits	Signed (two's complement)	-128 to 127
short	2 bytes	Signed (two's complement)	-32,768 to 32,767
int	4 bytes	Signed (two's complement)	-2,147,483,648 to 2,147,483,647
long	8 bytes	Signed (two's complement)	9,223,372,036,854,775,808 to +9,223,372,036,854,775,807
char	2 bytes	Unsigned (Unicode)	0 to 65,535

Representing Fractions:
In the decimal system, the value of each digit to the right of the decimal point (radix point) is calculated as 1/10, 1/10², 1/10³, and so on. These equate to 10^-1,10^-2,10^-3,and so on.

Similarly, in the binary system, the value of each successive digit of a binary fraction is the reciprocal of a power of 2 (1/2, 1/2², 1/2³… or 2^-1, 2^-2, 2^-3).

As you know, there are only two digits/symbols in the binary system – both are used for representing the numerical value of the number. So, how do you represent the radix point?

One solution is to make the radix point implicit: for example, assuming that the radix point is in the middle of a given 16 bits format. 8 bits would be dedicated to representing the integer part and the remaining 8 bits would represent the fractional part. Consider the number 5.25, as an example.

Integer portion 5: … 1x2²+ 0x2¹ + 1x2⁰= 00000101
Fraction portion .25 (=1/4): 0x2^-1 + 2^-2= 01000000

Decimal Number	Integer bits	Fractional bits
5.25	00000101	01000000

Fixing the radix point and dedicating a fixed number of integer bits and fractional bits, is what is known as "fixed-point" representation. Fixed-point representations are used by most calculators, spreadsheets and some computer hardware/software.

Now, suppose you wanted to represent 5.25. The binary representation would be:

312 = 100111000 (9 binary digits)
.25 = .01

The total number of binary digits required for the binary encoding is 11. The number can be physically stored in a 16-digit structure with five bits to spare. However, since this fixed point format assigns eight bits to represent the integer part of the number, 312.25 cannot be encoded because the integer part (312) requires nine binary digits.

Floating-point representations, on the other hand, do not use a fixed position for the radix point. It employs a scheme that contains a "floating" radix point (binary point in computers). We explore floating-point in the next section.

3) Floating-Point Numbers

There are times when you want to represent a number that is too large for your regular data types to handle. Say, you want to represent a very large number like 1x10¹⁰⁰ or a very small number like the mass of an electron: 9.1x10^-31. Enter floating-point representations.

Contrary to popular belief, floating-point representations are used to represent not only numbers that contain fractions, but also integers of large magnitudes. They essentially provide ability to represent a large dynamic range by means of an exponent while supporting reasonable accuracy by the use of a "mantissa"/"significand".

3.1 Floating Radix Points in Decimal Numeral System

Conventional scientific notation separates the position of the radix point from the significant digits. For example, the value 295.54 can be written as: 2.954 x 10², 29.54 x 10, etc.. In exponential form, the number can be written as 2.954E2, 29.54E1, etc.

Similarly, one can represent an integer value 8 as a floating-point 8.00 (8E0), 0.8 x 10 (or 0.8E1), and so on.

Numbers smaller than 1 can be represented by using negative powers of 10. For example 0.5 = 5E-1 (represents 5x10^-1) or 5E-2 (represents 50 x 10^-2).

When you multiply or divide any of these numbers by 10 or any powers of 10, only the position of the decimal point (radix point) changes: the radix point "floats".

Now, say you have a large number such as 9,999,999,123. You can represent it as 0.9999999E10, as long as the loss of the three low-order digits doesn't cause much harm in your given scenario. But it does allow you to represent a much wider range of values: for example, 0.9999999E99.

3.2 Floating-Point in Binary Numeral System

In the last section we discussed how a given number can be represented in multiple ways in the decimal numeral system, simply by varying the radix point, and changing the exponent. A binary floating-point number works in a similar fashion.

Let's consider the fractional number: + 11.1011 x 2^-3(or 11.1011E-3). It has:

A positive sign: This represents whether the number is positive or negative.
A Mantissa: 11.1001
- Where the integer part 11 = 2^{1 +}2⁰
- And the fractional part 1011 = 1/2 + 0 + 1/2³+ 1/2³(can be also represented as: 2^-1 + 0 + 2^-3 + 2^-3)
An exponent 3. The value of the exponent is negative.

One can also represent 11.1011 x 2^-3as 1.11011 x 2^-2. (In scientific notation these are 11.1011E-3 and 1.11011E-2 respectively.)

By varying the radix point and the exponent, one can represent a much wider range of numbers (both integers and fractions), even if the number of digits in the significand is much smaller than the range.

The most popular floating-point format is one defined by the IEEE 754 standard. The next sub-section describes the standard.

IEEE 754 Floating Point Formats:
Before the IEEE 754 standard was first published in 1985, there were several floating-point formats used by hardware and software. Therefore, it was difficult to port programs using floating-points from one system to another, as it would result in varying computed results.

The IEEE 754 specification is a widely adopted specification that describes how floating-point numbers are represented in binary (and decimal), how arithmetic operations and conversions needs to be done and how rounding and exception handling needs to be handled. It is widely adopted because it allows floating-point numbers to be stored in reasonable amount of space and computations to occur efficiently.

The standard defines five basic formats: a) three binary floating-point formats (which can be encoded in 32 bits, 64 bits and 128 bits), and b) two decimal floating-point formats. We explore the two key binary/base-2 floating-point formats: binary32 and binary64.

IEEE binary32 (also known as "single-precision" or "float" or "single") occupies 32 bits (4 bytes) in memory. It is implemented as "float" in Java, C, and C++, as "single" in Pascal and MATLAB, and as "real" in Fortran.

The layout of binary32 can be depicted as:

Sign

Bit index: 31 (1 bit)

Exponent

Bit index: 23-30 (8 bits)

Significand

Bit index: 0-22 (23 bits)

IEEE binary64 (also known as "double-precision" or "double") occupies 64 bits (8 bytes) in memory. It is implemented as "double" in Java, C++, C#, etc.

The layout of binary 64 can be depicted as:

Sign

Bit index: 63 (1 bit)

Exponent

Bit index: 52-62 (11 bits)

Significand

Bit index: 0-52 (52 bits)

Binary32 and binary64 have a similar layout and differ only on the number of bits devoted for the exponent and the significand.

Sign Bit: The sign bit (the high-order bit in the above tables) represents whether the number is positive or negative. 0 represents a positive number and 1 represents a negative number.
Exponent: The exponent represents the integer power of the base with which the significand must be multiplied.

The exponent needs to be able to represent both positive and negative exponents: positive for integer digits and negative for fractional digits, as we say earlier. Two's complement – the usual representation for signed values, would make comparison harder. Therefore, IEEE 754 standard establishes that the exponent be stored in biased form. A bias, in this context, refers to a constant that is added to the exponent in order to determine its final value.

Let's consider the single precision / binary32. Since the numerical range of 8 binary digits is 0 to 255 decimal, and half of this is approximately 127. Adding the constant 127 to all positive numbers would thus place them in the range 127-255. The negative exponents would fall in the range 1-126. Therefore, exponents of -126 to 127 are representable.

Exponents of -127 (all 0s) would be biased to 0, but is reserved to encode that the value is a denormalized number: a number that is very close to zero. It is also referred to as a "denormal". Denormalized numbers or denormals occur when the exponent of the number is too small to represent in the corresponding floating-point format. Why is the unbiased exponent of -127 reserved? Let us look at an example to illustrate the reason. The smallest non-zero normal number is 2^-126. Now say, a = 1.01 x 2^-126 x and b = 1.00 x 2^-126. The computation (a –b) would result in an exponent 2^-128 , which is not representable. This the computation would result in a value 0.0, which is not what you'd expect.

Similarly, exponents of +128 (all 1s) would be biased to 255, but are reserved for encoding an infinity or NaN (not a number).
- Infinities are generated by divisions by zero or by overflows. Overflows are generated when the result of an operation is a number so large in magnitude, that it cannot be represented. 1/0 = infinity, 1/inifinity = 0, and 0 (or any number) + inifinity = infinity.
- NaNs represent the result of operations that cannot have meaningful result in terms of a finite number or infinity. For e.g., 0/0 or √-1.
Significant/Mantissa: The significand is also known by other names too: mantissa, significant digits, fractional part, etc. We saw earlier that any given number, in base-2 or base-10, can be represented in multiple ways.

Floating point numbers are "normalized" such that they are represented as a base-2 decimal with a digit 1 to the left of the radix point, adjusting the exponent as necessary. Since every number (except for denormals) is represented with 1 is the first digit of the significand, IEEE 754 implementations can assume the bit, i.e., the first digit does not need to be stored explicitly. In the normalized form, the integer portion (the left of the radix point) is exactly one bit. Therefore, all stored digits in the bits dedicated to the significand hold only fractional parts. Refer to [13] page numbers 49-50, for a nice description on how a floating-point binary number is converted into a normalized form.

Also, by assuming a leading bit 1, the IEEE 754 single- and double- precision formats double the range of the representation. Rhe normalized significand's range is 1.00000… (all 0s, since 1 is assumed) to 1.111111… (all 1s, since 1 is assumed). The subnormals range from 0.0000…1 to 0.1111….

Thus, in general a given decimal number can be represented as:

(-1)^-sign x mantissa x 2 ^{(exponent-127)}

Let's take an example of a decimal number -55.625 to illustrate the above concepts.

Since the number is negative, the sign is 1.
55.625 can be represented in binary/base 2 as 110111.101 (equates to the integer: 32 + 16 + 0 + 4 + 2 + 1, and the fraction: 0.5 + 0 + 0.125).
Normalizing the binary value, we get 1.10111101 and exponent 5 (base 2). The significand is 1.10111101. The fraction is: .10111101.
Now, apply the bias to the exponent 5: this results into 132 (5+127).

Thus, -55.625 would be represented in binary32 as:

Sign	Exponent	Significand
1	10000100	1011110100000000000000

The effective ranges (excluding infinite values) of IEEE floating-point numbers are [17]:

	Binary Range	Decimal Range
Single Precision/ binary32	± (2-2^-23) × 2¹²⁷	~ ± 10^38.53
Double Precision/ binary64	± (2-2^-52) × 2¹⁰²³	~ ± 10^308.25

If you printed Java's minimum and maximum values for the corresponding float and double data types, you would get:

Float.MIN_VALUE = 1.4E-45 and Float.MAX_VALUE = 3.4028235E38
Double.MIN_VALUE = 4.9E-324 and Double.MAX_VALUE = 1.7976931348623157E308

4) Practical Issues in Representing Numbers

We now turn to some of the errors that can occur when we try to represent a given real number in the binary form.

4.1 Bit Patterns are Finite, Real Numbers are Not

Real numbers have the following properties (among others):

They have no upper and lower bounds: they go from –infinity to +infinity.
They have infinite density: There is a real number between any two real numbers. Consider 1.1 and 1.2. There are infinite real numbers between the two: 1.11, 1.111, 1.12, 1.112… and so on.

Now, let's consider a binary representation of 8 digits containing all bits as 1.
1111 1111 = 2⁷+ 2⁶+ 2⁵ + 2^{4 +} 2^{3 +}2^{2 +}2¹ + 2⁰= (128 + 64 + 32 + 16 + 8 + 4 + 2 + 1) = 256

Thus, using all 8 bits, you can get 256. In general, if you have n bits available, you have at most 2ⁿ bit-patterns, and so you can represent at most 2ⁿ numbers. For example, using Java's 32 bits integer data type "int", you can represent at most 2³²integers.

Most data types have a finite set of bits and as such the type will have to have a largest number and a smallest number (bounded). Also, computations must also not exceed the bounds.

4.2 Not all Real Numbers Are Representable Exactly

In the decimal numeral system (base-10), fractions such as 1/2, 1/4 and 1/5 can be represented exactly as 0.5, 0.25 and 0.2, respectively. There are others (irrational numbers) that cannot be represented exactly. For example: 1/3 = 0.333333333333333333… (3 repeats infinitely), 1/7 = 0.14285714285714285714285714285714… (and so on).

In the decimal numeral system, any decimal fraction whose value can be found with a sum of finite number of inverse power or 10, can be represented exactly as a decimal fraction. Others can't be represented exactly.

Similarly, in the binary system, some decimal numbers such as 0.5 are exactly representable ( 0.5 equates to 1/2). There are others such as 0.1 and 1/5 that cannot be represented exactly.
For example, the decimal value 0.1 can be represented roughly as a base 2 fraction 0.0001100110011001100110011001100110011001100110011..., where the fraction repeats inifinitely. Stop at any finite number of bits, and what you have is an approximation, rather than the exact value. For binary representations, any decimal fraction whose value cannot be found with a sum of finite number of inverse powers of 2, cannot be represented exactly.

4.3 Floating-Point Rounding Errors are Unavoidable

Arithmetic computations with integers are exact, except when they result in "overflows" (the result is outside the range of integers that can be represented. In contrast, floating point arithmetic is often not exact, because there are real numbers that cannot be represented exactly (as we discussed earlier) and so, must be approximated or "rounded off".

These roundoff errors can show up in unexpected ways, if you are not careful. Take the following code written in Java for an example.

double i=0.0;
while (i != 0.1) {
     i = i+0.01;
}

The while-loop in the above code will run infinitely, even though your expectation may be it should loop only 10 times. You can fix the code by changing "i !=0.1" to "i<=0.1".

Let's look at another example:

    double finalval = 0;
    for (int i=0; i<10; i++) {
       finalval +=0.1;
    }
    System.out.println(finalval);
    if (finalval == 1.0) {
       // do whatever
    }

The 'finalval' in the above example is not computed as 1.0 as you'd expect. It is: 0.9999999999999999. So, the code within the if block below that checks does an exact comparision is not executed.

Refer to [14, 15] for other great examples.

5) Summary

A floating point number is a finite representation that is designed to approximate (and not always represent exactly) a real number. It enables you to represent a much larger range of numbers, both integers and decimal fractions. Floating-point representation is done by breaking up a number into a fractional part and an exponent part and storing the parts separately.

By varying the radix point and the exponent, one can represent a much wider range of numbers (both integers and fractions), even if the number of digits in the significand is much smaller than the range. The most widely adopted floating-point standard is IEEE 754 and most software, including programming languages, implement the two binary formats defined by the standard: binary32 and binary64. A single-precision floating point number, i.e., binary32, contains 32 bits: 1 bit for the sign, 8 bits for the exponent and 23 bits for the significant digits. A double-precision floating point number, i.e., binary64, contains 64 bits: 1 bit for the sign, 11 bits for the exponent and 23 bits for the significant digits.

Although one does not need to know all the details of floating-point arithmetic used in hardware and software, having a basic understanding of how floating-point works makes it easier to represent real numbers, avoid potential issues due to rounding errors, and debug floating-point code.

6) References

[1] David Goldberg, "What Every Computer Scientist Should Know About Floating-Point Arithmetic"
[2] Brian Goetz, "Java Theory and Practice: Where's your Point?", Jan 2003
[3] http://en.wikipedia.org/wiki/Fixed-point_arithmetic
[4] Ronald Mak, "Java number cruncher: the Java programmer's guide to numerical computing", Prentice Hall PTR, 2003
[5] http://en.wikipedia.org/wiki/IEEE_754
[6] http://en.wikipedia.org/wiki/Computer_numbering_format
[7] David Monniaux, "The Pitfalls of Verifying Floating-Point Computations", May 2008
[8] http://www.mathsisfun.com
[9] Horstmann, Cay. "Appendix K - Number Systems". Big Java, Third Edition. John Wiley & Sons. © 2008
[10] Daniel B. Sedory, "What is Hexadecimal? And Where/How/Why is it used in Computers?"
[11] http://www.ibiblio.org/pub/languages/fortran/ch4-1.html
[12] http://en.wikipedia.org/wiki/Binary_numeral_system
[13] Julio Sanchez & Maria P. Canton, "Microcontroller Programming: The Microchip PIC", CRC Press
[14] Robert Sedgewick Kevin Wayne, "Introduction to Programming in Java", "9.1 Floating Point", http://www.cs.princeton.edu/introcs/91float/
[15] Bruce M. Bush, "The Perils of Floating Point", 1996
[16] Steve Hollasch, "IEEE Standard 754 Floating Point Numbers", 2005

Using JavaScript with Apache Ant

2010-09-28T14:35:00.000+05:30

Here is Eamonn McManus's post on using JavaScript with Apache Ant.

Java concurrency guidelines

2010-08-27T23:38:00.001+05:30

An extensive report on Java concurrency guidelines - by Fred Long et al. and published by SEI-CMU recently - can be found here.

Eponymous laws

2010-08-05T01:57:00.001+05:30

Eponymous laws are adages, laws, principles, theorems, etc. named after a person. Wikipedia has a fabulous list of such laws here.

A few from the above list (taken as is) that may be of interest to folks in software engineering:

Amara's law — "We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run."
Brooks' law — "Adding manpower to a late software project makes it later." Named after Fred Brooks, author of the well known book on Project Management, The Mythical Man-Month.
Campbell's Law - "The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor." Named for Donald T. Campbell (1916 - 1996)
Classen's law — Theo Classen's "Logarithmic Law of Usefulness" - 'usefulness=log(technology)'.
Conway's Law — "Any piece of software reflects the organizational structure that produced it." Named for Melvin Conway.
Dilbert Principle — Coined by Scott Adams as a variation of the Peter Principle of employee advancement. Named after Adams' Dilbert comic strip, it proposes that "the most ineffective workers are systematically moved to the place where they can do the least damage: management."
Gall's law — "A complex system that works is invariably found to have evolved from a simple system that worked."
Gibrat's law — "The size of a firm and its growth rate are independent."
Godwin's law — An adage in Internet culture that states, "As an online discussion grows longer, the probability of a comparison involving Nazis or Hitler approaches one." Coined by Mike Godwin in 1990.
Goodhart's law — "When a measure becomes a target, it ceases to be a good measure."
Kranzberg's First Law of Technology — "Technology is neither good nor bad; nor is it neutral."
Linus' law — "Given enough eyeballs, all bugs are shallow." Named for Linus Torvalds.
Murphy's law — "Anything that can go wrong will go wrong." Ascribed to Edward A. Murphy, Jr.
Pareto principle — States that for many phenomena 80% of consequences stem from 20% of the causes. Named after Italian economist Vilfredo Pareto, but framed by management thinker Joseph M. Juran.
Rothbard's law — "Everyone specializes in his own area of weakness."
Sayre's law — "In any dispute the intensity of feeling is inversely proportional to the value of the stakes at issue."
Wirth's law — "Software gets slower faster than hardware gets faster."
Dunning-Kruger Effect - Per this Wikipedia article: "... is a cognitive bias in which an unskilled person makes poor decisions and reaches erroneous conclusions, but their incompetence denies them the metacognitive ability to realize their mistakes. The unskilled therefore suffer from illusory superiority, rating their own ability as above average, much higher than it actually is, while the highly skilled underrate their abilities, suffering from illusory inferiority".

Computer and software history

2010-07-10T19:15:00.002+05:30

Cicero (Roman author, orator, & politician - 106 BC - 43 BC) is quoted to have said: "History is the witness that testifies to the passing of time; it illumines reality, vitalizes memory, provides guidance in daily life and brings us tidings of antiquity."

For those interested in the history of computers: The Computer History museum, located here on the Web, is dedicated to preserving and presenting the stories and artifacts of the information age. It was established in 1996 in Mountain View, California, USA. Also checkout videos of the many lectures and events at the museum and historic computer films, hosted on Youtube.

The museum has a special interest group on software history. Besides the history of software since the 1950s it has interesting anecdotes, first person essays, etc.

Keep in mind though, the Website is still in the making.

Rift between business and IT

2010-07-02T19:39:00.001+05:30

Found this particularly hilarious picture on one of Anne Thomas Manes' slides set (don't remember which one though).

Mappings Among Architecture View Sets

2010-07-01T19:42:00.000+05:30

This is my take on how some of the popular view sets map to each other:

"Selling" and "Order Handling" Overview (Communications Service Provider Environment)

2010-06-11T17:26:00.004+05:30

I often get queries related to basic aspects of selling and order handling business processes from teams that I work with. Here is a brief (very brief) overview of these processes, using TeleManagement Forum's Business Process Framework (widely known as eTOM) as a reference.

1. Selling

In sales terminology, a potential or a likely customer is called a "prospect". Prospects are typically determined by one of the following:

Leads – These are typically determined by product marketing teams and decision analytics systems supporting product marketing. For e.g., responses from email-, direct- or tele-marketing campaigns may provide good leads to potential customers. Similarly, website analytics tools may be used to analyze web page hits from a particular region to determine prospects.
Referrals – The customer might indicate an interest to a branch or call center representative who then directs the referral to sales specialists. Service providers also provide customer portals where customers can register an interest.
Transaction-Based Triggers – Prospects may also be determined by transaction-based activities combined with decision analytics. For e.g., based on customer's long distance calling pattern and its availability to a sales rep, the sales rep may be able to suggest a higher value unlimited long distance plan.

Prospects are turned into opportunities for further processing through the sales-pipeline, based on a number of factors, including but not limited to [TargusInfo]:

Which prospects are most likely to convert?
What is a prospect's potential lifetime value?
What size order would the prospect place if she were to become a customer?

The sales-pipeline supports the key selling related functions such as analysis of customer needs, identification of respective product offerings, sales-negotiation, quotes, proposals, sales forecast, etc. eTOM classifies key selling sub-processes as illustrated in Figure 1 below.

The sub-processes within "Selling" in eTOM are:

a) Manage prospect: This sub-process is responsible for matching assigned leads with the most appropriate products and ensuring that the prospects are handled appropriately [Etom_Gb921]. Activities include capturing and processing the probability of successful sales closure, estimating the total attainable revenue, analyzing the needs of the prospect, identifying potential solutions from the service provider's portfolio, etc.

b) Qualify opportunity: This sub-process ensures that the opportunity is qualified in terms of any associated risk, and the amount of effort required (e.g., response to a RFP vs. potential revenue) to achieve a sale [Etom_Gb921]. Also, customer's needs are analyzed and matched against available product offerings or customized solutions.

c) Negotiate sales/contract: Customers may agree with the standard prices and promotions or may want to negotiate pricing. For example, although the sales rep may have already offered 5% discount, the customer might be looking for a 12% discount. Also, the customer may want to negotiate non-standard terms and conditions/contracts. This sub-process includes activities supporting sales and contract negotiation.

d) Acquire customer data: This involves collection, association and organization of all pertinent data related to the customer, including those that are required to support the agreed product offerings and contract.

e) Cross/up selling: The purpose of this sub-process is to ensure that the value of the relationship between the customer and service provider is maximized by selling additional, or more of the existing products [Etom_Gb921]. Additionally, the ongoing analysis of customer's usage, problems or complaints can be used to identify when the current products may no longer be appropriate for the customer, or an opportunity for a larger sales arise [Etom_Gb921].

f) Develop sales proposal: The purpose of this sub-process is to develop a sales proposal to respond to customer's requirements [Etom_Gb921]. The development of a sales proposal may require selection of standard a product offering or may require the development of a non-standard offering [Etom_Gb921], leading to development of a unique solution design.

g) Manage sales accounts: This sub-process is responsible for managing sales accounts assigned to a sales channel and/or manager [Etom_Gb921]. These accounts are then used on a regular basis to develop appropriate relationships and contacts, to prospect for leads for other offerings, or to offer specific discounts based on criteria such as sales volume, length of relationship, etc.

Note that the sub-processes defined in eTOM, as shown in Figure 1 above, are in no particular sequence. eTOM refrains from defining sequencing of these processes as different service providers may need to implement them in different sequences, based on their particular environment. Figure 2 below depicts a sample flow that might be applied in a service provider environment.

2. Order Handling

Once a sales contract has been signed by the customer, its time to create and process customer orders for fulfilling the services that form the product offering. Enter order handling.

Although the terms product offerings and services are often used interchangeably, they have separate meanings. As described by [McFadyen], "This is best thought of as the box that everything is put in. Just as when you buy cornflakes it is the flakes of corn you want inside the box, but it is the box with its logo, brand name, and barcode that you actually purchase." The "flakes of corn" is the service while the "box of cornflakes" is the product offering. From the viewpoint of a product manager, a product offering defines what functionality is provided at what price to which market segments over which sales channel. A service on the other hand is the functionality that the customer gets. Product offerings can get complex in that many different services may be combined in a single offering bundle to make it attractive (cost-effective, for e.g.).

eTOM classifies key order handling processes/activities as:

The sub-processes within order handling are:

a) Determine preorder feasibility – This sub-process is responsible for determining whether the standard/customized product offering chosen for the customer is available and/or feasible. Examples of such checks include (but are not limited to):

Can the corresponding services be provisioned at the given location?
Are there enough circuits or lines available in the given region?
Can the services meet the specified SLA based on current network environment?

b) Authorize credit – The purpose here is to assess a customer's credit worthiness in support of managing risk and company's exposure to bad depth [Etom_Gb921]. Credit checks are often made against credit check service providers like Dun and Bradstreet, etc.

c) Issue customer order – This sub-process ensures that all necessary information about the customer order is captured and validated and that the customer has been issues a sales order. This sub-process typically kicks in after a purchase order has been received by the service provider.

A purchase order (PO) is a commercial document issued by a buyer to a seller, indicating types, quantities, and agreed prices for products or services the seller will provide to the buyer [Wikipedia_po], and constitutes a legal offer to buy products or services. Acceptance of the PO by the seller (service provider in this context) forms a contract between the buyer and the seller. A PO ensures the service provider is protected in case of a customer's refusal to pay for agreed products or upon unilateral cancelling of the order [Wikipedia_po].

A sales order (SO), on the other hand, is an order issued by the seller (service provider in this context) to the buyer (customer). Information captured as part of the sales order may include originating PO (one or more), service locations, product configuration data (such as bandwidth, circuit speed, etc.), bill-to customer account, customer contacts, and so forth. Both the customer and the service provider use the issued order to track the status, make edits to the order, etc.

d) Track and manage customer order handling – The objective of this sub-process is to ensure customer provisioning activities are assigned, managed and tracked efficiently to meet the committed availability date [Etom_Gb921]. This includes scheduling activities, generation of respective service order creation requests, tracking orders, adding additional information to existing customer order, cancelling customer order, monitoring jeopardy status of customer orders and escalating customer orders as necessary [Etom_Gb921].

e) Complete customer order – This process is responsible for ensuring that any customer information required by any other processes is updated as part of customer order completion [Etom_Gb921]. This may involve additional touch points with the customer. This may also involve sending relevant notifications to the customer on events such as service provider's acceptance of the order, commitment date, etc.

f) Report customer order handling – The objective of this sub-process is to monitor the status of customer orders, provide notifications of any changes and provide management reports [Etom_Gb921]. Specialized notifications to customers may include those indicating service provider's validation and acceptance of the order, availability date of the services, completion of the provisioning, activations of the respective services, etc.

g) Close customer order – The objective of this sub-process is to close a customer order when the customer provisioning activities have been completed.
As mentioned earlier, eTOM doesn't specify how the sub-processes are sequenced.

References

[McFadyen] Andrew McFadyen, http://telecomseim.blogspot.com/
[Etom_Gb921] GB921 Addendum D-Process Decompositions & Descriptions, Business Process Framework (eTOM), R 8.0, TeleManagement Forum (TM Forum), June 2009
[Wikipedia_po] http://en.wikipedia.org/wiki/Purchase_order
[TargusInfo] "Turning Prospects into Profits", TargusInfo, 2009

Paul Graham on Startups

2010-05-13T15:21:00.002+05:30

Paul Graham is famous for his plain-spoken, deeply insightful and thought-provoking essays. Some of his essays on startups:

A listing of his essays is here.

Philippe Kruchten et al. on Software Architecture Review & Assessment

2010-05-11T17:58:00.000+05:30

The report is here.

Bjarne Stroustrup on what should be taught to new software developers in college

2010-05-01T07:48:00.003+05:30

Bjarne Stroustrup's article titled "What Should We Teach New Software Developers? Why" (Communications of the ACM, Vol. 53 No. 1, Pages 40-42 ).

REST Links

2010-04-27T18:43:00.001+05:30

A nice list of REST links: Andre's REST Links.

TM Forum's MTOSI standard: Setting the Context

2010-04-06T13:36:00.009+05:30

MTOSI stands for Multi-Technology Operations Systems Interface. It is a standard specified by TM Forum. It is now part of TM Forum's Interface Program (TIP).

Why is it called "multi-technology"? Why do you need to worry about it? What does it really do? How does it relate to other TM Forum standards and initiatives (often, quite confusing for starters!)? This article attempts to explain the answers for these questions.

This article is organized as follows:

Section "1 - Intent" provides a summary of what MTOSI aims to do.
Section "2 - Related TM Forum Standards" provides a description of how MTOSI is related to some of the other TM Forum standards and initiatives.
Section "3 - Motivation" provides scenarios that illustrates the problem(s) MTOSI addresses.
Section "4 - MTOSI Description" provides more details on MTOSI.

1) Intent

The intent of TeleManagement Forum's (TMF) Multi-Technology Operations Systems (MTOSI) is to provide open and standards-based interfaces between OSSs (e.g., Service Activation systems, Inventory systems, Network Management Systems, Element Management Systems, etc.). For a detailed exploration of OSS/BSS systems, see my earlier post titled "Telecom OSS/BSS: An Overview".

2) Related TM Forum Standards

2.1 Relationship with MTNM

TeleManagement Forum's (TMF) Multi-Technology Operations Systems Interface (MTOSI) started as an MTNM (Multi-Tehchnology Network Management) substream sometime in 2003. Lets discuss MTNM first.

MTNM provides CORBA-based Element Management Layer (EML)-to-Network Management Layer (NML) interfaces for the management of "multi-technology" networks (SONET, SDH, PDH, ATM, Frame Relay, DSL, connectionless networks such as Ethernet, etc.). However, it can also be used to support a single technology such as ATM. Examples of management areas are: connection management, NE and EMS configuration, network inventory discovery, euqipment inventory management, performance monitoring, alarm surveillance and event reporting, etc. MTNM can also be used to manage a particular subset of a management area - say, alarm surveillance.

Around the time MTNM 3.5 commenced, it was realized that several vendors were using MTNM by converting the CORBA IDLs to XMLs, due to lack of XML based interfaces. Enter MTOSI.

MTOSI has gradually taken sets of MTNM interfaces, generalized the operations systems to operations systems interactions and created XML, SOAP Web services based interfaces - recasting and extending MTNM. Like MTNM, it provides a single interface for the management of many technologies. Moreover, MTOSI has had the goal to make XML interfaces independent of the underlying transport (HTTP, JMS, etc.)

2.2 Relationship with TIP and mTOP

MTOSI has been part of TM Forum Interface Program (TIP) since April 2008. Multi Technology OSS Program (mTOP) coordinates advances in the SOAP and XML based MTOSI interface standard, as well as the CORBA/IDL based MTNM standard. MTOSI service classifications are based on the eTOM model, while the information model is based on SID and MTNM data models.

2.3 Comparision with OSS/J

MTOSI APIs are focused towards providing interfaces at the service, network and element management layers. OSS/J on the other hand aims to provide open interfaces across the spectrum of both OSS and BSS. OSS/J APIs, unlike most MTOSI APIs are network technology independent [1]. OSS/J APIs are entity centric unlike MTOSI APIs that are task-centric. OSS/J provides a generic set of operations on "Managed Entities" [1]. There are efforts underway to draw the OSS/J and MTOSI communicaties together, and we may see a common goals and plans for migration for both OSS/J and MTOSI in the future.

3) Motivation

Element Management Systems (EMS) typically manage the functions of Network Elements (switches, CPU and remote hubs, gateways, routers, susbcriber line units, etc.) from a single vendor. Examples of such management functions include receiving and executing commands sent by upstream activation systems, feeding equipment status data back to upstream systems such as network and trouble management systems, etc. Moreover, the EMS may be able to manage functions related to a single technology (e.g., ATM, SONET, etc.) for a subset of management area (e.g., performance monitoring). Therefore a service provider environment typically has several types of EMS. Typically a Network Management System (NMS) manages functions of many EMSs. To be easily portable across different NMS systems, an EMS vendor can provide MTOSI based standards interface. This would enable easy integration of the EMS with various NMS that service providers could use.

Lets consider another example. A service activation system (an OS) may allow an order management system (also an OS), to request activation of a given service via a service request. The service activation system (an OS) may in turn need to send requests to multiple NMS and EMS OS's to complete the service activation request. A service provider environment in all likelihood contains multiple service activation systems as well as multiple EMS's. If each of those provided proprietary interfaces containing proprietary information models: a) the order management system must be integrated with each of the service activation interfaces, and b) the service activation systems must be integrated with the relevant EMS interfaces. With many services being added on frequent basis, this can easily turn into a nightmare. Utilizing standards-based interfaces provided by MTOSI the integration becomes lot more easier, reducing costs and increasing agility.

Lets consider yet another example. Say, an operator uses Telcordia TIRKS system to administer Layer 1 facilities and equipment while it uses a seperate OSS inventory system to administer Layer 2/3 (logical inventory) and higher services. TIRKS provides an open interface (data dictionaries and XML message definitions) using MTOSI which can be used by the OSS inventory systems to receive messages generated by TIRKS systems (such as inventory update notifications) and to send messages to the TIRKS system.

Similarly, a fault managent system may need to subscribe to alarm reports based on filtering conditions and may also handle subsequent reporting of alarms to subscribed OSs. Similarly, a manager fault management system may receive alarms from several technology-specific fault management system via active alarm sync.

4) MTOSI Description

MTOSI defines OS-to-OS (OS stands for operations systems) interfaces for service and resource management, where OS functionality can be at the levels of Element Management Layer, Network Management Layer or the Service Management layer, in the Telecommunications Management Network (TMN) Logical model.

Examples of such OS in this context include: Inventory Management, Performance Management Systems, Fault Management Systems, Service Activation, Service Assurance, CRM systems such as Order Entry system, Network Management systems, Element Management Systems (EMS). Refer to my earlier post titled "Telecom OSS/BSS: An Overview" for a description of some of these.

Note, that MTOSI is not concerned with EMS-to-NE API, which is typically based on protocols such as SNMP, TL or CMIP. Nor is it concerned with internal implementation details such as an internal inventory model - it only provides a data format and a number of operations around retrieving or modifying the data.

Management functions covered by MTOSI include (but not limited to):

Inventory Retrieval, Inventory Update Notifications, Enhanced Inventory Retrieval (attribute value matching based on XPath), Multi-Event Inventory Notifications (ability to send multiple inventory updates in a single notification)
Alarm Reporting, Active Alarm Retrieval
Service Activation
Service Inventory Management
Resource configuration and activation
Performance management and maintenance commands
Multi-Action and Request Transactions (MART) (i.e., ability to package multiple requests into a single message)

The output of the MTOSI effort can be downloaded from the TM Forum Web site. The output is partitioned into something called as Document Delivery Packages (DDPs) - often organized based on functional areas (such as resource inventory management). The framework DDPs contain the generic aspects applicable to all other DDPs while the Data Model DDPs contain aspects related to entities, data structures, attributes and states. Each DDP also contains the following set of documents:

TMF 518, MTOSI Business Aggreement (BA): Identifies the requirements and use cases to be supported by the interface.
TMF 612, MTOSI Information Aggreement (IA): UML description of the interface.
TMF 864, MTOSI Interface Implementation Specifications (IIS): Maps the UML model to XML (WSDL, XSD, bindings, etc.)
Supporting documentation

5) References

[1] "Bridging the Gap Between MTOSI and OSS/J" by Nigel Davis, John Reilly and Suresh Bhandarkar

[2] "Implementing NGOSS with MTOSI"
[3] "A very short history of mTOP" (Requires login to TM Forum site)
[4] Tim McElligott, "MTOSI brings OSS Integration a Step Closer", Nov 2005,
[5] "Designing a Standards-Based Application", March 2009,
[6] Stephen Fratini, Irene McGoey, Garry Grimes and Enrico Grosso, "Introduction to the TM Forum's Multi-Technology Network Model", National Fiber Optic Engineers Conference, 2001

On patterns of software degeneration

2010-04-01T17:42:00.000+05:30

Here ("Big Ball of Mud" by Brian Foote and Joseph Yoder) is a paper that explains some patterns of software entropy.

REST in practice and Richardson's RESTful maturity model

2010-03-24T13:04:00.000+05:30

Jim Webber, Savas Parastatidis and Ian Robinson presented "REST in Practice" at a QCon talk, in which they also presented a model of RESTful maturity developed by Leonard Richardson.

Here (Richardson Maturity Model: steps toward the glory of REST) is Martin Fowler's explanation of the maturity model.

Telecom OSS/BSS: An Overview

2010-03-19T12:05:00.020+05:30

1. Introduction

Before the initial 1970s, most of the support activities in a telephone company such as taking orders, maintaining network inventory, provisioning services (for example, line assignment and testing), configuring network components, managing faults and collecting payments were performed manually. It was realized that many of these activities could be replaced by computers. In the next few years, a number of computer systems and software applications were created to automate these activities. Examples include TIRKS, RMAS, SES, etc. Thus came the term Operations Support Systems (OSS).

OSS are “network systems” dealing with the communications network and supporting processes such as maintaining network inventory, provisioning services, configuring network components, managing faults.

Business Support Systems (BSS) is a newer term and typically refers to “business systems” dealing with customers and support processes such as taking orders, processing bills, collecting payments, sales and marketing, supporting customer care agents in response to service requests, trouble reporting and billing inquiries, etc.

OSS and BSS systems together are often abbreviated as BSS/OSS or B/OSS. The term OSS was historically used to include both network and business systems. Some industry analysts, system integrators and service providers still use the term OSS to include both network and business systems, which sometimes causes confusion.

This article provides an overview of some of the core areas in OSS and BSS such as Order Fulfillment, Service Assurance and Billing systems. The following BSS/OSS systems are covered:

Order Fulfillment – Order Management, Service Provisioning and Inventory Management
Service Assurance - Fault & Trouble Management, Network Performance Management, Topology & Configuration Management, Planning & Testing
Billing - Billing Mediation, Rating, Billing Systems, Interconnection Billing, Revenue Assurance

The article explains some of the basic functions of these systems. However, this article doesn't intend to provide extensive details. For an extensive overview of business activities, business process and functions, refer to standards such as eTOM and TAM at http://www.tmforum.org/.

2. The Realm of OSS/BSS in Order Fulfillment, Assurance and Billing

2.1 Order Fulfillment

Communications products/services could range from Voice services to IP and Data services to Hosting and CPE services. Some of the examples of communications products/services are:

Voice – Basic telephony, long distance, toll-free, Voice over IP (VoIP), Contact Center, Local Access, etc.
Internet Protocol (IP) – Internet Access, VPN, Contact Center, VoIP, Remote Access, etc.
Data – Layer 1 Wide Area Network (WAN) Services such as SONET, Layer 2 WAN services such as ATM, Frame Relay, Private Lines, Layer 2 VPN and Metro Ethernet, etc.
Hosting – Custom Application Environments, Disaster Recovery, Managed Services such as storage, security and network services, Web Site Hosting, etc.

Order Fulfillment functions are a critical set of activities performed in order to fulfill customer orders for services in a Communications Service Provider (CSP) environment.

The following figure shows a very high-level view of activities performed in a typical CSP environment.

After order entry, validation and submission, orders are decomposed and sent for provisioning. Upon fulfilling the decomposed orders and appropriate testing of the circuits, the orders are put into inventory. The following sub-sections explain the Order Fulfillment related functions and OSS/BSS systems.

2.1.1 Order Management

Order Management systems are complex systems that allow customer or customer service representatives to capture and process new orders, modify existing orders, process customer moves and changes, price quotes and orders, validate orders, etc., while supporting multiple channels such as Web, Order template documents and partner applications as well as multiple lines of businesses.

Order Management includes the following areas:

Order Entry and validation – The Order Entry process captures order details such as package or plan, service address, service details, customer accounts, relevant contacts and applicable contracts. Data entered during Order Entry is also validated against predetermined rules.
Orders can be validated as the data is entered and/or validation after all the data has been entered. Products/solutions that validate order data as they are entered and walk the user through the product configuration process are known as “Product Configurators”. One of such tools available in the market is Selectica COnfigurator.
Order Decomposition – A single customer order can be decomposed into one or more service requests, typically based on service types or quantities, in order to be able to fulfill an order.
For example, if a customer order contains both a VoIP order and a phone line order, two service requests would be created, one each for VoIP and the phone line, each of which would be sent to the appropriate provisioning systems.

One of the major problems service providers often grapple with is that, as new services are added to the offerings, led by different business units, the lack of flexible order management platform results in product/service specific OSS/BSS applications. These in turn result in higher time-to-market as well as increased costs of maintaining many different applications and systems. Product catalog based Order Management solutions attempt to solve these problems by storing and processing qualification rules for services based on customer profiles, ordering channels, service locations, product interdependencies, availability, customer eligibility and other business constraints.

2.1.2 Service Provisioning

Service Provisioning systems are systems used to setup products/services for the customer after an order for the services has been created and accepted by the CSP.

Service provisioning activities include specifying the pieces of equipment and parts of the network to fulfill the service, configuring the customer’s routing path, allocation of bandwidth in the transport network, setting up of wiring and transmission, etc.

Some of the systems that constitute provisioning systems are: Circuit Design & Assignment Tools, Activation systems, and Field Service Management systems.

Circuit design refers to specifying whether facilities exist to provide the service and which pieces of the network equipment and routes the service shall utilize.

One of the most widely used systems providing Circuit Design facility is Telcordia TIRKS. Apart from Circuit Design support, it also provides circuit order control, inventory record maintenance, selection and assignment of components from inventory, and preparation and distribution of circuit work orders. The order control module in TIRKS works with a circuit provisioning system and operates in conjunction with other TIRKS components to assign facility and equipment information for circuit orders and design circuits. TIRKS can then provide automated design criteria for certain circuit orders. The circuit design generated in TIRKS is then communicated to field operations or automated activation systems for implementation.

Circuit Design and Assignment tools these days often have graphical tools that allow a user to create services on a network map using mouse clicks and drag-and-drop rather than drawing maps by hand or using an abstract set of equipment identifiers displayed in a table.

After a service is designed based on the existing equipment and circuit inventory, it is ready to be activated. If new equipment or lines need to be configured manually, a Field Service Management (FSM) system is notified which in turn dispatches technicians.

Moreover, certain activations can be performed automatically. For example, issuing commands to ATM or circuit switches to provision circuits, to SONET terminals to allocate bandwidth, and to a wide array of access devices such as DSLAMS, Digital Loop Carriers (DLC), or cable modems. For such activations, Service Activation systems pass the device specific commands and configuration changes to the network elements, Element Management Systems (EMS), Network Management Systems (NMS) or application hosts.

EMSs are designed to receive and execute commands sent by activation systems on the devices. EMSs can also feed equipment status data back to network and trouble management systems. EMSs use protocols such as Common Management Information protocol (CMIP) or Transaction Language (TL) or Simple Network Management Protocol (SNMP) to communicate with activation and other systems.

Activation systems often comprise a library of adapters to various network systems. They usually also support transaction control, i.e. the capability to roll-back operations already performed, in case an error occurs.
It should be noted that Provisioning systems interact with the Inventory systems, both to verify that the required network elements and other facilities are available, and once the resources are provisioned - to reflect the changed on-line configuration of the facilities. Therefore, provisioning systems have close channels with inventory systems. As a result, some vendors have combined workflow capabilities with inventory management capabilities in their products.

2.3 Inventory Management

Tracking inventory involves tracking equipment, facilities and circuits.

Some examples of information tracked are: the location and quantities of the equipment, how a piece of equipment is configured and its status, etc.
Inventory Management Systems track both the physical network assets (such as equipment and devices) as well as “logical” inventory (such as active ports, circuit ids, IP addresses, etc.), although not all support both.

By relating usage of network assets to specific customers and services, an inventory system can help network operations determine the network usage and available capacity as well as enable automated network design and planning. Inventory Management Systems also enable Service Assurance systems to find the impact of a network fault on the customer’s circuits.

Some tools also have “auto-discovery” features to automatically check physical network assets and match the results with the information held in the inventory. However, these work only with some of the newer intelligent network elements.

2.2 Service Assurance

Communications service providers (CSP) strive to differentiate themselves from their competitors by implementing attractive Service Level Agreements (SLA). SLAs are formal contracts where the level of service delivered by the CSP to his customer is stipulated. An SLA may specify levels of service availability, performance, operation, etc. as well as penalties upon violation of the SLA.

Offering SLAs implies that the service provider has the ability to monitor, act and report the level of service, in order to assure the quality of services delivered to the customers. Service Assurance refers to all the activities performed for such an assurance. The goal of Service Assurance is to provide an optimal customer experience, that helps retain existing customers, attract new customers and prevent penalties arising out of violation of SLAs.

The following sub-sections introduce some of the common service Assurance systems.

2.2.1 Fault and Trouble Management

Fault Management Systems are designed for detection, isolation and correction of malfunctions in a communications network. They monitor and process network alarms generated by network elements (routers, switches, gateways, etc.). An alarm* is a persistent indication of a fault that is cleared only when the triggering condition is resolved.

Examples of trouble or fault in a network are damage to an optical fiber line, switch failure, etc. Such a problem in the network can result in a chain reaction where many network elements in a certain path produce alarms*.

Fault Management Systems may be either a component within Network Management Systems or as a standalone set of system and application software.

The following figure illustrates how fault management works.

Network Elements are designed to provide various levels of self-diagnosis. Older Network Elements might simply send an alarm notifying a problem while newer Network Elements can provide more precise and detailed messages. Fault Management Systems may collect alarms via SNMP traps, CMIP events or proprietary agents, via EMS. They use complex filtering systems to assign alarms to specific severity levels and correlate different alarms to locate the source and cause of a problem.

After a problem is identified, the FMS then notifies appropriate network operators as well as pass the problem information to a Trouble Management System that in turn logs the problem and issues a trouble ticket to start the repair process.

The Trouble Management System then sends commands to appropriate systems such as Field Service Management to schedule and dispatch technicians to repair the equipment and/or to EMS to reroute network traffic around the problem areas.

Trouble Management systems also handle automatic escalation, such as progression of a ticket from minor to major or major to critical, etc., and support a variety of notification methods such as paging, emails, synthesis voice dial-out.

Fault Management systems usually provide graphical network displays which are projected on large screens at the Network Operations Centres (NOC). NOC operators can see role-based views on their consoles, shortcuts to operations they perform the most as well as tools to quickly make connections to EMS to perform any testing or diagnostic operation.

2.2.2 Network Performance Management

Performance Management components in NMS and other Alarm Handlers monitor applications and systems and collect performance variables of interest at specified intervals. Performance variables of interest may be service provider network edge availability, customer premises availability, response times, packet delivery rate, packet losses, latencies, jitters and out of sequence packet reorder, etc., to name a few.

One way to capture performance metrics is collecting event logs, CDRs and other performance data such as counters or timers that the network and system elements maintain as part of their normal operation. This is referred to as passive measurement. Performance data is captured by polling MIB using SNMP or using syslog, (I & II), FTP, EMS feeds, etc. Most passive measurements report on a single network element.

For example, an Ethernet Switch may have a MIB which provides in and out data volumes of each port, histograms of frame sizes, number and types of erroneous frames, central processing unit (CPU) busy status. Associated Remote Monitoring (RMON) MIB-type data can then list ten most active users, etc. Performance Management tools can access the data by using SNMP to poll the MIBs at predefined intervals.

Statistics on performance variables can also be captured via dedicated network appliances known such as “probes” and “sniffers” that monitor or probe customer’s local loop connections, packet performance, etc. This form of performance testing is usually referred to as active testing.

Packet sniffers typically monitor signaling protocols such as SIP and RTP by inspecting packets on the wire/fiber, using pings, DNS, FTP, HTTP fetches, etc. Examples include WireShark and Geoprobes.

Probes such as Brix Networks BrixWorks Verifiers and Tektronix/Minacom IVR tools typically emulate customer traffic in order to test or probe specific paths to measure the quality of the services supported. Probes could be either placed into the network or could be built into network elements such as in the case of Cisco’s IP Service Level Agreements tools.

Note that active measurement measures a service, such as application response time, instead of the internal operation of a network element.

An example of active network performance test is injecting “ping” (short, network layer echo packet) into the network aimed at a remote IP address. Round-trip time is measured if the ping packet returns, and an error counter is incremented if it doesn’t.

Performance statistics captured by “active” or “passive” performance tests are normalized and routed to relational databases and/or data-warehouses. An alternative is to pass the performance data directly to Performance Management tools. For example, Concord eHealth could collect performance statistics from Netcool agents via SNMP polls at a pre-defined interval.

Performance statistics are initially analyzed to determine the normal (baseline) levels. Appropriate thresholds are determined for each of the interesting performance variable so that exceeding the thresholds indicates a problem.

Performance Management tools then measure the performance variables against SLAs defined as thresholds per application or service, on an on-going basis. In case of exceptions they report them to alarm handlers. This form of performance monitoring is reactive performance monitoring. Some tools also support proactive monitoring by way of providing simulation tools that helps network operators project how growth in network traffic will affect performance metrics and plan to take proactive countermeasures such as increase capacity.

Performance Management tools may also support real-time and historical reporting. Some CSPs have taken performance statistics of the network affecting customers’ circuits to their customer self-service portals.

2.2.3. Topology & Configuration Management

Older networks and systems were static and the network wiring was fixed in place, and sometimes required long outages while changes to the network and its configuration were being made. Any error or inconsistency in the configuration files of different network devices caused problem, and therefore these changes were well controlled [3].

According to [3], with the rise of IP-based, dynamically routed networks, network topologies started becoming dynamic. The topology of the network became dynamic because a few of routers might decide, on their own, to shift routing patterns, or because a network operator group might add a new router or switch to the network, possibly without everyone else in the network operations center being aware of the changes. Instead of static associations between users and network addresses (as was set in the old “hosts” file), DHCP and other techniques allowed users to appear, move, and disappear without providing prior notice to the network administration.

Most major NMSs therefore provide capabilities to automatically discover a network’s actual topology, which is critical to understand network performance or root cause of network alarms, etc.

Probes are placed into the network to automatically find devices and circuits. Also, most network elements provide MIBs that can be polled via SNMP to discover the network, although discovering the network topology in its entirety may not be guaranteed. Backup paths, virtual private networks, MPLS, etc., can make it very difficult to discover actual paths, through multiplexed links, patch panels, and test equipment [3].

Also, most Topology Management Systems allow the network operator to provide hints so that the system, for instance, in order that the system can ignore certain portions of the network. This makes it easier to discover relevant portions of the network more accurately.

Some service providers may run network discovery routines on a daily basis to discover any unauthorized changes to the network topology as a result of security intrusions or unplanned insertion of devices.

Moreover, network elements and computer systems have a variety of version information associated with them. For example, a workstation may have: Operation System, version 32, Ethernet Interface, version 5.4, TCP/IP Software, version 2.0 and SNMP Software, version 3.1. Since multiple engineers/network operators work on making changes to the network equipment, tracking the changes manually would be very tedious and error-prone. Configuration Management tools help automates the tracking of the changes. Configuration Management systems store the configurations in a database or LDAP server for easy access.

They also enable network operators to change configurations of the network elements as well as to roll back a change to a previous configuration, if required.

When a problem in the network occurs, network operators often search the Configuration Management database for clues that can help solve the problem.

2.2.4. Planning & Testing

Network Planning solutions help determine when a communication network needs an upgrade or additional equipment as well as to predict the impact of changes to a service provider’s network’s topology, configuration, traffic and technology. They provide simulation tools that help the network operators to project how growth in network traffic will affect the network performance. Based on the results and other planning activities, network operators can take countermeasures such as increase capacity.

Testing is an important activity in setting up a network or customer circuits. For simplicity in understanding the gamut of testing activities, let us divide them into the following:

1. Testing of existing network or a change
2. Integration testing of services configured for the customer
3. End-to-end testing of services configured for the customer

Testing the entire network platform - including the equipment, services and call quality – is critical for assessing the system prior to deployment and for service assurance in production environments [4].

Network testing tools usually simulate a production environment and generate synthetic voice, video and data traffic, which helps measure call/data quality, network performance, and the affects of any changes to the network or increasing traffic or adding new applications. These tests typically include tests like DNS, HTTP, RTP, Ping, etc. Also, during ongoing operations, these testing tools enable active testing of facilities.

Another form of testing is integration testing of network setup for the customer, i.e., routes, circuits, etc. configured for a customer. Network operators or field engineers perform integration testing of services upon completion of activations and other provisioning activities. Field engineers typically use equipment and network element specific applications to perform integration testing.

Upon completion of integration testing, field operations teams are notified to perform end-to-end testing. End-to-end testing includes testing of circuits, both within the CSP’s network as well as local access circuits between the CSP and the customer premises. Some service provider’s use craft access systems for the benefit of field technician’s access to their internal systems through a hand held terminal [5]. The hand held terminal helps them to access loop testing system and to view the complete test summary from remote locations.

2.3 Billing

IDC [6] defines Billing as: the processing and compiling of charges and enabling of revenue collection for network usage, feature transactions, and access charges of the services.Mediation systems collect network usage data from the network elements and convert to billable statistics.
The following figure depicts a simple Billing flow:

Traditionally, for phone calls, Call Detail Records (CDR) have been used to record the details of the circuit-switched phone call. CDR includes information on start time of call, end time of call, duration of call, originating and termination numbers. CDRs are stored until a billing cycle runs. For IP Based Services, a new standard is gaining acceptance called Internet Protocol Detail Record (IPDR). IPDR supports both voice and data.

Billing systems use mediation output to determine charges for the customers. It is also used to feed other downstream applications such as Fraud and Churn Management.

2.3.2 Rating

Rating systems calculate the charge for an individual call, IP usage event, etc. using the CDRs/IPDRs. Rating systems apply charges based on pre-configured pricing rules, applicable discounts and rebates from promotions.

This rating process has grown increasingly complex in recent years. In older times, it was solely a matter of taking the length of the call, assigning a price based on the mileage band (calculated by cross-referencing the prefix of the originating and terminating numbers in a table of values), and assigning discounts based on the time of day (peak, evening, night), day of the week, and holidays.

Modern rating systems can assign discounts based on calling circles, provide flexible rating plans based on size of accounts and increase switching costs [2]. These serve as strategic marketing tools but can be very complex to administer and operate.

2.3.3 Billing Systems

Billing systems aggregate rated calls, IP/data usage events, etc. and calculate customer invoices. In the United States, billing is usually performed once a month.

Billing systems combine rated records with prior balance information, payment records, recurring charges (such as line rentals), one-time fees (such as installation and service charges), promotions and discounts associated with the customer account, taxes and credits. Overnight billing batch jobs are among the largest batch environment at a CSP’s operating environment. Each customer is assigned a specific billing cycle.

According to Insight [2], the holy grails of the billing industry are unified billing and convergent billing. With unified billing, a customer gets a single bill for all services provided (or billed) by the service provider, appropriately rated, discounted, and taxed, and a single contact for inquiries and negotiation.

2.3.4. Interconnection Billing

In the competitive world of communications, service providers often tie-up with partners, in order to bundle their own products with their partners. This helps the service providers to provide attractive bundles of products and services. However, in order to successfully settle interconnect billing settlements an effective Interconnection Billing is required.

Interconnection Billing products support inter-working of a service provider’s billing systems with the corresponding systems of another service provider, based on interconnect agreements and contracts.

2.3.5. Revenue Assurance

Revenue Assurance & Fraud Management systems verify billing, detect and identify unauthorized usage of service provider network assets. Some of the kinds of frauds are Usage and Subscription.

Usage Fraud means that a customer uses the telecommunications network illegally. This is accomplished either by obtaining a service with no intent to pay or by obtaining unauthorized access to the network (i.e. “hacking” or “cracking”).

Fraud Management systems typically detect and prevent unauthorized access to a communications network by analyzing traffic patterns on the network. Some examples are provided in [8]:

One technique involves analyzing the average call duration or the number of calls placed to foreign countries to determine whether the traffic patterns are consistent with a subscriber's call history or pattern. If a call is inconsistent with the subscriber's call pattern profile, the subscriber is provided with a report of the abnormal call activity.
Other methods for dealing with the problem of unauthorized use involve automatically denying or blocking access to the network when abnormal use is detected to minimize the subscriber's financial loss.

Subscription fraud means that a customer obtains a service account by giving a false identity (name and/or SSN) or by giving a false address or false credit worthiness.

Detecting subscription fraud involves searching recent order and existing customer data for multiple orders and/or accounts with the same customer name, SSN, or service address.

Common subscription fraud patterns include:

Change of billing address within a few weeks of opening an account.
Substantial deviation of usage profile of a new user from an average new user.

Common techniques to control subscription based fraud include threshold based analysis, inference rules analysis, profile based analysis such as habitual user profiles and neural networks.

Fraud Management Systems typically read and store usage data from the service provider’s network switching equipment and allows queries to be executed against the data that detect suspicious usage patterns.

They also allow operators to review customer accounts that have suspicious activity, to track their investigation and record the final case resolution.

It should be noted that fraud is different from revenue leakage. Revenue leakage is characterized by the loss of revenues resulting from operational or technical loopholes where the resulting losses are sometimes recoverable and generally detected through audits or similar procedures [1]. Fraud, on the other hand, is characterized with theft by deception, typically characterized by evidence of intent where the resulting losses are often not recoverable and may be detected by analysis of calling patterns.

Another important class of Revenue Assurance tools includes Churn Management tools. Churn management is an important area for service providers that have subscription-based business - due to price wars, aggressive marketing and promotions from competing service providers, and customer’s expectations related to customer service.

Churn Management tools provide functions such as automated behavior analysis, forecasting and simulation, empirical profiling, churn metrics capture, that enable service providers to learn which customers are likely to leave and take appropriate countermeasures.

3. Conclusion

3.1 Summary

OSS/BSS systems and applications automate many of the day to day operations performed in a communications service provider’s operating environment. They optimize the time taken to perform these operations and make the business processes more efficient.

There are no all-encompassing OSS/BSS systems that can be installed, integrated, tested and allow the service providers to easily modernize their end-to-end operations functions.

Service providers, therefore, use all the different approaches: best-of-breed in some areas, off-the-shelf in some, and home-grown custom applications in the remaining areas, to modernize and optimize their operations.

More often than not, many of these OSS/BSS systems are integrated with the others in a point-to-point fashion, as part of discrete projects and programs, sponsored out of different business units. This leads to point-to-point integration of OSS/BSS systems unless the programs/projects are planned with a strategic goal.

A side effect of the difficulty in integrating the various OSS/BSS systems is many of the OSS/BSS systems in a service provider’s operating environment may not be integrated at all. For example, it is not unusual to find the following scenario: when a customer orders a new telephone line, the ordering system takes the details of a customer’s order, but a manual process is present to configure the telephone exchange using a switch management system. Details of the order entered in the Order Handling system is re-keyed manually by the technician into the Switch Management System – a process often referred to as “Swivel-Chair Integration”.

The article provided an overview of some of the core OSS/BSS areas in Order Fulfillment, Service Assurance and Billing.

3.2 References

[1] Who Makes What: OSS, http://www.lightreading.com/document.asp?doc_id=113052&print=true, Light Reading, Dec 2006
[2] The 2007 Telecommunications Industry Review, The Insight Research Corp., Dec 2006
[3] Eric Siegel, Architectural Overview of Network Management, The Burton Group, Oct 2005
[4] Arindam Banerjee, Network Management is the Key to the Success of Next-Generation Architecture, Yankee Group, Jan 2007
[5] Senthil K. Ramachandran, Order Fulfillment Core Processes and Pain Areas, TMFC 2122 White-paper
[6] Sterling Perrin et al., IDC's Service Provider Infrastructure Taxonomy, 2004, IDC
[7] Lars Andersson, OSS Solutions for Network Operators – white paper, 2002
[8] Telecommunications Fraud Detection Scheme, US Patent 5504810, http://www.patentstorm.us/patents/5504810/description.html, April 1996
[9] OSS Through Java Initiative, OSS/J Roadmap, TeleManagement Forum, Jan 2007
[10] Element Management Systems – Definition and Overview, Web ProForums, International Engineering Consortium (IEC)
[11] Divakara K. Udapa, TMN Telecommunications Management Network, McGraw Hill, ISBN:9780070658158, 1999
[12] Enhanced Telecom Operations map - eTOM: The Business Process Language of NGOSS, TeleManagement Forum
[13] Wikipedia, http://en.wikipedia.org/wiki/Operational_Support_Systems
[14] Elisabeth Rainge, Next-Generation OSS and Billing Market Taxonomy, IDC, Oct 2004
[15] Wikipedia, http://en.wikipedia.org/wiki/Fault_management
[16] Wikipedia, http://en.wikipedia.org/wiki/Management_information_base
[17] Network Management Basics, http://www.cisco.com/en/US/docs/internetworking/technology/handbook/NM-Basics.html
[18] Wikipedia, http://en.wikipedia.org/wiki/Network_planning_and_design
[19] Balan Nair et al., Method and system for planning a telecommunications network, United States Patent 5974127, http://www.freepatentsonline.com/5974127.html, Oct 1999
[20] Telecommunications Fraud Detection Scheme, US Patent 5504810, http://www.patentstorm.us/patents/5504810/description.html, April 1996
[21] Stephen Brown, Telecommunication Fraud Management, Jan 2005, http://www.waveroad.ca/ressources/Whitepaper_SB_Janvier2005.pdf
[22] Eric Siegel, Measuring Performance of Networks and Applications, The Burton Group, Feb 2007
[23] International Engineering Consortium (IEC), Tutorials, http://www.iec.org/online/tutorials/

Colin Powell on leadership

2010-03-15T18:25:00.000+05:30

For great leadership lessons, see "Colin Powell on leadership".

A proof that Microsoft technologies can be used for ultra-large-scale systems

2010-03-15T17:52:00.000+05:30

Here's a proof that Microsoft technologies can indeed be used to build ultra-large-scale systems: MySpace architecture.

Ultra-large-scale systems have the following characteristics, as Len Bass of Software Engineering Institure (SEI) describes here:

They are decentralized in terms of data, development, evolution, amd operational control
They have a wide variety of stake holders and therefore they have inherently conflicting and unknowable requirements
They continually add new functions and content
Software and hardware failure is a norm rather than the exception

Testing 1 million concurrent users on MySpace live site

2010-03-12T19:48:00.001+05:30

Here is Dan Bartow's account of how MySpace tested a live site with 1 million concurrent users.

Comparision of "wrapped"/"non-wrapped" conventions in Document/Literal (style/usage) Web Services

2010-03-03T19:02:00.000+05:30

For a good comparision on the bare/wrapped conventions while using Document/Literal Web services, see this post from Alexander Ananiev.

What every programmer must know about Unicode and Character Sets

2010-03-02T11:07:00.000+05:30

The following are great articles on the very important subject of Unicode and character sets.

Joel Spoesky's "The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)"
Mark Davis's "Forms of Unicode"

Craft, cooperative gaming and lessons from lean manufacturing

2010-02-25T17:59:00.002+05:30

Alistair Cockburn, in his thought-provoking publications titled "From Agile Development to the New Software Engineering", shows how software engineering is built on the following three legs and what to make of it. A few excerpts:

Craft - "Being in a craft profession brings with it certain expectations, such as lifelong learning, or “deepening” the proficiency in one’s craft. Programmers shouldn’t think that just because they once learned to program a computer, their programming skills are still sufficient. As craft professionals, programmers, project managers and others should always be looking to learn new tools, and techniques, and get better in the ones they have. [Cockburn]"
Cooperative gaming - "Every game and every situation are potentially different – there is no formula for winning the game. Different, even opposite strategies may be needed at any instant; The quality of a move in the game is not absolute, but rather, is only relative to how it improves the position of the team for its next move; The quality of community and communication among the team members matter enormously – they often make the difference between success and failure. [Cockburn]"
Lessons from lean manufacturing - "... people hand other people decisions; people wait on each other for decisions; and some people have a bigger backlog of decisions than they can handle at the moment. Examining the dependency network of decisions in play in an organization, we see that this network is very similar to the dependency network of parts in a manufacturing plant. Quite surprisingly, the same mathematics applies to the situation, and many of the same strategies apply: just-in-time, pull, continuous flow, and so on. [Cockburn]"

Software Engineering Method and Theory (SEMAT)

2010-02-25T17:24:00.002+05:30

Anyone who has worked in the software industry for a few years, in all likelihood, wouldn't find it hard to acknowledge that the state of practice in software development has a long way to go.

Visit Software Engineering Method and Theory (SEMAT) for a community working on developing a framework for improving the state of practice in software. Its heartening to see a list of great signatories on this initiative.

The SEMAT call for action states:

Software engineering is gravely hampered today by immature practices. Specific problems include:

The prevalence of fads more typical of fashion industry than of an engineering discipline.
The lack of a sound, widely accepted theoretical basis.
The huge number of methods and method variants, with differences little understood and artificially magnified.
The lack of credible experimental evaluation and validation.
The split between industry practice and academic research.

Visit here for SEMAT publications and references.

Data view as an architectural view

2010-02-25T12:04:00.002+05:30

For a nice roundup of data view as an architectural view, refer to Paulo Merson's tech note on the SEI site.

How we overvalue stuff

2010-02-24T17:47:00.002+05:30

Paul Graham on "Stuff". He says "Stuff has gotten a lot cheaper, but our attitudes toward it haven't changed correspondingly. We overvalue stuff." So true.

Enlivening discussion around static and dynamic types

2010-02-24T15:30:00.005+05:30

For a hilarious account of "Static-Typing Fundamentalist, the bearded and grizzled veteran of the Static/Dynamic Code Wars [Ted Neward] ", read Ted Neward's blog post on the comments posted on Scott Hanselman's intro to the C# dynamic types (don't miss the comments on Scott's post, either).

CWE's list of top 25 most dangerous programming errors

2010-02-24T13:51:00.002+05:30

Common Weakness Enumeration (CWE)'s study describing the top 25 programming errors that can open up security holes in your applications is here.

Examples of Interoperability Issues in SOAP-based Web services

2010-02-15T13:44:00.066+05:30

1- Mismatch in platform native data types

Data type compatibility is among the most important issue in Web services (SOAP based) interoperability. SOAP implementations convert XML schema types to language native data types during serialization and vice versa during deserialization. Different SOAP runtimes and underlying platforms/languages at sender and receiver end can therefore cause interoperability issues.

a) No direct native type maping. Some schema types such as xsd:unsignedshort, xsd:unsignedint xsd:unsignedlong do not always have a direct native type mappings on all languages. For example, in Java, unsigned types are not defined.

b) Difference in precisions. Different programming languages and platorms might support different precisions for XML schema defined floating point and decimal numbers such as xsd:decimal, xsd:double or xsd:float, leading to loss of precision during translation between xsd types to native types.
A good example of this issue is that of supporting Bigdecimal. XML Schema specification for decimal data types allows an arbitrary precision, upto 40 digits. Although Java BigDecimal also is defined as arbitrary-precision signed decimal numbers, it has an upper limit based on the underlying platform - Windows, Solaris, Linux, etc. The .NET implementation supports upto 29 digits.

Similarly, platforms use different precisions when interpreting native xsd:dateTime types. As per XML Schema specification, any number of digits after seconds may be coded into a dateTime data type. The Java platform supports a precision of upto three digits for the millisecond while .NET supports seven digits for the millisecond (can therefore support nanoseconds).

c) Null values in SOAP messages. Difference in how schema data types are mapped to native platform data types can cause issue in handling null values.
For example, lets say, a Java based Web service returns a date/time using java.util.Date or java.util.Calendar and a .NET consumer consumes the service. If the service returns a null, the .NET consumer fails, because in .NET System.DateTime is considered a value type (i.e., they reside in the stack) and value types cannot be a null. This is unlike Java where the two are classed as reference types (stored in the heap) and therefore can be null.

d) Processing byte arrays. SOAP allows exchanging data in the form of byte arrays (e.g., base64 encoded and placed on the XML document). Implementations may return inconsistent values due to a difference in native data types or other problems.

e) Processing nil and minOccurs='0'. These get handled differently in different platforms. For example, Java tools tend to serialize nulls explicitly as xsi:nil='true' unless told otherwise.

f) Weakly typed data structures: Some programming languages support Collection types such as hashmaps, vectors, sets, etc. Objects of these collection types, contains objects of other types. The receiving side may have trouble understanding SOAP messages containing such weakly-typed elements.
For example, an ArrayList may be taken as "anyType" in the XML schema, making it hard for the receiving side to map it to appropriate type.

2- Platform Idiosyncracies

a) Different interpretations of namespace declarations. Lets say you have two .NET services defined by unique URLs of the form: http://domain/branchname in the namespace declarations. The first service contains the URL: http://www.somecompany.com/ordering in its namespace declaration and the other contains the URL: http://www.somecompany.com/provisioning. Although both have unique URLs in the namespace declarations, a Java consumer consuming both services will face a naming conflict. This is because, the package name in generated stub code in Java is based on the domain name of the namespace, unlike in .NET where both domain name and the branch name are used to construct the namespace of the generated code.

b) Difference in platform generated proxies. Service proxies/stubs generated using one implemention do not typically work with other implementations. Moreover, code written against proxies generated using one version of the implementation may not work with another version of the implementation. For example, Code written against Axis 1.1 generated proxy may not work when the proxy is regenerated using Axis 1.2 (for example due to evolving xsd to Java type mappings).

c) Differences in expectations of type information. SOAP 1.1 defines a way to type each value explicitly using the xsi:type attribute. This is optional and typically not required if the type information is being expressed through the use of schema or any form of metadata exchange.
There were implementations that require explicit type information such as an earlier version of Apache SOAP. Other implementations such as .NET SOAP implementations didn't support it at all. This caused significant interoperability issues, since the implementation requiring type information would not be able to process a message from an implementation that doesn't include explicit type information.

3- SOAP and WSDL specific issues

a) Usage of mustUnderstand attribute. SOAP 1.1 defines a mustUnderstand attribute that can be specified for a SOAP header. The intent of this attribute is to ensure that if a SOAP processor/implementation doesn't understand it, it must fault. In other words, the recipient is required to understand and make use of the information supplied by the header. This helps in situations where a recipient can't perform its job unless it knows what to do with data provided by this header element.

SOAP 1.1 defines values for this attribute as 0 (false) or 1(true). The issues related to this attribute are:

Some toolkits have allowed the values "false" and "true". Latest SOAP versions may also be requiring implementations to support these.
Some SOAP runtimes do not support this attribute and rather offer work-arounds.

b) Out-of-sync SOAPAction headers. SOAP defines a "SOAPAction" request header. When bound to HTTP, SOAP mandates using this header, i.e., an HTTP client must issue this header field when issuing a SOAP HTTP request. According to http://www.w3.org/TR/2000/NOTE-SOAP-20000508/#_Toc478383528, this can be used to indicate the intent of the SOAP HTTP request. Legal values of this header's value are:

null or no value, meaning there is no indication of the intent of the message, e.g., SOAPAction:
Empty value, meaning the intent of the message is provided by the HTTP request URI, e.g., SOAPAction:""
An arbitrary URI, such as SOAPAction: "http://mydomain.com/whatever" or SOAPAction: "whatever.sdl". Implementations have also allowed unquoted URIs such as SOAPAction:http://mydomain/whatever.

The original intent of this header was to give hints about the content of the message to devices at lower levels (e.g., proxies, HTTP intermediaries, firewalls for filtering), without requiring the recipient to parse the XML. This allows implementations to identify, route or dispatch requests without having to parse the input request.

Some implementations also use it for application-level dispatch (e.g., routing the message to appropriate code).

Issues related to this header are:

Use of SOAPAction for transports other than HTTP, is not specified by SOAP 1.1., although some SOAP implementations do provide equivalent functionality for other transport protocols, leading to interoperability issues, since not all implementations implement or implement the same way.
SOAP implementations may interpret the "intent of the SOAP message" in different ways. This could be interpreted as the intended target for the message, the name of the target service, application level dispatch, and so forth. Therefore, a unless prior negotiation occurs between the sender and the receiver, full interoperability between implementations is hard to achieve.
Some SOAP implementations require that the values be quoted. However, some implementations do not quote their SOAPAction HTTP header values; this causes problems with implementations that assume SOAPAction values.
Also, if a receiver requires null-value SOAPAction, some HTTP client APIs have difficulty setting a null HTTP header value.
Some implementations are unable to parse the the Unicode byte order mark sent by other implementations.
Some SOAP implementations such as Apache SOAP do not support dispatching using the SOAPAction header. Instead they route based on the namespace URI of the first child element contained in the SOAP:Body element.

c) Identifying SOAP fault: Some implementations use only the HTTP status code to determine the presence of SOAP Fault. However, there are cases where the web infrastructure changes the HTTP status code.

d) WSDL Style and encoding. The WSDL "style" refers to the way WSDL describes the content of a SOAP message. It has two legal values - Document and RPC.

WSDL also has a "use" attribute (encoded/literal) that controls how web services messages are serialized (formatted according to a standard format).

SOAP 1.1 defines a SOAP encoding scheme (also refererred to as Section 5 encoding) that uses a set a rules based on the XML schema data types to encode programming language data types to XML and vice versa. However, the message doesn't conform to a particular schema. SOAP encoding is particularly useful while passing cyclic object graphs. For e.g., of one has an object graph that references a particular element many times, one would specify the element once and reference it at other places (similar to how one would do in a programming language such as Java). In literal encoding, on the other hand, one would have to repeat the element each time it is referenced. However, on the flip side, one cannot validate the message with an XML schema or transform it with XSLT.
SOAP also supports literal encoding, which means the application data such as objects, primitive types and methods are serialied to XML according to a particular schema, and therefore one can validate the message with an XML schema or transorm it using XSLT.

Key interoperability issues related to the value of the styles (RPC/Document) and uses (encoded/literal) attributes in WSDL are:

SOAP has no default encoding. Also, encoding can be defined through the encodingStyle attribute that identifies serialization rules. Some SOAP implementations default to SOAP encoding, while others default to using literal encoding. So, one side must be able to specify the encoding explicitly rather than use its default.
Based on the WSDL style and use attributes, these are these possible combinations, that implementations must agree to (or worked around) before two sides can interoperate: RPC/Encoded, RPC/Literal, Document/Encoded (almost never used) and Document/Literal (and Document/Literal Wrapped).

e) Mismatch using the "actor" attribute. A SOAP message sometimes passes through one or more intermediaries (such as a SOAP proxy service) before being processed. Therefore, SOAP headers may contain information intended solely for an intermediary, as well as that for the ultimate destination. If an "actor" attribute doesn't appear on the header element, it is assumed to be intended for the ultimate destination. SOAP requires that a header element intended for an intermediary is not passed on to the next SOAP processor in the path.

An actor is of the type anyURI and can have multiple URI identifiers. The ultimate destination can be designated either explicitly (targeting the ultimate destination using a URI identifier) or by the absence of the actor attribute, as described in the last paragraph. This leads to confusion and interoperability issues among different SOAP implementations.

Also, certain SOAP implementations put additional constraints on SOAP headers containing "actor" attributes. For example, .NET WSE2.0 server looks for the WS-Addressing header when the actor attribute is configured on the client, while the clients may not be requiring that, since its not required by SOAP specification.

For additional interoperability issues refer to http://www.xmethods.net/soapbuilders/interop.html.

Get started with Google App Engine

2010-02-08T16:49:00.004+05:30

Google App Engine allows you to deploy apps in Google's production infrastructure (for free upto a limit and then using pay-as-you-go model). Key advantages include:

You don't need to worry about handling scaling (delivery on a global scale and low latency), so you can concentrate on app's features and innovations
OpenSocial, Android devices, Facebook, etc. opens up your app to 100s of millions of target users
Typical to Google solutions, costs are low and fair

Get started with App Engine with Rick Hightower's three-part article series:
Google App Engine for Java: Part 1: Rev it up!
Google App Engine for Java: Part 2: Building the killer app
Google App Engine for Java: Part 3: Persistence and relationships
Ofcourse, you could also start here: http://code.google.com/appengine/

Management 2.0

2010-02-08T11:55:00.008+05:30

One of my favourites books on Management is "The Future of Management" by Gary Hamel and Bill Breen. A few excerpts from the book are as follows:

(To follow Gary via his blog, subscribe to his blog here.)

"Ask a group of your colleagues to describe the distinguishing characteristics of your company, and few are likely to mention adaptability and inventiveness. Yet if you ask them to make a list of the traits that differentiate human beings from other species, resilience and creativity will be near the top of the list. We see evidence of these qualities every day—in ourselves and in those around us. All of us know folks who’ve switched careers in search of new challenges or a more balanced life. We know people who’ve changed their consumption habits for the sake of the planet. We have friends and relatives who’ve undergone a spiritual transformation, or risen to the demands of parenthood, or overcome tragedy. Every day we meet people who write blogs, experiment with new recipes, mix up dance tunes, or customize their cars. As human beings, we are amazingly adaptable and creative, yet most of us work for companies that are not. In other words, we work for organizations that aren’t very human.
... The culprit? Management principles and processes that foster discipline, punctuality, economy, rationality, and order, yet place little value on artistry, nonconformity, originality, audacity, and élan. To put it simply, most companies are only fractionally human because they make room for only a fraction of the qualities and capabilities that make us human. Billions of people show up for work every day, but way too many of them are sleepwalking. The result: organizations that systematically underperform their potential.
.... The challenge: to reinvent our management systems so they inspire human beings to bring all of their capabilities to work every day."
"... Hierarchies are very good at aggregating effort, at coordinating the activities of many people with widely varying roles. But they’re not very good at mobilizing effort, at inspiring people to go above and beyond. When it comes to mobilizing human capability, communities outperform bureaucracies."
"... also hear virtually nothing that suggests the participants have hearts. Beauty. Truth. Love. Service. Wisdom. Justice. Freedom. Compassion. These are the moral imperatives that have aroused human beings to extraordinary accomplishment down through the ages. It is sad, then, that the vernacular of management has so little room for these virtues. Put simply, you are unlikely to get bighearted contributions from your employees unless they feel they are working toward some goal that encompasses bighearted ideals."

Architecture versus Design

2009-08-21T18:08:00.002+05:30

Rick Kazman and Amnon Eden's "Defining the Terms Architecture, Design, and Implementation" and "Architecture, Design and Implementation" provide a distinction based on Intension and Locality criteria.

Rick and Amnon argue that

Architectural specifications are intensional and non-local
Design specifications are intensional but local
Implementation specifications are both extensional and local

They demonstrate that:

Architectural styles are intensional and non-local
Design patterns are intensional and local
Implementations are extensional and local

A must read for architects and designers.

On Software Design

2009-08-19T14:11:00.004+05:30

Jack W. Reeves's classic essay, titled "What is Software Design" is a must read for software professionals.
Martin Fowler's "Is Design Dead?"
This is a good read on high level design and low level design.

What Every Web Programmer Needs To Know About Security

2009-08-12T19:18:00.001+05:30

Excellent presentations titled "What Every Web Programmer Needs To Know About Security".

Software Architecture Challenges in the 21st Century

2009-08-12T17:46:00.002+05:30

A workshop titled "Software Architecture Challenges in the 21st Century" took place some time back (June 2009).
Here is Diomidis D. Spinellis account of the talks. Presentations are located here.

Software reuse

2009-07-12T19:38:00.001+05:30

Reuse is a complex issue. It is about people and education, not just architecture, as Jeremy Meyer says here.

2009-05-08T11:05:00.005+05:30

A great blog post from Martin Fowler on a very difficult subject.

Nasa on complexity of flight software

2009-04-19T23:03:00.001+05:30

The document is here.

Excellent SOA paper

2005-12-07T17:58:00.000+05:30

Don Ferguson's paper on Services Oriented Architecture (SOA) published in the IBM Systems Journal (VOL 44, NO 4, 2005) can be found here.

Starting threads in EJBs

2005-12-06T12:12:00.000+05:30

Enterprise Java Bean (EJB) specs forbids EJB components from managing (starting, stopping, suspending, resuming, changing thread priority, changing name) threads.

According to Wayne Beaton, if you must use a thread inside an EJB, take a look at JSR 237 which defines Work Manager. It's not currently part of the J2EE specification, but it is supported by both BEA and IBM in their application server products.

Making things happen

2005-12-02T18:05:00.000+05:30

In this chapter excerpt from his book, The Art of Project Management, Scott Berkun illustrates the value of prioritization and how it helps to "make things happen".

During one of my earlier projects, that I worked on, at Hewlett-Packard, we would sometimes find ourselves in a tough argument. Half the team felt strongly for option 1 and the other half liked option 2. Pawan gupta, who was managing the team, would walk into the discussion, ask a few questions, align the discussion in line with the current goals in hand, and quickly get everyone to agree on one option.

Over 30 Cheatsheets

2005-12-02T16:29:00.000+05:30

I found Pete Freitag's post "Cheat Sheet Roundup - Over 30 Cheatsheets for developers" very useful.

You might want to check it out.

Components vs Services

2005-07-18T15:56:00.000+05:30

"If you need ACID - you need components, not services (in SOA). "

by Udi Dahan, http://udidahan.weblogs.us/archives/030112.html

Why smart people defend bad ideas

2005-06-30T19:44:00.000+05:30

Why smart people defend bad ideas -

http://www.scottberkun.com/essays/essay40.htm

Top programming languages

2005-06-23T12:10:00.000+05:30

An interesting read on the top OO languages -

http://rmh.blogs.com/weblog/2005/05/then_and_now_th.html

a += b is NOT the same as a = a+b

2004-12-01T15:29:00.000+05:30

Yeah, its not the same in Java.

int a = 2
float b = 3.2

a = a + b => leads to compile time error as expected. Explicit cast is required because of loss of preciiosn.

a += b => (Surprisingly!!) works fine - without even a warning!!

According to the Java Language Specification ----

A compound assignment expression of the form E1 op= E2 is equivalent to E1 = (T)((E1) op (E2)), where T is the type of E1, except that E1 is evaluated only once. Note that the implied cast to type T may be either an identity conversion (§5.1.1) or a narrowing primitive conversion (§5.1.3). For example, the following code is correct:

short x = 3;
x += 4.6;

and results in x having the value 7 because it is equivalent to:

short x = 3;
x = (short)(x + 4.6);

So its a feature of the language after all !!

And u thot they never preach u wrong.... Most of the books tell ya otherwise, don't they?

Grady Booch's Handbook of Software Architecture

2004-12-01T11:16:00.000+05:30

Handbook of Software Architecture by Grady Booch -

- http://www.booch.com/architecture/index.jsp

The site is not completely functional.

Booch's blog is at:

http://www-106.ibm.com/developerworks/blogs/dw_blog_rss.jspa?blog=317

2004-11-30T16:56:00.000+05:30

We know the rule on using GC.Collect() - Do not use it unless u must use it!

So when do we use it?

Go check this -

http://dotnetjunkies.com/WebLog/avnrao/archive/2004/11/30/34240.aspx

Java linkblog

2004-11-29T14:47:00.000+05:30

Found the following blog useful:

http://www.thauvin.net/linkblog/xml.jsp

Gives lots of links to Java related articles/software....

funny sig

2004-11-24T17:51:00.000+05:30

Found this funny sig in John Evdemon's blog...

"There are only 10 types of people in this world; those who understand binary and those who don't."