A "Cloudy" day at CSI Annual Convention 2009

I had a very interesting opportunity to be one of the speakers on the panel discussion on cloud computing at CSI Annual Convention 2009. As it turned out the entire day was "cloudy" with most topics and discussions being centered around cloud computing. Most people agreed that cloud is the next generation of computing but there are still doubts as to which form of cloud computing will take off. The conclusion is that there IS a lot of hype and when that has died down, the products and companies who solve real problems will survive. People who try to monetize the medium instead of the product, might end up failing. Here are some of the excerpts from the day.

The day started with a keynote address on "Cloud Computing - Challenges and Opportunities" by Girish Venkatachaliah from IBM. His take was that about 20% of IT will move to the cloud in next few years and currently its more hype than substance.

Dr. Srikanth Sunderrajan from Persistent gave a great talk on Google AppEngine, a Platform-as-a-Service offering. His company recently implemented a product on top of Google AppEngine. His take was that AppEngine lacks many features and is a strait-jacket environment with almost no flexibility. They had to write complex libraries to enable file-system like storage and ended up using Amazon EC2 to aid the short-comings of AppEngine. His take was that Google needs to open up the platform and be more like Amazon's cloud offerings. One good thing about AppEngine is that development and deployment is fast and easy.

The panel discussion on cloud computing included Monish Darda from Websym, Karan Gujral from BMC , Gireendra Kasmalkar from SQS, Vikram Rajkondwar from Microsoft, Samir Bodas from ICERTIS and yours truly. The discussions covered PaaS, IaaS, SaaS, testing for the cloud, how can startups leverage the cloud, managing the clouds and much more. Vikram's views which stemmed from his experience working on Microsoft Azure were extremely insightful.

Here are some of the take-away points from the discussion:

• The cloud phenomenon has been seeded due to the economies of scale. The cloud infrastructure providers use commodity hardware and use complex software to manage redundancy. The savings are passed on to the consumer making the cloud a very cost effective platform.
• Evolution of virtualization technologies has enabled cloud data centers to increase efficiency. All parts of the stack will be virtualized as we progress.
• Storage is an important aspect of the cloud. 3 copies of data are maintained by the cloud vendors so in terms of reliability to cost ratio, cloud storage is on par or cheaper than local storage. And unlimited storage is available on a completely pat as you go model.
• Cloud is very interesting medium for testing and QE since these phases are needed late in the SDLC and require investment in terms of hardware and provisioning. Clouds make it possible to do functional and scale testing without upfront investment.
• The most compelling use of cloud computing is when load and usage cannot be predicted. Cloud can be used to augment local data center - for scaling out when load exceeds certain levels. Such hybrid clouds will be the future of data centers. Another prime usecase is when loads are periodic - in-case of on-premise data centers this leads to low utilization and hence lesser ROI. Clouds can be provisioned as needed improving the ROI for such companies.
  
• Today even a 1-person startup can compete with Google and IBM in terms of infrastructure. If a good revenue model is in place, then startups can use the pay as you go model to their advantage. Companies like SmugMug, ElephantDrive has done just this to keep up with their phenomenal growth. Without clouds, their growth would have stymied as they would not have had scale out capability.
• The data center management companies will need to upgrade their products to manage the clouds. They will have to look at provisioning, job scheduling, profiling for the cloud along with the on-premise data center.
• Everyone agreed that on-premise data centers will never be replaced by the cloud. They will be augmented. A lot of web hosting will move to the cloud though.
  

The conclusion was that companies and consumers should try to look through the hype and try to identify solutions that actually solve their problems. Every little software when provided as Software-as-a-Service does not become a better solution. If you find your sweet spot in the cloud, you are poised for phenomenal growth.

Talk on Lustre at FOSS.in/2008

FOSS.IN is one of the world’s largest Free and Open Source Software (FOSS) events, held annually in India. The event is highly focussed on FOSS development and contribution. Over the years, it has attracted thousands of participants, and the speaker roster reads like a “Who’s Who” of FOSS contributors from across the world.

Last year I had the privilege of giving a talk on "Lustre: A Scalable Clustered Filesystem" at this event. This is one of the few events with a very techie agenda and I had some interesting discussions with the delegates. The breakout sessions where hackers sit together and actually code up a feature is really cool. Not many events would have people actually coding!

Here is my presentation. It describes the architecture of Lustre - a distributed, clustered filesystem which runs on 7 of the top 10 supercomputers. It goes on to describe some of the cutting-edge features that are being planned for future Lustre releases.

Inspirations from TiECon Delhi 2009

It isn't often that you get a chance to rub shoulders with industry leaders and successful entrepreneurs. TiECon gives you a chance to connect and interact with founders of successful companies, venture capitalists and budding entrepreneurs. It was great to be in presence of people with amazing clarity of thought and expression.

As I was contemplating what is the best way to structure this blog, I remembered attending some of the panel discussions where certain quotes and thoughts just resonated with my mind. It's like you have a gut feel of certain things but when they are put in perfect, concise words it becomes easy to put it into action. So the best way to express what I saw and learnt is in the form of quotes that I gathered personally or in discussions.

• "Entrepreneurship is a difficult word to define, an entrepreneur has a difficult path to choose and a difficult path to tread. Many succeed and many fail. An entrepreneur who trips and falls down once; if he is a true entrepreneur will pick himself up and walk the same path or a different path with greater determination. An entrepreneur chases a dream, pursues an idea, and seeks a goal. So I think there is much to be said about entrepreneurship and any organization which promotes entrepreneurship rather than simple businesses." Mr. P. Chidambaram, Home Minister, Ministry of Home Affairs, Government of India

• "Good entrepreneurs react differently to tough times" Deep Kalra, Founder & CEO, MakeMyTrip.com

• "Thinking how soon I can 'breakeven' is a big fallacy. We need to think how will we scale up" Achal Ghai, Managing Director, Avigo Capital

• "When the brand is in experimental stage, even spam works, especially in India" Manish Vij, Co-Founder and Business Head, Quasar Media

• "80-90% of requests for venture capital get rejected due to lack of a marketing plan" Achal Ghai, Managing Director, Avigo Capital

• "Only when you have done enough 'pilots' and have customers who can be brand ambassadors should you think of doing advertising" Manish Vij, Co-Founder and Business Head, Quasar Media

• "For your core team you need like minded people and people who can work with equity" Dhruv Shringi, CEO & Co Founder, Yatra.com

• "A part of your skill as an entrepreneur is to be a good salesman" Yashish Dahiya, Co Founder & CEO, Policy Bazaar.com

• "Its better to be No.1 in a niche market than No.20 in a large market"

• "You must be able to state your core value proposition in a single sentence"

• "Be non-conforming"

• "Leadership is about action not position"

Compression, encryption and deduplication

When you are doing backups or archival - compression and encryption is a must. Compression is necessary to save precious bandwidth while encryption is necessary when you store your data outside the boundaries of your data center/home. Now considering that you are archiving data for long periods, you would want to deduplicate it to reduce the amount of storage. But encryption/compression are quite incompatible with deduplication. Deduplication tries to find same blocks in the data set so that the blocks can be shared and storage space can be saved. But even a small change in your data could cause the encryption/compression algorithms to produce very different outputs, fooling the poor little deduplication engine. Deduplication loves patterns in the data and good encryption algorithms try hard to remove patterns from the data for better security.

Some googling led me to rsyncrypto and rsyncable-gzip - where the rsync algorithm is modified to be compression/encryption friendly. rsync has an excellent algorithm with which only the changed parts of the data need to be sent over the network for syncing data sets.

rsyncable-gzip is a patch to gzip which cause the compression to be done in chunks rather than processing the entire file in one go. This localizes changes within the compressed binary allowing rsync to do a better job. This can lead to lower compression ratios in some cases.

rsyncrypto modifies the standard encryption schema by localizing the effects of encryption to keep the side-effect changes minimal. This again allows rsync to work much more efficiently. Again this may reduce the efficiency of the encryption algorithm but it will still be good enough for most use cases.

Now obviously this problem has an easier solution - deduplicate first, then compress, then encrypt. But this flow may not be possible always.

Rackspace CloudFiles

CloudFiles is a cloud storage offering from Rackspace. You can use it for archiving and backing up your web-based storage data. The data is replicated across 3 locations giving you excellent redundancy.

I tried out their offering and here are some of my notes:

- Max file size of 5GB - most cloud storage vendors have these limits mostly due to protocol limitations
- Along with CloudFiles they provide Content Distribution Network(CDN) for distributing data across their data centers around the world.
- 15 cents/GB/month storage cost. 8 cents inbound and 22 cents outbound/GB
- APIs in multiple languages - PHP, Java, C#, Python, Ruby
- They have a "browser panel" which can be used to upload files and distribute using CDN. - Mozilla extension + Mac app to access CloudFiles storage is available. Not maintained by Rackspace though and *not* a backup tool, its just a front-end.
- Provides SSL for security
- Tokens expire every 24 hours and clients need to reconnect. Every operation must use a authentication token.
- FireUploader simulates a hierarchical file system structure which is not possible with CloudFiles.
- Cyberduck used on Mac is a GPL software and a good frontend for CloudFiles

- The only way user can access the content from another account is if they share their username/API access key or a session token.
- Files are called "objects" in Rackspace lingo. Data is saved as-is without compression/encryption. It does support metadata in form of key-value pairs so you can "tag" files and organize your data. Metadata is limited to 4KB and max of 90 individual tags can be stored.
- You can enable CDN on certain containers. Each CDN enabled container has a unique Uniform Resource Locator(URL). For example a CDN enabled container named "photos" can be referenced as http://cdn.cloudfiles.mosso.com/c3131 - if this container has an image named "baby.jpg" then that image can be served through LimeLight Networks CDN with the URL of http://cdn.cloudfiles.mosso.com/c3131/wow1.jpg. This is how we can enable SaaS applications to use data stored on our customer's accounts.
- Code snippets and documentation of APIs is good and code examples are available.
- The APIs are Restful i.e. Representational State Transfer protocol.

- You need to do a GET call on the account to get a list of all containers in the account. You can get this information in JSON or XML format as well.
- Max of 10000 container names can be returned at a time. You can make a continuation call with a "marker" if you want to retrieve later containers.
- HEAD call is used to find out number of containers in the account and the number of used bytes
- GET operation on a container is used to list objects in the container.
- Pseudo hierarchical folders/directories
- Users will be able to simulate a hierarchical structure in Cloud Files by following a few guidelines. Object names must contain the forward slash character ‘/’ as a path element separator and also create “directory marker” Objects, then they will be able to traverse this nested structure with the new “path” query parameter. This can best be illustrated by example: For the purposes of this example, the Container where the Objects reside is called “backups”. All Objects in this example start with a prefix of “photos” and should NOT be confused with the Container name.
In the example, the following “real” Objects are uploaded to the storage system with names representing their full filesystem path.

photos/animals/dogs/poodle.jpg
photos/animals/dogs/terrier.jpg
photos/animals/cats/persian.jpg
photos/animals/cats/siamese.jpg
photos/plants/fern.jpg
photos/plants/rose.jpg
photos/me.jpg

To take advantage of this feature, the “directory marker” Objects must also be created to represent the appropriate directories. The following additional Objects need to be created. A good convention would be to create these as zero or one byte files with a Content-Type of “application/directory”.
photos/animals/dogs
photos/animals/cats
photos/animals
photos/plants
photos

Now issuing a GET request against the Container name coupled with the “path” query parameter of the directory to list can traverse these “directories”. Only the request line and results are depicted below excluding other request/response headers.

GET /v1/AccountString/backups?path=photos HTTP/1.1

photos/animals
photos/cats
photos/me.jpg

To traverse down into the “animals” directory, specify that path. GET /v1/AccountString/backups?path=photos/animals

photos/animals/dogs
photos/animals/cats

By combining this “path” query parmater with the “format” query parameter, users will be able to easily distinguish between virtual folders/directories by Content-Type and build interfaces that allow traversal of the pseudo-nested structure.

- DELETE call on container will not succeed if it has objects
- HEAD operation on object is used to retrieve object metadata and standard HTTP headers
- GET call is used to used to retrieve object data. It supports headers like If-Match, If-None-Match, If-Modified-Since, If-Unmodified-Since. It is possible to get a range of bytes as well.
- PUT call is used to write or overwrite an object's metadata and content. End-to-end data integrity can be ensured by including an MD5 checksum of your objects data in the Etag header.
- Chunked requests can be sent if you do not know the size of the object you are PUT'ing but total size must be <5gb>

Short-comings mentioned by Rackspace themselves:
- cannot mount or map CloudFiles account as a network drive.
- files cannot be modified so block level changes cannot be done.
- containers cannot be nested
- No ACLs for security

cURL is a command line tool available on most UNIX environments and it allows you to transmit and receive HTTP requests and responses from the command-line or from within a shell script. So you can work with the ReST API directly instead of using client APIs. I used the cURL tool to test out the calls provided by CloudFiles.

auto-login using rsh

I know that rsh is deprecated but it is used all over the place in the test harness for the product I am working on. I needed to enable auto-login for rsh and rcp but Google didn't give definitive answers. So here is how you should do it:

- Install rsh-server RPM on the machines which will be communicating
- In /etc/xinetd/{rsh, rexec, rlogin} change "disable = yes" to "disable = no"
- Add rsh, rexec and rlogin to /etc/securetty
- Restart xinetd service:
service xinetd restart
- Add "hostname username" combinations in ~/.rhosts file. For example if you want to allow root@garfield access then add "garfield root".

Fun with assembly level debugging

Investigations into a failed regression test revealed that some allocations were breaking page boundaries and when these allocations were passed as buffers to bio's - the read operation failed. The reason for the failure was that DASD requires the bv_len to be on blocksize boundaries whereas if the earlier allocation was breaking across page boundaries without blocksize alignment. It was suspected that the failure was in the dasd_eckd module in dasd_eckd_build_cp():

rq_for_each_bio(bio, req) {
bio_for_each_segment(bv, bio, i) {
if (bv->bv_len & (blksize - 1))
/* Eckd can only do full blocks. */
return ERR_PTR(-EINVAL);

We needed to make sure that this was the actual point of failure but since we cannot modify the kernel on IBM zSeries machines, we needed to use console debugging to confirm the suspicion.

Firstly install kernel-debuginfo package which you will need for debugging.

* Get disassembly code of the module using objdump

objdump -d /lib/modules/2.6.9-67.EL/kernel/drivers/s390/block/dasd_eckd_mod.ko > dasd_eckd_dis

0000000000001ef0 :
1ef0: eb 6f f0 48 00 24 stmg %r6,%r15,72(%r15)
1ef6: c0 d0 00 00 00 00 larl %r13,1ef6
1efc: b9 04 00 ef lgr %r14,%r15
1f00: a7 fb fe 40 aghi %r15,-448
1f04: c0 c0 00 00 00 00 larl %r12,1f04
1f0a: e3 e0 f0 98 00 24 stg %r14,152(%r15)
1f10: e3 30 f0 e0 00 24 stg %r3,224(%r15)
.....
.....
2134: a7 0a ff ff ahi %r0,-1
2138: a7 aa 00 09 ahi %r10,9
213c: 18 51 lr %r5,%r1
213e: e3 e0 c0 00 00 04 lg %r14,0(%r12)
2144: e3 90 f1 18 00 04 lg %r9,280(%r15)
214a: 58 20 90 08 l %r2,8(%r9)
214e: 18 12 lr %r1,%r2
2150: 18 32 lr %r3,%r2
2152: 14 10 nr %r1,%r0
2154: 88 30 a0 00 srl %r3,0(%r10)
* 2158: a7 74 04 53 jne 29fe
215c: e3 10 90 00 00 04 lg %r1,0(%r9)
.....
.....
29fe: a7 29 ff ea lghi %r2,-22
2a02: a7 f4 06 1b j 3638

The line with the '*' was the suspected line which was doing the bv_len alignment check. The jump from this line to dasd_eckd_build_cp+0x29fe confirmed this. EINVAL = 22 and after loading the return value the function returns.

Now while setting breakpoints, remember not to use addresses or offsets from the objdump(if you do remember to subtract start addr of the function from the offset of the instruction). Or better yet you can use the disassembly output from GDB:

$ gdb /lib/modules/2.6.9-67.EL/kernel/drivers/s390/block/dasd_eckd_mod.ko

(gdb) disassemble dasd_eckd_build_cp
Dump of assembler code for function dasd_eckd_build_cp:
0x0000000000001ef0 : stmg %r6,%r15,72(%r15)
0x0000000000001ef6 : larl %r13,0x1ef6
0x0000000000001efc : lgr %r14,%r15
0x0000000000001f00 : aghi %r15,-448
0x0000000000001f04 : larl %r12,0x1f04
0x0000000000001f0a : stg %r14,152(%r15)
0x0000000000001f10 : stg %r3,224(%r15)
0x0000000000001f16 : stg %r2,216(%r15)
0x0000000000001f1c : mvi 359(%r15),134
0x0000000000001f20 : lg %r1,16(%r3)
0x0000000000001f26 : tml %r1,1
0x0000000000001f2a : mvc 232(8,%r15),88(%r2)
0x0000000000001f30 : je 0x1f40
0x0000000000001f34 : tml %r1,1
0x0000000000001f38 : mvi 359(%r15),133
0x0000000000001f3c : je 0x29fe
0x0000000000001f40 : lg %r2,232(%r15)
.....
.....

Remember that the offsets here are in decimal not in hex.

0x00000000000029fe : lghi %r2,-22

OK, so as you can see we are jumping to 0x29fe which is the above instruction for which the offset is 2830. So find out the address for dasd_eckd_build_cp from kallsyms

cat /proc/kallsyms > kallsyms.

Here it is 0x000000006088f27c, so we place a breakpoint on /x 0x000000006088f27c+ 2830

Go to the CP console and all the rest is to be done here:

#CP TR I PSWA 6088fd8a

- this will set a trace point at given address.

The reason we should not set a tracepoint on the above jump instructions is because we see many false positives and that makes debugging impossible.

You can query all tracepoints using #CP Q TR.

When you hit a tracepoint, you can do the following to continue #CP B

If you want to single step: #CP TR I
followed by #CP B
Each "#CP B" will take you forward one instruction.

When you want to end tracing and remove all traces do #CP TR END

If you want to break within a function address range:
#CP TR I R .

eg. #CP TR I R 6088f27c.200

To display registers you can use:
#CP D G - group registers
#CP D X - control registers
#CP D AR - access registers
#CP D PSW - display PSW

After insmod'ing a reproducer module, with the trace on - we hit the instruction which returns EINVAL thereby confirming the problem.