Compression, encryption and deduplication

When you are doing backups or archival - compression and encryption is a must. Compression is necessary to save precious bandwidth while encryption is necessary when you store your data outside the boundaries of your data center/home. Now considering that you are archiving data for long periods, you would want to deduplicate it to reduce the amount of storage. But encryption/compression are quite incompatible with deduplication. Deduplication tries to find same blocks in the data set so that the blocks can be shared and storage space can be saved. But even a small change in your data could cause the encryption/compression algorithms to produce very different outputs, fooling the poor little deduplication engine. Deduplication loves patterns in the data and good encryption algorithms try hard to remove patterns from the data for better security.

Some googling led me to rsyncrypto and rsyncable-gzip - where the rsync algorithm is modified to be compression/encryption friendly. rsync has an excellent algorithm with which only the changed parts of the data need to be sent over the network for syncing data sets.

rsyncable-gzip is a patch to gzip which cause the compression to be done in chunks rather than processing the entire file in one go. This localizes changes within the compressed binary allowing rsync to do a better job. This can lead to lower compression ratios in some cases.

rsyncrypto modifies the standard encryption schema by localizing the effects of encryption to keep the side-effect changes minimal. This again allows rsync to work much more efficiently. Again this may reduce the efficiency of the encryption algorithm but it will still be good enough for most use cases.

Now obviously this problem has an easier solution - deduplicate first, then compress, then encrypt. But this flow may not be possible always.

Rackspace CloudFiles

CloudFiles is a cloud storage offering from Rackspace. You can use it for archiving and backing up your web-based storage data. The data is replicated across 3 locations giving you excellent redundancy.

I tried out their offering and here are some of my notes:

- Max file size of 5GB - most cloud storage vendors have these limits mostly due to protocol limitations
- Along with CloudFiles they provide Content Distribution Network(CDN) for distributing data across their data centers around the world.
- 15 cents/GB/month storage cost. 8 cents inbound and 22 cents outbound/GB
- APIs in multiple languages - PHP, Java, C#, Python, Ruby
- They have a "browser panel" which can be used to upload files and distribute using CDN. - Mozilla extension + Mac app to access CloudFiles storage is available. Not maintained by Rackspace though and *not* a backup tool, its just a front-end.
- Provides SSL for security
- Tokens expire every 24 hours and clients need to reconnect. Every operation must use a authentication token.
- FireUploader simulates a hierarchical file system structure which is not possible with CloudFiles.
- Cyberduck used on Mac is a GPL software and a good frontend for CloudFiles

- The only way user can access the content from another account is if they share their username/API access key or a session token.
- Files are called "objects" in Rackspace lingo. Data is saved as-is without compression/encryption. It does support metadata in form of key-value pairs so you can "tag" files and organize your data. Metadata is limited to 4KB and max of 90 individual tags can be stored.
- You can enable CDN on certain containers. Each CDN enabled container has a unique Uniform Resource Locator(URL). For example a CDN enabled container named "photos" can be referenced as http://cdn.cloudfiles.mosso.com/c3131 - if this container has an image named "baby.jpg" then that image can be served through LimeLight Networks CDN with the URL of http://cdn.cloudfiles.mosso.com/c3131/wow1.jpg. This is how we can enable SaaS applications to use data stored on our customer's accounts.
- Code snippets and documentation of APIs is good and code examples are available.
- The APIs are Restful i.e. Representational State Transfer protocol.

- You need to do a GET call on the account to get a list of all containers in the account. You can get this information in JSON or XML format as well.
- Max of 10000 container names can be returned at a time. You can make a continuation call with a "marker" if you want to retrieve later containers.
- HEAD call is used to find out number of containers in the account and the number of used bytes
- GET operation on a container is used to list objects in the container.
- Pseudo hierarchical folders/directories
- Users will be able to simulate a hierarchical structure in Cloud Files by following a few guidelines. Object names must contain the forward slash character ‘/’ as a path element separator and also create “directory marker” Objects, then they will be able to traverse this nested structure with the new “path” query parameter. This can best be illustrated by example: For the purposes of this example, the Container where the Objects reside is called “backups”. All Objects in this example start with a prefix of “photos” and should NOT be confused with the Container name.
In the example, the following “real” Objects are uploaded to the storage system with names representing their full filesystem path.

photos/animals/dogs/poodle.jpg
photos/animals/dogs/terrier.jpg
photos/animals/cats/persian.jpg
photos/animals/cats/siamese.jpg
photos/plants/fern.jpg
photos/plants/rose.jpg
photos/me.jpg

To take advantage of this feature, the “directory marker” Objects must also be created to represent the appropriate directories. The following additional Objects need to be created. A good convention would be to create these as zero or one byte files with a Content-Type of “application/directory”.
photos/animals/dogs
photos/animals/cats
photos/animals
photos/plants
photos

Now issuing a GET request against the Container name coupled with the “path” query parameter of the directory to list can traverse these “directories”. Only the request line and results are depicted below excluding other request/response headers.

GET /v1/AccountString/backups?path=photos HTTP/1.1

photos/animals
photos/cats
photos/me.jpg

To traverse down into the “animals” directory, specify that path. GET /v1/AccountString/backups?path=photos/animals

photos/animals/dogs
photos/animals/cats

By combining this “path” query parmater with the “format” query parameter, users will be able to easily distinguish between virtual folders/directories by Content-Type and build interfaces that allow traversal of the pseudo-nested structure.

- DELETE call on container will not succeed if it has objects
- HEAD operation on object is used to retrieve object metadata and standard HTTP headers
- GET call is used to used to retrieve object data. It supports headers like If-Match, If-None-Match, If-Modified-Since, If-Unmodified-Since. It is possible to get a range of bytes as well.
- PUT call is used to write or overwrite an object's metadata and content. End-to-end data integrity can be ensured by including an MD5 checksum of your objects data in the Etag header.
- Chunked requests can be sent if you do not know the size of the object you are PUT'ing but total size must be <5gb>

Short-comings mentioned by Rackspace themselves:
- cannot mount or map CloudFiles account as a network drive.
- files cannot be modified so block level changes cannot be done.
- containers cannot be nested
- No ACLs for security

cURL is a command line tool available on most UNIX environments and it allows you to transmit and receive HTTP requests and responses from the command-line or from within a shell script. So you can work with the ReST API directly instead of using client APIs. I used the cURL tool to test out the calls provided by CloudFiles.

auto-login using rsh

I know that rsh is deprecated but it is used all over the place in the test harness for the product I am working on. I needed to enable auto-login for rsh and rcp but Google didn't give definitive answers. So here is how you should do it:

- Install rsh-server RPM on the machines which will be communicating
- In /etc/xinetd/{rsh, rexec, rlogin} change "disable = yes" to "disable = no"
- Add rsh, rexec and rlogin to /etc/securetty
- Restart xinetd service:
service xinetd restart
- Add "hostname username" combinations in ~/.rhosts file. For example if you want to allow root@garfield access then add "garfield root".

Fun with assembly level debugging

Investigations into a failed regression test revealed that some allocations were breaking page boundaries and when these allocations were passed as buffers to bio's - the read operation failed. The reason for the failure was that DASD requires the bv_len to be on blocksize boundaries whereas if the earlier allocation was breaking across page boundaries without blocksize alignment. It was suspected that the failure was in the dasd_eckd module in dasd_eckd_build_cp():

rq_for_each_bio(bio, req) {
bio_for_each_segment(bv, bio, i) {
if (bv->bv_len & (blksize - 1))
/* Eckd can only do full blocks. */
return ERR_PTR(-EINVAL);

We needed to make sure that this was the actual point of failure but since we cannot modify the kernel on IBM zSeries machines, we needed to use console debugging to confirm the suspicion.

Firstly install kernel-debuginfo package which you will need for debugging.

* Get disassembly code of the module using objdump

objdump -d /lib/modules/2.6.9-67.EL/kernel/drivers/s390/block/dasd_eckd_mod.ko > dasd_eckd_dis

0000000000001ef0 :
1ef0: eb 6f f0 48 00 24 stmg %r6,%r15,72(%r15)
1ef6: c0 d0 00 00 00 00 larl %r13,1ef6
1efc: b9 04 00 ef lgr %r14,%r15
1f00: a7 fb fe 40 aghi %r15,-448
1f04: c0 c0 00 00 00 00 larl %r12,1f04
1f0a: e3 e0 f0 98 00 24 stg %r14,152(%r15)
1f10: e3 30 f0 e0 00 24 stg %r3,224(%r15)
.....
.....
2134: a7 0a ff ff ahi %r0,-1
2138: a7 aa 00 09 ahi %r10,9
213c: 18 51 lr %r5,%r1
213e: e3 e0 c0 00 00 04 lg %r14,0(%r12)
2144: e3 90 f1 18 00 04 lg %r9,280(%r15)
214a: 58 20 90 08 l %r2,8(%r9)
214e: 18 12 lr %r1,%r2
2150: 18 32 lr %r3,%r2
2152: 14 10 nr %r1,%r0
2154: 88 30 a0 00 srl %r3,0(%r10)
* 2158: a7 74 04 53 jne 29fe
215c: e3 10 90 00 00 04 lg %r1,0(%r9)
.....
.....
29fe: a7 29 ff ea lghi %r2,-22
2a02: a7 f4 06 1b j 3638

The line with the '*' was the suspected line which was doing the bv_len alignment check. The jump from this line to dasd_eckd_build_cp+0x29fe confirmed this. EINVAL = 22 and after loading the return value the function returns.

Now while setting breakpoints, remember not to use addresses or offsets from the objdump(if you do remember to subtract start addr of the function from the offset of the instruction). Or better yet you can use the disassembly output from GDB:

$ gdb /lib/modules/2.6.9-67.EL/kernel/drivers/s390/block/dasd_eckd_mod.ko

(gdb) disassemble dasd_eckd_build_cp
Dump of assembler code for function dasd_eckd_build_cp:
0x0000000000001ef0 : stmg %r6,%r15,72(%r15)
0x0000000000001ef6 : larl %r13,0x1ef6
0x0000000000001efc : lgr %r14,%r15
0x0000000000001f00 : aghi %r15,-448
0x0000000000001f04 : larl %r12,0x1f04
0x0000000000001f0a : stg %r14,152(%r15)
0x0000000000001f10 : stg %r3,224(%r15)
0x0000000000001f16 : stg %r2,216(%r15)
0x0000000000001f1c : mvi 359(%r15),134
0x0000000000001f20 : lg %r1,16(%r3)
0x0000000000001f26 : tml %r1,1
0x0000000000001f2a : mvc 232(8,%r15),88(%r2)
0x0000000000001f30 : je 0x1f40
0x0000000000001f34 : tml %r1,1
0x0000000000001f38 : mvi 359(%r15),133
0x0000000000001f3c : je 0x29fe
0x0000000000001f40 : lg %r2,232(%r15)
.....
.....

Remember that the offsets here are in decimal not in hex.

0x00000000000029fe : lghi %r2,-22

OK, so as you can see we are jumping to 0x29fe which is the above instruction for which the offset is 2830. So find out the address for dasd_eckd_build_cp from kallsyms

cat /proc/kallsyms > kallsyms.

Here it is 0x000000006088f27c, so we place a breakpoint on /x 0x000000006088f27c+ 2830

Go to the CP console and all the rest is to be done here:

#CP TR I PSWA 6088fd8a

- this will set a trace point at given address.

The reason we should not set a tracepoint on the above jump instructions is because we see many false positives and that makes debugging impossible.

You can query all tracepoints using #CP Q TR.

When you hit a tracepoint, you can do the following to continue #CP B

If you want to single step: #CP TR I
followed by #CP B
Each "#CP B" will take you forward one instruction.

When you want to end tracing and remove all traces do #CP TR END

If you want to break within a function address range:
#CP TR I R .

eg. #CP TR I R 6088f27c.200

To display registers you can use:
#CP D G - group registers
#CP D X - control registers
#CP D AR - access registers
#CP D PSW - display PSW

After insmod'ing a reproducer module, with the trace on - we hit the instruction which returns EINVAL thereby confirming the problem.

Overwhelmed

Last week on 17th April was our first marriage anniversary. The last year has easily been the best of my life - filled with love, affection, laughter and lots of success for both of us. We definitely planned to celebrate it in a big way and Goa was an ideal place for a romantic getaway.

We stayed a Lemon Tree Amarante hotel and the stay was comfortable and leisurely. With the beach just about 500 meters away and an awesome pool(with a pool bar), this is just about the ideal Goa hotel. It was great fun to watch DVDs on the LCD TV provided in each room.

The Goa beaches were full of life with lots of shacks providing awesome cocktails and really tasty food. There was a cool breeze soothing us all through the day. Chilling out in each others company on the beach for hours together was really memorable.


The icing on the cake were the cute and brilliant gifts given to me by Komal. She gave me all of four gifts, each one better than the other:


A very nice photo of ours engraved in a crystal. The platform on which it is elevated beautifies it with different colors being dispersed.

XMini capsule speakers

A very compact and cute speaker for our iTouch with surprisingly good sound quality for its size. It has a cool battery life of 8 hours and can also be charged through USB port.






A magnificent diamond studded gold pendant with our initials. This design is Komal's handiwork - really creative.







The tale of our life as a conglomeration of pictures. Komal blew me away with her artistic skills and creativity with this one. She must have spent long hours making this amazing piece of work. Though to be honest, its beauty is not reflected in these pics - it looks much more beautiful in real life.








Cute collage of my pics when I was little and was actually sweet.










Collage of Komal's college life pics. Looking beautiful wifu!












A great pic with awesome expressions!











Dinner at the "Beach House" restaurant at the Taj. The ambience was exhilarating - overlooking the sea, with a beautiful breeze and awesome food. Gotta go and stay at Taj sometime - its a different experience which cannot be put into words.

Oracle buys Sun

That Sun was putting itself out in the market and someone would step up to pay a decent price was written in stone. And finally it has happened with Oracle buying Sun.

Personally I think this is a good deal for Sun. Before the talks with IBM, Sun stock was trading around $3-4 and now they are being sold at $9.50 per share. That is a very good deal for the investors and Sun employees. IBM and Sun had many similar products and would certainly have resulted in job losses. Oracle is flush with money and will surely continue to invest in Sun products since there is little overlap.

Some of my interesting thoughts about this deal:

• Oracle must be loving the thought of owning Java since its the crucial glue in their software offerings.
• Contrary to popular perception, Oracle won't kill MySQL - instead they will continue investing in it so that they get an entry in the low-end database market. Customers can then be upgraded to Oracle as they grow - a win-win situation for Oracle. Note that since MySQL is open-source if Oracle won't develop it, someone else surely will.
• Oracle now has access to the entire storage stack - Oracle database running on Java, ZFS, Sun hardware, Oracle backed Linux. That is a pretty strong offering.
• Every M&A deal brings with it some amount of layoffs, resignations, cultural changes but its this very churn that Sun has needed since a long time. Majority of the people will come of smiling eventually.

Don't vote for independents and regional parties

I have a fervent plea to all voters - please *DO NOT* vote for independents or regional parties for next 10 years or so. Even if these candidates may be honest, hard-working, result-oriented - please keep them out of power.

I know this logic seems inverted and downright stupid for the uninitiated. But think about it - the only way we can have a strong government is if one of the national parties receives a majority. Only then can the government go forward with development agenda instead of catering to the whims of individuals or factions. I personally don't care if its the BJP or the Congress which gets the majority - come to think of it both of them have tried to do a good job since 1998. BJP did this nation great service by going ahead with the nuclear tests and opening up the economy - in fact it laid the ground-work for the boom we saw post 2004. Congress also tried to make positive reforms in various sectors like insurance, telecom, nuclear deal, etc. but the fear of the Left kept the Congress from doing a stellar job.


Right now the national parties spend half their time hob-knobbing with their so-called allies and only the remainder of the time is given to the people. And I don't blame them. We as a people have failed if we vote on the following basis:

• caste - needless to say majority of votes are given/taken on the basis of caste and the percentage of people voting on the basis of caste must start reducing
  
• regional pride - Vote for the national government on the basis of regionalism is downright stupid. A good independent legislator is *useless*. He cannot push through reforms alone. Democratic governments run on collective ideology, not personal whims. So while it may *seem* like a good idea to vote for a independent candidate with a clean profile, it doesn't help you eventually.
• Short term promises - The only promises that should be respected are those that have the potential of empowerment - education, skill training, long term infrastructure projects through government spending, making bureaucracy accountable, transparent governance. Promises like reservations, subsidies, freebies only weaken the populace as I have indicated in my previous post - though I must admit it will look attractive to the minimum wage earner.
  

So my request to you is very simple - look who among BJP or Congress candidate in your region is more better and vote for him.

Ok, now for the skeptics: you will very rightly say that this cannot go on forever. You are right - we should do this only for this and the next election. And hopefully until then the national parties would have given a good account of themselves. Hopefully by then parties like Shiv Sena, MNS, Left, AIADMK, Trinamool, BSP, BJD, etc. would be relegated to the side-lines and we will have successive strong governments at the centre taking India where it should be.