Finding an entity in appengine console

Appengine has a nice "datastore viewer" - you can browse your data, but if you want to find a specific row you need to run a query.  The syntax is not obvious from the documentation if you're selecting by primary key:

SELECT * FROM DeviceInfo WHERE __key__ = Key('DeviceInfo', 'my_key')

I have a DeviceInfo table, with a primary key called 'name'. You can find about __key__ and Key in the docs, but I had to try many combinations to find that the first param is actually the type and the key name doesn't matter.

Server to Android messages

Also check: http://code.google.com/p/chrometophone

Froyo required.

Backup windows from linux

ntfsclone - to copy/backup to, included on ubuntu live/install CD. Tar doesn't seem to work very well.

This is for moving windows to a larger disk, or making a snapshot to restore later.

I got a new netbook, removed all the garbage, installed my stuff ( chrome, firefox, etc ), saved the 'restore point' and installed ubuntu. I only use windows for netflix and occasional testing - so I don't want to deal with any windows-specific backup program or to have to reinstall.

ZLib memory overhead

I was testing some compression code - and the load tests were running out of memory. Looks like ZLib allocates about  256K per compressor. I tried this with both  BEST_COMPRESSION and BEST_SPEED - looking at Shallow Heap in eclipse heap analyzer, 2x32k short[] and 2x64k byte[].

With SPDY you need to keep the header compressor around for the entire kept-alive connection. HTTP connections only use gzip for individual requests - the context and memory can be reclaimed, so a kept-alive connection has a very small cost - can be as low as just a socket in a selector, sometimes few extra buffers.In tomcat-lite there is one 8k buffer associated with the connection - not hard to get rid of it, but low priority.

Heap dumps - looking at all objects and fields in a live system, without a debugger

Do you have a java server - maybe in production - and you want to look at the value of some field ? You can't attach a debugger, this would require starting the server with various flags - but you can get a heap dump, which includes all objects, including the value of their fields. 

The other use is to generate a heap dump before and after running a load test - and look at garbage to evaluate how much work you put on the garbage collector and find leaks. There are profilers that can get better results - but this is pretty fast and free way to get the same result, and you can use it against running systems. 

Getting heap dumps:

 jmap -dump:format=b,file=heap3.bin PID_OF_JAVA_PROCESS

Using code ( in a servlet, etc ):

server = ManagementFactory.getPlatformMBeanServer();
server.invoke(new ObjectName("com.sun.management:type=HotSpotDiagnostic"),
              "dumpHeap",
              new Object[] {fileName, Boolean.TRUE}, 
              new String[] {String.class.getName(), "boolean"});

Looking at the data:

  Eclipse MemoryAnalyzer 

or 

  jhat heap3.bin 

  open http://localhost:7000

Flush() in a compressed stream

I finally understood the difference between 'sync' and partial flush: http://www.bolet.org/~pornin/deflate-flush.html

Since last byte may not end on a 8-bit boundary - you need to pad, i.e. to end the current compression block and start a new one.  The interesting part is that a flush with Z_SYNC_FLUSH inserts 0x00 0x00 0xFF 0xFF - but protocols like PPP can strip it.

Another good link: http://www.zlib.net/zlib_tech.html

Chrome, SSL and Tomcat - or other JSSE-based engines

If you try to use Chromium with a java server running https - be aware of  this bug:
http://bugs.sun.com/view_bug.do?bug_id=6728126

Chrome will use a nice TLS extension - RFC4507, SessionTicket - to allow servers to reuse the session without a second roundtrip and without having to cache the session ( which is quite tricky if you use load balancing - there is no easy way in java to do so ). Of course - JSSE doesn't support the extension, the bug is that it can't even ignore it. Besides saving the roundtrip this also saves some CPU time on both ends - good thing chrome added this.

I spent some time today trying to figure out what was happening - the good news seems to be that it's fixed in recent jdk builds. Unfortunately ubuntu doesn't have the fix yet (not even Karmic), and probably Mac is also behind.

Filed a chrome bug as well - I think reversing the order of SessionTicket and hostname extension would solve this.

The server name extension is another nice thing - it allows chrome to specify which host it's looking for in the first ClientHelo packet - so server can send the right certificate. The "Host" header will be sent after the encryption is started - this is why many https servers require one IP address per host and can't do virtual hosting with SSL.

Sipdroid, gizmo, android, NAT fun

Few notes on my setup - in hope others will not have to waste so much time. I have a couple of VOIP devices behind a router without SIP support, and SIP is very sensitive to UDP ports.

Network looks like this:

1. pretty dumb DSL router ( not modem !) - I'm setting it for my parents, that's what they have. 
2. a linux host running siproxd. Next I'll try with an even smaller box ( an openWRT router ) - to save power. 
3. One Grandstream box connected to a regular phone
4. Android device, with SipDroid. 

The firewall is set to forward all UDP and TCP to the linux box - I added a 'custom' rule, I don't want to deal with the dsl router, all real filtering in linux. You can specify the SIP range - 5060 + whatever RTP. 

I'm using Gizmo - I like that it's integrated with google voice, relatively easy to set software for most OSes, etc. Well - not so easy to get sound to get gizmo to work on ubuntu with pulseaudio - lot of time wasted there too - I set gizmo to OSS and used "aoss gizmo", this seems to work for me. 

One big problem: siproxd can't deal with the 2 identities, i.e. the ascii username and the phone number. 

As a rule, I ignore all the gizmo usernames and use only phone numbers. That means user@proxy01.sipphone.com will not work, neither "call gizmo" in the UI. To dial you must use 1234@proxy01.sipphone.com, or "call SIP" in their UI.

On Grandstream - normal registration, with outbound proxy set, no STUN.

On Sipdroid - there is no direct support for proxy, but you can set the "Server" to the IP address of the linux box running siproxd, and than set Domain to proxy01.sipphone.com. Note that older versions of sipdroid don't have the option - just upgrade with latest from market. 

So far no problems with the setup - getting my parents to figure out how to dial and where to connect the wires is another story.

Handbrake 0.9.4 on Hardy

Handbrake is a very nice tool - 'just works' in most cases, has good presets so you don't need to know arcane compression options. They just released a new version - which drops support for .AVI and few other things. I think it's great to see this happen - it's not that common for projects to cut features instead of half-supporting them. On the other hand some devices only work with .avi - so I'll keep the old version around.

Another remarcable thing about Handbrake - they link statically, the binary has only very basic dependencies.

The official download is for Ubuntu 9.10 - I had to compile from source, the CLI version is easy to compile - for GUI you need webkit and few other things, so I'll wait for an official backport.

The only trick for compiling CLI is also compiling current yasm from source, the version in hardy is too old. Simple configure/make install works, just make sure you remove the package with "apt-get remove yasm" if you have the old version installed.

Required packages (besides libtool/autoconf/build-essentials): zlib1g-dev libbz2-dev intltool libglib2.0-dev libdbus-glib-1-dev libgtk2.0-dev libhal-dev libhal-storage-dev  libnotify-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev

Compiled binary, if you want to skip the compilation -
 "gzip -d   HandBrakeCLI-0.9.4.gz ; chmod +x HandBrakeCLI-0.9.4"