http://neverssl.com

 Just discovered the super useful site - for hotels and other hotspots that require a Wifi signup page. Most sites auto-upgrade to TLS. 

Cool domain name too - hope they'll hold it. 

Same-node communication

For containers/VMs running on the same physical machine - including containers in the same Pod or in different Pods scheduled using affinity - it would be highly useful to use modern inter-process communication based on shared memory, DMA or virtio instead of keep copying bytes from buffer to kernel buffer to yet another buffer ( 3 copy is the best case - usually far more).

We have the tools - Istio CNI (and others) can inject abstract unix sockets, there are CSI providers that can inject real unix sockets. 

Unix sockets - just like Android Binder - can pass file descriptors and shared memory blocks to a trusted per node component - which can further pass it to the destination after applying security policies. 

I was looking into this for some time - I worked for many years in Android so I started in the wrong direction attempting to use binder ( which is now included in many kernels ). But I realized Wayland is already there, and it's not a bad generic protocol if you ignore the display parts and the XML. 

Both X11 and Wayland use shared buffers on the local machine - but X11 is a monster with an antiquated protocol focused on rendering on the client - and browsers are doing this far better. Wayland was designed for local display and security - but underneath there is a very clean IPC protocol based on buffer passing. 

How would it look like in Istio or other cloud meshes ? Ztunnel (or another per-node daemon ) would act as a CSI or as a CNI injecting an unix socket in each Pod. It could use the Wayland binary protocol  - but not implement any of the display protocols, just act as a proxy. If it receives a TCP connection - it can just pass the file descriptor after reading the header, but it would mainly act as a proxy for messages containing file/buffer descriptors. Like Android, it can also pass open UDS file descriptors from a container to another, after checking permissions - allowing direct communication. 

The nice thing is that even when using VMs instead of containers - there is now support for virtwl in kernel and sommelier - and this would also work for adding stronger policies on a desktop or when communicating with a GPU. 

Modern computers have a lot of cores and memory - running K8S clusters with fewer but larger nodes and taking advantage of affinity can allow co-location of the entire stack, avoiding slower network and slower TCP traffic for most communications - while keeping the 'least privilege' and isolation. Of course, a monolith can be slightly faster - but shared memory is far closer in speed compared with TCP.

I've been looking at this for few years in my spare time - most of the code and experiments is obsolete now, but I think using Wayland as a base ( with a clean, display independent proxy) is the right pragmatic solution. And simpler is better - I still like Binder and Android model - wish clouds would add it to their kernels...

Chrome and secrets on linux

 Wasted few good hours on this: if you want to move from gnome (and variants like cinnamon) to something else, like sway, and not have to re-enter all the passwords - ignore the man page and all the search results that suggest `--password-store=gnome`.

It is `--password-store=gnome-libsecret` instead.

The rest - installing/starting gnome keyring is still valid, validate with seahorse (i.e. gnome password manager) it is working. 

And add a desktop entry with the right flag. "--enable-logging=stderr --v" help to debug, look for key_storage_linux.cc

Changing settings for Crostini in ChromeOS

Found: Mount Block Devices in ChromeOS

Apparently it is possible to change the LXC config and get access to the real VM, which appears to be read-only. Combined with moving devices to the VM there is more control - but still limited by the small number of kernel modules in the VM.

I love the security model - the 'host' just handles display and a number of jailed services, all the apps in the VM with LXC on top. The problem is that it's too restrictive - and the linux apps are still all in the same sandbox with access to each other. Flatpak at least tries to isolate each app - but falls to the same trap that Java and early android did - the apps ask for too many permissions. 

I'm sticking with my less efficient setup - docker and pods with explicit mounted volumes, syncthing and remote desktop, with one container per app or dev project - but I've been looking to move from ChromeOS to normal linux set in a similar way. 

Installing and running linux using docker images

Found  Debootstick - previously used vmdb for same purpose.

Mainstream linux install is stuck in 1992, with a CD-compatible image running an 'installer' that asks 10 questions and installs old games and office applications in case you may need them.  And assuming you are lucky to have one computer - which you may want to dual boot with Windows - and will spend quality time manually configuring and and taking care of it.

Raspberri Pi, ChromeOS, Android, OpenWRT use 'image' install, where an image is just copied to disk. The booting 'glue' code can be simple ( EFI, kernel, firmware images ) or fancy - encrypted disk, A/B kernel, verified R/O rootfs. But after boot - you still have about the same rootfs you would run in docker or in a VM. 

That's what debootstick and vmdb can automate - a USB stick running a customized image with my SSH authorized keys and minimal set of apps to boot and 'dd' or flash on the few servers/laptops/routers I use. 

I use Debian and OpenWRT - the real value of distributions is still patching/building/testing the core libraries and kernels. 

OCI images are ubiquitous - there are plenty of automation tools and infra, can be tested, customized - and runs more securely on both home machines in docker and in K8S. 

Another recent find is KasmVMC - viewer/client is any browser, more optimized wire protocol ( but I don't think it's real WebRTC - and seems to be X11 only, no wayland yet). But most important the maintain nightly builds of common applications as docker images - install on any server or in K8S and use on any laptop.  The tricky part remains getting some ACME certificates - if only it had an Istio gateway in front... Very curious how it'll impact performance - but still looking for an equivalent using real WebRTC/Wayland and with same docker-image set.

Import k8s resources into Helm

Good doc - useful for example if you migrate an 'istioctl install' into helm, in particular Service resources which can't be deleted without losing the external IP.

 https://jacky-jiang.medium.com/import-existing-resources-in-helm-3-e27db11fd467

USB charging

USB charging info - going all the way back to LPT/COM ports, and lots of details on the protocols.

Idea: XDS to K8S API bridge

XDS is a gRPC-based protocol for pushing configs and updates. The data is represented by 'resource type' and 'resource names', and values are typically protocol buffers, but can be JSON or any other format.

K8S defines a similar API - based on Json, also defined as protocol buffers or OpenAPI schema. 

XDS and K8S APIs are quite similar - and serve a similar purpose, to allow controllers and other apps to get real-time notifications when anything changes in the config database. K8S also supports 'update/delete' - which are not present in 'official' XDS, but relatively easy to extend or support as a separate gRPC method.

It is possible to write a one or 2 way bridge between the two protocols: it would allow a simpler model for watching K8S resources compared to 'list and watch', and likely provide better performance. On the opposite side, it would allow kubectl to be used to debug and interact with XDS servers.

In general, there are many similar protocols using GET/LIST and with some form of 'watch' or events - creating bridges to allow users to pick one client library and interact with different protocols seems better than current model of having one heavy client library for each protocol.

K8S Events

A few notes on K8S Events.  

K8S at its core is a database of configs - with a stable and well defined schema. Different applications (controllers) use the database to perform actions - run workloads, setup networking and storage. The interface to the database is nosql - with a 'watch' interface similar to pubsub/mqtt that allow controllers to operate with very low latency, on every change.

Most features are defined in terms of CRDs - the database object, with metadata (name, namespace, labels, version ), data and status. The status is used by controllers to write info about how the object was actuated, and by users to find out. For example a Pod represents a workload - the controllers will write the IP of the pod and 'running' in status. Other controllers will use this information to update other object - like EndpointSlice. 

K8S also has a less used and more generic pubsub mechanism - the Event, for 'general purpose' events.

Events, logs and traces are similar in structure and use - but different in persistence and on how the user interacts with them. While 'debugging' is the most obvious use case, analyzing and using them in code, to extract information and trigger actions is where the real power lies.  

The CRD 'status' is persistent and treated as a write to the object - all watchers will be notified, the writing is quite expensive.  Logs are batched and generally written to specialized storage, and deleted after some time - far cheaper but harder to use programmatically, since each log system has a different query API. 

In K8S events have 1h default storage - far less than logs, which are typically stored for weeks, or Status - which is stored as long as the object lives. K8S implementation may also optimize the storage - keep them in RAM longer or using optimized storage mechanisms.  In GKE (and likely others) they are also logged to stackdriver - and may have longer persistence. 

Events are associated with other objects using 'involvedObject' field, which links the event to an object, and is used in 'kubectl describe'. This pattern is similar to the new Gateway 'policy attachment' - where config, overrides or defaults can be are attached to other resources.  

```

# Selectors filter on server side.

kubectl get events -A --field-selector involvedObject.kind!=Pod

kubect get events -A --watch

```

Watching the events can be extremely instructive and reveal a lot of internal problems - Status also includes errors, but you need to know to watch a particular object. 

As a 'pubsub' system the Events are far from ideal - both as storage, API and feature set - but they are close in semantics and easy to bridge to a real pubsub, and for K8S they are very useful. 

In the past I tried to add more Events to Istio - there was some interest but never got to finish the PR, maybe with Ambient we can try again. The real power of Events is not for debugging, but in synchronizing between applications in real time, for example propagate the IP address and info about a node as soon as it connects to the control plane. 

CNCF CloudEvents provides an API and integrations with various messaging and pubsub systems - it is a bit over-designed and more complex then it needs to be, but the integrations make it useful and it provides a basic HTTP based interface that is easy to work with. 

Istio also provides some events over XDS - and can also act as a bridge, to allow components using a control plane to get both configs and events. 

Links:

• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.19/#event-v1-core
• https://www.bluematador.com/blog/kubernetes-events-explained - how to watch and filter
• https://www.cncf.io/blog/2021/12/21/extracting-value-from-the-kubernetes-events-feed/

TODO:

• Evaluate CloudEvents integrations with K8S Events and 'real' pubsub
• Extend Istio XDS 'debug' bridge to Events, evaluate use for sync and ambient info if Events are as reliable as pubsub.
• Generate events from Istiod - connect/disconnect are clear. Warnings about bad configs are unlikely to be good unless frequency can be controlled.

Posts and comments

Publishing content is very easy - github, blogger, personal pages, countless social sites and fancy P2P networks. 

Reading content is harder - Reddit, Twitter, Feedly and few others are attempting to identify 'interesting' content and organize it. There is too much content published, and too little diversity if you stick with a list of subscriptions (old style blog readers) and too much garbage and propaganda if you want broader sources. 

Comments are the third problem - they usually are more interesting than the original post, and are valuable both to the reader and to the original poster.  Just posting some content on a page with no way to get any feedback is a waste of time.

I've started to look for an alternative to Twitter for reading - and while doing so I also started for a place to start posting/blogging/ranting. I rarely post on Twitter - but I keep a lot of notes on various projects and experiments - just too lazy to publish them except in comments and 'readme' in git repos I work on.

The options so far:

• blogger - I've been using it for a very long time, easy - but very unfriendly to code and markdown.
• reddit - for posting in specific topics. Great comments and community associated with the topic. I've seen many posting a 'blog' or page on github and linking it to reddit for discussions/comments.
• https://github.com/utterance/utterances - comments become github issues. Interesting idea - using search in the issue tracker to hold the post comments. Best associated with blogs hosted on github.
• disqus - adds on the free edition, $11 for add free. Work to set it up.
• cusdis - open source, selfhosted option - supports sqlite.
• ...

For now I'm (re)starting to use blogger - it is by far the least effort, since my goal is to publish my tech notes and rants and maybe get a bit of feedback if anyone stumbles on them - but also checking Reddit. I wouldn't spam Reddit with my rants - but most communities seem high quality and the moderation on each community seems like a far better way to keep the noise out than even old Twitter (very low bar, but that's what I used to use to read, when I somehow trusted they have a team fighting against garbage content and disinformation). 

If I get bored - I would try 'utterances' with github and cusdis.

Egress capture using TPROXY

Very low level notes on intercepting traffic for Istio and similar apps. IPtables provide 2 mechanisms to capture, REDIRECT and TPROXY. The first is buggy and not recommended by the kernel docs. TPROXY unfortunately requires NET_ADMIN (or root) and is only available in the `PREROUTING` chain, i.e. can only be used for packets received on an interface - not on packets sent by local apps ( OUTPUT ).

I've been playing with this for some time, and this is what I've found:

1. Use an OUTPUT chain to mark packets - just like we do for REDIRECT interception
2. Use a routing table with 'dev lo' to route all marked packets to loopback. 
3. Apply TPROXY capture on the loopback PREROUTING - if the dest IP is not 127.0.0.0/8

It looks like this: 

  # Anything with the mark 15001 will be sent to loopback
  ip -4 rule add fwmark 15001 lookup 15001
  ip -4 route add local default dev lo table 15001

  # Calling this chain will set the mark resulting in route to lo
  iptables -t mangle -N ZT_CAPTURE_EGRESS
  iptables -t mangle -A ZT_CAPTURE_EGRESS -j MARK --set-mark 15001

  # PREROUTING on loopback - anything routed by the route table 15001, based on OUTPUT mark
  # Ignore local source or dst - it's not egress
  iptables -t mangle -N ZT_TPROXY
  iptables -t mangle -A ZT_TPROXY -d 127.0.0.0/8 -j RETURN
  iptables -t mangle -A ZT_TPROXY -d 127.0.0.0/8 -j RETURN
  iptables -t mangle -A ZT_TPROXY --match mark --mark 15001 -p tcp  -j TPROXY --tproxy-mark 15001/0xffffffff --on-port 15001
  iptables -t mangle -A PREROUTING -i lo -j ZT_TPROXY

  # Table that determines who gets redirected
  iptables -t mangle -N ZT_EGRESS
  iptables -t mangle -A OUTPUT  -j ZT_EGRESS

The OUTPUT table is similar to regular Istio:

  # Exclude few ports that should not be captured
  iptables -t mangle -A ZT_EGRESS  -p tcp --dport 15001 -j RETURN
  iptables -t mangle -A ZT_EGRESS  -p tcp --dport 15009 -j RETURN
  iptables -t mangle -A ZT_EGRESS  -p tcp --dport 15008 -j RETURN

  # UID or GID of the app capturing - so it can originate egress without 
  # getting captured again.
  # Best is to use GID - so root user is also captured. However when debuggin
  # in an IDE like CLion/Golang it is very easy to set 'run as root' but not
  # 'using group id' - so using uid-owner.
  iptables -t mangle -A ZT_EGRESS -m owner --uid-owner 0 -j RETURN

  # For now capture only 10.0.0.0, private range, can be changed to 0.0.0.0/0
  # to capture everything.
  iptables -t mangle -A ZT_EGRESS -d 10.0.0.0/8 -j ZT_CAPTURE_EGRESS

This works for sidecars - and avoids the problems with REDIRECT, however it does require the sidecar to run with NET_ADMIN cap, which is not always possible. For Ambient Istio (ZTunnel) it may not be needed since eBPF or veth can be used instead.

The other major benefit of TPproxy is that it also allows UDP capture - REDIRECT 'original DST' does not work for UDP. I did a bit of testing with UDP and IPv6 - all seems to be working.

K8S TokenRequest and TokenReview

Links to various docs on using K8S as an identity provider for JWT tokens - TokenReview and TokenRequest APIs.

https://jpweber.io/blog/a-look-at-tokenrequest-api/

https://github.com/mikedanese/community/blob/2bf41bd80a9a50b544731c74c7d956c041ec71eb/contributors/design-proposals/storage/svcacct-token-volume-source.md

https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#tokenreview-v1-authentication-k8s-io 

https://github.com/kubernetes/community/blob/master/contributors/design-proposals/auth/bound-service-account-tokens.md

Istiod CRDs in IntelliJ

Quick post to remember the link:

https://raw.githubusercontent.com/istio/api/master/kubernetes/customresourcedefinitions.gen.yaml

This needs to be added to Languages&Frameworks / Kubernetes, and refresh when the API changes.

Istio injection

https://jimmysong.io/en/blog/sidecar-injection-iptables-and-traffic-routing/

Recovering .ecryptfs files

I have some old backups, including the .ecryptfs directories.

After several searches, what worked is:

1.  Check the key signatures: cat .ecryptfs/Private.sig 
2. Try to remember the password, and

   ecryptfs-unwrap-passphrase .ecryptfs/wrapped-passphrase

3. With the result, do

   echo $UNWRAPPED |ecryptfs-add-passphrase --fnek - 

If the result of ecryptfs-add matches the Private.sig, mount the disk with the magic command:

mount -i -t ecryptfs   -o ecryptfs_sig=$FIRST_ID,ecryptfs_fnek_sig=$SECOND_ID,ecryptfs_cipher=aes,ecryptfs_key_bytes=16   .Private/ $DEST

Read: low level kernel networking

In particular the UDP sections are very useful.

'via Blog this'

docs/manifest-format.txt - git-repo - Git at Google

Reference for Android repo format.


docs/manifest-format.txt - git-repo - Git at Google:

'via Blog this'

KSubedi/gomove: gomove is a utility to help you move golang packages by automatically changing the import paths from the old one to new one.

KSubedi/gomove: gomove is a utility to help you move golang packages by automatically changing the import paths from the old one to new one.:

'via Blog this'

Golang Concurrency Tricks

Golang Concurrency Tricks:

'via Blog this'

Debugging Go core dumps · Go, the unwritten parts

Debugging Go core dumps · Go, the unwritten parts:

'via Blog this'