End of Alcohol

Unknown — Tue, 04 Mar 2025 00:00:00 +0000

Most days we live on autopilot, programmed by habits defined by us or others. Drinking alcoholic drinks is a habit that most people acquire at a young age. It may be deeply ingrained over decades, forming a part of our identity. It takes awareness to stop reacting and question/interrogate this part. I argue it's worth it since it blocks us from our ideal life.</p>

A sound mind in a healthy body</h2>
My definition of an ideal life includes a sound mind in a healthy body. A sound mind means no brain fog, no sluggishness, less self-deception but clarity and awareness. In my view, it's the foundation for making optimal decisions and to experience life to the fullest. Shifting the view to a healthy body: there is a staggering amount of research findings that discuss alcohol significantly impacting physical health, we get into that.</p>
I started my journey in August 2023 as a self-experiment, and after a few setbacks, I am proud to have been abstinent for more than a year. The pride faded, and I adopted a new normal. This text serves as self-reflection and aims to inspire others to re-evaluate/challenge their long-held beliefs.</p>

Consistent small wins</h2>
While a transition to an ideal life is a long one, it starts with small consistent compounding changes and "small wins" to practice every day. To stop drinking for a week, a month, or for the entire next month to then revise if one wants to keep to it. With enough momentum and lessons learned, it can be easier to overcome setbacks and quit drinking alcohol completely. Before I outline the ugly effects on health, I'd like to start with the commonly believed "benefits" of drinking alcohol and dismantle them by perspective shift. And finally, if you're convinced I hope my takeaways at the end will support you in your journey.</p>

Alcohol destroys the Body and the Brain</h2>
The hangover the morning after is just the tip of the iceberg. Alcohol increases blood pressure and dehydrates the body (bad skin). It decreases gray and white matter in the brain</a>, weakens your immune system and prevents healing. It has negative effects on digestion and gut microbiomes</a> leading to gastritis</a> and leaky gut syndrome</a>. Excessive alcohol consumption can cause nerve damage and irreversible forms of dementia. Alcohol raises heart disease risk</a> even for young adults</a>.</p>
Long-term alcohol abuse can lead to permanent and irreversible scarring in your liver</a>. Moreover, alcohol is classified as a Group 1 carcinogen, meaning it is proven to cause cancer. In fact, any amount of alcohol consumption increases your risk of developing six types of cancer</a>: head and neck cancer, esophagus cancer, liver cancer, breast cancer, and colon and rectum cancer. Fortunately, the World Health Organization is pushing to have warning labels on alcoholic beverages</a> to alert consumers to these cancer risks. A more complete in-depth summary is provided by Andrew Huberman’s episode “What Alcohol Does to Your Body, Brain & Health</a>”.</p>

Aha Moments</h2>
My most important learning is to commit to stop entirely, because exceptions can become a rule with enough social pressure or life circumstances. What helps is to commit to a certain period such as 90 days to get into a new mindset, identify with a person who doesn't drink alcohol. And make it a rule not an opinion. I can't put it better than Shane Parrish from Clear Thinking</a>: "In a quirk of psychology, people typically don't argue with your personal rules. They just accept them as features of who you are. People question decisions, but they respect rules.". A good book on habit formation is James Clear’s Atomic Habits</a>. For the first 3 months, I employed a habit tracker, Strides</a>, to keep track of good and bad days. This helped to have an overview of progress and for keeping myself accountable.</p>
Shifting the focus on health, I barely have brain fog or headaches, much better sleep and with it more clarity, energy and less mood swings. I have much better skin (less dehydration and inflammation). While I don’t have personal before/after data from my tracking devices, the Oura ring data shows a community trend of +8% impact of "No Alcohol" on Heart Rate Variability (HRV, higher is better) among users.</p>
It became easier for me to approach people without alcohol. I have more deeper, authentic, honest and intellectually challenging conversations than I would have under the influence of alcohol. I can dance, sing and do whatever I do with alcohol but without it. Some situations remain challenging, I recall a wedding celebration of a friend, getting forcefully dragged onto the dance floor. Water is usually not served in wine glasses being the odd one out not clinking glasses. Every sober discomfort is a re-learning to approach situations anew.</p>
Let's talk supply. Once I stepped outside the matrix, I was astonished how much alcohol is sold in supermarkets. Germany's alcohol-free beers are decent</a>, but note that alcohol percentages vary from 0.5% to 0%. Alcohol-free white wine (Prosecco) is similar in taste, red wine not so much. Apart from alcohol-free versions the world has plenty of non-alcoholic beverages to explore. A healthy alternative that I’ve learned to love is Kombucha. It’s fermented tea with a complex tangy flavor that is also a great probiotic to support gut health.</p>

Getting Started</h2>
You might have cut back on alcohol, stopped drinking or succeeded only for a period to then return to your status quo. It’s important to acknowledge our struggles without judgment. Instead of fighting our inner demons, such as being too strict to ourselves, we can learn to understand and befriend them by understanding where the desire is coming from. In the beginning of this text, I outlined my why: a sound mind in a healthy body. Maybe we share this one - or maybe you have more concrete reasons and stories to remind you and to give you a refreshed conviction to stay on this path - until there’s no path but a new normal. I want to conclude this text with Confucius’ words: “You can overcome bad habits only today, not tomorrow”.</p>
Resources</h2>

EDIT: Kurzgesagt animation about Alcohol</a></li>
Acute alcohol consumption and arrhythmias in young adults: the MunichBREW II study</a></li>
An episode that has inspired me among other resources to take the leap is What Alcohol Does to Your Body, Brain & Health</a>.</li>
Glucose Revolution - How Alcohol Impacts your Body and Brain</a></li>
Associations between alcohol consumption and gray and white matter volumes in the UK Biobank</a> (Nature Communications</em>)</li>
Gut Microbiota at the Intersection of Alcohol, Brain, and the Liver</a> (Journal of Clinical Medicine</em>)</li>
Why Do Only Some Cohort Studies Find Health Beneﬁts From Low-Volume Alcohol Use? A Systematic Review and Meta-Analysis of Study Characteristics That May Bias Mortality Risk Estimates</a></li>
Tolerance to alcohol: A critical yet understudied factor in alcohol addiction</a> (Pharmacology Biochemistry and Behavior</em>)</li>
Associations Between Drinking and Cortical Thickness in Younger Adult Drinkers: Findings From the Human Connectome Project</a> (Alcoholism: Clinical and Experimental Research</em>)</li>
Moderate Alcohol Consumption and the Risk of Breast Cancer</a> (The New England Journal of Medicine</em>)</li>
Can alcohol promote aromatization of androgens to estrogens? A review</a> (Alcohol</em>)</li>
Living Sober: How Quitting Alcohol Changed My Life</a></li> </ul>

Investing into Drosera's Seed Round

Unknown — Tue, 11 Feb 2025 00:00:00 +0000

The original publication can be found at Greenfield Publications</a>.</p>
We at Greenfield are excited to lead Drosera’s $3.25M seed round with participation of Anagram, Arrington Capital, Paper Ventures, UDHC and Pulsar. This funding will be instrumental in further developing and deploying Drosera’s innovative decentralized incident detection and response solution to ensure the safety of onchain value. The team has an exceptional track record in relevant fields such as cybersecurity/malware analysis at the US Army Cyber Command, DeFi and MEV at Franklin Templeton, BitGo, Obol and Composable.</p>
Attacks and failures in DeFi</h2>
DeFi grew increasingly complex from 2020 onwards. The risk of economic attacks has been historically underestimated, leading to many cases of exploitation using a combination of economically interdependent protocols. Categories like flash loan attacks (Euler | $197M), price oracle manipulation (Mango markets | $117M), bridge related attacks (Wormhole Hack | resulted in $320M of WeETH not being backed) have emerged. Liquid staking tokens (LSTs), liquid restaking tokens (LRTs) and yield-bearing stablecoins (through, e.g., treasuries) have further increased complexity with staking (custom lockup and redemption), restaking (to stake the same asset multiple times), and dependence on (opaque) offchain yield.</p>
The risks of LSTs and LRTs are heightened due to complex factors that can lead to de-pegging. LRTs inherit the de-pegging risks of LSTs and add their own. This was evident in mid-2022 when stETH, typically pegged 1:1 to ETH, deviated significantly in price due to market fears following the Terra/Luna collapse, leading to cascading liquidations in DeFi lending. Shifting focus from LSTs/LRTs to stable coins, recently, Usual Money’s USD0++, a staked version of the USD0 stablecoin, updated its redemption policy without prior notice, causing a sell-off that dropped its price to $0.93 and impacted protocols using USD0++. Mispriced economic value in staking can further affect network security in proof-of-stake systems. Another example is Circle’s USDC de-pegging in 2023, driven by macroeconomic factors like inflation and interest rates. USDC fell to $0.88 after it was revealed that $9.7B of its reserves were at Silicon Valley Bank, which was facing a bank run. This led to a loss of over $6B in supply as investors feared a crash.</p>
A joint report from the European Banking Authority (EBA) and the European Securities and Markets Authority (ESMA) has summarised known risks of DeFi and recommended the introduction of disaster recovery and incident response mechanisms (2.4/95).</p>
Drosera</h2>
Drosera builds the immune system for Ethereum and its surrounding EVM ecosystem. It offers a unique solution by enabling the self-service creation of “traps” to detect and respond to potential threats and failures, as outlined above.</p>
Traps can be fine-tuned, activated or deactivated without a redeployment of a smart contract. The traps are defined by protocol developers or trappers (auditors and economic auditors) using Solidity code, targeting EVM-based ecosystems, incorporating both objective, verifiable facts, using zero-knowledge proofs, and subjective assessments, e.g. anomaly detection or proprietary models. The resulting incident response task for a trap is taken care of by adequately incentivized operators (and block builders). Incentivization and market forces ensure a required top-of-block execution to avert exploitation or failure. It is paramount to decentralize its operation over designing incident response in a centralized manner leading to single point of failure and corruption risk.</p>
The network has already made considerable progress, with more than 25 protocols, including Ion Protocol, EtherFi, and Gravita, committed to its Testnet. Additionally, it has established partnerships with various node operators like Infstones, Everstake, Cosmostation, a41, and Blockscape, and joined forces with security specialist GoPlus.</p>
We are excited about the experienced team and its relentless drive to redefine onchain security. In DeFi, economic audits and bug bounties are simply not enough. Drosera provides the crucial building block of holistic real-time threat detection and incentivized response.</p>

Leading Arcium's Strategic Round

Unknown — Thu, 09 May 2024 00:00:00 +0000

The original publication can be found at Greenfield Publications</a>.</p>
We at Greenfield</a> are excited to lead Arcium’s $5.5M strategic funding round to launch the first parallelized confidential computing network. Other investors include Coinbase Ventures, Heartcore Capital, LongHash VC, L2 Iterative Ventures, Staking Facilities, Smape Capital and Everstake along with angel investors including Anatoly Yakovenko</a>, co-founder of Solana, Keone Han</a>, co-founder of Monad, Mert Mumtaz, co-founder of Helius Labs, Santiago R Santos, Kenny Li, co-founder of Manta Network and Lucas Bruder</a>, co-founder of Jito</a> amongst others.</p>
Arcium was founded by Yannik Schrade</a>, Julian Deschler</a>, Nicolas Schapeler</a>, and Lukas Steiner</a>. They have already impressed us with Elusiv, their previous project, pioneering user-friendly and compliant transaction privacy</a> on Solana. Now with Arcium the team builds a foundational technology that unlocks a plethora of existing and new use cases, that require confidentiality, for Web3.</p>
Privacy is everywhere</h2>
Privacy is a fundamental human right and a core element of our daily lives. Examples where privacy matters can be found easily: payments, bank transactions, medical records, messaging, voting and legal contracts. Privacy is not only normal – but a prerequisite for exercising freedom of thought/speech and as a consequence essential for democracy. Anonymity and/or confidentiality are the two core components of privacy. Whereas anonymity ensures the identity of the user is unknown, confidentiality ensures the user’s action/information is unknown. To understand the pressing need for confidentiality within Web3 let’s first juxtapose it to Web2:</p>
The old world</h2>
In Web2, surveillance capitalism is a prevalent business model. The product seems to be free, but the hidden cost is the user’s information. The user gives up on anonymity and confidentiality toward the service company, but remains optionally anonymous/confidential toward other users. Collected information is stored in private silos and used in a variety of company-benefitting use cases: 1) to optimize the user’s timeline in such a way that the user spends more time on the platform 2) to target adequate customers with optimal advertisements (remember Cambridge Analytica) 3) to resell user information to the highest bidder for diverse downstream use cases e.g. training data for AI models, predictive policing, voter trend analysis and more. Hence, companies have a strong incentive to acquire more and to reinforce the information asymmetry by e.g., closing previously available APIs.</p>
The new world</h2>
This is in stark contrast to Web3: Information is stored publicly on-chain to ensure integrity and to be able to verify the authenticity thereof, be it a financial transaction, a loan, a limit order, a social graph, or a digital twin deed. Public data and interface standards allow streamlined cooperation between protocols in a permissionless manner resulting in strong network effects. The public traces of these on-chain interactions are stored on an immutable ledger such as Ethereum or Solana for anybody to utilize. Hence, confidentiality as a feature of these systems does not exist. When assessing anonymity one can easily make the mistake that these systems are anonymous. While it’s true that wallets/accounts can be created and used without sharing one’s identity, they can be easily linked to off-chain metadata such as IP or browser fingerprint and tracked across interactions with protocols/exchanges, etc. Hence, most blockchains are pseudo-anonymous and non-confidential.</p>
Enabling confidentiality in Web3 finally makes many existing use cases in Web2 (including information asymmetry) possible in Web3. The even more exciting use cases, however, will combine the old network effects of Web2 with the new network effects and capabilities of Web3: liveness, permissionlessness, composability, censorship resistance, verifiability, and, last but not least, ownership.</p>
Technology</h2>
Arcium is revolutionizing secure computation on encrypted data with its innovative “Multi-Party eXecution Environments” (MXEs). By integrating technologies like multi-party computation (MPC), homomorphic encryption, zero-knowledge proofs, and more, MXEs provide a customizable and secure environment for confidential computing. The team strives to have a frictionless developer experience providing adequate tooling and support.</p>

Cold Shower Therapy and Ritual

Unknown — Thu, 07 Mar 2024 00:00:00 +0000

As probably most of my readers, I've heard of cold showers before. But I never seriously made an effort to deliberately shift to this discomforting practice... until last October. The rewarding effect of dopamine came right after the first shower- and for three months up to now, icy water has been my daily wake-up call. What began as a self-experiment has become a discomforting, but rewarding, morning ritual that gives mental clarity and ignites unforeseen energy levels for the day ahead. But what has originally convinced me to try it at least once?</p>
Positive effects and risks</h2>
I heard that cold showers are good for the immune system but the most concise explanation I discovered was from Andrew Huberman</a> on this podcast</a>. Research studies</a> (added more below) have been performed using cold water submersion. The empirically found positive effects are mental clarity, feeling more energized in the morning and throughout the day, reduced inflammation, clearer skin, boosted metabolism, loosing weight, and improved mental resilience. Challenging yourself mentally every god damn morning makes the rest of the day feel like a piece of cake. And one more thing... Having this cold shock in the morning helps with clocking your circadian rhythm</a> and as a result with your overall sleep quality (no coffee needed). Just one little warning before going into specifics: After doing some research online for this article, I found negative side-effects</a> might occur so please be careful and ask your doctor if symptoms occur.</p>
Goal setting and habit formation</h2>
Some online references recommend starting with a warm shower and gradually changing to a cold shower. I've done that in the past and it didn't work for me, and others</a>. Seriously, I never dared to cool it down. Instead, this time I aimed to get rid of the warm shower option until I formed a strong habit of cold showers. For habit formation, I skimmed James Clear's Atomic Habits</a> again and committed to the following system:</p>
First, starting with a "Why", which is all of the positive effects above. Second, a cue/time/place: every morning after waking up, go to the bathroom and start the routine. Next, decide on a duration and temperature. The minimum duration for this cold exposure is three minutes. The hardest part is the (needed) initial shock, so don't worry about the duration. I went with the coolest setting coming out of the pipes- which after measuring turned out to be 7°C or 45°F. For me, this temperature is cold enough to trigger the desired physiological responses without being so cold as to cause me harm. The usual step for forming a habit is to define a reward for a dopamine spike afterward to wire the neurons. However, after stepping out of the discomfort the dopamine levels empirically increase 2.5x compared to the normal level. Hence, no other (external) rewards are needed. Last but not least, I recommend defining milestones like 7 days, 14 days, and 1 month to track progress. Streaks are important here because continuity overwrites habits. However, when you miss a day don't be too hard on yourself and think about the overall success rate in the long run, e.g. 9 out of 10 days, while also chasing a new streak.</p>
Before</h2>
What has changed since I started having cold showers? I've been loyal to my cold showers most of the time. I seldom skipped the daily cold shower, and sometimes I did not take it directly after waking up. It became a morning ritual that has a big positive impact on my life. Why? Let me explain by contrasting the before and after. In the past it was hard for me to wake up, I felt sluggish throughout the morning. I usually crawled out of bed and procrastinated in the comfort of a warm blanket in bed and in the shower. What I never realized, is after waking up in a warm bed, and having a warm shower, any other sensation after just feels cold. Plus, I was still half asleep when stepping out of the shower, and only felt awake when the caffeine finally kicked in. On cold winter mornings, I shivered, wrapped in blankets, rubbing my back against the heater.</p>
With Cold Showers</h2>
Now, let's visualize the alternative. I go to bed at 22:30 and wake up at 07:00 getting at least 8 hours of sleep. After, I go to the bathroom, brush my teeth, and step into the shower. At this moment, I consciously and unconsciously know that the early morning discomfort awaits, but I also know that I will feel unstoppable afterward. My heart rate climbs to 90. I put the timer on my shower radio for 3 minutes and start with cold watering my legs, arms, then my face and chest. Relaxation is key, aim for long deep breaths. Now comes the best part: 10 minutes after getting up from bed, and right after getting out of the shower my energy level is peaking, my mind is clear and I feel unrestrained.</p>
I think this is the perfect time to focus on the main goals in life, be it morning swimming, meditation, writing a book, sailing, surfing, learning the piano, or whatever you</em> have in mind. I continue this practice because, as outlined, it had a huge positive impact on my life, and I share this story to maybe inspire you to at least try it once.</p>
Extra: Cold and warm showers</h2>
I didn't have a warm shower for two months. I only had morning cold showers up to this point. I feared when having warm showers I would switch back immediately. So I decided to define a different</em> why/cue/time for warm showers. Warm showers are a great way to relax, and relaxation is best for winding down to sleep. But instead of having warm showers every day, I take them as a reward for physical exercise, such as running, or as a cure against stress.</p>
Research</h2>
Health effects of voluntary exposure to cold water – a continuing subject of debate, Sep 2022</a></li>
Health effects of cold water immersion and swimming and its influence on the human body, Jan 2024</a></li>
Beyond the cold baths: contemporary applications of cold-water immersion in the treatment of clinical depression and anxiety, Nov 2023</a></li> </ul>
Motivation helpers</h2>

Wim hof - No motivation to do cold showers?</a></li>
Wim hof - Wim Hof's Top 10 reasons to take cold showers & ice baths</a></li> </ul>

Obsidian and (Local) LLMs

Unknown — Fri, 22 Dec 2023 00:00:00 +0000

In this post, I introduce Obsidian</a>, compare it to Visual Studio Code, and highlight two notable LLM-powered Obsidian plugins namely TextGenerator</a> and Copilot</a>. I follow on with their respective synergies, and the ability to use both of them with online-proprietary and local-open LLMs. I end with the prospect of more multi-modal Obsidian-shaped extendable interfaces to leverage LLMs.</p>
Knowledge Tasks in Obsidian</h2>
LLM-powered tooling improved the productivity of my day-to-day research. The task categories include understanding concepts faster, introspection of my writing/thoughts, hardening arguments, brainstorming on options, extracting certain subjects for in-depth analysis, and compressing information. For most of the above, I use Obsidian</a> as my interface.</p>
For the ones who haven't heard of Obsidian yet: Obsidian is a personal knowledge management</a> (PKM) tool, some also call it a 'second brain'. Obsidian allows the interlinking of notes which over time results in a graph of connected knowledge. The notes are plaintext files stored locally in the Markdown</a> format. This ensures continuity/access to the data if you're offline (for instance on a boat) or even without Obsidian. The interface is intuitive, and general enough that it allows for many different use cases. If you haven't tried it yet, please take a stab at it- It's free for personal usage. All in all, it's a beautiful piece of software.</p>
Open Extensions</h2>
Visual Studio Code</a> has become the go-to IDE for many software developers and one important reason might be its vast extension library</a>, featuring among many more syntax extensions for the most obscure languages, integration with essential toolchains e.g. deployment, and code linting or manipulation. The latter has become surprisingly good with Github's CoPilot</a> or with plugins using OpenAI GPT-3/4 in the background (experiment</a> last year).</p>
Analogous to Visual Studio Code, Obsidian features extendability via Community Plugins</a>. The power/flexibility that this gives to a code literate is unparalleled. There are many helpful plugins that I use on my day-to-day, however, here I'd like to focus on the LLM-powered plugins TextGenerator and Copilot (not to be confused with GitHub's CoPilot) and use cases, such as the ones described at the beginning of the article.</p>
Plugins for Online and Local LLMs</h2>
The first plugin, I tried is nhaouari's TextGenerator</a>. The most basic feature allows you to write prompts inline or select existing text to be evaluated as a prompt using various models. The plugin allows extendability with a prompt template system allowing users to create reusable prompts that can parse Markdown and Obsidian syntax, see documentation</a>. If the underlying model supports it, it even allows multi-modal prompt input such as PDFs, audio, and images. Supported are proprietary models from OpenAI</a>, Anthropic</a>, and Google</a> but also- and this is great for privacy, offline-first, and independence: TextGenerator features local open-source LLM integration with Ollama</a>. Ollama is a user-friendly application to download and run open-source LLMs such as Mistral</a>, Nous Research</a> Hermes and many others. And even better: you don't even need a beefy GPU: e.g. Hermes 13B runs fast on a MacBook Pro M1 Pro.</p>
The second plugin I want to highlight in this introductory post is Logan Yang's Obsidian Copilot</a>. One set up with OpenAI or local LLM backend, you have a chat side pane to prompt generally or to prompt specifically towards the current note. Compared to TextGenerator, this option is more suitable for on-demand prompting and prototyping of prompts/questions. You can even save the chat log as a note to do further introspection. Therefore, using both in combination is ideal: first, determine the most effective approach and then create a reusable prompt template for TextGenerator.</p>
Conclusion</h2>
The integration of LLM-powered extensions within open platforms such as Obsidian and Visual Studio Code represents a significant leap forward in how we manipulate information in natural language or code. Importantly, these new LLM enablers come in two different flavors: online-proprietary and local-open. While general proprietary LLMs are leading the general purpose benchmarks, the scores of open general LLMs caught up fast. Also- open LLMs allow for crowd-sourced fine-tuning and specialization, as seen on HuggingFace</a>, which is similar to the extension/plugin architecture of Obsidian and Visual Studio Code. I'm personally excited to see more extendable interfaces to make use of the coming plethora of (specialized) open LLMs.</p>
To end with some ideas one could think of a collaborative/multi-user photo-editing such as Photoshop or Gimp, DAW</a> such as Ableton, a whiteboard such as Miro or Mural, or even a "Blender" (VR editing) that allows the easy creation of template prompts (for inserting, manipulating or commenting) based on online-proprietary or crowd-sourced local-open models.</p>

Starting to Sail

Unknown — Thu, 21 Sep 2023 00:00:00 +0000

I spent my last summer sailing a lot in Berlin. Last autumn, I finally did my in-water license, after almost 25 years not sailing. Well as a small kid, without any formal education, I believe I didn't know how sailings works. 'Going back sailing' for me was one of those goals/dreams that you don't dare to realise because you fear it might not be what you expect.</p>
Reconnecting</h2>
I remember well, on the first day of sailing school, the teacher familiarised us with the parts of a sailing boat. I sat in one, and while the boat was lifting up and down from the water, I realised this is going to be so much fun. The first time out on the water, we had strong winds, I enjoyed everything of it: the wind in the sails, feeling it pulling the lines, controlling the tiller and cutting our way through the water. I had great teachers and met interesting people in random groups on lots of weekends.</p>
Being in nature</h2>
This year, after a long winter of not sailing, I started early getting back on the boat in April. It was around the time when birds just came back from the south. One early morning out on Wannsee, a group of hundreds of birds flew over us, and landed all around the boat. Like black dots on a silver plane. It was a magical moment. This year, I mostly sailed with family and friends in lakes around Berlin. And had my first experiences being on a sailing yachts in the Baltic Sea/North Sea.</p>
Continuation</h2>
It's almost the end of the year, hence I want to spend as much time on the water as possible. I also decided to continue learning with a costal water license, which includes learning how to navigate with compass, understand sea maps, pyro and the short radio certificate.</p>
I'm not sure where this journey will take me, but I love sailing I'm thanking my past self to have taken the leap.</p>

Enablers and Catalysts in Lightning Network

Unknown — Fri, 30 Jun 2023 00:00:00 +0000

The original publication can be found at Greenfield Publications</a>.</p>
Bitcoin, a different category</h2>
Bitcoin was created in the turmoil of the banking crisis in 2009. Satoshi inscribed the message</a> “The Times 03/Jan/2009 Chancellor on brink of second bailout for banks.” into the first block, referencing these turbulent times. Back then, the US bank Washington Mutual Bank failed with $307B in assets</a>. From then until now, 14 years later, we’re seeing US bank failures again with Silicon Valley Bank with $209B</a> in assets. And the following failure of Signature Bank with $110</a>.</p>
In the meantime, Bitcoin gained strength in an antifragile</a> manner from various regulatory attacks such as straight-out explicit bans e.g. China</a>, Qatar</a>, Saudi Arabia</a>, and various implicit bans</a>, to mining bans in e.g. China</a>, to exchange bans, and technical attacks such as forking the chain, e.g. Bitcoin Cash in 2017</a> and others</a>. This young asset has proven to be resilient over the last 14 years with a 100% uptime, a higher hash rate</a> than ever before, countries adopting it as legal tender e.g. El Salvador in 2021</a>, Central African Republic in 2022</a>, and more and more people holding bitcoin for the long-term, see Total Amount of Holders Chart</a>. With it came institutional adoption such as companies and funds taking Bitcoin onto their balance sheet</a> e.g. MicroStrategy, Tesla, Block Inc. and many others. And just this month BlackRock filed for a spot Bitcoin ETF, and Fidelity is planning to do the same.</p>
Some interesting advancements</h2>
At Greenfield we're excited about recent developments regarding ZK rollups</a>, space chains</a>, discreet log contracts</a> (DLCs), Ark</a> and Fedimint</a> – see Fig.2. A good way to stay updated with these advancements is the Bitcoin Optech newsletter</a>. One of the areas the team at Greenfield is particularly interested in is the Lightning Network (LN) and major advancements, like Taproot Assets</a>, among others, building on top of it.</p>
Lightning Network</h2>
In 2015, Joseph Poon and Thaddeus Dryja</a> proposed the Bitcoin Lightning Network (LN): Scalable Off-Chain Instant Payments. This layer is built on top of Bitcoin to allow for instantaneous, private off-chain settlement in a decentralized way while leveraging Bitcoin’s security and asset. It is horizontally scalable since it’s not using a ledger, but rather a P2P network in which payments are routed. Instant settlement, low fees (<$0.01) and its privacy traits make it attractive for (micro) payments and for companies in the payment space to build on top of to compete with e.g. Visa and Mastercard, the world’s biggest payment processing networks. In a world of IoT and smart grids, the LN can be leveraged as a machine-to-machine payment system. For an analysis of its viability as a substitution for traditional global payment networks, I'd like to refer to Lyn Alden’s A Look at the Lightning Network</a>.</p>
Enablers and catalysts</h2>
In the period from the year 2015 of the inception of the idea until 2021, LN was only used by enthusiastic early adopters. The biggest obstacles were running an own Bitcoin full node, running a LN implementation, finding peers in the network to manually connect to and manually balancing lightning channels including finding a source for inbound liquidity</a>. That’s a lot of friction considering almost nobody was accepting LN payments. With this in mind, let’s have a look at what has changed and led to the explosion of renewed interest in the LN:</p>

Massive reduction of payment failure and an increase in connectivity:
River Financial</a> states in its latest report</a> that in the early days of Lightning in 2018, due to limited liquidity needed for routing payments through channels, people had failed $5 transactions around ~48% of the time. In 2021 however, the network grew to 1,000 BTC of liquidity. By now, this has increased to even more than 5,500 BTC, a rough 5x growth within two years. River states that nowadays payment failure has dropped below 2%. Other growth metrics besides total channel capacity include channel count (but important to keep in mind channel quality over quantity) and. For total capacity and channel count, Coinmetric’s dashboard</a> provides live data.</p> </li>

Advancements in UX and developer experience:
On the user side next generation non-custodial wallets such as Phoenix</a> and Breez</a> (also a PoS solution) allow for frictionless user onboarding to Lightning by leveraging Lightning Service Providers</a> (LSPs) for both automatic and instant (zero-confirmation</a>) LN channel opening. The fundamental problem of inbound liquidity to receive funds has only been partially solved however, see the challenges below. Custodial wallets, such as Wallet of Satoshi</a> (predominantly used in El Salvador), and CashApp</a> and Strike</a> are other important catalysts for the usability of LN (while the use of custodial wallets is debatable). On the developer side, SDKs such as BDK</a> and LDK</a>, Breez SDK</a>, River Financials’ API</a> accelerate the development of nodes, wallets, and services sending/receiving lightning payments. Other UX improvements include new transaction types such as the BOLT12</a> offer type allowing users to receive (instead of sending) payments by scanning a QR code among others.</p> </li>

Renewed interest in Bitcoin through experimentation and Taproot assets:
This year the so-called Ordinal Theory</a> enabled the issuance of inscriptions (think of it as NFTs + on-chain content, more than >13M) on the Bitcoin base chain. It is a standard on how to interpret the existing blockchain data, without any modification, and attribute a new meaning off-chain to individually tracked on-chain satoshis. Another idea, the BRC-20 concept, leveraged inscriptions to enable fungible asset transactions on top of Bitcoin. Its market cap is by now >$100M. Both approaches are more of a hack, an experiment, rather than a serious proposal to extend Bitcoin but they proliferated new interest in Bitcoin. An upcoming and serious proposal to extend Bitcoin to support custom assets is Taproot Assets</a> (formerly known as Taro). These custom assets enable issuance on the Bitcoin blockchain and the use of them in LN (while using the critical mass of liquidity of bitcoin locked in the network). A similar effort is RGB</a>. Both enable centrally-issued stablecoins on Bitcoin and LN, which currently is a large chunk of liquidity settled on e.g. Ethereum. The ability to have stablecoins on Bitcoin/LN is another controversial topic for the community.</p> </li> </ol>
Bitcoin’s resilience, institutional adoption, and status as the most adopted cryptocurrency for payments and store of value make it a significant and differentiated player in the industry. The Lightning Network, built on top of Bitcoin, has emerged as a superior crypto payment system, offering horizontal scalability, low fees, and enhanced privacy. Recent developments in liquidity, connectivity, and user experience have fueled renewed interest and adoption of LN. However, challenges such as reliability, non-custodial scaling and easier ways to provide routing liquidity still need to be addressed.</p>
Overall, Bitcoin’s unique attributes and the growth of the Lightning Network position them as transformative forces that might become, if challenges are accordingly addressed, in a couple of years time a viable alternative to upcoming centralized digital currency payment systems worldwide.</p>

From Scratch in Zola

Unknown — Wed, 31 May 2023 00:00:00 +0000

The 10-year journey of static site generators led me to Ruby (Octopress), Python (Pelican) ultimately to Javascript (NextJS). Minimalism was always dear to my heart, but in this iteration of my site I practiced ultimate minimalism, not only in aesthetics but also for accessibility by low power, low bandwith devices.</p>
When I started my blog in 2012 I used Octopress</a> (I was a Ruby fan back then, last release in 2015), then Pelican</a> (Python) and ultimately NextJS</a> (Javascript) to generate my static website. NextJS is powerful, however, it's too complex to add simple functionality such as RSS and Javascript library dependencies are just scary (security). On top of that, I optimised the site for aesthetics using heavy fonts such as Cardo</a> (~400kb from Google servers) and quiet complex layout that was hard to translate to mobile. It looked great on a Macbook, but accessibility via low power, low network bandwidth, devices was shit.</p>
Smol web & Permacomputing</h2>
You might ask why all of a sudden the change to become a smol/handcraft web, no javascript enthusiast? I got inspired by the concept of Permacomputing</a> and folks like 0xff</a> and Gavin</a> who managed to pull off a beautiful, clean and functional site with few KBs. Other examples include XXIIVV's webring site</a>, 250kb club</a> (madewithtea.com is less than 10kb now, without images), Emersion</a>, Max</a> and the classic Motherfuckingwebsite</a>. My made up requirements were modest: very simple CSS, dark mode, RSS feed, responsive design, no web font and of course markdown input.</p>
A sharp tool</h2>
So I finally started this site from scratch again. And instead of building my own static site generator how all the cool kids are doing it (and what I initially planned to do, with the help of kerkour</a>), I turned to Zola</a>. Zola is a static site generator written in Rust and it's basically just a binary. The documentation is clean and easy to grok. Just took me less than an hour to create a theme from nothing. It worked so well that I decided to write a piece about Zola. It just took me a few hours to set everything up, and will probably (hopefully) last for many years to come.</p>

The Multi-Chain Future

Unknown — Wed, 30 Jun 2021 00:00:00 +0000

Different communities that form around blockchain projects view other projects in the crypto space primarily through the lens of competition and one-shot-winner-takes-all games. This is good because it drives innovation. However, it is shortsighted. In the long-term, the wider crypto ecosystem will thrive by experimentation, diversity and synergies. Hence, I argue that the future is multi- rather than single chain and should be viewed as a repeated, cooperative game. A multi-chain future with monetary value and trustless data flowing freely and inter-chain, would be more adaptable to changing conditions in the environment, more scalable, more resilient and naturally synergies would emerge. Let’s go through it one by one.</p>
Adaptability</h2>
The whole is more adaptable. The decentralized instantiation of blockchain networks precedes test nets and waterfall-like phase gates. Design choices like Proof-of-Work</a>, Proof-of-Stake</a> or Proof-of-Space-Time</a> consensus are fundamental and can’t be easily changed after mainnet instantiation. This means that a single-chain future inhibits experimentation. In a multi-chain world, however, each blockchain can be its own one-shot experiment competing in a multivariate selection process driven by market forces.</p>
Scalability</h2>
The whole is more scalable, because in a multi-chain world, the chains balance traffic between one another. You have similar chains specializing for the same use case/target audience. Vega</a> and Serum</a> on Solana</a> both provide decentralized fast derivatives markets. If one of them reaches its maximum throughput of transactions or orderbook size the unsatisfied demand would shift to the under-utilized competitor.</p>
Resiliency</h2>
The whole is more resilient, because a single-chain is ironically a central point of failure which is vulnerable to e.g. hacks (flaws in the low-level runtime or consensus implementation), decreasing adoption (aging language e.g. Solidity</a>), and miner/validator support (mining, staking becomes unprofitable). Here a competitor chain could easily step in as a fallback. Analog to having a risk-spread basket on different (e.g. collateralized, synthetic) stable coins on one chain, one could easily imagine utilizing different chains (with their own unique securities characteristics) to de-risk chain-wide failures.</p>
Synergies</h2>
And most importantly, synergies will emerge. Different chains, each with their unique characteristics can work together to build something that wouldn’t be possible alone. Bitcoin</a> is the reserve crypto currency. It has by far the most proof-of-work securing it as well as the longest track record of operation as a crypto currency. In it’s wrapped form it has been used on automated market makers, as collateral for stablecoins or loans. It's even possible to take a leveraged position on bitcoin without touching a centralized exchange, have a look at Sovryn</a>. This wouldn't have been possible two years ago.</p>
Building Bridges</h2>
I hope that I could convince you with these brief examples. They are just the tip of the iceberg. With this article, I’d like to encourage a view that embraces diversity within groups and shared commonalities among groups alike. A view that helps us better understand shared values and visions and to have mutual respect and cooperate with one another.</p>

Trust in Blockchains and Smart Contracts

Unknown — Wed, 18 Mar 2020 00:00:00 +0000

This is a piece on trust, so let’s start this with the status quo of trust in a centralized (silos) architecture of the internet. In a centralized scenario, one has to trust the centralized entity. Looking back on the last ten years of security breaches and privacy scandals, we can clearly summarize, that either people (users) don’t care enough about data privacy and security to stop trusting centralized entities or they have no viable alternative. It is this lack of clear alternatives that has allowed centralized infrastructure providers to exploit trust, the effect of which is often opaque to those exploited.</p>
Big companies have sold hundred of millions of user data to hundreds of companies. The biggest effect of user data monetization has been to energize even more companies to bait you for your clicks, likes, poll responses, IQ tests, real-time location and much more, exploiting that data for avaricious ends. The masses happily ever after use and abuse their little pocket casino, it’s a lost cause</a>.</p>
So why should the masses care about decentralized contracts?</strong></p>
Do we need Trust-less?</h2> In a decentralized ecosystem, trust, or the absence of it, is central. Entities in the system don’t need to trust other entities. Statements like: “We don’t trust the government, because it is printing money out of thin air.” or “We want to cut out the middle-man and decentralize the infrastructure.”, are pointing towards lacking trust into a centralized entity. The first statement caused Bitcoin</a>. The second statement is the motivation of Ethereum</a>. Now consider the following statement: If we need to trust somebody or a company (due to the lack of transparency or literacy) to run Bitcoin (e.g. exchanges, custodians) or Ethereum (e.g. contract developers or Infura</a> and MetaMask</a>), then the whole discussion about true decentralization is pointless</strong>.</p>
Is Everybody On Drugs?</h2> DAO</a> hackers gonna hack, DAO hippies and missionaries keep on preaching of a post-privacy or privacy-conserving world in smart contracts. Don’t get me wrong, I love this techno-utopianism</a> vibe. The religious connotation is inspiring. A clever nash equilibrium</a> based on proof-of-work</a> (or other approaches) allows eliminating the trust into a central party. This works great on the consensus layer (consensus about forming blocks, the source of truth). But the need for trusting in protocols in other layers, such as contracts, wallets, and external providers still remains. The essential topic of trust, the reason this all started, has fallen short in the space.</p>
Clients</h2>
On the foundational layer, we have node clients, the software that runs on decentralized blockchain nodes. Clients are open source, and users could potentially verify if they agree with the consensus rules. The client propagates the blockchain state and enforces these rules. The CVE-2018–17144 bug, that was unintentionally merged in 2016 into Bitcoin core, allowed miners to crash any single node that they’re connected to. But even worse, it allowed the miner to create new bitcoins at will, exceeding the 21 million hard cap limit that is currently in place, read the full disclosure</a>.</p>
Another more recent bug was found just a couple of day before the Constantinople release of Ethereum. The smart contract auditing firm ChainSecurity</a> discovered the flaw in Ethereum Improvement (EIP) 1283 that would let attackers steal funds from users and make it “cheaper to do certain things on-chain, especially things that are currently ‘excessively’ expensive.”. Geth and Parity quickly released new versions reverting the future-fork on a particular block & encouraging updating to that version to prevent a chain split.</p>
Data Providers</h2>
The second layer could be seen as blockchain data providers, such as node operators and APIs to blockchain data. A very central, and very trusted entity in the Ethereum space, is Infura</a>. Infura is used as a backend for any MetaMask</a> wallet. It’s the single-point-of-failure. Here, it shows the power of the default. As an Infura or wallet provider, you don’t only have tremendous access to contextual data of your users, but you’re also shaping how people understand and integrate this new technology into their workflows and lives.</p>
Smart Contracts</h2>
The third layer, smart contracts. I remember people saying contracts are immutable and unstoppable. Let me set something straight, Ethereum contracts are Turing complete, that means there will be bugs (such as the one exploited in the DAO hack</a>) and intentional backdoors. We cannot even say with certainty, who owns the contract. Who has the power to upgrade functionality, stop functionality, mint and burn tokens? To understand what are the rules defined in the contract, a computer program, one has to be able to read contract code written in Solidity</a> or some other language.</p>
Wallets and Exchanges</h2>
The fourth layer, decentralized wallets and exchanges. The assets and functionality supported by wallets determine what exists. If a wallet supports ERC20, but not ERC721, then the latter does simply not exist for the user. In the case of exchanges, they have power over what chain is behind the brand “Bitcoin” or “Ethereum”. Further, listing or de-listing of blockchains and coins corresponds to let live or die for 99% of the users.</p>
For better user experience, users trust the service, letting exchanges hold custody of their funds. History showed that exchanges get hacked, loose liquidity and steal the user’s money. As Andreas, the Bitcoin Jesus, likes to say “Not your key? Not your Bitcoin.”. He advises people to store their cryptocurrency on their own wallets, see 2018, record-breaking year for exchange hacks</a>. And that’s not all, wallets and exchanges have tremendous contextual data (Know-Your-Customer) of the users and can profile user liquidity, link funds from bank wire transfer to trades to potential dark-net transactions. Selling user data has always been a profitable business it seems.</p>
Measuring Decentralization</h2>
For shedding light on how decentralized the wealth of a network is, we at TokenAnalyst wrote an article, Who invests in ICOs?</a>, on the level of centralization inherent in holders of ICO tokens. Another related piece is Classifying Ethereum users using blockchain data</a>.</p>
Balaji S. Srinivasan, CTO of Coinbase, wrote an interesting article on Quantifying Decentralization</a> and proposes the Nakamoto coefficient as a simple measure to quantify decentralization, incorporating a distribution of wealth over a population, or distribution of commits among the developers. Based on this measure, Bitcoin might be “more decentralized” than Ethereum. However, more decentralization, based on the distribution of ownership in the community, does not imply that people don’t need to trust the system, with all its sub-systems and layers above.</p>
Trust and Transparency</h2>
The goal of this post is to increase awareness of the layers of trust still involved. We would like to see trust and transparency as central as decentralization in discussions and conferences to come. Further, instead of preaching tech-utopianism (true decentralization), let’s be real and educate users about what’s possible, what’s not and what information & power asymmetries exist, to enable them to make the right (trustworthy) choice.</p>

Kafcache: Memcached and Kafka Streams

Unknown — Thu, 03 Oct 2019 00:00:00 +0000

Head over to the Kafcache</a> repository and try it out.</p>
Kafka Streams</h2>
Kafka Streams is a wonderful library for stream processing building on top of a resilient Kafka cluster. Kafka Streams applications, think of it as stream processors or microservices, consume from topics and write to Kafka topics. Either stateless or stateful. For the latter, one can persist state with an out-of-the-box RocksDB or in-memory backend. RocksDB writes state to disk which can, depending on the setup, survive a restart of the Kafka streams application or restart of the host instance.</p>
Why does it matter?</h2>
However, in certain scenarios, writing to disk is simply too slow and results in higher processing latency. At TokenAnalyst, we take low latency serious and try to minimize performance bottlenecks where possible. We're running our processing services on memory-optimized AWS instance storing our full state in memory. Instead of using the provided in-memory implementation we're experimenting with a epheremal memcached storage backend, I call the bridge library Kafcache</a>. The provided Kafka Streams in-memory implementation is designed for small state that can easily be held in the JVM heap space. However, our state ranges easily from 100GiB to 200GiB.</p>
Kafcache</h2>
Kafcache is built on ScalaCache</a> allowing other supported cache backends, such as Redis, as well. On the Kafka side, it's implemented using the KeyValueStore backend serializing data with ByteArray serdes for key and values. Due to the basic nature of memcached operations, range queries e.g. for window queries are not implemented. As an effect, the memcached backend cannot be used for windowed KTables.</p>

Reliable Fast Access to On-Chain Data

Unknown — Tue, 18 Jun 2019 00:00:00 +0000

At TokenAnalyst</a>, we are building the core infrastructure to integrate, clean, and analyze blockchain data. Data on a blockchain is also known as on-chain data. We offer both historical and low-latency data streams of on-chain data across multiple blockchains.</p>
How we use Apache Kafka and the Confluent Platform</h2>
Apache Kafka® is the central data hub of our company. At TokenAnalyst, we’re using Kafka for ingestion of blockchain data—which is directly pushed from our cluster of Bitcoin</a> and Ethereum</a> nodes—to different streams of transformation and loading processes.</p>
Loading involves batching and storing data in Avro</a> for replay and schema evolution, as well as in Parquet</a> for optimized batch processing in AWS Athena. A big challenge is to support and manage multiple semantically enriched data models for the same underlying data, e.g., into a graph data model to trace value flow</a> or into a MapReduce-compatible</a> data model of the UTXO-based</a> Bitcoin blockchain.</p>
One can easily imagine that the Confluent Schema Registry is extremely helpful for typeability and systematic evolution of our models. Access to the data lake and raw data streams is self-provisioned which allows us to work in parallel, and to scale to support multiple protocols (e.g., the decentralized exchange protocol 0x</a>) that are built on top of different chains.</p>
On the event streaming side, we offer reliable low-latency data streams for financial applications based on Kafka Streams. Kafka Streams allows us to focus on our business logic leveraging its out-of-the-box high availability and fault tolerance. A very central and custom component of our data pipeline, the block confirmer, is built on top of the Processor API</a>, as we will discuss later in the Block confirmer based on Kafka Streams section.</p>
Why does on-chain data matter?</h2>
A public ledger could potentially serve not only as a publicly accessible ledger for money or asset transactions but also as a ledger of interactions on a shared decentralized data infrastructure. Any event, from IoT-supported delivery, trade, real estate transfer, to a bet in a prediction market is timestamped, censorship-resistant, and provable.</p>
The blockchain as a data structure is, in essence, a giant, shared immutable log, lending itself perfectly for event sourcing and (replayed) stream processing. Interestingly, blockchain technology is reducing the need for trust, but doesn’t make it obsolete</a>. The required trust comes from transparency. And transparency is realized by surfacing and decoding the data that is stored on the blockchain.</p>
Without looking too far into the future—blockchain companies, public entities, and governments rely on analytics of blockchain data for different motives, ranging from simple runtime monitoring of a smart contract to live trades made on a decentralized exchange, or revenue reports of potentially decentralized autonomous organisations or companies.</p>
The importance of on-chain data will become ubiquitous, prompting potential on- and off-chain data joins, such as on-chain value transactions or token holdings with traditional user or customer profiles. For example, we recently examined data on Ethereum smart contract interactions</a> and clearly identified patterns of usage that could inform future development in what is essentially a machine dominated ecosystem.</p>
Accessing on-chain data requires setting up nodes, which turns out to be not as easy as we thought, due to overcoming different quirks</a> we encountered or data discrepancies between versions. Furthermore, another challenge is keeping them in sync with the network. For zero downtime, and to ensure the highest standards in data quality and reliability, we decided to use Kafka and the Confluent Platform.</p>
</p>
Cluster of Ethereum nodes, Ethereum-to-Kafka bridge</h2>
Right now, our focus lies on Ethereum. To interface with the peer-to-peer network, we have node templates written in Terraform, which allow us to easily deploy and bootstrap nodes across the planet in different AWS regions. We use the Geth</a> and Parity</a> clients. Both provide a JSON-RPC</a> interface that enables us to subscribe to recent block updates (recently mined blocks). A new block is mined on average every 15 seconds. Complementary data types such as transaction receipts, event logs, and state diffs</a> are also extracted.</p>
To bridge the gap between different Ethereum clients and Kafka, we developed an in-house solution named Ethsync, written in Scala, which allows us to propagate the data in a reliable at-least-once manner. Each node plus Ethsync is pushing the data to its corresponding Kafka topic. It’s important to note that we need to run multiple nodes for redundancy because nodes can run out of sync or simply crash. The block updates pushed to the topics are the blocks that the client accepts as a new valid block. However, due to the nature of the blockchain, forks of the blockchain might happen (an alternative chain becomes longer and causes the other chain to be invalidated). Therefore, previously valid blocks can become invalid.</p>
Block confirmer based on Kafka Streams</h2>
To prevent using invalid blocks in downstream aggregate calculations, for example, we developed a block confirmer component that is based on Kafka Streams in Scala. It resolves reorganisation scenarios by temporarily keeping blocks, and only propagates them when a threshold of a number of confirmations (children to that block are mined) is reached.</p>
For this, we keep a tree of parents and children blocks (with forks), as well as the original data, in a Kafka Streams state store as seen in Figure 1. We’re storing the state as updates to the same key. In production, each new block produces a state record update of 150 KB on average. When forks happen, we usually see two forks, but sometimes up to four forks in a six-confirmation time window.</p>
The confirmer not only solves the challenge of verifying the canonical chain but also outputs every confirmed block with effectively once semantics by discarding already registered and already confirmed blocks. Building that component on Kafka Streams allowed us to leverage on fault tolerance recovery mechanisms, such as failover instances and state recovery. This is great for rolling deploys for zero-downtime environments.</p>
Our API and software development kit (SDK)</h2>
In the big picture, our API and SDK</a> allow customers to retrieve both historical data and reports, and to subscribe to low-latency data streams of raw and contextualized on-chain data. The API backend and SDK are both written in TypeScript. To access the result records of Kafka Streams processors and KSQL processing queries like filtering, aggregates, and anomaly detection, we use the kafka-node and avro-schema-registry JavaScript libraries. The latter enables us to verify the correct output types to prevent schema inconsistencies, inconsistencies with the SDK, and implicitly with external customer infrastructure.</p>
Conclusion</h2>
The hype is currently omnipresent, and it’s not going away soon. We foresee a demand for on-chain data, or trusted data, going forward. It doesn’t just come from the perspective of financial transactions and regulations but, more importantly, from the use cases that smart contracts will enable in the near future. As a startup, we try to maximize the impact of our work effort in the right direction under very limited resources. The latter requires leveraging proven components such as the Confluent Platform to build resilient data pipelines.</p>
Remark</h2>
This article was published first on the Confluent blog</a></p>

Weird Quirks we found in Ethereum Nodes

Unknown — Tue, 18 Dec 2018 00:00:00 +0000

At TokenAnalyst we are experts at hosting, running, and extracting valuable information from blockchain nodes. Through our time working with Ethereum, we encountered some interesting quirks that we'd love to share with the broader community. We work with the two top most-used Ethereum clients, namely Geth (go-ethereum), built mainly by the Ethereum Foundation, and Parity, built by Parity Technologies.</p>
The following quirks have been found by my coworkers Ankit</a>, Sid</a> and myself</a>. If you have any questions regarding the following findings, we would be happy to help. There are not too many people trying to get data from nodes, but if they do, they'll inevitably encounter these quirks at some point. The following list is anecdotal, for further references links are supplied. The first two points are particularly important to know when you set up your own node.</p>
1. Run SSD</h2>
Magnetic hard-drives are not optimal for running nodes; they are just too slow. SSD is a much better option. We started our full nodes on AWS t3.xlarge instances, having EBS storage attached. However, it turned out, that NVMe</a> instance storage in RAID-0 is much more cost-effective, with a 30x improvement on I/O throughput when writing files.</p>
2. Responsiveness</h2>
When Geth is not in sync with the latest block, the RPC interface is unresponsive, meaning you cannot query for e.g. blocks, transactions, receipts etc. This works in Parity.</p>
3. No Error Response Standards</h2>
Even though there is a standard what kind of RPC methods and parameters exist. There are no standard error responses across nodes. Hence, if you want to write software that integrates with different clients you have to parse responses dependent on the client.</p>
4. Response Fields not Consistent</h2>
The fields, r, s, v of a transaction are not consistent on different clients. They're different on Geth, Parity and Infura, see reference</a>.</p>
5. Parity vs. Geth</h2>
Running the latest versions of Geth and Parity on the same AWS t3.xlarge instance types with EBS storage, in the same regions, Geth is more out-of-sync than Parity. This is particularly unfortunate since Geth is unresponsive while syncing. The comparison was pre-1.8.19. After we updated to 1.8.19 we experienced better performance.</p>
6. Typesafe</h2>
The JSON-RPC interface is not type-safe. That means, for data pipelines building on top of this data source, post-processing e.g. validation and typecasting is required. The "standard" on the wiki</a> is not very enforcing.</p>
7. Traces and Statediffs</h2>
For data, like internal transactions, state diffs, there is not an RPC standard yet. Geth is not supporting them yet, Parity has implemented them: https://wiki.parity.io/JSONRPC-trace-module.html. The state diff result format is a bit weird, and it's not fully consistent: see</a>.</p>
8. Data Bugs</h2>
There are two ways of retrieving the latest blocks. Through HTTP polling or a kept alive web socket connection. For both ways block re-organization (a wrong block is forwarded and later discarded) is not correctly implemented on Geth and Parity, see</a>.</p>
9. Smaller Differences</h2>
Multiple smaller differences between the interfaces of Geth and Parity: Uncle size is null in Parity, but does exist in Geth. Filter IDs on Geth and Parity are different. Geth assigns random IDs, and Parity uses an incremental counter.</p>
10. In General</h2>
The consensus rules among clients have to match in order to make Ethereum work, but everything else, e.g. the exposed API, or RPC interface can be totally different. Take each client with a grain of salt.</p>
To sum it up.</p>
Through the process of setting up our infrastructure. We learned that when building fault-tolerant, highly reliable, and mission-critical application on top of Ethereum, it is not enough to rely on the functionality of specific Ethereum clients. An extremely deep understanding of how nodes work, what they need, and how to orchestrate them is required to ensure effectively zero downtime along the whole data pipeline.</p>

Using Monix with Kafka, Avro and Schema Registry

Unknown — Sun, 30 Sep 2018 00:00:00 +0000

Working with Confluent's schema registry solves problems like where to store schemas in a polyglot environment, how to keep track of versions and how to evolve schemas over time. It's also required to have implicit schemas for your Kafka topics when using Kafka connectors</a> or KSQL</a>. If you don't need the former, you might as well just use a type of Array[Byte]</em> and serialize with Avro4s</a> or avrohugger</a>.</p>
So what's Monix? Monix</a> is a high-performance Scala / Scala.js library for composing asynchronous, event-based programs. It's leveraging concepts of Scalaz' task and provides further out-of-the-box compatibility with the Reactive Streams protocol. The developers behind Monix also created monix-kafka</a>, which is a wrapper for Kafka to easily integrate with Monix-based services. In addition, a consumer can also be turned into an Observable</a>, see also this comparison</a> to Akka Actors, Akka Streams and FS2.</p>
Add Dependencies</h2>
When using monix-kafka, you instantiate your Kafka producer by passing a configuration and type parameters of how your records look like:</p>
val producerCfg = KafkaProducerConfig.default.copy(bootstrapServers = brokers.toList) private val producer = KafkaProducer[Array[Byte], Array[Byte]](producerCfg, scheduler) </code></pre> The type for both key and value is the supported Array[Byte]</em> type. However, let's serialize with Confluent's Avro serializer. For this, first, we need to add the missing link, a new resolver URL and the serializer dependency. Be sure you add the right version depending the schema registry you're running.</p> resolvers += "confluent.io" at "http://packages.confluent.io/maven/" libraryDependencies ++= "io.confluent" % "kafka-avro-serializer" % "5.0.0" </code></pre> Extend Monix Kafka</h2> Thanks to implicits we can easily add it to extend the capabilities of Monix' Kafka producer and consumer. Let's create an Object that contains functions to create implicit MonixSerializer</em> and MonixDeserializer</em> values given a serializer, deserializer configuration and a boolean parameter to indicate whether it is the record key (needed by Confluent's Kafka Avro Serializer). In the configuration we can now pass the schema registry URL.</p> import io.confluent.kafka.serializers.{KafkaAvroDeserializer, KafkaAvroSerializer} import monix.kafka.{Serializer => MonixSerializer} import monix.kafka.{Deserializer => MonixDeserializer} import collection.JavaConverters._ object AvroSerializer { def serializer(cfg: Map[String, String], isKey: Boolean): MonixSerializer[Object] = MonixSerializer[Object]( className = "io.confluent.kafka.serializers.KafkaAvroSerializer", classType = classOf[KafkaAvroSerializer], constructor = _ => { val serializer = new KafkaAvroSerializer() serializer.configure(cfg.asJava, isKey) serializer } ) def deserializer(cfg: Map[String, String], isKey: Boolean): MonixDeserializer[Object] = MonixDeserializer[Object]( className = "io.confluent.kafka.serializers.KafkaAvroDeserializer", classType = classOf[KafkaAvroDeserializer], constructor = _ => { val deserializer = new KafkaAvroDeserializer() deserializer.configure(cfg.asJava, isKey) deserializer } ) } </code></pre> Monix Kafka Producer</h2> Let's use these implicits now to instantiate a Kafka producer which serializes to Confluent's Avro format and uses the schema registry to lookup the schema for a specific topic, great for typesafetyness in Kafka topics. Here's how the rest of the code looks like. It's analogous to use Avro in the key field or instantiating a Kafka consumer.</p> case class RecordValue(someInt: Int) val serCfg = Map("schema.registry.url" -> "http://schemaregistry:8081") implicit val serializer: Serializer[Object] = AvroSerializer.serializer(serCfg, false) implicit val format = RecordValue[ValueFormat] val producerCfg = KafkaProducerConfig.default.copy(bootstrapServers = brokers.toList) val producer = KafkaProducer[String, Object](producerCfg, scheduler) val recordVal = format.to(RecordValue(1)) val record = new ProducerRecord[String, Object](topic, 0, "key", recordVal) val task = producer.send(record) </code></pre>
Via Negativa Minimalism Unknown — Sat, 25 Aug 2018 00:00:00 +0000 Readers of Antifragile by Nassim Nicholas Taleb should be familiar with the concept of ‘via negativa’ (literally 'negative path'). ‘Non-doing’ is an active action and has side effects like doing. ‘Non-doing’ is generally unnoticed and undervalued. However, not doing something negative, has a neutral or even positive outcome.</p> Smart avoidance can therefore improve a situation. The requirement, however, is understanding or imagining how the inverse action would effect the situation. A concrete example certainly everybody would agree with, is not-smoking.</p> Hoarding Principles</h2> Two other concepts, cognitive biases, I would like to introduce before connecting them with minimalism are loss aversion and the endowment effect. Both were introduced in Daniel Kahneman’s bestseller Thinking, Fast and Slow.</p> Loss aversion in a nutshell: It refers to peoples tendency to strongly prefer avoiding losses to acquiring gains. I would call it the first Hoarding-principle. The endowment effect is a hypothesis that people ascribe more value to things when people own them. That’s the second hoarding principle. The experiments underpinning the concepts are very interesting reads.</p> Choices</h2> From a functional perspective, objects are utility. The decision of buying or owning something relies on the a-priori estimation whether the object would provide utility. We don’t know, and usually we are not taking other people as reference (except for signalling purposes) since we believe we’re so special, read Daniel Gilbert’s Stumbling on Happiness.</p> So, choices are not-buying or buying (let’s simplify rent to buy). Given a finite amount of money and time, we need to select objects that have a good utility prospect. We are exploring the space of goods. We are window shopping. Endless possibilities, choices, nudged by ads, friends, idols and so on. Got something? Good. Let’s integrate it into your life.</p> Integration</h2> Depending on how accurate you were estimating the long-term utility value of your acquired object, it paid off, or it was throwing money out of the window. The spectrum ranges from introspective consumers to buyers on a FOMO basis. Sure, personal traits and influences play a big role here, but exploration is required. How else can we determine if it provides utility?</p> But here we should remind ourselves to the first concept: Via Negativa. What if we would not buy into the desire. Would it have a positive effect on our life? Clearly, it would go unnoticed, even if it was an active decision of not-buying. Can you imagine your life without the object, but something else taking its place with more utility value? What would this other object be?</p> Exploration and Exploitation</h2> It’s tough, and trying it out is the only choice. Otherwise we would be stuck not-buying anything new, because of fear of being disappointed. In other words, we’d be stuck in a local optima. It’s the balance between exploration and exploitation. Exploration refers to buying new things, without a-priori knowledge whether it provides the desired utility value (taking risk), and exploitation refers to the long-term benefit of utility value provided by the object (not-taking risk, but might be stuck in a local optima).</p> Hoarding Bias</h2> In Minimalism, we ask ourselves the question, do I need this, does it make me happy, do I really want to focus on this, do I want to maintain it? In the rational case after an evaluation period, we decide to keep or sell it. This happened to me recently with a fitness tracker which I decided to sell (it didn’t make me happy, and I didn’t like to wear it all-the-time).</p> However, as I referenced earlier, the hoarding principles are deeply integrated in us humans. The cognitive biases of loss aversion and endowment effect bias us to simply keep our stuff. With both, a just-in-case (loss aversion) and a ‘it’s more worth than it’s actually worth’ mindset (endowment effect). Overcoming this, is the path towards minimalism. Put the objects in doubt into a box, and put it in the attic or basement. If you don’t miss it, get rid of it. Even throw it away (it’s worth less then you actually think).</p> References</h2> I wrote an article about minimalism, which offers questions you can ask yourself when you consider buying new things. I hope both articles inspire you to become a more conscious consumer and owner. And please share the article if you enjoyed it.</p>Countermeasures Against Phone Addiction Unknown — Mon, 02 Apr 2018 00:00:00 +0000 In my previous article, I wrote about the pleasures of using your smartphone, specifically using addictive apps, see Pleasure, Happiness and Feedback-Loops</a>. Addictive apps use dopamine-driven feedback loops. Dopamine</a> corresponds to short-term pleasure, and to much of it leads to addiction</strong>. It is typically experienced alone, and makes the brain say “This feels good, I want more”. Serotonin</a> on the other hand is closely related to long-term happiness, like contentment. It’s generally shared, like spending time with friends, family, colleagues etc. and it makes the brain say: “This feels good, and it’s enough”. Too little of it leads to depression, see The Hacking of the American Mind</a> and This Is Why our Phones are Making Us Miserable</a>. Fundamentally, seeking for pleasure doesn’t make us happy, it addicts us, going down a spiral of long-term dissatisfaction.</p> Form a Habit Every Single Day</h2> And when I write addictive apps</em>, I mean even Reddit, Mail, WhatsApp, Telegram, Signal, Snapshot and what not. I like to outline five essential principles that are based on my experience and which everybody can potentially integrate in their lives. After that list, I’ll describe my ideal day routine. I’m by no means close to forming a habit - there are exceptions, I just try to avoid those as hard as I can. Let's start the list with a bold clever quote.</p> "You will never change your life until you change something you do daily. The secret of success is found in your daily routine."</cite> — John C. Maxwell</p> </blockquote> Principles</h2> Don’t take your phone into your sleeping room or bath room.</strong> There is no excuse. If you’re using it as an alarm, buy a radio alarm clock (just buy a used one, if money is your excuse). I know a lot people take showers while listening to Spotify or Apple Music. Just buy a shower radio. It’s simple, always works and sometimes water-proof. Don’t use your phone on the toilet, that’s just the worst. Yep, guess we all do or did it. Take a magazine, news paper whatever, or just enjoy the quiet time.</li> When going to bed, put your phone on airplane mode.</strong> (when you’re not expecting a super important call) put it in a drawer (outside sleeping room and bath room). I’m putting mine in a draw right under my desk. Out of sight, out of mind.</li> Disable all notifications</strong> for messages apps etc. disable also notification badges and push notifications. I believe it's a much healthier approach to encourage people to call you for the important parts in life.</li> When you meet somebody e.g. in a cafe</strong>, don’t put your phone on the table. No matter if you put it upside down or not. Putting your phone on the table makes the opposite person feel less important to spend attention to.</li> Put your phone on grayscale mode.</strong> It really makes a difference, just try it out, you’ll feel that you’re less attracted to the phone. iOS devices can be put on a grayscale mode easily without any additional app</a>.</li> </ol> Day Routine</h2> My ideal routine is the following: Waking up by a radio alarm clock. Getting a shower. Afterwards, I'm having breakfast. After breakfast, I actively decide if I want to check my phone for messages. I disable airplane mode and get the latest updates. Hopefully I don’t have to write back to someone, in urgent cases I plan to call people to give a response. Putting it back into the drawer or taking it with me. At work, don’t put it on desk, keep it in your pocket or bag. After work, put it in the drawer. Before sleep set it to airplane mode.</p> I hope that article inspired you to rework your phone habits. Let’s enjoy life more by watching our phones less.</p> Reading List</h2> Humane Technology Reading List</a></li> Have Smartphones Destroyed a Generation?</a></li> How a Handful of Tech Companies Control Billions of Minds Everyday</a></li> How Better Tech Could Protect Us From Distraction</a></li> What I Learned in 24 Hours Without My Phone</a></li> This Panda is Dancing - Time Well Spent Video</a></li> Your Brain Is Getting Hacked Video</a></li> Simon Sinek on Millennials in the Workplace</a></li> </ul> Monitoring Kafka Streams Metrics via JMX Unknown — Thu, 09 Feb 2017 00:00:00 +0000 The documentation on monitoring of Kafka Streams</a> is a bit sparse, so I will shed some light on interesting metrics to monitor when running Kafka Streams applications. In addition to Kafka producer, consumer metrics, each Kafka Streams application has stream-metrics</em>, stream-rocksdb-state-metrics</em>, and stream-rocksdb-window-metrics</em>.</p> In this article, let's focus on the stream processing metrics only. I'll present a way to access the metrics using the command-line application jmxterm</a>, and I'll give some insights about the semantics of these metrics, so you can use them for monitoring and alerting. For an introduction into monitoring, I highly recommend the Monitoring 101</a> in the resources section at the end. Among other valueable resources</a>.</p> JMX-exposed Stream Metrics</h2> Let's dive into the metrics. Start jmxterm and open a connection to the local JVM with the Kafka Streams application running. For connecting you can use the open</em> command. The command beans</em> will show you all exposed beans. In this article I will only cover the global stream metrics. The same metrics are defined for each processor. When using jmxterm, it is important to switch client-id</em> and type</em> in the bean name when querying, otherwise, jmxterm responds with bean name invalid, see my problem description</a> and Sachin Mittal's workaround on the helpful Kafka users mailing list</a>.</p> $>info -d kafka.streams -b kafka.streams:type=stream-metrics, client-id=app-fdddffde-5b10-44c8-82dc-3e343cdab829-StreamThread-1 #mbean = kafka.streams:type=stream-metrics, client-id=app-fdddffde-5b10-44c8-82dc-3e343cdab829-StreamThread-1 #class name = org.apache.kafka.common.metrics.JmxReporter$KafkaMbean # attributes %0 - commit-latency-avg (double, r) %1 - commit-latency-max (double, r) %2 - commit-rate (double, r) %3 - poll-latency-avg (double, r) %4 - poll-latency-max (double, r) %5 - poll-rate (double, r) %6 - process-latency-avg (double, r) %7 - process-latency-max (double, r) %8 - process-rate (double, r) %9 - punctuate-latency-avg (double, r) %10 - punctuate-latency-max (double, r) %11 - punctuate-rate (double, r) %12 - skipped-records-rate (double, r) %13 - task-closed-rate (double, r) %14 - task-created-rate (double, r) </code></pre> The latencies are in milliseconds, the rates are per-second. To get the values of an attribute use the get</em> command. For the original descriptions of these metrics, have a look at this code</a>.</p> Semantics</h2> To understand what these metrics are measuring, we need to know the answers to the following questions in Kafka jargon: What is a commit request? What is a poll? What is a process? What is a task? What is puncate? And finally: What are skipped records?</p> A commit or commit request</em> is a request from a Kafka consumer to the broker to commit a certain processed offset to mark the latest offset it read from.</li> A poll or long poll</em> is the process of fetching records from the broker to the consumer. If there are no new records available it will block (busy-wait). Metrics for polling are only updated if long polling is used in the consumer. However, it will be initialized with zero.</li> A process is a function evaluation of a StreamTask in a StreamThread associated with a PartitionGroup, compare with documentation on stream partition</a> and thread model</a> and code</a>.</li> The punctuate</em> method is part of the Low-Level Processer API</a>. The punctuate method is executed periodically based on elapsed time. When you use the High-Level Streams DSL</a> you don't have to bother with this metric, since it is not used. However, it will be initialized with zero.</li> The record is skipped when it is added to the StreamTask record queue. In the code documentation it says, a record is skipped when it has an invalid timestamp, see StreamTask code</a> and StreamThread code</a>. So this should be zero, otherwise, you might want to check your timestamps.</li> </ul> Custom Metrics Reporter</h2> You might want to push your metrics to e.g. statsd or some SaaS to monitor and alert on anomalies or thresholds. There are some libraries available for Kafka, which might work for Kafka Streams metrics as well. Otherwise, you can fairly simple add your own custom metrics reporter by creating a class which implements the MetricsReporter interface and passing the class name to the metric.reporters</em> parameter in the StreamsConfig</a>.</p> If you found a mistake, or like to add something, please feel free to send me a mail under contact</a>.</p> Kafka-related Resources</h2> Introducing Kafka Streams: Stream Processing Made Simple</a></li> Collecting Kafka Performance Metrics</a></li> Monitoring Kafka Performance Metrics</a></li> Official Kafka Streams Documentation Monitoring</a></li> Stream Processing with Kafka Streams by Hugo Picado</a></li> Kafka Papers and Presentations Wiki</a></li> </ul> Resources</h2> Monitoring 101: Alerting on What Matters</a></li> Monitoring 101: Collecting the Right Data</a></li> Monitoring 101: Investigating Performance Issues</a></li> All the things we can do with JMX</a></li> Free Tier: Error Reporting</a></li> Free Tier: Monitoring</a></li> List of Analytics Tools</a></li> </ul> Released Mocked Streams for Apache Kafka Unknown — Mon, 24 Oct 2016 00:00:00 +0000 Kafka Streams</a> is a deployment-agnostic stream processing library written in Java. Even though Kafka has a great test coverage, there is no easy way to write unit-tests for processing topologies, until now. I released Mocked Streams</a> for Scala, which simply allows you to unit-test multi-input and output stateful stream topologies (since Apache Kafka >= 0.10.1) without Zookeeper and Kafka Brokers in your favorite Scala testing framework e.g. ScalaTest</a> and Specs2</a>.</p> import com.madewithtea.mockedstreams.MockedStreams val input = Seq(("x", "v1"), ("y", "v2")) val expe = Seq(("x", "V1"), ("y", "V2")) val strings = Serdes.String() MockedStreams() .topology { builder => builder.stream(...) [...] } .input("topic-in", strings, strings, input) .output("topic-out", strings, strings, expe.size) shouldEqual expe </code></pre> Mocked Streams</h2> Mocked Streams 1.0</a> is a library which allows you to do the latter without much boilerplate code and in your favourite Scala testing framework. It wraps the org.apache.kafka.test.ProcessorTopologyTestDriver</a> class, but adds more syntactic sugar to keep your test code simple. The features of the 1.0 release include:</p> Multiple input and output streams</li> Specify your topologies as a function: (KStreamBuilder => Unit)</li> Use .config to pass your Kafka Streams configuration (e.g. Timestamp extractor)</li> Use .outputTable to match against the compacted table output as Map[K,V]</li> Support for Scala 2.11.8</li> Support for Apache Kafka 0.10.1.0</li> </ul> Multiple Inputs and Outputs</h2> val ms = MockedStreams() .topology { builder => builder.stream(...) [...] } .input("in-a", strings, ints, inputA) .input("in-b", strings, ints, inputB) .stores(Seq("store-name")) ms.output("out-a", strings, ints, expA.size) shouldEqual(expectedA) ms.output("out-b", strings, ints, expB.size) shouldEqual(expectedB) </code></pre> Getting Started</h2> See the README</a> for using Mocked Streams. For examples check out the test code of Mocked Streams</a> itself. Mocked Streams is also located in the Maven Central Repository</a>, hence you just need to add:</p> libraryDependencies += "com.madewithtea" %% "mockedstreams" % "1.0.0" % "test" </code></pre> If you have any questions or issues regarding Mocked Streams, get in touch via Github</a>. Alternatively, you can get in contact via mail</a>.</p> Stateful Streaming in Spark and Kafka Streams Unknown — Sat, 15 Oct 2016 00:00:00 +0000 TLDR: This article is about aggregates in stateful stream processing. It covers two concrete examples</a> in Apache Spark (using the Streaming API</a> with mapWithState</a>) and Apache Kafka (using the high-level DSL</a> in Streams). Spark Streaming and Kafka Streams differ much. Therefore the reader can get to know both approaches and can decide which fits best. I discuss what features for real-time aggregates</a> besides reliability and scaling requirements we would like to see covered (e.g. to be queryable, re-processable, versionable, composable, unlimited updates, providing retention times, downsampling and having end-to-end guarantees). In the end, I will discuss the example code, also along those requirements.</p> Overview</h2> I will start this article by giving an overview of streaming frameworks</a>. After, I will discuss requirements on aggregates</a>. I would like to present an example in Spark</a> using Spark Streaming's mapWithState</a> and one example in Kafka Streams</a> using the high-level DSL</a>. Spark as a cluster computation framework relying on HDFS</a> and external databases such as Cassandra</a> or HBase</a> is very different from Kafka Streams, a topology-based deployment-agnostic processing library, which heavily relies on the distributed log system Kafka</a> and a key-value store (e.g. RocksDB</a>). Spark Streaming and Kafka Streams differ much. Therefore the reader can get to know both approaches and can decide which fits best.</p> For a complete example application in Kafka Streams, check out my previous article Processing Tweets with Kafka Streams</a>. For the examples some experience with Spark and Kafka is needed, I will refer to introduction articles.</p> Streaming Analytics</h2> Data-driven decision making in companies is becoming essential to stay competitive (adoption rates nearly tripled from 11 - 30% between 2005 and 2010, see this article</a>). Analytics are valuable for both business, business partners or customers. Data warehousing, data mining and learning from archived data is essential to measure KPIs, gain insights and to plan a strategy to reach the companies goal. With a market size of $5.1 billion in 2015, it is forecasted to $53.7 billion by 2017, see here</a>. However, in recent years, analytics have been transitioning from offline (batch) to online (streaming) approaches.</p> Streaming analytics is extensively used in a wide variety of domains such as healthcare, e-commerce, financial services, telecommunications, energy and utilities, manufacturing, government, and transportation. This paper</a> summarizes growth forecasts in the listed fields.</p> Streaming Frameworks</h2> For a conceptual introduction into stream processing, I like to refer to the introduction articles Streaming 101</a> and Streaming 102</a>. A list of vast streaming framework can be found here</a>. The following table shows Apache streaming frameworks. The original article where I got this useful comparison from is: An Overview of Apache Streaming Technologies</a>.</p> </a></p> The key differences of streaming frameworks are, when they process events (event-at-a-time or (micro-)batches, how the architecture looks like (cluster framework, library or distributed workers), what processing guarantees they have, e.g. at-least-once (duplicates possible), exactly-once processing and small latency or near-zero latency. When it comes to stateful streaming, it is also important how and where the state is stored and for how long. General problems like scalability</a>, elasticity</a>), fault tolerancy</a>, high-availability</a> and sub-problems like state migration, versioning of data / computation and reprocessing are also important to compare. All in all, as usual it highly depends on the requirements. As I wrote in the beginning, I would like to focus on aggregates with Apache Spark Streaming and Kafka Streams. But first, let's define what an aggregate is and what features we would like to have on aggregates.</p> Aggregates</h2> A streaming computation could be simple as a shell command, a web request, a database insert, any arbitrary execution, or output data in form of an updated aggregate, a stream of transformed events, new data triggered by input events, new data dependent on external lookups in other tables (e.g. machine learning models</a>) and so forth. The use case of maintaining a realtime aggregate is however the most interesting one since it needs to store a state somehow, somewhere.</p> For example, an aggregate could be the total amount of viewers of a page in a certain time window. This aggregate depends on events fired in this event time window. Desired features of such an aggregate, besides reliability, scalability and fault tolerancy, would be: to be queryable (like a database, give me the current state of the aggregate), reprocessable (if the aggregated was computed in the wrong way), versionable (we have multiple versions of the aggregate since the computation changed), composable (one aggregate depends on another aggregate), unlimited updates (no watermarking needed, you should be able to always update an aggregate, some use case exceptions later on), sending updates (changes) of aggregates downstream (for consumers, e.g. analytics UI), providing retention times (when is this aggregate obsolete or not useful anymore), downsampling (different resolutions of an aggregate value) and having end-to-end guarantees (is this event used for the current aggregate?).</p> In the next sections I will give the two concrete examples and will then refer to those features some of them rely on design decisions of the overall architecture and some directly depend on the streaming framework.</p> Aggregates in Spark Streaming</h2> Even though Spark 2.0.0 introduced Structured Streaming</a>, in which you should be able to perform SQL queries on infinite data frames representing streams, Structured Streaming is an ALPHA RELEASE, and the APIs are still experimental. Many important features are not available yet, e.g. joins between streams</a>, the distinct operation</a> and the update output mode</a> for aggregate changes. Therefore, for general purpose stream processing, I highly recommend to stick to the Spark Streaming API, which before 1.6 supplied the updateStateByKey</a> DStream function, and since 1.6 the mapWithState</a> DStream function. See Faster Stateful Stream Processing in Apache Spark's Streaming</a> for an official comparison. As another reference, I also like to refer to the online book Mastering Apache Spark, chapter Working with State using Stateful Operations</a>.</p> Don't Use UpdateStateByKey</h2> The big caveat of using updateStateByKey is, that on each new value the transformation iterates over the entire state store, regardless whether a new value for each key has been consumed or not, see here</a>. Smaller caveats are: no timeout mechanism (if you need it) and it will only return what is stored. In Spark 1.6, mapWithState was introduced, which has a built-in timeout mechanism, partial updates, arbitrary return types and an initial state. A well-written comparison can be found here: Exploring Stateful Streaming with Apache Spark</a>. Okay, let me show a simple example employing mapWithState: Maintaining a distinct set of strings per key.</p> mapWithState</h2> mapWithState has the following function signature: mapWithState[StateType, MappedType](spec: StateSpec[K, V, StateType, MappedType]). K, V are the types of the input stream, the StateType the type of the state stored and MappedType the returning stream type. The StateSpec can be defined by a function. In the following example, let us assume an input stream of type DStream[(SiteId, String)] representing page view events partitioned by SiteId. The string can represent a user id.</p> def stateSpecFunc(batchTime: Time, key: SiteId, value: Option[String], state: State[List[String]]): Option[(SiteId, Int)] = { val v = value.get if (state.exists()) { val currentSet = state.get() if (currentSet.contains(v)) { None } else { state.update(currentSet.++(List(v))) Some(key, state.get.size) } } else { state.update(List(v)) Some(key, state.get.size) } } val spec = StateSpec.function(stateSpecFunc) </code></pre> As you can see, we have to do the state initialization and retrieval manually. As discussed, we can return an arbitrary DStream with mapWithState. We now use this trackStateFunc in our processing graph.</p> val stateSpec = StateSpec.function(trackStateFunc _).numPartitions(2) val stateStream = input.mapWithState(stateSpec) val stateSnapshotStream = stateStream.stateSnapshots().map { state => (state._1, state._2.size) } stateSnapshotStream.print() ssc.checkpoint("file:/tmp/") ssc.start() // Start the computation ssc.awaitTermination() // Wait for the computation to terminate </code></pre> Features</h2> Let's re-visit the desired features of aggregates which we would like to have. First, the aggregate should be queryable. The state is stored virtually in an RDD and physically spread over the cluster worker nodes. Besides there is no official API to query the state. Therefore we need to store the aggregates somewhere downstream e.g. database. We can reprocess the data if we stored the original events (for instance in a Kafka log) and reprocess by event time. For event time processing we need to time-bucket events manually by a groupByKey operation. If reprocessing is a common use case, the iteration is an important variable when it comes to storing and querying. Versioning in the database and in the query API needs to manually implemented. Since we can return whatever data type in the state function, we can compose aggregates.</p> Let's discuss unlimited updates to an aggregate. So, if you deal with aggregates which are temporary and will be closed (e.g. an event window, any events after a certain watermark</a> will be discarded for that window, even if the events lies in that window), you can push the final aggregate downstream and save into e.g. a database. In the case when the aggregate is not temporary, the RDD will grow, and gets eventually spilled onto disk. Retention times and downsampling needs to be done manually. Exactly-once processing guarantees can be achieved with extra effort, see here</a>. As already discussed, both functions mapWithState and updateByStateKey the state is implemented using the RDD concept, therefore checkpointing</a> for recovery in case of failure is also supported by the state (in the file system as well as HDFS).</p> Towards Structured Streaming</h2> Tathagata Das mentions in his talk Apache Spark 2.0: A Deep Dive Into Structured Streaming</a> some pain points which led to the development of structured streaming, including not having the concept of event-time processing, using the same engine for batch and streaming, and the difficulty of reasoning about end-to-end guarantees. Interestingly, he also mentions the table stream duality</a>, originally described here</a> by Jay Kreps. This concept inspired the Kafka people to fundamentally rethink stream processing.</p> Aggregates in Kafka Streams</h2> The latest version 0.10 of Kafka introduces Kafka Streams. It is a deployment-agnostic stream processing library with event-at-a-time</em> (not micro-batch) semantics written in Java. It scales via partitioning and tasks</a>, is fault-tolerant</a> and has an at-least-once</a> guarantee when it comes to processing records. The framework tries to focus on simplicity in the semantics. Especially the 'Tables and Streams are Dual' concept is very interesting, see Jay Kreps</a> or Guozhang Wang</a> explaining it. To give you a short example:</p> A stream can be aggregated into a table, e.g. an aggregation of a stream in a time window can be seen as a table for a time span in a local state. The change log contains every update to that table. So you can transform the windowed aggregate to a stream of changes, which in term can be used in other aggregates or even joins. If you have not already read it, I highly recommend reading the introduction article</a>.</p> Without further introduction let us examine the example. In this example, we do same processing: Maintaining a set of strings by key. The processing is defined in the high-level DSL in the latest Kafka 0.10.1, see the documentation</a>.</p> class SetInitializer[V] extends Initializer[Set[V]] { override def apply() = Set[V]() } class DistinctAggregator[K, V] extends Aggregator[K, V, Set[V]] { override def apply(k: K, v: V, t: Set[V]) = t + v } builder .stream(Serde[String], Serde[String], topic) .groupByKey(Serdes.String(), Serdes.String()) .aggregate( new SetInitializer[String], new DistinctAggregator[String, String], Serde[List[String]], "store-name") .toStream .to(Serde[String], Serde[Set[String], output-topic) </code></pre> The return type of the aggregation is a KTable</a> reference. Since we are interested in the change stream, we transform the KTable back to a change stream. The operation after is the most interesting part, also check this example</a> of Guozhang Wang. Aggregate() on a KStream can be used to store an aggregate state. One aggregate per key is maintained. The aggregate itself must be defined by the user with a custom Initializer</a> and an Aggregator</a>. You might also want to check out other articles. Here are the official Confluent Scala</a> and Java examples</a>. The documentation for these examples can be found here</a>. Another good resource is the Kafka papers and presentations</a> wiki page.</p> Conclusion</h2> If you want to start using it seriously, I recommend you to read the referenced articles. If you have any questions or suggestions on my article, feel free to send me an email. See the contact page</a> for contact information.</p> Resources</h2> Streaming 101</a> and Streaming 102</a></li> Awesome Streaming List</a></li> </ul> Spark-related Resources</h2> An Overview of Apache Streaming Technologies</a></li> Structured Streaming in Spark</a></li> Spark Streaming Programming Guide</a></li> Faster Stateful Stream Processing in Apache Spark's Streaming</a></li> Working with State Stateful Operations in Mastering Spark</a></li> Exploring Stateful Streaming with Spark</a></li> </ul> Kafka-related Resources</h2> Introducing Kafka Streams: Stream Processing Made Simple</a></li> Official Kafka Streams Documentation</a></li> Processing Tweets with Kafka Streams</a></li> Kafka Papers and Presentations Wiki</a></li> Applying Kafka Streams for internal message delivery pipeline</a></li> Table-Stream Dualism Video</a></li> </ul> Testing Topologies with Kafka Streams Unknown — Fri, 30 Sep 2016 00:00:00 +0000 UPDATE: I released the first version of Mocked Streams. See Released Mocked Streams for Apache Kafka</a>.</p> TLDR: Kafka Streams</a> is a deployment-agnostic stream processing library written in Java. Even though Kafka has a great test coverage, there is no helper code for writing unit-tests for your own Kafka Streams topologies. I wrote a little helper library Mocked Streams</a> in Scala, which allows you to create lightweight parallelizable unit-tests for your topologies without running a full Kafka cluster neither an embedded one. See this code example</a>.</p> Kafka Streams</h2> The latest version 0.10 of Kafka introduces Kafka Streams. It is a deployment-agnostic stream processing library with event-at-a-time</em> (not micro-batch) semantics written in Java. It scales via partitioning and tasks</a>, is fault-tolerant</a> and has an at-least-once</a> guarantee when it comes to processing records. If you have not already read it, I highly recommend reading the introduction article</a> to Kafka Streams.</p> Unit-Testing Topologies</h2> If you had your first hands-on Kafka Streams already, you might have noticed that there is no easy way to unit-test your topologies. Even though Kafka is documented quite well, it lacks, for now, of good documentation of how to test. In general, you can choose between running a full Kafka cluster including Zookeeper in your environment, running an embedded Kafka broker and Zookeeper, like in this Kafka Streams integration test</a>, or you use mock producers and consumers for lightweight unit-tests of your topologies. While the first two approaches are the way to go for integration tests, we should also be able to unit-test our topologies in a simple way:</p> val input = Seq(("x", "v1"), ("y", "v2")) val exp = Seq(("x", "V1"), ("y", "V2")) val strings = Serdes.String() MockedStreams() .topology { builder => builder.stream(...) [...] } .input("topic-in", strings, strings, input) .output("topic-out", strings, strings, exp.size) shouldEqual exp </code></pre> Mocked Streams</h2> Mocked Streams</a> is a library which allows you to do the latter without much boilerplate code and in your favourite Scala testing framework e.g. ScalaTest</a> and Specs2</a>. It wraps the org.apache.kafka.test.ProcessorTopologyTestDriver</a> class, but adds more syntactic sugar to keep your test code simple. The example above is testing the following topology:</p> builder.stream(strings, strings, "topic-in") .map((k, v) => new KeyValue(k, v.toUpperCase)) .to(strings, strings, "topic-out") </code></pre> In the example, we have only one input and one output topic. However, we are able to define multiple inputs and multiple outputs. Assuming we have a topology MultiInputOutputTopology</a> which consumes two input streams, does an aggregation with a local state, and sends records to two output topics, we can test it like this:</p> val mstreams = MockedStreams() .topology { builder => builder.stream(...) [...] } .input("in-a", strings, ints, inputA) .input("in-b", strings, ints, inputB) .stores(Seq("store-name")) mstreams.output("out-a", strings, ints, expA.size) shouldEqual(expectedA) mstreams.output("out-b", strings, ints, expB.size) shouldEqual(expectedB) </code></pre> Note, that the order .input(...) calls is very important: When calling .output(...) Mocked Streams produces to the Kafka input topics in the same order as specified. In the example, it would produce all messages to topic "out-a" first, then to "out-b". Each output call will start an isolated run, fetching from the specified output topic. For a better understanding I like to refer to the tests</a> of Mocked Streams itself.</p> Usage in the Next Release of Kafka</h2> Personally, I work on next still unstable version 0.10.1 of Kafka. I was experiencing some issues back-porting Mocked Streams to the stable 0.10.0.1 release. Therefore, I decided to only support and distribute JARs for the next stable release. However, if you are interested in scratching your own itch, contribution & collaboration would be great! Unfortunately, for now, if you want to use it, you need to add Mocked Streams manually. But I will add it to the Maven Repositories</a> when the next Kafka version is released!</p> Minimalism and Elegant Code Unknown — Sun, 22 May 2016 00:00:00 +0000 TLDR; The minimalism lifestyle is a mindset in which you let go of unused belongings and unimportant attachments. But how is elegant code defined? I would define it similiar to Antoine de Saint-Exupéry's infamous definition of elegant design</em>: Elegant design is achieved not when there is nothing left to add, but when there is nothing left to take away – In this sense, the best code is the one never written, but it does fulfill the job.</p> Minimalism</h2> The minimalism lifestyle focuses on the true important needs, and letting go of unused belongings or unimportant attachments. It can have a huge impact in your life and I recommend to anyone who has a different relationship with old stuff, a toxic attitude of buying too much new stuff, to learn more about minimalism</a>. I am a minimalist, and for me it became a very important mindset which counterbalances the status quo, the world of more. I wrote two other articles about that topic: Minimalism: The Art of Living with Less</a> and Mindfulness in the Culture of Distraction</a>. Even though, it became quite popular in terms of a way of living, it is also applied to a lot of other domains like music</a>, architecture</a> and the visual arts</a>, see Artsy Minimalism</a>.</p> </p> Elegance</h2> Before we dive into minimalism and code, I want to start with an infamous definition of elegance</em> also referenced in the hacker jargon file</a>, also see where it comes from</a>. In my opinion it connects elegant engineering and minimalism very well:</p> The French aviator, adventurer, and author Antoine de Saint-Exupéry</a>, probably best known for his classic children's book The Little Prince, was also an aircraft designer. He gave us perhaps the best definition of engineering elegance when he said “A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away.” – Jargon File</cite></p> </blockquote> Elegant code is what remains</em>. Without decoration, without bloat. The pure functionality to fit the requirements – and the requirements only</em>. On StackExchange</a>, there are other possible definitions of elegant code, which I like to reference to give a broader perspective. But for the next sections, let me explain why minimalism is central to elegant code. Hence, let us start with avoiding code in the first place.</p> Avoid Code</h2> You start something new</em>. You get the requirements right and begin outlining the infrastructure, components and the protocol which might be needed to get the job done. This is design work, and it should be effective</em> and efficient</em>, but also I think, as simple as possible</em>. Lean. The same applies on a code level. Every statement you write, could be another bug</em>, could distract</em> you from the core of the logic, or could even be redundant or useless</em>. The devil is in the details, so writing less, makes him easier to spot</em>. Plus, every line you write, somebody else or even you have to read and understand over and over</em> again.</p> Legacy</h2> Months or years after, the solution gets outdated, still it works. A possible sceneario could be in terms of the Pareto principle</a>, that only twenty percent of the code base is needed for 80 percent of the business use cases. So why don't refactor? Maybe you have to get feature complete for the next release and the sales team keeps saying that only new features sell. But also on the engineering side, of course, working on new features is more fun. We need to get the priorities right. If the error handling or monitoring code for critical functionality became cryptic and untrusted over the years, you should decide slowly about refactoring it or not. Legacy code tells a story. Maybe a good, maybe a bad one. Most importantly, we should have learned something useful in retrospective.</p> </p> We all know that technical debt can be a big burden, it slows down the development cycle, scares off new employees, scares off old employees, seriously, nobody wants a share of that old pie. Even when this old pie is still making money, even when it looks shiny and new. Removing code makes space for better, more suitable and elegant code. Even if it might feel like it, it is not equivalent of burning money or business value, see Sunk cost fallacy</a>.</p> Experience over Lines of Code</h2> The real value is in the heads of the people who worked on it. Programming is mind work, and writing code is not only translating the requirements into code, it is creative work which involves the experience in coming up with a precise design: More efficient, and also simpler to understand. Think about what is actually needed, here and now, not what was needed in the past or will be needed in the future (YAGNI - you arent gonna need it</a>), you might want to write minimal code which only fits required functionality, not more - not less.</p> Striving towards Elegance</h2> Of course, you should not forget about the principles of writing clean code: Design patterns, test-driven development, continous integration, abstraction principle and so on. Having a good test coverage will help you a lot getting rid of bloated or even unused code. I strongly believe, that striving towards elegant design is a very important trait every good engineer or programmer should have or should develop to excel in her or his profession. Similiar to Occam’s Razor</a> of chosing the hypothesis with the least assumptions, we should write minimal code without code bloat</a>, otherwise it fails to work, is hard to use, hard to maintain, and is less applicable.</p> Emptiness</h2> Let me give you another perspective, which came into my mind. I will start explaining it for the minimalism lifestyle. By buying stuff or gaining attachments you remove emptiness in your life. You need the extra time to maintain and spare attention to these things which are nice to have but not ultimately important for you, like a second car, second bicycle, or whatever crosses your mind now. On the contrary, emptiness does not need to be maintained, and you would have more time for significant parts of your life. Shifting the perspective to code: While writing code, you remove emptiness. And emptiness does not need to be maintained. Emptiness can be seen as a reservoir or budget of time. Hence, if you remove too much emptiness, you end up having legacy code you are not able to maintain in time while also writing new code.</p> </p> Last Words</h2> In my opinion, the principle of elegant code should be taught in lesson one of programming. It is simple but powerful like minimalism, the life-changing trait, itself. I hope this article will fire up your motivation of thinking smart, writing less and reflecting on requirements and code. But keep in mind, that nothing will change, until you respect and integrate it into your daily working routine. Therefore I am glad that we cultivate this attitude on a daily basis at work. Spread the article if you like it and let us discuss the idea.</p>

End of Alcohol

Investing into Drosera's Seed Round

Leading Arcium's Strategic Round

Cold Shower Therapy and Ritual

Obsidian and (Local) LLMs

Starting to Sail

Enablers and Catalysts in Lightning Network

From Scratch in Zola

The Multi-Chain Future

Trust in Blockchains and Smart Contracts

Kafcache: Memcached and Kafka Streams

Reliable Fast Access to On-Chain Data

Weird Quirks we found in Ethereum Nodes

3. No Error Response Standards</h2> Even though there is a standard what kind of RPC methods and parameters exist. There are no standard error responses across nodes. Hence, if you want to write software that integrates with different clients you have to parse responses dependent on the client.</p>

Using Monix with Kafka, Avro and Schema Registry

Via Negativa Minimalism

References</h2> I wrote an article about minimalism, which offers questions you can ask yourself when you consider buying new things. I hope both articles inspire you to become a more conscious consumer and owner. And please share the article if you enjoyed it.</p>

Countermeasures Against Phone Addiction

Monitoring Kafka Streams Metrics via JMX

Released Mocked Streams for Apache Kafka

Stateful Streaming in Spark and Kafka Streams

Testing Topologies with Kafka Streams

Minimalism and Elegant Code

3. No Error Response Standards</h2>
Even though there is a standard what kind of RPC methods and parameters exist. There are no standard error responses across nodes. Hence, if you want to write software that integrates with different clients you have to parse responses dependent on the client.</p>

References</h2>
I wrote an article about minimalism, which offers questions you can ask yourself when you consider buying new things. I hope both articles inspire you to become a more conscious consumer and owner. And please share the article if you enjoyed it.</p>