<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:media="http://search.yahoo.com/mrss/"><channel><title><![CDATA[Aaron Mak]]></title><description><![CDATA[Figuring out life and software]]></description><link>https://arnmk.com/</link><image><url>https://arnmk.com/favicon.png</url><title>Aaron Mak</title><link>https://arnmk.com/</link></image><generator>Ghost 5.28</generator><lastBuildDate>Fri, 13 Mar 2026 04:56:10 GMT</lastBuildDate><atom:link href="https://arnmk.com/rss/" rel="self" type="application/rss+xml"/><ttl>60</ttl><item><title><![CDATA[Software engineering feedback loop]]></title><description><![CDATA[How software engineers explore and execute.]]></description><link>https://arnmk.com/software-engineering-feedback-loop/</link><guid isPermaLink="false">61c53310953fcb0001b17eca</guid><category><![CDATA[software]]></category><dc:creator><![CDATA[Aaron Mak]]></dc:creator><pubDate>Tue, 04 Jan 2022 09:46:41 GMT</pubDate><media:content url="https://images.unsplash.com/photo-1512758017271-d7b84c2113f1?crop=entropy&amp;cs=tinysrgb&amp;fit=max&amp;fm=jpg&amp;ixid=MnwxMTc3M3wwfDF8c2VhcmNofDE1fHxmZWVkYmFja3xlbnwwfHx8fDE2NDAzMjI0NjU&amp;ixlib=rb-1.2.1&amp;q=80&amp;w=2000" medium="image"/><content:encoded><![CDATA[<img src="https://images.unsplash.com/photo-1512758017271-d7b84c2113f1?crop=entropy&amp;cs=tinysrgb&amp;fit=max&amp;fm=jpg&amp;ixid=MnwxMTc3M3wwfDF8c2VhcmNofDE1fHxmZWVkYmFja3xlbnwwfHx8fDE2NDAzMjI0NjU&amp;ixlib=rb-1.2.1&amp;q=80&amp;w=2000" alt="Software engineering feedback loop"><p>Too many times, software engineers, myself included, write code before even thinking enough about it. This usually results in time wasted both in rewrites, refactors, or worse, removal. </p><p>I&apos;ve been thinking a bit about when we should give feedback and the consensus is early, frequent and actionable. Yet, we also need to balance it with the cost of giving feedback because it takes effort and time.</p><p>Everyone should first ask this question, &quot;Do I need to write code?&quot; Sounds like a stupid question but when your job is writing code, it&apos;s easy to jump straight to it. If we don&apos;t ask ourselves that question, we likely have not fully understood the problem we are trying to solve. Writing code has to be the best way to solve the problem. Once the problem is understood, and the tradeoffs of writing and maintaining the code we write are worth it, then it&apos;s time to get some feedback. </p><p>Now, down to the nitty-gritty of it. There are many ways to get feedback on software engineering tasks. A non-exhaustive list includes meetings, code reviews, code pairing and RFCs. How should we decide when to use which form of feedback?</p><p>Going back to balancing cost vs benefit, it depends on how much we already know about the problem we need to solve, and what kind of feedback we are looking to get. Questions that we ask should help us determine how clear we are on the scope and how concrete this scope is. Some examples of questions that we should ask are as follows.</p><ul><li>What is the overall direction and strategy?</li><li>Who are our stakeholders?</li><li>How do we measure success or failure?</li><li>Are we the best team or person for the task?</li><li>Have we as a team done something similar before?</li><li>How familiar is the task to the one executing?</li></ul><h2 id="explore">Explore</h2><p>If this problem hasn&apos;t been solved before and it&apos;s entirely new, meetings or RFCs are likely worth doing. Code shouldn&apos;t be written before the team is clear on the direction it&apos;s heading. Timelines need to be flexible and milestones, shorter because all of them could change once execution starts as we learn new information. You might not have all the details sorted out; what you&apos;re looking for is early feedback and agreement between stakeholders in the direction you are heading. If stakeholders are not technical, you&apos;ll need to simplify your explanations or split up the meetings and RFCs for different audiences.</p><p>Imagine having to change scope or direction once code is already written. From experience, it&apos;s costly and you would want to avoid it at all costs. </p><p>The length of this phase depends on the entire scope of the project. There might be various stakeholders from the C-suite down to the engineers in your team. Depending on your role and the size of the project, you might want to also split up meetings for discussions and meetings for decisions. This helps increase the quality of discussions and you can have more information on the table before proposing or making a decision.</p><h2 id="execute">Execute</h2><p>Once everyone is clear on the direction of the project, it&apos;s down to the next most important bit &#x2013; execution. I won&apos;t go into details about what good software engineering practices are because that would take up a lot more than one post but I&apos;ll assume that your team is aligned on those. If they aren&apos;t, no one can help you here.</p><p>If you or your team has not done this task before, it&apos;d be worthwhile to start with code pairing. Feedback would be real-time and any decision would have to be taken by at least two people, which in software engineering, is usually already a good proportion of your team. If at least one person in the team has done a similar task before, it&apos;ll be worth having this person pair with someone that hasn&apos;t. A good team shares information to make better decisions and pairing helps in transferring this informal knowledge that might be difficult to write down. Every line of code written influences other lines of code so inconsistency is obvious and confusing.</p><p>The cost of code pairing is undeniably time, and probably more exhausting as it requires constant communication between two people. That said, most of the time, it&apos;s worth it as code quality is higher and the team communicates more effectively. When the pull request is finally created, it&apos;s just a formality.</p><p>If a task is very familiar to the team and the one writing the code is very familiar with it, it could be worth just jumping straight to creating the pull request. Feedback in the pull request is likely slower and less frequent but that&apos;s okay if there are only one or two minor comments. </p><p>Warning signs of creating the pull request too early is a long list of comments or one major comment that would cause a huge rewrite. If we observe that, to save everyone&apos;s time, it&apos;s better to take a step back and go back to the earlier stages of feedback.</p><h2 id="tldr">TLDR</h2><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2021/12/explore-and-execute.png" class="kg-image" alt="Software engineering feedback loop" loading="lazy" width="1162" height="769" srcset="https://arnmk.com/content/images/size/w600/2021/12/explore-and-execute.png 600w, https://arnmk.com/content/images/size/w1000/2021/12/explore-and-execute.png 1000w, https://arnmk.com/content/images/2021/12/explore-and-execute.png 1162w" sizes="(min-width: 720px) 720px"><figcaption>Explore, Execute, Repeat</figcaption></figure><p>How much we explore depends on how many unknown unknowns we think there are. And once we are clearer on what we don&apos;t know and we are confident enough, we start writing code, learning more as we start executing. If we discover more unknowns along the way, we may need to take one step back and explore again. It&apos;s seldom a one-way street but more like a cycle of exploring and executing because there will always be something that we didn&apos;t realize we don&apos;t know.</p><p>As a software engineer, feedback comes in various forms. And we need to know the tradeoffs between each form before deciding which is best. Sometimes, ego comes in the way and we&apos;ll need to learn how to navigate others and our own ego to gather feedback and respond to it well. No one is perfect at it but like anything else, it gets easier with practice.</p>]]></content:encoded></item><item><title><![CDATA[Some Expectations at Work]]></title><description><![CDATA[Since I'm about to lead a team, so thought it'd be helpful to have these things written down. They might change over time but hopefully not too much. I'd also want to be transparent and clear about what I'd expect from my team.]]></description><link>https://arnmk.com/some-expectations-at-work/</link><guid isPermaLink="false">5fc0ed2d9bbf620001022695</guid><category><![CDATA[software]]></category><category><![CDATA[management]]></category><dc:creator><![CDATA[Aaron Mak]]></dc:creator><pubDate>Fri, 27 Nov 2020 13:14:51 GMT</pubDate><content:encoded><![CDATA[<p>Since I&apos;m about to lead a team, thought it&apos;d be helpful to have these things written down. They might change over time but hopefully not too much. I&apos;d also want to be transparent and clear about what I&apos;d expect from my team.</p><h2 id="work-isn-t-the-most-important-thing">Work isn&apos;t the most important thing</h2><p>Somewhat contrarian, especially in Singapore. No one should prioritise work above all else. It&apos;s <u>inhumane</u> to expect one to do so. Life is so much more than just work. Even if you love your job, sometimes you&apos;ll be bored, tired or just think it isn&apos;t worth it. And that&apos;s okay, it&apos;s normal. Sometimes we feel that way because of people we work with or a project isn&apos;t completed the way we&apos;d like it to be. </p><p>If you&apos;re someone who thinks work is the most important thing, have a read of this excerpt from a book in the Bible, written by a king around 3000 years ago who was richer than Bill Gates. </p><blockquote>I hated all my toil in which I toil under the sun, seeing that I must leave it to the man who will come after me, and who knows whether he will be wise or a fool? Yet he will be master of all for which I toiled and used my wisdom under the sun. This also is vanity. So I turned about and gave my heart up to despair over all the toil of my labors under the sun, because sometimes a person who has toiled with wisdom and knowledge and skill must leave everything to be enjoyed by someone who did not toil for it. This also is vanity and a great evil. What has a man from all the toil and striving of heart with which he toils beneath the sun? For all his days are full of sorrow, and his work is a vexation. Even in the night his heart does not rest. This also is vanity. <br>- Ecclesiastes 2:18-23</blockquote><p>To clarify, this book doesn&apos;t say work (toil) isn&apos;t important at all, though it might make you wonder what&apos;s important then? You&apos;ll have to <a href="https://www.esv.org/Ecclesiastes+1/">read on</a> to get the full picture or you can ask me for a fuller explanation. </p><p>And for most of us, we&apos;d naturally think there are a lot of things more important, like our religion, family, friends, hobbies. That&apos;s normal and I don&apos;t expect otherwise. I don&apos;t mean you should do a bad job at work of course. I&apos;d expect good work but not at the expense of what you think is more important. And I&apos;d argue that we need to prioritise things we believe are more important so that we can do good work. We are made to work but it&apos;s not the most important thing. Even rest at times could be more important than work.</p><h2 id="be-an-owner">Be an owner</h2><p>See a bug, fix it. See a spelling mistake, fix it. See bad documentation, write good ones. If you own something, you&apos;d take good care of it. I like the idea of a <em>Maintainer</em> coined in software engineering<em>. </em>You are responsible for maintaining what you write, especially in software where every commit is signed by you. Don&apos;t write sloppy code and say you have to merge it because of a deadline. <a href="https://www.youtube.com/watch?v=pqeJFYwnkjE">Sloppy code is not tech debt</a>. Made bad decisions or merged a bug? Take responsibility, own up and do something about it. And be proud of your work.</p><h2 id="we-are-a-team-first">We are a team first</h2><p>Since we are working together towards an arbitrary goal, don&apos;t be afraid to say what&apos;s on your mind. Don&apos;t be afraid to say hard truths we need to hear, even to your manager. Sometimes our opinions turn out to wrong, and that&apos;s okay. As a team, there is room for mistakes. As long as we own up, fix it, learn and move on. </p><p>If you are having personal issues, or unable to deliver some time in the promised time, let us know and trust that we will fill in the gaps for you. </p><p>Of course, all that assumes that we trust each other enough over time.</p><h2 id="learn-how-to-learn">Learn how to learn</h2><p>It&apos;s easier to teach people what they should do than teach people to figure that out for themselves. Knowledge is everywhere but to discern what knowledge is worth learning is a difficult skill.</p><p>The smartest people aren&apos;t those that know the most, but those that can figure out how to best leverage on the knowledge of others. </p><p>I think it boils down to three things. How to ask good questions, how to think critically, and to be comfortable being wrong. And that&apos;s what I&apos;d expect. Not writing the most efficient algorithm, writing the most elegant code or even scoping the most impactful project. But I&apos;d think you&apos;ll get there if you ask good questions and think critically in a way that doesn&apos;t assume you&apos;re always right.</p><p>In software engineering, most people think that you need to be the most logical person, or the smartest one to be good at it. On the contrary, I found that the people I most enjoy working with, are humble, curious and communicate well. </p><hr><p><em>If you are in my team, let me know what you think. If you have any feedback regarding any of my expectations that were mentioned, you&apos;re more than welcome to talk to me about it.</em></p>]]></content:encoded></item><item><title><![CDATA[Don't be afraid of the command line]]></title><description><![CDATA[We are so used to graphical user interfaces that we forget computers started off without it. As a software engineer, there are many reasons why using the command line can help with development. ]]></description><link>https://arnmk.com/dont-be-afraid-of-the-command-line/</link><guid isPermaLink="false">5f71941f4dbbc40001ee8b3b</guid><category><![CDATA[software]]></category><dc:creator><![CDATA[Aaron Mak]]></dc:creator><pubDate>Mon, 28 Sep 2020 10:05:15 GMT</pubDate><media:content url="https://arnmk.com/content/images/2020/09/scary-technology.png" medium="image"/><content:encoded><![CDATA[<img src="https://arnmk.com/content/images/2020/09/scary-technology.png" alt="Don&apos;t be afraid of the command line"><p>We are so used to graphical user interfaces that we forget computers started off without it. As a software engineer, there are many reasons why using the command line can help with development. </p><ul><li>Less context switching -&gt; more focus</li><li>Lightweight -&gt; faster development</li><li>Deeper understanding of tools and the computer</li><li>Higher interoperability between computers</li></ul><p>On an aside, the recent trend of containerization requires you to be familiar with the command line. With most containers being in linux and entering them would mean that there&apos;s no user interface so you have to depend on the good old fashion command line. </p><h2 id="cli-basics">CLI Basics</h2><p>I&apos;ll assume you&apos;re either using a Mac or a Linux because it might work differently in Windows. And most development machines are based of UNIX anyway. </p><p>With that, I&apos;ll be explaining some common command line commands that are present in most computers which I commonly use. They won&apos;t be exhaustive but I encourage you to read more of them in the <a href="https://www.amazon.com/Linux-Pocket-Guide-Essential-Commands-ebook/dp/B01GGQKXRG">Linux Pocket Guide</a>.</p><h3 id="cd">cd</h3><p>It allows you to go to a relative or absolute path.</p><pre><code class="language-bash">cd child_directory  # relative path to child_directory 
cd /usr/foo/directory  # absolute path
cd ../sibling_directory  # go to parent directory then into sibling directory
cd  # without arguments returns you to the home directory</code></pre><h3 id="export">export</h3><p>Creates environment variables. Note that you&apos;ll need to define these environment variables in a <code>.zshrc</code>, <code>.bashrc</code> or any file that loads when the shell is created if you want this environment variable to always be present. Otherwise, this environment variable would not persist beyond it&apos;s current shell.</p><pre><code class="language-bash">export FOO=bar
echo ${FOO}  # would show `bar`</code></pre><h3 id="printenv">printenv</h3><p>Helpful for debugging to figure out what are the current environment variables.</p><h3 id="history">history</h3><p>Prints the commands you have used previously. Helpful when used with <code>grep</code>.</p><h3 id="alias">alias</h3><p>Does what it says. </p><pre><code class="language-bash">alias gp=&apos;git push&apos;
gp  # would be equivalent to typing `git push`</code></pre><h3 id="ls">ls</h3><p>Short for &quot;list&quot;. Lists attributes of files and directories.</p><pre><code class="language-bash">ls  # lists files in current directory
ls directory  # list files in that directory
ls -a  # lists all files, including hidden ones</code></pre><h3 id="mv">mv</h3><p>Short for &quot;move&quot;. Move / Rename file(s) or directories.</p><pre><code class="language-bash">mv foo.txt bar.txt  # renames foo.txt too bar.txt
mv blah.txt foo.txt destination_directory  # moves blah.txt and foo.txt to destination_directory</code></pre><h3 id="cp">cp</h3><p>Short for &quot;copy&quot;.</p><pre><code class="language-bash">cp original.txt copy.txt  # creates copy.txt which is a copy of original.txt
cp original.txt directory  # creates a copy of original.txt into `directory`</code></pre><h3 id="rm">rm</h3><p>Short for &quot;remove&quot;.</p><pre><code class="language-bash">rm foo.txt  # removes foo.txt
rm -r directory  # removes anything in this directory</code></pre><h3 id="pwd">pwd</h3><p>Prints the absolute path of your current working directory.</p><h3 id="mkdir">mkdir</h3><p>Short for &quot;make directory&quot;.</p><pre><code class="language-bash">mkdir foo  # creates foo directory
mkdir -p foo/bar/bar  # creates any parent directories if necessary</code></pre><h3 id="echo">echo</h3><p>Prints arguments passed into it.</p><pre><code class="language-bash">echo foo  # prints `foo`</code></pre><h3 id="cat">cat</h3><p>Concatenates and print files. Basically just means printing the whole file.</p><h3 id="less">less</h3><p>Something like <code>cat</code> but good for larger text and if you want to browse by pages.</p><h3 id="grep">grep</h3><p>Given some files, print lines which match a regular expression pattern. Pretty powerful stuff if you&apos;re well versed in <a href="https://en.wikipedia.org/wiki/Regular_expression">regular expressions</a>.</p><p>Given this file <code>a_text_file.txt</code>.</p><pre><code class="language-txt">80 days around the world, we&#x2019;ll find a pot of gold just sitting where the rainbow&#x2019;s ending.
Time &#x2014; we&#x2019;ll fight against the time, and we&#x2019;ll fly on the white wings of the wind. 
80 days around the world, no we won&#x2019;t say a word before the ship is really back.
Round, round, all around the world. Round, all around the world. Round, all around the world.
Round, all around the world. </code></pre><pre><code class="language-bash">grep time  # Time &#x2014; we&#x2019;ll fight against the time, and we&#x2019;ll fly on the white wings of the wind.</code></pre><h3 id="touch">touch</h3><p>Usually used to create empty files since it creates the file if it doesn&apos;t exist. It also updates the modification timestamp and access timestamp of the file.</p><h3 id="-pipe-">| (pipe)</h3><p>Passes the output of the previous command to the next. This is the key to mastering the command line. Commands on it&apos;s own don&apos;t seem that powerful but combine them with each other and the possibilities are endless. And that&apos;s what the unassuming <code>|</code> (pipe) does.</p><pre><code class="language-bash">cat file.txt | grep text  # finds `text` in every line in file.txt
history | grep git  # prints git commands you have used previously</code></pre><h3 id="xargs">xargs</h3><p>Another powerful command that reads each line of text from the input and turns passes each of them to the next command and executes them. It&apos;ll be clearer when you try out the examples.</p><pre><code class="language-bash">ls | xargs ls  # runs ls on all child directories and prints the result
grep foo --files-with-matches | xargs rm  # removes all files with `foo`</code></pre><h3 id="pbcopy-macos-">pbcopy (MacOS)</h3><p>MacOS clipboard. Helpful when you want to pipe something to the clipboard.</p><pre><code class="language-bash">cat file.txt | pbcopy  # copies file.txt contents to clipboard
grep foo | pbcopy  # copies any line with `foo` to your clipboard</code></pre><h2 id="exploring-tools">Exploring tools</h2><p>Now you know the basics, but you&apos;ll need to figure out how to use these commands on your own. Most commands have a <code>help</code> option.</p><pre><code class="language-bash">ls --help
grep --help</code></pre><p>If not, they would likely have a <code>man</code> or short for manual.</p><pre><code class="language-bash">man ls
man cat
man echo
man grep</code></pre><p>You might find the manual too verbose but I found this helpful tool called <code>tldr</code> &#xA0;(<a href="https://github.com/tldr-pages/tldr">https://github.com/tldr-pages/tldr</a>) which summarizes the manual to useful cheatsheets for me.</p><h2 id="other-useful-commands">Other useful commands</h2><p>These require a separate installation that are not found by default and to me, are better versions of default commands, but require an understanding of them.</p><ul><li><a href="https://github.com/rupa/z">z</a> - a faster cd based on your history</li><li><a href="https://github.com/tldr-pages/tldr">tldr</a> - a quicker manual</li><li><a href="https://github.com/BurntSushi/ripgrep">rg</a> - a faster grep written in rust</li><li><a href="https://github.com/peco/peco">peco</a> - a more interactive filtering tool</li><li><a href="https://github.com/eradman/entr/">entr</a> - runs a command when input changes. I mainly use this for TDD to run tests when any file changes.</li><li><a href="https://linux.die.net/man/1/tree">tree</a> - list contents of a directory in a tree-like format. Comes default with Linux, but you&apos;ll have to install it on MacOS.</li></ul><p>These commands are just scratching the surface of what you can do, so the best way to try them out is to use them in your daily workflow.</p><p>I&apos;ve also created a small practice file at <a href="https://gist.github.com/aaronmak/dcc479d30ec9e844d6756933f0f6886e">https://gist.github.com/aaronmak/dcc479d30ec9e844d6756933f0f6886e</a> if you&apos;d like to get more familiar with your command line.</p><h2 id="references">References</h2><ul><li><a href="https://www.amazon.com/Linux-Pocket-Guide-Essential-Commands-ebook/dp/B01GGQKXRG">Linux Pocket Guide</a> - similar commands to MacOS but there are slight differences.</li></ul>]]></content:encoded></item><item><title><![CDATA[Building a Dactyl Manuform keyboard with hot-swappable sockets]]></title><description><![CDATA[<p>Thinking it was similar to building one had normal switches, I tried to build my first keyboard with hot swappable sockets. I was so wrong. Not only could I not find build logs of anyone who has built one, but soldering wires to the <a href="https://kbdfans.com/products/mechanical-keyboard-switches-kailh-pcb-socket">Kalih hot swappable sockets</a> was so</p>]]></description><link>https://arnmk.com/building-a-dactyl-manuform-with-hot-swappable-sockets/</link><guid isPermaLink="false">5ec0d5c86ba5270001f76586</guid><category><![CDATA[keyboard]]></category><category><![CDATA[dactyl manuform]]></category><dc:creator><![CDATA[Aaron Mak]]></dc:creator><pubDate>Sun, 17 May 2020 15:09:43 GMT</pubDate><media:content url="https://arnmk.com/content/images/2020/05/627F1F6F-B62D-499F-9015-95C09DB79916_1_105_c-1.jpeg" medium="image"/><content:encoded><![CDATA[<img src="https://arnmk.com/content/images/2020/05/627F1F6F-B62D-499F-9015-95C09DB79916_1_105_c-1.jpeg" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets"><p>Thinking it was similar to building one had normal switches, I tried to build my first keyboard with hot swappable sockets. I was so wrong. Not only could I not find build logs of anyone who has built one, but soldering wires to the <a href="https://kbdfans.com/products/mechanical-keyboard-switches-kailh-pcb-socket">Kalih hot swappable sockets</a> was so much harder. So please read on if you&apos;re as clueless as I was.</p><p>There are many build logs out there so I&apos;ll distill what I thought was helpful and share some mistakes I&apos;ve made to prevent you from repeat them before continuing with the build log.</p><h2 id="helpful-but-not-obvious">Helpful but not obvious</h2><h3 id="soldering-the-copper-tape-first">Soldering the copper tape first</h3><p>Start with soldering the copper tape and not the wires, but leaving the top most switch unsoldered. The topmost switch should be soldered together with the wire.</p><p>I made a mistake of soldering the diodes first because most guides suggested it. To my horror, I found out later that the copper wire cannot be wrapped around the hot swappable sockets. So I had to sneak the copper tape under the existing wires, which probably added at least an hour or two to my build time.</p><h3 id="solder-the-diode-as-close-to-the-socket-as-possible">Solder the diode as close to the socket as possible</h3><p>None of the guides I read told me to do this even though some images I&apos;ve seen hinted that. Or maybe everyone already knew that except me, a soldering noob. </p><p>Soldering the diode close to the socket (or switch) ensures that the diode doesn&apos;t move around too much. There&apos;s also less of a chance that the row wire would touch another.</p><h3 id="understand-what-you-re-building">Understand what you&apos;re building</h3><p>There&apos;re many guides out there and some have subtle differences of how they wire the thumb cluster or the materials they use. Since it&apos;s my first time soldering a keyboard, I just followed a mishmash of the popular guides. I&apos;m glad it worked out but I just didn&apos;t know what the options were and the tradeoffs I was making. </p><p>Took me a while to understand how the wiring allows the controller to know which key was pressed. So even though I followed the popular wiring guide at <a href="https://github.com/abstracthat/dactyl-manuform">https://github.com/abstracthat/dactyl-manuform</a>, I might not need to in the future. The QMK guide to split keyboard also suggested an <a href="https://github.com/qmk/qmk_firmware/blob/master/docs/feature_split_keyboard.md#i2c-wiring">alternative wiring</a> to connect the 2 controllers which I didn&apos;t know existed until I flashed the controller.</p><h3 id="wiring-diagram">Wiring Diagram</h3><p>Not drawn to scale but hopefully it&apos;s clear. It&apos;s an <code>svg</code> file so you&apos;ll be able to zoom in as needed. I found that most diagrams don&apos;t have the reset switch and they only show one side at a time.</p><p>To understand this diagram, it is drawn from the perspective of the underside of the keyboard. The brown and black rectangles represent the diodes and the direction they need to be in for the current to flow the right way. I used elite-c&apos;s because I&apos;ve read that pro micros are fragile but they work as well.</p><figure class="kg-card kg-image-card kg-width-wide kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/09/Wiring-Diagram-1.svg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy" width="1707" height="1074"><figcaption>Wiring Diagram</figcaption></figure><h2 id="building-the-keyboard">Building the keyboard</h2><figure class="kg-card kg-image-card"><img src="https://arnmk.com/content/images/2020/05/CD5B9A75-6425-4122-9649-07523878F36C.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"></figure><h3 id="materials-and-tools">Materials and Tools</h3><ul><li>3D printed case with hot swappable kalih switches, base, wrist rests and elite-c holder printed by reddit user <a href="https://www.reddit.com/user/crystalhand/">crystalhand</a></li><li>2 Elite C controllers</li><li>2 3.5mm TRRS sockets</li><li>Reset button switch</li><li>64 Diodes (a few spare would be handy just in case)</li><li>Solder core</li><li>Solid core copper wires (from ethernet cables)</li><li>5mm width copper tape</li><li>Soldering Iron</li><li>Soldering Iron Tip Cleaner (An old wet cloth or sponge will do as well)</li><li>Pliers</li><li>Tweezers</li><li>Digital Multimeter (optional, but useful for debugging and troubleshooting if something goes wrong)</li><li>Screws, nuts and screwdrivers (optional, only needed if you want to cover the bottom or to attach the printed wrist rests)</li></ul><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/05/IMG_0184-1.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"><figcaption>what have I gotten myself into</figcaption></figure><h3 id="soldering-the-connections">Soldering the connections</h3><p>I started with the diodes because I wasn&apos;t expecting to use copper tape later. So don&apos;t repeat my mistake and start with the copper tape instead. I was building the 5x6 version of the dactyl manuform so it wasn&apos;t too difficult finding wiring diagrams for it. As said in other build logs, winding the diode wire around the node would help. However, it&apos;ll be trickier to wind it around the hot swappable socket.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/05/80B93FF5-DD20-4172-A9FA-B39138B4007C_1_105_c.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"><figcaption>winding the diode around the socket</figcaption></figure><p>When you get to the corners, you&apos;ll likely have to use the tweezers to finish the job.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/05/IMG_0186.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"><figcaption>After what seemed like eternity</figcaption></figure><p>As you can see, some diodes are pretty far away from the sockets, which I&apos;ll live to regret later on.</p><p>Since I&apos;ve started on the diodes, I continued by soldering the row wires. After removing the individual wires from the ethernet cable, I got to work. The wire stripper I got was too big so I had to return it, leaving the only other option which was to use the soldering iron to melt the insulation of the wires. That didn&apos;t leave me with the cleanest cuts but it worked. </p><figure class="kg-card kg-image-card"><img src="https://arnmk.com/content/images/2020/05/9153895F-EF68-4750-9CC4-0F69046C04A6_1_105_c.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"></figure><figure class="kg-card kg-image-card"><img src="https://arnmk.com/content/images/2020/05/2B25950A-C7E3-4125-B1E6-1F42449EE43E_1_105_c.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"></figure><p>Now for the columns! I assumed I could wind the wire around the socket like all the guides for the switches mentioned but I was clearly mistaken. The copper wires were a little thicker than the diode wires, which resulted in me being unable to wind them around the sockets. I saw some guides that used copper tape so I ordered them online.</p><p>After waiting for a couple of days for the copper tape to arrive from a local hardware store, I realised that it&apos;ll be difficult to get the tape under the wires. After much patience and perseverance, it was complete. </p><figure class="kg-card kg-image-card"><img src="https://arnmk.com/content/images/2020/05/166F33E9-6EE8-48E9-A211-70016A6863F9_1_105_c.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"></figure><p>I found the thumb cluster to be insanely difficult with the copper tape so a tip is to plan where you&apos;re going place the tape. </p><p>Soldering the wire to the copper tape was also tricky because don&apos;t forget, I couldn&apos;t wind it around the socket. What I found helpful was to tape one side of the wire such that pressure was applied to the end of the wire in contact with the tape. I know it sounds vague so hopefully this next photo would help.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/05/B02822E9-0327-4AFB-8ACF-A3972A896806_1_105_c.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"><figcaption>Masking tape to hold the wire in place while soldering</figcaption></figure><p>The exposed end of the wire was bent to be in contact with the socket and copper tape at the same time. With the masking tape helping it stay in place, soldering was more manageable.</p><h3 id="soldering-the-components">Soldering the components</h3><p>Once the rows and columns are complete, it was time to solder the components. Before soldering to the elite-c, you&apos;ll need to wind the exposed end of the wire around the side. It&apos;s not that clear but the wire needs to enter from the top and curl at the bottom because the soldering will be done below the controller. Ensure that this is done for all wires so it&apos;s consistent.</p><p>One mistake I made here was to not give enough slack for the wires so I had to solder 2 wires together because I realised one was too short. If you have the holder for the elite c, ensure that you have enough slack so that elite c can peek out through the cavity for the holder to fit.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/05/21A1F656-2117-46A4-B64E-C61C0D5D71FA_1_105_c.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"><figcaption>pardon the messy wiring</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/05/IMG_0212.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"><figcaption>Soldering below the controller, with the wires entering from the top</figcaption></figure><p>Quite a few guides would suggest a specific <code>GND</code> position for the TRRS wire but any one of them would do as long as it&apos;s labeled <code>GND</code>. Since I was going to solder the reset switch to <code>GND</code> and <code>RST</code>, I used the <code>GND</code> adjacent to <code>RST</code> for the reset switch. </p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/05/8681E917-E293-419A-A466-50ED446083F4_1_105_c.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"><figcaption>Soldering wires to the reset switch</figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/05/661B0183-FDF8-4CD2-A8CA-FFA2F7F2AD3F_1_105_c.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"><figcaption>Soldering wires to the TRRS socket</figcaption></figure><p>And before I knew it, I was finally done! Now came the easiest part, adding the switches and keycaps.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/05/7AA1B6CE-32EB-4F96-B6DE-E768305B175E_1_105_c.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"><figcaption><a href="https://zealpc.net/products/tealios">Tealios Switches</a></figcaption></figure><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/05/50E7522F-8B33-45CE-8402-421590498E6B_1_105_c.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"><figcaption>SA Vilebloom from MechSupply (originally for Ergodox)</figcaption></figure><h3 id="flashing-the-controller">Flashing the controller</h3><p>The open source program <a href="https://github.com/qmk/qmk_firmware">QMK</a> is what you&apos;re looking for to configure it. For the first flash, I just got the defaults working so I could test out if any keys weren&apos;t working. Following the instructions, I used homebrew to install it on my mac. It was surprisingly easy to install. </p><pre><code class="language-bash">brew tap qmk/qmk
brew install qmk</code></pre><p>QMK is well documented and I&apos;d advise you to read the <a href="https://docs.qmk.fm/#/">documentation</a> before continuing. After reading the documentation, I flashed the left side first and connect the right side for the default settings. And it seemed everything was going well.</p><p>It turned out I celebrated too early. After testing the keyboard at <a href="https://config.qmk.fm/#/test">qmk</a>, I found 4 keys that weren&apos;t working. Immediately, I rushed to get the multimeter to diagnose the issue. Finding out that the circuits were working fine which meant no more soldering, I heaved a sigh of relief. It was the switches&apos; pins that were bent. After straightening them out, it was perfect.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/05/627F1F6F-B62D-499F-9015-95C09DB79916_1_105_c.jpeg" class="kg-image" alt="Building a Dactyl Manuform keyboard with hot-swappable sockets" loading="lazy"><figcaption>Just another shot of the completed keyboard &#x1F60E;</figcaption></figure><p>After spending some time tinkering with the online configurator, I arrived with one which mimicked the layout of my <a href="https://www.ergodox.io/">ergodox</a>. Don&apos;t worry if you don&apos;t know how to use a text editor or read <code>C</code>. The online QMK configurator and toolbox doesn&apos;t require you to use any text editor. </p><p>And after pressing the reset button followed by one final flash, that&apos;s it! </p><p>A few days in and I&apos;m pretty sure I&apos;ll be using this keyboard for a long time to come. </p>]]></content:encoded></item><item><title><![CDATA[Learning Vim in 30 days]]></title><description><![CDATA[<p>If you&apos;d like to learn vim, I&apos;ll be sharing what I found helpful while picking it up. Vim seems ancient with all the fancy new IDEs like Visual Studio Code. However, I&apos;ve found vim to greatly improve my productivity. Within a month, I started</p>]]></description><link>https://arnmk.com/learning-vim-in-30-days/</link><guid isPermaLink="false">5ead6c3f515393000118ebed</guid><category><![CDATA[tools]]></category><category><![CDATA[vim]]></category><category><![CDATA[software]]></category><dc:creator><![CDATA[Aaron Mak]]></dc:creator><pubDate>Sun, 03 May 2020 01:00:00 GMT</pubDate><content:encoded><![CDATA[<p>If you&apos;d like to learn vim, I&apos;ll be sharing what I found helpful while picking it up. Vim seems ancient with all the fancy new IDEs like Visual Studio Code. However, I&apos;ve found vim to greatly improve my productivity. Within a month, I started to use vim full time and never looked back.</p><p>The timelines are just a guideline so feel free to skip ahead or slow down.</p><h2 id="why-vim">Why vim</h2><p>I saw a few colleagues using vim and I was intrigued. I wondered how a text-only interface can work better than an IDE. At first, vim seemed difficult to use and I thought it was only for the nerdiest. The bar was too high, or so I thought.</p><p>I can&apos;t find the article but I remember it talking about how programming is like surgery. In a sense, most of the time, we make minor tweaks just like surgery. But we might also repeat the same change throughout the repository. And after learning that vim is built to do those things without any fancy tooling, I was convinced to give it a shot.</p><h2 id="week-1-vim-tutor">Week 1 - Vim Tutor</h2><p>Vim tutor comes with every UNIX computer. To figure out whether you have it, simply go to your terminal and enter <code>vimtutor</code>. If you&apos;re using a windows machine, refer to <a href="https://superuser.com/questions/270938/how-to-run-vimtutor-on-windows">https://superuser.com/questions/270938/how-to-run-vimtutor-on-windows</a>.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://arnmk.com/content/images/2020/05/Screenshot-2020-05-02-at-10.29.23-PM.png" class="kg-image" alt loading="lazy"><figcaption>The beginning of vimtutor</figcaption></figure><p>The beauty of the file is that it teaches you through practice. Getting used to the keybindings in vim requires some unlearning of what you&apos;re used to. And there&apos;s no better way than to simply practice. Just like learning a musical instrument, if you are struggling with any chapters, repeat them until you are confident.</p><p>Spend a couple of hours in the first week to play around with vim tutor. And once you&apos;re comfortable with the basic modes and keys, you&apos;re ready for vim.</p><h2 id="week-2-vanilla-vim">Week 2 - Vanilla Vim</h2><p>At this point, you should be able to use vim to edit files but you might find the vim defaults a bit of a hassle. I&apos;d recommend tpope&apos;s <a href="https://github.com/tpope/vim-sensible">vim-sensible</a> plugin to help start you off with some defaults that make sense while still allowing you to explore and use vim.</p><p>Learn how to use buffers or tabs and how to navigate using vim commands and motions.</p><h2 id="week-3-cli-tools">Week 3 - CLI tools</h2><p>If you are familiar with the command line and Linux, you can probably skip this. Vim and the terminal work hand in hand so it&apos;d feel like typing with one hand rather than two if you didn&apos;t know how to use command-line tools.</p><p>A place to start would simply be your terminal. Basics include <code>ls</code>, <code>mkdir</code>, <code>cp</code>, <code>mv</code>, <code>touch</code>, <code>cat</code>, <code>rm</code> and many more. A book I&apos;d recommend would be <a href="https://www.amazon.com/Linux-Pocket-Guide-Essential-Commands-ebook/dp/B01GGQKXRG">Linux Pocket Guide</a>.</p><h2 id="week-4-customise">Week 4 - Customise</h2><p>Vim is built to be infinitely customisable. Every software engineer&apos;s vim is different. Some people like to use vanilla vim so it&apos;s easy to be as efficient on remote machines. Others might install so many plugins that their vim might be mistaken for an IDE.</p><p>Whatever the case, you would still require a .vimrc file to add your configs. I&apos;d recommend slowly changing that file and only adding things when you&apos;ve gotten used to your current setup. A mistake I&apos;ve made was to add too many things at the beginning without fully understanding what I was adding. That caused some issues later down the road when things broke.</p><h2 id="practice-makes-perfect">Practice makes perfect</h2><p>If you have come this far and aren&apos;t convinced, you could always return to your IDE. But you are convinced, practice makes perfect. There&apos;s no better way to be more proficient than using it for your daily projects.</p><p>There are endless things to learn about vim so it&apos;s quite likely I&apos;ve only scratched the surface. I&apos;d also recommend using vim with <a href="https://github.com/tmux/tmux">tmux</a> once you&apos;re confident and try out <a href="https://neovim.io/">neovim</a> as well.</p>]]></content:encoded></item><item><title><![CDATA[Maintainable SQL in Data Warehousing]]></title><description><![CDATA[<p>Since SQL isn&apos;t a programming language, and say you don&apos;t have the option of using a programming language to generate SQL, how do you go about making it maintainable?</p><p>If I had the opportunity to build a new data warehouse from scratch, I&apos;d probably</p>]]></description><link>https://arnmk.com/maintainable-sql-in-data-warehousing/</link><guid isPermaLink="false">5ea142c429788f0001ec277a</guid><category><![CDATA[sql]]></category><category><![CDATA[data]]></category><category><![CDATA[software]]></category><dc:creator><![CDATA[Aaron Mak]]></dc:creator><pubDate>Sat, 25 Apr 2020 03:26:46 GMT</pubDate><content:encoded><![CDATA[<p>Since SQL isn&apos;t a programming language, and say you don&apos;t have the option of using a programming language to generate SQL, how do you go about making it maintainable?</p><p>If I had the opportunity to build a new data warehouse from scratch, I&apos;d probably tell myself these things to prevent past mistakes I&apos;ve made along the way.</p><ul><li><a href="#strict-style-guide">Strict style guide</a></li><li><a href="#clear-naming-conventions">Clear naming conventions</a></li><li><a href="#write-tests">Write tests</a></li><li><a href="#use-table-views">Use table views</a></li><li><a href="#arrange-the-repository">Arrange the repository</a></li><li><a href="#use-version-control">Use version control</a></li><li><a href="#describe-your-sql">Describe your SQL</a></li></ul><h2 id="strict-style-guide">Strict style guide</h2><p>Enforcing a style guide early on is a no brainer. The earlier you do it, the better. If you&apos;re lucky, there&apos;s an open-source formatter that you can use. But you&apos;d still need to decide on some things that would help your SQL be more maintainable. To my dismay, there wasn&apos;t one for BigQuery SQL so the style guide had to be enforced manually.</p><p>Some things to consider could be the level of nested subqueries, or whether to have commas at the end, the number of spaces and many more. It sounds trivial but it&apos;ll improve maintenance if there&apos;s consistency in SQL styles. No one likes to point out a missing space or comma in code reviews. But without a formatter, these comments help in the long run.</p><p>A good example of what a style guide might look like can be found at <a href="https://www.sqlstyle.guide/">https://www.sqlstyle.guide/</a>.</p><h2 id="clear-naming-conventions">Clear naming conventions</h2><p>Part of a style guide could include naming conventions. Suffixes and prefixes could help describe columns better. For example, order_count has a count prefix. If standardised, it would provide additional information for the reader even before the reader explores the data.</p><p>How tables and schemas (or datasets in BigQuery) are named are equally important. Are tables grouped by end-users or products? Would you be using the fact and dim prefixes for fact and dimension tables? Would reporting tables be in a separate schema? In the end, you should have some rules and the reasons behind them.</p><p>A caveat is that conventions have to be followed or else it would have the opposite effect and make it more unmaintainable and more confusing for everyone.</p><h2 id="write-tests">Write tests</h2><h3 id="validity">Validity</h3><p>The SQL needs to be able to run, to begin with. You can start with a static parser and some data warehouses like BigQuery might even allow you to call their API to check if your SQL is valid. This would easily prevent major issues in your pipeline.</p><h3 id="data-integrity">Data Integrity</h3><p>Monitor the quality of the data is essential so errors can be caught as early as possible. Trust me, the last thing you&apos;d want is your stakeholders reporting data quality issues.</p><h4 id="expected-row-count">Expected row count</h4><p>This prevents bad joins for downstream tables. You won&apos;t have that issue with SQL databases because you&apos;ll likely define the relationships between objects using an <a href="https://en.wikipedia.org/wiki/Object-relational_mapping">object-relational mapping</a> (ORM) library. But when it comes to data warehouses, there aren&apos;t any restrictions. You could expect a one-to-many relationship, but the join revealed a many-to-many relationship. These issues are made worse when data is unclean or duplicates are hidden in distributed microservices.</p><h4 id="expected-columns">Expected columns</h4><p>Certain columns contain the primary key or the foreign keys which you&apos;d expect to be in the table. Or even for reporting tables, you might expect the date to be there. Ensuring such columns exist will prevent careless mistakes. A static parser would help with this or the output table can be monitored instead.</p><h4 id="uniqueness">Uniqueness</h4><p>Ensure columns which contain the primary key to be unique. It&apos;s common to have duplicate rows in streaming pipelines or wrong joins. </p><h3 id="recency">Recency</h3><p>Ensures the tables are up to date. It&apos;s more like a failsafe than anything. An external monitoring tool would be great for this.</p><h2 id="use-table-views">Use table views</h2><p>Views allow you to breakdown a large SQL into smaller chunks. Views can also be shared by downstream tables, reducing the SQL you need to write. Long SQL scripts are more likely more difficult to maintain.</p><h2 id="arrange-the-repository">Arrange the repository</h2><p>If you&apos;re using a repository to store the SQL, a structure is necessary so people know where to find their pipelines. Grouping them by their data pipelines or domain is a good start. It&apos;d be helpful as well if there&apos;s a document to show how the repository is organised.</p><p>For example, I might want to group the first layer transformations in one directory and the reporting transformations in another directory.</p><h2 id="use-version-control">Use version control</h2><p>I&apos;m going to go out on a limb and say that analysts need to use <a href="https://git-scm.com/">git</a>. If you could learn SQL, you can learn git. An <a href="https://en.wikipedia.org/wiki/Integrated_development_environment">IDE</a> would help if the terminal gives you nightmares, though I&apos;d still recommend the terminal. Version control not just helps you to figure out what happened historically but also why things are done a certain way if git is used well, which in my opinion is more important.</p><h2 id="describe-your-sql">Describe your SQL</h2><p>Most data warehouses allow you to add a description to your columns so use it! There are times when the logic for a column is complicated and cannot be described with just the column name, especially for reporting tables. But maintenance of the descriptions is key so the description is up to date with the logic of the column. Looking at SQL is no fun for users of your table that don&apos;t know SQL.</p>]]></content:encoded></item><item><title><![CDATA[Continuous Integration and Deployment for Data Pipelines]]></title><description><![CDATA[Although continuous integration and deployment (CI/CD) are not new concepts, new tools such as Docker, Kubernetes, and Jenkins have allowed CI/CD to be more easily implemented in recent years. However, there are interesting challenges when applying CI/CD to data engineering.]]></description><link>https://arnmk.com/continuous-integration-and-deployment-for-data-pipelines/</link><guid isPermaLink="false">5ead24ad515393000118ebc8</guid><category><![CDATA[devops]]></category><category><![CDATA[data]]></category><category><![CDATA[software]]></category><dc:creator><![CDATA[Aaron Mak]]></dc:creator><pubDate>Tue, 03 Dec 2019 07:44:00 GMT</pubDate><content:encoded><![CDATA[<blockquote>Continuous integration and continuous deployment (CI/CD) is a practice that enables an organization to rapidly iterate on software changes while maintaining stability, performance and security.</blockquote><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://cdn-images-1.medium.com/max/800/1*j0eOmGczi2QyfiNipHgZxg.png" class="kg-image" alt loading="lazy"><figcaption>CI/CD Pipelines interface on&#xA0;Gitlab</figcaption></figure><p>Although continuous integration and deployment (CI/CD) are not new concepts, new tools such as Docker, Kubernetes, and Jenkins have allowed CI/CD to be more easily implemented in recent years. However, there are interesting challenges when applying CI/CD to data engineering.</p><h3 id="challenges-with-ci-cd-in-data">Challenges with CI/CD in Data</h3><h4 id="replication-of-environments">Replication of environments</h4><figure class="kg-card kg-image-card kg-width-wide kg-card-hascaption"><img src="https://cdn-images-1.medium.com/max/1200/1*pBR1uQR9MN-Q0BaF7TkmwA.jpeg" class="kg-image" alt loading="lazy"><figcaption>Credit: CommitStrip.com</figcaption></figure><p>In modern software engineering workflows, there are usually <strong><em>development</em></strong>, <strong><em>staging</em></strong> and <strong><em>production</em></strong> environments. In software engineering, development and staging databases usually mimic a fraction of the production database. This assumes that the differences in data would not matter. In software engineering, this is usually the case.</p><p>In data engineering, we are unable to create a staging environment for some 3rd party data sources. Yet, our development and staging environments should try to mimic our production environment to ensure consistency in environments. But replicating data environments can be difficult, depending on the size of your data and existing infrastructure.</p><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://cdn-images-1.medium.com/max/800/1*AYbmscaJizdaCTCVy_LzzQ.png" class="kg-image" alt loading="lazy"><figcaption>A typical set of data pipelines in 90&#xA0;Seconds</figcaption></figure><h4 id="tests-take-too-long">Tests take too long</h4><p>Tests are designed to give quick feedback in CI/CD, ensuring as many iterations as possible so that bugs can be caught and fixed quickly. However, testing data pipelines might require testing queries that take more time than an average integration test.</p><h4 id="legacy">Legacy</h4><p>Existing data pipelines might have been in place before CI/CD was even considered. This makes things complicated because a lot of code would need to be redesigned to consider different environments. Other than technical legacy, it might also come in the form of organisational legacy where people are not used to having multiple environments. Educating different stakeholders and engineering teams to get used to a CI/CD workflow would not be easy. Besides, it would also require significant changes in processes.</p><h3 id="a-typical-data-architecture">A typical data architecture</h3><p>Instead of hypothetical examples, let&#x2019;s take a look at a simplified data architecture which follows the <a href="https://blog.panoply.io/etl-vs-elt-the-difference-is-in-the-how" rel="noopener">ELT principle</a>. For this example, we will be using Google Cloud Platform, which is similar to other cloud providers.</p><figure class="kg-card kg-image-card kg-width-wide"><img src="https://cdn-images-1.medium.com/max/1200/1*dXEPOXDU4PC2ch2qgDnQKQ.png" class="kg-image" alt loading="lazy"></figure><p>Looking at this architecture, we notice some challenges if we were to fully adopt the modern development workflow. For one, it&#x2019;d be impossible to create a BigQuery instance for each data engineer on their development environment since this is likely a local setup. And if you were to create one BigQuery instance on the cloud for each engineer, it will likely be too costly.</p><p>Say you wanted to test your pipeline. How would you go about doing an integration test? With any modern application, there&#x2019;s usually a library for that but how do you verify the data integrity in development and pinpoint the source of the error without a complete copy of the production environment?</p><p>If you had thought about these issues, you&#x2019;d be surprised that some companies had never thought of it. Some teams might not be experienced or have an engineering mindset. So they started building things from the get-go, without considering how best to control technical debt. Once enough things are built and the process is settled, it&#x2019;ll be difficult to make any significant changes.</p><p>Some of these issues convince some companies to simply go with one environment&#x200A;&#x2014;&#x200A;production. But there can be a compromise and I&#x2019;d argue that multiple environments encourage higher code quality and data integrity.</p><h3 id="a-better-workflow">A better workflow</h3><p>Since we are unable to strictly adopt a software development workflow, we will have to make some compromises.</p><p>Instead of having a dedicated BigQuery instance to each data engineer for the development environment, we will have a shared one. For this to work, all tables have to be easily torn down and rebuilt&#x200A;&#x2014;&#x200A;following a <a href="https://medium.com/@maximebeauchemin/functional-data-engineering-a-modern-paradigm-for-batch-data-processing-2327ec32c42a">functional approach to data engineering</a>. The BigQuery API integration with Airflow also helps us with that.</p><p>With every merge request, a series of automated tests will be run. BigQuery gives us some <a href="https://cloud.google.com/bigquery/docs/best-practices-costs#performing_a_dry_run" rel="noopener">cool API calls</a> to check the validity and cost of our queries which helps us with testing. Though not a strict unit test, it gives us quick feedback from automated testing so we could catch errors before they are pushed to production.</p><p>A shared BigQuery instance in development and test environments will inevitably result in unexpected errors, especially when the state of the shared BigQuery instance is dependent on the last executed Airflow DAG. But since we have followed a functional approach to data engineering, resetting the state of the data warehouse is quick and easy.</p><p>Applying CI/CD as best as we can gives us a nice workflow to test our pipelines and logic before pushing them to production.</p><figure class="kg-card kg-image-card kg-width-wide"><img src="https://cdn-images-1.medium.com/max/1200/1*Mi44U6YhEiWFOJOuwLpdAg.png" class="kg-image" alt loading="lazy"></figure><p>Though this is similar to our current setup at 90 Seconds, every company has a different data architecture and constraints. I&#x2019;d expect our setup to evolve as data needs change and better tools are being developed.</p><h3 id="data-engineers-need-to-think-like-software-engineers">Data Engineers need to think like Software Engineers</h3><p>In a lot of ways, data engineers are software engineers, though with different constraints. Concepts such as functional programming, containerization and <a href="https://en.wikipedia.org/wiki/Test-driven_development" rel="noopener">TDD</a> might not be applicable all the time but it&#x2019;s a good place to start. Let&#x2019;s start with first principles and maybe, just maybe, there&#x2019;ll be a new set of conventions created by the data engineers built upon the shoulders of software engineering.</p>]]></content:encoded></item></channel></rss>