Minimal Engineering

Setting up local Spark Cluster

EMR presents a convenient method for rapidly deploying a cluster. However, we frequently require an ability to explore locally stored data rapidly. This guide walktlhoughts setting up a minimal local spark cluster fit for stated purpose. The completed project can be found here. GitHub - varokas/spark-demoContribute to varokas/spark-demo

UV - Poetry Replacement

UV is an extremely fast dependencies and project manager. It has roughly equivalent in functionality with Poetry, but is much faster. Installing $ curl -LsSf https://astral.sh/uv/install.sh | sh New Project Uses uv init to initialize a project with pyproject.toml configuration file. Use uv add to add

Using Tensorflow and Keras on M1 Macs

Following the instruction here Tensorflow Plugin - Metal - Apple Developer [https://developer.apple.com/metal/tensorflow-plugin/] Python (Miniforge) First, we need to install miniforge. Easiest way to do that is to use pyenv [https://github.com/pyenv/pyenv]. The tool allow us to install multiple version of pythons. More

Living with M1 Mac

These are my current account of roadblocks and workaround in using the tools I frequent within Apple Silicon Mac. Mostly I'll try my best to make native works before falling back to using rosetta. I'll keep this page updated as things improves. I'll note

AWS Lambda Function from Docker Image

Since Dec 1, 2020, AWS Lambda allow developers to uses any docker images to be executed as lambda functions. This obviously bring tons of benefits and conveniences. New for AWS Lambda – Container Image Support | Amazon Web ServicesWith AWS Lambda, you upload your code and run it without thinking about servers.

AWS Lambda Functions in Scala

Initialize Scala Project Assuming we are familiar with SBT and have that installed $ sbt new scala/scala-seed.g8 A minimal Scala project. name [Scala Seed Project]: Lambda Scala Seed Template applied in /Users/vpanusuwan/projects/./lambda-scala-seed Add Lambda Library The libraries contains event classes which is a typed input for

Define AWS Infrastructure in Python with AWS Cloud Development Kit

With AWS CDK [https://aws.amazon.com/cdk/], we can define an infrastructure using familiar languages (Python, JS, Java, etc) . The code will be compiled into a cloud formation JSON and deployed as a cloud formation stack. Later we can diff the changes made before redeploy. Install brew install node

Algorithms: Sorting

High level review of classic sorting algorithms with Haskell. What do we mean by Sorted? For a sorted list, every element is smaller or equals to the next one. From this very definition we can check if a list is sorted by 1. Checking that the first element is less

Secrets in Code with Mozilla SOPS

Secrets presents a challenging dilemma for infrasture-as-a-code. Solution today converge mostly on storing these secrets in some external trusted system (Kube Secrets, Docker Secrets, Build System Secrest, Vault) outside of the code. Using SOPS, we can check in the encrypted secrets (e.g. connection passwords) along with the code. The

Self Hosted Ghost Blog setup with Docker

How to set up a self-hosted ghost blog with docker, mysql, TLS, and backup script.

Easy HTTPs with Caddy

Nginx and certbot too complicated? Use Caddy.

Consistent Python Build with pipenv

Managing dependencies in Python is kind of a mess. Let's try to make that better with pipenv

Using AWS as Terraform Backend

Terraform [https://www.terraform.io] is great at provisioning cloud infrastructure. Terraform needs to keep the current state somewhere. Local filesystem can be used in many cases, but not ideal if there are multiple people sharing this state file. In case of AWS, we usually wants to keep it in

JDK 14 Feature Overview

JDK 14 was officially GA released [https://openjdk.java.net/projects/jdk/14/] on 17 March 2020. These major releases are much more frequent in past years. However, I still find the release notes not an ideal source in providing a quick overview of what is new in the release.

Using Kubernetes Locally in MacOS

Install Docker brew cask install docker Install Kubernetes in Docker * Go to menu bar, select the whale icon -> preferences * Select Kubernetes tab, then "Enable Kubernetes" Check that docker is working ~ ❯❯❯ docker run hello-world Hello from Docker! This message shows that your installation appears to be working

GPU-accelerated Machine Learning on MacOS with Keras and PlaidML

There aren't a lot of GPU-accelerated Machine Learning Framework in MacOS besides CreateML [https://developer.apple.com/documentation/createml] or TuriCreate [https://github.com/apple/turicreate]. Luckily, we could use PlaidML as a backend for Keras [https://keras.io] as it implements Metal Performance Shaader. Install Keras Easiest

Manage Java Versions with JEnv

Experimenting with different versions of Java made easier by a handy tool. Install First, install JEnv itself. brew install jenv JEnv simply assist in managing Java versions. We have to install various Java versions by ourselves. brew tap homebrew/cask-versions brew cask install java11 Put these two lines in ~/.bash_

SystemD Quickstart

A quite handy framework built-in to most Linux distro. SystemD helps in setting up a process that restarts itself on boot, and when it crash. This is what many packages do when apt-installed. Doing it by hand is not complicated. But I couldn't find a single quickstart source

javascript

Generate QR Code for WiFi

Instead of having your guests hunting for WiFi name and type in your weird password, let them scan a QR code and connect with ease. Change wifi-name , wifi-password below and save the QR code. const wifi_name = "wifi-name" const wifi_password = "wifi-password" escape = (s) => s.

Interactive ClojureScript in a blog with Klipse

I've always been looking  for ways to put interactive code in a blog. For Javascript, there's CodePen. For ClojureScript, there's KLIPSE [https://github.com/viebel/klipse] First, try fiddling with the ClojureScript expression below and be in awe at the marvel of modern technology!

blog

From Medium to Ghost 2.0

Ghost has advanced so much this year. At this point, I highly recommend it to everybody writing a technical blog. It is time to stop fiddling with Medium and jump over.

Creating Single secure Airflow instance on Ubuntu 18.04

Airflow installation beyond the basics are pretty involved. This is my notes on how. This note assumes the reader wanted to access airflow…

Zeppelin — Notebook สำหรับApache Sparks

สำหรับคนที่ติดใจเขียนโปรแกรมแบบ notebook ผ่าน jupyter ตอนนี้มีโปรเจคที่ค่อนข้างใหม่ชื่อ Apache Zeppelin เหมาะมากสำหรับสาย data ที่ใช้…

สร้าง Kubernetes Cluster บน Google Cloud Platform

เนื่องจากเป็นโปรแกรมเมอร์งบจำกัด เพิ่งมีคนใจดีให้เครดิตมาลองเล่น GCP ก็เลยสร้าง Kubernetes Cluster ซะเลย ง่ายสุดก็เข้าไปกดคลิ๊กๆ…

Manage Digital Ocean with Terraform

สำหรับคนที่ชอบเปิดเอา Compute Node มาเล่นแต่ยังไม่ใช้ Kubernetes ถ้าใช้ Terraform เราก็จะสามารถ สร้างและเชื่อมกับ DNS ได้เลยเร็วๆ บน…