Iury's Thoughtbox Blog

From GitHub Actions to Cloud Build

Wed, 03 Jun 2026 12:00:00 GMT

Back in 2024, I wanted a project where I could try building an agent-like app without getting buried in infrastructure. I ended up building the AI Audio Guide app: point the camera at a landmark, and get back a short explanation with narrated audio.

In 2025 I also wanted a dedicated backend where I could explore RAG, image analysis, different models, tool calls, retrieval, and audio generation without stuffing all that product logic into the mobile client.

The first version worked. It had a mobile client, a backend API, and enough moving pieces that deployment stopped being a side quest and became part of the product.

Cloud Run felt like the right place for that backend. That view got stronger after I had the chance to join an AI Paris event and talk with people from Google about Cloud Run. I did more research afterwards, especially around agents, and the product started to make sense: a real container, HTTPS, IAM, logs, Secret Manager integration, and request-based scaling without running a platform team for an experiment.

I wanted containers, but I did not want to run servers or manage a Kubernetes cluster just to ship an experiment. I also did not want to squeeze the app into serverless functions. This was a backend with a few moving parts, model calls, retrieval, and audio generation. Cloud Run let the container stay the unit while Google Cloud handled the serving layer.

Chatting about cloud run with its founder: Steren Giannini

mobile client
  -> Cloud Run container
  -> Google Cloud APIs + RAG storage

Once I had a containerized app, deployment automation became the next problem. I was iterating fast, so I reached for the closest tool: GitHub Actions. The code was already on GitHub, I knew the workflow model, and wiring a release event to a build script was easy. It was a good first solution.

I also wanted the boring parts scripted. I was already familiar with the Twelve-Factor App ideas around config, build/release/run separation, and dev/prod parity. So I put most operations behind commands: start the app, run the vector store, build the container, push the image, bump the version. I wanted repeatable commands, not a checklist I had to reconstruct every time.

It worked, but there were maybe too many moving parts and components spread across different services.

Part of the system	Owner
Source code	GitHub
Release trigger	GitHub release
Release runner	GitHub Actions
Container images	Artifact Registry
Runtime	Cloud Run
IAM and logs	Google Cloud
Secrets	GitHub Secrets decoded into `.secrets/` during release

Secrets were not committed to the repository. The shortcut was to have GitHub Actions decode them during release, write them into .secrets/, and make those files available to the container.

That was okay for moving fast. But I did not want the build path to keep acting like a secret delivery system. Secrets should be runtime concerns. Images should be deployable artifacts. If rotating a key makes me think about image rebuilds, the boundary is wrong.

There was also the runner question. GitHub Actions works for my use case, but its runner queue has been stretched pretty thin lately. Since this service already ran on Google Cloud, I would prefer not tying production deploys to an external runner queue if Cloud Build could handle that part natively.

The old release path

The previous flow started from a GitHub release:

flowchart TB
  subgraph GitHub["GitHub"]
    direction TB
    Release([Release published])
    Actions[GitHub Actions runner]
    Secrets[(GitHub Secrets)]

    subgraph Runner["Inside the release job"]
      direction TB
      Checkout[Checkout source]
      SecretFiles[Write .secrets files]
      Build[Build Docker image]
      DeployScript[Run deploy scripts]
    end
  end

  subgraph GoogleCloud["Google Cloud"]
    direction TB
    Auth[Authenticate to Google Cloud]
    Registry[(Artifact Registry)]
    CloudRun[Cloud Run service]
  end

  Release --> Actions
  Actions --> Checkout
  Actions --> Auth
  Secrets -.->|decoded during workflow| SecretFiles
  Checkout --> SecretFiles --> Build --> Registry
  Registry --> DeployScript --> CloudRun

  style GitHub fill:#f6f8fa,stroke:#57606a,stroke-width:1.5px
  style Runner fill:#fff8e1,stroke:#f9ab00,stroke-width:1.5px
  style GoogleCloud fill:#e8f0fe,stroke:#1a73e8,stroke-width:1.5px

A simplified version looked like this:

name: release-main

on:
  release:
    types: [published]

jobs:
  release:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Write credentials
        run: |
          mkdir -p .secrets
          echo $SERVICE_ACCOUNT_JSON | base64 -d > .secrets/service_account.json
          echo $MAPS_API_KEY | base64 -d > .secrets/maps_api_key.txt

      - name: Authenticate Google Cloud
        uses: google-github-actions/auth@v2
        with:
          credentials_json: ${{ secrets.DEPLOYER_SERVICE_ACCOUNT }}

      - name: Build image
        run: uv run container --build --env prod --tag $TAG

      - name: Push image
        run: uv run container --push --env prod --tag $TAG

This was a valid first release path. The scripts made local and release behavior predictable, and the release event was explicit.

Still, the workflow carried too much Google Cloud context: secret files, GCP auth, Docker config, image build, registry push. All of it was headed back into Google Cloud anyway.

So earlier this year I started thinking: why not let Google Cloud do that part?

What is Cloud Build?

Cloud Build is Google Cloud’s managed build and release service. Instead of renting a generic runner somewhere else and then teaching it how to talk to Google Cloud, Cloud Build runs the release work inside the same platform that already owns the runtime, registry, logs, secrets, and IAM.

A build is defined as a sequence of containerized steps: build the image, push it to Artifact Registry, deploy a new Cloud Run revision, and record the whole story in Cloud Logging. For this project, that made Cloud Build less of a replacement for GitHub Actions and more of a natural release layer for a Google Cloud-native backend.

The Cloud Build release path

GitHub Actions and Cloud Build have different shapes.

GitHub Actions thinks in workflows, jobs, hosted runners, marketplace actions, and repository secrets. Cloud Build thinks in triggers, builds, container steps, service accounts, Artifact Registry, Secret Manager, and Cloud Logging.

If your application runs somewhere else, GitHub Actions may still be the better general-purpose tool. For this backend, Cloud Build made sense because the deployment target was already Cloud Run. The closer the release path got to Google Cloud, the simpler the system became.

%%{init: {
  "theme": "base",
  "themeVariables": {
    "fontFamily": "Inter, system-ui, sans-serif",
    "lineColor": "#5f6368",
    "primaryTextColor": "#202124"
  },
  "flowchart": {
    "htmlLabels": true,
    "curve": "basis"
  }
}}%%
flowchart LR

  subgraph NEW["Cloud Build owns build + deploy"]
    direction TB
    Tag([Git tag vX.Y.Z]) --> Trigger[Cloud Build trigger]
    Trigger --> BuildNew[Docker build]
    BuildNew --> PushNew[(Artifact Registry)]
    PushNew --> DeployNew[Deploy Cloud Run]
    SecretManager[[Secret Manager]] -.->|runtime mounts| DeployNew
  end

  classDef old fill:#e8f0fe,stroke:#1a73e8,stroke-width:1.5px,color:#174ea6
  classDef new fill:#e6f4ea,stroke:#34a853,stroke-width:1.5px,color:#137333
  classDef awkward fill:#fce8e6,stroke:#d93025,stroke-width:1.5px,color:#a50e0e
  classDef store fill:#fef7e0,stroke:#f9ab00,stroke-width:1.5px,color:#8a5c00
  classDef secret fill:#f3e8fd,stroke:#a142f4,stroke-width:1.5px,color:#681da8

  class Actions,BuildOld,DeployOld old
  class Trigger,BuildNew,DeployNew new
  class PushOld,PushNew store
  class SecretFiles awkward
  class SecretManager secret

  style NEW fill:#f7fff9,stroke:#34a853,stroke-width:2px

The source stayed on GitHub. The release runner moved into Google Cloud, where the image registry, Cloud Run service, IAM, logs, and runtime secrets already lived.

The core cloudbuild.yaml is not complicated:

substitutions:
  _REGION: europe-west3
  _REPOSITORY: backend-images
  _IMAGE_NAME: audio-guide-api
  _SERVICE_NAME: audio-guide-api
  _RUNTIME_SERVICE_ACCOUNT: cloud-run-runtime@PROJECT_ID.iam.gserviceaccount.com

steps:
  - name: gcr.io/cloud-builders/docker
    env:
      - DOCKER_BUILDKIT=1
    args:
      - build
      - -t
      - ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:${TAG_NAME}
      - -f
      - Dockerfile.api
      - .

  - name: gcr.io/cloud-builders/docker
    args:
      - push
      - ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:${TAG_NAME}

  - name: gcr.io/google.com/cloudsdktool/cloud-sdk
    entrypoint: gcloud
    args:
      - run
      - deploy
      - ${_SERVICE_NAME}
      - --image
      - ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:${TAG_NAME}
      - --region
      - ${_REGION}
      - --platform
      - managed
      - --service-account
      - ${_RUNTIME_SERVICE_ACCOUNT}
      - --port
      - '8080'
      - --allow-unauthenticated

images:
  - ${_REGION}-docker.pkg.dev/${PROJECT_ID}/${_REPOSITORY}/${_IMAGE_NAME}:${TAG_NAME}

options:
  logging: CLOUD_LOGGING_ONLY

The tag supplies TAG_NAME. Cloud Build builds the image, pushes it, and deploys Cloud Run. Build logs live in Cloud Build. Runtime logs live in Cloud Logging. The deployed image lives in Artifact Registry. The IAM model is Google IAM instead of a GitHub secret that impersonates its way into Google Cloud.

At that point, the release path matched the runtime instead of pretending GitHub was the center of a Google Cloud deployment.

Improving the secret situation

One useful side effect of moving to Cloud Build was cleaning up how secrets reached the app.

The backend needed several of them: a Maps API key, a Firebase service account, a Google Cloud service account, and another API key for model-related tooling. None of these were committed to the repository, but the old release path still made GitHub Actions responsible for turning GitHub Secrets into files under .secrets/.

The official security issue name is: Unsecured Credentials. In plain English: exposing keys in places they do not need to be. In a normal backend, that is already bad, but in an agent-like backend, it gets much worse. If a tool path can be nudged into reading files, every readable credential becomes another key on the ring.

This does not make runtime secrets magically harmless. If the app process can read a mounted file, compromised code can read it too. But at least now CI is out of the secret delivery chain and keeps the image build away from runtime credentials.

Local development still works like this:

uv run secrets --pull
uv run start --dev-server

Docker Compose mounts the same directory for local container runs:

services:
  api:
    volumes:
      - ./.secrets:/code/.secrets:ro

Cloud Run gets the files from Secret Manager:

- --set-secrets
- /code/.secrets/gcp/service_account.json=app-service-account:latest,
  /code/.secrets/maps/maps_api_key.txt=maps-api-key:latest

The Docker image doesn’t carry secret files anymore. Production secrets live in Secret Manager and local development keeps the same app-facing shape.

Build identity and runtime identity are different jobs

The migration also cleaned up identity.

Instead of giving a default service account broad powers, we created a dedicated build/deploy account:

cloud-build-deployer@PROJECT_ID.iam.gserviceaccount.com

That account can run builds, push images, deploy Cloud Run, write logs, and act as the runtime service account.

The Cloud Run service uses a separate runtime account:

cloud-run-runtime@PROJECT_ID.iam.gserviceaccount.com

That account reads only the runtime secrets it needs.

A small release rehearsal

I kept one pre-production check: a separate Cloud Run service wired to smoke tags. A smoke-v... tag deploys the same build shape to the rehearsal service; a vX.Y.Z tag deploys production.

After each deploy, I run one paid end-to-end request with a real Firebase token and a fixed Brandenburg Gate fixture:

AUDIOGUIDE_AUTH_TOKEN="<firebase-id-token>" uv run smoke --target smoke

That is enough for the release flow. The smoke test is not the main story; it is just the guardrail that lets me test the path before touching production.

What broke

A few things failed in useful ways.

Cloud Build could not create the trigger until the GitHub repository was connected through the Cloud Build GitHub App. That step is obvious after you know it. Before that, the CLI error just looks like another setup problem.

The first Docker build failed because the Dockerfile used BuildKit cache mounts:

RUN --mount=type=cache ...

BuildKit is Docker’s newer build engine. It enables features like cache mounts, better layer reuse, and more efficient builds. My Dockerfile already relied on it locally, but the Cloud Build Docker step did not enable it by default.

The fix was one line:

env:
  - DOCKER_BUILDKIT=1

The first Cloud Run deploy failed because of the Secret Manager mount layout. Different secrets needed isolated paths.

The runtime environment also needed to be explicit. The image had defaults, but production should not depend on whatever happens to be inside a tracked .env file. Cloud Build now sets the Cloud Run runtime environment during deploy.

None of these issues were dramatic. They were normal migration friction, and having a rehearsal service meant I could find them without poking production.

The result

The final shape is less about individual steps and more about ownership:

flowchart TB
  Repo[GitHub repo] --> Tag[Git tag]
  Tag --> Trigger[Cloud Build trigger]

  subgraph BuildDeploy[Cloud Build]
    direction TB
    Build[Build container image]
    Push[Push image]
    Deploy[Deploy Cloud Run revision]
    Build --> Push --> Deploy
  end

  subgraph GoogleCloud[Google Cloud]
    direction TB
    Registry[(Artifact Registry)]
    Run[Cloud Run service]
    Secrets[(Secret Manager)]
    Logs[(Cloud Logging)]
    IAM[IAM]
  end

  Trigger --> Build
  Push --> Registry
  Registry --> Deploy
  Deploy --> Run
  Secrets -.->|runtime mounts| Run
  IAM -.-> BuildDeploy
  IAM -.-> Run
  Run --> Logs

  classDef source fill:#f6f8fa,stroke:#57606a,stroke-width:1.5px,color:#24292f
  classDef build fill:#e8f0fe,stroke:#1a73e8,stroke-width:1.5px,color:#174ea6
  classDef runtime fill:#e6f4ea,stroke:#34a853,stroke-width:1.5px,color:#137333
  classDef secret fill:#f3e8fd,stroke:#a142f4,stroke-width:1.5px,color:#681da8
  classDef store fill:#fef7e0,stroke:#f9ab00,stroke-width:1.5px,color:#8a5c00

  class Repo,Tag source
  class Trigger,Build,Push,Deploy build
  class Run,Logs,IAM runtime
  class Secrets secret
  class Registry store

GitHub still owns the source. Cloud Build owns build and deploy. Artifact Registry owns the image. Cloud Run owns the runtime. Secret Manager owns the secrets. IAM owns the permissions.

For a small backend, this is the shape I wanted: no platform-engineering ceremony, no maze of console clicks, and no release runner pretending to be the cloud it deploys into.

The takeaway

A picture is worth a thousand words.

Having this centralization gives a lot of peace of mind. When something fails, I do not have to jump between a bunch of different services. So, I can actually vouch for Google Cloud Build. It’s a stable, straightforward solution that made implementing this workflow much simpler, and I can definitely recommend it, especially if you’re already using something like Cloud Run.

My 26-Q1 Agentic Coding Workflow

Thu, 29 Jan 2026 12:00:00 GMT

Cover: The gang and I building an overengineered CLI.

I’ve been doing more and more agentic-coding for the past year. And some patterns keep showing up. Not rules, just things that seem to work for me across the board.

Closing the Loop

The agent needs to verify what it built. Not just write code and hope. But actually check if it works. Like a self healing mechanism.

For an app, that could mean:

Build the code
Lint/format
Run tests
Run the app
Inspect the UI
Debug failures
Check the logs

You need tooling that lets the agent execute and observe. Sometimes that means writing a small script, sometimes it’s just about adding proper instructions to its context. Sometimes it’s a browser automation tool, and sometimes you build a custom skill, like I did with android-use.

Closing the agent loop is something I’m really leaning into now. Basically, an agent needs to be able to: Given a goal, it will derive a plan. Now, it needs to get the world state, then reason on what action to take to achieve that goal, make an action, and VERIFY the state again. Then loop.

Without being able to verify what happened it can easily get lost. It’s not its “fault” if you think about it, you gotta help it out.

We’ve briefly talked about that on our Fragmented podcast last week, skip to 09:03.

Multitasking

Agents take time. While one’s working, you may jump to something else. Maybe a different part of the codebase. Maybe a different project.

I try to keep my parallel work under three different things. Backend change here, frontend tweak there, docs update somewhere else. When your system is modular enough, you can hop between them without much hassle.

It’s oddly addictive. There’s always something you could be doing, but BEWARE. This can fry your brain faster than manual coding+multitasking would.

You Don’t Have to Read All the Code

Well, it depends on what you’re doing. Please don’t do it on your banking app or in any critical system. I’d like my planes flown without vibe coding for now, thank you.

But ON SOME INSTANCES, doing that is fine. Think of your home automation scripts, or some fun side project for tracking your run stats.

The thing is, if you structure things right, you really don’t have to read it all. Focus on the boundaries. Make sure data formats and validation between modules are sound, then test and trust the interior.

Like a staff eng would trust his/hers peers to implement something, you might eventually build enough trust that if your agents have built and tested like you asked them to, things will work. Also, there’s absolutely no reason to avoid writing tests anymore. Use that.

This focus on bondaries idea maps to functional core, imperative shell. Which to me always sounded like a nimbler version of hexagonal arch. Separate calculation from action:

Functional Core: Pure functions, immutable data. Takes input, returns output. No side effects. No touching the outside world.

Imperative Shell: Thin layer that does the I/O. Reads user input, calls the core for logic, writes to database, updates UI.

Side effects live in the shell. Logic lives in the core. Testing becomes straightforward. So does working with agents (and humans).

Semantic Maps

I have a folder I call .ai-artifacts. It sits at the root and at relevant units throughout the project.

A unit is whatever has enough meaning to you. Could be the project root, a package, a module, or a feature folder. Like unit tests, you define what this boundary means. I could probably write a full article on the things I like to keep in here.

Inside each .ai-artifacts folder, the main file is the semantic map. It describes what’s in that unit. Everything you’d want an agent to know before touching this code. In my case, they can look like this.

It’s independent from the global AGENTS.md/CLAUDE.md, so I control when they load. When I want deeper context about src/auth/, I run /load-semantic-map to inject that folder’s map plus parent folders up to root.

I also have an agent to create or update a semantic-map. It uses git hash to check if anything changed. Generates new maps or updates existing ones. They’re opt-in. I decide when the context is worth the tokens.

OpenCode is building something similar:

This is pretty cool to see!

I was manually doing by combining claude rules and a few commands to generate a "semantic mind map" for each "unit" of the project.

It worked pretty well. Having it baked into the harness is awesome. https://t.co/mWwCddfxq8
— iury souza (@IurySza) January 28, 2026

They also just added a /learn command to capture session insights back into scoped context files. Which is similar to what I do when I update the semantic-map.

We might be on to something.

Specialized Agents

I’m playing with different agent types for specific areas of work. I don’t use all of them all the time, but these are the categories where I’ve found it useful to spawn a dedicated agent:

Coder: Main agent. Writes most code, very basic skills/tools, and perhaps an MCP for connecting to the IDE of choice.
Planner: Read-only mode. Explores codebase, designs approach before implementation.
Reviewer: Fresh eyes after changes are done. This guy is a PITA.
Tester: Writes tests for what Coder built. Picky. Looks for edge cases.
Researcher: Connected to a few MCPs to find info online. Meant for scouring the web, also has a browser-tool.
Scribe: Maintains architecture docs and semantic maps.

Instead of having every skill and MCP globally available, each agent gets equipped with just the subset it needs. Minimal tooling per agent. Focused scope, fewer ways to go wrong. I manually spawn these. No automatic orchestration, I decide when I need a different perspective.

The agentic trap

Don’t overdo it. You don’t need an army of subagents. If you’re not at the frontier of this, I recommend avoiding getting sucked into this rabbit hole.

Im leaning towards this opinion for some time.
Occam's razor, man... https://t.co/4kqObYCTFk pic.twitter.com/yQrj9XB6wI
— iury souza (@IurySza) February 2, 2026

Workflow High level View

Start on plan-mode: describe what you want in plain English using a lot of context and as much detail as you can.
- protips:
  - Build your own.
  - Use dictation tools for this talking to the model here. There are tons of options, I’ve settled with VoiceInk some time ago.
Iterate: Iterate on that plan as much as needed. Get an output file. Open it. Read through.
/tracker - Use a command/skill/prompt to break down the work into phases.
Code: Code it however you prefer. Serial or parallel (don’t overdo it), human in the loop or not.
- You can have one instance coding, another reviewing, another writing tests. Or do one thing at a time.
- protip: You don’t need the SOTA model to do all of the work (unless someone else is paying for it.). Find a balance of what tasks can be offloaded to faster/cheaper agents. I’m in a kimi2.5-coding + GPT-5.2-Codex with opencode phase now (jan 26).
Manage the context: Know when to clear the context and start over. above 100k context is iffy territory.

The other day I found out there’s a framework that kinda emcompasses a lot of this. They call it spec-kit. I’m not a fan of such frameworks yet because I don’t like losing control of how this works, but I like it when they show that my experience is somewhat aligned with them.

Confirmation bias, am I right?

In any case, the concept is very simple and powerful: You just gotta build good enough context (md files) for the agent to keep working.

Spec-Kit workflow

Hope this is useful!

PS: This will likely change before next month :D

The Craftsman's Case for AI

Tue, 20 Jan 2026 12:00:00 GMT

“Craftsmanship starts with the tools. Your setup shapes the work as much as your skill.”

Software is no different.

I stumbled onto the idea of software craftsmanship very early on in my career. Probably something I picked from the Agile dinosaurs back when I started coding. Whatever you think of the figures behind it, the concept stuck with me. A clear north star: get good at your craft. Treat code like a trade worth mastering. That felt compelling then. It still does.

The Toolbox

One piece of this puzzle was getting good at the tools you use. Historically, this meant mastering the Unix philosophy. It meant preferring the terminal and the CLIs, for direct manipulation of the file system, the actual state of the computer, rather than a proxied UI layer. It meant knowing IDE shortcuts and some obscure settings.

Also, craftsmanship involves ownership. You shouldn’t rely just on the tools your OS or your company provides. You need a personal toolbox which can travel with you from job to job and computer to computer. It’s the accumulation of your specific workflow optimizations over years.

Opportunity Cost

Until recently, there was a hard limit to this toolbox. We collected aliases and configurations, but we rarely built full-fledged software to solve our own minor inconveniences. The opportunity cost was simply too high.

I’ve already written about how AI coding unlocks us for building tools that can automate tedious workflows at our workplaces, improving internal tooling and what not. But another point where I think there’s massive leverage is in improving our personal workflows.

You can script complex automations for your specific development environment that maybe no one else needs, but which make you significantly more effective. Because ideally, software should fit you like a craftsman’s tool fits their hands. Now that code is relatively cheaper, we can finally do more of that.

Compounding

I’ve been curating my dotfiles for many years now. Aliases, neovim setup, zsh tweaks, tmux keybindings and much much more. Everything looks and feels very familiar to me. You spend enough time somewhere, you make it yours.

But that used to evolve really slowly. It would take months for me to even attempt to adopt some tool or to try and customize a workflow. Now, it takes me minutes. I can spin up an agent and set things up in no time. Undo, change, integrate with other tools, anything is possible. And it’s fast.

Besides, I can now create small tools or scripts, and build experiments like Shadow Claude.

Another thing I built recently was a window/workspace manager for macOS, which I was never happy with (having tried many alternatives over the years). I spent an afternoon with an agent and built exactly what I needed: iurysza/ultra. Just portable, simple, mine.

These are not that impressive, I know. But these things compound nicely. Over time what you get is a system that’s familiar, which can massively augment your capacity to investigate a bug, to analyze some state, to have a clear grip of what’s going on.

To seamlessly jump between different parts of a system in a smooth and natural way. Tasks that require multiple steps can be reduced to a single command. This composition is the real unlock.

Nerd Sniping

There is one major caveat. With the barrier to entry lowered, the risk of nerd sniping yourself increases.

xkcd356: Nerd Sniping

Just because you can build a tool for everything doesn’t mean you should. It’s now even easier to fall into the trap of spending hours building things just because.

Conclusion

You are no longer limited to the tools you can buy or the scripts you have time to painstakingly write. If you treat your development environment as a product you own, AI allows you to forge exactly the tools you need to be a master of your craft.

That’s honestly one of the most compelling use cases of AI-coding to me. The meta layer. Improving your ability to do your work. This also helps to approach another major point now: developer grief in the face of these changes. But that’s the topic for another article.

Non-Deterministic Interfaces or How AI is Coming for the Frontend

Mon, 29 Dec 2025 22:51:00 GMT

xkcd: Can’t wait to find the intersection of debugging and philosophy

Last month the Flutter team dropped a GenUI SDK. The idea is you register a widget library, give an LLM the component definitions and let it decide what to render and with what data.

At its core, the GenUI SDK for Flutter is an orchestration layer. This suite of packages coordinates the flow of information between your user, your Flutter widgets, and an AI agent, transforming text-based conversations into rich, interactive experiences.

And it’s not about Flutter. Vercel’s AI SDK first announced GenUI support back in March 2024. Google is experimenting with that on search and the Gemini App. They recently published this interesting paper on the topic.

Server-driven UI, but the server is now a language model. I don’t think this is particularly revolutionary as a concept, but IMO they’re signals.

Something’s cooking.

The chat interface ceiling

We’ve been interacting with LLMs through chat windows for a few years now. Text in, text out. Then we got multimodal input/output, but still very limiting. Markdown became the de facto UX improvement, and if you’re lucky you get a Mermaid diagram integration here and there.

It works. But it’s also obviously limiting. The LLMs already are good at generating structured information (structured output). Why are we constraining the output to a text box?

The logical next step is letting the model decide not just the content, but the presentation. Which components to render. How to lay them out. Based on the query, the context, whatever signals are available.

This isn’t a prediction. It’s already happening in various forms.

Take Claude Code’s AskUserQuestion tool. A criminally underrated feature (which every other agentic coding tool was quick to copy). It’s so simple and illustrates this idea well:

Instead of a lot of back and forth with the agent, I ask it to generate a multi-step wizard UI on the fly. Here’s a snippet of how I use it with a simple gemini-image-gen skill I built.

> [!IMPORTANT] YOU MUST USE AskUserQuestion tool for gathering user feedback before starting generation.
>
> - The questions should attempt to cover the parameter space for the given task. Identify the key dimensions, constraints, and optimization targets that a specialist would consider when creating such an image.
> - After gathering the feedback from the user you must launch another AskUserQuestion round, but this time offering suggestions that should attempt to offer enough variability to the output, but still adhering to the confirmed constraints. One of the suggestions should be a bit out of the box.

What you get is this:

Claude Code’s AskUserQuestion tool generating a wizard UI for image creation

The AI reads the context (this blog post), then dynamically generates relevant style options, each with descriptions. A multi-step flow with Style → Palette → Concept → Submit.

No one coded this specific wizard. The structure emerged from the prompt and context. This is the whole idea in a nutshell.

What this actually means

Here’s the part that’s been rattling around in my head.

We’ve spent decades building (or attempting to build) deterministic systems. Same input, same output. That’s the contract. That’s what users expect. That’s what testing assumes.

Generative UI breaks that contract. Ask the same question twice, and you potentially get different layouts. Different component choices. Different emphasis. The system is working correctly. It’s just that “correctly” now includes probability.

And I think that’s… fine? For certain problems? Maybe?

LLMs work best when there’s no single correct answer. When the space is ambiguous. So that may work for a set of use cases.

The interesting bit

The implication is that we’ll have products where the interface itself is non-deterministic. Probabilistic by design.

That raises questions I don’t have answers to:

How do you test this? Are we going to start benchmarking user satisfaction?
What’s the fallback when the generated UI is wrong or confusing?

I suspect the patterns will involve things like user correction loops, graceful degradation to deterministic components, and some form of transparency about uncertainty. But the industry hasn’t figured this out yet. We’re in the “interesting problem” phase.

Not everything, obviously

Most apps don’t need this. CRUD with well-defined flows doesn’t benefit from probabilistic UI decisions. If you know exactly what the user needs to see, just show it.

But there’s a class of problems where this might make sense. Where the “right” UI genuinely depends on runtime context that’s hard to enumerate in advance.

The point isn’t that generative UI is the future of all frontends. The point is that it seems to have found a legitimate niche. And that niche implies a shift in how we think about building certain kinds of products.

The stack is splitting

There’s an interesting framing that’s been floating around that talks about how some software could be decoupled into three layers.

At the bottom, you have the system of record, your data models, APIs, etc. The durable stuff that doesn’t change much. Above that you’d have an agentic layer that would abstract the application logic and at the very end you would have a generative UI layer.

Pixels generated at runtime for a specific user and goal. A one-off chart. A temporary decision panel, etc.

The interesting implication: if the interface layer becomes ephemeral, in that scenario you would stop building screens and start curating component libraries that agents can assemble on demand. Weird.

Weird times

We’re quietly moving toward a world where some products are non-deterministic by design. And that is really weird.

Memory Portability: Owning what matters

Mon, 22 Dec 2025 10:00:00 GMT

What if your AI assistant could remember everything you’ve told it? Across sessions, across tools, across providers. Not locked into one company’s servers. Portable. Yours.

Why This Matters

I’m not sure who’s winning the LLM wars. But I know this:

If I’d been feeding ChatGPT my data for the past two or three years, you probably couldn’t pay me to switch now.

The more it knows you, the more indispensable it gets. The friction to change providers becomes enormous.

Sure, people stay with providers for other reasons: model quality, integrations, habit, pricing. But those factors shift. A better model launches. Prices drop. APIs get abstracted away. Memory is different. It compounds over time. The longer you use a system, the more context it accumulates, and the harder it becomes to replicate that elsewhere. That’s what makes it the stickiest form of lock-in. Almost like a natural monopoly.

The Manual Approach

I’ve been trying to maintain multiple system prompts: one for health data, one for financial context, some for general coding preferences. I share them across multiple providers/clients: Gemini, Claude, OpenAI, Msty Studio, Raycast, AI Studio, etc. Each context file gets tweaked over time as I learn what works and the data evolves.

It’s obviously tedious, but provides enough benefit that I keep doing it. Still, every time I switch tools or start fresh, I need to surface that context somehow. Each tool has a way to inject relevant context, and sometimes you just have to paste it. There’s gotta be a better way.

I remember reading about memory layer tools like Mem0 back in 2024. At the time it felt like too much work, maybe targeting devs building agents.

So I started thinking: what would a basic memory system look like? Something I could prototype using the hook system from my Shadow Claude project?

The Memory Model Basics

I sketched out what a minimal memory system would need. Two core flows:

Memory Storage

%%caption: Hooks to capture memories and store as vectors
flowchart TB
    subgraph Capture["Capture"]
        S[Session Ends] --> H[Hook]
        H --> LLM[LLM Summarizer]
        LLM --> F[Structured Facts]
    end

    F --> |Append| TXT

    subgraph Storage["Storage"]
        TXT[(Memory Store)] --> V[Vectorize]
        V --> VDB[(Vector DB)]
    end

This summarizes what happened in a session: user preferences, decisions, facts. Vectorize for semantic search.

Memory Retrieval

%%caption: Sub-agent decomposes requests into multiple memory queries
flowchart TB
    Q[User] -->|query| A[Main Agent]

    subgraph SubAgent["Recall Sub-Agent"]
        SA[Decompose Request]
        SA --> Q1[Query 1]
        SA --> Q2[Query 2]
        SA --> QN[Query N]
    end

    A --> |delegates|SA
    Q1 & Q2 & QN --> VDB[(Vector DB)]
    VDB --> AGG[LLM Aggregates Results]
    AGG --> A
    A -->|response| Q[User]

A sub-agent could decompose requests and trigger multiple retrieval attempts from different angles. The depth of retrieval scales with memory size. More memories mean more creative search strategies. I know I could use GraphRAG or something similar, but that would perhaps defeat the simplicity goal.

Two Use Cases

Explicit Recall: “What’s my marathon time PB?” Direct query, standard RAG.

Implicit Augmentation: You ask “Who’s playing tonight?” and the agent knows you support a specific team because you mentioned it months ago. Or you ask for a training plan and it factors in your VO2 max, injury history, and recent race times without you having to re-explain your fitness level. No explicit query. The context just shows up when relevant.

MCP Memory Service

Turns out someone already built this. MCP Memory Service is an open-source tool that implements this pattern and more.

flowchart LR
    subgraph acq [Acquisition]
        A1[Session-End Hook]
        A2[Skill Call]
    end

    subgraph store [Storage]
        S1[Vector DB + Embeddings]
    end

    subgraph cons [Consolidation]
        C1[Dream Cycle]
    end

    subgraph ret [Retrieval]
        R1[Session-Start Hook]
        R2[Mid-Conversation]
        R3[Skill Call]
    end

    A1 --> S1
    A2 --> S1
    S1 --> C1
    C1 --> S1
    R1 --> S1
    R2 --> S1
    R3 --> S1

What I liked:

Hook-based capture: Enables passively capturing memory.
No LLMs for extraction: The hooks are extractive, not generative. It relies on pattern matching to identify what is worth saving, so it runs often and fast.
Dream-inspired consolidation: Async process that decays old memories, clusters related ones, compresses redundant information
Session-start injection: Surfaces relevant context when you start a new session
Local-first: SQLite-vec backend with super fast (5ms) reads
You own the data: Everything stays on your machine unless you choose to sync

The “dream cycle” is a nice touch. Instead of memories accumulating forever, the system periodically consolidates them. Like how sleep helps humans consolidate learning.

What I got working

I set up MCP Memory Service with Claude Code and Msty Studio. The dashboard lets me browse memories by tag (auto-generated).

MCP Memory Service dashboard showing memories organized by tags

The real test: querying from a different client. Here’s Msty Studio (local chat client) consuming the same memory:

Msty Studio querying memory service

Same memory, different clients. That’s the portability I was after.

Hosted Alternatives

Two other projects worth mentioning:

Letta: Developer platform for stateful AI agents with persistent memory. Their API lets you build agents that remember and improve over time. Works with OpenAI, Anthropic, Gemini. Hosted solution with memory built into the agent layer.

MemSync: Portable memory platform focused on cross-application context. Handles semantic and episodic memories, smart categorization, can import data from ChatGPT, social media, documents. This has probably the biggest feature set.

Both are integrated solutions where you use their infrastructure. MCP Memory Service is more DIY: hooks, local storage, your own vector DB. Different tradeoffs.

Conclusion

Memory is really important for these systems. And perhaps we’ll go in a very different direction in the near future (as some papers already hint), but I really don’t like the idea of having something as powerful as an LLM owning all that information about me.

Social media companies already know a lot about us, but my interaction with Meta or X is usually defined by the content (and ads) they serve. With LLMs, it is literally a chat box. The information the model owns could potentially steer interactions in ways that would feel very uncomfortable without explicit control.

That’s why I think it’s so important, now more than ever, that we own the data. That’s real LLM portability. The memory stays with you, not some company’s servers.

As models become commodities, this might become a big deal.

When Knowing How to Code Is Not Enough

Sun, 14 Dec 2025 12:55:00 GMT

Disney’s 1940s cautionary tale on how NOT to instruct an agent.

For a lot of people AI assisted coding is basically fancy autocomplete. Understandable. Copilot was some point the biggest AI product ever. Most devs had their first exposure to AI-assisted coding with that. And we all know that’s not that incredible as a product. You’re saving a lot of keystrokes, that’s for sure. But it doesn’t fundamentally move the needle.

Then, at some point circa 2024 people started talking about coding agents and maybe you tried it again. “Ok… that’s cute, I guess”. But you could still argue that it removes the “craft” from coding, or that it could stil not handle the complexity of a bigger codebase. That’s also a fair take.

Nonetheless, I’m here to tell you that if you think agentic coding isn’t happening, it is likely because you have not seen what is currently possible or you lack a bit of imagination.

The engineer’s actual job

We need to be honest about why we are on the payroll. You are not hired type code. You’re hired to contribute to the bottom line using technical skills. Software is simply a means to an end.

I mean, I know that can sound awfully dark and sad from an IC POV. I have to say that I got into this because I love coding. I like looking at it. I’ve spent countless hours writing the same thing in different ways and deciding what feels best. What looks more elegant as a solution. The good old 100% pure, old-fashioned, ~~homegrown human~~ handcrafted code.

But I also have two eyes. And I’d wager that if you’ve given cursor or claude-code (these are the most popular) an honest try by now (late 2025), you’re convinced they’re good enough for a LOT of things. And that set of things is not decreasing.

Gatekeeping, Changes and Pragmatism

In my first semester at Uni I learned basic C. Mallocs, pointer arithmetic, manual memory management. Next semester: Java and Python. Oh, you don’t need to do that anymore?

But then, the online forums were full of people saying that garbage collection created freezes and lead to memory leaks and that software was getting slow and bloated. Real programmers managed their own memory. But we know how that played out. Eventually, CPU cycles became cheaper than developer hours and soon enough we had enough CPU to spare.

I’m not here to tell you we’re losing our jobs or that hand written code is going away, but we need to be aware of what’s up. If you can contribute to the bototm line without manually writing every line, you’re still doing your job. In fact, in some cases you might be doing it better.

The bottom line is that this shift opens up massive opportunities. Look at some of the internal tools at your company. They are probably terrible. You know that. Think about the friction caused by bad internal software or the recurrent tasks that eat up your week.

Previously, you might not have justified the time to build a tool to fix these flows. With agents, the cost of writing software for these “small” problems drops to near zero. You can improve workflows and automate the mundane work that gets in the way of the interesting engineering problems. And in some cases it will let start coding in a higher abstraction level.

There are places where you don’t want that and I could make an argument for where that’s the case. This is not the point of this article though, so I won’t go there.

Context engineering

I know a lot of good programmers with years of experience who are still unsure how LLMs can actually build useful code.

The thing is that being a great coder does not automatically make you good at agentic coding.

Getting results from an agent requires a different set of muscles. Some people call this context engineering. It is a mix of technical understanding and being able to codify that into the LLMs workflow. You have to take the mental model inside your brain when approaching a specific project and put it onto ~~paper~~ markdown effectively (and timely). And for that you need to be aware of the constraints and of some of its internals.

You cannot just let an agent loose on a repo and hope for the best. You need to build guardrails. Ideally, you have to instrument the agent for your specific codebase and your specific workflows.

Tools like Claude Code work well because they have heavy instrumentation built on top of it. It’s more than just the model. So to get the most out of an LLM in a codebase, you often need to build your own harness on top of that foundation. You need to customize how it interacts with your environment to mitigate damage and ensure it behaves as you expect it. Things like commands, skills, rules, subagents and concious context window management make a huge difference in the quality of results.

Claude Code orchestration and context flow diagram

Side Note: I’ve put this together after dumping some of my notes into gemini-3-pro-image, aka Nano Banana and playing around with prompts. I find it amazing how much info it can compress in a single image like that.

The problem is that this requires actual effort. It’s not free. Coding, as you may know, is something very personal. You pick 10 devs and look at how they code and you’ll likely find 10 different workflows. And it’s the same thing with agentic coding. You’ll likely have to build your workflow preferences into the agent before it you start trusting it, and that will require a LOT of trial and error.

Getting on the train

We are still in the phase where we are figuring this out as we go. There are no standard textbooks on context engineering yet. The industry is still settling on the patterns and it seems every other week someone comes up with a different .md file you have to manage.

But I recommend you gradually start incorporating the new agentic features I mentioned before as you learn how to use the agents. Build something and see how far you get. Give it a go.

Because, to me, the direction is clear. The ability to instrument agents and build high-quality context is just as important, in this new way of coding, as knowing the syntax of the language itself.

Where to go from here?

I find that you can learn a lot by understanding the mental model of the people that actually build the agent harness, then you can distill some of those ideas into your workflow. Some of interesting links:

Conclusion

If you approach this topic like a curious engineer approaches any other tech I think you may find out that there’s a LOT of engineering involved in getting the best out of it. This non deterministic coding is definetly challenging and has many quirks, but can also be incredibly fun to do when you looked at from the right angle.

I’ll share some of personal workflows in another article. Cheers!

Cover image: “Spellbound” (Sorcerer’s Apprentice / Fantasia) by Denyse Klette

AI Coding Is Just Coding Now

Tue, 09 Dec 2025 12:00:00 GMT

I’ve been avoiding writing about coding with LLMs. There’s just so much noise in the space that it’s easy to sound like you’re an AI maximalist (whatever that means). But I think that a few months ago we crossed a line which closes this debate.

Things are really polarizing these days and I know some people who have put themselves on the AI=BAD corner since the start. And that makes me want to write this.

We don’t say that we’re researching online for how to fix a bug. It’s just implied that you’ll use the internet for that. AI coding is heading the same direction. Having code written by AI in some capacity will be the defacto way of doing it.

Slowly and then suddenly

I’ve been experimenting with these tools since the faraway 2021 when you’d write a comment describing what you wanted and hope the model would fill in the function body. But in the last six months, the majority of my coding has been done with LLMs. Don’t get me wrong. I still manually write a lot of it. The thing is that when I count personal projects + code at work, my total output has increased by a LOT and that’s mostly due to AI.

The biggest jump for me was Claude Code, about six months ago. I try to keep my finger on the pulse, because though I could see how these tools could be useful I also knew they weren’t good enough yet. Over the years I tried Cline, Aider, Cursor, Windsurf, Gemini CLI then finally Claude Code + Sonnet 3.7. That was it. Things started to click.

With claude it was now the first time I could see how the agent’s tool calling worked. This transparency changed everything. Instead of mindless prompt engineering loops, I could look at the agent’s thinking traces, understand what went wrong when it failed, and actually optimize my workflow rather than trying to force a square peg into a round hole. Alongside that, the new models have unlocked a really compelling and useful workflow that at this point is completely pointless to argue against.

The point

Rogers Innovation Adoption Curve

I have a lot of thoughts on agentic coding that I’ll write about eventually. But one thing is clear: it’s not going anywhere. It’s not a fad. The way we use them is definitely still going to change, but when you cut through the noise from some people in the industry, and you give this an honest try you realize that this is just another way of coding and it’s here to stay.

Shadow Claude: A Second Opinion While You Code

Sun, 30 Nov 2025 21:46:37 GMT

What if you’ve had an LLM agent looking at your chat with Claude Code and calling bullshit on some of your bad decisions or maybe pointing out tradeoffs that you’re not considering?

Wait what?

When you’re deep in a coding session with Claude, you’re in the flow. You’re making decisions, accepting suggestions, moving forward. But sometimes you should step back and ask things like:

Is this the right approach?
What am I missing?
Are there gotchas I should know about?

The thing is, being self aware is hard. Stopping to ask these questions breaks your flow or you can get some decision fattigue and honestly, you just forget to do it.

What I Tried Before

I’ve experimented with some workarounds:

Tmux splits with forked conversations. I’d resume the same Claude session in a second pane and ask meta-questions there. “What do you think about this direction?” “Let’s think out of the box here.” It worked, but:

You’re stuck with the same bloated context
You have to remember to actually do it
It’s just one more thing to manage

Sub-agent patterns. There’s an MCP called Zen that lets your agent spin up other LLMs, send them context, get feedback. Similar idea, but it’s the agent asking, not me getting passive insight.

Neither was quite right.

The Click 💡

Then I saw that Msty Studio just released a feature called Shadow Persona. It’s essentially a side panel where a pre-configured agent reviews your main conversation and gives feedback.

That was the click. This is a real thing people are building. Now I wanted the same, but for claude.

So I did. At just before sleep. In bed. As one does.

Solution: Shadow Persona

A side-cart style agent that passively monitors the primary conversation and gives feedback without disrupting flow.

Version One: Validating the idea

The first prototype was simple:

Tmux split
Send a prompt to Claude programmatically via claude --print
Dump the response into the side pane with tmux send-keys

No formatting, no syntax highlighting. Just text. But it worked.

Version Two: Actually usable

This is what I built the next morning.

Design Constraints

Zero friction: No keystrokes required once activated
Model independence: Shadow LLM can differ from primary (e.g., Claude primary, Gemini shadow)
Persona specialization: Static system prompts define critique style
Context limits: Aggressive summarization before transmission

Workflow:

I’m in a claude-code session
I type !shadow
A vertical Tmux split opens
The split runs Neovim rendering a markdown buffer
Every three conversation rounds, it triggers automatically
A separate LLM (configurable) evaluates the conversation using a predefined persona prompt
The response renders live with full markdown syntax highlighting

The result: I’m working on the left, and on the right I’m getting ongoing commentary.

“This approach has these trade-offs…”

“You might want to consider…”

“Watch out for this edge case…”

A second opinion, automatically delivered, without breaking flow.

The Recipe

Tmux for pane management
Neovim for rendering markdown with syntax highlighting on the terminal
An LLM provider wrapper
Persona prompts that define what kind of feedback you want

How It Actually Works

Here’s what’s happening under the hood:

┌───────────────────────────────────────────────────────┐
│              Claude Code Session (tmux)               │
│                                                       │
│ UserPrompt ──► trigger.py ──► FIFO ──► loop ──► LLM   │
│                                                  │    │
│ ┌────────────────┬───────────────────────────────▼──┐ │
│ │  Main Pane     │  shadow-claude Pane (nvim)       │ │
│ │                │                                  │ │
│ │  claude code   │  nvim --listen .../observer.sock │ │
│ │                │        │                         │ │
│ │                │  observer.lua (buffer mgmt)      │ │
│ │                │        │                         │ │
│ │                │  render-markdown.nvim            │ │
│ └────────────────┴──────────────────────────────────┘ │
└───────────────────────────────────────────────────────┘

The flow is kinda simple: Claude Code fires a hook when you submit a prompt. That hook triggers a script that writes to a named pipe. Another script runs a loop in the background that reads the pipe, it eventually hits the gateway, and streams the response into a Neovim buffer on the right.

When you end the session, a different hook kicks in and generates a summary.

The Moving Parts

Hooks are how Claude Code lets you intercept events. I use two:

trigger.py fires on every prompt, counts rounds, decides when to ping the shadow
summary.py fires on stop, sends the transcript off for a summary

The FIFO pipe is just old-school Unix IPC. The hook writes JSON, the loop reads it. No sockets, no complexity.

The gateway is a thin wrapper around LLM providers (Anthropic, Google, whatever). It handles caching, loads the persona prompt, streams the response back.

The observer buffer is a scratch buffer in Neovim. A small Lua plugin manages it—append new content, clear when needed. render-markdown.nvim makes it look nice.

State I Have to Track

A few things live in ~/.cache/shadow-claude/:

Counter — how many prompts since last shadow trigger
Last position — where I left off reading the transcript (so I don’t re-send the whole thing every time)
Events log — mostly for debugging
Nvim socket — so the loop can talk to the observer buffer

This works surprisingly well and Nano Banana Pro visualized its architecture like this:

Shadow Claude Architecture

And here’s Shadow Claude giving feedback on its own implementation

Shadow Claude Demo

As you can see form its comments, this is not quite there yet, but I want to publish this once I’ve get to use it for longer and have refined it a bit more. It’s a fun little side project.

Running on the Edge: What, why and how

Sun, 06 Apr 2025 14:22:20 GMT

I recently gave a talk about Running LLMs locally on Android at the GDG Berlin Android, and I thought it would be useful to break it down into an article. This article breaks down what ‘Running an LLM on the Edge’ means, why you’d consider it for Android apps, and explores the current options available.

Edge Inference

An LLM is basically a large file containing learned language patterns represented as numbers. Running an LLM is often called inference. That just means we’re actively calculating and predicting the output based on its learned internal model. That is: performing statistical inference.

To run an LLM your text prompt is converted into number tokens, fed into the model which calculates the most likely next token. This process repeats, using the newly generated token as part of the input for the next step, allowing the model to generate (hopefully coherent) text. That’s why they call them autoregressive models.

So, when you run an LLM locally, you’re essentially using a program to perform these tokenization, calculation, and step-by-step prediction steps using that massive model file stored on your machine. It gets complex under the hood, but the core idea is just that: predict the next piece of text based on what came before.

You can get a bit more of the intuition what that looks like when using the transformers library to run Gemma3. If you find that confusing, don’t worry. We can go up another abstraction layer.

Probably the most famous open source inference engine is the llama.cpp. There are alternatives, and even abstractions built on top of it like the ollama project, which makes running an LLM as easy as:

❯ ollama run gemma3:27b
>>> Send a message (/? for help)

So that’s Inference. What about Edge? Well, that’s just means the consumer hardware. Your laptop or your phone.

Ok, with this new knowledge we can reinterpret the article’s title. It’s about what are the alternatives for Inference Engines on Android.

But y tho?

Weren’t LLMs Super Hardware Intensive?

That’s one of the questions that can pop up. We hear about LLMs needing data centers full of GPUs, so running them on a phone sounds… ambitious. Plus, we already have powerful options like ChatGPT and Gemini available in the cloud. Why go through the hassle of running them locally?

Well, there are some reasons:

Cost: Cloud LLM APIs often charge per token (input and output). If your feature gets popular, those costs can skyrocket. Running on-device means the inference cost is essentially offloaded to the user’s hardware (the edge), potentially saving you a lot.
Latency: Calling a cloud API involves network roundtrips. On-device inference can be significantly faster, leading to a much snappier user experience.
Privacy: This is a big one. If your app deals with sensitive user data (think personal messages, health info, finances), sending it to a third-party cloud might be a non-starter. On-device processing keeps that data on the user’s phone.
Offline Availability: Self explanatory.

Buried in the AI News Hype Cycle:

When we talk about AI breakthroughs, the news usually focuses on the massive, state-of-the-art (SOTA) or frontier models. These are the Geminis, Claudes, Deepseeks and GPT-4s of the world. They crush benchmarks every week and dominate the headlines. But they’re just the tip of the iceberg.

LLM iceberg with SOTA models

But beneath the surface, something else is happening.

Efficient small models

There’s a fast growing category of smaller, highly efficient, and often open-source models. I like to call these the “Top Left Corner” models because when you plot performance against resource usage (GPU requirements or size), they show up on that top-left spot: high performance with low resource needs.

These models are catching up fast. Models like Mistral 24b, Llama 3.1, and Google’s own Gemma3 family are hitting benchmarks today that were SOTA just 6-12 months ago. Benchmarks show these smaller models achieving performance levels comparable to much larger models from just 6-12 months prior, often on significantly less demanding hardware. Think about that for a second. Models that can run on consumer hardware right now are just as capable as huge models were just last year.

Mistral 24b latency results

Gemma3 ELO Score vs Size

Gemma3 ELO Score vs GPU requirments

Gemma ranking on lmarena.ai

And importantly, some of these are efficient enough to run locally, even on mobile hardware.

Your Options for On-Device Android AI

Okay, if I this your attentionc, then you might be wondering how to do that. How to run LLMs on a device? Right now, there are essentially two officially supported paths for integrating on-device Generative AI into your Android app:

1. The System-Wide Approach: Google AI Edge SDK

This is Google’s integrated solution, designed to be the easiest path for developers. It leverages components built directly into the Android platform on compatible devices.

Pros:
- Simple API, very similar to other Google AI SDKs (Vertex AI, Generative AI SDK).
- Platform handles model downloads, updates, and hardware optimization.
- Built-in safety features.
- Supports LoRA adapters for efficient fine-tuning.
Cons:
- Limited Device Support: Currently only available on select high-end devices (Pixel 8/9 series, some flagship Samsungs).
- Significant Storage: Requires specific AICore and PCS APKs to be installed and updated, and the base Gemini Nano model takes up ~4.3GB of storage before any LoRA adapters.
- Model Choice: You’re currently limited to using Google’s Gemini Nano model provided by the platform.

2. The Self-Managed Approach: MediaPipe LLM Inference API

This option gives you more control and flexibility but also puts more responsibility on your app. MediaPipe is Google’s broader framework for on-device ML pipelines (vision, audio, text).

Pros:
- Wider Device Compatibility: Doesn’t rely on AICore/PCS, so it can run on many more Android devices (even on iOS).
- Model Flexibility: You can run almost any text-to-text LLM, provided you can convert it to a supported format and the device has the enough resources (more on that later).
- Full Control: Bring your own base model, your own fine-tuned model (supports LoRA).
Cons:
- App Handles Everything: Your app is responsible for downloading the model, managing updates, initializing the inference engine, and managing resources (memory!).
- More Involved: Requires a bit more understanding of ML concepts and model formats.
- Potential Quality/Size Constraints: You need to choose models small and efficient enough to run well on target devices.
- No Built-in Safety (from the API): You might need to implement safety filters yourself if required.

Gotchas and Limitations

Whichever path you choose, be aware of some current realities:

Fine-tuning is Likely Needed: Base open-source small models can be quite “raw” or generic. You’ll likely need to fine-tune them (using LoRA or other methods) for your specific task to get good results.
Hardware Requirements: These models still need significant resources:
- Space: As we saw, the AI Core APK takes over 4GB just for the base model. Self-managed models vary, but even smaller ones are hundreds of MBs or several GBs.
- RAM: Inference, especially with larger context windows, consumes a lot of RAM.
- Processing Power: While optimized, inference still needs decent CPU/GPU/NPU power for acceptable speeds.
Context Window Limitations: On-device models typically have smaller context windows than the big cloud ones. Passing too much text (long conversations) will lead to signifcant performance degradation.
Narrow Use Cases: Current on-device capabilities are good at tasks like text rephrasing, smart reply, proofreading, and summarization. More complex reasoning or generation is still challenging for the smaller models (though fine tuning can help a lot)

Why Should We Care About This Now?

You might look at the limitations and think:

I can’t see these smalls beating cloud proprietary ones. Why bother?

And you’d be right. For now. History doesn’t repeat, but it often rhymes.

From Mainframes to the PC

Remember mainframes? Well, I haven’t seen one. But back in Uni I heard that those were huge supercomputers meant for serious work. They were the only thing that could run actually useful computer programs. It would take a dedicated refrigerated room with super expensive accessory hardware. And people could communicate with them via a terminal. A real terminal. Those had a keyboard and a screen, mainly handling input/output for the central machine.

From wikipedia

In the early days of computing, with the advent of interactive computing, the prevailing model involved a central computer connected to multiple terminals. This configuration, known as the centralized or mainframe model, featured a powerful central computer that performed all the processing tasks, while terminals served as input/output devices for users to interact with the system.

1970s-1980s mainframe computers

Computer terminal

Then came microcomputers, first seen as hobbyist toys on which one could play around, but not good enough for any serious work. We know how that turned out.

Early personal computer

Does that sound familiar?

I could go on and mention smartphones vs PCs and how there’s now a huge cohort of the population that uses these handhelds as their main computing device. Smartphones are still not faster than laptops or desktop computers, but they’re good enough for many people. You get the point.

The pace of improvement in AI, both in model capability per size and in hardware efficiency (TPUs/NPUs on chips), is incredibly fast. The gap between cloud and edge in some areas is shrinking fast. What seems like a “toy” or niche capability today could be a standard, expected feature on mid-range phones in 18 months.

So in my opinion, getting familiar with the concepts, APIs, and challenges now may pay off in the future. Besides, I’d argue it’s pretty fun to build with.

Getting Started

So, what’s the takeaway? While the AI Edge SDK offers the simplest integration, its device limitations are significant right now. The MediaPipe LLM Inference API offers broader compatibility and model choice but requires more heavy lifting from your app.

My recommendation?

Prototype & Validate: Use tools like Google AI Studio or cloud APIs Vertex AI in Firebase, to validate your feature idea.
Consider On-Device: If your feature requires privacy, offline use, low latency, or if cloud costs are a concern, then explore the on-device options.
Choose Your Path:
- If targeting only the latest high-end Pixels/Samsungs is acceptable, AI Edge SDK is a good choice.
- If you need wider device reach today or want full control over the model, MediaPipe LLM Inference API is a route, but be prepared for the extra work of model management.

The “genie” of powerful AI is definitely starting to squeeze into the “bottle” of our mobile devices. It’s not fully comfortable in there just yet, but it’s making progress fast. IMO it’s an exciting time to be an Android developer and lead the way. Make no mistake, competition will soon innovate and make a huge fuss about this too.

Other Resources:

Google AI Edge SDK Documentation (Official Docs)
MediaPipe LLM Inference API (Official Docs)

AI Edge SDK Overview
Gemini On Android - The Story So Far
Prototyping an AI APP Series. Parts 1, 2 and 3

Putting the Genie in the bottle

Sat, 09 Nov 2024 14:22:20 GMT

On October first, Google opened up the public beta of the AI Edge SDK. This SDK promises smooth integration with the new on-device AI components on Android. So, in case you want to build AI features that deal with sensitive user data or that do not require internet access, this will likely be the way forward.

If the above sounds like a bit of gibberish, or maybe if you haven’t been following what’s going on with AI on Android and how we got here, you may want to check my previous article, Gemini On Android - The Story So Far, where I talk about that.

This time I went through a bunch of docs, official blog posts, and announcements about the foundational on-device AI components the Android team is cooking up and finally tried it out on my own device. Here are some of my impressions and my initial hands-on experience with them.

Quickstart

The docs are pretty good to get you started, but here’s the gist of it. As of now, using the AI Edge SDK requires just three things:

A compatible device.
A valid beta version of the AICore APK.
A valid version of the Private Compute Services APK.

To achieve this, follow the prerequisite steps on the docs, and then add the Maven coordinates in your Gradle build file and you’re good to go. To speed things up, you can download their sample app (this fork includes a small fix to the build setup) and run it.

In my case, the compatible device is my new Pixel 9 Pro, so my impressions come from using the SDK on it.

P9P 🔥

Once you’re done with the setup, you’ll get to do this:

Apparently Kant hates Dagger

Hmm… maybe not ideal. Let’s try something else.

Translation got commoditized

Ok, that’s impressive. My app can translate stuff locally now. What else?

Right…

Alright, let’s get back on track. In just these few paragraphs, you may have noticed a new word salad: AI Edge SDK, AICore, Private Compute Services. Also, notice that we haven’t mentioned Gemini so far. Let’s break that down.

On-device AI Architecture

There are basically three components in this architecture.

1. AI Edge SDK

So, that’s how Google is calling it in the docs. But, you know, naming is hard. So although the “product” is called AI Edge SDK, internally, this particular artifact seems to be called aicore. Anyway, it’s a good thing that it has a very generic name so we don’t tie ourselves down to specific model names or technology. Remember Bard? RIP.

implementation("com.google.ai.edge.aicore:aicore:0.0.1-exp01")

If you’re wondering about where this edge comes from, it’s related to Edge Computing. This is basically the idea of offloading work that usually happens on a centralized server to the edge of the network. So in this case, we’re talking about moving the model inference from the in Google servers to your device.

This SDK is how our apps can get access to on-device AI infrastructure, and it should be the only part that our app will be directly interacting with. These are the things you can do with the SDK:

Check if the device is supported.
Get access to the Gemini Nano model.
Tune safety settings.
Run inference at high performance.
Optionally, provide a fine-tuning adapter.

The APIs for accessing Gemini Nano currently support text-to-text modality, but they mention that support for audio and image is coming soon.

2. AICore

This is the OS-level integration that Google has built. But why, you may ask? Good question!

A huge problem for running inference on mobile is that AI models and weights take up a lot of space, and we’re talking multiple gigabytes of storage. So one of the main goals of AICore is to manage the deployment of models and weights and their updates (via the Play Store, like a normal app). This also allows a way for multiple apps to share the same model, saving a lot of storage.

On top of that, we get built-in safety features (probably required by regulators), so you don’t have to deal with users trying to jailbreak your app’s LLM. Besides that, you get abstractions to deal with different hardware interfaces, so you don’t have to. According to Google:

AICore takes advantage of new ML hardware like the latest Google TPU and NPUs in flagship Qualcomm Technologies, Samsung S.LSI, and MediaTek silicon.

You see, you don’t really need Google to run on-device inference. People have been doing it for some time now, but it’s far from an ideal flow, especially for anything production-ready.

That architecture looks a bit like this

AI Edge Architecture

3. Private Compute Services

AICore is PCC compliant, meaning it’s super secure, and cannot directly access the internet. Any requests to download models or other information are routed through a separate, open-source companion APK called Private Compute Services.

Privacy through Data Isolation

This component helps protect your privacy while still allowing apps to benefit from the underlying models.

A good example is the Smart Reply feature which recently rolled out to WhatsApp. It uses Gemini Nano to suggest replies and without PCC, it would be able to read the conversation context directly. With PCC, the code that has access to the conversation runs in a secure sandbox and interacts with Gemini Nano to generate suggestions on behalf of the keyboard.

This is also how Live Caption and Now Playing run. You can find out more about how in the this is possible in the PCC Whitepaper. If you understand this bit, you’ll finally be able to explain to your friends how these apps are able to do this without being creepy 🙃.

With that out of the way, we can finally get to this full diagram.

AI Core Architecture Diagram

Now that we understand roughly how it works, let’s dive in.

Hands On

If you try to launch that sample app without meeting the prerequisites, you’ll get this error message.

Failed to check model availability.
com.google.ai.edge.aicore.UnknownException: AICore failed with error type 2-INFERENCE_ERROR and error code 8-NOT_AVAILABLE:
Required LLM feature not found

Then, after updating the AICore, the Private Compute Services apps, and opening that sample app again, you should see in the logs that the model starts downloading:

com.google.ai.edge.aicore.demo       D  Downloading model: 3063135863 bytes
com.google.ai.edge.aicore.demo       D  Downloaded 132720 bytes
com.google.ai.edge.aicore.demo       D  Downloaded 2090361 bytes
com.google.ai.edge.aicore.demo       D  Downloaded 2090645 bytes
...
com.google.ai.edge.aicore.demo       D  Downloaded 4358020858 bytes
com.google.ai.edge.aicore.demo       D  Download completed.

So the model is around 2.8 GB, but there’s more being downloaded, I hooked it to GlassWire and saw this.

GlassWire Network Monitor

It shows that just over 3 GB of data was downloaded using Private Compute Services, as they promised they would. Then, checking my AICore App, I you can see that it’s now taking around 4.3 GB of what is probably the uncompressed model + weights.

The bottom line here is that a minimum of 4.3 GB of local storage is going to be required for this to work.

AICore Storage Usage

Show me the code already

For something so “exciting” the code you need for using is a bit underwhelming. I mean, it’s very basic. There are just a couple of things you need to do to make this smoother for the user.

The main object is the GenerativeModel. It takes two parameters: DownloadConfig and GenerationConfig.

GenerationConfig is where you define the main settings, like maxTokens, temperature, etc. These parameters are standard across LLM models and frameworks, so I’ll leave them out of this article’s scope. A quick Google search should get you in a good place.

Putting things together we get:

val downloadConfig = DownloadConfig(
        object : DownloadCallback {
            override fun onDownloadStarted(bytesToDownload: Long) {}
            override fun onDownloadFailed(failureStatus: String, e: GenerativeAIException) {}
            override fun onDownloadProgress(totalBytesDownloaded: Long) {}
            override fun onDownloadCompleted() {}
        }
)

private val model = GenerativeModel(
    generationConfig = generationConfig {
        context = applicationContext
        maxOutputTokens = 600
        temperature = 0.9f
        topK = 16
        topP = 0.1f
    },
    downloadConfig = downloadConfig
)

This is it. Some weird parameters, but that’s all there is to it. Now, before we’re good to go we need ensure the model is ready to use. Checking things like:

Is this a compatible device?
Is the model downloaded?

To do that, we can call the prepareInferenceEngine and handle the exceptions.

runCatching {
    model.prepareInferenceEngine()
}.onFailure { e: GenerativeAIException ->
    Log.e("TAG","Model not available: ${e.message}")
}

Finally, you can generate the response synchronously.

val response = model.generateContent(input.buildPrompt())

Or do it asynchronously; for simplicity, I’m just appending the newly generated text as it’s being streamed, like so:

var result = ""
val job = scope.launch {
model.generateContentStream(input.buildPrompt())
     .collect { responseChunck ->
        result += responseChunck.text
        println(result)
    }
}

// Which you can cancel midway
job.cancel()

Now that’s it. If you run that code, Gemini will start streaming down text. That is of course, if you ask it nicely. What do you mean nicely? Well… it’s complicated.

Mic check, 1,2… is this thing on?

You see, I think it’s fair to say that most of things you’ll try with work straight away, but it’s likely you’ll hit some snags on the way. Eg.:

✋ Stop! We don’t do that here.

It’s unclear what exactly is considered out of bounds, and I think they’re being very conservative with these settings, which is likely to avoid some PR disaster. For an experimental product, I think that’s fair.

If you do ask it nicely, then I have to say that it’s pretty fast! Especially if you remember that this is the worst it’s ever going to be.

What 45 tokens/second looks like

Apart from security, you’ll notice another an issue when you start testing the limits of its .

First of all, I couldn’t find any explicit information about Nano’s context window size. In the paper, they say that the models support a 32k context window, but I assume that’s obviously not the case for Nano running on a phone.

Now, the paper also says that Nano is a 3.25B parameter model. As we saw on my last article, the Pixel 9 Pro has 3 GB of RAM reserved for LLM inference.

Back of the napkin estimation - don’t take this seriously!

3.25B × 4 bits = 13 billion bits = ~1.625GB for model weights. Given we have 3GB of RAM available, that means we need to stay within 1.375GB.

I did some research and an uninformed estimate could gives us between 2048 and 4096 tokens.

Empirical trials

Or Throwing things at it and seeing what sticks.

I prompted it with an excerpt of 4k tokens from the Gemini paper to ask questions about it. It worked reasonably well. For reference, ~4k tokens is what you read from the beginning of the article until here.

Then, I tried increasing that to around 7k tokens by adding excerpts from another paper and it started logging some weird errors, and I couldn’t get it to answer it. Finally, I tried duplicating the original prompt, getting close to about 10k tokens and it worked(?)! Though it took way longer.

As you can see, my math was off (as usual), but so were the docs and the error messages. Actually, that’s another big pain point when working with this, but at least they’re aware of it.

AiCoreInferenceHelper
com.google.android.apps.aicore.base.InferenceException: null (statusCode = 4)
at biw.a(PG:1049)
at dgx.a(PG:13)
at duz.g(PG:3)
at dvb.run(PG:42)
at cou.run(PG:3)
at dsl.run(PG:50)
at amg.run(PG:50)

These are not super helpful. The internal components (things inside com.google.android.gms.internal.aicore) are obfuscated, so the SDK really works like a black box, which is somewhat expected since LLMs are, by definition, black boxes. Though, I’d like to see better API documentation, including the recommended context window size and information on performance degradation for larger contexts.

Making it smarter

Not gonna lie. Nano is not the smartest kid in the room, and it shows. The main problem is that On-device models have some hard constraints, like the limited parameter size and smaller context windows. To improve their performance we can use fine tuning. Or more specifically, fine tuning, which is like “fine-tuning on a budget”. You get similar results with way less memory and computation.

Press X to doubt

It’s no doubt that things are still very rough around the edges. You can certainly build products with it, like Google and other OEMS are doing, but there’s a few blockers that we can need point out:

Device support: Just a few devices currently have the hardware to support this and those are top of the line ones. Android is known for cheaper models which make it ideal for users on a budget (most people). It’s still unclear to me how well the new hardware and increased memory requirements are going to rollout to these cheaper devices or how long would that process take.

Base models lower quality: LoRA can help, but Nano is still not going to match what you can do with the models running on the cloud which limits the use cases a lot.

‘This ain’t going nowhere’

It’s a movie, not a picture

Now, the counter argument to that is that while those points are true, we need to understand how this environment is playing out.

For example, what are some of the core constraints:

Compute: How hard we train each generation of models.
Parameter size and training data: More parameters generally mean that the model have a better understanding of the input, and more training data gives it a better understanding of how different concepts relate to each other in meaningful ways.

Despite all the valid criticisms you can have, what’s happening in AI right now is nothing short of absurd. This stuff puts Moore’s Law to shame. Doubling the number of transistors in a chip every 18 months? Neah… how about a 10 fold improvement over these core constraints every 12 months?

Training compute (FLOP) used over time

Parameter vs training dataset size

There’s a counter argument to these graphs, which say that we may run out of data before to train them soon. However, if you connect the dots with what Google specifically is doing for this to work on Mobile, then you start getting where this is still likely going. Just look at this.

Recent break throughs that made mobile inference possible

Mobile = ~Cloud/10

Mobile hardware constraints are about 2 orders of magnitude from the workstations or the cloud and judging from the data above it starts looking clear that it’s a matter of time before local models are just as capable as the current generation of cloud models.

Look, I’m not making any claims about AGI or AI improving exponentially forever. I’m just pointing out that what we can do in the cloud now, will be feasible on devices soon, which I think it’s reasonable and backed by the data.

Building for the future

Now, if you’re convinced that all of this makes sense and you’re interested in building something what to do now? Should you get a new device just for that?

Well, I don’t think so. At least not yet. The AIEdge SDK is still experimental and supported on just a few devices, so I recommend another way of going about this.

In my opinion, at least so far, Google has adopted what it looks like a very coherent approach for their AI dev tools. The APIs are very similar and there’s options for gradually going up the ladder of investment. Just look how similar they are.

AI Edge SDK

import com.google.ai.edge.aicore.GenerativeModel
import com.google.ai.edge.aicore.generationConfig

private val model = GenerativeModel(
    generationConfig = generationConfig {
        context = applicationContext
        maxOutputTokens = 600
        temperature = 0.9f
        topK = 16
        topP = 0.1f
    }
)
val response = model.generateContentStream(input.buildPrompt())

Google Generative AI SDK

import com.google.ai.client.generativeai.GenerativeModel
import com.google.ai.client.generativeai.type.generationConfig

val generativeModel = GenerativeModel(
    modelName = "gemini-flash-1.5",
    apiKey = BuildConfig.GEMINI_API_KEY,
    generationConfig = generationConfig {
        maxOutputTokens = 600
        temperature = 0.9f
        topK = 16
        topP = 0.1f
    }
)
val response = model.generateContentStream(input.buildPrompt())

Firebase with Vertex AI

import com.google.firebase.vertexai.GenerativeModel
import com.google.firebase.vertexai.type.generationConfig

val model = Firebase.vertexAI.generativeModel(
    modelName = "gemini-1.5-flash",
    generationConfig = generationConfig {
        maxOutputTokens = 600
        temperature = 0.9f
        topK = 16
        topP = 0.1f
    }
)
val response = model.generateContent(input.buildPrompt())

Moving from one to another is almost as easy as changing the imports.

So I think a good game plan would be to validate your AI feature idea using AI Studio. If that works, then you can prototype it with the Generative AI SDK. If you’re confident that it can be shipped, roll it out to your users (probably A/B testing) using Firebase with Vertex AI.

If you get positive feedback and you think price will be a problem, or rather, if the AI input data is privacy-sensitive, then you can try your chances with the AIEdge SDK.

Wrapping up

The article stretched out more than planned, but if you’re still here, you’re either really into this or just jumped to the conclusion. In any case, the bottomline is that rate of improvement is significant enough that local models could be 12-18 months away from being capable of handling many use cases locally. You can start thinking about it or even start building now.

If you’re interested, you can read this three-part series on Prototyping an AI App, where I documented my process.

We’re not there yet, but in my opinion this is a preview of what will likely become a standard capability of future apps. Just like GPS, Bluetooth, or NFC, many apps will build on this. Not every app will need it, just as not all apps require GPS or Bluetooth, but certainly, many will. You can read some more thoughts about that

What’s next?

There’s yet another SDK for LLM inference on Android that, as pointed out by Paul, I have not talked about. It’s the MediaPipe LLM Inference API and hope to play with that in the near future.

For now, I want to get some hands-on experience with LoRA and how to apply to Gemini Nano. So you can expect an article on that soon.

Besides that, earlier this year I’ve pushed a container to GCP with a Python backend and a very basic RAG implementation using Vertex AI and LangChain. Now that Gemini 1.5 is out, I need to try it with the new 1MI or 2MI tokens context window options and eventually hook it up with my AudioGuide AI App I built last year. Let’s see if can put those together.

I’ll keep you posted either way. Cheers!

Gemini on Android, The story so far

Thu, 17 Oct 2024 17:39:24 GMT

Here we go again

Right. Another article about AI. I’m sure you’ve read enough about it so far, but chances are that if you landed here, you have at least some vague interest in it. I, for one, am very interested on the tech and with the direction this is going, especially with what has just started to roll out. FYI: if I say AI in this article, I probably mean LLM.

On-Device LLMs are here!

Google just announced their Android AICore beta program, and with it, we finally got access to Gemini Nano with AI Edge SDK via AICore. Did Android smartphones just got smarter? Well.. kinda.

AICore Beta Program

Now, honestly, this is nothing short of amazing. The thing is that we’re on 2024, and so overwhelmed with information that it is sometimes hard to appreciate just how incredible this last paragraph reads. Especially if you’re sceptic (and rightfully so) about all the AI bros selling snake oil out there, but I digress.

It was just in November 2022 that OpenAI released ChatGPT-3.5, the first version to become wildly popular, and here we are now, less than two years later, with a somewhat comparable model running ON DEVICE!

But how did we get here?

Android 🤝 LLMs - Quick Recap

AI at the core

This story started back in 2023 with the Pixel 8 lineup, the first smartphones to ship with Google Tensor processors that were good enough for LLM inference.

This enabled the devices to run LLMs locally, though initially limited to just a handful of apps and features, like smart replies and the recorder transcript summarization. There was a program that you could enroll to get access to Gemini Nano, but that was kinda restrictive. So no open access yet.

But of course, on-device generative AI is really complex: 150 times more complex than the most complex model on Pixel 7 just a year ago. Tensor G3 is up for the task, with its efficient architecture co-designed with Google Research.

Source

Generative AI SDK

A few months later, around December last year, Google started gearing up and stealthily releasing the Generative AI SDK, which made it easier to play with the Gemini API. This one was not running on device, though.

It was still very limited, and clearly not a lot of resources were put into it. Heck, it was not really announced anywhere. Besides, it was still blocked in a lot of countries (Europe) by that time.

I remember using a VPN to access Bard early on, but it was a one-time thing.

Now, I suppose I'll have to wait until it's available in the EU. I wonder how OpenAI managed to push generative AI apis here while Google is still restricted. https://t.co/yzHV6rIrve pic.twitter.com/5dzuRrXpfv
— iury souza (@IurySza) December 29, 2023

That happened right around the time I was building my first Gen AI app. You can read more about that here: Building an AI Audio Guide - Part 1 . I was mainly targeting OpenAI’s API back then, but I was able easily hook up Gemini using that SDK. Quality was still not there yet, but it worked.

Vertex AI for Firebase

Fast forward another six months, and we’re at Google I/O Connect 2024. Now, a joint venture with Vertex AI and Firebase was announced. Vertex AI is Google’s Cloud solution for AI products and now, app clients could tap into it without having to deal with GCP.

What I got from this was that while the Generative AI SDK was OK for prototyping, Firebase Gen AI was meant to be a secure and more robust channel for production use.

Any Local LLMs Yet?

During that same Connect event, Google announced that Android would soon open up access to core APIs for on-device LLM inference. Some architecture diagrams were shown about this new AI Core thing. But still nothing for us devs to play with.

Then, a couple of months later, the Pixel 9 family was launched with more Gen AI stuff and a slightly better Chip in terms of raw performance, but with a new TPU (Tensor Processing Unit) for faster LLM inference, reportedly at 45 tokens (which is pretty decent) per second now. Also, it’s interesting to notice that these devices shipped with min 16GB RAM out of which 3GB reserved just for LLM inference.

Pro models lock 3GB RAM exclusively for AICore and the Tensor’s TPU.

Yes, you read that right. Almost 20% of device memory reserved for AI. Also, you should expect multiple GBs of model data stored locally (more about that later). This should not be taken lightly. They mean business!

Ok, so finally, on October 1st, Gemini Nano with Google AI Edge SDK was open for us devs 🎉. Well, at least for beta testing.

Gemini Nano experimental access is designed for developers seeking to test enhancement of their apps with cutting-edge on-device AI capabilities.

Gemini Nano

The Gemini Nano model is specifically optimized for on-device use, which means that it can runs completely locally.

Nano is the smallest of the Gemini lineup, with Flash, Pro and Ultra being the other ones. It is the most efficient version and because of that it needs to be much smaller, with Nano-1 and Nano-2 models being only 1.8B and 3.25B parameters, respectively. For comparison, Gemini 1.5 Pro has 1.5 trillion - with a T (note that these things don’t exactly scale linearly though). Due to this size difference it had to make some compromises, but still managed to perform remarkably well.

How? Quoting straight from the Gemini paper:

The power of Nano lies in its training process, which involves “distilling” knowledge from the larger Gemini models. This approach allows Nano to deliver substantial computational power in a smaller form factor, making AI more personal and accessible.

This model is not as general as the other ones. Being smaller, is better suited for some specific tasks, also quoting the paper:

These models excel in summarization and reading comprehension tasks with per-task fine-tuning.

You can see these features in action on some of the new Pixel apps like the AI Weather Report of the new Pixel Weather App and the Pixel Screenshots app.

To get started with it you have to enrol on the beta program. This will enable you to update the AICore app which comes installed on the Pixel 8 and 9. After that you can start using the SDK.

Note: I’m purposefully skipping the MediaPipe LLM Inference API, released early 2024, because using it is more involved and requires us to manually push models (non Gemini) to the device. I’ll try to revisit it later.

What’s next?

Welcome to the new normal. Some people loathe it, but this has been an open secret for years. This vision of a shift from a mobile-first to an AI-first world was announced by Sundar himself back in 2017 and I can only see it accelerating.

The Big Shift

I could write another article just about this vision and what it means for mobile dev, especially us Android Developers, which for years were used to be in the very center of everything Google announced (remember IO 2018?) and now are left feeling a bit alienated by these changes. But let’s do that another time. For now, I’ll just say that mobile is still the platform. AI will be just another tool in our belt.

Welp... you know what's up tomorrow.
A erm... — a SHIT TON of AI is about to drop!

This quote is from a 2017 article from Sundar, I but thnk they're now positioned to make that happen. Don't worry, mobile isnt going anywhere, I just thnk that they're about to dial up some knobs https://t.co/oJRy1MtZ0Y pic.twitter.com/qqSiln5pZJ
— iury souza (@IurySza) May 13, 2024

This was a just a short recap of how we got here, I’m trying to keep these in a shorter format, so I’ll have to wrap this up. But if you’re interested in learning more, like and subscribe 😛.

I’m currently playing with the Nano model on the Pixel 9 and hope to put together some of my notes into another article about my first impressions and some ideas of how you can try implementing LLM based features on any device.

Kotlin DEVEX is not great, but Amper could fix it.

Thu, 13 Jun 2024 20:05:37 GMT

I like working with kotlin. Great language, great features, super nice community, and java interop can be a superpower.

That said, Kotlin Developer Experience (DEVEX) gives me mixed feelings. On the one hand you have Jetbrains, the maker of arguably the best suite of IDEs in the market behind it. This automatically gives us, kotlin devs great support for language features directly in our editor. Debugging and refactoring Kotlin code is a breeze comparing to most other languages.

Now, the same cannot be said about other parts of the typical day-to-day workflow. For example, say you want to build a tiny Kotlin library targeting the JVM and distribute it for other devs to use. That’s a standard workflow that in the year of our lord, 2024, should be more than straight forward, right? Let’s take a look.

Getting Started

Ok. How would you go about doing that? Well, on Kotlin’s Getting Started page, Jetbrains states:

Install Kotlin

Kotlin is included in each IntelliJ IDEA and Android Studio release. Download and install one of these IDEs to start using Kotlin.

Fair enough, we’re going to use IntelliJ anyway, so yeah. Fire up your IntellijIDEA and use the New Project menu.

That works, but IMO, far from ideal. Not a big fan of clicking through a GUI just for this. What I prefer instead is using a CLI tool. Luckily, Gradle helps with that. Being the most common build tool for kotlin projects these days, I think fine to have it handle this part of the flow. This requires you to have Gradle installed globally on your system, which you can do by using sdkman. This is what it looks like to create a new kotlin library project using the Gradle CLI:

$ gradle init
Starting a Gradle Daemon (subsequent builds will be faster)

Select type of build to generate:
  1: Application
  2: Library
  3: Gradle plugin
  4: Basic (build structure only)
Enter selection (default: Application) [1..4] 2

Select implementation language:
  1: Java
  2: Kotlin
  3: Groovy
  4: Scala
  5: C++
  6: Swift
Enter selection (default: Java) [1..6] 2

Enter target Java version (min: 7, default: 21): 18

Project name (default: gradle-kotlin):

Select build script DSL:
  1: Kotlin
  2: Groovy
Enter selection (default: Kotlin) [1..2] 1

Generate build using new APIs and behavior (some features may change in the next minor release)? (default: no) [yes, no] yes


> Task :init
To learn more about Gradle by exploring our Samples at https://docs.gradle.org/8.7/samples/sample_building_kotlin_libraries.html

BUILD SUCCESSFUL in 37s
1 actionable task: 1 executed

This is what you get:

$ tree -L 3
.
├── gradle
│   ├── libs.versions.toml
│   └── wrapper
│       ├── gradle-wrapper.jar
│       └── gradle-wrapper.properties
├── gradle.properties
├── gradlew
├── gradlew.bat
├── lib
│   ├── build.gradle.kts
│   └── src
│       ├── main
│       └── test
└── settings.gradle.kts

Ok, this is kinda nice. There is a way to get started via CLI, like all the other cool kids do. Check ✅.

Adding a dependency

Now, what if you want to add a new dependency? How do you go about that?

Say for example, I need to use okhttp for creating a small HTTP client. How do I do it? First of all, we need to check which version are we going to use and what are its maven coordinates.

You can find out by navigating to maven and searching for okhttp. Click through the UI and find out the version you want to use.
Then you would navigate to the build.gradle.kts file in your project and add this snippet to it.

dependencies {
    implementation("com.squareup.okhttp3:okhttp:4.12.0")
}

Then, you would sync your project, by running ./gradlew or clicking through the GUI and you would be able to use the library.

Done! Added a dependency. And now you’re ready to code your small client application. This felt a bit convoluted though. If you’re a beginner you probably won’t even know what implementation stands for. What about that coordinate? You’re not gonna memorise them, are you?

Now, let’s stop right there and go the other side.

The Python way

Let’s contrast that to Python when using Poetry.

If you’re not familiar with it, Poetry is a dependency management and packaging tool for Python. It provides a standardised way to manage Python projects, including: environment and dependency management, lock files, packaging and more.

Getting Started and Adding a dependency

$ poetry new pytho_demo && cd python_demo
Created package python_demo in python_demo

$ poetry add flask
Creating virtualenv python-demo in /Users/iury.souza/projects/python-demo/.venv
Using version ^3.0.3 for flask

Updating dependencies
Resolving dependencies... (0.3s)

Package operations: 7 installs, 0 updates, 0 removals

  - Installing markupsafe (2.1.5)
  - Installing blinker (1.8.2)
  - Installing click (8.1.7)
  - Installing itsdangerous (2.2.0)
  - Installing jinja2 (3.1.4)
  - Installing werkzeug (3.0.3)
  - Installing flask (3.0.3)

Writing lock file

You’ll get this:

$ tree -L 3
.
├── README.md
├── poetry.lock
├── pyproject.toml
├── python_demo
│   └── __init__.py
└── tests
    └── __init__.py

3 directories, 5 files

Done. Much simpler, right?. You can immediately start coding. This highlights a few key differences between the two development workflows and their developer experience:

What Poetry does better

Ease of Use: Python’s Poetry offers a straightforward, CLI-driven approach to managing dependencies. In contrast, Kotlin’s reliance on manually configuring Gradle files is more involved and error-prone.

Impact on Development Workflow: Python developers can rapidly add and manage dependencies without stepping out of their development flow. On the other hand, Kotlin developers often find themselves fiddling with build scripts and build syncs.

Reproducibility and Environment Management: Poetry’s management of pyproject.toml and poetry.lock files ensures that builds are reproducible and environments are consistent across different machines and setups. With Kotlin there’s no such elegant solution. You could use the gradle lock options, but that’s not a standard workflow and doesn’t actually work the same way.

Why a CLI approach is better

Using the CLI is a better approach for a few reasons:

Consistency: The CLI provides a consistent interface for developers to interact with the build system. This reduces the cognitive load on developers and makes it easier to remember how to perform common tasks.
Automation: The CLI can be easily automated using scripts or other tools. This allows developers to perform repetitive tasks more quickly and efficiently.
Replicability: The CLI commands can be easily shared with other developers, making it easier to replicate workflows across different machines and environments.

What’s all the fuss about?

You might be thinking:

- Uh… I don’t see the big deal. I’m fine with the GUI. It’s just a few clicks.

Well, yeah. But consider this: How many times do you add or remove dependencies from a project? Besides, I didn’t even touch other aspects like testing, or publishing Kotlin libraries as it would make this post too long, and would hit the same points. All of these are also more convoluted than they should be and could benefit from a CLI based approach.

Besides, a lot Kotlin (especially Android) get so rely on the IDE for managing their environment and end up not getting familiar with a CLI based workflow. Missing all the benefits that come with it.

An obvious counterargument is that Kotlin has many constraints that Python and other languages doesn’t have, like the JVM, and the need to be compatible with the whole Java ecosystem, but maybe there’s some tradeoffs people would be willing to make to have a better DEVEX in at least certain use cases. I think that either the Kotlin or Gradle team could take a page from Poetry’s book and work to make the development experience more streamlined and user-friendly.

Moving Forward

I think there’s a huge opportunity here for Kotlin to improve its developer experience by providing first-class support for the command line for common workflows. Jetbrain’s Amper kinda looks like a step in the right direction; I hope we devs can tap into that system via the CLI.

If you've been paying attention, you might have noticed that both @gradle and @jetbrains are working on transitioning from imperative to declarative approaches for configuring builds. ⚒️
— iury souza (@IurySza) May 31, 2024

A Glimpse into a Better Kotlin DEVEX

What Poetry does for Python, Amper could do for Kotlin. When you run Poetry init, it generates a pyproject.toml file. This is what a typical pyproject.toml file looks like:

[tool.poetry]
name = "audioguide-ai-api"
version = "0.0.1"
description = "The backend for the audioguide-ai project"
authors = ["Iury Souza <iurysza@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12.2"
Flask = "^3.0.3"
python-dotenv = "^1.0.1"
gunicorn = "^22.0.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

This is a declarative way to configure your Python project. You can either modify this file directly or use poetry commands to add or remove dependencies, etc.

Now, this is an example of Amper’s module file, which is also declarative way to configure your project:

product:
  type: lib
  platforms: [jvm, android, iosArm64, iosSimulatorArm64, iosX64]

  # Shared Multiplatform dependencies:
  dependencies:
    - org.jetbrains.compose.foundation:foundation:1.5.0-rc01: exported
    - org.jetbrains.compose.material3:material3:1.5.0-rc01: exported

  # Android-only dependencies
  dependencies@android:
    - androidx.activity:activity-compose:1.7.2: exported
    - androidx.appcompat:appcompat:1.6.1: exported

  # iOS-only dependencies with a dependency on a CocoaPod
  dependencies@ios:
    - pod: 'FirebaseCore'
      version: '~> 6.6'

  settings:
    kotlin:
      serialization: json

    # Enable Compose Multiplatform framework
    compose: enabled

Imagine now, if you could run something like amper init and have a project scaffolded for you. You could then run:

$ amper add okhttp -android

And have a new dependency added to your project, and a lockfile generated. After that you could run:

$ amper install

Which would read from the lockfile, download the dependencies, and sync the project for you. Maybe you would have your credentials in the module.yml file and you could run amper publish to publish your library.

Conclusion

These are just some thoughts I’ve put together. I think that Kotlin is a great language, but there’s a lot of room for improvement in the developer experience. I think that embracing the CLI more fully could be a step in the right direction.

If you ever wrote a Node module, a Ruby gem or a Cargo package, you know how easy it is out there. I think it’s high time we have that too.

Note: I chose to compare Kotlin with Python just because I recently had to work with it and it was fresh in my mind, but the same could be said about other languages like Rust, Go, JS, Ruby and many others.

Sketch, refine, rinse and repeat

Sat, 20 Jan 2024 19:48:07 GMT

Let’s get to it!

This is the final installment in my three-part series on prototyping an AI audio guide app. If you missed them, here’s where it started: Part 1 and Part 2.

Following my mantra of fast iteration, I decided to just build the most basic thing first. This will help you cut through the writers block issue and get you into writing the code faster.
That’s usually what we devs want to do, but it can result in poor decisions and harming your project from the start. Because of that, I prefer a middle-ground solution that addresses both problems. I call this the coding equivalent of Sketch and Refine.

In art and design, ‘Sketch and Refine’ is a method where an initial sketch is refined over time to create a final piece, improving precision and depth in the design process.

This is an example of that in practice. Source: Digital Art Timelapse.

Image	Description
	Initial outline for a semi-realistic digital painting.
	Coloring and detailed features added.
	Well-defined portrait with attention to facial nuances.
	Clothing, textures and hair refinement added.
	Final stage: last details, textured background and sunlight highlighting.

The idea is to iterate through multiple versions of the product rather than trying to get it right on the first try. This approach enables collecting real-world feedback, making adjustments, validating assumptions, and gradually improving the product.

A similar concept, often credited to Kent Beck, has long been associated with the Unix Way.

Make It Work. Make It Right. Make It Fast.

So, that’s the plan: to build a very basic prototype and keep iterating on it.

Basic Prototype

Having the “Sketch & Refine” idea in mind, I decided to plan the basic app flow with three simple screens: Home Screen, Preview Screen, and Audio Guide Screen. This is what it looked like:

App Flow Sequence Diagram

Next, I went for the most simple implementation. A single screen handled asking for all permissions right at launch. Once those were granted, I’d grab the user’s last known location using Android’s play-services-location library and feed those coordinates into Google’s reverse geocoding REST API to translate them into an address. At the same time, I used Google’s NearbyPlaces REST API to get the landmarks close to those coordinates.

Tapping a button would send a camera intent. From this, I got a low-resolution snapshot, converted it into a base64 string format, and sent all that data to the LLM. Not pretty, but it got the job done!

This was implemented in the quickest and dirtiest way possible. Just make it work. Even the commits were done in a quick and dirty style.

$ git logline

* 35f817e - Implement bot (11/12/2023) <Iury Souza>
* b474f43 - Add support for taking pictures (11/12/2023) <Iury Souza>
* a421a34 - Implement location data fetching (10/12/2023) <Iury Souza>

Refining

Once you’ve got that initial prototype working, refactoring the code and seeing the big picture becomes much easier.

Those shortcuts you took suddenly scream at you, and you get the space to make important architectural decisions.

Delaying decisions

So now, on one hand, you have the opportunity to apply all the fancy architectural patterns you’ve learned, the latest modularization approaches, and even decide which esoteric libraries you’d like to have on the project. Or, go full wild west – a few messy files, a bunch of singletons… why not? Now really, finding the balance is key. There’s this spectrum I always think of:

These choices will shape the entire coding experience from now on, so it’s important to get them right.

The sweet spot depends on your style and the trade-offs you’re willing to make. My rule of thumb is: Do not rush these decisions.

The YAGNI principle is your friend – decide when you have to, when you fully understand the problem and have viable solutions in mind. Similarly, you should let your project’s requirements mature before making decisions that are hard to reverse – if you do, you’ll be sure you’re optimizing for the right goals. And finally, don’t create abstractions just for the sake of it!

Cutting the bullshit

Still under the same idea, these are some of the things I’m not doing at this stage:

Single module. I don’t need a multi-module architecture. There’s absolutely no benefit in bringing a multi-module setup to the table so early on.
No custom gradle setup. Getting fancy with gradle build config is also something people waste way too much time. KISS: Keep it simple stupid
No code gen. I’m avoiding code gen libs, particularly KAPT is out question.
DI? I went cowboy with a simple Service Locator. No frameworks.

Obviously, if I have to change any of these decisions I can, but I don’t need to commit to any of them yet.

This ain’t no Wild West either

I’m not gonna go full cowboy either. I’m still following basic coding principles and standards. Separating concerns, writing tests (where it’s useful at this point), and keeping the code clean (such a loaded term) and readable, of course. I’m even using a simple MVI architecture to keep things organized and consistent.

The MVI thingy looks a bit like this. Consumers can send events, listen to State and Effect changes. It gets the job done, without being overcomplicated. It also forces the implementations to be consistent and this is actually what I’m looking for here. I generally shouldn’t have to spend a lot of brain cycles whenever I need to implement something simple. There should be a standard way of doing common things in this part of the app.

class UserContextViewModel(...) : MVIViewModel<
        UserContextViewEvent,
        UserContextViewState,
        UserContextViewEffect
        >() {

    override fun setInitialState() = {...}

    override fun handleEvent(event: UserContextViewEvent) {...}
}

Besides that I didn’t add the standard clean architecture components like repositories or usecases. I settle with simpler abstractions like LocationProvider or GoogleMapsApi, a simple HTTP Client wrapper that exposed the Google’s REST APIs.

I didn’t want to add a lot of layers here because I just didn’t know exactly what I was building. So making things easy to refactor was important.

To me, this series of decisions and compromises is what makes the project enjoyable to work with. It’s a side project, you don’t need to make your life harder than it needs to be. It’s all about finding the right balance between speed and quality.

Tiding things up

At this point we’re still only dealing with the happy path, but now we add:

A permissions pager with explanations, etc
A Gradle versions catalog (only Gradle change, promise)
A Navigation library

I also Refactored the code to use service locators and that basic MVI architecture and finally define theme and colors properly.

It still looked ugly, but internally it was much better.

Dealing with LLM Prompts

Or how I learned to stop worrying and love the ~~bomb~~ AI.

We’re all used to dealing with HTTP-based APIs, specially RESTful APIs or any kind of RPC. This is the long-standing norm in most software.

These days we have more sophisticated protocols like gRPC or GraphQL. While they introduce new features and improvements they’re fundamentally under the same paradigm. In this case, remotely accessing data stored elsewhere.

LLMs on the other hand unlock a different monster. We as a client application can now effectively tell this black-box what we want it to do, via a prompt and it will happily answer us. But not only that. Via some prompt engineering techniques you can alter the way it answers. Change for example the format of the answer. Maybe you want the answer to contain some metadata about the text, so you can ask for it to answer in Markdown format. Or maybe you want it to answer in a particular json structure. You can even ask it for things it doesn’t know, and it will happily hallucinate an answer for you.

This is a brave new world for anyone, like me, who has never worked with this kind of tech before.

Down the Prompt Engineering rabbit hole ⤵️

Well, like I said in Part 1, one could say that I was an early adopter of this tech, as a normal user at least. So I knew about the basic prompting text style and was able to come up with this prompt.

My initial prompt:

You are a tour guide. You have a laid back personality and like to
enthusiastically
talk about monuments and history. You're good at
providing facts and bits of trivia of the places people ask you about.
You can also understand city addresses and can pinpoint any location
of the city of `input.city` just with its address.

A tourist just sent you a picture and this address: `input.address`
of a place in `input.city`. The place is near these
locations: `input.getLocations()`

The tourist asks you "what's in the picture"?
Please answer this person in the style of a tourist guide
who's enthusiastic about history.

- Immediately start talking about the monument or historic place.
- Don't start the text with: "Ah, what a splendid
  sight to behold!" or similar phrases.
- Do not repeat the address or the city name in your answer.
- It's important that you answer in a way that's
  informative and interesting.
- Try to figure out which particular part of the monument
  the picture is focusing on and talk about that.

Nothing outstanding there, I just provided it with the context and gave it some instructions on how to talk. For a particular address that I provided it would return this kind of output.

Output:

Right before your eyes, you're looking at a powerful photographic
display located near the historic Bornholmer Strasse in Berlin,
which is part of the larger Place of 9 November installation.
This particular image captures an extraordinary moment in history,
an intimate glimpse into the jubilant faces of East Berliners
at the Bornholmer Strasse border crossing on the
night of November 9, 1989 – a night that altered the course of history.

This was the first crossing to open during the fall of the Berlin Wall, marking
a pivotal point in the reunification of Germany. People streamed through here,
in disbelief and elation, as the barriers that once divided a city, and
symbolically the world, were lifted.

In this surrounding area, the picturesque Kirschblutenpfad, or Cherry
Blossom Path, awaits you during the spring, standing in contrast
to the Bosebrucke, another witness to the changes that swept
across this city.

This installation invites contemplation and serves as a portal to the past,
allowing visitors to stand amidst history retold through imagery. As you wander
alongside this precious slice of the city's timeline, you can almost hear
the echoes of cheers and the rustle of footsteps that marked the beginning
of a new era. it's not just a photograph, it is a testimony to unity, to change,
and to the enduring spirit of berliners.

Decent response, but I thought it could do better.

Time to read the manual.

Prompt Engineering Techniques

At this point I started to look into the different techniques to improve the quality of the responses and I found a few of them that I thought were interesting.

Role prompting

This is the one is where you give the AI a role and ask it to play it, which usually increases the quality of responses quite a bit. This is a basic one, and I realized that I was already doing it.

You are a tour guide.

The role you assign serves as a guideline for the kind of response you’re looking for. It’s good for bringing out specific styles, tones, or levels of complexity in responses.

System Roles

Instead of sending the role prompt via the standard user prompt field, you can set it as a system prompt. Which is said to be injected into every question.

It can be instructions guiding the LLM Behavior like:

Assistant description: profession, persona, etc
Context: information, preferences, etc
Response format: email, summary, json
Boundaries: don’t talk about this, don’t repeat that, etc

Solo Performance Prompting (SSP)

This is a method where a single Large Language Model (LLM) acts like multiple experts (or personas) working together to solve complex tasks.

This is important because it improves the model’s ability to handle tasks that require deep knowledge and complex reasoning, making it more effective and versatile.

A basic example of its use could be an LLM playing the roles of a historian, a scientist, and a travel guide to answer a question about the historical and scientific significance of the Eiffel Tower.

The model, using SPP, would combine insights from these different personas to provide a comprehensive and well-rounded answer.

Few shot prompting

This is a technique where you provide a few examples that demonstrate the kind of output you’re looking for before sending the actual query.

These few examples (or “shots”) give the AI with a pattern to follow when dealing with the actual task.

Zero shot prompting

By appending words like “Let’s think step by step.” to the end of a question, LLMs are able to generate a chain of thought that answers the question. From this chain of thought, they are able to extract more accurate answers.

What worked for me

You see, this still feels more like invoking incantations than doing actual software development. Besides, the qualitative measurement of these changes are super subjective and hard to assert. Most of the times you’re left with going with your feeling.

But some simple tricks can make working with this way more manageable. For example, asking for an output format is super useful to interface with the unstructured nature of the LLMs outputs and the structured nature of our code base.

Role Prompting

This is the most basic one and think we’re all pretty much doing it at this point. Tell the AI its job, its role

You are a tour guide. You have a laid back personality and like to
enthusiastically talk about monuments and history. You've been
living in `input.city` for the past 30 years and know the city like
the back of your hand. Your main target audience are tourists aged from
18 to 45, so you have a casual and friendly speech tone.

Output formatting

This is the best example of managing this complexity.

Your output format is a json file without the code block
(`json content`)

{  
 "title": "string",  
 "content": "string"  
}

The `title` should describe the monument
or historic place. Keep it short, 5 words max.

This was great. Now with the title I could actually create a screen instead of justing rendering a text.

There was still a problem though. Whenever I would send a picture of something that’s not a monument, it would start hallucinating (technical term for talking gibberish) and going into tangents just to keep talking. So I added this to the prompt:

Your output format is a json file without the code block
(`json content`)

{  
 "isValidRequest": "boolean"  
 "title": "string",  
 "content": "string"  
}

The `isValidRequest` field should be `false` if the image is **NOT**
related to a monument or historic place and `true` otherwise.  
When `isValidRequest` is `false`, the `title` and `content` should be omitted.
The `title` should describe the monument or historic place.
Keep it short, 5 words max.

Now, I had error handling built-in the output.

Few shot prompting

This idea worked pretty well. What I did was basically pick one of the outputs and change the parts which I didn’t like. Replaced some words, changed the tone of the writing and etc. This caused the other answers to follow a general underlying pattern that it picked from that writing. Which I feel is a great idea.

Sometimes it’s hard to explain exactly what you want, so an example fills in the gaps for you.

Out of the prompt rabbit hole ⤴️

You can spend quite a while improving your prompts, there are many other techniques that you can and should try. My final prompt had quite few more things into it, but I feel there’s a point where adding new things stop being effective, so it’s a balancing act too. And like I said in the beginning all this few more like throwing a bunch of shots into a wall and seeing what sticks. So keep that in mind.

Back to refining - Making it pretty-er

Prototypes are nice and all but a nice app is much better. But what kind of UI should we have here? This is sort of an unusual app so I couldn’t ~~just copy~~ take inspiration from another bigger app.

I as a user just wanted to go from camera to audio guide as fast as possible. The main problem was big loading times and how to display the audio guide in an interesting way. This app UI should be minimal anyway so this shouldn’t be rocket science, not for a side-project.

I had some notes about what I wanted to build:

- Home: catchphrase: (CallToAction)
  - Start camera
  - Take picture
    - Show fancy animation for uploading picture
    - Request audio content and stylized img based on the picture uploaded
    - Show blurred out stylized pic as background for text streaming.
    - play audio on top of blurred background

So I had 3 main UI problems to solve.

Home
Loading
Playback

Massive loading times ong loading times are a real pain point – my app was taking an average of 35 seconds to generate text and audio! That’s a lot of waiting. The plan was to eventually implement text streaming chatGPT which improves the perception of waiting a lot. Implementing that was part of the plan, but not at this stage.

I wanted to get the basics down first, and since I would still need to wait for the audio to be generated anyway, so I would still need a decent loading screen.

So I went with a combination of lottie animations and displaying randomly selected sentences in the style of Maxis (from The Sim & Sim City series) loading screens (try to spot them on the app demo video). This was fun bit that I added, it may start to feel repetitive after a while, at this stage it got the job done (making waiting easier).

Random technical-sounding loading sentences

Karaoke Playback

The second thing I wanted to solve was the Playback screen. I decided to implement a feature similar to spotify’s synced lyrics. Here, I had the audio and the text that would be played, but I didn’t have the actual speech rhythm and you can’ t expect every word to be said following the same speech rhythm pattern, otherwise it would sound awfully like the old windows TTS from 20 years ago. These days we have more natural sounding voices, which bring this problem back.

My solution was to get the current audio playback percentage and sync that to the scrollState of the component that would display the text. This component would have fading text borders in a way that the current sentence being played would be displayed in the center and the rest of the text would be faded out. This kinda creates the illusion of a karaoke style component, even when it’s not perfect. Good enough for me.

The next piece of the puzzle was making the text more easier to parse for us. For that, I just modified the prompt to get it format the text with markdown, so I could render it in a more interesting way. I added this to the prompt

The text is in the content field should use bold markdown formatting for the
following cases:
- Dates.
- Names of places.
- Addresses.
- Names of people.
- Historic events.
- Name of monuments.
- Names of historic figures.

To do that I used the nice compose-rich-text library which literally takes of everything.

Dev Mode

Geolocation coordinates are crucial in this app. To fine-tune the output, you’ll typically provide a picture and a lat/lng. Real-world testing gets repetitive fast – you don’t want to take a new picture for every test.

That’s why I created a Dev Mode screen. Here, you can enter a picture URL and paste some coordinates, skipping the usual photo capture and location tracking. Later I even expanded this system to stub out the ChatGPT API for better testing the karaoke flow.

This setup was a lifesaver during UI development. With quick tweaks and no cumbersome photo-taking, I could iterate on the design much faster.

Ironing it out

Now I only wanted to do a couple of improvements like finally adding text streaming support and supporting Google’s Gemini SDK.

Text streaming was pretty easy to support, since it was built into both SDKs and my implementation was flexible enough to support it. So while text was streaming I displayed it like a loading page, and only showed the final audio guide karaoke after the TTS was loaded.

Finally, Gemini was released just a week prior to this stage, so I wanted to check that out. Living in the EU I knew it was blocked, but that didn’t stop me from trying. So I just added a dev-mode option to enable it and it worked as expected - a bit underwhelming at this point, but I wanted to have the freedom to switch engines if I wanted to.

App Pitch

Here’s a quick pitch of the app. It’s not the final version, I posted this before it had text streaming support.

I've been toying with the chat-GPT vision API since last week and I've put together this prototype.

I'm still exploring what it's capable of, but so far I'm impressed.

Apart from improving the rubbish UX/UI, the next step would be to use word embeddings to improve its accuracy pic.twitter.com/cABPZ2mY5U
— iury souza (@IurySza) December 22, 2023

Final touches

This is the final version of the app I shared just before taking my yearly 20 days off to travel to Brazil.

I finished ironing out some things here like rendering partially loaded content while streaming text and improving the "karaoke" audio guide screen. I try to keep the highlighted text in sync with the audio and added Netflix inspired skipping button animations 😛.
Also Gemini (+) pic.twitter.com/yQLzF6FJBP
— iury souza (@IurySza) January 9, 2024

Next steps

Okay, I’m excited to get this app published! Before a full release, I’ll definitely take a better look into the pricing model and try to get it published for beta-testers only the coming weeks.

This is far from a finished product and I know what you might be thinking:

It doesn’t get things right all the time. It hallucinates or it is just stealing content from other people!

And I hear you. I also think it’s not perfect. That’s why I’m not monetising it.

Although there are a few things I could do to improve the quality of the answers by a LOT.

RAG (Retrieval Augmented Generation): In sum, this lets the LLM query a vector database for content related to the question and inject that content into its context, which enables it to pretty much solve the hallucination problem in a lot of cases. You can read more about that here.
Google Gemini 1.5: This recent update massively increases the context window. More context means better answers and could help with many of the problems RAG addresses. I’m curious to see where this leads!

Curious about the App?

The app is still not ready for a full release, but I’d be happy to get some feedback from you. If you’re interested in trying it out, send me an email and I’ll include you in the beta.

Conclusion

This was such a fun project! I can’t remember the last time a new technology blew me away like this. When ChatGPT went mainstream in 2022, it felt like magic. Even now, I’m still amazed by some of the things it can do.

Of course, there are plenty of issues, and it raises serious questions about how we’ll adapt as a society, but part of me can’t help but feel excited about the prospects of integrating this into our apps and tools.

Something I felt about this project from the start was that this will get built by someone. I’m sure that in a few months, a big company will pick this up. Even Zuck talked about how he wants Meta’s Ray-Ban Glasses to be able to answer historical data of a building, so I’m pretty sure that will cover this audio guide use case too. Anyway, it is nice to play around with the idea while there’s still nothing out there.

I learned a ton about LLMs and how to turn them into real products. Plus, it was fun to build some weird UI components!

Update: I eventually built the dedicated backend for this prototype to explore RAG, retrieval, and audio generation properly. I wrote a follow-up on the deployment side here: From GitHub Actions to Cloud Build.

Planning and documenting a side project

Sat, 13 Jan 2024 22:39:07 GMT

This is the second part of a series of articles about prototyping an AI audio guide app. Missed the other parts? There you go: Part 1 and Part 3.

In the first part we talked about how to go from an idea to a basic prototype. Now, in this short article I’d like to give my 2 cents on how to document and plan the execution of that idea, or any side project.

Mind you that this is not a tutorial, but rather a case study of how I approached this topic. Besides, it’s important to say that this works best for very early stage projects, where you have a lot of questions and not many answers.

What’s next?

I was sure that the core functionality worked and I could start building an app around that idea. But at this point of the project there are a lot of open questions, like:

How would I structure the project and what architecture to use.
Which libraries would be needed?
How would the app flow would look like? What about the design? The UX?
What permissions do I need? When to ask them?
How could I improve the prompts? Is prompt engineering an actual thing?
How would I display the actual output to the user? Show a text? Just play the audio? How does the UI for an AI audio guide look like?

The more you think about the more questions you get.

Obsidian and Mind Maps

The first thing you need to do at this point is putting all those questions in a place where you can see them all at once. Then you start answering and organizing the answers into categories. This is where mind maps come in handy.

Notes help a lot, but you’ve probably been at this place where you write down a bunch of notes, but you can’t make sense of them. Or they don’t seem to connect, and you can’t see the big picture. To fix that we can use something like Obsidian.

It is already my go-to note-taking tool, but it’s also great for mind mapping with the canvas feature. Each note is a node, and you can link them. Notes can even reference each other with hyperlinks, so you can easily navigate between them. It’s all going there - diagrams, designs, ideas, case studies, laid out in a big mind map with all those markdown files. So, let’s get to it.

The two starting nodes were these:

From there you can start to break down the project into smaller parts and try to focus on each one at a time.

Research and documentation

Those starting nodes would prompt questions I had to answer, which lead to other topics I had to research and document. For example, I would research which APIs I could use, then document the steps to use each of them, think API request examples, dashboard panels, links, api keys, and so on. Then, when I was ready to integrate it to the project I had a clear path to follow.

For example, to answer the question “how to add location context to the LLM prompt” I had a simple idea: Use the device’s GPS to get exact coordinates, then use the Geocoding API to get the precise address, finally we could fetch the top 5 tourist attractions near those coordinates.

After some research I figured that the Places SDK could solve that, but after testing I realised it doesn’t filter for tourist attractions, so I had to resort to using the plain REST API.

Having done that, I could step back and take a look at the big picture. I could see the whole process of getting the data I needed to build the experience I wanted.

This process also worked when defining how I wanted the app flow to look like. Starting from sequence Diagrams, wireframe sketches, and finally a higher-fidelity design.

All this information linked in the same space, makes it easy to navigate and understand the project as a whole.

Here is the full mind map. I included information about decisions and documentation on prompt engineering, design, UX, sequence flows, even a Kanban board for tracking the feature backlog. Having worked on other side projects where I didn’t document anything, I can say that this approach is a game changer.

Wrapping up

This whole documentation topic might seem like a chore or just irrelevant, but especially when it comes to side projects, they can be crucial for taking an idea to fruition. But we usually just don’t pay enough attention to it, which often leads to wasted effort and sometimes giving up on an idea altogether (been there, done that). Besides this, there are the obvious benefits of having a clear plan and an organized place to find important information.

All of this is vital for keeping side projects alive. It’s easy to lose interest and move on when you hit roadblocks, so having a dedicated space to store your thought process makes it much easier to pick up where you left off.

In the next and final article, we’ll talk about a mental model for architecture decisions, trade-offs, learnings, and guidelines. See you there!

Jump to Part 3

From idea to code

Tue, 09 Jan 2024 22:39:07 GMT

This is the first part of a series of articles about prototyping an AI audio guide app. Looking for the other ones?

Happy new year! A bit late, but here we are. This is the first part of a series of articles where I’ll be writing about my first experience building an AI-powered mobile app. This was a fun learning experience and I hope to share something useful here.

Now, first, a bit of a background story. You can safely skip this part if you want.

Background story

I’m the kind of person who’s naturally curious, I can nerd out about literally anything and that’s probably why I’ve picked up so many hobbies over the years. It happens that I got into tech, so I’ve been nerding about that the longest. Always keeping an eye on the ever-changing trends in the industry.

My curiosity naturally drew me to AI around 2017, although from a distance. I remember talking about it with a flatmate (and great friend) of mine every evening. We’d watch videos on the new developments and the new papers coming out. I even convinced him to join me in Udacity’s AI Foundations Nanodegree, which we completed in 2018, basic stuff obviously, it was clear to me that this field would boom pretty soon.

Now, you can imagine how amazed I was when ChatGPT 2 came out, let alone ChatGPT 3. In 2023 AI went mainstream like we have never seen before. The hype was on par with – maybe even crazier than – all that crypto buzz, but this time actually useful products were being built.

The Chat-GPT Hype-train

I signed up as soon as it launched. For a while I obsessed with prompt engineering hacks and trying to see what this thing could do and how I could squeeze this new tech into my workflow. Writing, translation, coding, learning new things, you name it.

The next AI milestone for me was the Raycast Pro which added a chat-gpt wrapper interface. Now I could definitely incorporate it to my day to day tasks. Having tried lots of chrome extensions, Obsidian plugins and what not, this one felt natural. Now I was really using AI a lot for mundane tasks.

In my experience I think you really need to try and use this tech for you to really see what it’s capable off. A lot of smart people around me were writing it off because they had and incompatible mental model of it.

Sure, I hadn’t yet had that mind-blowing lightbulb moment, especially for anything I could build as a mobile engineer. Most apps I saw were about chatbots or image generating, which didn’t really click with me.

Then, comes Q4 and the multimodal models start getting released. So, now ChatGPT can understand pictures? Ok. This is interesting! I took a look at the OpenAI API docs – that vision API seemed crazy simple. So one chilly December night, I figured, why not give it a shot?

Picking the right tools

So, the first decision is which language to use, right? OpenAI has official SDKs for Python and JavaScript. I could work with both, but they always feel a bit like speaking a second language – I get by, but there are too many little mistakes and one thing I’ve learned over the years is that learning multiple things on a new project is going to slow you down. So, before picking one of those I researched Kotlin SDKs and found the openai-kotlin library: A kotlin multiplatform alternative that offers almost every feature of the official sdks, so I went with it.

✅ Check!

Inspiration

Ok, now what to build?

Well, I had just spent a week in Andalusia, South of Spain and had the fresh memory of walking around the city and juggling between multiple apps to figure out the monuments and what not.

Imagine this: You’re walking around Granada and want to know more about you’re looking at. You’re visiting, the Alhambra? Great, they’ve got an audio guide app web app. Next day, you’re at a medieval church in the city. Great, another app. Then you stumble upon this amazing monument – no app, so you’re stuck Googling (or asking ChatGPT). And sometimes you just see something incredible but have zero clue what it is – cue frantically searching Google Maps to even find the name!

Okay, sure, this is what happens when you don’t plan ahead. Guilty as charged. But still, there’s a problem to solve here.

I knew what to build.

What if there was an app where you snap a photo and ✨! – instant audio guide?

Honestly, even if I was the only one who used it, that sounded awesome. Alright then, time to roll up the sleeves!

1st Prototype: Just use ChatGPT

Now, the beauty of this Generative AI thing is that it’s super easy to prototype. This is what I did:

Went to open AI platform page. (This is important because, the ChatGPT ui that most people are used to is a product built on top of their APIs, so you need to use this platform to simulate the answers you’d get from their apis).
Uploaded a picture of the Berlin Brandenburg Gate,
Added a few instructions to the prompt

And there you have it! Not perfect, but a surprisingly solid answer. I tested a few other landmarks and it kept working well enough.

I was sold on this idea and these tests gave me confidence to start building the app. But not a mobile app, not just yet.

2nd Prototype: A Kotlin App

Instead of jumping head first into building the android app, I wanted to first iterate on the core implementation using the simplest approach.

This should be really straight forward:

Call the API passing an image file and a prompt.
Explore the ergonomics. E.g.:
- File uploading via the api
- How does text streaming works?
- How long does it take to answer
- How to integrate with the TTS (Text-To-Speech) API

You see, spinning up an emulator, or the basic infrastructure to test this basic flow of on a native android app is not as straight forward as calling a main function and that’s what I wanted.

And this is literally what the code looked like:

object TTS {
    suspend fun run(
        text: String,
    ): Result<Path> = runCatching {
        val openAI = getInstance()
        val request = speechRequest {
            model = ModelId("tts-1")
            input = text
            voice = Voice.Shimmer
            this.speed = 1.15
        }
        val audio = openAI.speech(request)
        writeToFile(audio)
    }
}


object AudioGuide {
    suspend fun run(
        input: TourGuideInput,
    ): Result<String> = runCatching {
        val openAI = getInstance()
        val request = chatCompletionRequest {
            model = ModelId("gpt-4-vision-preview")
            messages {
                user {
                    content {
                        text(buildPrompt())
                        image(input.imageUrl)
                    }
                }
            }
            maxTokens = 600
        }
         openAI.chatCompletion(request)
            .choices
            .first()
            .message
            .content
            .orEmpty()
    }
}

suspend fun main() {
    val input = Input(
        city = "Berlin",
        imageUrl = "https://static.dw.com/image/52796179_605.jpg",
    )
    AudioGuide.run(input)
        .onFailure { it.printStackTrace() }
        .onSuccess { response: String ->
            log(
                """
                        | Parameter | Value |
                        |---|---|
                        | City | ${input.city} |
                        | ImageUrl | ${input.imageUrl} |
                        | Output | $response |
                """.trimIndent()
            )
            TTS.run(response)
        }
}

And that was it. I built this on that very evening and was impressed with the results.

Improving the prototype

So that prototype worked, but as soon as you start using pictures that are not super clear, or basically whenever you ask about not so famous sights, then the magic starts to break off.

Which makes sense. This thing is not magic, it has lots of flaws, but if you give it enough context things start to improve. On the example above you can see that I already was telling it about the city where it was. This helped clear out a lot of confusion because it reduces the search space from anywhere in the world to just one city. But, can we do better?

Well, the idea was to eventually build a mobile app and if we’re an app, then we can get the user’s geolocation coordinates. We can also do Reverse geocoding (address lookup)! Telling it the city and street name would be nice… but upon testing this (by manually putting the address into the prompts) I saw that it still didn’t solve the problem.

Next idea: Google Places SDK! I could use that to find the tourist attractions nearby. It’s a plan! Now we:

Send the AI a picture
create a nice prompt telling it:
- The address where it was taken
- The top attractions nearby

Also, I decided that I was going to focus on a single city for this prototype. Since I was dependent on google places api, I couldn’t trust the quality of its answers for just any city (same for chat-gpt). So, I decided to focus on Berlin, the city I live in.

Validating the Solution

To validate all these assumptions, I still didn’t want to invest the time into implementing it, so I hardcoded the data.

I researched a bunch of hidden landmarks examples and used their data to feed into the LLM and then recorded the answers. Then, looking at the output, I was again happy with it.

This is an example of the output I got:

City	Berlin
Address	Bornholmer Str. 61, 10439 Berlin
TopLocations	Place of 9 November, Bosebrucke, Kirschblutenpfad, Scheifengarten
ImageUrl	https://berlintraveltips.com/wp-content/uploads/2021/07/Platz-des-9-November-Berlin-free-activities.jpg
Output	Right before your eyes, you’re looking at a powerful photographic display located near the historic Bornholmer Strasse in Berlin, which is part of the larger Place of 9 November installation. This particular image captures an extraordinary moment in history, an intimate glimpse into the jubilant faces of East Berliners at the Bornholmer Strasse border crossing on the night of November 9, 1989 – a night that altered the course of history. This was the first crossing to open during the fall of the Berlin Wall, marking a pivotal point in the reunification of Germany. People streamed through here, in disbelief and elation, as the barriers that once divided a city, and symbolically the world, were lifted. In this surrounding area, the picturesque Kirschblutenpfad, or Cherry Blossom Path, awaits you during the spring, standing in contrast to the Bosebrucke, another witness to the changes that swept across this city. This installation invites contemplation and serves as a portal to the past, allowing visitors to stand amidst history retold through imagery. As you wander alongside this precious slice of the city’s timeline, you can almost hear the echoes of cheers and the rustle of footsteps that marked the beginning of a new era. It’s not just a photograph, it is a testimony to unity, to change, and to the enduring spirit of Berliners.

Now I felt confident that this would at least make a good prototype.

Wrap up

Starting small, we can use generative AI like ChatGPT to play around, refine ideas, and turn them into something interesting. The key is - prototype and iterate!

By now, the mobile app is finished. But these series of articles are intended to document this process and hopefully give you insights along the way.

Jump to Part 2

Grounding Beliefs

Thu, 16 Mar 2023 11:46:37 GMT

If you came for the docs, head over to the repo.

Grounding Beliefs

I saw a tweet last year that stuck with me:

"A diagram about the current system is only useful if it's generated. If it is produced by hand it documents the author's belief, not the system." 💯 https://t.co/aenK2eRW07
— Marcello Galhardo (@marcellogalhard) March 5, 2023

That’s exactly what I’d been doing—manually drawing module graphs, watching them go stale, then redrawing them. Even after switching to Mermaid.js for version control, I was still maintaining them by hand. The tweet forced the question: why not generate them from the source code itself?

The Evolution of Tooling

I’ve used diagrams for years. Started on whiteboards, moved to diagrams.net for remote work, then Excalidraw for cleaner output. The “as-code” tools like Mermaid and PlantUML were an improvement—I could diff them, version them, and check them in.

But they were still static artifacts I had to remember to update. The tweet made me realize that was a bug, not a feature.

Prior Art

Before building anything, I looked for existing solutions. Jake Wharton had a script that parsed Gradle projects and emitted Graphviz DOT files.

It worked, but added a binary dependency and produced JPGs—useless for diffs and clunky in CI. I wanted something lighter that output plain text and ran anywhere.

From Script to Plugin

The result is a Gradle plugin that introspects your module dependencies and writes a Mermaid graph directly into your README. No external binaries, no manual steps.

I started with a POC and eventually published the full plugin to the Gradle Portal.

To get started, apply the plugin and run:

./gradlew createModuleGraph

You get a live module diagram that’s readable on GitHub and GitLab:

It should look something like this

Wrapping Up

It’s a single task that hooks into the build graph without getting in the way. The code is on GitHub. CI integration is trivial—just run the task and commit the delta.

This is my first plugin, so I expect rough edges. Issues and PRs welcome.

From macro to micro

Wed, 13 Jul 2022 15:30:37 GMT

There’s this interesting idiom in English:

to miss the forest for the trees.

From the Merriam-Webster dictionary: to not understand or appreciate a larger situation, problem, etc., because one is considering only a few parts of it.

I think this idea translates very well to problem-solving. Sometimes we fail to see the big picture because you’re trapped on the small details. Let’s consider for example the case of algorithm coding problems.

Algorithm coding problems

Whether you agree with solving these as part of a hiring process is beyond the point of this article. But if you visit a website like Leet Code or HackerRank you can easily spot that there are multiple categories of problems.

There are problems with trees, graphs, sliding windows, etc. Each of these categories represent a sets of problems which you can solve by applying a certain algorithm.

These algorithms are, at the highest level, the main tool you’re going to use to solve a problem. It is only natural that first you need to have these tools on your tool belt to be able to use them.

Your Tool belt

Knowing when to use an algorithm and knowing how to implement it should be the first step. That seems kinda obvious, right? But, the point is that if you’re solving a sliding window problem, creating a sliding window, needs to be trivial for you.

It can be really frustrating to forget about a certain check, or getting some kind of out of bounds exception because you can’t remember what a sliding window looks like. When you’re solving a problem, a sliding window has to be just a tool. When you decide to use it, it’s gotta be ready to go.

Knowing your tool

The other thing is understanding how to use that tool. Think of it as just any other tool… like a drill. It’s got some buttons and some switches that can alter its behavior. You can also use different bits depending on surface.

Having a drill by itself is not enough. You have to know what can be changed or configured on it for different scenarios.

a drill

For example, a sliding window can have a dynamic, or static width. Or maybe you need to keep track of different things while you’re moving across the dataset. Knowing how to work with the tool is key.

Sharpening your tool

The next thing is the language and its data structures and idioms. You have to be comfortable with the language that you’re going to work with. Compilation errors shouldn’t catch off guard you. You have to make sure that you know how to use its data types. Working with Arrays, and maps should really not be a problem.

Planning and Zooming

If you have these out of the way, then you can really focus on what the problem is about. In my experience, ideally you should to have a plan before you start coding. Drawing, sketching, writing, etc are all useful at this point. This way you can piece together each part of the idea and gradually zoom in, from the bigger picture, to the small details of the implementation.

Then, you can validate your assumptions at each step of the way:

Is my sliding window iterating over the dataset correctly?
Is this validation correct?
Is the content of that data structure correct?

But, if you start having issues with your sliding window implementation because you forgot how to keep track of the pointers, or if compilation errors keep popping up, or you can’t add new items to your hash-map, then this will take up your energy, and you won’t be able to be to focus on the actual problem-solving.

Since you can only keep a certain amount of subjects in your working memory, it’s important to use it efficiently. Avoiding constant changes in this zoom level helps to manage the amount of information you’re going to have to work with.

Wrap up - Keeping the forest in sight

Keeping context switching to a minimum is a key part of problem-solving for me. I usually trip over myself when I’m trying to think through something only to find out that some basic assumptions were wrong. Or if I keep getting weird compilation errors, or if my hash-map is not working as I thought. When you start to jump around fixing different parts of the implementation, at different “zoom levels”, you lose sight of the forest. For that reason, when studying, I think it’s important to check each of these stages before jumping into a problem.

Nesting React Components in Native UI

Tue, 05 Jul 2022 18:46:37 GMT

React Native has support for creating Native UI components which are implemented using the native ui frameworks, and exposed as react-native components. This is the most common use case, but there’s also a not so common one, which allows you to nest React Native components inside the native UI.

Now, let’s skip the whole discussion about whether you should do that or not. Since there’s not a lot of info about this on the web, the main point of this article is to give you a brief overview of what nesting react components looks like in practice and how to implement that on both platforms.

Native UI Components in React-native

I’m breaking down this feature in two main use cases:

Basic Native UI component

You have a native ui component which can be shown inside a React Native app, side-by-side a react component.

This one is well covered, you can follow the official docs to implement it.

Nesting React Native components in Native UI

In addition to the basic use case, this native-ui also supports having react components being laid out inside it.

Let’s take a second to visualize what we want to build:

So, what are we looking at?

We have a React Native App which has a Native UI Component.
This Native UI Component has a native view, which acts as slot in its view hierarchy where we place the nested react-native component.
Finally, we have a simple React Native Component as a sibling of the Native UI Component.

From the react-native side, this could look like this:

import { requireNativeComponent } from 'react-native';

export const YourNativeComponent = requireNativeComponent(
  'YourNativeComponent',
);

const App: () => JSX.Element = () => {
  return (
    <View>
      <YourNativeComponent>
        <View nativeID="aWildNativeID">
          <Button title="Nested React native Button" />
        </View>
      </YourNativeComponent>
      <Button title="React native Button" />
    </View>
  );
};

We’ll revisit this later, but for now, let’s take a look at the native code.

Native Changes

Let’s quickly go over the basic building blocks of native-ui integration with react.

On android we have the ViewGroupManager class, which you need to extend from to create a native ui component.

class YourNativeComponentManager : ViewGroupManager<YourComponentView>() {

	override fun createViewInstance(context: ThemedReactContext): FrameLayout {
	  return YourComponentView(context)
	}

	override fun addView(parent: YourComponentView?, child: View?, index: Int) {
  	  TODO("Add child to view hierarchy")
	}

	override fun getName() = "YourNativeComponent"

	// ...
}

On iOS that looks a bit different:

@objc(YourNativeComponentManager)
public class YourNativeComponentManager: RCTViewManager {
    public override func view() -> UIView! {
        return YourNativeComponentView()
    }
}

public class YourNativeComponentView: UIView {
    public override func insertReactSubview(_ subview: UIView!, at atIndex: Int) {
		// TODO: add subview to view hierarchy
    }
}

When the react-native side starts nesting components, as we showed before, you’ll start receiving calls to the addView method on android and to the insertReactSubview on iOS.

These methods will allow you to actually lay out the views on your component’s view hierarchy.

Advanced Layout Strategy

If you plan to nest multiple kinds of components in different places of your view, you will need to change things a bit.

Imagine you have react-component A and a react-component-B and they need to be placed in different places in your view hierarchy.

When addView or insertReactSubview are called, you’ll just get a new view, and you have to do something with it (put it somewhere).

Identifying react subviews:

To identify each view, you can tag them on the react-native side with the nativeID property.

This way, when you get that view on the native side, you can use that to figure out where should you place it.

On android, you need to use the react’s android library to do that via the ReactFindViewUtil.findView(view, "nativeId"). It will help you identify if a particular view has the given nativeID.

On iOS that’s a bit easier. You have access to the view’s nativeID directly via subview.nativeID property.

Laying Them Out:

To lay out these components accordingly, what I suggest is limiting them to a slot on your native implementation. This way, the native side can stay in control of its own ui. You assign each of these possible subviews a slot on your view hierarchy.

Kotlin Multiplatform under the surface

Wed, 18 Aug 2021 08:00:00 GMT

The mobile development community always had many alternatives for creating cross-platform apps. Since the days of webview-based solutions we’ve come a long way, but we still have many alternatives. The most recent one is the Jetbrains led, Kotlin Multiplatform Mobile project.

If you’re curious about how does it stand against the competition and what are its pros and cons, this brief overview may help you get started.

Cross-platform vs Multiplatform

As you may have noticed, Jetbrains calls its solution: Kotlin Multiplatform Mobile. This might get overlooked by a lot of people, me included, up until I decided to write this down. But then it hit me: what’s the difference between Cross-platform and Multiplatform?

After some research, I got the answer. From the user’s point of view, this is plain simple. In short: there’s no difference. Multiplatform and Cross-platform implementations achieve the same thing. It simply means that the application will run on - you guessed it. Multiple platforms.

Since we’re focusing on mobile dev, Multiplatform means Android and iOS. But this can translate to web, desktop… whatever. Just… multiple platforms. It doesn’t say anything about how the app was created. If you have a code-base for an app written in Kotlin targeting android and another written in swift targeting iOS, you have a multiplatform app.

Now, cross-platform is different. Cross-platform means that the same code you wrote will run on, for example, android and iOS. This is usually done via some kind of runtime that needs to be shipped to the hosting platform that enables that code to be interpreted for that particular environment it is running on.

So what about Kotlin Multiplatform?

The main idea behind Kotlin Multiplatform is code-sharing. If you enter the home page, you’ll see them mentioning how you can write the code once and have it running on both platforms. So -you may be asking, isn’t that cross-platform… like Flutter or React? What gives?

The main advantage of Kotlin Multiplatform is that it delivers native code for multiple target platforms/architectures by taking the same code as input. It is able to do this because of the Kotlin/Native Compiler, so to understand how Kotlin Multiplatform works we first need to understand what is Kotlin/Native.

Kotlin/Native

Quoting from Kotlin’s landing page:

Kotlin/Native is a technology for compiling Kotlin code to native binaries, which can run without a virtual machine. It is an LLVM based backend for the Kotlin compiler and a native implementation of the Kotlin standard library.

There’s a lot to unpack here. Let’s take the first sentence in:

So Kotlin/Native will give us native binary executables that can run on an OS using no virtual machine.

A binary is the machine code, ready to be run on the specified target architecture. So it’s the machine code for, for example, the iOS targeting a specific CPU architecture, like arm64 (which is the official iPhone architecture name)

Then it goes on to mention an LLVM based backend for the Kotlin compiler. This one is dense.

LLVM

The LLVM Compiler Infrastructure Project is probably the biggest open-source compiler project that exists to build native binaries. Languages such as C, C++, Haskell, Rust & Swift compile into native binaries through LLVM.

Ok, fine… but what about the backend? More specifically, what is a backend for a compiler?

The Kotlin Compiler, frontends, and backends

The Kotlin compiler first compiles your Kotlin source code into an Intermediate Representation, or IR. This is what they call the Frontend.

Then, the LLVM compiler takes this IR as input and generates machine code for several supported CPU architecture (like the aforementioned arm64). This is the compiler backend.

Batteries included

(…) and a native implementation of the Kotlin standard library.

This last part refers to the stdlib. Kotlin multiplatform also includes a sound standard library. This allows us to get up and running with it. So when writing common code, we’ll be using things like kotlin.String instead of java.lang.String. The Compiler Backend will automatically map the IR a Kotlin.String into the correct type.

If you think about it, since the native side doesn’t have access to, for example, List, this means you would have to roll your own when targeting that platform. Not cool. So the Kotlin team built this to enable us to hit the ground running when targeting native. They also did that for things like IO, networking, and more - all of which are included.

Kotlin.String   ----jvm-----> java.lang.String
                -----js-----> string
                ---native---> KString

Long story short

The Kotlin Compiler compiles code into an Intermediate Representation with the Compiler Frontend
Then, it takes this IR and sends it to the LLVM based Compiler Backend.
Kotlin has a stdlib that covers most basic use-cases, like IO and networking which is compiled to the appropriate target representation.

Not Cross-platform, but truly Multiplatform

Write once, run anywhere (with the jve). > -java

Kotlin, probably:

Kotlin Multiplatform takes advantage of its great tooling paired with the amazing LLVM project to create true native code that you only need to write once.

It is not Cross-platform because it doesn’t need a runtime to be able to run anywhere. It is just plain native code.

Cool, right? I would say that Kotlin is able to take that direction because of its tooling/architecture and the stage that LLVM project has achieved. Let’s see what the future holds for this tech. There are certainly many challenges that we haven’t discussed here, but anyways, the foundation of the tech looks pretty interesting to me.

There’s this talk from KotlinConf18 by Kevin Galligan where he goes on to make the case for this multiplatform idea and why he thinks this is the future. I also recommend this great [post](https://blog.londogard.com/gradle/kotlin/workshop/multiplatf

A Continuous Integration approach for Kotlin projects

Sun, 24 Jan 2021 23:46:37 GMT

Continuous integration

Continuous integration is one of the words that you often hear nowadays. I remember not really getting it when I first read about it. I think that most people don’t get it right away. It’s one of those things that you don’t see the benefit before you need it, like an insurance or a sun lotion.

So, let’s try to tackle this one once again. What is Continuous Integration? Integrate what? And why should we continually do it?

As you can see, there’s clearly something missing in this term: the subject. Maybe it should be called Continuous ** CODE** Integration, but that may not sound so appealing. Now you might be wondering what this integration is. Here it refers to code merging. Let’s jump to a full definition:

Continuous Integration (CI) is a phase in the software development cycle where code from different team members or different features are integrated together. This usually involves merging code (integration), building the application and carrying out basic tests all within an ephemeral environment.

Successful Continuous integration means new code changes to an app are regularly built, tested, and merged to a shared repository. It aims to improve the quality of software, and to reduce the time taken to deliver it, by replacing the traditional practice of applying quality control after completing all development.

It’s also a solution to the problem of having too many branches of an app in development at once that might conflict with each other. The main benefits here is making sure that the code quality doesn’t decrease while keep this process small and automated.

You can check my DevOps series articles, where I go deeper about some concepts behind CI and automation. [A Primer]https://github.com/iurysza/thoughtbox/devops-primer/) The First Way

Pipelines

The pipeline is another loaded term. We lend its usage from the lean manufacturing process. The basic intuition is that each software change should go through an automated code quality verification assembly line instead of using a manual, artisan like approach to review the changes.

To achieve that we need a clear process to guarantee software quality after every change. CI pipelines can vary a lot depending on tools, type of project and many other factors, but a common flow has these steps.

Pushing to the code repository
Running Static code analysis
Unit testing
Manual code revision
Merge code on the main branch

An android library example

CI/CD tools can help a team automate their development, deployment, and testing. Some tools specifically handle the integration (CI) side, some manage development and deployment (CD), while others specialize in continuous testing or related functions.

One of the best known open source tools for CI/CD is the automation server Jenkins. Jenkins is designed to handle anything from a simple CI server to a complete CD hub.

You can even use GitHub with Jenkins, via the ghprb-plugin to set up a pipeline workflow to be triggered after a Pull Request is opened.

Let’s take a look at how this flow works on an android library project.

Our Pipeline

This is an overview of our final pipeline. Now let’s see what each stage does.

Opening a Pull Request

This is the starting point. Developers should open a pull request where they explain the context of ther code changes. Every project have their own pull request process and that should be clear to everybody contributing on to the code-base. This pull request should trigger a CI build. While that happens, other developers can contribute to the PR making questions and reviewing the code.

Merge

Your integration server will then checkout the code on your branch and merge with the main branch. If there are issues here, your build will fail and that is reported directly on the PR.

Static analysis

In this pipeline this stage does two things using the same tool:

Code Review Automation

To improve the code review process you can automate some parts of it. To do that you can use Danger. This tool “provides another logical step in your build, through this Danger can help lint your rote tasks in daily code review.” This way you can codify your team norms. E.G.: PRs with no description or with no tests are instantly reported back directly on the PR, speeding up the review process.

Actual static analysis

Static analysis is done on the code base of the application without the need to run the software. The goal here is to ensure the code doesn’t have bugs and conforms to standard formatting and styling. You might want to check this article I’ve written about it with my approach for Kotlin projects.

Ideally, developers should have fast feedback for issues during this step, so they can start fixing them ASAP. To do that, I’ve forked the danger-checkstyle-format plugin and added some changes to enable reporting issues from both ktlint and detekt directly on the PR, making developer iteration faster. Check it out.

Build

In this step we use a gradle task to build our code.

Test

Here we run all unit tests, also with gradle.

Publish reports

This step happens after the pipeline has finished. It runs jacoco jenkins plugin, a java code coverage tool to create coverage reports, then we store the test results in Jenkins. This provides a way for us to track code coverage and investigate flaky tests later on.

Bonus

You can also add some basic commands to your Jenkins pipeline, like the rebuild or rerun-tests commands. To help you work around flakiness on your project’s build or testing stages.

Conclusion

The benefits you get from having a CI pipeline in place pile up like Loan Shark interests. I strongly advise that you and your team consider creating one. If you already have one, I’d love to hear some other stages or ideas you’ve added to it. Please share in the comments =)

Publishing an obfuscated kotlin library

Sun, 27 Dec 2020 23:46:37 GMT

Being an android developer, I’ve heard about code obfuscation pretty early in my career. Since it’s preconfigured in android’s gradle plugin (it’s actually proguard under the hood), but you don’t really mess with it unless you want to. On the other hand, when you’re developing a java/kotlin library and you want to use code obfuscation, things get a little different. In this article I wanted to give a brief overview of my strategy for publishing obfuscated kotlin libraries.

Code obfuscation

If you’re not sure what that means, code obfuscation is the modification of the source code to make it unintelligible, meaning it is impossible for a third-party to understand. It’s makes for a great AppSec initiative and often takes care of the bare minimum security you can have when publishing a source code out in they wild. By making the code hard to reverse engineer, you ensure that your library’s intellectual property is guarded against security threats, unauthorized access, and discovery of application vulnerabilities.

Proguard

ProGuard is the most popular optimizer for Java bytecode and it also provides obfuscation for the names of classes, fields and methods.

Proguard is basically a command line tool that takes input jars (aars, wars, ears, zips, apks, directories) and then performs shriking, optimization, obfuscation and preverification to output a new jar.

Here we’re just interested in the obfuscation step which will rename the classes, fields, and methods using short meaningless names making it really hard to read the code.

An obfuscation strategy

Choosing our tools

There are probably many ways to do this, but since I have a mobile development background, we’ll be doing this using proguard’s gradle plugin, which will make things are little more simple, specially during the publication part. Also, I’ll use gradle’s groovy dsl, just because I like its convenience better than kotlin’s now (for gradle scripts). I may need another article to justify that, but, bear with me. This is how we integrate proguard’s gradle plugin:

buildscript {
    repositories { jcenter() }
    dependencies {
        classpath "com.guardsquare:proguard-gradle:7.0.1"
    }
}

We now need to use it to obfuscate our library’s jar. To do that we’re going to use gradle’s syntax to create a task that does just that.

task("obfuscateArtifact", type: ProGuardTask, dependsOn: jar)

Let’s dissec the task signature for a second. Here we’re declaring that this is a task of type ProguardTask and that it depends on jar task, meaning it has to run after it. This jar task comes from either the kotlin or java plugin, that you’re already using and it is responsible for packaging your application artifacts into a jar file. The reason our obfuscateArtifact task needs to run after this one can be seen in the image above. Proguard takes a jar file as the input, so we need this task to create the jar and then we’ll apply our task on top of it.

Now let’s configure the input/output of our task:

def artifactName = "$libraryArtifactId-${libraryVersion}.jar"
def obfuscatedFolder = "$buildDir/obfuscated"

injars "$buildDir/libs/$artifactName"
outjars "$obfuscatedFolder/$artifactName"

The jar task will create, by default, the outputfile file in buildDir/libs folder. By default, its name follows the pattern: artifactId-version.jar. The artifactId is defined in the publishing task that we’ll see later on. It’s useful to have those values as variables declared in a gradle.properties file.

Next properties are for handling deobfuscation. These are meant to be used when you want to understand your stacktraces.

printseeds "$obfuscatedFolder/seeds.txt"
printmapping "$obfuscatedFolder/mapping.txt"

Handling dependencies

After that, we have to make proguard understand our project’s dependencies. To do that we need to provide it with java’s standard library and ours runtime, and compile-time dependencies. A quick explanation of the last two:

The compile-time classpath contains the classes that you’ve added in order to compile your code. This is the equivalent of the classpath passed to javac (or any other java compiler).
The runtime: classpath contains the classes that are used when your application is running. That’s the classpath passed to the java executable. This is all the code your library is using when it’s running.

In proguard we can use the libraryjars property to provide these dependencies:

libraryjars "${System.getProperty('java.home')}/lib/rt.jar"
libraryjars configurations.runtime
libraryjars sourceSets.main.compileClasspath

The first line has a path pointing to a rt.jar. This stands for runtime JAR and it has all the classes from the Core Java API. This is the path you use when you’re targeting java 8 and below. The other two lines handles runtime and compile-time dependencies that we get by using gradle’s dsl.

Wrapping up

The last piece of this puzzle is proguard’s configuration file. I prefer doing that in a separated file. If you’re using intelij you can get nice syntax highlight for this file, which in my book is always a nice to have.

configuration files("$projectDir/gradle/configuration.pro")

The configuration you’re going to use is up to you, even if you don’t add anything in the config file, proguard will handle code obfuscation by default. We’ll come back to this file again.

Ok, now if you want to obfuscate your code you just need to:

run jar task
run obfuscateArtifact task

Problem solved! Or maybe not.

Automating the publication pipeline

Assuming you get the artifactId right, what we did until now may actually work for youu. The obfuscate task will trigger a build and then obfuscate the jar. But that’s not ideal. We don’t want to manually run this specific task every time. To me, the obfuscation step should be just a part of our library’s publication pipeline now. We shouldn’t have to remember about it anymore. In order to fix this we need to take a look at this so called, publication pipeline.

If you already have one, it may be just a simple publishing task, something like this:

group = "com.myLibrary.group."
version = "$myLibraryVersion"

publishing {
    publications {
        create(MavenPublication) {
            artifactId = "$myLibraryArtifactId"
            from components["java"]
            name = "My Library's name"
            description = "My Library's description"
        }
    }
}

The key here is the from components["java"] declaration. Quoting from gradle’s docs:

Components are defined by plugins and provide a simple way to define a publication for publishing. They comprise one or more artifacts as well as the appropriate metadata. For example, the java component consists of the production JAR — produced by the jar task — and its dependency information.

So, this is how our publication task gets the project’s artifacts and dependencies. Therefore, if we want to publish an obfuscated library we need to remove that line and provide our publication with the output produced by obfuscateArtifact task.

So, how can we do that? As we saw in the documentation the components["java"] is automatically defined by the java/kotlin plugin and includes all classes from the default sourceSets. If we want to define our own artifacts that we want to be published, we can use the artifact property. This is how you do it, inside the create block, add:

artifact("$buildDir/obfuscated/${libraryArtifactId}-${version}.jar") {
    builtBy obfuscateArtifact
}

Now, let’s read this backwards:

builtBy tells which task is going to build the artifact
the argument to the artifact function is the path where this task is going to output it. This path was defined earlier by the outputJar property, in our obfuscateArtifact task.

Almost there! Now, we need to provide the dependency information. This comes automatically when using the components["java", but here we need to manually provide it. Still in the create block, add:

pom.withXml {
    def dependencies = asNode().appendNode("dependencies")
    configurations.implementation.allDependencies.each { dep ->
        def depNode = dependencies.appendNode("dependency")
        depNode.appendNode("groupId", dep.group)
        depNode.appendNode("artifactId", dep.name)
        depNode.appendNode("version", dep.version)
    }
}

And that’s about it! Now, we can forget about obfuscation. We just run the publish task and get our obfuscated code published seamlessly.

Naming conflicts

If you’re using this library in another codebase that will also obfuscate its code you might get some naming conflicts due to proguard using the same name for classes/methods in that projects own obfuscation process. To fix this you can provide proguard with a custom dictionary.

//class-dictionary.txt
Lorem
ipsum
dolor
sit
amet
consectetur
adipiscing
...

You’ll then need tell proguard about it in the configuration.pro file, by adding the line.

-classobfuscationdictionary class-dictionary.txt

Now, proguard will use the words in that file to name our classes during the obfuscation step, avoiding naming conflicts.

The last thing I like to do is to create a different artifactId for the unobfuscated version of the library and to publish both versions. You can see the complete script in this gist

This diagram shows our end product.

References

Proguard home page

Continuous Kotlin Static Analysis

Sun, 20 Dec 2020 23:46:37 GMT

What is it anyway?

Static analysis is a way to find errors and other issues in a source code without actually executing it (hence, static). The process provides an understanding of the code structure and can help ensure that the code adheres to industry standards and capture issues early.

In sum, it’s a glorified spellchecker.

This is not a new idea, the first one of these tools is lint which dates back from the late 70’s and is the most common kind of static analysis tool. Most languages have a linter nowadays. The automation provided by these tools is fundamental to improve code quality, as it greatly increase code reviewing speed while also being less prone to human error.

Let machines do what they’re best at!

What can it do?

Static analysis can be broken into formal, cosmetic, design properties, error checking and predictive categories.

Formal checks if the code is correct;
Cosmetic checks if the code syncs up with style standards;
Design properties check the level of complexities;
Error checking looks for code violations;
Predictive guesses how code will behave when run.

But who needs this anyway?

In short, everyone does. Quoting John Carmack, lead developer of Wolfenstein, Doom, Quake and their sequels:

The first step is fully admitting that the code you write is riddled with errors. That is a bitter pill to swallow for a lot of people, but without it, most suggestions for change will be viewed with irritation or outright hostility. You have to want criticism of your code.

Automation is necessary. It is common to take a sort of smug satisfaction in reports of colossal failures of automatic systems, but for every failure of automation, the failures of humans are legion. Exhortations to “write better code” plans for more code reviews, pair programming, and so on just don’t cut it, especially in an environment with dozens of programmers under a lot of time pressure. The value in catching even the small subset of errors that are tractable to static analysis every single time is huge.

First Quality Gate

Quality Gate - It’s something

Static code analysis is a common DevOps practice and makes for a great first quality gate, since it provides early feedback for developers making it easier to fix issues. So, if you’ve reached this far, I’m hoping that I was able to convince you, but how can we get static analysis added to your project?

Kotlin static analysis landscape

So, in the Kotlin community two tools usually come up when talking about static analysis: ktlint and detekt.

ktlint prides itself for taking a no-configuration approach to linting. The idea is that it will always reflect kotlin’s official code style so the idea here is to be a cosmetic static analysis tool. Detekt on the other hand provides many other features like complexity reports, code smell analysis while also being extensively configurable.

Basic integration pitfalls

Both tools provide many ways to integrate in your project. You can add a plain gradle task, a gradle plugin or install the command line interface. The recommended route is via a gradle plugin. After integrating these tools in your project you can use tasks like detektCheck and ktlintCheck to run over all your project and report the issues. And that’s it, you’re done!

Yes, but also no.

The problem with the basic integration is that in real life things tend to be a little different. If you’re adding static analysis to a greenfield project, it’s usually fine to run those tasks over the whole project, but if that is not your case, you’ll probably want to avoid auto-formatting your whole project or getting 5k+ warnings every time you run one of those tasks. Warnings just turn into noise. You could use a baseline, which is a configuration that helps incremental adoption on a project, but you’re still going to stumble in some issues related to running these over your whole codebase.

A better solution

If you stop and think about it to be able to get the benefits I’ve mentioned in the beginning of this post, ideally:

You want these tasks to run fast, so you can run them often.
You want to incrementally adopt this in your codebase
And last, but certainly not least: you don’t want to keep forgetting to run these tasks.

Configuring both tools is really similar, so I’ll highlight some points that happen in both scripts. You can get detekt’s here.

So, the magic here is to get the files that were changed in our commit

There’s git command that does just that: $ git diff --name-only origin/main --relative In line 38 we call it by using groovy’s api. This will output all the change files names.
In line 44 we process the output so that we can pass the filename list to ktlint/detekt.
From lines 13 to 20 we decide whether we need to run static analysis or just ignore all files.

ktlint.gradle-gist

And that’s about it! The rest of the code (from both scripts) just handles the report output location.

Good! Almost there now!

We just need a way to make this a part of our workflow. This way we won’t forget to run these tasks. And yeah, we can use a git-hook like the pre-commit hook to do just that.

If you’re not familiar with git-hooks, you can check them here, but if you want to keep reading, just think about them as a piece of code that is going to be called when you do something in git.

The pre-commit hook will run when you try to commit something. It runs before git actually creates the commit, so the idea is to only allow a new commit when our static analysis passes successfully, effectively creating our first quality gate. Woot woot!

precommit-hook-gist

That’s it! This makes sure that every piece of code you commit has to follow the minimum quality standards that you and your team agreed upon. This is not yet a final solution as people can bypass those checks by running git commit --no-verify, but it’s a solid first step towards a better codebase.

Besides, if you think about it, these scripts can be easily integrated to a CI environment. What if it could send those warnings as inline comments in your pull request? Stay tuned for more =)

References

Carmac’s blog Static analysis definition

The First Way

Thu, 17 Dec 2020 23:46:37 GMT

The First Way of DevOps: Automating our Value Stream

Have you ever heard the phrase “it works on my machine”? It’s one of the biggest pain points for any software development team. But what if I told you that there’s a way to solve it? That’s where the First Way of DevOps comes in.

The idea behind the First Way is simple: by identifying, measuring, and improving the flow of work through our value stream, we can optimize and refine the ability to deliver value to our customers.

Let’s start with pipelines

You’re probably already using a build system to automate some parts of our value stream, but we can do better. Quoting from the Devops Handbook:

In order to create fast and reliable flow from Dev to Ops, we must ensure that we always use production-like environments at every stage of the value stream. Furthermore, these environments must be created in an automated manner, ideally on demand from scripts and configuration information stored in version control, and entirely self-serviced, without any manual work required from Operations. Our goal is to ensure that we can ** re-create** the entire production environment based on what’s in version control.

The authors go on to say that we should enable on demand creation of dev, test, and production environments so developers can run production-like environments anywhere, even on their own computers, created on demand and self-serviced.

This way, developers can run and test their code in a production-like environment as part of their daily work, providing early and constant feedback on code quality.

We’ve all been there before:

The “It works on my machine!” issue comes from the fact that in any complex system there’s just too many variables to manage. If you google The Matrix of Hell you’ll might see something like this:

Different kind of applications will have different rows and columns in this table, but you get the idea: - Multiple environments with multiple third party dependencies, create a combinatorial explosion that no sane human being can manage.

Most companies will have a test and production environments, maybe you even have a dev environment where you can safely run your app with the hottest new features. But as the project evolves, non-production environments tend to degrade. It’s hard for them to keep up with prod because of lack of automation or the discipline of constantly updating it. After a while the dev environment will have so many small things that work differently that it becomes hard to trust it. Some months down the line and people will stop believing anything that happens in the dev environment and only newcomers, still unaware of these issues, will be using it at all.

How to fix that?

The issue is that if these environments were manually created, you can’t expect people to maintain them equally. By automating their creation we enable anyone to quickly spin up a new environment on demand, enabling every developer to have their own local environment when they need one. The automation scripts will make sure that the correct configuration is loaded on each environment, avoiding hundreds or thousands of hours of unneeded headaches over a misconfiguration due to human error.

This way we’re effectively giving the development team access to production-like environments by treating your infrastructure and configuration as code, making immutable infrastructure, ensuring where we operate, develop, and test is comparable and a reliable indicator of system operation for customers — such that new value contributed by developers can easily and predictably be deployed to customers.

It’s hard, but it pays in the long run!

This is not a simple issue to solve. And you should be mindful about when and where to start the process.

In my experience, small startups or software studios tend to overlook this, because it’s too much to worry about this stuff at their product stage.

If your project doesn’t have to scale, fine. You can get away from following these ideas in smaller projects, but the engineering team has to stay vigilant for when they should stop and implement a solution.

If you get big enough the cost of not having on-demand, dev/prod environment parity will also scale, even faster than your product. And hiring more people will only make the cost increase. Hundreds or thousands of development hours go down the drain because of this issue. Bugs that are impossible to reproduce make debugging even harder, fear of changing “buggy” parts of the application causes code quality to suffer and everybody suffers.

Every team and organization is different and deciding when to start with this approach has to made in case by case scenario, but I’d argue that it’s certainly cost-effective to gradually implement this idea in the early stages of a project.

Create our single source of truth for the entire system

To ensure that we can restore production service repeatedly and predictably (and, ideally, quickly) even when catastrophic events occur, we must check in all the assets needed to recreate the system state.

That includes:

All application code and dependencies (e.g., libraries, static content, etc.)
Any script used to create database schemas, application reference data, etc.
All the environment creation tools and artifacts
Any file used to create containers (e.g., Docker or composition files)
All supporting automated tests and any manual test scripts

Ideally this should happen using immutable infrastructure where no manual changes occur - make it easier to destroy and rebuild than to configure.

Encoding knowledge and sharing it through a repository is one of the powerful mechanism for propagating knowledge. Particularly when it includes every library (internal or external), and those each have an owner responsible for ensuring it successfully passes all tests.

Wrapping up

We’ve seen how dev/prod parity begets automation. If you’re not yet embracing these concepts, well, you quite frankly should, or at least be aware of the issues that are coming your way. But fret not! We’ve all been there and those who came out alive have some very interesting ideas about how to solve it. We just have to listen to them.

A primer

Wed, 16 Dec 2020 23:46:37 GMT

Hey there, folks!

Recently, I’ve been on a mission to level up my understanding of good CI/CD practices, and you know what kept popping up? DevOps. At first, I was like “cool, so those are the people who keep our infrastructure running and manage our pipelines, right? But, man, was I mistaken!

So, I took it upon myself to get a better understanding of this whole DevOps thing. I read a book, scoured the interwebs, and even talked to some experts. And now, I want to share my newfound knowledge with you all!

The book I picked up was “The DevOps Handbook” - it’s a real treasure trove of information, written by some of the smartest folks in the DevOps community. And while I was at it, I also stumbled upon “The Twelve-Factor App” - a methodology for developing software as a service (SaaS). It’s a great read, even if you’re not in the SaaS game.

Anyway, let’s get to the good stuff! What is DevOps, you ask?

Well, according to good ol’ Wikipedia:

DevOps is a set of practices that combines software development and IT operations. It aims to shorten the system’s development life cycle and provide continuous delivery with high software quality. DevOps is complementary with Agile software development; several DevOps aspects came from Agile methodology.

In simpler terms, DevOps is like Agile, but at a macro level. When you think about all the Agile methodologies, best practices, tools, etc., the ultimate goal is to deliver value to the customer. And that’s where the concept of a “value stream” comes in.

A value stream is the process of converting a business hypothesis into a technology-enabled service that delivers value to the customer.

And the DevOps Handbook breaks down DevOps practices into three ways:

The First Way: fast flow of work from Development to Operations to the customer. To make this happen, work needs to be visible, batch sizes and intervals need to be reduced, quality needs to be built in, and everything needs to be optimized for global goals.
The Second Way: fast and constant flow of feedback from right to left. This means amplifying feedback to prevent problems from happening again and detecting and recovering from problems faster.
The Third Way: creating a generative, high-trust culture that supports experimentation and risk-taking. This leads to organizational learning, both from successes and failures, and enables the organization to learn faster than the competition.

The book goes into more detail about each of these ways, with case studies from companies like Netflix and Amazon.

But enough about the book - let’s dive in into the topics I found most interesting!

Type safe scripts

Sun, 26 Jul 2020 10:00:00 GMT

A kotlin script is a way to compile and run Kotlin code easily. Scripts are a powerful tool when you want to automate any kind of work. The most used languages for scripting tend to be python or bash/shell but, what if you could get the same task done with Kotlin? But not just that. What if you had access to any java or kotlin library in your scripts?

With Kscript you can easily do that! Besides it also offers a lot of features.

Scripts caching: Running the same script will be way faster the second time.
Maven dependencies
IntelliJ support
Bootstrap header
Check the kscript repo for the full feature set.

Trying it out

Install it with sdk man:

$ sdk install kscript

Now let's create a `.kts` file and try importing some libraries to create an embedded server:

#!/usr/bin/env kscript

@file:MavenRepository("bintray-ktor","https://dl.bintray.com/kotlin/ktor")
@file:DependsOnMaven("io.ktor:ktor-server-netty:1.2.6")

import io.ktor.application.*
import io.ktor.http.*
import io.ktor.response.*
import io.ktor.routing.*
import io.ktor.server.engine.*
import io.ktor.server.netty.*

println("starting server...")

val server = embeddedServer(Netty, port = 8080) {
  routing {
    get("/") {
      call.respondText("Hello World!", ContentType.Text.Plain)
    }
  }
}
server.start(wait = true)

Now, lets run it!

$ kscript kscript-server.kts

Open your browser on localhost:8080/

Cool, right?

But if you’re like me you probably do most of your work on the IDE. So, let’s try that now.

If you have Intelij Idea on your machine you can type

$ kscript --idea kscript-server.kts

and it will generate a Gradle Kotlin project with all dependencies declared with the @file convention and add them to a Gradle file.

#Protip: If syntax highlighting is not working properly, just right click on the build.gradle file select Import Gradle project

A Twitter-Telegram bot in Kotlin

Thu, 09 Jul 2020 11:53:37 GMT

Have you ever wondered how Twitter bots work? Can we integrate them with Telegram? Is it possible to do this only with Kotlin?

Fret not! Follow me while we uncover these mysteries!

What we’ll be creating

The idea here is pretty straight forward: A Twitter bot that downloads videos from tweets and sends them to me on Telegram.

Whenever I reply to a post on Twitter that has a video, tagging the bot’s username, the bot will download the video and send it to me on telegram. Simple enough, right? You might have even seen something similar in the wild.

Getting a little more technical

Now that we know what we want, how can we achieve this?

We’ll have an application running locally on our machines that will somehow listen to tweets that mention another user. After being notified of this event, we will need to fetch the video URL from the tweet we’re replying to, download the video and finally send it over to a telegram user using the a bot.

After reading the Twitter developer docs I’ve found they offer 3 possibilities:

(If you’re not familiar with these concepts you can check this article to understand the basics.)

Choosing our tools

Now, since we’re planning to use Kotlin to develop this, the first thing I did before diving-in was a little research (as usual). It seems we have some library options to make this task a little smoother.

Here’s what I found:

There’s an unofficial Java library for the Twitter API called Twitter4j
There’s a Java HTTP client for consuming Twitter’s realtime streaming API called HBC
There’s a wrapper for the Telegram bot API written in Kotlin called kotlin-telegram-bot

Sweet! The job just got a whole lot easier now that we don’t need to build the HTTP clients for any of these.

The only issue is that we don’t have any java/kotlin library to work with the WebHooks api, so we’ll be using the ones above.

Let’s code!

First of all we need to get the keys to play with these services.

To get Twitter keys we’ll need to create a Twitter App, you can get yours here To get Telegram’s token is just as simple as talking to the BotFather

Hiding keys, tokens and secrets

First, we need a mechanism to load keys, secrets, userIds and tokens used in the project. It is a good practice to keep our keys out of version control.

To do that, we create a json file with all the keys and put it in the resources folder.

{
  "consumerKey": "abc1234DEF1234GHI456",
  "consumerSecret": "abc1234DEF1234GHI456abc1234DEF1234GHI456",
  "token": "0123456789-abc1234DEF1234GHI456",
  "tokenSecret": "abc1234DEF1234GHI456abc1234DEF1234GHI456",
  "telegramToken": "0123456789:abc1234DEF1234GHI456-abc1234DEF1234GHI456",
  "twitterUserId": "012345678901234567890123456789",
  "telegramUserId": "01234567890123456789"
}

Then, we can use Moshi to simplify deserialization and the AuthLoader class to wrap it up.

Add this to your gradle file

implementation("com.squareup.moshi:moshi:1.9.3")
implementation("com.squareup.moshi:moshi-kotlin:1.9.3")
kapt("com.squareup.moshi:moshi-kotlin-codegen:1.9.3")

object MoshiWrapper {
    val instance: Moshi = Moshi.Builder()
        .add(KotlinJsonAdapterFactory())
        .build()

    inline fun <reified T : Any> fromJson(type: KClass<T>, json: String): T? {
        return instance.adapter(type.java).fromJson(json)
    }
}

@JsonClass(generateAdapter = true)
data class AuthData(
    val consumerKey: String,
    val consumerSecret: String,
    val token: String,
    val tokenSecret: String,
    val telegramToken: String,
    val twitterUserId: Long,
    val telegramUserId: Long
)

object AuthLoader {

    fun toOAuth(authData: AuthData): OAuth1 {
        val (consumerKey, consumerSecret, token, tokenSecret) = authData
        return OAuth1(consumerKey, consumerSecret, token, tokenSecret)
    }

    fun getAuthDataFrom(fileName: String): AuthData {
        val fileContent = getFileFromResources("/$fileName")
        return MoshiWrapper.fromJson(
            AuthData::class,
            fileContent
        )!!
    }

    private fun getFileFromResources(fileName: String): String {
        val resource = this::class.java.getResource(fileName)
        return if (resource == null) {
            throw IllegalArgumentException("file not found!")
        } else {
            File(resource.file).readText()
        }
    }
}

Filtering real time tweets

The docs tell us that we can filter tweets using some parameters like keywords and userIds

In order to do that we can use the HBC-core and HBC-Twitter4j integration.

So, we’ll be using a Twitter4jStatusClient to listen to filtered status updates. This class is a wrapper of HBC’s BasicClient . It is responsible for opening the connection with the Streaming API and to pass filter parameters. The Integration with Twitter4j removes the need to model and deserialize the json responses when receiving status updates.

First we add the dependencies

implementation("com.twitter:hbc-core:2.2.0")
implementation("com.twitter:hbc-twitter4j:2.2.0")
implementation("org.twitter4j:twitter4j-core:4.0.7")
implementation("org.slf4j:slf4j-log4j12:1.6.0") //needed for twitter4j
implementation("io.github.kotlin-telegram-bot.kotlin-telegram-bot:telegram:5.0.0")

This is the implementation I came up to simplify consuming this stream:

class FilteredStatusStream(
    private val authData: AuthData,
    private val filterParams: FilterParams,
    private val statusListener: (Status) -> Unit
) {

    companion object {
        private const val CAPACITY = 1_000
        private const val THREAD_POOL_SIZE = 4
    }

    var statusClient: Twitter4jStatusClient? = null

    fun start() {
        val (followings, keywords) = filterParams

        val filters = createStatusFilters(followings, keywords)
        val msgQueue = LinkedBlockingQueue<String>(100 * CAPACITY)
        val client = createBasicClient(filters, msgQueue, CAPACITY)

        statusClient = createStatusClient(client, msgQueue, THREAD_POOL_SIZE).apply {
            connect()
            process()
        }
    }

    fun stop() {
        statusClient?.stop()
    }

    private fun createStatusFilters(followings: List<Long>, terms: List<String>) = StatusesFilterEndpoint().apply {
        followings(followings)
        trackTerms(terms)
    }

    private fun createStatusClient(
        client: BasicClient?,
        msgQueue: LinkedBlockingQueue<String>,
        poolSize: Int
    ) = Twitter4jStatusClient(
        client,
        msgQueue,
        listOf(DefaultStatusListener(statusListener)),
        Executors.newFixedThreadPool(poolSize)
    )

    private fun createBasicClient(
        endpoint: StatusesFilterEndpoint,
        msgQueue: LinkedBlockingQueue<String>,
        eventQueueSize: Int
    ) = ClientBuilder()
        .hosts(HttpHosts(Constants.STREAM_HOST))
        .authentication(AuthLoader.toOAuth(authData))
        .endpoint(endpoint)
        .processor(StringDelimitedProcessor(msgQueue))
        .eventMessageQueue(LinkedBlockingQueue(eventQueueSize))
        .build()
}

It looks like a lot, but it is really just configuration code. We’re just hiding the complexity of setting up a Twitter steaming client. Now we can listen to the tweets from the user with the twitterUserId whenever he mentions the tweeter handle @MandaProZap just by calling:

val authData = AuthLoader.getAuthDataFrom("auth_data.json")
val filterParams = FilterParams(listOf(authData.twitterUserId),listOf("@MandaProZap"))

FilteredStatusStream(authData, filterParams) { newStatus ->
        println(newStatus)
    }.start()

Awesome, right?!😁

But… how do we get the video data? First of all, just a quick recap: We want to get the video from the tweet that we’re replying to.

This newStatus we’re getting from the stream is our own tweet. So we still need to get the parent tweet and it’s video.

Getting media from the parent tweet

Inspecting the Status returned from the stream we can see it’s properties:

private Date createdAt;
private long id;
private String text;
private int displayTextRangeStart = -1;
private int displayTextRangeEnd = -1;
private String source;
private boolean isTruncated;
private long inReplyToStatusId; //This is the one we need
.
..
...

Now we can use Twitter4j to fetch the parent tweet.

Again, we can simplify it’s usage with this class:

class TwitterClient(authData: AuthData) {

    private val twitter: Twitter = TwitterFactory().instance

    init {
        twitter.setOAuthConsumer(authData.consumerKey, authData.consumerSecret)
        val accessToken = AccessToken(authData.token, authData.tokenSecret)
        twitter.oAuthAccessToken = accessToken
    }

    fun getStatusById(statusId: Long): Status? {
        return runCatching { twitter.showStatus(statusId) }.getOrNull()
    }
}

Finally we can get the video url like this:

val twitterClient = TwitterClient(authData)
twitterClient.getStatusById(newStatus.inReplyToStatusId)
            ?.mediaEntities
            ?.mapNotNull { it.videoVariants.firstOrNull()?.url }
            ?.forEach { url -> println(url)}

Almost there now! We just need to send the video to Telegram! But wait… if we send this url… we’ll be sending just an url… not an actual video. We can do better!

Downloading the video

Since we’re using the kotlin-telegram-bot, and it has transitive dependencies to retrofit it seems like a good idea to using it to download the video file.

object FileDownloader {

    private val service = Retrofit
        .Builder()
        .baseUrl("http://localhost/")
        .build()
        .create(FileDownloadService::class.java)

    suspend fun downloadAndWriteToFile(url: String): File? = withContext(Dispatchers.IO) {
        val body = service.downloadFileFrom(url).await()
        writeToFile(body)
    }

    private fun writeToFile(responseBody: ResponseBody?): File? = runCatching {
        File("${System.currentTimeMillis()}.mp4").apply {
            createNewFile()
            Okio.buffer(Okio.sink(this)).write(responseBody!!.bytes()!!)
        }
    }.getOrNull()
}

interface FileDownloadService {
    @GET
    fun downloadFileFrom(@Url fileUrl: String): Call<ResponseBody?>
}

Here we’re creating a FileDownloadService with Retrofit and use it to download the url.

After that we write the data to a file using Okio (also comes bundled with retrofit).

You can see that we’re using coroutines to switch threads while it still looks like synchronous code.

This was achieved with a simple extension function on retrofit’s Call.

suspend fun <T> Call<T>.await(): T {
    return suspendCancellableCoroutine { continuation ->
        enqueue(object : Callback<T> {
            override fun onResponse(call: Call<T>, response: Response<T>) {
                if (response.isSuccessful && response.body() != null) {
                    continuation.resumeWith(Result.success(response.body()!!))
                } else {
                    continuation.resumeWithException(IOException(response.message()))
                }
            }

            override fun onFailure(call: Call<T>, t: Throwable) {
                continuation.resumeWithException(t)
            }
        })
    }
}

Ok, now we’re downloading the video file. We just need to send it over to Telegram!

Sending the video to Telegram

This is the easiest part thanks to the amazing kotlin-telegram-bot library.

val telegramBot = bot { token = authData.telegramToken }
telegramBot.sendVideo(authData.telegramUserId, videoFile)

Putting it all together

fun main() {
    val authData = AuthLoader.getAuthDataFrom("auth_data.json")
    val filterParams = FilterParams(listOf(authData.twitterUserId),listOf("@MandaProZap"))

    val twitterClient = TwitterClient(authData)
    val telegramBot = bot { token = authData.telegramToken }

    FilteredStatusStream(authData, filterParams) { newStatus ->
        twitterClient.getStatusById(newStatus.inReplyToStatusId)
            ?.mediaEntities
            ?.mapNotNull { it.videoVariants.firstOrNull()?.url }
            ?.forEach { url ->
                GlobalScope.launch {
                    FileDownloader.downloadAndWriteToFile(url)?.let { videoFile ->
                        telegramBot.sendVideo(authData.telegramUserId, videoFile)
                    }
                }
            }
    }.start()
}

fun main() {
val authData = AuthLoader.getAuthDataFrom("auth_data.json")
val filterParams = FilterParams(listOf(authData.twitterUserId),listOf("@MandaProZap"))

    val twitterClient = TwitterClient(authData)
    val telegramBot = bot { token = authData.telegramToken }

    FilteredStatusStream(authData, filterParams) { newStatus ->
        twitterClient.getStatusById(newStatus.inReplyToStatusId)
            ?.mediaEntities
            ?.mapNotNull { it.videoVariants.firstOrNull()?.url }
            ?.forEach { url ->
                GlobalScope.launch {
                    FileDownloader.downloadAndWriteToFile(url)?.let { videoFile ->
                        telegramBot.sendVideo(authData.telegramUserId, videoFile)
                    }
                }
            }
    }.start()
}

Summing it all up!

Yes, we can create a twitter-telegram bot with Kotlin while keeping it fairly simple.

Taking advantage of these libraries make the experience a lot smoother and adding some wrappers on top of it helps to hide the complexity and enable us to create a declarative API to let us focus on what matters.

The next step here is to implement this idea using the WebHook API, but this will have to be in a another post.

I hope you learned something along the way, I certainly did!

You can check the project on github.

Setting up MockServer

Tue, 21 Apr 2020 07:03:47 GMT

Main goals:

Simple and declarative testing environment setup.
Runs everywhere

Strategy

Set up a replicable local server.

Docker - Containerized applications (actually write once, runs anywhere)
Provide an expectations json where all the requests are mapped out.

MockServer - Create local server with complex request/response expectations through a json resource or a REST API.

Making it happen

Install Docker.
Install Docker Compose.
Make sure your docker daemon is up and running.

Create a folder for your MockServer setup.

$ mkdir /path/to/mockserver_container/ && cd /path/to/mockserver_container

Create the docker-compose.yml .

This is where you’ll setup your container for MockServer.

$ touch ./docker-compose.yml && nano ./docker-compose.yml

Paste this code snippet:

version: '2.4'
services:
mockServer:
image: mockserver/mockserver:latest
ports:

1080:1080 environment: MOCKSERVER_PROPERTY_FILE: /config/mockserver.properties MOCKSERVER_INITIALIZATION_JSON_PATH: /config/initializerJson.json volumes:

type: bind source: . target: /config


This is saying:

- Create a service named `mockserver` using the docker image from `mockserver/mockserver:latest`.

- Assign the `host port` 1080 to the `container port` 1080.
- Set the `enviroment variables` to the provided paths.

- Create a `volume` binding from the `current directory` (this file's dir) to `/config` inside the container.

Setup the Expectation Initializer JSON.

Create the initializerJson

$ touch ./initializerJson.json && nano ./initializerJson.json

Paste this code snippet:

[
  {
    "httpRequest": {
      "method": "GET",
      "path": "/api/v1/weather",
      "queryStringParameters": {
        "code": ["10969"]
      }
    },
    "httpResponse": {
      "body": "{\"wheather\": 20.5}",
      "statusCode": 200
    }
  }
]

Full documentation reference can be found here

Running your container

Now that we’ve got the plumbing out of the way, it’s time to run it.

Before running your container check if have any other containers in your environment:
```
$ docker ps -a
```
This should print an empty table.
Build your container:
Inside the folder you created the .yml type:

$ docker-compose up

Your terminal should print something like this:

Creating mockserver_mockServer_1 ... done
Attaching to mockserver_mockServer_1
mockServer_1  |
mockServer_1  | java  -Dfile.encoding=UTF-8 -jar /opt/mockserver/mockserver-netty-jar-with-dependencies.jar  -server
mockServer_1  |
mockServer_1  | 2020-04-11 15:52:30  org.mockserver.log.MockServerEventLog  INFO  creating expectation:
mockServer_1  |
mockServer_1  |   {
mockServer_1  |     "id" : "3f3962d2-9c03-4dce-95cd-2522962ceccb",
mockServer_1  |     "priority" : 0,
mockServer_1  |     "httpRequest" : {
mockServer_1  |       "method" : "GET",
mockServer_1  |       "path" : "/api/v1/weather",
mockServer_1  |       "queryStringParameters" : {
mockServer_1  |         "code" : [ "10969" ]
mockServer_1  |       }
mockServer_1  |     },
mockServer_1  |     "times" : {
mockServer_1  |       "unlimited" : true
mockServer_1  |     },
mockServer_1  |     "timeToLive" : {
mockServer_1  |       "unlimited" : true
mockServer_1  |     },
mockServer_1  |     "httpResponse" : {
mockServer_1  |       "statusCode" : 200,
mockServer_1  |       "body" : "{\"wheather\": 20.5}"
mockServer_1  |     }
mockServer_1  |   }
mockServer_1  |
mockServer_1  | 2020-04-11 15:52:30  org.mockserver.cli.Main  INFO  logger level is INFO, change using:
mockServer_1  |  - 'ConfigurationProperties.logLevel(String level)' in Java code,
mockServer_1  |  - '-logLevel' command line argument,
mockServer_1  |  - 'mockserver.logLevel' JVM system property or,
mockServer_1  |  - 'mockserver.logLevel' property value in 'mockserver.properties'
mockServer_1  | 2020-04-11 15:52:30  org.mockserver.log.MockServerEventLog  INFO  started on port: 1080

Sending requests

Open a new terminal window and enter:

To see info about your available containers enter again:

$ docker ps -a

You can now see some info about your running container.

$ curl localhost:1080/api/v1/weather\?code\=10969

You should see the response:

{"wheather": 20.5}

And thats it!

You’ve got yourself a local server where you can mock simple and complex requests and responses.

Useful links:

REST API Swagger docs