Blog - Inside Programium | Software Development, Product and Digital Transformation Insights

The Scarcity of Engineering Craftsmanship

Vasilis Christopoulos — Sat, 03 Oct 2026 18:51:07 GMT

To me, engineering craftsmanship is the care that goes into deciding what to build and owning every line of the result. The current market is trying to prove that AI made code cheap to produce. Judgment and the risk of accountability still cost what they always did, because judgment only forms when you put real effort into creating. Skip the effort, and the craft gets scarce.

I've been trying to find the time to put my thoughts on engineering craftsmanship into a post: what I believe it really is, and what it means to me.

Last October I gave a talk with the same title at Patras Tech Talk (slides). This post picks up where the talk left off, one year later.

I am old enough to remember how software engineers treated software before the mass adoption of Large Language Models (LLMs). It was a period when people were truly intrigued by new ideas, and the limits of hardware pushed them to find creative ways to move technology forward without compromising quality.

It was a time when smart, passionate and dedicated engineers were necessary for a product to truly succeed.

They invented new frameworks, new paradigms and new languages. They spent countless hours, across many different mediums, discussing and co-authoring Request for Comments (RFC) documents, the specs that laid the foundation of what we call technology today.

From small companies to big ones, many individuals, scattered across every level of their organizations, shared one common goal: to design and deliver efficient, well-crafted software they could be proud of, software that either improved people's lives or inspired other engineers to grow their craft.

But today, that time is long gone.

Generative chaos

AI has been adopted explosively across many markets. I don't mean traditional artificial intelligence, but the huge demand for, and supply of, LLMs and LLM wrappers. As a result, we live in a time where theories like the dead internet theory are becoming more and more plausible.

Most content is now either generated or augmented by AI. People rely heavily on LLM tools to automate their work, to do things they couldn't do before, or to "improve" the quality of their work.

Companies found new metrics to track, mostly vanity ones like tokens used and lines of code (produced by AI, this time). That pressure has forced employees at every level to rely on AI to some extent. It is now hard to tell what is generated and what isn't, and that has changed how people interact with each other.

The vibe era and the Dunning-Kruger effect

In the talk I called this the vibe era. Low-code, no-code and now AI-code dropped the barrier of entry to writing code to almost nothing. We can now produce a lot of code really fast, often with limited understanding of what it does, and the push for speed has never been higher.

We got lazy. We always wanted to be lazy, and LLMs looked like a way out of hard work. Or maybe they are a way to amplify our Dunning-Kruger effect, the bias where people with limited skill in an area overestimate their ability in it. A tool that answers every question with confidence makes it very easy to feel competent without being competent.

Pushing the responsibility to the recipient

With so much of our work shifted to LLMs, a hard battle starts: reviewing the work.

In the past, writing code took time. There was a ceiling on how much code one developer could write in a day. Writing several variations of an implementation just to decide which one worked best was expensive.

So people spent more time thinking, planning and designing their idea on paper. By the time they started implementing, the result was usually good, because so much of the thinking had happened early. They also knew their implementation details well, because they had designed them.

But the moment the effort needed to produce quick variations dropped, surprisingly, so did the thinking. People spent more time prompting for more variations and delegated every part of their work they could. They refactored heavily, iterated aggressively and kept generating until something looked like it worked.

The thinking moved to whoever has to review the result. A pull request that took minutes to generate can take far longer to review properly, and the reviewer now carries the responsibility the author skipped.

The culture decay

The damage reaches past the code. We are slowly but steadily losing mentorship, architecture and intent: the senior engineer explaining why, the design that holds a system together, the reason behind each decision. AI is taking away our effort, and effort is where most of that was learned.

AI also amplifies chaos built on unstable foundations. A model will build on top of a bad design as readily as on a good one, and much faster than any team could. Craftsmanship gets replaced by convenience.

One year later

Since I gave that talk, LLMs have improved quite a bit. In the same year, software engineers have grown huge dependencies on them, to the point where many are unable to do their own work well without them. The effort moved to the model, and the skill followed it.

I watched it happen in my own work

When I first tried to integrate AI into my own workflow, I watched the quality of my own work drop, and it dropped together with the effort I was putting in. Less effort to produce the work meant less thinking about the work, and the result showed it.

That convinced me the problem was what I had stopped doing.

Judgment is earned by effort

Judgment only shows up when you put enough effort into creating something. When the effort is reduced, judgment is allowed to be poor, and nothing in the process stops it.

That is why I think craftsmanship is becoming scarce. Producing code got faster. Whether it got cheaper is yet to be proven, except maybe in the short term. Knowing whether the code is right still costs effort, the same effort AI makes easy to skip. The habits we rely on to keep code maintainable on fast-moving projects depend on that effort too.

How we work now

None of this means we shouldn't use AI. We use it every day at Programium, on our own code and on the custom software and AI features we build for clients. What changed is where the thinking happens:

We think the idea through on paper or in a doc before anyone writes a prompt.
Nothing ships unless we can explain how and why it works.
Tests define what correct means, whoever or whatever wrote the code. More on that in automated testing as a first-class citizen.
Core logic, and anything on unfamiliar ground, gets written by hand on purpose, so the skill stays sharp.

As many others have said before me, the awesome part of using tools like LLMs is that they reduce the effort you spend on things that don't matter in your role: the things you shouldn't be trying to be the best at, not the ones you should. Use them to become better, not worse.

The call to craft

This is how I closed the talk:

Don't be lazy. Take ownership.
Keep writing software.
Keep trying, failing, and trying again.
Take pride in your work.
All effort, no matter how small, moves you forward.
AI might look convenient, but it's far from perfect.

With discipline comes experience. With experience comes clarity. With clarity comes quality. With quality comes pride. And with all of the above comes craftsmanship.

Building for Growth: A Guide to Application Scalability and Performance

Vasilis Christopoulos — Wed, 06 Aug 2025 08:00:04 GMT

Application scalability starts with measuring the application, and adding servers comes after. Profile it, load test it until it breaks, and find out what every dependency costs you. Then watch latency, traffic, errors and saturation in production, and fix the bottleneck you can see. On Compot, a sub-250 ms p95 at launch was a target we set and measured against.

This guide walks through how we do that, in the order we do it.

Measure before you scale

Effective scalability begins with a data-driven understanding of your application's operational needs. Accurately forecasting resource requirements is essential for preventing both over-provisioning, which incurs unnecessary costs, and under-provisioning, which risks system failure.

Adding servers is the most expensive way to find out where your application is slow. The cause is often a handful of operations doing far more work than they need to: a file processed inside the request, a query that scans a whole table, two threads fighting over one lock. Measuring first tells you which one you have and how much capacity you really need. It also stops you paying for capacity you'll never use, which is the quieter way to get scaling wrong.

Profile the application

Start with the profiler for your stack: JProfiler or the IntelliJ profiler for Java, cProfile for Python, pprof for Go. Run the application and walk through every user flow while it records. Upload and download files, trigger the heavy processing, submit the forms.

You're looking for two things. The first is where time and CPU go, which tells you where optimisation will pay off: a thread race, a slow query, a file pipeline that needs re-engineering. The second is memory that gets allocated and never released. A leak shows up as memory that keeps climbing the longer you use the application, and it's far cheaper to find on a laptop than in production.

A memory dump in the IntelliJ profiler

Load test to find the breaking point

Profiling covers one user at a time, so the next step is a load test. We use k6; Apache JMeter and Gatling do the same job. Raise the load step by step and record two numbers: the requests per second a single instance handles comfortably, and the point where latency or errors start to climb. Those two numbers are your baseline, and every scaling decision afterwards gets measured against them.

Most of these tools can record a session in the browser and turn it into a replayable script. Record the real user flows once, then edit the script or replay it as often as you need.

load testing report using the tool k6

Map your dependencies

Your application is only as fast as the slowest dependency on its critical path: the database, the cache, a third-party API. If a payment provider takes 400 ms to answer, a 250 ms p95 is off the table for that flow, however fast your own code is. Map every dependency and measure what it costs per request. Do it before launch, while the architecture can still change.

Availability works the same way. A service that needs three dependencies to answer a request can't be more available than the three of them together, and AWS shows the maths. Check the SLAs of what you depend on before you promise one of your own.

A dependency map in Azure Application Insights

Monitor the four golden signals

Google's SRE book names four golden signals worth watching on every user-facing service, and we build our dashboards around them:

Latency: how long a request takes. Watch p95 and p99 as well as the average, because the average hides the slow requests, and those are the ones users notice under load.
Traffic: how much demand reaches the system, usually in requests per second.
Errors: the share of requests that fail. A sudden rise is usually the first sign of a fault.
Saturation: how close CPU, memory and disk I/O are to their limits. High saturation means a service is running out of headroom.

We use Axiom for metrics and the dashboards. Putting all four signals for every service on one screen is what makes patterns visible. A drop in requests per second next to a spike in CPU, for example, usually points to a blocking operation.

Trace requests end to end

Metrics tell you something is slow. Tracing tells you where. One request can pass through a frontend, an API, two internal services and a database, and the delay can sit in any of them. Distributed tracing follows the request through each hop and shows how long it spent in each one.

For open-source tracing, look at Jaeger, Zipkin and Grafana Tempo. Commercial APM platforms such as Datadog, New Relic and Sentry bundle tracing with metrics and logs.

How we troubleshoot a traffic surge

When traffic spikes, the order of the checks matters more than speed. We go through the same steps every time:

Dashboards. The four golden signals show what changed and when: more traffic, slower responses, more errors, or a resource at its limit.
Logs. Query the central log store for errors and warnings around the moment it started, and look for a pattern: one endpoint, one client, one service. Ours go to Axiom.
Recent deployments. A fresh code or configuration change is one of the most common causes of sudden instability. Check what shipped in the hours before.
The bottleneck. Traces and metrics narrow it to one component, and its profile tells you what kind of problem you have:
- CPU-bound: inefficient code, or not enough compute.
- Memory-bound: a leak, or instances that are too small.
- I/O-bound: database contention or slow disks. Query optimisation or a missing index usually fixes it before new hardware does.

Not every surge is good news.

On a large platform where I was technical lead, a sudden spike in traffic turned out to be a DDoS attack on our customer. We traced the attack to its origins and blocked them at the firewall. The attackers rotated to new IP addresses, so blocking them one at a time stopped working. We moved to rate limiting per origin instead: every source got a request budget, and each new address the attackers rotated in hit the same ceiling as the last one. Blocking dealt with the first wave. The rate limit is what held when the attack changed shape, and it costs very little to set up before you need it.

Which fix for which bottleneck

Once you know where the bottleneck is, the fix is usually one of a few well-known patterns. The useful question is what each one costs you.

Symptom	Fix	Trade-off
Servers busy delivering images, CSS and JavaScript	A CDN serves static assets from edge locations close to the user	Cache invalidation becomes part of every release
A predictable peak, such as a planned campaign	Manual scaling: more instances (horizontal) or a bigger machine (vertical)	Someone has to watch and act, but costs stay predictable
Unpredictable load	Autoscaling on CPU or request count, with Kubernetes or AWS Auto Scaling Groups	Without caps, costs scale with traffic, attacks included. Set limits before you turn it on
One instance overloaded while others sit idle	A load balancer spreads requests across instances	One more component to configure and monitor
The same expensive reads, over and over	A cache such as Redis or Memcached	Stale data, and an invalidation strategy to design
One client, or one attacker, flooding an API	Rate limiting per client or origin	Legitimate bursts can get throttled, so the limits need tuning
A failing downstream service dragging its callers down	A circuit breaker stops calls until the service recovers	Callers need a fallback for when the circuit is open

Scale for the load you can forecast

Building for traffic you may never get has a daily cost: more moving parts, slower changes, and a system that is harder to hire for. We design for the load we can forecast, leave seams where growth would go, and measure before we tune. That's how Compot launched with a sub-250 ms p95, a target we set up front and measured against.

Scalability also depends on the codebase. The habits that keep code maintainable on fast-moving projects are the same ones that let you change a system safely when the load changes. If you're planning for growth, see how we approach custom software development.

Automated Testing as a First-Class Citizen

Vasilis Christopoulos — Thu, 17 Jul 2025 17:52:11 GMT

Introduction

In the race to deliver software faster, it's tempting to skip steps and tell yourself you'll do them later, in some milestone further down the roadmap. Testing is often one of them. Skipping tests does save time in the short term, but testing is the foundation of reliable, maintainable and scalable systems.

At Programium, automated testing is a first-class citizen. It's part of everything we build, from small APIs to enterprise platforms.

In this post, we'll share why testing gets that status and how it pays off for our clients.

Building Confidence at Speed

Modern development moves fast. Features ship quickly, architectures evolve, and teams often work across time zones.

Without automated testing, every change introduces uncertainty:

"Will this break something unexpected?"
"Is the integration still working?"
"Can we safely deploy to production?"

Automated tests give us the confidence to say yes to these questions, and allow us to deliver faster without compromising quality.

Catching Problems Early (When They're Cheaper to Fix)

Bugs caught in production are costly. Not just in time, but in trust.

Automated testing helps catch issues early in the development lifecycle:

Unit tests ensure individual components behave as expected.
Integration tests confirm systems work together properly.
End-to-end tests validate entire workflows from a user perspective.

By investing upfront, we prevent problems from becoming expensive disasters later.

Supporting Continuous Delivery and Agile Teams

Our clients expect agility. Whether we’re building a custom application or expanding an existing one, continuous delivery allows us to iterate quickly.

Automated tests are the backbone of this process:

Every commit is tested automatically.
Deployments become routine, not risky events.
Engineers focus on solving business problems, not firefighting regressions.

This leads to faster feedback loops and happier clients.

Enabling Cleaner Code and Smarter Design

There’s a hidden benefit to automated testing: it encourages better code architecture.

When you write testable code, you naturally:

Keep functions small and focused.
Use dependency injection for flexibility.
Avoid tightly coupled components.

The result is a system that is easier to understand, extend and maintain, which is what a long-lived product needs. Testing is one of several habits behind that; the rest are in writing maintainable code for fast-moving projects.

Keeping Coverage High with AI Coding Agents

AI coding agents are useful for testing once a codebase has structure. Before that, they mostly produce noise.

We learned that the hard way. Our first AI-written tests mostly mocked files they touched, so all they proved was that the mocks worked. Every file came out in a different style. Coverage went up, and our confidence stayed where it was.

The fix was in the codebase. Three things make AI-written tests worth keeping:

Interfaces and API contracts are typed, so integration tests run against the real contract instead of a mock of it.
Every module is laid out the same way, so one good test is a template for the next.
Engineers write the first tests in a module by hand, and the agent follows their shape.

The split of work stays fixed. An engineer decides what correct means by naming the cases and behaviours a test must cover. The agent writes the test bodies, backfills coverage on code that has none, and suggests edge cases we missed. Every test gets reviewed before it merges, like any other code.

Unit and integration tests carry the breadth: the combinations, the edge cases, the error paths. That keeps our end-to-end suite thin, a few Playwright flows over the user journeys that matter most, written by the agent under the same rules.

On one of our current projects, this setup keeps line coverage above 86% with little overhead, because the structure does most of the work before the agent starts. Tests still define what correct means, whoever or whatever wrote the code. We wrote more about where the thinking stays in the scarcity of engineering craftsmanship.

If you’re thinking about scaling your platform, modernizing legacy systems, or ensuring your next release goes smoothly, let’s talk!

Mail us now

Protecting Long-Term Product Health

Your software is a living system.

As features are added and team members change, tests become your safety net.

They act as living documentation, ensuring new engineers can make changes confidently, and that your platform can evolve without fear of breaking mission-critical functionality.

Conclusion: Quality Shouldn't Be Optional

At Programium, we believe quality is not an add-on, it’s built in.

Automated testing helps us deliver software that is:

Faster to deploy
Safer to change
Easier to maintain
Trusted by our clients

It’s how we uphold our commitment to engineering excellence, and why our clients trust us with their most important projects.

Flow at Your Fingertips: The Quiet Power of Muscle Memory in Software Development

Artur Rozalski — Tue, 06 May 2025 15:08:23 GMT

As software developers, we're often drawn to the flashiest tools, newest frameworks and the most performant languages. We convince ourselves that staying on the bleeding edge means staying ahead. I know I did. For a long time, I believed I was achieving maximum productivity simply because I was using the latest and greatest tools.

Somewhere along the way, I began to notice something. My hands were flying all over the keyboard. My movements weren’t deliberate, they were chaotic. I'd reach for the mouse, then back to the keyboard, then fumble through shortcuts I hadn't truly internalized or look endlessly through the context menus. When pressure mounted or fatigue set in, I’d start making the same small mistakes over and over again.

That’s when it hit me: my workflow wasn't inefficient because of the tools I was using, it was inefficient because I hadn’t trained my body to use them well.

Muscle Memory: The Developer’s Silent Ally

Managing cognitive load is a well known aspect in programming, keeping track of logic, naming things, understanding business rules. But there’s a hidden load most developers carry without realizing: the physical friction of how we interact with our tools.

Muscle memory, the unconscious, trained motion of your fingers, is rarely discussed in developer circles, yet it plays a massive role in shaping how fluidly we work. Think of a pianist who has to glance at the keys every few notes. Or a gamer who can’t quite hit the right button under pressure. Their technical ability isn’t the issue, it’s their lack of automaticity.

When you have to think about how to navigate, open files, switch panes, or reach for the right key, your attention is no longer fully on the code. It’s split. And over a day, a week, or a career, that cost adds up.

Typing: The Foundation Everyone Skips

This all starts with typing, and typing properly.

It’s ironic that we spend our entire careers at the keyboard, yet most developers never formally learn how to type. Not just fast, but accurately. Ergonomically. Without looking.

It might seem like a small thing, but poor typing technique creates a subtle stream of friction: inconsistent rhythm, constant corrections, breaking flow to check key positions. Over time, these micro-delays interrupt not just speed, but thought.

Once I committed to retraining my typing, with deliberate repetition and patience, I noticed something shift. My fingers weren’t guessing anymore. They were moving with my mind, not just after it.

Keyboard-First Workflows: More Than a Trend

There's a reason developers who master keyboard-centric tools like Vim, Tmux, or modal editors seem so fast. It’s not just that the tools are efficient (though they are). It’s that using them well forces you to build habits.

Habits that rely on repetition. Habits that reward consistency. Habits that, over time, become muscle memory.

And here’s the truth: those habits are transferable. Whether you're in an IDE, a browser-based editor, or your terminal, if your hands know what to do without you thinking about it, you're not just faster, you're calmer. You're not chasing the cursor or scanning the screen for buttons. You’re staying in the problem space.

The Slow Phase Is the Important Phase

The difficult part of all this? You get slower before you get faster.

When I first started retraining how I typed, or forcing myself to stick to keyboard shortcuts and avoid the mouse, I felt like I was moving through molasses. I was less productive. I made more mistakes. I doubted the point of it all.

But that discomfort wasn’t failure, it was unlearning. It was the messy transition from unconscious inefficiency to conscious, intentional action.

And like any physical skill, whether it's playing an instrument, doing martial arts, or learning to drive, there’s a breakthrough that only comes on the other side of consistent, sometimes frustrating repetition.

Less Friction, More Flow

At some point, the friction faded. I stopped thinking about my hands. Navigation became intuitive. My typing smoothed out. And that quiet mastery had ripple effects I didn’t anticipate.

I could stay in flow longer. I felt less tired at the end of the day. I made fewer typos, fewer wrong turns, and fewer decisions driven by interface fatigue.

The work didn’t get easier. I just got less in my own way.

Final Thought: Mastery Is Physical, Too

There’s a quiet kind of power in mastering the physical side of development. It’s not glamorous. It won’t show up on a résumé. But it pays off in every session, every sprint, and every late-night debug session.

So if you’ve ever found yourself in your favourite IDE, reaching for the mouse out of habit, making the same navigation mistakes, or watching your hands dance aimlessly across the keyboard, maybe it’s time to slow down.

Not to do less. But to do better.

Train your hands like you train your mind. The results will speak for themselves, without you even needing to think about them.

Writing Maintainable Code for Fast-Moving Projects: Our Approach

Vasilis Christopoulos — Wed, 16 Apr 2025 16:02:48 GMT

In fast-moving projects, speed is everything.

Whether you're validating an MVP, scaling a successful product, or racing to meet market demands, the temptation is always there: "Ship now, fix later."

But later rarely comes. A common misconception in the software industry is that moving fast is doing everything in a rush.

At Programium, we believe that speed and maintainability aren't mutually exclusive.

In fact, they go hand-in-hand.

In this post, we’ll share how we balance delivery velocity with sustainable code quality, and why this balance is key to long-term success.

Maintainability as a Design Principle

Before we write a single line of code, we ask:

"How easy should it be to change this in six months?"

Maintainable code is:

Easy to read and understand.
Predictable in behaviour.
Modular, so changes are localised.
Well-documented, with clear intent.

We treat maintainability as an investment, not an afterthought.

Every decision from project structure to naming conventions is guided by this principle.

Clean Code Matters

Clean code might feel "slower" at first, but it’s actually the opposite over time.

Our practices include:

Descriptive naming: so code reads like a story.
Small functions: Each function does one thing well.
Consistent formatting: Tools like Prettier and Checkstyle enforce clarity.
Eliminating dead code: If it's not used, it’s gone.

💡 Tip: We treat our codebase like a garden: continuous small cleanups prevent wild overgrowth.

Automated Testing: as a Safety Net

We rarely compromise on automated testing.

Why? Because tests give you the confidence to move fast without breaking things. Untested code moves the work to manual validation and manual regression testing, which costs the speed you skipped the tests to gain.

Our typical setup includes:

Unit tests for fast feedback.
Integration tests for system behaviour.
End-to-end tests for critical user flows.

This layered approach means we catch issues early, when they’re cheap and easy to fix.

"If tests are green you should be good to go, if they are not you better fix them first."

Using The Right Architecture

Even in MVPs, we apply architecture thinking:

Domain-Driven Design (where applicable) to reflect business logic cleanly.
Separation of concerns to avoid tangled systems.
Architecture Decision Records (ADRs) to document why decisions were made.

We don’t "over-engineer", but we leave clear paths for future growth.

We keep the second-system effect in mind: the pull to load a new system with everything the last one left out. So we keep the design simple and make the decisions the project needs now, not the ones it might need someday.

All Code Is Reviewed

In fast-paced projects, code reviews can feel like bottlenecks.

We make them accelerators instead.

Our approach:

Mandatory peer reviews, used to mentor each other, not only to gatekeep.
Focus on high-impact feedback (readability, structure, logic).
Automated pull request decoration that focuses on the basics (e.g. linting, test coverage, Sonar scans, vulnerability scans) so reviews focus on real improvements of the business logic and code quality.

This aims to keep our velocity high while maintaining quality.

Conclusion: Sustainable Speed Wins

Our experience taught us that fast growth doesn’t have to mean technical chaos.

By embedding maintainability into our processes, from architecture decisions to daily coding habits, we help our clients scale confidently, knowing their software can evolve as fast as their business does.

If you're building fast and want to avoid growing pains later, see how we approach custom software development, or let's talk.

Building Software That Moves Businesses Forward

Vasilis Christopoulos — Mon, 07 Apr 2025 09:14:05 GMT

Welcome to Inside Programium, our space to share ideas, engineering stories, and lessons from the systems we build.

At Programium, we started with a simple belief: software should solve real problems, scale with your ambitions, and be built with care.

Over the years, we’ve seen how the right software choices can transform businesses. But too often, companies are stuck between off-the-shelf tools that don’t fit, or over-engineered solutions that create more problems than they solve. We built Programium to be a different kind of partner — one that delivers custom, pragmatic systems with a focus on real-world impact.

Our work spans:

Custom Software Development — Frontend, backend, web apps, APIs, and developer tools.
Distributed Systems — Cloud-native, scalable infrastructure for modern businesses.
Web3 Development — Full-stack solutions for EVM-based platforms.
AI & Automation — Smarter workflows and intelligent systems that unlock efficiency.

Through this blog, we’ll share:

How we design systems that scale
Insights from our engineering practices
Honest lessons from the field — what works, and what doesn’t
Explorations in Web3 and AI
And, of course, occasional experiments and things we’re just plain curious about.

If you’re building something ambitious, we hope this space brings you ideas, clarity, and inspiration.

Thanks for reading — and welcome to Programium.