Released filepush 1.0

I am proud to announce that I launched my first open-source project. Filepush is a self-hosted (for now) service that allows you to upload and download files from your app. It gives the user one endpoint where he can send files. The app takes care of important stuff like security and storage.

Multiple filesystem support

Filepush is built around a Filesystem. It can handle multiple adapters. It ships with a LocalAdapter and an AWS S3Adapter. Support for other filesystems can be easily added.

Image support

Filepush supports the resizing of images. When uploading a new image, you can pass a width and height and it will resize the image before storing it on the filesystem.

Security first

Filepush requires for every action a valid API key. Credentials are never stored by the application. Filepush is built on the shoulders of giants, like NestJS.

What’s ahead

More features are added:

  • Configurable caching for the download endpoint. Why hit AWS or the filesystem when the file has not changed?
  • Configurable rate limitting for uploads and downloads.
  • Resizing support for downloads.

Let me know what you think or follow the development at github.

Event-Driven Architecture

A simple event emitter

So I have some actual experience with Event-driven architecture myself but this week I followed an interesting webinar from ThoughtWorks about this specific topic. Here are my main takeaways.

It is nothing new

Events are nothing new. Events are everywhere and life itself is actually just a set of events.

It is valuable to the business

It is a really natural way to describe your business. For example, a customer adds an item, drops an item, and adds again a new item to their basket. Having these events stored, gives great insight into a customers behaviour.

It adds structure to the system

Events are a functional way to design workflows with immutable events. They provided a mechanism to easily extend other workflows in a loosely coupled way. Events also let you focus on what valuable information your domain has and what information is relevant for other domains.

Event-sourcing

A great example of event-sourcing is Git. Instead of storing the latest state, Git stores events that contain what changed. You can easily go back to a previous state or browse through all the changes that happened over time.

Event orchestration

Event orchestration is when you have a single service that is telling the other services what to do and when to do it. This is like the conductor in an orchestra who shows the musicians what to do.

Event choreography

Event choreography on the other hand is more decentralized. The services communicate with each other and know what they have to do in relation to the other. This does not require supervision and instructions.

Event streaming

Event streaming is taking it to the next level where you have real-time event streams and processes, and smaller systems to do other things like filtering streams, produce new streams, etc. What an event stream actually is, was not really clear after the webinar. But after doing some research I found out that an event stream has to do with persisting events and can be used to get a view of the events. For example, one can have two event streams, one which contains every change made in chronological order and one which provides the latest view of an entity.

Challenges of an Event-Driven Architecture

Every architecture has challenges and working with events requires a little bit of a different mindset.

Handling failure

Things will go wrong. Let’s say you store something in a database and then publish an event. Either of these processes can fail. What can we do in this case? Did subscribers already process our event? And what to do with missing events?

The first question you should ask yourself is, is it okay to miss the event or is it crucial and do we put in the extra work?

One pattern you can use is the outbox pattern. In this pattern you store transactions in the database and in a separate process you read from the database, scan for updates, and publish events in a reliable way to a broker.

Some frameworks also offer this out of the box. So service A publishes an event and service B is offline. When service B comes back online, it starts reading events from where it stopped.

Observability and debugging

In a complex system, it is crucial to be able to debug the system and have a way to see what’s happening inside. The developers from Thoughtworks came with a few options.

  • Have correlation ids on all events.
  • Have standardized log formats.
  • Have a good tool for searching logs with a correlation id.

Versioning

Business changes often and our software as well. Here are a few things to consider.

  • Have contracts in place. Contracts say what an event looks like.
  • Have good communication between teams, so that you know when things change.
  • Communicate with all the consumers about the change of the event update.
  • To avoid breaking or having to update other systems, you can also decide to publish using another schema.

Slicing the work

Generally, we want stories to center around the business value that they add. Because events often require changes in many different services, we could end up with monster user stories.

To avoid this we can slice the work in smaller parts and still keep the ability to show progress to the stakeholders. We can for example publish to the event broker without having the consumers. This gives us the opportunity to share progress and show what the change would be. The next step will be to actually build the consumers.

Legacy components

Event-driven architecture can be a useful tool to refactor legacy systems. Focus on getting the data out of the legacy system. You can do this by reading from the database and publishing events into a new event stream to integrate with the old system.

Commands

Commands tell us that we have to do something. Often people use passive-aggressive events. An example: the teapot is empty. Some subscriber picks up this event and fills the pot. What you actually want to say is: fill up the pot.

Events are just notifications, letting the consumers know something has happened. No one needs to do anything about it. You can do something about it, but I could just be shouting into the wind.

When we are talking more about the state, this is the state and this has to happen, then we are looking more at commands. By using passive-aggressive events we are losing the choreography of events.

Final note

When deciding if you are going to use techniques like CQRS or Event-sourcing, you do not have to follow them to the letter. It is ok to just pick what you need and what works for you and just drop the parts that are not useful.

It was an interesting webinar, where I learned some new terms and techniques that can start my further research into some topics related to Event-driven architecture. For example, having the right names for event orchestration, choreography, etc is really steering me in the right direction.

My MobX findings

For one of my React Native projects, I started to use MobX as a state management tool. I have used Redux before, but I have to say that MobX feels a lot simpler in terms of updating and managing state via code.

Mobx is using an observer pattern to signal state changes and on every change, it will send a message to every subscriber that is observing some observable variables.

Changing the state is as easy as just changing a property on your model. All subscribers of the property will be updated. This reactive model can be used in any JS framework or even in plain JS.

An example store

class GameStore {
  @observable
  loading: boolean = false;
  @observable
  games: Game[] = [];

  @action.bound
  async addGame(game: Game) {
    runInAction(() => {
      this.games.unshift(game);
    });
  }

  @action.bound
  removeGame(game: Game) {
    runInAction(() => {
      this.games = this.games.filter(g => g !== game);
    });
  }
}

This is the simplified version of my GameStore. It has a few actions and some observable properties.

Best practices for using MobX

While using MobX I follow to the following (best) practices:

Use many specific stores

In Redux you are used to having only one store for all your application state. MobX suggests to have as many stores as you want. So in my application, I have a GameStore, a NoteStore, a LocationStore, and a UserStore. A store is nothing more than a class holding reference to the piece of state that you want to manage.

So my GameStore has a simple Array of Game models and exposes some actions like addGame() or removeGame() that just push or pop the game on the internal Array. MobX takes care of updating all the observers.

Change state only via actions on the model

Because changing state in MobX is so easy, you could do that easily in your view components. However, this is, in general, a bad idea as it becomes hard to track where in your application the state is changed. A better practice is to define actions on your model. Actions are simple methods on your model that define behavior. So when I want to change the name of a user, I could simply set the name property of my User model in the Profile view. But instead, I will define a method called changeName() on my User model and just call that. This way we have all the state changes nicely together in our model.

A pattern I am just figuring out is to pass the GameStore to my Game model so that I can just call remove() on my model. This remove action will then tell the GameStore to remove itself from the list of games. This makes my code really easy to read.

About autorun

MobX makes it easy to react to state changes in your model by using reactions. Autorun is one of them. Reactions make it easy to observe some state and react to it. At the moment I am working on using autorun to implement a sync mechanism with a remote server. So when a game is changed, it is updated in the local database and in the API.

Conclusion

I really like this reactive model of MobX. It takes a lot less code to change the state compared to Redux and it’s reactions make it really easy to react to changes. I am sure there will be good reasons for when to use Redux or when to use MobX and that MobX probably has some drawbacks as well. I did not find any yet, but I will be looking out for them in the future when I come further in the development of the project.

https://mobx.js.org/README.html

Actions and reducers for a PHP developer.

Redux is a predictable state container for Javascript apps. It manages state via actions and reducers.

Actions

The state cannot be updated with setters. This is done so that your code will not update the state arbitrarily, causing bugs. To change the state you have to dispatch an action. An action is a Javascript object that describes what happened. For example:

{
   type: 'ADD_TODO',
   text: 'Go to the swimming pool'
}

Enforcing that the state only changes via actions, gives us a clear understanding of what’s going on in the app.

Reducers

To tie state and actions together, you have to write a function, called a reducer. It’s just a function that takes the state and an action, it applies the changes from the action and returns the new state.

export default (state = [], action) => {
    if (action.type === 'ADD_TODO') {
        return state.concat({text: action.text, completed: false});
    }

    return state;
}

The action contains only the minimum information about the new change. The reducer is responsible for making that change correctly. The reducer should manage other logic like whether the item is completed. It would be a bad idea to add the completed flag to the action. Keep actions as simple as possible.

View components in your app are connected to the Redux state. When the state is updated, the component needs some logic to update itself.

Xdebug issues on macOS

Recently I spend a lot of time trying to configure Xdebug on my local machine. I am working on macOS Mojave with PhpStorm.

The standard installation guide from JetBrains is a really good start and works 99% of the time. But this time I could not get it to work at all.

The problem

Whenever I started the debug button, the debugger would never hit any breakpoint in my code. I was getting the following error:

Connection with Xdebug 2.6.1 was not established.
Xdebug 2.6.1 extension is installed. Check configuration options. 

The solution

After troubleshooting the issue and a lot of Googling, I checked which other processes were listening to the port 9000.

sudo lsof -iTCP -sTCP:LISTEN -n -P

It turned out the php-fpm was listening to port 9000 and was receiving the debug connections from my editor. It turned out that I had installed php with php-fpm via brew and brew had configure the php service as autostart.

Stopping the brew php service fixed the issue.

brew services stop php           

Happy debugging!

What makes a good Unit test?

According to Clean Code, the three things that make a clean unit test are readability, readability, and readability.

Readability in tests is probably even more important than in production code. So what makes a test readable? The same standards that make other code readable: clarity, simplicity, and density of expression.

Clean tests will contain a lot of your domain language. That means it will not focus on the technical terms of your testing framework, but rather on the business case the code is implementing.

F.I.R.S.T.

Clean tests should follow this acronym.

Fast

Fast means fast. Tests should run fast because you want to run them as frequently as possible. If your tests are slow, you will not run them often enough and the feedback loop will be longer. Lately, I have been working on a codebase that has around 5000 tests and performs around 8300 assertions in less than 3 seconds. I think that’s a good example of fast tests.

Independent

Tests should not depend on each other. It should be clear why a test fails and where you should go to fix the problem. If tests depend on each other, one failing test can cause a lot of depending tests to fail which makes diagnosis more difficult.

Repeatable

You should be able to run your tests in all environment. In your production environment, in your QA environment, and on your local machine. They should not depend on a network connection. If your tests are not repeatable, you will have an excuse for when they fail. Plus you will not be able to run them when the environment is not available.

Self-Validating

A self-validating test is a test with a boolean output. It either fails or passes. You should know it directly. You should not have to go through a log file to see which tests failed.

Timely

Tests should be written just before the production code. If you write your tests after the production code, the change that you will write code that’s hard to test is much bigger. Writing your tests before the production code always results in code that’s easy to test. And code that’s easy to test is also often clean, clear and simple.

How to test protected and private methods

Testing public methods with any testing framework is easy because we have access to the public interface of the test subject. Unfortunately, we don’t have access to the protected and private methods of our test subject. They are hidden to us (the client) for a reason. They are implementation details the class does not want to leak to potential clients.

Generally speaking, it is a bad idea to test private and protected methods. They are not part of the public interface and they are usually executed via the public methods of a class. So when we test the public methods, they implicitly test private and protected methods.

In TDD we are testing the behavior of a class, not the specific methods. So when we make sure we test all the things a class can do, we can rest assured that the private and protected methods are tested.

If you find yourself having large private methods which need their own test, there is probably something wrong with your design. Always aim for designing small classes that do one thing.

If you still think this is nonsense and you really want to test those methods, you can take a look here on how to do that with PHPUnit.

 

How to handle time in your tests?

Handling time in Unit Tests

Often we face scenario’s where a process we are modelling is depending on time. Take for example an expiring coupon. A coupon is no longer valid when it has expired. In PHP we could implement this as follows:

class Coupon
{
    /**
     * @var \DateTimeImmutable
     */
    private $expiryDate;

    public function __construct(\DateTimeImmutable $expiryDate)
    {
        $this->expiryDate = $expiryDate;
    }

    public function isValid(): bool
    {
        $diffInSeconds = $this->expiryDate->getTimestamp() - time();
        return $diffInSeconds > 0;
    }
}

class CouponTest extends TestCase
{
    public function testANotExpiredCouponIsValid(): void
    {
        $coupon = new Coupon(new \DateTimeImmutable('2018-10-28T07:14:49+02:00'));
        self::assertTrue($coupon->isValid());
    }

    public function testAnExpiredCouponIsNotValid(): void
    {
        $coupon = new Coupon(new \DateTimeImmutable('2017-10-28T07:14:49+02:00'));
        self::assertFalse($coupon->isValid());
    }
}

The problem

When we run the test for a valid coupon someday after 2018-10-28 (the date it will expire), it will actually fail because the coupon has actually expired. That’s not what we want because we need to test coupons with an expiry date in the future. So we need a way to fix that.

One way to resolve this issue would be to just change our test and pass in a date that’s 500 years in the future. It will fail again in 500 years but it will not be our problem anymore.

A better way to resolve this is to actually pass the current time as an argument to the isValid() method. Look at the new implementation:

class Coupon
{
    /**
     * @var \DateTimeImmutable
     */
    private $expiryDate;

    public function __construct(\DateTimeImmutable $expiryDate)
    {
        $this->expiryDate = $expiryDate;
    }

    public function isValid(\DateTimeImmutable $now): bool
    {
        $diffInSeconds = $this->expiryDate->getTimestamp() - $now->getTimestamp();
        return $diffInSeconds > 0;
    }
}

class CouponTest extends TestCase
{
    public function testANotExpiredCouponIsValid(): void
    {
        $coupon = new Coupon(new \DateTimeImmutable('2018-10-28T07:14:49+02:00'));
        self::assertTrue($coupon->isValid(new \DateTimeImmutable('2018-10-01T00:00:00+02:00')));
    }

    public function testAnExpiredCouponIsNotValid(): void
    {
        $coupon = new Coupon(new \DateTimeImmutable('2017-10-28T07:14:49+02:00'));
        self::assertFalse($coupon->isValid(new \DateTimeImmutable('2018-10-01T00:00:00+02:00')));
    }
}

That’s better. Now we have frozen the current time and are no longer depend on the real system type. Our tests will always pass.