RxJava Observable tranformation: concatMap() vs flatMap()

After a while I decided that was time to get back for some writing. As you may know at @SoundCloud we do a strong use of the reactive approach, but to be honest, I am not here to talk about RxJava itself because there are great articles out there to read about it (here and here) and great people to follow as well such as Ben Christesen, Matthias Käppler and many others.
I also consider myself a ‘newbie’ in reactive programming and now I am at that stage where you start seeing the benefits of this approach and want to make every single object reactive, which is very dangerous, so if you are in the same level as me, just keep an eye on it, and use it wherever makes sense, you are advised.
Let’s get started with the article then…

Observable transformation

There are times where you have an Observable which you are subscribed to and you want to transform the results (remember that everything is a stream in Reactive Programming).
When it comes to observable transformation, the values from the sequences we consume are not always in the format or shape we need or each value needs to be expanded either into a richer object or into more values, so we can do this by applying a function to each element returned by your observable which will convert all of the items emitted by it into Observables and merge the result. Do not worry if you do not understand yet (it took me a while to think in reactive), we will see an example in a bit.

The problem

I was retrieving a set of values from the database and applying a function to each of them that was suppose to both transform them in other objects asynchronously and also preserve their order. Last step was to convert them into a list needed by the UI to display the results. The behavior I had was not the expected one and here is why: I was using Observable.flatMap() which does not preserve the order of the elements.

A simple example

Let me put a simple example to demonstrate the mentioned behavior. Let’s say we have an Observable emitting a set of Integers and we want to calculate the square of each of those values:

Here our DataManager class has a method that returns an Observable which emits numbers from 2 to 10. Then we want to calculate the square of those values so here is our function to apply to each of them:

This will take an Integer as entry, will generate an Observable<Integer>, merge them and emit the results. As you can see we are using a call to dataManager.squareOf() method which is asynchronous (for demonstration purpose) and looks something like this:

Of course this works, but not as expected (at least the way I wanted), the order of the elements is not preserved (logcat output):

flatMap_logcat

Observable flatMap() vs concatMap()

Both methods look pretty much the same, but there is a difference: operator usage when merging the final results. Here is some stuff from the official documentation:

flatMap

The flatMap() method creates a new Observable by applying a function that you supply to each item emitted by the original Observable, where that function is itself an Observable that emits items, and then merges the results of that function applied to every item emitted by the original Observable, emitting these merged results. Note that flatMap() may interleave the items emitted by the Observables that result from transforming the items emitted by the source Observable. If it is important that these items not be interleaved, you can instead use the similar concatMap() method.

concatMap

As you can see, the two functions are very similar and the subtle difference is how the output is created (after the mapping function is applied). flatMap() uses merge operator while concatMap() uses concat operator meaning that the last one cares about the order of the elements, so keep an eye on that if you need ordering :).

Merge operator

Combine multiple Observables into one.
merge

Concat operator

Concatenate two or more Observables sequentially.
concat

Problem solved

Observable concatMap() for the salvation! The problem was easily solved by just switching to a concatMap() method. I know you may argue why I did not read the documentation first, which is very well explained by the way (kudos to the RxJava contributors!!!), but sometimes we are lazy or that is the last place we look into. Here is a picture with the final results and some test I did (you can find the sample code below):

final_results_concatMap

References

That is my two cents and hope it helps. As always here is the sample code of the sample app and other useful information that worth read.

  1. Source code: https://github.com/android10/Android-ReactiveProgramming
  2. Functional Reactive Programming on Android With RxJava
  3. Grokking RxJava
  4. Top 7 Tips for RxJava on Android
  5. Mastering Observables
  6. React Conference London

Remember that any feedback is very welcome, such as better ways of addressing this problem or any issue you may find.

Architecting Android…The clean way?

Over the last months and after having friendly discussions at Tuenti with colleagues like @pedro_g_s and @flipper83 (by the way 2 badass of android development), I have decided that was a good time to write an article about architecting android applications.
The purpose of it is to show you a little approach I had in mind in the last few months plus all the stuff I have learnt from investigating and implementing it.

Getting Started

We know that writing quality software is hard and complex: It is not only about satisfying requirements, also should be robust, maintainable, testable, and flexible enough to adapt to growth and change. This is where “the clean architecture” comes up and could be a good approach for using when developing any software application.
The idea is simple: clean architecture stands for a group of practices that produce systems that are:

  • Independent of Frameworks.
  • Testable.
  • Independent of UI.
  • Independent of Database.
  • Independent of any external agency.

It is not a must to use only 4 circles (as you can see in the picture), because they are only schematic but you should take into consideration the Dependency Rule: source code dependencies can only point inwards and nothing in an inner circle can know anything at all about something in an outer circle.

Here is some vocabulary that is relevant for getting familiar and understanding this approach in a better way:

  • Entities: These are the business objects of the application.
  • Use Cases: These use cases orchestrate the flow of data to and from the entities. Are also called Interactors.
  • Interface Adapters: This set of adapters convert data from the format most convenient for the use cases and entities. Presenters and Controllers belong here.
  • Frameworks and Drivers: This is where all the details go: UI, tools, frameworks, etc.

For a better and more extensive explanation, refer to this article or this video.

Our Scenario

I will start with a simple scenario to get things going: simply create an small app that shows a list of friends or users retrieved from the cloud and, when clicking any of them, a new screen will be opened showing more details of that user.
I leave you a video so you can have the big picture of what I’m talking about:

Android Architecture

The objective is the separation of concerns by keeping the business rules not knowing anything at all about the outside world, thus, they can can be tested without any dependency to any external element.
To achieve this, my proposal is about breaking up the project into 3 different layers, in which each one has its own purpose and works separately from the others.
It is worth mentioning that each layer uses its own data model so this independence can be reached (you will see in code that a data mapper is needed in order to accomplish data transformation, a price to be paid if you do not want to cross the use of your models over the entire application).
Here is an schema so you can see how it looks like:

clean_architecture_android

NOTE: I did not use any external library (except gson for parsing json data and junit, mockito, robolectric and espresso for testing). The reason was because it made the example a bit more clear. Anyway do not hesitate to add ORMs for storing disk data or any dependency injection framework or whatever tool or library you are familiar with, that could make your life easier. (Remember that reinventing the wheel is not a good practice).

Presentation Layer

Is here, where the logic related with views and animations happens. It uses no more than a Model View Presenter (MVP from now on), but you can use any other pattern like MVC or MVVM. I will not get into details on it, but here fragments and activities are only views, there is no logic inside them other than UI logic, and this is where all the rendering stuff takes place.
Presenters in this layer are composed with interactors (use cases) that perform the job in a new thread outside the android UI thread, and come back using a callback with the data that will be rendered in the view.

clean_architecture_mvp

If you want a cool example about Effective Android UI that uses MVP and MVVM, take a look at what my friend Pedro Gómez has done.

Domain Layer

Business rules here: all the logic happens in this layer. Regarding the android project, you will see all the interactors (use cases) implementations here as well.
This layer is a pure java module without any android dependencies. All the external components use interfaces when connecting to the business objects.

clean_architecture_domain

Data Layer

All data needed for the application comes from this layer through a UserRepository implementation (the interface is in the domain layer) that uses a Repository Pattern with a strategy that, through a factory, picks different data sources depending on certain conditions.
For instance, when getting a user by id, the disk cache data source will be selected if the user already exists in cache, otherwise the cloud will be queried to retrieve the data and later save it to the disk cache.
The idea behind all this is that the data origin is transparent for the client, which does not care if the data is coming from memory, disk or the cloud, the only truth is that the data will arrive and will be got.

clean_architecture_data

NOTE: In terms of code I have implemented a very simple and primitive disk cache using the file system and android preferences, it was for learning purpose. Remember again that you SHOULD NOT REINVENT THE WHEEL if there are existing libraries that perform these jobs in a better way.

Error Handling

This is always a topic for discussion and could be great if you share your solutions here.
My strategy was to use callbacks, thus, if something happens in the data repository for example, the callback has 2 methods onResponse() and onError(). The last one encapsulates exceptions in a wrapper class called “ErrorBundle”: This approach brings some difficulties because there is a chains of callbacks one after the other until the error goes to the presentation layer to be rendered. Code readability could be a bit compromised.
On the other side, I could have implemented an event bus system that throws events if something wrong happens but this kind of solution is like using a GOTO, and, in my opinion, sometimes you can get lost when you’re subscribed to several events if you do not control that closely.

Testing

Regarding testing, I opted for several solutions depending on the layer:

  • Presentation Layer: used android instrumentation and espresso for integration and functional testing.
  • Domain Layer: JUnit plus mockito for unit tests was used here.
  • Data Layer: Robolectric (since this layer has android dependencies) plus junit plus mockito for integration and unit tests.

Show me the code

I know that you may be wondering where is the code, right? Well here is the github link where you will find what I have done. About the folder structure, something to mention, is that the different layers are represented using modules:

  • presentation: It is an android module that represents the presentation layer.
  • domain: A java module without android dependencies.
  • data: An android module from where all the data is retrieved.
  • data-test: Tests for the data layer. Due to some limitations when using Robolectric I had to use it in a separate java module.

Conclusion

As Uncle Bob says, “Architecture is About Intent, not Frameworks” and I totally agree with this statement. Of course there are a lot of different ways of doing things (different implementations) and I’m pretty sure that you (like me) face a lot of challenges every day, but by using this technique, you make sure that your application will be:

  • Easy to maintain.
  • Easy to test.
  • Very cohesive.
  • Decoupled.

As a conclusion I strongly recommend you give it a try and see and share your results and experiences, as well as any other approach you’ve found that works better: we do know that continuous improvement is always a very good and positive thing.
I hope you have found this article useful and, as always, any feedback is very welcome.

Links and Resources

  1. Source code: https://github.com/android10/Android-CleanArchitecture
  2. The clean architecture by Uncle Bob
  3. Architecture is about Intent, not Frameworks
  4. Model View Presenter
  5. Repository Pattern by Martin Fowler
  6. Android Design Patterns Presentation

Aspect Oriented Programming in Android

Aspect-oriented programming entails breaking down program logic into “concerns” (cohesive areas of functionality). This means, that with AOP, we can add executable blocks to some source code without explicitly changing it. This programming paradigm pretends that “cross-cutting concerns” (the logic needed at many places, without a single class where to implement them) should be implemented once and injected it many times into those places.

Code injection becomes a very important part of AOP: it is useful for dealing with the mentioned “concerns” that cut across the whole application, such as logging or performance monitoring, and, using it in this way, should not be something used rarely as you might think, quite the contrary; every programmer will come into a situation where this ability of injecting code, could prevent a lot of pain and frustration.

AOP is a paradigm that has been with us for many years, and I found it very useful to apply it to Android. After some investigation I consider that we can get a lot of advantages and very useful stuff when making use of it.

Terminology (Mini glossary)

Before we get started, let’s have a look at some vocabulary that we should take into account:

  • Cross-cutting concerns: Even though most classes in an OO model will perform a single, specific function, they often share common, secondary requirements with other classes. For example, we may want to add logging to classes within the data-access layer and also to classes in the UI layer whenever a thread enters or exits a method. Even though each class has a very different primary functionality, the code needed to perform the secondary functionality is often identical.
  • Advice: The code that is injected to a class file. Typically we talk about before, after, and around advices, which are executed before, after, or instead of a target method. It’s possible to make also other changes than injecting code into methods, e.g. adding fields or interfaces to a class.
  • Joint point: A particular point in a program that might be the target of code injection, e.g. a method call or method entry.
  • Pointcut: An expression which tells a code injection tool where to inject a particular piece of code, i.e. to which joint points to apply a particular advice. It could select only a single such point – e.g. execution of a single method – or many similar points – e.g. executions of all methods marked with a custom annotation such as @DebugTrace.
  • Aspect: The combination of the pointcut and the advice is termed an aspect. For instance, we add a logging aspect to our application by defining a pointcut and giving the correct advice.
  • Weaving: The process of injecting code – advices – into the target places – joint points.

This picture summarizes a bit a few of this concepts:

Aspect Oriented Programming

So…where and when can we apply AOP?

Some examples of cross-cutting concerns are:

  • Logging
  • Persistance
  • Performance monitoring
  • Data Validation
  • Caching
  • Many others

And in relation with “when the magic happens”, the code can be injected at different points in time:

  • At run-time: your code has to explicitly ask for the enhanced code, e.g. by using a Dynamic Proxy (this is arguably not true code injection). Anyway here is an example I created for testing it.
  • At load-time: the modification are performed when the target classes are being loaded by Dalvik or ART. Byte-code or Dex-code weaving.
  • At build-time: you add an extra step to your build process to modify the compiled classes before packaging and deploying your application. Source-code weaving.

Depending on the situation you will be choosing one or the other :).

Tools and Libraries

There are a few tools and libraries out there that help us use AOP:

  • AspectJ: A seamless aspect-oriented extension to the Javatm programming language (works with Android).
  • Javassist for Android: An android porting of the very well known java library Javassist for bytecode manipulation.
  • DexMaker: A Java-language API for doing compile time or runtime code generation targeting the Dalvik VM.
  • ASMDEX: A bytecode manipulation library as ASM but it handles the DEX bytecode used by Android executables.

Why AspectJ?

For our example below I have chosen AspectJ for the following reasons:

  • Very powerful.
  • Supports build time and load time code injection.
  • Easy to use.

Example

Let’s say we want to measure the performance of a method (how long takes its execution). For doing this we want to mark our method with a @DebugTrace annotation and want to see the results using the logcat transparently without having to write code in each annotated method. Our approach is to use AspectJ for this purpose.
This is what is gonna happen under the hood:

  • The annotation will be processed in a new step we are adding to our compilation fase.
  • Necessary boilerplate code will be generated and injected in the annotated method.

I have to say here that while I was researching I found Jake Wharton’s Hugo Library that it is suppose to do the same, so I refactored my code and looks similar to it, although mine is a more primitive and simpler version (I have learnt a lot by looking at its code by the way).

AspectWeaving

Project structure

We will break up our sample application into 2 modules, the first will contain our android app and the second will be an android library that will make use of AspectJ library for weaving (code injection).
You may be wondering why we are using an android library module instead of a pure java library: the reason is that for AspectJ to work on Android we have to make use of some hooks when compiling our app and this is only possible using the android-library gradle plugin. (Do not worry about this yet, cause I will be giving some more details later).

Creating our annotation

We first create our Java annotation. This annotation will be persisted in the class (RetentionPolicy.CLASS) file and we will be able to annotate any constructor or method with it (ElementType.CONSTRUCTOR and ElementType.METHOD). So our DebugTrace.java file will look like this:

Our StopWatch for performance monitoring

I have created a simple class that encapsulates time start/stop. Here is our StopWatch.java class:

DebugLog Class

I just decorated the “android.util.Log” cause my first idea was to add some more functionality to the android log. Here it is:

Our Aspect

Now it is time to create our aspect class (TraceAspect.java) that will be in charge of managing the annotation processing and source-code weaving.

Some important points to mention here:

  • We declare 2 public methods with 2 pointcuts that will filter all methods and constructors annotated with “org.android10.gintonic.annotation.DebugTrace”.
  • We define the “weaveJointPoint(ProceedingJoinPoint joinPoint)” annotated with “@Around” which means that our code injection will happen around the annotated method with “@DebugTrace”.
  • The line “Object result = joinPoint.proceed();” is where the annotated method execution happens, so before this, is where we start our StopWatch to start measuring time, and after that, we stop it.
  • Finally we build our message and print it using the Android Log.

Making AspectJ work with Android

Now everything should be working, but, if we compile our sample, we will see that nothing happens.
The reason is that we have to use the AspectJ compiler (ajc, an extension of the java compiler) to weave all classes that are affected by an aspect. That’s why, as I mention before, we need to add some extra configuration to our gradle build task to make it work.
This is how our build.gradle looks like:

Our test method

Let’s use our cool aspect annotation by adding it to a test method. I have created a method inside the main activity for testing purpose. Let’s have a look at it:

Executing our application

We build and install our app on an android device/emulator by executing the gradle command:

If we open the logcat and execute our sample, we will see a debug log with:

Our first android application using AOP worked!
You can use the Dex Dump android application (from your phone), or any any other reverse engineering tool for decompiling the apk and see the source code generated and injected.

Recap

So to recap and summarize:

  • We have had a taste of Aspect Oriented programming paradigm.
  • Code Injection becomes a very important part of this approach (AOP).
  • AspectJ is a very powerful and easy to use tool for source code weaving in Android applications.
  • We have created a working example using AOP capabilities.

Conclusion

Aspect Oriented Programming is very powerful. Using it the right way, you can avoid duplicating a lot of code when you have “cross-cutting concerns” in your Android apps, like performance monitoring, as we have seen in our example. I do encourage you to give it a try, you will find it very useful.
I hope you like the article, the purpose of it was to share what I’ve learnt so far, so feel free to comment and give feedback, or even better, fork the code and play a bit with it.
I’m sure we can add very interesting stuff to our AOP module in the sample app. Ideas are very welcome ;).

Source Code

You can check the example app here (using AspectJ): https://github.com/android10/Android-AOPExample
Also I have another AOP example for Java (you can use if for Android as well) using a Dynamic Proxy: https://github.com/android10/DynamicProxy_Java_Sample

Resources

Unit testing asynchronous methods with Mockito

After promising (and not keeping my promise) that I would be writing and maintaining my blog, here I go again (3289423987 attempt). But lest’s forget about that…
So in this occasion I wanted to write about Mockito…yes, this mocking framework that is a ‘must have’ when writing your unit tests ;).

Introduction

This is article assumes that you know what a unit test is and why you should write tests.
Also I strongly recommend this famous article from Martin Fowler that talks about test doubles, it is a must read to understand about test doubles.

Common case scenario

Sometimes we have to test methods that use callbacks, meaning that they are asynchronous by definition. These methods are not easy to test and using Thread.sleep(milliseconds) method to wait for the response is not a good practice and can convert your tests in non-deterministic ones (I have seen this many times to be honest).
So how do we do this? Mockito to the rescue!

Let’s put an example

Suppose we have a class called DummyCaller that implements a DummyCallback and has a method doSomethingAsynchronously() that delegates its functionality to a collaborator of the the class called DummyCollaborator that has a doSomethingAsynchronously(DummyCallback callback) as well, but receives a callback as a parameter (in this case our DummyCallback), so this methods creates a new thread to run his job and then gives us a result when is done.
Here is the code to understand this scenario in a better way:

Creating our test class

We have 2 options to test our asynchronous method but first we will create our test class DummyCollaboratorCallerTest (for convention we just add Test at the end of the class so this becomes part of its name).

So here we are using MockitoAnotations to initialize both Mock and ArgumentCaptor, but don’t worry about them yet, cause this is what we will be seeing next.
The only thing to take into account here is that both mock and class under test are being initialized before each test is executed in the setUp() method (using the @Before annotation).
Remember that for unit testing all the collaborators for a CUT (class under test) must be test doubles.
Let’s take a look at our 2 test solutions.

Setting up an answer for our callback

This is our test case using a doAnswer() for stubbing a method with a generic Answer. This means that since we need a callback to return immediately (synchronously), we generate an answer so when the method under test is called, the callback will be executed right away with the data we tell it to return.
Finally we call our real method and verify state and interaction.

Using an ArgumentCaptor

Our second option is to use an ArgumentCaptor. Here we treat our callback asynchronously: we capture the DummyCallback object passed to our DummyCollaborator using an ArgumentCaptor.
Finally we can make all our assertions at the test method level and call onSuccess() when we want to verify state and interaction.

Conclusion

The main difference between both solutions is that when using DoAnswer() we are creating an anonymous inner class, and casting (in an unsafe way) the elements from invocation.getArguments()[n] to the data type we want, but in case we modify our parameters the test will ‘fail fast’ letting know that something has happened. On the other side, when using ArgumentCaptor we probably have more control cause we can call the callbacks in the order we want in case we need it.
As interest in unit testing, this is a common case that sometimes we do not know how deal with, so in my experience using both solutions has helped me to have a robust approach when having to test asynchronous methods.
I hope you find this article useful, and as always, remember that any feedback is very welcome, as well as other ways of doing this. Of course if you have any doubt do not hesitate to contact me.

Code Sample

Here is the link where you can find this example and others. Most of them are related with Java and Android because this comes from a talk I gave a couple of months ago.
The presentation is in english but the video is in spanish (sorry for those who do not understand my argentinian accent…haha…BTW I will try to upload an english version as soon as possible…)

Further Reading

I highly recommend to take a look at Mockito documentation to have a better understanding of the framework. The documentation is very clear and has great examples. See you!

NFC on Android

Hi all! I’m so proud that finally I’ve created my new blog. Since this is my 12372983 attempt for having a development blog, I can only promise you that will put all my effort to keep it up to date. So welcome!

And now that this is my first post I just wanted to share with you a talk about Near Field Communication I gave last year which was requested by a lot of colleagues. Here it is guys! (take into account that the video is in spanish, anyway I promise to explain some examples and use cases soon).

Enjoy and feel free to comment and give very useful feedback.

You can find the source code of the example application of both the presentation and video here: https://github.com/android10/nfc_android_sample

Here is the presentation (english):

 

Here is the video (spanish):

See you!!!