Sunday, January 27, 2013

Top 10 Technologies I Want My Software Engineers to Know In 2013

Last year I blogged about the top ten technologies I wanted my engineers at Berico Technologies to learn in 2012.  The post was so popular, I've decided to make it a tradition.  In addition to providing my new top ten list, I want to provide a little retrospective on the technologies that made the list and those that fell off this year to explain why I've increased or devaluated their importance.

Without further adieu, I present:

Top 10 Technologies I want my Software Engineers to Know in 2013.


1.  The Modern Client-Side Web Development Stack
2.  Node.js
3.  Modern Messaging Frameworks
4.  Hadoop + HBase
5.  Clojure + Leiningen
6.  Twitter Storm
7.  Lucene/Solr
8.  Graph Databases
9.  A Platform as a Service Offering
10. Apache ZooKeeper

Rationale.


1.  The Modern Client-Side Web Development Stack:  Let's face it, the faster growing "sector" of the Enterprise stack is the Client-Side.  We have not only seen an explosion of new client-side Application/MVC frameworks, but also the adoption of a whole new process for composing and building web applications.  And we should emphasize the term "application"!  Modern websites are as complex as their desktop predecessors (perhaps even more), as browsers continue to be come more capable, and user expectations grow.

Unfortunately, no single framework or library is worthy of being in the top ten on its own.  I will say, however, that the combination of these frameworks represents a fundamental shift in our community away from Server-side MVC and GWT-like frameworks towards hyper-capable clients.

There are a number of frameworks/libraries of note:

 

-  Application/MVC Frameworks:  Ember.js, Batman.js, Angular.js, and Knockout.js.

Please note that I've left Backbone and Sammy.js off the list in part because I think their popularity are starting to wane as the newer breed of frameworks offer more capabilities.  Another framework generating a lot of excitement is Meteor.js, which attempts to provide a seamless stack (client-server-database) with a simple API.  I have only glanced at Meteor's documentation, but it looks promising.


-  Application Composition:  Require.js.


-  Build Tools:  Mimosa.js, Yeoman.

Mimosa.js is a new build/deployment framework developed by one of my friends David Bashford.  While it hasn't gotten a lot of attention from the community, our company uses it for nearly all of it's projects and the number of people forking/starting it on Github seems to double each month.
      

-  Languages:  CoffeeScript, ClojureScript, TypeScript.

At this point, I don't think there is any dominant JavaScript alternative.  My opinion is that a team (not an individual) should adopt the one that best fits their collective personality.

-  Visualization:  D3.

D3 scored us a number of big wins in the visualization department last year.  Our company is also beginning to cultivate a number of D3 proficient engineers.  I see us continuing to spread the D3 goodness around our team indefinitely.
2 Node.js:  In my mind, last year Node.js (well Express and other web frameworks) proved it was a viable alternative to Ruby on Rails.  I believe this will be the year in which Node.js unseats Rails.  Numerous PaaS vendors began supporting the platform last year (OpenShift, Cloud Foundry, Heroku, Azure) and I'm sure many more will follow suit (where are you Google App Engine?).

More importantly, Node.js IS NOT JUST A WEB FRAMEWORK!  Applications and libraries in a number of different domains are being written on top of the platform, and I can't wait to see what comes next (a NoSQL database?, a distributed processing framework?). 



3.  Modern Messaging Frameworks:  one of the top ten technologies last year was RabbitMQ.  Let start by saying that my respect, admiration and appreciation of RabbitMQ has only grown during 2012 (there is no other AMQP broker in my book).  This year, I'm broadening the scope by including "modern messaging frameworks" as one of the top ten competencies in 2013.

The shift to include more messaging technologies came from the realization that there is a need in modern cloud architectures to have multiple messaging platforms.  AMQP, and specifically RabbitMQ, is representative of the "reliable messaging" tier of brokers (like JMS, MSMQ, TIBCO).  However, there is a distinct need for "high-throughput" and "batching" message brokers that sacrifice security and reliability for performance.  The dominant brokers I'm looking at in each tier are:

Reliable Messaging:  AMQP/RabbitMQ
High-Throughput:  ZeroMQ
Batching:  Apache Kafka

  

4.  Hadoop + HBase:  If it isn't obvious that the Hadoop ecosystem is incredibly important right now, you are living under a rock.  Almost every RDBMS vendor is embracing Hadoop with tie-ins to their database.  Many, including Oracle and SQL Server, will allow you to ship SQL tables to Hadoop, execute a MapReduce job, and pull the results back into the database.  As for HBase, it remains critically important as an operationally-proven petabyte scale NoSQL database.

This year, I don't expect any revolutionary changes to occur on the Hadoop + HBase platform, though I think the software will continue to mature as vendors adopt the platform and create commercial extensions.  The thing to watch for is the frameworks built on top of Hadoop, like Impala (released by Cloudera last year).  There's also the potential for the start of an Apache project attempting to clone "Spanner", Google's "Globally-Distributed Database".

 

5.  Clojure + Leiningen:  This will probably be the most upsetting item on this list, particularly from the Scala crowd.  Last year I wanted my engineers to learn a JVM language.  This year, I only want them to learn one: Clojure.  The decision to buck all other languages from this list came from the collective frustration our engineers faced last year using both Scala and JRuby (and our great experiences with Clojure).

So what happened with Scala and JRuby?

I think the success of JavaScript last year may have deemphasized Ruby's importance, which in turn deemphasized JRuby.  I personally like the Ruby language, but I constantly found myself struggling to find a good use-case for its application.  Another problem I think Java developers have with Ruby is needing to learn Ruby Gems in addition to dealing with Java dependency management.  Frankly, not too many people wanted to learn JRuby, favoring Scala or Clojure.

Scala, on the other hand, is a language I think many of us learned to hate last year.  On the surface, it appears to be a decent language.  However, the more we learned it's syntax, the more we realized how needlessly complex and obtuse it could be.  In the hands of a really great engineer, it can be very elegant.  In the hands of everyone else, it can be difficult to read and understand.  More importantly, we didn't particularly enjoy Scala's mechanisms for interoperability with Java (it seemed strange in many cases).

The big eye-opener was Clojure.  Once getting past all of those parentheses, I think many people realized a how simple and elegant the language was.  Interop with Java is extremely clean.  I personally found my self up and running in a couple of hours, using all of my favorite Java libraries without any issues.  This year we will continue to evangelize the language, pushing people and projects toward the platform.

6.  Twitter Storm:  Storm is doing for real-time computing, what Hadoop did for batch processing.  I think we are just now seeing the Storm rise in popularity and I expect it will become even more popular as developers start building frameworks on top of it.  Our company, Berico Technologies, already has plans for building an data ingestion framework and a real time data manipulation framework on top of it.  I would imagine many other developers are actively doing the same as I write this.
  

7.  Lucene/Solr:  This entry was at risk of falling off the top ten list, but stayed on the list primarily because of the promise of Solr Cloud this year.  Search is no longer a feature, but rather a requirement for many applications, and Lucene-based indexes will remain the dominant implementation.
 

8.  Graph Databases:  As graphs become more mainstream, I think engineers are starting to realize the value of databases optimized for joins.  The clear leader in this market is Neo4j, but I think it will start to see some competition from highly-scalable, distributed implementations like Titan.

More importantly, there has been a trend towards polyglot architectures (combining a graph database [for joins] with a relational database [for queries]).  Frameworks like Spring Data Neo4j simplify the development of these systems by providing annotation-driven Data Access Layer functionality similar to Hibernate and JPA.

In terms of usability, a framework to highlight is Tinkerpop's Blueprints, an abstraction for graph databases.  In addition to Blueprints, Tinkerpop has also written a number of complimentary frameworks (Pipes, Gremlin, Frames, Furnance, and Rexter) enhancing the usability of your graph.
 
  

9.  A Platform as a Service Offering:  Platforms like RedHat OpenShift, VMWare Cloud Foundry, Windows Azure, Heroku and Google App Engine are the way of the future in my opinion.  Being able to compose an application and not worry about installing and maintaining the services it relies upon is quite liberating for an engineer.  There is a clear cost and time savings in employing such solutions.  More importantly, I want my engineers thinking about PaaS design and it's implications for applications so they can build and/or employ them for our customers who don't have the luxury of using a commercial offering.

10.  Apache Zookeeper:  This probably seems like an odd addition to the list.  ZooKeeper is a framework that enables reliable communication and coordination between nodes in a cluster.  The framework is used to perform centralized configuration, key-value storage, distributed queues, leadership election, synchronization, failure detection and recovery, etc.  ZooKeeper is a key component in number of important distributed platforms like Apache Hadoop, Apache Solr, Apache HBase, Twitter Storm, Neo4j HA, and Apache Kafka to name a few.  Simply put ZooKeeper is underpinning of a number of important applications and knowledge of it couldn't hurt.  More importantly, there's nothing the prevents you from building your own distributed application with ZooKeeper.

Favorable mention.

Finally, I wanted to give a "favorable mention" to a number of frameworks that didn't make the list:

- Spring Framework:  Still the trusted backbone of all of our Java applications, and I just as dependent on it as I was three years ago.
- Redis: We're using it successfully on a couple of projects, the only downside is the lack of security which prevents us from using it on all of our projects.
- MongoDB:  We use MongoDB on a couple of projects.  It's certainly proven itself to be the document database of choice for our company.
- Riak:  Incredibly interesting NoSQL offering.  We aren't using it at the moment, but a couple of our engineers have used it at other companies and we're genuinely fascinated by it.
- Datomic:  Another incredibly interesting database from the key contributors and creator of the Clojure language.  Datomic offers the ability to keep a temporal history of mutations to a records stored within, making it uniquely suitable for some of our client's problem-sets.
- LMAX Disruptor:  A framework for performing multithreaded, no-lock operations on a ring buffer.  Developers at LMAX have optimized the framework to work with the underlying hardware, like ensuring variables are cached effectively in the L1.

Sliding off the list from last year.

These are the frameworks that feel off the list this year and why.

- Spring Framework:  Spring is still incredibly important, but it's at the point that we are taking it for granted.  Knowledge of the framework is practically mandatory in our company, so it's not as important of a technology to be learned this year.
- Ruby on Rails:  We are no longer building on Rails.  Rails is still a great framework, but it's being overshadowed by the prospect of an end-to-end JavaScript web stack.  We've also had some issues with Ruby's thread model, making it incredibly difficult for us to integrate messaging into the Rails stack.
- Redis:  Redis is still a great key-value store, but it's lack of security features makes it difficult for our company to use in our client's architectures.  We still love it, however.
- CoffeeScript:  I still write in CoffeeScript all the time, but its time to acknowledge that there are a number of great new compile-to-JavaScript languages out there.  For my .NET developers, I can't consciously recommend them learn CoffeeScript when they get great support for TypeScript in Visual Studio.
- OSGi:  OSGi became a great frustration last year for us.  The API is antiquated (i.e.: no use of Generics, registration of components is not straight forward), containers function inconsistently (we gave up on JBoss 7's OSGi support), and it's a real pain in the ass to have to bundle libraries you don't write.  I think the idea of a component architecture is awesome, but I think it needs to be apart of the core Java platform and not an external framework.
- RabbitMQ:  We still love and use RabbitMQ all the time.  In fact, I just wrote 8 posts on the topic!  RabbitMQ was not so much as demoted as expanded to include ZeroMQ and Kafka.
- A new JVM Language:  I don't advocate staying with the Java language, but I want to warn you that you may be frustrated with your options.  As you've already seen, I'm advocating you learn Clojure above all other languages.  If you don't learn Clojure, try give Scala a try.  We may even be surprised this year with a resurgence in Groovy's popularity as optimizations of the JVM makes the language much faster.  Outside of those choices, I think you will find learning a language off the platform more rewarding.


5 comments:

  1. Hello,

    A very nice post, exactly what I was looking for. It would be helpful if you could also give the best resources to learn some of these technologies. Great work!

    ReplyDelete
  2. plz provide the resource to learn above mentioned technologies

    ReplyDelete
  3. Great post this. As is your post for 2012. I've worked on the same products for some years now and it's been easy to get stuck in using the same technologies, albeit the latest versions of them. So when you are faced with trying to find out what everyone else is doing/using out there, such a list is very useful.

    Any plans to do a 2014 one too?!

    ReplyDelete
    Replies
    1. Thank you. I'm working on the 2014 one; your response is motivating me to get it done today!

      Delete
    2. Great - looking forward to it!

      Delete

Note: Only a member of this blog may post a comment.