How can we help?

Design? Check. Strategy? Check. Bulletproof code? Check! People who can manage it in an agile and efficient manner? Check. Someone to help you create your next big product? Of course.

Denver

- 1201 18th St. Suite 250, Denver, CO 80202

Grand Rapids

- 125 Ottawa Ave NW Suite 270, Grand Rapids, MI 49503

Blog

BLOG:Retrospective on a Long-Term Large Javascript Project

Retrospective on a Long-Term Large Javascript Project

We recently finished up a fairly large Javascript/HTML5 project that spanned the course of about three years. Large is a somewhat relative term, so to put things into perspective, we’re talking about a project with around 700 Javascript files ( 75k lines ), 300 HTML templates ( 13k lines ), and around 100 Less files ( 18k lines ) which excludes any 3rd party libraries that we brought in.

In the Javascript world, a lot has changed in the past three years, and before a lot of this information escapes my mind, I thought I’d compose a list of some of the most important things we learned.

Backbone

We started this project in the winter of 2013, right around the time the Javascript framework wars were really picking up. Angular was rapidly gaining popularity, as was Ember. The client we were working with already had some experience with Backbone, and preferred to stick with that, so we used that for the project. While not the hottest thing at the time, it quickly became one of our favorite frameworks to work with.

Backbone is a lightweight framework that has been around since 2010. One of the best things about it is that it hasn’t changed much over the years - it’s established, used in a lot of very large websites, and most of the updates during our development were very minor. There is very little magic in it and the source is readable in a few hours. But because it’s so simple and lightweight, it doesn’t force you into a certain pattern of how to use it, which can be good or bad depending on the developers that are working with it. In our case, we had three incredible developers lay down the initial architecture and patterns, and we were then able to scale that code during the project. I know this is not always the case with development, and sometimes more guidance from your framework is desired, but I enjoyed the freedom of architecting our own solution that really fit their requirements.

It’s worth mentioning that the freedom of crafting that solution usually involves bringing in some of the best of breed libraries to fill the gaps that the main framework you chose leaves out. There seems to be a trend in Javascript development where either you build up your solution using building blocks of libraries that do certain things particularity well (Backbone/React/Flux/Redux/etc) or a more comprehensive solution that makes these choices for you (Angular/Ember/Sencha). Each way has its pros and cons, and the best choice really depends on the project you are building. If you don’t want to have to pick the best router or data management library and want the framework to provide stricter guidance on how things should be done, a more comprehensive framework might make more sense for you. If you want the freedom to swap out different pieces and lay down your own architecture, using the building block approach would be better. In our case, we wanted to have that finer level of control of what we were using and why. In general, we wanted to stay as close to the Javascript specs as possible and not bring in any libraries unless we absolutely needed them. This helped reduce the pain of upgrading when other libraries changed. After three years of development, this app is still very maintainable and has a solid foundation for the future. As ES6 nears, they will be able to easily bring in a lot of great features it has to offer.

Dependencies

It’s amazing to think how complicated Javascript development is becoming. There is a growing number of libraries you end up bringing in via Bower or NPM and become dependent on. Hopefully, we won’t get to the point of DLL hell, but we are becoming more dependent on other developers maintaining these libraries and making sure they are not making breaking changes to their APIs.

I have to admit that over the course of the project, the number of issues we had with changes in third party libraries breaking our build was probably less than 10. Given the number of dependencies we had and builds we did, I would have expected more. After we got a burned a few times, we learned to stick with exact versions of libraries using specific version numbers instead of using the conventional tilda in the package.json and bower.json to grab any patch release updates. This way there weren’t any surprises during simple patch fixes because we got those surprises; it’s likely you will as well.

Code Reviews

When we first started the project, we didn’t do a whole lot of code reviews as we were rapidly trying to build out the skeleton of this new application. But after seeing a coworker describe their GitHub pull request workflow, it seemed like something very beneficial and didn’t seem to add a lot of overhead. At first, we were worried at the amount of time it would take to do the reviews and what it would do to velocity, but after we were able to catch some potential issues during a review, we knew this was something very beneficial to the project.

The basic idea is to follow the Gitflow style pattern where a developer works in a feature branch of the code repository and completes their task there. When they completed that feature, they’d create a pull request from that branch to the main develop branch. Then someone would have a chance to review it and make sure it was working and met the requirements of the ticket.

At first, we did this on just some features that we wanted an additional set of eyes on to help test, but it quickly turned into something we did all the time. Especially as we grew as a team, we wanted to ensure that the quality of code stayed high. This also cross trained developers into different sections of the app that they might not have been as familiar with. Some developers knew certain sections of the app better than others, by assigning them the pull request, they were able to share some additional edge cases in testing that maybe you didn’t think of. Pretty soon, anytime you committed something directly to develop, it immediately felt wrong and you’d have to wear the cowboy hat.

One thing worth mentioning is that it’s best to keep your pull requests small and focused. There was a saying that if the pull request had less than 10 lines of code changed, that we’d be more likely to reject it, but if it was fairly large touching a bunch of files, it would likely be approved. It’s not that we didn’t test it, but it’s much harder to be comprehensive on a review of a large piece of work. Time crunches happen, and if it works, it sometimes gets in.

Embracing New Paradigms

While we were working on this project, React came out and we paid close attention to what the community was doing with it. We even had one of our developers go to React Week, immersing himself in it there. After dabbling around in it, we really wanted to bring it into our project, but by that time, we already had a solid architecture in place using Backbone and even had developed a shared library for our service layer that additional projects were depending on. Introducing a new dependency on something that was still changing fairly rapidly was not something we decided on doing.

Stepping back from the immediate desire to bring in the hot new library, we thought more about why we wanted to bring it in. Sure the virtual DOM stuff was pretty awesome, and being able to create React components would have been useful, but we really didn’t need them at the time. Our performance was fine, and we already had most of our views somewhat ‘componentized’. Some things that we thought we could benefit from were some of the ideas of React manages data for its components and how it renders that state to the DOM. We ended up creating a React-like Backbone base view that added in the idea of state/options and a basic system to control how you wanted to render the view based on those data changes. Options would be things that were passed into the component and always read only, and state would be the data that the component managed and could change. When the state or options changed, you could attach your own handler to determine how to process that data and potentially update the DOM. Now you could target certain pieces of the DOM to update instead of re-rendering the whole thing. Someday we could potentially swap in a Virtual DOM implementation to make that even more magical, but for now, it allows developers to have fine grained control over what they want to update. This type of data management in the component also helped keep state out of the actual DOM and in the Javascript so there was only one place that controlled what was displayed.

While we never got to bring in React, I think we ended up with something pretty special that gave us the best of both worlds. Developers got familiar with the concepts and enjoyed working that way. If they ever move to a new project that’s using React, I’m sure they’ll be quick to pick up the rest. This is just an example of how you could potentially bring in new programming paradigms into your own code without the explicit dependency on it. When working in technology that is changing so often, you should be watching the current trends and be open to bringing in some things into your project.

Code Structure

Code organization is one of those things that you can do several different ways. I simply wanted to mention this as one of the things we initially did one way, and then later in the project realized that maybe we shouldn’t have done it that way. You won’t always get things right the first time, so be prepared to recover, rethink, and redo.

When we first started the project, we had a ‘src’ directory, and that had subdirectories for ‘less’, ‘js’, and ‘html’ holding the corresponding file types. Initially, our ‘js’ folder stayed somewhat organized into folders for each of the different sections of the app, but the ‘html’ templates folder and ‘less’ folder ended up become flat with a bunch of files in them without organization. The lack of organization is one thing, but what was extremely painful later on was finding what files went together for particular sections of the app. Especially for new developers that joined the project, it wasn’t always clear what went with what.

We used RequireJS to bundle our code into a handful of modules that got loaded at different times as you used the application. When you created new views, that view would require a bunch of Javascript to control the view, but also brought in some HTML templates for those views as well. As a developer working on these views, it felt more natural to see all the associated files together regardless of type in the same folder. By moving everything into these folders, you’d see it’s JS, HTML templates, Less files, and even some localizations that it might have used. It was nice to see all the pieces of what put that particular view together in one place. While we didn’t go back and update everything to this pattern, we’d highly recommend starting this way on the next project.

If you are interested in reading more about code organization, I’d recommend reading a post by one of our coworkers, Cliff Myers - ‘Code Organization in Angular/Javascript Projects’.

CSS

CSS in a large project can become very problematic if you don’t architect it well initially. Without guidance, it’s easy to have styles trickle into certain selectors that you may have had no idea about. Then it becomes difficult to see why some things aren’t styled as you intended.

Initially, the developers would try to scope their css within certain sections of the app and then they had a special file for anything that was truly a global style. This became problematic as we built more and more sections; we really needed better guidance on how to keep this under control.

Later we adopted a new pattern in correlation with the code structure changes we made where your styles would be scoped to the views we were working on. Each view would have a class where it would scope everything under that. For example, in the following HTML, we have an OrderDetail view, and here is how we might style that.

HTML

<div class="MainWorkspace">
<div class="OrderDetailView">
<div class="header"></div>
<div class="orderContainer"></div>
</div>
</div>

CSS

.OrderDetailView {
.header { … }
.orderContainer {
.detailText { … }
}
}

This ensures that the things you styled for OrderDetailView would only be scoped to that view. If you needed to make something common, you’d move up to the highest level of the scope that included the pieces you wanted to make common and then define the style using that selector. For example. if you wanted to make header common to all the potential detail views, you would do something like:

.MainWorkspace {
.header {...}
}

If you had to override that style for the ‘OrderDetailView’ for example, you could do it within the scope of that view. While in this example it seems straightforward, when you are building a larger app, maintaining this hierarchy is not often as easy as it seems. Setting the rules at the beginning will help keep this consistent as your app grows.

It’s worth noting that we knew that performance of the browser takes a hit as you increase the number of selectors in your style, but we felt that the security it provided to the style outweighed the performance impact of it.

Another thing related to CSS and styling that I wanted to share is that you should make sure that you never hard code any colors in your Less/Sass. You should work with your designers to decide on a color palette for the app and ensure that you define variables for those colors and stick to them. Ideally, you’d have some kind of style guide, but not all projects have the time/budget for one, so sometimes your color variables become the only guide to keep new designers from bringing in other variations of those colors into your app. It seems minor but it can become a huge headache trying to figure out which shade of that color is the right one. We had an issue where we had way too many versions of a particular color and during the refactor of fixing that, we wrote a script to color match the questionable color against the known set of colors to determine its closest match. Using that script we were able to reel in all the rogue colors.

There seems to be a lot of room for improvement in how CSS is used in large apps. CSS Modules look promising, and provide answers to some of the issues we have with CSS, but only time will tell where this goes. If you are interested in CSS Modules, we just posted an article all about them here.

Pragmatism

During the course of the project, we scaled in team size from around three to twelve developers. One thing that all developers commented on when joining the project was how readable and understandable the code was that they were working with. We focused on clarity of code over cleverness. One of our developers always said, ‘know that the next developer to look at this code will probably be yourself so be polite and focus on clarity’.

When working on features, we tried not to over-engineer them, focusing only on the most immediate requirements. We were working in an agile environment and tried to keep things reasonable per sprint. Sometimes we ended up butting heads with the same pieces of code that were becoming problematic. By the third time we hit something, we’d regroup as a team and see if there was a better solution. The majority of the time it really revolved around creating a more generic solution, which makes you think ‘well why didn’t you make it more generic the first time?’ Making a generic solution doesn’t always make sense, it adds extra development time, additional cost, and might not be as performant as something targeting the exact features. What we found is that over time it sometimes took those intermediate versions to really figure out all the features that were really intended for it.

Conclusion

When you work on something for three years, there will always be things you did right the first time and others that you didn’t. Being able to look back and see things that you can improve will help you grow and help on the next project you work on. That process will likely repeat throughout your career.

As Javascript and Web Technologies continue to evolve, the thought process on how you build these large apps will change. Trends will come and go and it’ll be up to you to decide what you think will work best for your project. Just because they make sense to you doesn’t mean they are a good fit for your project. Sometimes it might make sense to bring in a certain library or it might just be that you adapt some of those new ideas into how you develop your own code.