Enterprise web development is focused around the web server. You can change the functionality running on the server and test it fairly quickly out of the box. To avoid breaking what is already there when adding new functionality mature projects normally set up large numbers of automated tests. These can take several hours to run but typically hovers under an hour. This means that to verify that a change you made is Ok you have to run tests on your machine for perhaps an hour during which you cannot work on other changes, so it is both an interruption and a drag on productivity.
With increasing need for interactive Web User Experience a lot of effort is in making changes to code running in the browser. Enterprise web development servers are not made for this, so making a change (think changing a sentence) can easily take minutes rather than seconds before you can confirm the effects. As a developer makes hundreds of such changes per day this is a significant drag. Typically the functionality running in the browser is 80% of the development effort and the server 20%.
This has been the trend in the past 4 development projects we have done. A trading software vendor, a Swiss Bank, a Swiss Telco.
As staff moves around between projects and people join from outside at regular intervals it is important to use tools that are well known. In web development this means using common Open Source solutions rather than proprietary ones. When a build tool has 20k stars on GitHub it means a lot of people know it. We intentionally went for command line friendly tools that are broadly appreciated.
Improving Web App Development
In one project 2 years into development making a change in the browser content. Say setting a color somewhere takes seconds. To update the server takes 1m 25s to build and 35s to re-start the server.
To get around this we added a special web server just for development that shows the latest in-browser functionality changes and passes other requests to the Enterprise Web Server. While it does take 35s to start this development server, it doesn’t have to be restarted for every change made. Once a change is made a part of the test suite is run and the web page reloads. This takes around 25s, so it is four times faster to see changes.
The psychology of waiting is actually a bigger issue than the numbers indicate. Once you have to wait several minutes you are more likely to go for a coffee causing an even bigger delay, but when it is seconds your focus isn’t interrupted.
A nice detail is that changes to CSS source update the CSS bundle which in turn is updated in the web browser. However it doesn’t cause the page to reload, so it doesn’t interrupt the state you are currently looking at. You can be in the middle of a workflow and update the styling on the fly.
With the times it is clear that a modern computer should be able to update the web page much faster. So with a new project we decided to replace some technology choices and revise the approach.
- The development web server can be started immediately while the changes are updated (build process)
- The tests can be bundled on the fly and sent to the web browser to be run (no writing to disk)
- Putting the browser bundles directly into the destination on the disk drive avoid copying overhead
- Support ad-hoc simulation of configuration differences that may be present in production
UX prototypes could be tied into the development server to provide fast access to compare the implementation with the visual mock-up.
Speed is a competitive parameter. We often explain away how it is not, but just think about how quickly you get impatient stuck in a traffic jam or waiting in line at the supermarket. If you have that feeling as a developer every day, it affects your mode of working.
When things are snappy we not only get things done faster, we feel better about everything.
When you get confirmation that nothing broke due to a change that you just made you can focus on the problem at hand and only react if you get notified that something broke. While developing it is important to focus on what you are doing and not get sidetracked by technical tool details.
Web Build Technology Details
The Server build is run with Maven. One artefact in Maven is to make the Front-End using Grunt. The build,
- Pull in Angular, formatting, Bootstrap libraries
- Pull in our libraries for common navigation frame and components.
- Concatenate JS dependencies in vendor.js
- Merge JSON translation files
- Bundle CSS from Sass sources and enhance with Autoprefixer
- Bundle JS from sources
- Minify and Uglify bundles
- Copy build output tmp folder to exploded WAR target folder
- Rename bundles with unique ID to bust caching of old bundles
- Run 300 Unit Tests with Karma and Jasmine against the un-minified JS
The model for Grunt is based on the idea that each step reads files and updates other files making a lot of changes on disk. In comparison Gulp is based on the Stream principle that pipes one step into the next. The switch to Gulp did cause some improvement, but no an order of magnitude. The main advantage is the general feeling that it is leaner and enables doing whatever is needed in the build.
The main speedup for the Maven build is the splitting of build and test. If Maven is run with tests disabled the Karma unit tests shouldn’t be run, this saves ~20s for something that will be done a lot. Even more importantly you sometimes want to do a Maven build even though a couple of tests fail. This happens often while you are making changes.
With the Gulp build the working directory is the Maven target directory. This means that running a Maven build while running the development server will replace the bundles. That isn’t a real use case. Normally you run the maven build to start the server and then start the development server. Since Maven has already deployed the target folder is free to update. This removes a copying step and gives a simple answer to the question where is the stuff I see in the browser stored?
For development the
grunt live command does a full build and then starts the development server using connect. Connect serves out of a temporary build directory that contains the bundles for the current source code. The bundles are automatically updated when source changes and a page reload is triggered.
With Gulp the development server is started immediately serving static bundles out of the target directory. This means that bundles are updated while the browser starts and loads the initial page which in our case is a launcher that allows simulating different use cases. Browser-sync(browsersync.io) is used for the development server. It supports trying out the site in multiple browsers at the same time. While browser sync runs, Karma is also run independently. That gives you feedback on unit tests passing as you develop. It doesn’t affect the running of the development server if tests fails, this is much better than the old Grunt build.
As an improvement BrowserStack(browserstack.io) can be used to test a long list of browsers that you probably haven’t installed on your local machine. This is great for mobile devices.
Although we use Gulp that is a detail best hidden. Some tasks don’t need Gulp and can be run with other tools. So the commands to run are all configured as npm scripts. In the
scripts section of package.json all the commands you need are listed
- npm run build: Make the bundles and copy assets to target
- npm test: ESLint(syntax check) sources, Run Karma tests bundling on the fly
- npm start: Start Karma Server, run build and BrowserSync
- npm run modules: Update node_modules to avoid npm install on CI Build Servers
Due to time constraints some features were not finished.
- The JS and CSS bundled must be minified and renamed for production to ensure cache busting. This can be done in place.
- There should be an option to run tests against the minified source
- There should be an option to use the minified source with the development server
NodeJS has become the defacto choice for building web assets. Other languages offer some support, but nearly all the best tools aremade with Node. The needed tools are,
- ES6 to ES5 transpiler (buble,babel,traceur)
- Angular injection transpiler (ngannotate)
- CJS/ES bundler (rollup,browserify,webpack,jspm)
- CSS+ bundler (Sass,Less,Stylus,PostCSS)
- CSS compatibility processor (autoprefixer)
- Image minifier
- JS minifier(uglify)
- CSS minifier(cssnano,clean-css)
- Development server and proxy (connect,express,browsersync,webpack-dev,jspm)
- Builder (gulp,grunt,brunch,webpack,jspm)
- Dependency manager(bower,npm,jspm)
To get the fastest transpile and simplest ES6 support we picked Buble. Rollup isn’t an option as there is no ng-annotate option. The possibility of using babel+rollup with ng-annotate run by babel was not explored. Our main source is in SASS format, but using Gulp we combined both a SASS stream and a Less stream and used PostCSS to improve the css for older browsers and shrink it for faster loading.
Initially we investigated jspm as it is quite forward looking delivering many great features out of the box. Unfortunately it didn’t manage to download dependencies through out internet gateway and there seemed to be no way to set it up to get around it.
Getting sourcemaps to work well is very important.
- They must refer to the correct line
- The path to the original source must fetch it in the browser
In Grunt we had the path for the sources wrong for most of a year, possibly due to a bug in the Grunt plugin. In Rollup we’ve seen issues with matching lines in the web debugger. Angular annotations can also mess up line locations.
ngAnnotate is very fickle. It doesn’t do source maps so well. We tried doing the Angular injection transpile with a Browserify plugin but it failed to build source maps. We were not able to reduce the problem or work around it.
Buble is a much faster transpiler than Babel. It was tested for ES6 support, but assed seconds to the build time.
Babel disabled support for decorators (@Inject) this means that the ngAnnotate plugin doesn’t work for Babel 6.
Browserify is an ecosystem that allows for switching in many options. It does however require maintaining a lot of version choices that may conflict. A wide range of combination options are possible.
Webpack is a monolith with makes version choices much easier. The configuration is however very convoluted supporting a network of plugins where it’s hard to see what is the right solution. For instance there is a 3rd party Bower plugin and a builtin plugin. There is also the resolver which can be used without a bower plugin if the main entry point doesn’t have to be relied on.
By keeping vendors, app and test source different islands the tests can be run in different models. Individual source. Concatenated and minified. If tests import the app source the minified bundle cannot be tested.
When source maps are not configured quite right it impacts development productivity as you cannot set debug breakpoints correctly.
Setting up a project with a good build process for web assets is a significant task. It can provide a lot of value to do it well, but it also takes time. Spending 2-3 weeks getting everything working isn’t unusual. Once a good setup is achieved it can be used as a basis for other projects.
As tools evolve and new features arrive you will want to keep up. Be sure to spend up to a week per year to improve the build process.