We’re finishing our first year with SCRUM and I thought I’d recap the highs and lows that I’ve observed.  For me, this is the most enjoyable thing I’ve done in my 20+ year career in IT.

Highs

Lows

So that’s my take after one year of really doing agile.  In the next 12 months, our plans are to gain more consistency across teams (artifacts & ceremonies) and just get better at it.  Organizationally, we’re still newbies, but there’s a real commitment from the top, which makes all the difference.

So, to my friends, peers & mentors, what advice do you have for me? What land mines should I watch out for?

If you wonder why I’m posting to a blog instead of Facebook, I’m keeping work & social networks separate. LinkedIn, which connects to my blog, is for work & Facebook/Google+ is for friends. I think mixing work associates into my friend network isn’t a good idea.

Have a Merry Christmas! Hope you can enjoy time with friends & family.

The project is web forms (yes I know that MVC is cooler), VS 2010, IIS Express and SQL Server 2008.  So, I figured this would be an easy process when I do a production release:

1. Check-in code & tag it in subversion
2. Update my prod code directory from the tagged version in subversion.  Compile app and find what I didn’t check-in correctly.  I call this the oh-sh!t phase.
3. Apply DB updates from local dev DB to local prod DB. 
4. Run unit & smoke tests, aka, run nUnit.
5. Problem:  The web app defined in the solution file & is tied to the port in the IIS Express config file, “C:\Users\[name]\Documents\IISExpress\config\applicationhost.config”.  So, even though there’s 2 separate physical paths on my laptop, the solution file is locked to a port in another config file.  Argh!
   

      1: Project("{E24C65DC-...}") = "localhost", "http://localhost:8082", "{ACF39ED5-...}"

      2:     ProjectSection(WebsiteProperties) = preProject

   You just have to “learn” that the port # in the solution file magically ties to the port # in the IIS Express config file.  Bleh. 

So intelligent fellows, how have you solved this situation?

Options that I thought about

1. Use separate virtual machines – this just seems like overkill.  Why do I need a separate OS per code branch?  I want a more Java-ish solution that doesn’t require configs all over the place; I want a one-stop solution!
2. Update the solution files – Yuck!  Each time I update the “prod” branch from the tagged prod version, I have to remember to go update the port # in the solution file.  Guess how often this will be forgotten…
3. Maintain 2 solution files inside the project – Yucky, yuck!  The idea is to keep 2 solutions files; one for dev and one for prod.  Maybe worse than #2, because each time I add another project or other solution item, I have to update the other solution file.
4. Please, please someone have a better idea!!!!!!!

What I’m doing now

I’m using option #2 above and it is painful.  I can test prod fixes in one branch and update the repository (trunk or branch) with the changes.  It isn’t a huge deal to remember to change the port in the solution, but I have to remember to NEVER check-in the prod branch solution file and each time I overwrite it, I have to manually edit it.  There just has to be a cleaner way.

Phil’s got a great eye at making things look good.  Not me.  I’m a table with boxes and lines, no round corners or any of that fancy schmancy stuff.  Take a look; I think the differences are striking and I was so blown away by the differences that I decided to write this.  Nice clean look; no harsh lines; less text and a nice pretty book cover image.

In a week or so, I’ll have the site updated with the new cart.  We’re heading into our Christmas rush, so we’ll see if we get more compliments or complaints.  My bet is on compliments. 

What do you think; like the new cart look?

Why you should care

Privacy

Google, Bing and others have this “nice” feature where they send your search terms, location, browser information and other details to a site when you click on a link from their search page.  With enough searching and enough sites tracking you, you’re identity isn’t secret anymore.  Here’s a blurb from WSJ about how insurance companies are thinking of using the data.

Insurers have long used blood and urine tests to assess people’s health—a costly process. Today, however, data-gathering companies have such extensive files on most U.S. consumers—online shopping details, catalog purchases, magazine subscriptions, leisure activities and information from social-networking sites—that some insurers are exploring whether data can reveal nearly as much about a person as a lab analysis of their bodily fluids

I’m not a big conspiracy person, but I don’t trust folks at Google to take care of my information.  Besides, any organization that hires Eric Schmidt shouldn’t be trusted; he’s a snake.  There’s just something creepy about him.

Here’s duckduckgo’s presentation on the subject.

Filtering your data

Wikipedia: http://en.wikipedia.org/wiki/Filter_bubble

While I’m sure we have vast political differences, this is a good presentation on the “filter bubble”.   There’s a fair amount of disagreement on his views, but it is interesting to listen to his hypothesis.

Here’s duckduckgo’s take on it, dontbubble.us.

It ROCKS!

I know I started off with the same heading last time, but it totally rocks!  We’re 7 months into our first major SCRUM project and I can tell you that waterfail is now so painful that I can’t stand it.  I’m spread across several projects and I gravitate towards the scrum project, because it is fun.  There’s real teamwork; people working hard and having fun; progress being made and business folks believing in the work.  Who wouldn’t want to work in that environment!

What I’ve observed lately

Attitude

I have to admit that I totally missed what scrum/agile would do for attitude.   In waterfail, it seems that you’re always fighting between teams and trying to figure out how not to do things.  6 months into a waterfail project and your talking about descoping work; can it get more miserable than that?

Now contrast the attitude with an agile project and it is the complete opposite.  In agile, it is all about what we can get done; what we can do to add value and “yes”, “yes”, “yes”.  Don’t get me wrong, there are tough decisions, high tension discussions about features, but in the end; it is teamwork and partnership.  Once the team gels and everyone’s on the same page; the positive vibe can’t be hidden.  Other teams will look at you like you’re crazy because you are feeling GOOD.  When’s the last time you saw that vibe in a 3 hour long requirements session; YUCK!

The team became a team

Huh, what?  What I mean is that the group of folks put together really became a team.  About 4 to 5 sprints into the project, there was this sudden shift of everyone pulling together; really pulling together.  The team is comprised of folks from 8 different disciplines/areas (.net, java, DB, ba, qa, pm, architecture [me] & business).  Initially, everyone worked well together, but there were these “walls” between us.  The dev guys had differences of opinion about the DB guy and we weren’t really honest when he was in the room; that sort of walls.  Just not really trusting each other and being honest.

One day, it all just changed.  Yes, it really did happen gradually over time as we learned to trust each other, but it was visibly noticeable about sprint 5.  Suddenly, people gathered around to discuss design or other issues.  Everyone talked through how to solve a problem; it wasn’t just dev types; it was everyone.  It wasn’t my solution or her solution; it was our solution.  That’s when the team became a team.  We trust each other; we help out whenever we can and we’ll protect each other to all ends.  Just like in sports; a cohesive team often times beats the better talent, because they work together.  

Artifacts work

When we started SCRUM; I really didn’t get the power of the simple artifacts: velocity chart, burn-down chart & burn-up chart.  Whoopie; 3 graphs; who cares.  Well, I was 100% wrong in my attitude.  I learned the power of velocity and burn-up (progress towards the release) when the steering committee for the project was concerned that the team wasn’t delivering enough.  First, we walked through the velocity (points completed per sprint).  For the first 5 sprints, it was 31 points and there was a zero point sprint in there.  The nest 3 sprints the velocity jumped to 70.  That proved to the executives that a major shift in productivity was being sustained.  Closed that issue in 10 minutes; beat that time to solving an issue with directors and VPs!

Second, the burn up chart proved to them that the project was on track and even trending ahead of schedule to deliver what was required for the first release.  Why did they believe the graph?  We just showed them the velocity and that it was increasing.  They could see that every 3 weeks we were tracking progress and they didn’t have to question the numbers.  The proof was right there.  Closed that issue in less that 10 minutes.

The artifacts work; do them; trust them. 

Ceremony works 

In SCRUM, there are 4 ceremonies as they are called: sprint planning, daily stand-up, sprint review and sprint retrospective.  Hopefully you know what these are; if not, go look them up.

Our daily stand-up meetings were horrible at first.  We didn’t stick to: “what I’ve completed”, “what I’m working on today” and “what are my impediments”.  The stand-ups should last about 15 minutes and ours was taking a minimum of 30.  We’ve made it a point to stick to the 3 question and get done in 15 minutes.  Wow, does it go fast; people know what’s going on and everyone’s back to work quickly.  We reserve an additional 15 minutes for issue resolution, but if you’re not need or interested; back to work you go.  Quick, clean, simple and very effective. 

Sprint reviews are your time to show off what you’ve done.  Knowing that business stakeholders, directors, VPs and even “C” level execs are invited and show up really keeps you on your toes.  Do you want to tell them we don’t have a review, because we didn’t complete anything?  I know we don’t.  The power of showing the product every 3 weeks can’t be underestimated.  Ideas start flowing; people get excited and the engineers get to actually talk to real live people on the business side.  When a top level exec shows up at a review; it is amazing how energized the team becomes.  How often does the President of your company show up to your project meetings?

The retrospective has become my favorite meeting for selfish reasons.  We’ve kept every retro’s output on large sheets.  Looking back sprint by sprint at what we’ve done well and what we can improve upon is amazing.  I’ve learned so much about human behavior from this one activity.  Once the team becomes a team and is honest about what’s working and what’s not; real change happens.  Most companies do a mid-year and annual performance review for people.  Imagine if every 3 weeks you looked at your work and really evaluated it; think how different your work habits would be.  It is very, very hard to be honest, open minded and willing to accept real criticism of one’s work.  However, the growth that comes out of this exercise is truly amazing.

Gory Details

Each page in a book is rendered using either a common page handler or a “specialized” page handler.  I use a config (xml) file that contains entries like the following to wire-up the specialized pages.

<add key="mapgraphic"   value="klw.Render.Data.PageMapDataHandler" />

So, the natural thing to do is hide the ugly details behind a factory class and that’s where my problem hid undetected for years.

So let’s see if you can find the bug in an asp.net application, which is naturally multi-threaded.

Look at this code snippet:

   1: if (cache.ContainsKey(key))

   2: {

   3:     result = cache[key];

   4: }

   5: else

   6: {

   7:     if ((PageData.SpecializeKey != null) && (lookupTable.ContainsKey(PageData.SpecializeKey)))

   8:     {

   9:         Type handlerType = lookupTable[PageData.SpecializeKey];

  10:         result = Activator.CreateInstance(handlerType) as PageDataHandler;

  11:     }

  12:     else

  13:     {

  14:         result = new PageDataHandler();

  15:     }

  16:     result = Activator.CreateInstance(handlerType) as PageDataHandler;

  17:     cache[key] = result;

  18: }

  19: return result;

Did you find it?  Ok, I’ll give you a hint; what happens if there’s class level state for anything created in line 10 or 14?  Look a little further down.  Did you find it?

Okay, times up.  My CODESOD moment is line #17.  I really meant to cache the “Type”, NOT the instance for “specialized” pages.  So there’s my multiple threading issue.  If multiple threads are requesting an instance at about the same time, thread 2 can get the instance that thread 1 is using.  Bunch of thread switching and bad things happen.  I was in a hurry, so I just commented line 17 and the problem went away. 

There you have it; enjoy my folly.

It ROCKS!

As surprised as I was about the impacts of SCRUM and agile on development, I’m even more surprised about the impact it has on project delivery.  Now, to be clear, SCRUM doesn’t cover the entire lifecycle; SCRUM is about getting quality code out the door.  Some will disagree with this, but SCRUM doesn’t cover business valuation of the project, what happens after it is released, call center impacts, etc..  There are additional agile project management approaches that do.  Read this book; I can’t cover the material as well as the authors did. 

Getting Rid of the Bureaucracy

The biggest problem with the waterfail methodology is it inherently adds a lot of paperwork, sign-offs, meetings and a ton of bureaucracy.  SCRUM and agile is all about doing the least amount necessary to produce a quality product.  If I never see another 3 ring binder full of what’s called requirements, I’ll be a happy, happy man.  So how does SCRUM achieve this?  Well, by applying common sense:

It ROCKS!

I rarely get excited about project development approaches, but SCRUM and agile just plain rock.  We are seeing dramatic changes from a project management perspective and a development perspective.  Honestly, seeing it in action is 10x more impressive than reading testimonials or “how-to” books.  I wish they would have called it “Common Sense Approach” instead of SCRUM, but oh well.

Development Impacts

Quality

Our #1 development priority is quality.  Not features, not time to deliver, but quality.  To achieve this, we’re spending significant effort building automated unit tests.  Everyone thinks they know how to build automated unit tests, but until you do it, you won’t appreciate the complexity.

The first step is to make unit testing one of the most important steps in you dev effort.  To do this, we approach building our app as follow:

1. Build the entities (we’re using nHibernate)
2. Build out the repository/data access (crud operations, searches, etc.)
3. Build the unit test (lots of them)
4. Finally, build the UI or services

By sticking to this approach we’re pretty much guaranteeing that business logic stays out of the UI.  Duh, of course “we do that”, but be honest, how many times does a little business logic leak into the UI.  Yeah, I thought so.  Some may argue to unit test the entities, but they are just to hold data, so there’s nothing much to test.

Good unit test are hard to write; there’s books written on the subject.  Here’s what we’re doing:

1. Full CRUD operational tests.  Using an ORM, you’ll find a few WTF moments after looking at the generated SQL.  While we don’t automate looking for the WTF SQL, creating the unit tests and logging the SQL makes it real easy to spot.  Besides, it is good practice to ensure that our data access can perform the basic CRUD operations.
2. Test the various search operations with varying parameters.  Test with valid and invalid parameter; test several boundary cases and throw in a few “gut feel’ tests.  You’d be surprise at how many gut feel tests discover the hard problems.
3. When we find a bug in QA or in UI testing, stop immediately, write a unit test that covers the bug; fix the code and you’re golden. 
4. A few other rules that are just good unit testing practices:
   1. Guarantee that each test can be run independent of the other tests.
   2. Guarantee that all tests can be run in any order.
   3. Do your best to construct your own test data; don’t rely on pre-existing data.  All good frameworks have test setup and destroy (tear down) methods; use them!
   4. When you need large amounts of data for a test, such as data paging tests, either bulk load data before your tests or pick a range of data in your DB and tell everyone to LEAVE IT ALONE.  TDD purist; back off; I know that you can test without the DB and it is overhead, blah, blah.  In my world, keeping a few hundred thousand rows in the DB is a much easier approach. 
   5. Find tools that will help generate the test code.  We’re just starting to investigate this and are looking at T4 templates, MyGeneration and other tools. Let me know if you’ve got a good one.
5. Each test should do ONE and ONLY ONE thing.  Don’t try to test all the crud operations in one test; you’re just wasting your time.
6. We are NOT following full test driven development (TDD).  We’re making some first steps and hope to get there eventually.

All the rest of SCRUM and Agile/XP approaches are very valuable, but I’m finding this to be the most valuable.  If you’re not ready to dive head-long into full SCRUM and agile, take up automated unit testing.  And while your at it, help your QA team to automate their testing. 

Other Benefits

• Reduces the ever growing boat anchor effect.  By this I mean that as you add features, you have to regression test, which becomes a bigger burden, aka boat anchor.  Automated unit tests is insurance against the boat anchor and introducing bugs.  Make changes, add unit tests, re-run ALL tests and when everything is green, you know that you didn’t break anything.
• The ability to refactor your code and really trust that you haven’t broken anything is enormous.  Change a few classes, run the test, all checkmarks turn green.  DONE!  Eliminating code debt over time provides tremendous payback in the long run.

My final word on automated unit tests is that your comfort level in not breaking things is ONLY as good as the tests you write.  And you can’t test everything and you will leave a few holes.  However, the benefits so far outweigh the time commitment that you should start now.

Small Chunks of Work

The second piece of SCRUM and agile that is dramatically changing how we work is breaking down the work into little pieces and getting them to QA quickly.  Honestly, this is just how I work anyway, so it wasn’t a change for me.  I’ve always tackled things one screen at a time or one business function at a time, so this just works. 

We’re finding that breaking tasks into 6 hour chunks works really well.  Knowing that something needs to be done by tomorrow is a huge motivator.  As an example, day one may be add a new page skeleton, including CSS and style.  Day 2, add basic functionality.  Day 3 add ajax and fancy UI (jquery anyone) transitions.  

Study after study has shown that tackling three tasks in a row is far faster than tacking all three at once.  It takes an average of 30 – 45 minutes for you to get back into the groove, so stick to one thing, do it right and get-r-done.

Final Thoughts

There’s a lot more to agile than I can cover, know or am experienced enough to discuss.  Here’s a few other key points to consider/research:

• Pair programming – I thought this was the biggest waste of time on the planet when I first heard of it.  However, if you have one engineer focus on the unit tests while the other is building the entities or repository classes; BINGO!  They will refactor on the fly, discuss problems as they arise and generally you’ll get far better code.  
• Continuous Integration – Just DO IT!  It is hard work, takes a lot of discipline, but is well worth it.  Java has had it for a long time and .NET is starting to catch up.  Just do it; really!
• Check-in code frequently – If you break work into 6 hour chunks, there’s no reason for code to be sitting on an engineers workstation.  Check it in, so we can all see what’s changing; no more secrets.
• Try things; break things; it is YOUR TEAM, so do what works best for you.

Project Methodology Impacts

I’ll cover our initial experiences in Part 2.

References

http://www.extremeprogramming.org/rules/unittests.html
http://www.scrumalliance.org/
http://www.testdriven.com/
http://www.methodsandtools.com/archive/archive.php?id=103

First Attempt

The first attempt was a very, very basic asp.net page with a drop down for the log files, a text box for number of lines to read, a text area to display the lines and one button “View Log”. Seemed very simple, until I needed to read a log file that was already opened for writing.

fs = new FileStream(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite);

Next, I needed a way to read a file backward. Gee, how hard can that be… Luckily, I was able to find the code (forgot where) and here’s the guts of it. Search on c# backward reader and you’ll find it.

   1: public string Readline()

   2: {

   3:     byte[] line;

   4:     byte[] text = new byte[1];

   5:     long position = 0;

   6:     int count;

   7:     fs.Seek(0, SeekOrigin.Current);

   8:     position = fs.Position;

   9:

  10:     //do we have trailing \r\n?

  11:     if (fs.Length > 1)

  12:     {

  13:         byte[] vagnretur = new byte[2];

  14:         fs.Seek(-2, SeekOrigin.Current);

  15:         fs.Read(vagnretur, 0, 2);

  16:         if (ASCIIEncoding.ASCII.GetString(vagnretur).Equals("\r\n"))

  17:         {

  18:             //move it back

  19:             fs.Seek(-2, SeekOrigin.Current);

  20:             position = fs.Position;

  21:         }

  22:     }

  23:

  24:     while (fs.Position > 0)

  25:     {

  26:         text.Initialize();

  27:         //read one char

  28:         fs.Read(text, 0, 1);

  29:         string asciiText = ASCIIEncoding.ASCII.GetString(text);

  30:         //moveback to the charachter before

  31:         fs.Seek(-2, SeekOrigin.Current);

  32:         if (asciiText.Equals("\n"))

  33:         {

  34:             fs.Read(text, 0, 1);

  35:             asciiText = ASCIIEncoding.ASCII.GetString(text);

  36:             if (asciiText.Equals("\r"))

  37:             {

  38:                 fs.Seek(1, SeekOrigin.Current);

  39:                 break;

  40:             }

  41:         }

  42:     }

  43:     count = int.Parse((position - fs.Position).ToString());

  44:     line = new byte[count];

  45:     fs.Read(line, 0, count);

  46:     fs.Seek(-count, SeekOrigin.Current);

  47:     return ASCIIEncoding.ASCII.GetString(line);

  48: }

So, to read an open log file and dump it to the text area:

   1: protected void Button1_Click(object sender, EventArgs e)

   2: {

   3:     tbData.Text = "";

   4:     int lineCnt = 1;

   5:     List<string> lines = new List<string>();

   6:     int maxLines = int.Parse(tbLines.Text);

   7:     string logFile = Server.MapPath("~/" + ddlLogFiles.SelectedItem.ToString());

   8:     BackwardReader br = new BackwardReader(logFile);

   9:     tbData.Text = "";     while (!br.SOF)

  10:     {

  11:         string line = br.Readline();

  12:         lines.Add(line + System.Environment.NewLine);

  13:         if (lineCnt == maxLines) break;

  14:         lineCnt++;

  15:     }

  16:

  17:     for (int i = lines.Count; i > 0; i--)

  18:     {

  19:         this.tbData.Text += lines[i - 1];

  20:     }

  21: }

Initially, this worked well for a while, but I got tired of clicking “View Log” over and over and realized that I wanted a web based Tail program, like Baretail, which I use on my laptop.

Second Attempt

My next attempt was to simply use javascript’s setTimeout to refresh the page. While this worked, it seemed clunky, so I decided to ajax it. But, being a lazy programmer, I didn’t everything that WCF offers; I needed something simple and quick. I remembered reading a couple of articles about adding a static, “Webmethod” to a page. So, that’s what I did. In a previous post, I mentioned using Rick Strahl’s serviceproxy, which makes the process as easy as can be.

So the details for implementing my web based tail applications goes like this:

• Create the ajax method as part of the web page.
• Add serviceproxy.js, json2.js and js_extensions.js to enable ajax & displaying the logs
• Replace the text area with a standard div; more on this below…
• Finally some handy, dandy css to make it look nicer.

Step 1 – The Webmethod

   1: [System.Web.Services.WebMethod]

   2: public static IList<string> GetLogTail(string logname, string numrows)

   3: {

   4:     int lineCnt = 1;

   5:     List<string> lines = new List<string>();

   6:     int maxLines;

   7:

   8:     if (!int.TryParse(numrows, out maxLines))

   9:     {

  10:         maxLines = 100;

  11:     }

  12:

  13:     string logFile = HttpContext.Current.Server.MapPath("~/" + logname);

  14:

  15:     BackwardReader br = new BackwardReader(logFile);

  16:     while (!br.SOF)

  17:     {

  18:         string line = br.Readline();

  19:         lines.Add(line + System.Environment.NewLine);

  20:         if (lineCnt == maxLines) break;

  21:         lineCnt++;

  22:     }

  23:     lines.Reverse();

  24:     return lines;

  25: }

It worked much better once I added “lines.Reverse()” before returning the data. So the data returned is standard json, except there’s a “d” object that’s prepended to the data. Rick’s serviceproxy handles this, so no worries.

Step 2 – JavaScript

serviceproxy.js and json2.js are used to deal with the asp.net peculiarities of ajax of asp.net. See the links above; Rick’s site covers these in great detail.

js_extensions.js is my growing collection of javascript extension methods. Currently, this is pretty small and contains other peoples work for formatting dates and HTML encoding and decoding. My goal is to keep this small.

The heart of the whole thing is some jQuery that ties everything together, here’s the code. Instead of going through all of the code, let’s just go over a few key points:

The heavy lifting is done in the getLogData method.

function getLogData(numRows, logFile, isTail) {
    var proxy = new ServiceProxy("ViewLog.aspx/");
    proxy.isWcf = false;
    proxy.invoke("GetLogTail", { logname: logFile, numrows: numRows },
        function (data) {
            displayLog(logFile, data);
            if (isTail) {
                var method = "getLogData(" + getNumRows() + ", '" + getLogFileName() + "', "+ isTail +")";
                logTailId = setTimeout(method, getNumSecs());
            }
        },
        function (errmsg) {
            clearTimeout(logTailId);
            alert(errmsg);
        },
        false);        // NOT bare
}

As you can see, there’s not much to it. By setting “isWcf = false” and passing in false for “bare” setting, the heaving lifting is done by the serviceproxy. if successful, then display the data and call “setTimeout” for the next ajax call. Okay, so I can hear you asking, “Why not use setInterval?”. For 2 reasons, I decided to useSetTimeout: first, the precision isn’t important, and second, if the ajax method is slow, I don’t want to hammer the web set for log reading. That’s all.

If there’s an error, clearTimeout is called, so we don’t keep calling the ajax method when things to bad.

displayLog is mostly

var $log = $('#logDiv');
var logStr = "";
for (var property in data) {
    logStr += formatLine(data[property]);
}
$log.html(logStr);

So, you probably would like to see the magic that is formatLine…

function formatLine(line) {
    line = line.encodeHtml();
    if (line.toLowerCase().indexOf('info') === 0) {
        return "<span class='info'>" + line + "</span><br/>";
    } else if (line.toLowerCase().indexOf('error') === 0) {
        return "<span class='error'>" + line + "</span><br/>";
    } else if (line.toLowerCase().indexOf('warn') === 0) {
        return "<span class='warn'>" + line + "</span><br/>";
    } else if (line.toLowerCase().indexOf('debug') === 0) {
        return "<span class='debug'>" + line + "</span><br/>";
    }
    return "<span>" + line + "</span><br/>";
}

The only interesting part of this is the “encodeHtml” method, which is one of the extension methods.

So when it is all done, this is how it looks. To turn on the “tail” operation, click tail; that’s about it.