Invitation email: Subject: XP Bootstrap Team I'm interested in organizing an XP team in the SF Bay Area. My idea is, we get together on a regular schedule -- let's say, Tuesday and Thursday nights, and/or Saturday afternoons -- and work on a project in the XP style. Note that although I want to help organize the team, I'm pretty sure I don't want to "manage" it -- it would be awesome if someone else stepped up to fill the role of coach and/or tracker and/or manager. Also note that there's no budget yet -- this would be strictly a learning exercise. Naturally, if it works out, we would be well-positioned to start a company or whatever. In thinking about the problems I've had with self-motivation, I realized that the biggest is that I'm my own customer. So I went out looking for a customer! A friend of a friend is publishing a photography book; our first idea for a product is an online photo database containing the pictures from the book. I think having a "real" deliverable will crystallize the project and motivate us. She'd be able to come to meetings etc. (Of course, if she falls through for whatever reason, we can find a different customer.) If you're interested, please email me at alex@jguru.com. Also, let me know your availability -- weekends, week days, week nights, etc. (I'm available pretty much whenever.) I live in San Francisco, and my apartment and bandwidth and 802.11 network and CVS server are ready for action, but depending who signs up we might want to pick a different meeting place. - Alex --- From: "Erik Hanson" My goals in joining the project are: to do 100% XP, or at the very least, the 20% of XP that I don't do at work; to be in a better position to get/create new (XP) employment for myself; to pair with people who are smarter than me and/or are better programmers so I can learn. --- Iteration notes: I0 no notes I1 Mon 2/25 poor Ian and William had to configure Gazoontite Wed 2/27 Mon 3/4 servlets are a mess -- after hallway discussion w/ Erik, proposed naming convention to fix UI needs more back-and-forth before settling, IMHO We probably need different colored cards for tasks and stories Wed 3/6 Last night I was pair programming with two different people. I didn't type a line of code. It was very creative/enjoyable for me, and I caught many typos, and at two separate points, when my partner was going off on a tangent, I stopped him. Once Erik was writing all sorts of code and getting confused, so I made him give me the keyboard and wrote four comments -- "// submit the thing... //check the response... //submit the next thing... //check the response" -- that clarified the task. And Jim was in the middle of writing error-checking code, and I told him, "You know, we should bail out right now, since we haven't written tests for these error cases, and we should get the base case working first." He agreed immediately and commented out what he'd done, and the rest went more smoothly. We finished the base case just before leaving; the errors will wait until next time. My point is, the voices in your head are good, but an actual voice in your ear is better. OK, so we're killing Task Estimates / Individual Velocity. And every task Monday 3/11 Task Cards are falling by the wayside. However, we want to get a UI MOCKUP task and an ACCEPT task for each story. If we're casual with tasks, then this could be bad, but I think it'll be OK. Maybe we'll do a talk, maybe based on these notes. Wed 3/13 "What to do if the customer's not here yet?" "Work, but if a customer issue comes up, note it down." (Ian said some jargon -- "operate buffer"?) First big refactoring. Jim had to be convinced to do teeny little steps (or maybe he was just being taciturn). We merged the silly BookListFactory and BookRequestFactory into one Library. Unfortunately, this caused a hellish merge, since we modified a file that Ian and John had deleted, and they had to do CVS magic to ressurect it and then kill it for good. We were quartet programming trying to resolve that. "cvs add" in this case means "cvs really remove" (it can also mean "cvs ressurect") Mon 3/17 Customer test note - split per URL or per story? Ian and Alex kept refactoring Library John and Jim flipped coin to decide which task card to pull Customer was not finished with acceptance tests again. Maybe we should allocate 2nd-priority developer tasks to helping customer do this. Then when we're done with our 1st-priority story, we go help the customer instead of picking a different story or refactoring. Mon 3/25 I3 wrapup: again with the unfinished customer tests I4: Decided to make customer pair a priority -- make sure he's got a developer with him to finish the tests New people, here for JavaOne XP seems *very* resilient to turnover. New people, missing people, get absorbed. I attribute this to (a) pairing promiscuously -- the more experienced one helps the newbie and still gets advantage of monitor (b) small tasks -- you can pick up a task at will, and finish within 3 hours, so the team doesn't have to wait for you to come back Alex: paired with Erik on customer tests Wed 3/27 Alex, Erik, Charlie, Jim, Alex2 (new) Eclipse UG (Charlie left early) Erik and Alex refactored customer tests Customer becomes developer -- Erik and Alex added an id to the browse form. This is OK because we are a pair, so we can futz with production code. Mon 4/1 Wed May 8 Customer tests - we're on a serious lag, still... tracking needs improvement... we honestly need people who finish-- no, *start* a story to write the acceptance tests for it... UI - last week, we did a pairing (Tom + Alex) to make a mockup on Monday, then on Wednesday Alex implemented it in Java while Tom did a spike design session solo. Then this week, Monday Tom and Jim turned the solo JPG into a mockup; then Wednesday Tom and Lawrence paired on finishing the mockup, then starting the Java. The theory is, the UI designer is a developer (with special skills). For each UI story, he will do a spike solo, then pair to make an HTML mockup. Then a different pair (not necessarily including a turns the mockup into functional code; also, a different pair turn Story: Foo Task: UI Spike for Foo Task: HTML mockup for Foo Task: Acceptance Test for Foo Task: UI Java code for Foo Task: Implementation of Foo This may get easier if we use JSP or XMLC, but not much. 3rd time's the charm -- Walter and Alex refactored Library, trying to extract an Item (Book) class in order to (eventually) add 4 new fields. Previously, two different pairs had attempted this but failed to check anything in. So we kept it *REALLY* small steps. First did lots of refactoring-to-understand of Library, and renamed methods with the word "Book" in them. Then left all the title-based stuff exactly as-is, but added a getItem(title) method that returned an object with one field (title). Then we FITYMI until we got an itemList -- no, an itemMap -- and now we're set to just make persistence work for that map and its item contents. cf. Refactoring p. 175 Coding Standard - We seem to have decided on a simple standard, based on one or two emails. Valerie reacted, but we didn't... --- Wk. of 5/14 Apparently our supershort iterations have led to a few bad effects: - Acceptance lag (no time to write tests) - Spaghetti code (no time to refactor) Maybe we should spend *as much* time refactoring as coding -- e.g. spend 5 minutes writing test, 5 minutes wrtiting code, 5 minutes refactoring I need to switch off more when pairing. I started using the kitchen timer: 15 minutes, ding, switch. --- Date: Mon, 20 May 2002 21:24:45 -0700 Subject: [bootstrap] status and insight Reply-To: bootstrap@yahoogroups.com * I'm assuming the thankless customer role for the time being. * We'll have a "real" iteration meeting next time. * Thanks to a new "RemoveAll" servlet, and the gratuitous calling of it, the customer tests are now down to a consistent <1min. Persistence dropped from 20-40 seconds per test to <5 sec/test. * Jim and I were discussing the duplication between Customer and Unit tests for UI, and we seem to agree that it would be better for all concerned if we made the following policy: For strictly Web-based UI tasks, don't bother writing a separate unit test -- just go straight to the customer tests. This will take care of the duplication -- which becomes a pain to maintain when we /change/ the UI, as we've done a few times lately. (Also, if it's a new story, make a CustomerTest object named after the story, to keep it clear what story they apply to.) * The process of (A) implement story then (B) immediately bug the customer and don't go away until the customer tests are written worked great tonight! I (as customer) would fiddle and refactor and work on old tests until a pair finished a story, then we'd work together until it was done. This actually uncovered a few bugs, which the developer went off and immediately fixed. Maybe this means that there shouldn't be a spot on the board for "Completed But Not Accepted" -- i.e. until it's accepted, you can't pick a new card. Date: Tue, 21 May 2002 09:39:36 -0700 Subject: Re: [bootstrap] status and insight Reply-To: bootstrap@yahoogroups.com On Mon, May 20, 2002 at 09:24:45PM -0700, alex@jguru.com wrote: > > * Jim and I were discussing the duplication between Customer and Unit > tests for UI, and we seem to agree that it would be better for all > concerned if we made the following policy: > > For strictly Web-based UI tasks, don't bother writing a separate > unit test -- just go straight to the customer tests. > > This will take care of the duplication -- which becomes a pain to > maintain when we /change/ the UI, as we've done a few times lately. Clarification: by "UI" here I should say "HTML", since tests like making sure form-handling servlets actually change the data model may be important to unit test as well (by looking directly into the data objects, which can only be done by unit tests, not customer tests). So maybe the above policy should apply to *Page* servlets only... It's a judgement call where to draw the line, but I think we agree the duplication is becoming unmanageable, so it may be time to move the line closer to "developers can write customer tests." Another thing this may do is encourage more eyes to look at the customer tests, which could help refactor it so it's not such a tangled mess. --- 5/22 Charlie and I refactored the customer tests -- extracted a Robot class. Getting easier to read. "Customer is there to answer questions -- writing acceptance tests is secondary." --- 5/29 Alex and Alex kept working on ctest -- managed to move asserts to MyTestCase, much cleaner. Lawrence and Skot still couldn't make much progress on Blue UI. There's something wrong with the process. HTML sucks. --- 6/4 Since the customer doesn't have to write all the customer tests any more, it frees him up for other stuff. So I get to pair with someone, as time allows. I also get to play coach and/or tracker. New policy: if there's a config problem, or a bug, ASK FOR HELP. We don't want anyone to get bogged down. Is HTML hard, or is acceptance testing hard? We should track attendance -- it'll be fun. Why did the release slip? Jim says "bad scoping" -- we identified that we could do /either/ UI changes, /or/ add multiuser. We ended up trying to do both, /plus/ we had a whole new great-looking UI. That's three releases worth (?). We also need to be better about divvying stories into tasks and discussing the tasks before jumping in. Emacs is hard for some people, so we need to make an effort to have at least one non-emacs machine. My G4 Cube is running Project Builder, so that's a start. New motto: Don't believe the mockup! Even if it looks like something, ask the customer. E.g. the mockup said to sort by status (available items, then lent items), but the customer doesn't care; I'd rather have you add the icons or whatever. "Sort by Status" then become a new story I can (de)prioritize. We now need to start refactoring xpaths out from the unit tests, following the lead of the customer tests. 6/10 Jeffrey is bringing some good terms. "Hang a bag" = put some quick-and-dirty code on the side of a cleaner module. "Law of Demeter" = a method should refer only to objects that are (a) passed in, (b) created by it, or (c) returned by methods it calls (including factories). That is, don't follow pointer chains. The test refactoring is proceeding. It's tedious to pull out the xpaths, let alone to unify them. Alan Shalloway stopped by. We had a nice long chat. Some highlights: Why Refactor? Increase Clarity and Cohesion, Reduce Coupling and Redundancy (Duplication) Patterns and Refactoring lead to the same place via different routes. It turns out some people's brains work better at a low level (Alex) or at a high level (Alan), but the goal is to improve the above (clarity et al.). XP's concept of "simple" means those principles too, plus testable. Let's say you had a deep hierarchy embodying two entangled concepts. (Chip -> (ChipA -> (TCPChipA, FTPChipA), ChipB -> (TCPChipB, FTPChipB))) Patterns would say you need a Bridge. Refactoring would say, extract the similar methods (either into a helper class or the superclass), then unify them, then move them to the side, then extract a hierarchy, and what you end up with looks exactly like a Bridge. Jim (and Scott) did some refactoring of Library and ItemList, in preparation for multi-user, but also just to repay some design debt. Refactoring needs to be continuous, or it's too much of a chore and you end up not doing it often enough. Without Refactoring -- when you're just doing abstract design -- it's hard to notice where these common principles can be applied. But with refactoring, with your nose to the ground, you can tell where and how to improve the design. Refactoring and Design Patterns are bottom-up and top-down views of the same thing. 6/12 ... 6/24ish: "If it *almost* makes sense now, it *may* make sense in the future, so let's leave it in." -- totally non-YAGNI refactoring advice from Alex (turned out it did make sense, by the end of that session) Refactoring works: Jim and I wanted to add persistence to Users as well as to Library. We had already decided that it would be written to the same file. After we wrote the tests, Jim suggested we go to the place in Library where it called saveSelf() and just add a call to Users.saveSelf(). I demurred, suggesting instead that we perform a refactoring: extract the persistence methods into a separate LibraryPersistence object, then rename the object Persistence, then finally add the call to Users.saveSelf(). Jim agreed, and we're glad we did it that way. In extracting the persistence code we found it was tangled, and ended up with a better design. What to do about stories that last too long for one iteration? Normally, one would think that it means we should work more on breaking them down. However, we're already going on absurdly short iterations, so this is probably a function of that: the stories are already broken down as small as they can reasonably be, but the natural length is still greater than one iteration. (Remember, most XP projects have 80-120 hour iterations; we have 12-hour iterations.) It's still a problem for tracking/accounting velocity. What about bugs? Do we go zero-defect or not? --- 8/6 Refactoring is Fun (or, Refactoring is Hard) The past three sessions, my pairs and I have been refactoring the Library and Item classes to be more object-oriented. This follows one of my big revelations, which I'm sure every Extreme Programmer must learn, and must constantly relearn: Simple Design trumps YAGNI (meaning that writing clean, clear, object-oriented code is very important, and that if you have a choice between (a) leaving your code smelly and implementing a new feature, or (b) refactoring and letting the feature wait a little while, you should refactor.) Here, we were fixing some less-than-OO stuff that happened to be left over from a long long time ago: - passing Item object references instead of Title strings - passing User object references instead of Email strings - mapping from Item object to Request List instead of from Title string to Request List - storing User/Item in the Request object, instead of Email/Title Eventually, these will lead up to some refactorings that will be actively useful, like - using unique IDs for Items, instead of using Title as the key (this will allow us to have multiple items with the same title, which is a story) - changing the unique part of a User from Email to Username (this will allow us to keep email addresses private, which is a story) The initial refactorings have fairly large scope -- each method signature change had a lot of usages. Hopefully, once we get all these done, and we're passing around objects instead of key strings, the later refactorings will be trivial. That is, once we're passing around Item references, most of the code won't care /how/ we identified the item in the first place. We'll just have to change the code that creates/loads items to use the new ID key instead of the old Title key. There are basically two ways to do a refactoring with fairly large scope. The first way I call "Nose Job" -- in order to do a nose job, the plastic surgeon first breaks the nose, then reassembles all the broken bits into the new design. So the Nose Job Refactoring means, first change *all* the method signature(s) to match the new design, then compile and see what breaks, and fix all the broken callers. If you're lucky, and the refactoring is straightforward enough, then once it compiles, all tests will pass and you're done. (In this case, the Nose Job would change *all* methods in Library that take a parameter "String email" to "User user", then walk through *all* the calling code and clean up.) Unfortunately, this only seems to work about half the time. The other half, even once you change all the callers, some unit tests fail. If you're lucky, you can easily fix these, but if not, you have to roll back *all* your changes (sob) and do the more laborious, incremental, Martin Fowler Approved kind of refactoring: change one signature, fix all callers, pass all tests, check in, repeat. The reason Nose Jobs are preferable is that you can get into a Flow state walking through a file and changing things to the new design everywhere. In the Incremental refactoring, you're often in a file changing one caller, and you see a line of code calling a different method the wrong way, and you just *itch* to clean that up too, and you *know* it would take just one second... But you have to control yourself, since you know that that one just might be the evil caller where the semantics changed enough to cause a test to break. (What do I mean by causing a test to break semanticaly? In one case, an XPath expression failed to find a match, and it wasn't a simple thing like accidentally forgetting to change out.print(user) to out.print(user.getEmail()) (leading to "expected: but was:"), but a non-obvious bug.) The bigger lesson, of course, is Don't Check Your Brain At The Door and that we should have been doing these OO refactorings all along. Maybe even as soon as we got the code to work using "String title", we should have taken half a step back, applied the "Replace Value With Object" refactoring, and been satisfied that we had made the code more aesthetically pleasing and therefore better -- -- *even though* it wasn't to satisfy a user story. IOW, refactoring is a *technical* decision, not a business decision, and the customer has no right to tell us not to do it (even if he does so implicitly, or if it's just our own puritanical super-egos saying "Work! Be more productive! Don't refactor, write a new story!"). --- 8/6 Me posting to the XP list: On Tue, Aug 06, 2002 at 10:29:58AM -0700, C. Keith Ray wrote: > > A programmer's first realization during pair programming is (typically, I > think) discovering that he is not smart as they think they are. He make > terrible stupid mistakes, and this other person sees it all. > > Then his second realization is that the other programmer isn't that smart > either -- you see her making stupid mistakes too. This happened to my pair last night! I was paired with someone with less pairing experience than me [skot]. At one point I was driving, and I heard him sort of sighing, or grunting, or at least not being as vocal as he usually is. I stopped and asked him, "What's wrong?" He said, "Oh, nothing." I persisted. "No, really, what are you thinking?" He admitted, "I'm just not sure why you're doing things that way." I didn't understand. "What way?" It turned out that I had mistakenly declared a new local variable when I should have been using the already-declared instance variable. He'd found a bug! But his first impulse was to assume that I was too smart to make such a simple (if insidious) mistake, and then try to understand the brilliant new algorithm I'd come up with, and to berate himself (or at least to take pause) for not following along. I trust that next time, he'll realize what a doofus I am, and catch my bugs more vocally :-) --- 8/12 Erik says Valerie is *really* close to actually using it. (I've heard *that* before... :-)) We got zero points done last iteration. Time was mostly spent refactoring. Adjusting to Local Conditions Department: it has become clear that we are not refactoring enough. Our code is smelly and oddly-designed. Furthermore, with people coming and going, it is difficult for individuals to keep track of the code base. As a result, features that should be straightforward to implement end up taking too long, because of the effort required to "wrap your brain around" the code. As an attempt to solve this, we started off Monday's coding session with a Code Review. All four of us (Alex, Erik, Skot, Angela) sat down at a computer and looked over the Data Model objects (Library, Item, User, Request, Persistence), making a UMLish diagram as we went. It was simply amazing how code smells leapt off the monitor at us. We got some good refactorings done as a group. Although the common wisdom on pairing is, "two's company; three's a crowd," it didn't seem to hinder us that there were four of us. Perhaps this was because we weren't working on new code, which would have required more thought => more communication => more interference with four. This is my big current XP lesson: what Refactor Mercilessly really means. Several months ago, I thought R.M. meant to spend as much time refactoring as writing new code. Now I think I was off by at least a factor of 2 or 3. You should spend *twice* as much time refactoring, and you should do it *before* writing new code (as well as after, and, if it doesn't interrupt your train of thought, during). Current Stories: * Replace "email" with "username" * About Page (information and credits) * Secure Passwords Upcoming Big Tasks/Refactorings: * Use XML (or just text) Persistence (instead of serialization) (the customer says it's OK to break backwards compatibility for this change, since we don't have any real data to save yet) * Move data objects to polonius.data package (depends on XML Persistence -- I'll tell you the gory details on why if you really want to know) * Pull out the Request management methods from Library (right now there are about 10 public methods and a handful of privates dealing solely with requests, and Library feels cluttered; I suggest either extracting them into a Librarian object or extracting the others into an Inventory object... maybe we need a 2-minute Metaphor brainstorming session to come up with the right object meodel) * Change the item map into a list (since lists are easier to understand, at the cost of making the getItemByTitle slightly more complicated) * Change Users from a static into an instance or a factory. In addition to just being cleaner, this would allow us to test more efficiently. Right now every test writes out everything to disk. If instead the Library could accept a Users from outside (probably passed into the constructor), then the unit tests could just make an in-memory Users containing only our test users. * Change Library from a static into an instance or factory. For similar reasons as above. This would also help prepare for the "same title, many owners" story. * Finish implementing new HTML ID scheme * Clarify ID concept -- right now we have many meanings of ID (HTML, UserID, ItemID, LinkableID) ----------------------- 9/16 First Release Planning meeting Attendance: Notes: Went well 9/17