Does your company need help testing new software? In the book, How Google Tests Software, legendary expert James Whittaker and two Google experts show you new techniques and the best practices you can use for testing software.
Here’s a excerpt from the book as the experts interview YouTube test engineer Apple Chow. Enjoy!
An Interview with YouTube TE Apple Chow
‘Apple Chow is a TE for Google Offers and before that a test lead for YouTube in Google’s San Francisco office. Apple likes new challenges and is constantly looking to leverage the latest tools and techniques for testing.
The authors recently chatted with Apple about her thoughts on testing in YouTube.’
HGTS: Apple, what brought you to Google? And with a name like yours, surely you thought about employment elsewhere?
Apple: Ha! firstname.lastname@example.org is very tempting! But I came to Google because of the breadth of our product offerings and the opportunity to work with really smart and knowledgeable people. I like to change projects and I love a wide variety of challenges, so Google seemed like the right place for someone like me. I get a chance to make a difference to millions of users across a lot of different product areas. Every day is a new challenge and I never get bored. Of course the free massages are definitely a plus.
HGTS: What did you think of the interview process for TEs and SETs?
Apple: Google focuses on finding generalists who can learn, grow, and tackle a wide variety of problems. This goes for TEs, SETs, and SWEs in my opinion. A lot of places interview for specific roles on a specific team and the people you meet are all going to be people you will work with closely. A Google interview doesn’t work that way. The people interviewing you are from a variety of different teams, so you get a lot of different perspectives. All in all, I think it’s a process designed to get good people who can work on almost any team at Google. This is important, too, because it’s easy to move around within Google so you can always choose a new product area to work in, a new team to work with. Being a generalist is important in this kind of a structure.
HGTS: You have worked at many other tech companies. What would you say was the most surprising thing about software testing at Google?
Apple: A lot of things are different. Perhaps I am biased because I like Google so much, but I would say that our TEs and SETs are more technical than at most other companies. At other large companies I worked for, we had specialized automation teams and then a bunch of manual testers. SETs at Google have to write code; it’s their job. It’s also rare to find a TE who can’t code here. These coding skills allow us to be more impactful early on when unit testing is far more prevalent and there’s nothing really to test end-to-end. I think our technical skills are what make us so impactful here at Google.
Another thing that makes Google unique with respect to test is the sheer volume of automation. Most of this automation executes before manual testers even get hold of the product. When they do, the code they get to test is generally of very high initial quality.
Tooling is another difference. In general, we don’t use commercial tools. We have a culture where tooling is greatly appreciated and 20 percent time makes it so that anyone can make time to contribute to the internal Google toolset. Tools help us get past the hard and repetitive parts of testing and focus our manual efforts to really impact things where a human is actually required.
Then, of course, there is the developer-owns-quality and test-centric SWE culture we have here that makes it easy to relate to SWEs. We’re all in the quality game together and because any engineer can test any code from any machine, it makes us nimble.
HGTS: What things would you say are pretty much the same about testing at Google?
Apple: Software functionality that is hard to automate is just as hard to test or get right as at any other company. When there is a huge rush to get features out, we end up with code that isn’t as well tested as we’d like it to be. No company is perfect and no company creates perfect products.
HGTS: When you were a TE for YouTube, what feature areas were you responsible for?
Apple: I’ve worked with many teams and helped launch many features at YouTube. Some notable mentions would be the launch of the new Watch page that is a complete redesign of the YouTube video page, one of the most viewed pages on the Internet I am happy to say! Another memorable project is our partnership with Vevo. It is a new destination site for premium music content with YouTube powering the video hosting and streaming. It’s a joint venture with Sony Music Entertainment and Universal Music Group. On day one of the launch, more than 14,000 videos went live, and they averaged 14 M views on VEVO premium videos on YouTube.com for the next three months, following the December 8, 2009 launch. I also coordinated test efforts for the major rewrite of the YouTube Flash-based video player during our move from ActionScript 2 to ActionScript 3, and the launch of the new Channel and Branded Partner pages.
HGTS: So what does it mean to be a lead in testing at Google?
Apple: The lead role is a coordination role across the product, across the team, and across any product that our work might impact. For example, for the Vevo project, we had to worry about the YouTube player, the branded watch component, channel hierarchy, traffic assignment, ingestion, reporting, and so on. It’s definitely a “forest and not trees” mindset.
HGTS: How have you adapted the concepts of exploratory testing to YouTube?
Apple: With a product that is so human-oriented and visual as YouTube, exploratory testing is crucial. We do as much exploratory testing as we can.
HGTS: How did the YouTube testers take to the idea of exploratory testing?
Apple: Oh, it was a huge morale booster. Testers like to test and they like to find bugs. Exploratory testing increased the level of engagement and interest among the testers. They got to put themselves in the mindset of the person in the tour and, with those specific angles, got creative with the types of tests they conducted to break the software. This made it more fun and rewarding as adding more tests revealed interesting and esoteric bugs that would have otherwise been missed or discovered through more mundane and repetitive processes.
HGTS: You mentioned tours. Did James make you use his book?
Apple: When James first came to Google, that book was new and he did a couple of seminars and met with us a few times. But he’s up in Seattle and we’re in California, so we didn’t get much hand holding. We took the tours in the book and ran with them. Some of them worked, some didn’t, and we soon figured out which ones were the best for our product.
HGTS: Which ones worked? Care to name them?
Apple: The “money tour” (focus on money-related features; for YouTube, this means Ads or partner-related features) obviously got a lot of attention and was important for every release. The “landmark tour” (focus on important functionalities and features of the system) and the “bad neighborhood tour” (focus on previously buggy areas and areas that we find to be buggy based on recent bugs reported) have been most effective in uncovering our most severe bugs. It was a great learning experience for each one to look at the bugs others in the team had filed and discussing the strategy in finding them. The concept of tours was really helpful for us to explain and share our exploratory testing strategy. We also had a lot of fun joking about some of the tours such as “antisocial tour” (entering least likely input every chance you get), “obsessive compulsive tour” (repeating the same action), and the “couch potato tour” (provide the minimum inputs possible and accepting default values when you can). It was not only helpful to guide our testing; it built some team unity.
HGTS: We understand you are driving a lot of Selenium testing of YouTube. What are your favorite and least favorite things about writing automation in Selenium?
Least favorite: It’s still browser testing. It’s slow, you need hooks in the API, and tests are pretty remote from the thing being tested. It helps product quality where you’re automating scenarios that are extremely difficult for a human to validate (calls to our advertising system backend, for example). We have tests that launch different videos and intercept the Ad calls using Banana Proxy (an inhouse web application security audit tool to log HTTP requests and responses). At a conceptual level, we’re routing browser requests from browser to Banana Proxy (logging) to Selenium to Web. Thus, we can check if the outgoing requests include the correct URL parameters and if the incoming response contains what is expected. Overall, UI tests are slow, much more brittle, and have a fairly high maintenance overhead. A lesson learned is that you should keep only a few such high-level smoke tests for validating end-to-end integration scenarios and write as small a test as possible.
HGTS: A large portion of YouTube content and its UI is in Flash; how do you test that? Do you have some magic way of testing this via Selenium?
HGTS: What was the biggest bug you or your team has found and saved users from seeing?
Apple: The biggest bugs are usually not that interesting. However, I recall we had a CSS bug that causes the IE browser to crash. Before that we had never seen CSS crash a browser.
One memorable bug that was more subtle came up during the new Watch Page launch in 2010. We found that when the user moves the mouse pointer outside of the player region, in IE7, the player would freeze after some time. This was interesting because users would encounter this bug if they were watching the same video for an extended period of time and moving the mouse around. Everything got slower until the player finally froze. This turned out to be due to unreleased event handlers and resources sticking around and computing the same things over and over again. If you were watching shorter videos or being a passive viewer, you wouldn’t observe the bug.
HGTS: What would you call the most successful aspect of YouTube testing? The least successful?
Apple: The most successful was a tool to fetch and check some problematic URLs. Although it was a simple test, it was really effective in catching critical bugs quickly. We added a feature to make the problems easier to debug by having it provide stack traces that the engineers could then use to track down problems and develop fixes. It quickly became our first line of testing defense during deployment and brought along considerable savings in testing time. With only a little extra effort, we extended it to hit the most popular URLs from our logs plus a list of hand-picked ones. It’s been very successful.
The least successful is probably our continued reliance on manual testing during our weekly pushes. Given that we have a very small time window for testing (code goes out live the same day it’s frozen) and we have a lot of UI changes that are hard to automate, manual testing is critical in our weekly release process. This is a hard problem and I wish we had a better answer.
HGTS: YouTube is a very data-driven site as much of the content is algorithmically determined; how do you verify that the right videos are displayed the right time and place? Does your team verify the video quality? If so, how do you do this?
Apple: We measure how much and which videos are being watched, their relationship to each other and a whole lot of other variables. We analyze the number of buffer under-runs and cache misses, and we optimize our global-serving infrastructure based on that.
We have unit tests for video quality levels to make sure the right quality is used. After I changed groups, our new team wrote a tool to test this in more depth. The tool is open-sourced29 and it works by having FlexUnit tests that use the embedded YouTube player to play a variety of test videos and make some assertions about the player state and properties. These test videos have large bar codes on them to mark frames and the timeline that are easily recognizable despite compression artifacts and loss of quality. Measuring state also includes taking snapshots of the video frames and analyzing them. We check for the correct aspect ratio and/or cropping, distortion, color shifts, blank frames, white screens, synchronization, and soon—issues found from our bug reports.
HGTS: What advice do you have for other testers of Web, Flash, and data-driven web services out there?
Apple: Whether it’s a test framework or test cases, keep it simple and iterate on the design as your project evolves. Don’t try to solve everything upfront. Be aggressive about throwing things away. If tests or automation are too hard to maintain, toss them and build some better ones that are more resilient. Watch out for maintenance and troubleshooting costs of your tests down the road Observe the 70-20-10 rule: 70 percent small unit tests that verify the behavior of a single class or function, 20 percent medium tests that validate the integration of one or more application modules, and 10 percent large tests (commonly referred to as “system tests” and “end-to-end” tests) that operate on a high level and verify the application as a whole is working.
Other than that, prioritize and look for simple automation efforts with big pay-offs, always remembering that automation doesn’t solve all your problems, especially when it comes to frontend projects and device testing. You always want smart, exploratory testing and to track test data.
HGTS: So tell us the truth. YouTube testing must be a blast. Watching cat videos all day …
Apple: Well, there was that one April Fool’s day where we made all the video captions upside down. But I won’t lie. Testing YouTube is fun. I get to discover a lot of interesting content and it’s my job to do so! And even after all this time,
I still laugh at cat videos!
This excerpt is from the book, “How Google Tests Software’, authored by James Whittakeer, Jason Arbon and Jeff Carollo, published by Pearson/Addison-Wesley Professional, March 2012, ISBN 0321803027, Copyright 2012 Pearson Education, Inc. For more info please visit the publisher site, www.informit.com/swtesting
Over the past few weeks, rumors surrounding RIM’s declining financials continue to be a huge topic in the mobile device industry.
According to DailyFinance, RIM recorded a net loss of $125 million last quarter and revenue declined by 25 percent. Reports are surfacing that RIM has hired a law firm to work out a restructuring plan for the company.
Where did RIM go wrong? I remember just a few years back, everyone had a BlackBerry. You weren’t a part of the “cool crowd” if you didn’t have one.
Now, where has it gone? All you see is iPhones and Androids. Recently, a friend of mine got rid of his BlackBerry and bought an iPhone. Why?
“Because it’s better,” he said.
That’s the problem for RIM. As Apple and other smartphone companies were coming out with new phones or upgraded features, RIM hit a standstill.
In December 2011, RIM needed to postpone the BlackBerry 10 because their operating system was a complete mess.
“RIM is simply pushing this out as long as they can for one reason, they don’t have a working product yet,” a high-level RIM employee told Boy Genius Report.
RIM released a statement to All Things D regarding the BlackBerry 10:
RIM made a strategic decision to launch BlackBerry 10 devices with a new, LTE-based dual core chip set architecture. As explained on our earnings call, the broad engineering impact of this decision and certain other factors significantly influenced the anticipated timing for the BlackBerry 10 devices. The anonymous claim suggesting otherwise is inaccurate and uninformed. As RIM has previously explained, and as Mike Lazaridis reiterated on the earnings call, we will not launch BlackBerry 10 devices until we know they are ready and we believe this new chip set architecture is required to deliver the world class user experience that our customers will expect. Any suggestion to the contrary is simply false.
Even with RIM’s explanation, users can assume the BlackBerry 10 won’t be coming out for months.
So this is what were left with: a once promising mobile company being eroded by its competitors and themselves.
Could there be a comeback for RIM?
Well, I would put its chances at slim but not none. RIM CEO Thorsten Heins is very innovative but also is inclined to make mistakes and says what’s on his mind.
It could very well come down to the BlackBerry 10. If it succeeds with users, then it might be back in business. But, for now, RIM is in jeopardy.
Michael Tidmarsh is the Assistant Community Editor for ITKnowledgeExchange.com. He can be reached at Mtidmarsh@techtarget.com.
Despite grand ambitions at the start of the recession, Cisco’s been in a slump: Its stock price has struggled, its consumer ambitions have been shattered and a few rounds of layoffs have tried to focus the company on what matters to its core business, network dominance.
That focus will be put to the test soon as new competitors from above and below take aim at Cisco. The marquee name in this battle is none other than Google, which recently previewed its OpenFlow initiative to Wired.
Cybercriminals are on the attack as they have set their sights on a new target: cloud-based payroll service providers.
According to the security firm Trusteer, they have come across a Zeus malware configuration that targets Ceridian, a payroll service provider.
Trusteer’s chief technology officer Amit Klien explains in a blog post how the Trojan is attacking these cloud service providers.
“Zeus captures a screenshot of a Ceridian payroll services webpage when a corporate user whose machine is infected with the Trojan visits this site. This allows Zeus to steal the user id, password, company number and the icon selected by the user for the image-based authentication system,” he said.
“These attacks are designed to route funds to criminals, and bypass industrial strength security controls maintained by larger businesses,” Klien said.
Ceridian released a statement emphasizing that no security breach on its own servers had occurred and that the vulnerability targets customers’ computers and targets a wide range of SaaS services.
“Ceridian has not experienced a security breach as implied by this article,” Donna Teggart, Ceridian’s director of communications wrote in a statement. “A Zeus infection happens at the customer computer location and will capture all the user’s keystrokes, regardless of the application they are logging into. Ceridian encourages all individuals and organizations to ensure they are protecting their computers and networks from all threats and virus attacks such as this.”
This could only be the beginning as Trusteer reports cybercriminals are attacking small cloud-based providers to create easier ways to attack larger businesses. Continued »
Taking cracks at bad business software design is beyond beating a dead horse (although I still love the famous tree swing comic), but Microsoft looks like their trying really, really hard to turn that around. Leading the charge: Microsoft’s ERP package, Microsoft Dynamics GP. A beta Metro-ized version of it was shown off recently, and design is gorgeous to look at. Let’s play a quick game of before and after:
GP 2010 R2, the most recent version
The Metro-ized UI, demoed recently
My first reaction: Microsoft, the same company inflicting us with the Office Ribbon, made this?
My second reaction: But will it blend?
Well, Windows 8 is now officially slated for an October released and the earlier reviews are positive (I’ve downloaded the release candidate but haven’t installed it yet). Already, Mr. Denny is putting together excellent troubleshooting tutorials and IE10 is getting excellent marks from Ed Tittel.I don’t think a Windows release has garnered so much excitement and enthusiasm since WindowsXP, released 10 years ago. And for good reason:Windows 8‘s “Metro” Interface represents the biggest departure from the traditional windowing paradigm since Windows launched, and WindowsPhone 7 has proven that Microsoft is capable of making a well-designed OS with it, even if it’s not a complete market success yet.
But that excitement and those revisions comes at a cost: Radically different means what has worked for years is heading the way of the Dodo, and retraining, rebuilding and restructuring are all going to become part of the upgrade, especially difficult for the OS ecosystem that has bent over backwards for backwards compatibility. Yes, a more traditionalWindows 7-ish interface lies just beneath Metro, but that’s a bandaid, or as Tony Bradley snarked, “Windows 8 feels like Windows 7 with Metro added as an additional, frustrating layer I have to work through to get to the features and capabilities I actually want to use.”
I bet that will be a common thread among two groups in particular: Power users and computer novices used to things exactly the same as they’ve been (See viral video for demonstration). Change is inevitable and I think Microsoft is making the right strategic choice, but it’s also a good opportunity for enterprises to ask themselves which platform will they embrace for the future: Windows 8‘s bold but uncertain moves? Apple’s polished, pricey and enterprise-indifferent strategy? A web application suite that they can better deploy – but at the cost of endpoint control?
I’d love to hear your thoughts, since the only thing I know for certain is that there are no easy answers. E-mail me at Michael@ITKnowledgeExchange.com, and if we like your response we’ll even try and select a great book or other swag to send you.
Recently I spoke with John Horn, president of RACO Wireless, about the past, present and future of wireless connectivity.
RACO, a T-Mobile partner, specializes in wireless data solutions for machine-to-machine (M2M) industries, which allows wireless systems to connect with other devices. John explains how M2M is now being used all across the world.
“Each industry has become integrated with M2M technology. From the insurance industry which uses it for monitoring discounts to even the ice industry monitoring their inventory. It’s important to show that M2M has become mainstream,” he said.
As Brad Pitt said in Inglorious Basterds, ‘Business is a boomin’. Raco has seen a 300% growth in 2011 and has provided new solutions for providers in less than a day.
How has RACO been able to perform and grow at such a high level? John explained the keys to his success came down changing the way he looked at wireless altogether.
“The team has been together for over 9 years and has 200+ years of M2M experience. This translates into a wealth of knowledge to help our customers build successful business models. We have seen what works and what doesn’t and know what trouble spots to look out for,” Horn said.
The keys, Horn said, are flexibility and the ability to quickly turn around a solution that’s right for the business.
“The old business model was to have companies go through long certification processes in order to put a new device or product on the carrier network,” he said. “This takes thousands of dollars and many weeks. Then you have to buy a rate plan that exists from a carrier or wait for weeks for them to build a new one. Then it takes weeks to connect into whatever management platform they are using and tie up IT resources to get it accomplished.”
Now, it takes John and Raco only a few short hours to complete the process.
“We created a whole new business model which allows us complete flexibility and shortens the business model to one day. We have eliminated or simplified these steps”
With all the company’s growth, it still faces challenges. While M2M is growing, its still a niche many people don’t know about, and RACO is struggling to become a household name.
“The biggest problem we have right now is marketing, getting our name out there,” he said. “We print ads and appear at trade shows but we’re also looking into other avenues to explore.”
However, Horn sees a bright future for the M2M industry. “We have seen massive growth. It comes down to if you support it or not. If you support it, you will come out a winner.”
One story in particular drifting out of South By Southwest caught my attention: The outrage and indignity over a trial/marketing stunt program which gave Austin-area homeless individuals a 4G “hotspot” that nearby techies could log in to and browse the web, while introducing the wireless vendor and asking users for a small PayPal or cash donation.
The reaction was as swift as it was predictable. Wired’s excellent Tim Carmody blasted the Damning Backstory Behind ‘Homeless Hotspots’ at SXSW, while others took the initiative as another sign of the tech conferences jumping the shark – or worse, how out of touch the digerati are with real world problems.
To quote Carmody:
It sounds like something out of a darkly satirical science-fiction dystopia. But it’s absolutely real — and a completely problematic treatment of a problem that otherwise probably wouldn’t be mentioned in any of the panels at South by Southwest Interactive.
This program and the immediate media backlash reminded me of why so many promising, innovative projects inevitably sputter out, whether its in the world of startups, social work or plain vanilla corporate IT.
RSA 2012 has come and gone from the caverns of Moscone, and I’ve had a (short) chance to digest this year’s event, leaving a little more educated and a lot more wary about the risks facing modern IT when it comes to security.
The biggest wake up call? Security expert Bruce Schneier’s timely reminder that outsourcing, whether to India or the Amazon Cloud, has ripple effects on security and privacy, and that right now the trend is to cut costs and complexity – in exchange for control. That’s not necessarily a mistake, particularly for businesses that are rapidly expanding, businesses that were hit hard by the recession, businesses that need to quickly adapt to a mobile landscape, or pretty much any other business that can benefit from the agility the cloud offers. In other words, the cloud offers a little of something to everyone.
But the allure of the Google way, or even the Microsoft Azure or Amazon S3 way, costs something, whether it’s an increased chance a competitor can sneak a peak at your proprietary data, that a government can subpoena your records or simply that you can’t control when and where outages hit home.
The biggest threat to Internet freedom isn’t traditional “bad guys” like cyberterrorists and hacking groups, says Bruce Schneier, security researcher and author, but the slow, creeping advances of Big Data companies like Google and Amazon that are quietly rewriting the fundamentals of how security is managed.
Schneier explained his fears to a packed room at RSA 2012, outlining how he saw individuals, companies and governments effectively outsourcing security to cloud providers, abdicating ultimate control in exchange for convenience and cost savings.
The result is a state of “security serfdom” where fealty is pledged to one of a few centralized data gatekeepers who promise and deliver great benefits – but upon whom the user becomes completely reliant for basic security. Apple’s legion of adoring gadget geeks and people who live the “Google lifestyle” through GMail, Google Voice and more now rely on those companies to make critical security decisions for them.
It’s not an all together negative trend, particularly since “average users” historically do the bare minimum of backup, encryption and other information security hygiene possible, but it does create a more monolithic landscape that is likely to get harder and harder to opt out of.
“There’s a war on general purpose computing, because companies realize they gave up too much control,” said Schneier.