1. Purpose: The main purpose of the application for which it is built need to be defined.
2. Load Test goals: Identify what you want to achieve by load testing, what are your primary goals to go for load testing. Define all your goals. One example could be – login time for 100 concurrent users should be less than 2 seconds.
3. Architecture: The major concerns in the overall architecture about achieving these goals
4. Simulations: Think of all possible scenarios in which the end user will be working when application is put live. This could include some batch processing, some different client types etc.
5. Past experience: don’t forget your past experience with load testing solutions. The results, the shortcomings, bottlenecks – all could lead to a new learning.
6. Topology: Define all infrastructure components and the network which links them.
1. Framework: What is the framework being used in the application? For example it may be .Net or J2EE or any other.
2. Servers: What all server services you are using to run/ use this application? What standard server applications are used for Presentation, Application and Database level? Some examples could be MS SQL Server, Oracle, Apache, Websphere, IIS etc.
3. Number of machines: Are these servers running on same machine or different machines. If different machines how many machines and what servers specfically on which machine.
4. Details of these machines: The details of the machines running all these server applications or server services. Details would include Operating System, Hardware configuation – like Hard disks, RAM, processors etc.
5. Access details: Last but not the least important point is how the users will be accessing the application – via LAN, through dial-up, through internet, through some specific port and third party tool etc.
Which Server: Where is the load testing intended to be performed? Is it the test server, production server or staging server where load testing is to be performed? If it is being performed on Production Server, it is ok. Otherwise if it is to be performed on test or staging server, be careful that it is as near to production server in terms of configuration and setup as possible. It may give wrong projections if there is a wide gap in environment which is to be used in real versus the environment on which the test is performed.
How many Tiers: It is very important to understand how many tier application is, on which the load testing is to be performed. The n-tier count should include the client also.
What Browsers: Be very clear and specific on defining what all browsers are meant to be used for running this application (if this is a web application). The browser and its version is very important to conduct the load testing as each browser behavior, architecture and performance varies. Even between the different versions of the same browser.
What Protocols: Whether this is a client server application, web based application or n-tier application – identifying what protocols you are using to run the application is very important.
A love for music – if you can move to music, you can dance: Same applies to testing also. If you have love for testing, and you can move to its music, you can be a good tester.
A sense of rhythm: If you understand the product well and move along well, in a right rhythm, you win the game.
Willingness to give it a go: A true willingness is a must for a tester to be successful.
A bit of natural ability doesn’t go astray: Don’t forget that anybody can not be a good tester. There is a natural ability that gives you an extra edge than others to be a good tester. Only you need is to catch this ability in you and keep polishing it.
Work hard enough: Hours of hard work and dedication is a must to be a good tester.
The responsibility of a tester is to ensure the peace of mind of the end users who are going to use the software product. Another target should be to safeguard customer’s investment in the product. In order to discharge this responsibility, the tester should focus all his skills on understanding customer requirements and measures to map it with product built by writing appropriately comprehensive test cases, performing test and preparing a detailed self-explanatory test report. Complex business requirements will require extraordinary level of understanding. Keep in mind the budget and time constraint for performing testing without compromising with the coverage or quality.
The overall impact of testing should be clearly visible on the product built. The bugs reported should bring a sense of belonging with the developers and customer users. Clarity of bugs in testing report should not only help developer to fix defects properly but guide him not to repeat the same type of issues in future releases. This may happen slowly but gradually should help in increasing developer’s performance and quality of code writing. After all the development team should start taking testers as their best friends and well wishers instead of someone who is trying to find out the weakness in code writing.
People are not wrong, processes are. People in an organization do what they are told to do. Organizations who hit on people at the time of failure are not doing the right thing. It is the process that is to be blamed not the person with Q-tag. As long as quality is considered to be the child of people with Q-tag in the organization, and the culture is so around, the non Q-tag people will keep themselves enjoying building a product without quality. Why not blame the developers for developing so many bugs along with the requirements building. Why not blame the analysts who could not understand or translate the requirements correctly. Why not blame the management for not putting the right people and process at right place. This blame game itself is merely a scapegoat in any organization until the focus shifts from people to process enhancement.
Analyze the failure as a team involving all the stakeholders. Management, business analysts, project managers, project team, development team, testing team, implementation team have to sit together and brainstorm. Failure is to learn and not fail again. Mainly failure can occur due to following three factors:
Teams do not work and deliver in desired fashion – required innovation, awareness, training, cohesiveness, team building, and collaboration
Wrong people in the team – HR and management has to play a major role and look into the recruitment process.
Failed to build right product – needs training, demonstration, pairing, indentifying right people for right job.
In any software development and implementation company there is always a need of quality assurance and quality control people who own the responsibility of setting the right methodology and framework for development and implementation (QA), bugs identification and product approval (QC). Usually everyone in the organization has an inherent feeling that the quality is the responsibility of only these few persons belonging to this Q-department of the organization. Business analysts understand the customer and business requirement, hand it over to development team for building the product. Development team develops the product, and hands it over to QC team. QC team tests the product, finds out the bugs, and submits the report to development team. Development team fixes the bugs and returns the product to QC persons for re-testing and verification. After few exchanges between development and QC team, the product is declared as defect free and is released or launched for implementation.
If top management, development team, business analysts, implementation team and all other stakeholders think that quality is just the responsibility of only the persons belonging to quality department, they all are wrong. If Q-tag is limited to only a limited persons belonging to Quality department among all stakeholders, a product can never be built with great control on quality aspect.
Q-tag has to be on each of the stakeholder in a software project. When each and every person wears a Q-tag – the analysis, building, testing and implementation will be more justified. Otherwise there will always be a big question at the time of failure of a product build – that who is responsible?
As in a software project, the complete project lifecycle comprises of different phases. Similarly the performance testing lifecycle also comprises of various phases. Performance testing is usually, as the name suggests, is done to evaluate or examine the performance of the software product with the help of some tools and certain parameters. The performance testing may include components based on the requirements that include – load, volume, stress etc.
Five prominent phases of performance testing can be:
a) Test Plan
b) Test strategy
a) Load Modeling
c) Benchmarking criteria
a) Benchmark execution
b) Other executions
c) Reports generation
d) Reports Analysis
a) Reports reconciliation
b) Reports compilation
c) Final Test Report
d) Bottlenecks identification
a) Execution of recommendations
In my June 15 2009 post – “Do’s (+) and Don’ts (X) in Project Management” (https://itknowledgeexchange.techtarget.com/quality-assurance/do%E2%80%99s-plus-and-don%E2%80%99ts-x-in-project-management/), I got a query and am posting it here as it is – As mentioned by you in one of your post “Single constant in business is Change”. But often software vendors are too bound by the requirement documents that they fail to gauge the change in the underlying business need that they are trying to cater to. I understand that there are time and cost considerations, but many a times the attitude is not conducive. Could you throw some light on this in your writings…?
First of all let us be clear – we can not just record requirements and seal it. And then stay isolated from customer in building the software. One fine morning communicate to the customer that the product is ready and you are reaching customer site for product launch. If you feel that during the conceptualization, build and test phase customer has no role to play – you are wrong. If the customer thinks just by giving the requirements they will get a good product meeting all their expectations – customer is wrong. Both have to be in tune during the product development. If vendor fails to gauge the change in underlying business need that they are trying to cater to and think if they do so, it will increase their cost and time – again the vendor is wrong. By not doing so – they call for more time and cost. By not doing so – they surely call for more discrepancies, ambiguities, delays and failures. A small investment of time and cost during the build phase to involve customer and take his consent at every step will definitely lead to later on investments, time delays etc.
If attitude is not conducive, it has to be. Customer has to realize that. And customer has to realize the importance of getting involved in each phase of the project rather than assuming that it is vendor’s call and everything will go fine.
The requirements keep changing, as the business. Only thing that varies is the pace of change in requirements. Overlooking changes happening in the business process at customer end after freezing requirements will result only in undesirable product. Both customer and vendor have to understand this and keep overseeing the changes rather than overlooking.
If changes are not recorded and incorporated in time, customer is always going to blame the vendor and may be vendor gets exchanged with another vendor for the same project.
At the birth (inception) of a new software project the project manager is puzzled and confused just trying to gather and understand customer requirements. He starts like a wanderer in the dark islands of customer for collecting various requirements and understanding their business norms. The moment he is able to collect this information, he aggregates to get the stock of ‘total requirements’. Understanding this makes him getting into ‘catching the rhythm’. Now comes his planning phase where he has large ‘in depth’ discussion with his development teams. By identifying different milestones for various project stages, he prepares a project plan. At this stage he is supposed to discuss this plan with customer and get ‘in sync’ with him. Once approved from customer, he breaks this plan into different stages plans to hand over respective plans to their ‘stage owners’. Development plan goes to development manager. Quality plan goes to QC head. Implementation plan goes to implementation head (himself mostly). Accordingly each stage milestones are identified which might be more than overall project milestones shared with the customer.
Then starts the actual war phase – the development phase. Meetings, discussions, brainstorming, logs, monitoring are all the weapons of this war phase. A product gets birth during this phase which gets vetted by quality control team. Documentations are also integral part of this phase.
Various test phases occur during and post development or build phase. Smoke testing, Unit testing, module testing, performance testing, security testing, load testing, functional testing… to name a few.
Now the baby has started walking. So baby is dressed well to take it to the grand function taking place at customer site. The function is ‘Implementation’. This is a long function, comprising of baby show, meetings, discussions, demonstrations, recordings, UAT, training etc. Once the baby is able to walk neatly in front of all the guests at the function, all praises fall on baby. The ceremony closing takes place. Project manager adds one more bullet of ‘confidence’ in his gun.
And starts over again for a new project.
A software tester evaluates software based on certain parameters. These parameters are set as per product, customer and organization requirements. Testing could be just of functional features or include load, performance and security. For any parameters a tester has to work as quality analyst to understand requirements, features and accordingly build test cases and perform test. This is the quality control part. On quality assurance front the quality team has to build standards for requirement freezing, planning, development, implementation and post implementation phases of a project.
A software tester at various stages of a project gets on to the job of a Quality Analyst by performing following tasks:
Analysis of customer requirements: The first and foremost analysis required is that of the customer requirements to ascertain if it is complete, detailed and free from any confusions, ambiguities or equivocalness. Any flaw in requirements will certainly lead to a big disaster at a later stage. Unclear requirements are not difficult to build, but are difficult to manage. Every requirement should be in black and white. Each line should be very clearly documented such there should be nothing hidden between the lines.
Analysis of Product Functionality: Requirements documented and product built has to go hand in hand. It should not happen that requirements and product speak differently even a single line. Usually while testing functionality of a product, tester forgets to refer to requirements documented, or asks developer about the functionality. The developer will certainly explain him the functionality he has built not what exactly has been mentioned in the requirements document. If this happens, it will certainly cause a big blast at implementation or acceptance stage.
Analysis of Product related documents: There are many documents prepared during the project. Some are meant for internal use, some are prepared for customer. All these documents need to be inspected thoroughly and neatly.
Analysis of test results: Test cases are built to perform tests resulting in bugs report or test results report. A thorough scan is must to ensure complete coverage and thorough testing. The report should be detailed in all respects in terms of clarity and coverage.
Analysis of Testing Process: The testing process once establishes need to be revisited again and again to improve further at every go. Once established does not mean it is ultimate and best. Improvement has always a scope howsoever best your process or product is.
You, as a project manager are most crucial factor of a project being the key driver. Your commitment matters most as it can do any wonders. Your real commitment can put fire in your team member’s belly to run the project full swing. But if your commitment is merely Pretence, it can wipe off the energy and zeal of others in the team. It can trigger on the pace by acting as a catalyst or adversely trigger off the pace by decelerating it.
There have to be certain checkpoints in your thought process that may ascertain if your commitment is in right direction or not. Let us see what those points as mentioned below are (against each I am mentioning if the commitment is PRETENCE or REAL).
1. Believe – “All stages of a project are crucial and important as far as the overall success of a project is concerned.” – Your commitment is REAL.
2. Believe – “Each milestone matters and need to be converted to win-win for all for a smooth sailing towards this bigger goal.” – REAL
3. Believe – “All stakeholders have a stake and a part to play in the project. Overall every task and success has to cross ‘mine-yours’ barrier to reach at ‘our task’ and ‘our success’ stage.” – REAL
4. Are – “Intentionally creating conflicts.” – PRETENCE
5. Are – “Avoiding ‘important to be resolved’ conflicts.” – PRETENCE
6. Are – “Unnecessarily trying to prove your point (imposing your point) (putting yourself over the project). – PRETENCE
7. Have – “Lack of knowledge and understanding.” – PRETENCE
8. Are – “Giving more importance to emotions and feelings over logics.” – PRETENCE
9. Have – “High level of actions but poor planning.” – PRETENCE
10. High level of planning but with poor actions. – PRETENCE
11. Framework is not important – people, tools and management are. – PRETENCE
12. High performing teams are different from highly organized teams. – PRETENCE
13. Best practices do not require improvements. – PRETENCE
14. 100% test coverage means correctness. – PRETENCE
15. Automated testing is a silver bullet – PRETENCE
Projects do happen with the beliefs and actions mentioned above. Whether they end up in good or bad state, I am not sure. PMs do exist with these beliefs and actions. But they have to change themselves over a period of time, I am sure.
We learnt in earlier two posts about the strategic decision of a management to outsource a complete project or part(s) of a project depending on certain factors, and the factors respectively. In this post let us see at the various components of a project that are most widely outsourced or otherwise we can say these are the components of a project which can be outsourced. It is very less often that a project awarded to a company is totally outsourced.
We are talking about outsourcing an activity of a software project. The most important components of a software project can be listed as:
Requirement analysis, gathering and freezing
Design, development and testing
Implementation, training and hand-holding
Post implementation support
These can further be broken into various sub-components under each component thereby creating a tree like structure to have a bird’s eye view of any project. Planning and execution for each phase or component comes next with control everywhere.
Outsourcing mainly is the resultant of constraints in an organization.
Outsourcing is neither bad nor good. It is a strategic decision based on certain parameters taken by an organization to offload some projects or part of a project to a third party. Offloading certainly comes with many boons and many banes. Offloading should be managed very carefully as the project delivery ownership still remains with you even if you are offloading complete project. Insertion of seriousness in the outsourced vendor regarding the project, requirements and its timelines is very important. There are many parts of a project that can be outsourced and as said above, the decision is purely management’s to decide what part of project is to be outsourced and to be awarded to whom.
Definitely in offloading the vendor-customer chain becomes longer depending on number of components of a project outsourced and the number of vendors involved. Even for a single component of a project there can be more than one vendor. And for a number of components of a project there can be a single vendor.
Offloading or not will depend on many factors that can be listed below as:
1. High number of projects
2. Shortage of skilled manpower
3. Shortage of right skilled people
4. High engagement of required persons
5. Lack of resources (in case of testing tools or otherwise)
As already has been discussed, User Manuals play a crucial role in any software project and are the solid bonding between product, users, customer management and vendor project team. The stronger this bond is the more comfortable and happy each stakeholder will be. Each player has to play a crucial role during the game of building of User Manuals. Both Project Management teams have to work hand in hand for that.
The quality of User Manuals has also been discussed in my earlier blog (Six Pack User Manual – how to build).
Let us discuss below the role to be played by the respective project managers in this regard:
Vendor Project Manager has to ensure that the user manuals are built along with the product development and are vetted/ approved by their QC team via a walkthrough. If need be they should as for draft manuals in build phase. If there is a foreign language issue, the Project Manager should also ensure that the user manuals are required in their local country language.
Customer Project Manager has to check that user manuals are complete in all respects, and checked thoroughly before handing it over to the respective key users
Six major key points that should be kept in mind while preparing User Manuals can be as listed below:
1. Simple – Never expect your product users to be as knowledgeable about your product as you are. Remember when you first time tried working on this product, how ignorant or untrained you were. The expertise that you have on your product has come over a period of time on repeated usage (in terms of implementation and training) of this product. So keep your User Manuals as simple as possible. Don’t use technical jargons, or tough language clauses. And even if you have to use them, break them in fractions or explain them in detail.
2. Extensive – Prepare User Manuals as detailed as possible. Use less text and more visuals/ diagrams/ flow charts/ process charts/ drawings/ illustrations/ screenshots/ pointers, highlights, captions, bold/ caps and italics to make is more user friendly, more understanding and less cumbersome.
3. Crisp and Fluent – On the other hand the User Manuals that you are preparing for your product users have to be crisp, up-to-the-mark, relevant, in-tune with the product, devoid of useless stories.
4. Quality – Imagine an airplane handed over to a person who doesn’t know ABC of even kite flying keep aside an airplane. And with the help of a user manual if he is able to take off and landing, the whole credit goes the User Manual. Build your user manuals like this.
5. FAQ – This is an important section of your User Manuals simply explaining two things. First is what – if situation that means what will happen if you do a particular activity. And second is if – what that means if you want to perform a particular task, what you are supposed to do. Understand the difference between what – if and if- what and build your FAQ section accordingly.
6. Right Person – Don’t imagine. Not at least that a good actor will be a good story writer or script writer. These are different jobs. Let developer do the development and chose the right person for understanding the product correctly and design User Manual.
The product owners or stakeholders might be many in a software project, but the real frontrunners who drive, run and use the software product post implementation are the key users and other users. It is their feedback that matters most. They must be the most comfortable lot on usability, functionality, reliability, stability, durability, resultant outputs in terms of reports and analytics, feel and look of the product. They are the one who are going to matter most in the success or failure of a product during and post implementation. The best tool to give them comfort, satisfaction, confidence and support is the User Manuals that they refer to most of the time during post implementation live run.
Even if the product is excellent and trainings are most rigorous ones, in absence of the project implementation team, every now and then the key users will be seeking help from User Manuals. User Manuals are the supporting agents for them at all times. That is why User Manuals have to be perfect in all aspects.
Infact a user manual prepared for product users should be such treated as a replacement of implementation and training team right from the moment the project is signed off and users start using the product in real business scenario.
+ Single constant in business is Change
X Single constant in business is resistance to change
+ Change means shifting to a better comfort zone
X Change means shifting away from comfort zone
+ Recruit well in advance before the project start time
X Recruit at the time of start of project
+ Give to new recruits the ample time and guidance to charge them well
X No time for relaxation should be given to employee
+ Relaxation re-energizes, recharges the employee
X Relaxation destroys the enthusiasm and fire of an employee
+ Retention of good employees is worth spending in case of no projects
X No Projects No Retention
+ Importance, Encouragement, Attention, Satisfaction, Opportunity to grow all these
are important for employee and it pays back to an organization
X Organization pays, Employees work
+ It is better to monitor new employee to check its readiness for appropriate job
X Recruit and assign directly to project
+ Shortcut is the longest route to achieve success
X Omit steps, adopt shortcuts and attain results
10. When you start getting increased number of bugs as compared to earlier releases
9. When your developers stop thinking about quality in code
8. When bugs pass through QC unnoticed
7. When you stop acknowledging quality efforts and start giving more importance to speed and volume of writing a code
6. When quality and development teams are not in sync
5. When you start feeling QC is a waste of money, efforts and time
4. When quality aspect is given least time in overall project
3. When management starts overlooking then overseeing
2. When customer does not get grossly involved in performing UAT
1. When above mentioned in point numbers 2 to 10 starts spreading in other projects
In the previous post we learnt what all could a software project could lead to in presence of higher scope of software development at customer site during implementation phase. Let us see what all factors are responsible of insufficient requirement gathering during business study phase of a project.
1. Incompetent team (vendor): Any project calls for a project team. A good team at both ends is important. Both teams have a substantial role during a project. If key users selected at customer site for delivering process knowledge to requirement analysts (vendor), there are higher chances of wrong information driven. The key users chosen should rightly be the actual process owners not essentially be at the higher level in the organization.
2. Incompetent team (customer): A right selection of project manager and analysts is equally crucial. The team has a short duration stipulated for gathering requirement information and understanding business processes. And mind it, this is not an individual race, it is a relay race, and weakest link will decide the overall result of the project.
3. Non commitment (customer): Well the process owners are the key users, but if are not serious or have not received the right message from their management may not be seriously committed to the vendor team. A non committed team will not be able to impart right knowledge, processes, practices and information required critically for a project.
4. Non commitment (vendor): A team of good committed project manager and analysts is equally crucial for gathering information. The previous experience on business domain and success factor definitely counts in.
5. Lack of time (customer): Key users identified if are engaged in other important projects running in the organization will be always short of time for providing right information to vendor project team.
6. Inappropriate discussions: In a limited timeframe of business study, there will be limited discussions. So all discussions should be crisp, purposeful, well driven and result oriented.
7. Improper documentation: Well explained requirements if not documented properly may lead to a wrong, incomplete product. Right people with right business and process knowledge have to acquire a good documentation skill also to lead the project to right direction and grand success.
8. Top Management involvement (customer): If customer management thinks that providing key users completes their job in the project, they are wrong. Their involvement in all important meetings is as important as the business is.
9. Project Organization: A right architecture of project team is very important. The project sponsor, project directors, project managers, key users, developers, implementers, technical leads from customer and vendor respectively should be the right mix of people.
10. Change in requirements: Any change in business or process during or after the requirement study phase has to be communicated to vendor team well in time, so that by following the change management procedure, the changes are taken care of properly.
Ideally, in a software project, for an offshore customer, the requirement gathering phase should be given an extra care to understand to a maximum extent so that the product developed and tested when ready for implementation at customer site requires no development. Practically, it is very difficult to achieve. The more it is left open to be handled at customer site, the higher is the chance of customer and vendor getting affected.
So that means the higher scope of development at customer site will change the team requirement and affect the overall project. Let us see what major factors will get affected and where could those lead the project to:
5. Money Factor: The most important factor is money. Any scope of development at customer site will need extra developers in the team besides the implementation functional leads. This will lead to an extra burden in terms of cost of technical staff not only in terms of developers but also testers.
4. Time Factor: Delay in project is inevitable in these circumstances when
requirements will be clearer at customer site at the time of implementation and
thus will change the overall implementation plan to accommodate development
and testing in between.
3. Quality Factor: Poor quality will be a major concern. Everyone knows you can not
take a bundle of testers at implementation site. That means limited testers will be
in under a tremendous pressure to release the product at the earliest and hence
may not justify with their job. This may lead to lot of holes in the pot –
intentionally left or unintentionally skipped.
2. Exuberance Factor: Enthusiasm, tempo, momentum, pace will all start
diminishing and in turn start creating frustration and dissatisfaction at both teams
1. Product Factor: Product Failure, if not, then delays and unpolished product is
always visible on the platter under these circumstances.
All these factors, being inter-related will have recursive effect on each other thereby increasing each by manifold and everything may turn into a complete mess in place of a successful project.
The Post implementation review is usually filled collectively by customer project team and signed by the project sponsor, who is usually one of the senior members of management. This form is to be submitted back to vendor (after filling) to serve the purpose of – sharing experience, product evaluation in live scenario, user’s satisfaction, assessment of management itself in understanding the goals assumed or perceived versus the actual goals achieved. The components of this review and WHAT, WHY and WHEN have already been discussed in earlier posts.
Let us now understand – what are the guidelines for management (customer), project team and key users for filling in the Post Implementation Review:
This is not a sports match: The project should not be taken as a sport match where one has to be declared winner and the other loser. In a software project either both teams win or lose. So intention of the review should be taken in right spirit and with right approach.
Be Courteous: Don’t be too blunt while filling in the replies. It could hurt the sentiments of vendor. After all if something has gone wrong, it needs to be analysed and resolved rather than throwing stones at each other. Remember that someone had made a choice at your end for this product and that is why it is here. On the other hand if all has gone well, be generous in praising.
Don’t be a critic: Don’t analyse this product with a critic’s eye. Your goal while selecting this product was that you had a trust in the vendor and this product. Evaluate more on the basis of your key user’s confidence, satisfaction, results, improvements, and product strengths.
Be Transparent: It is good to be courteous, non-critical and not-blunt. But don’t shy away in exposing any weaknesses in the product, the methodology adopted, trainings imparted, or vendor’s team efficiencies. Strike the right ball in right hole at right time. After all it is learning for both the vendor and customer that is helping in product and people maturity.
Ask for more: If you are happy with the vendor, don’t hesitate in asking for more features or functionality (may be as a bonus, if you are able to pamper the vendor enough!)
Both vendor and customer have to understand that the real journey of a product starts only after a successful implementation of software at customer site. The challenges get a new meaning once the project team leaves the customer site after project sign-off. Both the managements can celebrate the successful completion of project. But it is better to hold on this celebrations for a while, till customer really learns to live with the product, starts tasting the real fruits of the product committed earlier. Let the key users start managing the show on their own, run the complete process in live scenario and get satisfied with themselves and the product. The project team that comes at customer site for implementation has a role related to post implementation review prior they go back home after successful closure of project. Similarly the management (customer) have to clearly understand their critical role once the vendor project team goes back after the completion of project. It is the key users and management (customer) now to run and manage the show. No doubt that the vendor team is always at the back to support and assist them in achieving this. Let us assume that the project is over and team is about to leave the customer site. At this juncture it is the prime responsibility of vendor and customer project managers to shake hands and understand the importance of post implementation review in following terms:
The Project manager from vendor side has to hand over the post implementation review plan to customer at the time of implementation sign off and explain clearly each and every agenda of the post implementation review. The clarity in terms of product, managements, users, usage and product is very important for assessment purposes after a stipulated period.
The project manager from customer end has to ensure to get the post implementation plan for feedback purposes at the time of implementation sign off. This will not only leave the project open for another some time in terms of providing a lease for customer team to assess and experience the product in complete isolation in their own terms.
The purpose ultimately is to share the overall health of product, the key users, and the management.
What is post implementation review? When it should be done? Why is it required? All this has been discussed in last three posts. Let us now understand what ideally would be the components of a post implementation review. As discussed earlier, some components can be answered immediately at the completion of a project with a formal closure. But for answering some other components, customer needs some time for project to run in the absence of project team, experience it, feel it, see the user’s reaction on the way things are happening in the product when it is in use.
Infact before filling the post implementation review, it is advisable to use the software at its maxima, as extensively as possible, involving all key user’s areas, using all inflow and outflow procedures. The basic guidelines required while filling a post implementation review can be take up in my next post. Let us also understand that this questionnaire has to be quite elaborative and descriptive. Let us first understand the essential components of Post Implementation Review for a software project. The questions can be built as per the need of the vendor and product. The essential components to be covered are:
1. Effectiveness of the Project Team
2. Effectiveness of Customer management in managing Product understanding and implementation
3. Effectiveness of Customer management in understanding the requirements, building the product, selecting the right team and procedure
4. Change management during the complete cycle
5. Issues management
6. Preparedness of key users and management for accepting the product
7. Communication skills and management of vendor team and management
8. Risk perception, risk management
9. Product effectiveness, usefulness, maturity and friendliness
10. Customer management eagerness for future business to the same vendor
These are the core components based on which the elaborative questionnaire can be build to assess customer satisfaction over the product, vendor and team.
When is to perform a post implementation review? A witty answer could be – obviously after implementation. Ha! Definitely a successful closure of implementation could declare a project closure with a formal project acceptance report or project sign off. So shall we have a post implementation review as soon as we have a project sign off? Nah! That would not solve the purpose. Give an appropriate time to the customer team key users to settle down as the captain of the ship and sail it smoothly. One day or one week smooth sailing will not tell you the turmoil or undercurrent storm about to come in future. Correct. Then future is too long. That means keep waiting for turmoil. But mind it, all ships sailing in the sea do not experience storm. Similarly all products after implementation and project sign off do not guarantee a serious disaster.
Product performance in actual sense requires a certain timeframe to establish and to give confidence to end users. Some part of post implementation review related to team performance (implementation) can be answered quickly, maybe immediately after the project closure. But the other part needs a considerable amount of time to understand the product from different perspectives and accordingly present a right picture in the review report.
Certainly, then, atleast a period of minimum three months is required to experience the product and then fill in the post implementation report. Ideally, I would say, wait for six months, use product in all respects, aggressively, and then the top management need to sit with their key users and project board to evaluate, assess and fill the post implementation review.
A successful implementation does not ensure the completion of project. The reality starts when the implementation team has gone back and users have started using the project in full swing with the help of training material, learning, knowledge and product. The health of users in respect of using the product is sustained, deteriorated, or improved will depend on many factors. A post implementation review is always important to understand the users understanding, pains and delights during this tenure. This will translate further into management’s pains and delights. The overall delight weightage has to be higher than the pain. In a blow of project implementation phase users might feel quite confident regarding the product features, functionality and ease. When the whole thing falls upon them, it usually drive them in confusion, wrong actions or stoppages. Or a smooth drive.
A post implementation survey will help in a real measurement on two fronts. One front will be users’ understanding, ease and comfort (or vice versa). Second front will be product’s stability, performance and functionality. It also will assess the after effects of a successful project implementation.
The conclusions could be misleading although and will require a deep analysis. A user’s lack of understanding may spell into products inefficiency or the opposite of it.
A Post implementation review is conducted after a substantial period from implementation sign-off. This review is to ascertain customer management’s and users’ experience on product in absence of product implementation team. The product has been implemented successfully and the team is gone. Ofcourse sufficient learning, knowledge, exposure and material have been imparted to users by the implementation team.
Post implementation review by the customer management with key users will spell out – user’s depth in using the product and product’s reliability and stability. The format of post implementation review document should be designed by the vendor management for the purpose of understanding the status of project after a certain period of usage post successful implementation. Filled by customer management in agreement with key users’ experience, the report is sent to vendor for their assessment.
As stated in previous two blogs, top level expectations gathering is very crucial during the business study and requirement gathering phase. And respectively I mentioned how vendor and customer can be careful (and should be) about that. Although it is rare and unexpected, but there are instances where customer organization top level management may not involve in a new software development project. This could have various reasons but could lead to only a single road – where the end is DISASTER. If this so happens, the chances of project getting hurt are manifold. It will precisely and ultimately lead to project failure. The reasons could be many, but to my mind following come at the top:
1. Customer Top management presumes that their involvement requirement is only for signing agreements, papers, reviews and sign-offs. If that so, they will have almost negligible awareness that how critical it is for them to get their expectations and requirements (top level) during requirement study phase. After all it is an investment of time, resources and money. Instead of getting a jolt at a final stage about getting the unexpected or not getting the expected, it is better to explain right in the beginning their own requirements and expectations from the product in detail.
2. Vendor’s management role is very crucial during this phase. They have to take initiative in explaining the top management about the benefits of the product being proposed. How it is going to enhance the processes in the organization, the change in roles etc.
3. Assume a situation where there is a pressure from user level for a solution or a product at customer end. The management is not sure about the product. Without horizontally analyzing various solutions available, they decide to go for a particular vendor or product just for the sake for fulfilling requirement. Although it will be rare, but even it is there at a very low intensity, it need to be addressed.
4. Customer top level assumes that the solution being invested into is meant only for user level and not organization level. This is a wrong assumption. Any investment – small or large- has to have benefits for the organization and top management on the cards.
5. Customer top management overestimates their users and assume that they will be able to drive the project without top management’s involvement. This does happen but very rarely, in very organized structures.
Usually it is the customer top level person who is project sponsor for a software development project, be it in-house or from a software development vendor. A Project Sponsor may presume that his/her roles during the project would be – sign agreement and papers, assign roles down the line for the project, monitor/review project at various stages etc. A critical and crucial role is overlooked that of getting into the project especially during atleast vendor’s business study and requirement gathering stage. It is not only important for project sponsor and directors to be part of this stage but equally important is to involve all top management in that.
In brief, Customer Top management has to fully understand the benefits being proposed by the vendor that will be produced by the software when it is in use. They should also participate fully in business study and requirement gathering meetings to define and freeze their own expectations from the project/product/software/vendor.
The most critical stage in software project lifecycle is business study and requirement gathering. Vendor has to be very cautious and careful in understanding all levels expectations from the product they are going to build for the customer. Skipping top level at this stage could be disastrous for both. As a vendor, if you don’t involve customer top management while gathering requirements – you are inviting a mishap!
Customer top Management involvement is very critical during the business study and requirement gathering phase of a software project. The expectations of top management shall invariably be different as compared to other stakeholders of the software project at customer end. Assuming that the requirements gathering from process owners or end users will be sufficient for developing software will be a misconception. A detailed discussion for capturing requirement and understanding top management perception is critically important to lead to a successful completion of the project.
At the Vendor end – the Project Manager has to ensure that besides capturing user level requirements, it is essential to highlight the benefits to the top management being proposed for them from the product. It is not desirable but mandatory to freeze top level expectations at business study and requirement gathering stage.
Scope defined and decided upon initially by vendor and customer mutually has a large impact on timeline, progress and success of a project. A change in scope at a later stage may call for a big impact on project schedule and progress. Let us see the roles of vendor and customer respectively in this aspect of project.
Vendor – Project Manager has to ensure that the scope of project defined in the Business Study Document has to be adhered to. Any change in the scope (increase or decrease) has to be escalated to both the managements and the project plan has to be re-designed thereupon.
Customer – The management and project manager has to ensure that they define the Right scope for the project (number of locations, pilot site etc.), also understand that any change in scope will adversely effect the project plan and hence seeks redefinition. The pilot site chosen should be the best possible site in terms of testing the software completely and rigorously.
Sign-off at various stages has a significant importance during project lifecycle. Everyday sign-off can be a headache for customer, no sign-off can cause headache for vendor, so there has to be a balance of sign-offs of milestones, achievements and stages of project so that the sanctity of sign-offs is retained thereby earmarking the progress of project. Both, vendor and customer have to be very careful in this aspect of project management as it is crucial for all stakeholders.
At Vendor side the Project Manager has to educate the customer management and project manager over the benefits of timely sign offs. Sign-off should not be there just for the sake of it. It should add substantial value to the project.
At Customer end the Project Manager and management should respect the timely sign-off practice but also ensuring that the activity which is being signed off has finished in actual, rightfully and rightly.
Change management is a subset of project management. In any software project change is to be managed during the whole cycle of development and implementation. Requirements once specified by customer at the business requirements study phase does not mean that there will be no change in requirements later. Vendor who is not open to mange or cater to this change in requirements will not be able to deliver the product to customer upto his satisfaction. To mange these changes Change Management is essential and both have to play their respective roles in managing the project. Changes could be in terms of specifications, process being re-defined, or change in business rules. The two aspects for change management are – vendor and customer.
At Vendor level the Project Manager should accept changes only that are business critical and not cosmetic or wishful in nature. He should redefine the project/ development/ implementation plan in terms of time and revenues in consultation with his management taking customer management in confidence. He should incorporate changes only after getting it approved by customer top management.
The Project manager has all rights to challenge any change in requirements that is fanciful, not business critical and is impacting the software largely in terms of efforts and time.
At Customer level the Management and Project team have to understand the impact of any small change on the software thereby asking for a change only if it is a business critical change, analysing if the change can be avoided, and understanding the time and cost effect of the change.
Although it can not be avoided in real life scenario and that is why there is a risk plan and risk management to avoid such circumstances. But still a lot can be done to atleast minimize the risk and thereby mitigation.
Vendor – Management has to ensure the bare minimum changes (preferably NO CHANGES) in terms of project manager and team members during the project lifecycle.
They also have to ensure that the project is handled in such a manner that any change(s) happening therein will not affect or delay any of the stage.
Customer – Management has to ensure the bare minimum changes (preferably NO CHANGES) in terms of project manager and key users during the project lifecycle.
They also have to ensure to adhere to the guidelines specified by vendor project manager to manage such change(s) so that it does not affect or delay the project during any stage of the project.
Project initiation starts with the marriage of vendor and customer to produce the desired software product. The desire comes from customer, with a promise from vendor to deliver the desired product. The time and investment (in terms of people, equipments, and expenses) calls for a superior quality management so that the product can be built as per requirements and in time. Project can be managed well with the best possible project management practices.
There can be two scenarios – the customer is already having best practices in place and in that case asks vendor to follow those practices as these practices have already produced desired results in past. That means the practices are well established at customer end. The other scenario can be that customer is not aware of any best practices and looks at vendor to follow as per their standards set for other projects. In this case the vendor has not only to bring those practices in place at customer’s place but also primarily educate customer about them.
That means the role of Vendor Project Manager will be to design the best practices suited for the project and educate the customer management and project manager to adhere to those practices. He also has to highlight the benefits that will be coming out of those practices.
Customer Project Manager and the management, on the other hand, have to understand the practices designed for the project and ensure their adherence at all levels.
Project management is a joint effort of vendor and customer teams. Project Organization members have to play their respective roles timely and religiously to get the best of the results. Both have to go hand in hand right from the start of the project till end and even beyond. The relationship does not end with the successful completion of the project. Rather a new journey starts with the project sign-off. The baby borne by the vendor team with the help customer team changes the hands with the project sign-off. If these responsibilities are not well understood well in advance, it may lead to overrun and may end to the total failure. To avoid a project overrun the vendor and the customer have to trigger the alarm well in advance as soon as they sense a sign of overrun arising out of any reason.
At Vendor end the core responsibility of project manager is to train the customer project manager in project management so that customer project manager takes the lead in project and ensures that there is no overrun in terms of time and revenues.
At Customer end the customer Project Manager has to be pro-active in his approach to escalate the matter to his top management in case he feels in advance that project is going to overrun (with reasons identified and agreed upon mutually). Some suitable metrics can be used as project plan to trace the progress of the project in accordance with the project plan.
Project overrun is not unknown for any software organization. Every organization who has tasted it would acknowledge that it is always painful – in terms of time, cost and resources. Ultimately when the other two factors are also converted to cost, it comes to a voluminous figure. Nobody wants it, but in many projects despite all good efforts it becomes unavoidable and then some valid reasons are listed to justify it. Usually as the project overrun starts – simultaneously starts a hidden tug of war between the vendor and the customer.
Customer gives a different set of reasons all pointing towards the vendor to prove that it is the vendor who is responsible. Vendor on the other hand tries to justify that there were n number of shortcomings at customer end that caused it.
And that becomes a major cause of more overruns… sometimes endless…
Instead even at this juncture, when an overrun starts, if both the parties sit together and find out the root cause instead of finding out reasons to blame each other, it would be helpful in stopping further delays.
A classic scenario happened in an organization recently as told to me by a project manager of that organization engaged in software development and implementations.
It is related to project overrun.
A new project started with a set of requirements from a customer for development and implementation. It was an overseas project so project cost was comparatively higher than the domestic project. The respective teams for business requirements, development, configuration and implementation were formed. All went well till the implementation phase. The implementation team was ready to take the charge for on-site visit with the product to launch there.
The implementation phase planned was 4 months. Somehow due to a mix of reasons, it took 18 months to complete the implementation.
The team came back after successful implementation. The customer paid the full project cost as was accepted upon in the beginning.
The project was declared as successful without any overrun. For overrun cost was taken the criteria and since full cost was recovered, it was treated as not overrun project.
Is that right?
Throwing some points to ponder upon:
The cost that had to come 14 months back came now.
The team that has to return 14 months back arrived now.
In this period of 14 months atleast 3.5 projects of similar nature and size would have been completed.
The project manager is over-optimistic.
Project monitoring was very poor.
Project overrun is simply a project crossing its boundaries set by the organization. These boundaries may vary from organization to organization depending on how they blindly or how over-extensively (both extremes) they want to look at it.
5 ways to control project overrun could be:
2. Customer engagement: At customer site (or earlier as and when customer involvement is required) if customer project team is not justifiably involved in project by means of specifying requirements, providing master inputs, in training, timely sign-offs at various stages, hands-on exposures, etc. effects project drastically and plan may go haywired, without anybody’s accountability to prove, if alarm is not raised well in time.
3. Milestones: If appropriate milestones are not identified and monitored at every stage of the project, it affects the project finish off in time.
4. Management involvement: If management let the project go off without their involvement in it, it has high chances to overrun.
5. Celebrations: No celebrations of achievements during the project can decelerate the tempo and momentum of the team at both ends to finish off the project in time.
All projects are prone to overrun. An overrun acceptance is directly proportional to an organization’s fault absorption capacity. Accordingly the definition of overrun is framed to demonstrate an overrun project as rightly completed project.
5 myths about Project Overrun could be:
4. Manpower: Project Plan is made after which additional manpower is inducted in the project, but no need to revise the plan.
3. Cost: Customer is ready to pay the full payment to complete the project, even if it overshoots the timeframe decided as per plan.
2. Time: A project had to complete in 5 months, but it took 10 months to complete. Imagine the manpower engaged in this project that could have finished another project if this project finished in time.
1. Customer: Customer is not able to cope up with plan but not ready to pay for extra efforts being done by the project team on behalf of customer thereby overshooting cost and time. We have a valid reason for this overshoot.
The customer project manager has to take the lead and ownership of product as soon as it is launched at customer site for implementation. The UAT, training and implementation process can only be effective in case customer project team gets fully involved into each and every activity of the implementation phase. Infact the implementation plan prepared by vendor project manager should be the responsibility of customer project manager to execute.
Customer Project manager and management has to clearly understand the risks involved during the business study, implementation and post implementation phase as highlighted by the vendor Project Manager and to act thereupon to overcome those risks with suggestions from vendor project manager.
These risks could be in terms of consequences involved:
the involvement of users and process owners during business study, implementation, UAT, masters creation, transaction entry, reconciliation etc.,
if sign-offs not happening in time, etc.
Even if the sign-off is given and product is not put in use, there is a chance of project failure at both ends.
He has to make the customer management totally convinced that it is their product which they have to use in real life scenario and hence they have to be fully confident in that. The vendor management has to be with him all the time for this.
Project managers (the customer end and the vendor end) have to work hand in hand during the implementation stage of a software project happening at customer site. The key responsibility of both the project managers working on the project is to ensure successful implementation and project closure.
It is the customer project manager who has to take the charge, be in full lead and command to drive the project and keep the responsibility and onus to him and his management. The implementation plan has to be his baby all the time. Unless and until he takes the lead, he will not be able to put the potential fire in his team to accept the product. And then only he will be able to ensure a fully hassle free implementation.
No project manager can claim there was not a single problem in any of his projects. But then he is to tackle them. To tackle them he should be aware of them. To be aware of them, he has to have an ability to foresee them than to overlook them. The earlier he envisages those problems, the more time he will get to dig into and find out a countermeasure. No software project is problem free, but then… the project manager has to be alert and have a perfect vision to oversee them, accept them, plan accordingly and win over each of the issues.
Problems will definitely be there, in all phases of project lifecycle. It is the vision that you envisage them in advance and prepare yourself to overcome them.
The deficiencies do not get so highlighted during in-house development and testing phase of the product. But it becomes prominent any problem that arises at customer site during the implementation phase, or say post implementation period.
It is not only the project manager but all stakeholders who get affected by the project over-run or failure. It could happen due to any reasons. One of the major reasons that have emerged is the lack of vision of the project manager, project sponsors, project directors and other stakeholders to foresee the problems faced by customer during implementation or post implementation while using the product.
The product may have been developed well, tested well and built well to launch, but what happens if some soft issues that may arise during implementation or even post implementation period are overlooked. It could lead to a disaster…
A risk is a bigger than its size if it is not identified well in advance. An identified risk is as risky as unidentified if its assessment is not done. Risk assessment is useless if there is no impact analysis. Impact analysis has no worth if its countermeasure is not identified.
Let us understand the different stages of risk in software application development and testing phase:
2. Risk identification
3. Risk Assessment
4. Risk Analysis
5. Impact Analysis
6. Risk Classification
7. Risk Plan
8. Risk Plan Analysis
9. Risk Plan Execution
10. Risk Closure
Similar to SDLC (software development lifecycle management), there is RLC or Risk lifecycle management in a software application in which there are different stages involved. The different stages could be risk identification, risk assessment, impact analysis, countermeasure identification, countermeasure assessment, risk plan etc. There are certain facts about Risk:
2. All risks have an impact: All risks have an impact – large, medium or small, but they have. It is the impact that makes its severity high, medium or low and accordingly a plan is prepared to handle the risk, when it happens.
3. Same risk in different circumstances will have different impact: The same risk will vary in terms of its severity under different circumstances of usage, user base, geographic location, type of application etc.
4. No application is 100% risk free, whatsoever countermeasures are taken for it, and only thing that gets done with the countermeasures is lowering of risk: A risk plan to countermeasure a risk never fool-proofs a risk’s impact, only it helps in lowering its impact to a certain level.
5. Risk Impact Cost vs. Countermeasure cost: It is very important to have an analysis of both before deciding on the plan. Some risk may be very severe but its countermeasure cost could be unaffordable.
6. The biggest risk in any application is identification of wrong risks, impact, and plan: Identification of wrong risk with right estimation of impact and countermeasure is useless. Equally useless is identification of right risk with wrong impact analysis (thereby underestimating or overestimating the impact) and arriving at a wrong countermeasure. Right risk identification with right impact analysis but with wrong countermeasure also is a waste of efforts.
Any activity is never without risk involved in it. Risk could be classified in different categories like – low, medium or high depending on its impact, software’s requirements and purpose, software usage, and software user volume. Accordingly the risks are identified or rather perceived. Their impact is measured or assessed, and based in the category in which it falls into – its countermeasures are designed or defined. The right identification of risk is as important as its classification or category.
A classic example could be a bank application being used by all its customers for their account maintenance, for transactions and for various other purposes. The risks involved in this sort of application could be: availability of application to all its users all the time, the speed of the application, the security of transactions, the ease and comfort of usage, the user account vulnerability of hacking and so on.
Some applications required all time availability whereas other demand high performance and high availability at peak time. Say, for example there is a website of a university where the peak usage is only at the time of admission or enrollment, at the time of fee payments, and at the time of release of results. At these times the volume of this site usage will be extremely high. So not only it demands availability of application at critical or peak times but also to be ensured is that it does not crash down due to high volume of use.
So it is very important to identify the right risks, to understand the right impact size, and to derive at a right countermeasure.
Once upon a time there was a programmer, a very good programmer. Good here means skilled, learned, experienced, and serious. He was appointed by an organization for a large project for a job to write quality programs. He was doing it well and was able to prove his point that he is good for the purpose he has been appointed for.
All of a sudden the large project was required to be closed. Now, the good programmer had to be absorbed in some other project, as company was in favor or retaining him. He was moved to another project requiring different platform development skills. The development platform is different than the one on which he was working in the previous project. He was master of this platform also and assured the management that he is fit for this new requirement and would be able to deliver well as he was (in his earlier project).
But somehow, his new project manager started feeling that – he needs lot of spoon feeding, he is not very productive and is quite slow in delivery, he is not self driven, he is not enthusiastic enough and does not report well.
Would someone like to share – who needs to do what in this case?
Let me start with the classic story –
This refers to the team of a Project Manager. The team size may vary from project to project and organization to organization, but the story remains the same. Story is quite short and interesting. A Project manager assigns different set of tasks to his team on 5 members individually. Each member has to start the work on Monday morning and finish it off by Friday evening. One member finishes her all assigned tasks on Thursday evening (without compromising with the quality of work), goes to her project manager, reports about the completion of task and requests for an off on Friday with a genuine reason. The project manager refuses although he admits that the work is complete, it is quality work, and the reason for seeking off is also genuine. The reasons for not sanctioning her the day off given by him were – it will spoil the culture and discipline and it may lead to non-quality work production. He was more into favor of her sitting with other team members and helping them in finishing off their individual tasks.
Well said, but this is only one side of the coin. Although project manager trusts all team members but out of fear he is not ready to do a favor to one member as it may spell out wrong signals for others.
But has the project manager understood that there are always HARE and TORTOISE in a team. All have just one responsibility – finish their task in assigned timeframe and produce quality work. It is not HARE’s responsibility to help others. If HARE is able to finish off her tasks earlier than stipulated time, it is a credit and she deserves a reward for that. And above all what about her trust getting hurt and she getting demotivated by not getting a reasonable favor.
I am sure, the reader would have their individual opinion on this – would love to hear!!